Jump to content
barrygordon

Rebuilding an array

7 posts in this topic Last Reply

Recommended Posts

I just removed this topic and reposted my original post as It was getting a little out of hand.  Let me start by saying I am new to unraid but have been in the software hardware field probably longer than most of this community has been alive. I started in 1960 and retired in 2005. I was a part time adjunct Prof Teaching computer Science at the Graduate level, and when I was in grad school myself I was privledged to have Richard Hamming (of Hamming code fame) as one of my instructors.  I well remember his lecture on data protection by using "Parity" versus using the concept of Hamming codes and Hamming space.

 

So lets start again.

 

Lets say you run a partity check, the one that does not adjust the parity bit to correct the parity; how do you go about rebuilding the real bad drive assuming the problem was not on the parity disk?

 

The Parity check can not tell you which disk had the parity error so I guess that is the only solution, to correct the parity disk. What else can you do? That is why in a time long ago (in a galaxy far away) we used a two dimensional parity check so that the location of the failed bit could be discovered.

 

I assume the only way to correct the "failed disk" is to read the syslogs. If there is something there pointing to the failed drive, then try and ascertain which file on that drive has to be relaoded.  I don't see any way of auto rebuilding the failed drive if it is a matter of a couple of bytes of corruption, yet I constantly see words in the posts re "rebuilding the drive".  I am a little confused (my normal state of mind these days) but I am sure community members will assist in bringing me clarity.

 

If you know the failed drive and assume that all parity errors occurred on that drive, you can zero out all its bits (clear it) and then rebuild it so the total array has correct parity. That is, you recompute parity treating the parity drive as a data drive and the cleared drive as the parity drive for the purposes of the computation.

 

I completely understand the concept of parity and how to use it to detect a single bit error in a bit stream.  However you can not correct that single bit error without additional information.  In unRaid's case you do not know on which drive (parity or data) the single bit error occurred you only know one has.  That is why in another post I brought up the concept of "two dimensional parity" which allows for the correction of a large number of errors syndromes, but not all.

 

How does unRaid go about rebuilding a failed non-parity disk, if there is a single bit error on other than the parity drive? In essence how does it know which drive to rebuild? If there is a double bit error in the same "row" of the parity computation, it will go undetected.  By "row" I mean all the corresponding bits from each of the drives that make up the corresponding bit on the parity drive. You just don't know which bit to correct, i.e. on which drive it is. That is probably why the parity bit is adjusted as there is nothing else you can really do without knowing which drive had the error.

 

If the parity bit is not adjusted the error should show up on every check, and if more errors show up you probably have a failing drive; but which one?  

Share this post


Link to post

I just removed this topic and reposted my original post as It was getting a little out of hand.  Let me start by saying I am new to unraid but have been in the software hardware field probably longer than most of this community has been alive. I started in 1960 and retired in 2005. I was a part time adjunct Prof Teaching computer Science at the Graduate level, and when I was in grad school myself I was privileged to have Richard Hamming (of Hamming code fame) as one of my instructors.  I well remember his lecture on data protection by using "Parity" versus using the concept of Hamming codes and Hamming space.

 

So lets start again.

 

Lets say you run a parity check, the one that adjusts the parity bit so parity is once again is correct; how do you then go about rebuilding the real bad drive assuming the problem was not on the parity disk? Lets say you do not allow for the correction of the parity disk. The question still remains.

 

The Parity check can not tell you which disk had the parity error so I guess that is the only solution, to correct the parity disk. What else can you do? That is why in a time long ago we used a two dimensional parity check so that the location of the failed bit could be discovered.

there are several other "checks" in place today that are invisible to us.  The data recorded in the disk platters has a CRC checksum per sector.  The disk itself will re-try reading a sector several times before deciding it cannot get a good CRC checksum.  Only then will it report to the OS that it is unable to read a sector.  It will report that as a "media error"

 

In the same way, a bad SATA cable might get induced noise... it too has a CRC checksum in its serial protocol.  Again, the failure is reported to the "md" drive only after the OS has attempted to reset the drive and drive controller.

I assume the only way to correct the "failed disk" is to read the syslogs if there is something there pointing to the failed drive, then try and ascertain which file on that drive has to be reloaded.

If a "read" error is reported by a single data drive, the unRAID "md" device drive will read parity and all the other data disks to re-construct the contents of the sector that could not be read and supply that to the program attempting to read the disk.  It will ALSO write that same re-constructed data back to the disk that reported an un-readable sector.

 

When a disk reports a sector un-readable that sector is placed on a list of sectors pending re-allocation.  When that sector is subsequently written, first an attempt will be made to write the data to the original sector on the disk.  If that works, the un-readable sector is now readable, and the sector is not re-allocated. If that fails, then an alternate sector on the disk is used from a pool of spare sectors. (most disks these days have several thousand spare sectors).

 

So... a disk read error is self-healing, with the contents supplied by the other data disks in combination with parity.

 I don't see any way of auto rebuilding the failed drive if it is a matter of a couple of bytes of corruption, yet I constantly see words in the posts re "rebuilding the drive".  I am a little confused (my normal state of mind these days) but I am sure community members will assist in bringing me clarity..

If you physically remove a single disk from your array you will still be able to read and write to it.  It is "simulated" by reading all the other data disks and parity.  In the same way, the parity disk is updated as needed when writing to the simulated disk.  If you now install a replacement disk, it can be completely rebuilt from the re-constructed contents based on parity and all the other data drives.

I completely understand the concept of parity and how to use it to detect a single bit error in a bit stream.  However you can not correct that single bit error without additional information.  In unRaid's case you do not know on which drive (parity or data) the single bit error occurred you only know one has.  That is why in another post I brought up the concept of "two dimensional parity" which allows for the correction of a large number of errors syndromes, but not all.

In unRAID's case, the initial parity calculations always assume the data disk is correct.  In the same way, subsequent "Checks" always assume the parity disk is wrong and the data disk right.  It is for that reason we, as users requested the addition of a non-correcting version of a parity check.  That is partly implemented.   (There is no web-interface to it other than through unMENU, and it does not tell you the location of errors)

How does unRaid go about rebuilding a failed non-parity disk, if there is a single bit error on other than the parity drive?If there is a double bit error in the same "row" of the parity computation, it will go undetected.  By "row" I mean all the corresponding bits from each of the drives that make up the corresponding bit parity bit on the parity drive. You just don't know which bit to correct, i.e. on which drive it is. That is probably why the parity bit is adjusted as there is nothing else you can really do.

Individual bit problems are typically detected by the disks themselves with their internal CRC checksums on the sector.  The only time "parity" errors show is if the array was not cleanly stopped.  When re-started, the journal system of the reiser file-system re-plays the transaction not yet committed.  The parity disk has no equivalent journal.  Typically, it will show a few parity errors after a non-clean shutdown as its housekeeping area is brought into sync with the data disks.

If the parity bit is not adjusted the error should show up on every check, and if more errors show up you probably have a failing drive; but which one?  Also what do some mean when they say "... the failed drive is simulated..." when discussing rebuilding the drive?

The parity calculation is performed several billion times and it is possible to both read and write to a completely failed drive. When reading the data is re-constructed by reading all the other data disks and parity.  When writing, all the other disks are read to first learn the old contents of a failed drive's sector, then parity is updated to reflect the new.  (obviously, we cannot write to a failed data disk... but we will once it is replaced with a working one)

 

To experiment do this:

Stop the array

While it is stopped, copy your config folder to old_config.  This is a good idea regardless since it lets you get back to a clean stop state.

 

Go to the devices page and un-assign a data disk.  (if you have a choice, you might use your smallest disk)

 

Go back to the main page and start the array.

 

Now, try reading from the un-assigned disk.  (un-assigning it is exactly the same as having it fail)  I've been able to play multiple movies from the un-assigned disk. I actually got up to 5 different movies before I ran out of media-player clients.  

 

You can even write to the un-assigned drive.

 

Now, stop the array once more, and re-assign the drive.   The entire drive will be re-written from parity in combination with all the other data drives.  That same parity calc that determines the value of a missing bit will be performed trillions of times to re-construct the data.

 

When the entire drive is re-written, you will have a fully parity protected array once more.

 

Joe L.

Share this post


Link to post

That explains it.  I was not aware of the operations at the disk level involving the CRC.  Thank you very much for the complete and detailed explanation.  I am sure others who read this will also be "de-mystified".

 

Thanks once again.

Share this post


Link to post

Might I suggest adding some of this goodness to the unOfficial unRAID Manual? Perhaps in a how your data is protected by unRAID section.

Share this post


Link to post

Might I suggest adding some of this goodness to the unOfficial unRAID Manual? Perhaps in a how your data is protected by unRAID section.

 

It seems a bit too technical for the manual (remember, the manual is intended for first-time unRAID users), but I added it to the 'Best of the Forums' section of the wiki.  I added it under Misc because I didn't see another section in which it would fit.  I would prefer to put it in something like a 'how unRAID works' section.

 

Joe. L, I titled it 'How does unRAID know if an error is on the parity drive or a data drive?'  Do you agree with that title?

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.