Parity Drive Question

[DigitalDivide]

I really need to wrap my head around this whole parity thing for the Unraid.  Let’s say I have 2 data drives and one parity drive, all 3 drives are 500gb.  When I copy data to one of the data drives, it only copies data to that drive and data to the parity drive.  If that data drive fails, and I swap it out with another drive and the data is restored from parity, wouldn’t that mean that the parity drive contained duplicate data for that drive?  With 2 data drives it would mean that the parity drive must contain duplicate data in order to restore any of the drives should they fail.  How is this possible that one single parity drive can contain all the data for up to 12 drives?  Is it compressed in some manner?  Can anyone point me to a good article on this?  I’d really like to get a better understanding on this works exactly.  It's driving me crazy!

[hypyke]

It's not a copy of the data.  Parity is all math.  It is a calculation based on the data that you have.

 

This is NOT how it works but it is an example of how it is possible.

 

In your scenario you have 3 drives.  They will be represented by the numbers 4, 5 and 9.  Your two data drives are the 4 and the 5 and the parity is a calculation based on that data.  In this case the calculation is 9.  4 + 5 = 9.  If the number 4 dies all you have are 5 and 9 but you know what the number was that died because 9-5=4 and the 4 can be rebuilt. 

 

BUT if both 9 and 4 die and all that is left is 5 how can you tell what the other 2 numbers are?  You can't.  Therefore data loss. One drive failure, or number failure in this case, can be compensated for.  Two cannot.

 

Here is the other bit about parity that may be confusing you.  Yes, everytime you write to the data drives parity gets written to as well.  That is because as soon as one of those numbers changes in any way the calculation is wrong and must be re-done.

 

Side note:  If you ever have to rebuild a drive in an array you will notice that ALL drives in the array are being read while the new drive is being rebuilt.  This is another example of how parity uses the calculation it has in addition to the data you have to come up with the missing data.

 

Make some sense?

[mutant_matt]

It might have actually been clearer explaining exactly how it does work with binary!

[limetech]

At the end of this article there is a good explanation of 'parity' as it's used in RAID systems, including unRAID.

[hypyke]

It might have actually been clearer explaining exactly how it does work with binary!

 

I didn't claim to understand HOW it works.  Only that it can.  Looking at that wikipedia article gave me a headache.

[Joe L.]

Let me try to explain how it works.

 

I wrote this a while ago in the thread about unRaid on AVS.

 

Lets try an example... Lets say I set up a new UnRaid server with 5 disks for data and 1 for parity.  I'll take for granted that you know that the data on disk drives is stored as a series of "bits" and each bit can have a value of either "0" or "1".

The file-system we create on the disks allows us to organize collections of these bits  in files and folders.  I'm going to ignore the file-system in my example, as parity on it works exactly the same as parity on files.  For now, lets assume the files on the disk start at the beginning of the disk, and the file system overhead is at the end. (in reality they are interleaved)

 

For arguments sake:

the very first file located on the first disk is a text file joe.txt

the very first file located on the second disk is a picture of my cat

there are no files on the third disk... it is currently empty

the very first file located on the fourth disk is a MS-Word document

the very first file located on the fifth disk is a PDF document

 

Now... I also have a parity disk and I need to store data on it so if I lose any single drive I can still rebuild its data.

 

So I start by reading the first bit of each file on each disk.

 

the very first bit in the text file joe.txt on the first disk is a "1"

the very first bit in the picture of my cat on the second disk is a "0"

for the third disk the spot on the disk where the first file would be is a "0"

the very first bit in the MS-Word document on the fourth disk is a "1"

the very first bit in the PDF document on the fifth disk is a "1"

 

Now... Tom has said he stores EVEN parity... that means he stores either a "1" or a "0" on the parity disk in each bit position so we will have an "EVEN" number of "1s" in that position across all the drives, including the parity drive.    So far, we have an ODD number of ones in this bit position across all the drives holding our data. Since unRaid stores EVEN parity we need to store a "1" on the parity drive for this bit position. (for a total of four "1s", an EVEN number, in this bit position across all the drives, including the parity drive)

 

Now... lets say one of the drives fails and the unRaid software detects the failed drive... since we have a parity drive we can figure out what would have been on the failed drive if we could have read it... To do this we need to read the first bit from all the drives still available.

 

we read the very first bit on the first disk as a "1" (joe.txt)

we read the very first bit on the second disk is a "0" (picture of cat)

we read the very first bit off the empty third disk is a "0"

we CANNOT READ the fourth disk as it has failed - this is the bit we need to recreate

we read the very first bit on the fifth disk is a "1" (PDF document)

we read the very first bit in parity drive and it is a "1"

 

Now... we know the missing bit on the fourth disk could be either a "0" or a "1" (unRaid needs to figure out which)

We also know that when we originally calculated and stored the bit on the parity drive we had an EVEN number of "1s" in that bit position across all the drives. With that knowledge we can figure out what the missing bit would have been if we could have read it.

 

Since in all the bits we can still read (data bits in the working drives, and the bit in the parity drive) we have an odd number of "1s", the missing bit we cannot read, because of the failed drive, must have also been a "1" to make it an even number of "1s." If we guessed the bit on the failed drive was a "0" we would have three bits set to "1" in that bit position (an odd number) so it can't be that. Since we are originally calculated and stored EVEN parity... the missing bit has to be a "1" for a total of 4 bits set to "1" (an EVEN number) 

 

Hey... unRaid just re-created a missing bit from a failed drive!!!  Pretty easy, since it could only have been one of two values, and we could use the remaining data drives bit values in that bit position in combination with the parity bit value in that position to figure out the missing bit's value.

 

So... the algorithm does let us recreate any single missing bit by using the data on the parity drive in combination with all the other data drives that can still be read.

 

Use the same algorithm for each bit position in turn on the entire set of drives and you can recreate the entire set of bits that existed in the failed drive.

 

We can even simulate writing to the failed disk...  Lets say we edit the MS-Word document on the fourth disk and now the first bit on it has been changed from a "1" to a "0" (we are able to read and write the MS-Word document, even though the disk it is on has failed)  Now, unRaid will probably attempt to write as usual... It first pre-reads what that bit held on disk 4.  (it will be re-created from parity, it learns it was a "1". ) It then looks at the bit it is going to write as we update the document.  (It is a "0")

 

Since there will be one less "1" in the data in this bit position, unRaid's parity routine knows it must also change the bit on the parity drive from a "1" to a "0" to have an EVEN number of "1s" in that bit position across all the drives.  It will then write a "0" to that position on the parity drive. (Not sure if it will also attempt a write to that same position on the failed drive, it might or might not, but that does not matter, it is still marked as "failed" so it will not be used when reading)

 

Once the parity drive has been written to, it once again can re-create the bit when the file on the failed disk is attempted to be read. It does this by reading that bit position on ALL the working drives, including the parity drive.  This time however, there will be an even number of "1s" in all the working drives it can read, therefore, the bit on the failed drive must have been a "0"  Again, unRaid re-created the data from the failed drive.

 

Joe L.

[DigitalDivide]

Thanks guys!  Now I understand.

 

So, I guess in order to start with Unraid, I need to start with 3 drives, one parity and two data.  The reason I am asking is I have a number of drives with data on them already.  I was going to buy two 320GB drives today.  I was hoping to make one parity with the other data.  Once Unraid is installed I wanted to copy data from my existing drive to the new one.  Then format the old one and add it to the array.  Then do the same with my other drives.  I'm assuming it's the only option I have other than buy 3 drives.  I could use the drive I have the OS on but I'd rather not wipe out my OS just yet.