The squeaky wheel gets oiled.  If there are enough cases documented where a rebuild fails due to a second drive failure I'm sure it would provide leverage in the development effort.

 

I was kind of bit by this just recently.  My Parity drive went down with some write errors.  I reseated the cables and it came back but it wanted to do a parity check so i let it.  During that time one of my data drives went down.  So now the server thinks parity is bad and I have a data drive that went down. Hopefully i can bring the drive back up long enough to pull the info off of it...

 

Do you do periodic cron based parity checks?

 

What's the rule of thumb for retiring drives?

Do you do periodic cron based parity checks?

Yup, it runs on the first of the month every month.  The check finished no problem 2 weeks ago.

 

What's the rule of thumb for retiring drives?

Generally when they start showing signs of wear and tear, and when the power on hours gets to a certain mark.  The drive that died was fairly new (less than a year old if I remember correctly).

Maybe a poll for hot spare or dual parity should occur.

 

Why choose between dual parity and a hot spare?

Why can't we have both?

 

Keeping my data safe is very important to me.  I want as many layers of security for my data as possible.  But, I don't really want to maintain two separate RAID systems.

 

1. Cost - 2 extra drives per array. (for people who are budget and space minded)

2. Programming time and effort (While I believe both provide value, the time and effort involved needs to be prioritized. if this were to happen in stages, Hot spare, then q-parity or vica versa, there are all sorts of efforts and betas that need to occur).

 

Also, I never said one without the other.

 

 

Hey Weebo ... I certainly agree with your second point.  All projects must prioritize.  So instead of thinking about whether we should have one or the other we should consider which we would prefer to see first 

 

I do, however, disagree with your first point.  If people are budget minded, they can simply run their array with a single parity drive.  I see no reason why unRAID couldn't offer both single parity or dual parity as a configuration option.

 

 

 

 

 

I see no reason why unRAID couldn't offer both single parity or dual parity as a configuration option.

 

A programmer or a mathematician will disagree with you.

 

You can't just say "make 2 parity disks" and voila -- you have double parity protection and can survive 2 simultaneous drive failures.

 

Traditional striped RAID can do ReedSolomon or other error correction on the fly for parity because all disks are always spinning.

 

unRAID recalculates the parity across N drives when only 1 drive changes, and can do it without spinning up the other drives.  It can do that because it can take the parity, and "subtract" the old data fir that sector on the active drive, and then "add" the new data being written to that sector on the active drive, and thus determine the new value for the parity for that sector without accessing the other N-1 drives.

 

AFAIK, you cannot do that with R/S or other polynomial error correction algorithm.  So if you change 1 disk, you have to re-read all other N-1 disks to recalculate the parity for that sector.

 

So to do Q-parity and have the ability to survive 2 drive failures, you have to leave all drives spinning all the time when writing to the array.

^^ Very interesting! I like this thread. Its very informative..

I see no reason why unRAID couldn't offer both single parity or dual parity as a configuration option.

 

A programmer or a mathematician will disagree with you.

<snip>

So to do Q-parity and have the ability to survive 2 drive failures, you have to leave all drives spinning all the time when writing to the array.

Plus, you have to understand that is will NOT speed up disk writes... If anything, it will slow things down.

 

In any case, check out the EVENODD parity paper I linked to.  It is mostly over my head, but a quick read seemed to indicate it did not need to spin up all the other drives, just the few involved by doing a read/write operation similar to what unRAID does now.

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.88.332&rep=rep1&type=pdf

EVENODD technique for "recovery of 2 failed disks" describes read-modify-write timing and how it offers better performance over REED-SOLOMAN.

 

Even/Odd is mathematically no different than just making 2 arrays, and then you have a 50-50 change of a 2-drive failure happening in the same array (data loss) versus just 1 in each array (no loss).

 

With R/S, theoretically you can chose any size of redundancy you want.... just like with PAR2 files.

 

Say you have 10 drives total, each 1TB.  You can have 9TB data and 1TB of ECC (survive loss of up to 1TB of data) or 8TB of data and 2TB of ECC (survive up to 2TB of loss (i.e. 2 drives)), and so on.

 

There are other polynomial ECC mechanisms, but they are more computationally expensive than R/S and none that I know of allow you to "subtract" a participant in the ECC and add another w/o accessing the source data from all the other participants.

 

I would love to be proven wrong, but I don't see it happening unless you go to a "spin up everything on writes" system.

 

Now there is something I though about while driving the last couple of days.

 

There are certain discrete outcomes from any binary parity system like unRAID:

 

- you know that both missing disks have a "1" in that bit  (definitive)

- you know that both missing disks have a "0" in that bit  (definitive)

- you know that both disks have the opposite value, but don't know which disk has a "1" in that bit and which has a "0"  (ambiguous)

- you know that both disks have the same value, but don't know if both are "1" or both are "0"  (ambiguous)

 

So what if you stored a bit on a second parity drive that told you which to resolve the ambiguous result?  I think that could be done if you require a convention, and reliance, you the order of the drives.

 

I haven't worked out the math yet, but you might be able to create a bitmap of the pattern of transitions... i,e, when you go from disk n to disk n+1, the transition is either 0-0, 0-1, 1-1, 1-0.  Since all the drives are there except 2, you can check the transitions and I believe you may be able to disambiguate the result from the standard parity.  You could also be able to "subtract" and "add" one participant.  I don't do this kind of math regularly, so I may also be smoking stems...... but usually on the long 1000 mile drives when I am on autopilot I do my best theoretical work

 

If you can do it on a bit-level, it would just take some math to make it work on the byte level, and then unRAID could recover from a 2-disk failure by using conventional parity, plus a second "helper" parity to resolve bit order for an ambiguous result from standard parity.  It is a concept sometimes known as a disambiguation bit.  So I will call this the Disambiguation Parity or D-Parity

 

This D-parity would, however, be invalidated if you ever changed the order of the drives!!!!!! However I don't believe that would be a huge hurdle to deal with... if you changed order of drives, the standard parity would still be fine... only the D-parity would be invalidated.  unRAID could easily detect a change in drive order, or could use an artificial order determined and recorded by unRAID.

 

Adding or removing ANY drive would potentially require recalculation of D-Parity, unless you essentially included all 20 drive slots -- including empty slots -- in the D-Parity.  Perhaps if unRAID did its own ordering of drives, new drives could be added to the end of the list, so you could do a quick add of a cleared drive.... I haven't though about that problem much yet.

For a second parity drive, keeping disks in a particular order is a non-issue I think. Its a very small inconvenience if at all.

 

The thing about the D-parity drive is that you would need to spin up a min of 3 drives because you would need to calculate the parity using the preceding and proceeding bits from the before and after drives, not so?

The thing about the D-parity drive is that you would need to spin up a min of 3 drives because you would need to calculate the parity using the preceding and proceeding bits from the before and after drives, not so?

 

I'm not so sure.... I'm still thinking about it.  You can certainly do it without re-reading the adjacent values by using a positional coefficient, and then reducing the result to binary... the issue is will that give you the necessary resolution, or whether you need 2 bits to do it (there will be collisions, but they may be logically exclude-able).... and even with 2 bits, you may be able to do it because you don't always need disambiguation.  And it also may depend on which ambiguity you are looking to eliminate. 

I haven't worked out the math yet, but you might be able to create a bitmap of the pattern of transitions... i,e, when you go from disk n to disk n+1, the transition is either 0-0, 0-1, 1-1, 1-0.  Since all the drives are there except 2, you can check the transitions and I believe you may be able to disambiguate the result from the standard parity.  You could also be able to "subtract" and "add" one participant.  I don't do this kind of math regularly, so I may also be smoking stems...... but usually on the long 1000 mile drives when I am on autopilot I do my best theoretical work

 

I actually understood most of that... after i read it a second time.

 

 

Maybe smoking stems would help in this situation... free the mind as it be.  Though i hope you are not trying to do this math, smoke stems, and drive at the same time; that could get a little dangerous.

I see no reason why unRAID couldn't offer both single parity or dual parity as a configuration option.

 

A programmer or a mathematician will disagree with you.

 

You can't just say "make 2 parity disks" and voila -- you have double parity protection and can survive 2 simultaneous drive failures.

 

Traditional striped RAID can do ReedSolomon or other error correction on the fly for parity because all disks are always spinning.

 

unRAID recalculates the parity across N drives when only 1 drive changes, and can do it without spinning up the other drives.  It can do that because it can take the parity, and "subtract" the old data fir that sector on the active drive, and then "add" the new data being written to that sector on the active drive, and thus determine the new value for the parity for that sector without accessing the other N-1 drives.

 

AFAIK, you cannot do that with R/S or other polynomial error correction algorithm.  So if you change 1 disk, you have to re-read all other N-1 disks to recalculate the parity for that sector.

 

So to do Q-parity and have the ability to survive 2 drive failures, you have to leave all drives spinning all the time when writing to the array.

 

Bubba, I actually am a programmer and yes, I disagree with myself all the time 

I'm just now starting to read about various RAID algorithms, so there are still a lot of gaps in my knowledge.

I hadn't considered the issue that multiple disks would need to spin up for RS coding. 

 

Of course, as a programmer, I still say that dual parity can easily be made a configuration option.  I plan to do a lot more reading from my array than writing.  So having all drives spin up for a write doesn't bother me much.  Regardless, I plan to go off and read more about dual parity systems and algorithms.

 

 

 

 

The thing about the D-parity drive is that you would need to spin up a min of 3 drives because you would need to calculate the parity using the preceding and proceeding bits from the before and after drives, not so?

 

I'm not so sure.... I'm still thinking about it.  You can certainly do it without re-reading the adjacent values by using a positional coefficient, and then reducing the result to binary... the issue is will that give you the necessary resolution, or whether you need 2 bits to do it (there will be collisions, but they may be logically exclude-able).... and even with 2 bits, you may be able to do it because you don't always need disambiguation.  And it also may depend on which ambiguity you are looking to eliminate. 

I'm not sure your analysis of the non-ambigious bit patterns is correct

 

For three data disks, the patterns for 1 bit are here.  I also added (in blue) the "even" parity bit

 

000  0

001  1

010  1

011  0

100  1

101  0

110  0

111  1

 

Now.  Let's assume one of your definitive (non-ambiguous) 2 disk failures, represented by XX

0XX  0

 

Tell me, what were the two bits that failed....  We know the parity value is a zero. We know the remaining data bit.

Unfortunately, there are two different patterns of bits that would result in a zero parity bit.

000  0

011   0

 

Not so definitive in my mind.

 

Joe L.

Parity tells you the only two results possible are:

 

000

011

 

 

If you know the number of transitions (from 0 to 1 or 1 to 0 as you move left to right) you can disambiguate.

 

 

000  0 <----Transitions = 0 (even)

011  0 <----Transitions = 1 (odd)

bubba, I think your technique may fail for a larger number of disks.

For example, if you have 10 disks (disks 1 though 10) with the following bit pattern

 

0001110001

 

It's easy to see there are 3 transitions.  However, what if two failures occur at transition boundaries as follows:

000x1x0001.

 

The first transition may have happened between disks 3 and 4, or it may have happened between the disk 4 and disk 5.  Same goes for the second transition.  The transition may have occurred between disk 6 and 7, or it may have occurred between 5 and 6.

 

In other words, the above can resolve to either of the following and still satisfy both P and D.

0001110001 (correct)

0000100001 (incorrect)

 

 

 

Even/Odd is mathematically no different than just making 2 arrays, and then you have a 50-50 change of a 2-drive failure happening in the same array (data loss) versus just 1 in each array (no loss).

I don't think that is how EVENODD works. 

 

It uses two "parity disks", one with the normal "exclusive-or" of the horizontal rows of bits, and the other with an exclusive-or of the diagonal rows of bits.

 

Again, I'll need to do a lot more reading to understand how EVENODD works, but their example specifically deals with reading and writing a subset of the disks when writes occur.  Their example only shows three disks being involved, implying only three need spin up.

I don't think that is how EVENODD works.  

 

I didn't have your link, so I was just assuming.

 

bubba, I think your technique may fail for a larger number of disks.

 

Yes, it does, which is why I was considering some positional coefficient, that would address that... I'm still cogitating on it.

bubba ... have you looked at Row-Diagonal Parity?

I just came across it, and it seems to be a pretty efficient approach for dual parity.

 

http://www.usenix.org/events/fast04/tech/corbett/corbett.pdf

 

 

bubba ... have you looked at Row-Diagonal Parity?

I just came across it, and it seems to be a pretty efficient approach for dual parity.

http://www.usenix.org/events/fast04/tech/corbett/corbett.pdf

I agree.  Looks very interesting, and from their description similar to EVENODD, but slightly more efficient.

I always thought there was a hole in RDP (Row-Diagonal Parity) where you had (theoretical) collisions, but that is not mentioned in this article..... interesting.

 

Note that RDP also has the same restriction I discussed earlier that it will be invalidated if you change the disk order.... so unRAID has to address that.

Note that RDP also has the same restriction I discussed earlier that it will be invalidated if you change the disk order.... so unRAID has to address that.

 

Good point.  Changing disk order would require a rebuild of the diagonal parity (but not the horizontal parity).  A small price to pay for a bit more peace of mind.  I wonder how often people change the order of their disks.  If it's not too often, then I don't see this as a major issue.

 

You can probably get around this limitation, but it would require an extra level of complexity for unRAID to deal with.  

Essentially, the idea would be that:

(1) a disk has an actual disk # based on the slot to which you assign the disk (may be changed)

(2) a disk has a virtual disk # which remains static for the lifetime of the disk in the system, from which the diagonal parity is computed.  The virtual disk # doesn't change when a disk is moved

   (2-a) the virtual disk # can be disassociated from a disk when a disk is cleared and/or formatted

   (2-b) a new disk added to the system will be assigned the lowest available virtual disk #

   (2-c) if there is a hole in the virtual disk array because a disk was removed and not replaced, diagonal parity can be calculated by assuming all 0's for the missing virtual disk

   (2-d) a new pre-cleared disk can be added to the system and assigned to the gap in the virtual disk array without the need to compute a new diagonal parity

 

While this does add a new level of complexity to unRAID, it could likely be implemented on top of the current unRAID software architecture.  That is, only the new diagonal parity portion of the unRAID software would even have to know that these virtual disk numbers exist.  There should be no need to modify the existing/working functionality of unRAID.

 

Just a quick thought  

 

 

The "md" devices are, in effect, virtual slots.  They map to physical devices.  Parity is done on the "md" devices, it would not matter if the physical disks change, as they would still map to the same "md" device.  Now, if a persons assigns disk3 to disk4, and disk4 to disk3, then yes, we need another layer of abstraction or, just make it easier, re-calculate the diagonal parity in that situation.

The "md" devices are, in effect, virtual slots.  They map to physical devices.  Parity is done on the "md" devices, it would not matter if the physical disks change, as they would still map to the same "md" device.   Now, if a persons assigns disk3 to disk4, and disk4 to disk3, then yes, we need another layer of abstraction or, just make it easier, re-calculate the diagonal parity in that situation.

 

Hmm .. I've only been reading the unRAID forums for a short while.  I'm not sure what you mean by "md" devices.  Can you explain or point to a reference?  When I search, I get hundreds of results for AMD.

 

disk3 to disk4, and disk4 to disk3

 

Can you explain why this case is special?  I suppose I don't know enough about the underlying implementation of unRAID to understand this point.

 

 

The "md" devices are, in effect, virtual slots.  They map to physical devices.  Parity is done on the "md" devices, it would not matter if the physical disks change, as they would still map to the same "md" device.   Now, if a persons assigns disk3 to disk4, and disk4 to disk3, then yes, we need another layer of abstraction or, just make it easier, re-calculate the diagonal parity in that situation.

 

Hmm .. I've only been reading the unRAID forums for a short while.  I'm not sure what you mean by "md" devices.  Can you explain or point to a reference?  When I search, I get hundreds of results for AMD.

 

disk3 to disk4, and disk4 to disk3

 

Can you explain why this case is special?  I suppose I don't know enough about the underlying implementation of unRAID to understand this point.

 

 

You asked for it.  The is the linux overview, the only difference is that the unRAID md driver is completely rewritten for unRAID.

You asked for it.  The is the linux overview, the only difference is that the unRAID md driver is completely rewritten for unRAID.

 

Thanks prostuff ... just what I was looking for.

 

 

I wonder how often people change the order of their disks.  If it's not too often, then I don't see this as a major issue.

 

Some things you may not realize do it....

 

You have disks 1-8, and you add a new disk 9, and copy everything from disk 5 to disk 9, then remove disk 5 and leave it empty.

 

You move everything off disk 5 since it is getting old and you want to remove it, and then remove it leaving slot 5 empty.

 

You have drives 1-4 and 6-8, an empty slot 5, then put a new drive into slot 5.