I just read the Corbett article fully, and the way they deal with the collision hole I mentioned is by substituting a data disk with the parity disk for the diagonal parity.  Including the regular parity disk in the diagonal parity means that every diagonal will have one data disk excluded (and some diagonal stripes will not have a parity block either)... so with a 2-data disk failure, there will be a diagonal parity that includes only one of the 2 failed disks, so the first block is repaired for that disk using diagonal parity, then the regular row parity can repair the other. 

 

So for n data disks, you have n+1 diagonal patterns.... that's the trick.

 

 

 

I re-read the Corbett article, and the diagonal parity is definitely doable for unRAID, and presents a number of advantages over using Reed-Solomon error correction Q-parity.  In fact, the only advantage for R/S, is that it could be adjustable to provide varying degrees of protection (protect against 1-disk, 2-disk, 3-disk, ... up to n-1 disk failures) but I can't see that as a something that unRAID would realistically employ, particularly with all the other advantages.

 

Creating the DP should be relatively simple to implement in the existing unRAID parity calc. Same with updating the DP on writes.  Recovery would be trickier, particularly handling writes while double-degraded (particularly writing to an emulated (i.e. failed) disk), but it is eminently doable.  It's not all that complex, but it can be tricky.

I understand standard XOR parity -- but I'm having trouble wrapping my head around the diagonal parity in RDP, even after reading the Corbett article multiple times.

 

Edit:

It finally sunk in after re-reading the paragraph on recovering data from 2 failed drives a few more times.

 

 

Figure1:

D1

D2

D3

D4

RP

DP

Row A

0

1

2

3

4

0

Row B

1

2

3

4

0

1

Row C

2

3

4

0

1

2

Row D

3

4

0

1

2

3

 

The operation of the algorithm can be seen by example. Assume that data disks 1 and 3 have failed in the array of Figure 1. It is necessary to reconstruct from the remaining data and parity disks. Clearly, row parity is useless in the first step, since we have lost two members of each row parity set. However, since each diagonal misses one disk, and all diagonals miss a different disk, then there are two diagonal parity sets that are only missing one block. At least one of these two diagonal parity sets has a stored parity block. In our example, we are missing only one block from each of the diagonal parity sets 0 and 2. This allows us to reconstruct those two missing blocks.

 

 

 

Calculating Diagonal Parity is very similar to the two-dimensonal Row-Parity I describe here:

http://lime-technology.com/forum/index.php?topic=5342.msg81842#msg81842

 

The key advantage of DP is you can recover from 2 failed drives.

 

The key advantage of RP is that you can continuously monitor for corrupt sectors as you read a data drive without powering up any other data drives.

 

The second advantage of RP is that you can use a much smaller disk second parity. The DP parity disk must be the same size as the first parity disk, ie both parity disks must match the largest data disk.

 

RP cannot recover from 2 failed drives, but the monitoring ability lets you detect when a single disk is about to fail just by reading files on it (normal day-to-day stuff). This may help you avert the 2-drive failure disaster by replacing disks sooner.

 

RP gives you a use for that nearly retired but rock-solid disk that is too small to be used for data disk, and too slow for cache. Or, RP finally gives you an excuse to put that SSD into your array, which you can use for both cache and RP by partitioning it.

 

Both DP and RP allow you locate an error on a single data drive (a single parity disk only lets you detect an error, but you aren't sure if the error is in the parity disk or one of the data disks; DP and RP allow you to pinpoint exactly which disk has erroneous data).

 

DP and RP should have similar performance. They both use only XOR calculations, and both are similar in write overhead (they both require reading a chunk of old data from the disk to be written, meaning they can incrementally update the parity without reading the other disks).

 

Both DP and RP depend upon disk ordering, but it would be easier to program an on-the-fly reordering with RP, and since RP uses a smaller disk it means you can rewrite this new ordering back to the 2nd parity disk more quickly. In both cases, with some programmer effort you can automatically detect reordered disks, but for a large number of disks the number of permutations is huge. For 10 disks, DP must search 10! = 3.6 million ordering comparisons, but RP can search each disk position independently, requiring just 10*9/2 = 45 comparisons.

 

RP is slightly easier to program, so it is less likely to have bugs in beta versions of the code. DP can be added after RP -- most of the code would remain the same, except for the extra DP logic you add in.

 

Guy

RP cannot recover from 2 failed drives, but the monitoring ability lets you detect when a single disk is about to fail just by reading files on it (normal day-to-day stuff).

 

Yes, and no.  It does have some predictive capabilities in some situations, much like SMART.  But the large-scale field tests (e.g. Google, et al) have shown a large percentage of drives fail w/o warning, and w/o being predicable by any predictive algorithms.

 

The key advantage of DP is you can recover from 2 failed drives.

 

And that's why anything less is inferior.

 

The second advantage of RP is that you can use a much smaller disk second parity.

 

I don't see this as worthy of mentioning.  The delta-$ between a small disk and your largest disk is small compared to the total cost of the array.

 

Disk ordering is a simple issue to address.  unRAID already manages disk slots by serial number.  The DP order can be recorded with the persistent array config, and at array-start, a simple array can be set up, to abstract the drives in the proper order, and the DP calculation uses the abstracted array for calculation/recovery.  So now, disk slot is independent of the disk order used in DP calculation

 

 

The key advantage of DP is you can recover from 2 failed drives.

And that's why anything less is inferior.

 

The second advantage of RP is that you can use a much smaller disk second parity.

I don't see this as worthy of mentioning.  The delta-$ between a small disk and your largest disk is small compared to the total cost of the array.

 

Disk ordering is a simple issue to address.  unRAID already manages disk slots by serial number.  The DP order can be recorded with the persistent array config, and at array-start, a simple array can be set up, to abstract the drives in the proper order, and the DP calculation uses the abstracted array for calculation/recovery.  So now, disk slot is independent of the disk order used in DP calculation

 

Yes, delta-$ savings is small. This is not to save money this, but find a use for all those old disks you keep throwing out and also to give you an excuse to use an SSD instead (and share it with your cache).

 

But the point is RP is more flexible than DP, and it is a bit easier to implement.

 

Like DP, RP offers full protection for 2 simultaneous failed disks, assuming you have installed an RP the same size as needed for DP.

 

RP has the option of using a smaller disk. This adds greater flexibility: you don't have to add another physical disk for RP. Instead, it can be located as a separate partition of your cache drive, which might be an SSD for low-power reasons. Or, it can be a 2nd partition on your existing data drives in the array (reserving <5% from each of them is enough space). In the latter case, for performance reasons, you should store RP parity partition on a different physical disk than the data disk it is protecting (eg store parity for disk N on disk N+1).

 

Suppose you lose 3 or more disks, then you have definitely lost some data. In this case, you must quickly protect your remaining data by regenerating your first parity disk from scratch. During this time, however, DP offers no protection and it must be regenerated from scratch as well. However, RP can continue to detect if there are integrity problems with the remaining disks while the new parity disks are brought online. Unfortunately, neither RP nor DP can recover if a third disk actually fails during that time.

 

Storing UUID disk order in a superblock on the DP disk is a great idea. To be fully protected against 2 simultaneous drive failures, you need to store it on 2 other disks as well -- probably the first parity disk and data disk #0. This is not needed for RP.

find a use for all those old disks you keep throwing out

 

Disks are retired for a reason -- PM.  Putting them back in a server is exactly what I do not want to do with them.

 

RP may be easier to do from scratch than DP, but this unRAID not being built from scratch and I do not agree that RP is any easier to implement in unRAID than DP.  Indeed, several of the options you mention would require much more work, such as embedding the RP on a portion of each data drive or a separate partition.

 

Another reason I do not think that RP is any easier to implement than DP is that DP is nearly identical to the current parity scheme, except for the sector offsets (and that is easily handled with an abstraction array) and skipping one disk in each DP stripe (which can also be handled with the abstraction array).  Reuse and modification of existing parity code in unRAID can make DP easier to implement in unRAID than RP.

 

Zeroing a drive and inserting it will also maintain DP parity protection but it will not with some implementations of RP you described.

 

RP can continue to detect if there are integrity problems with the remaining disks

 

As I said before, this predictive ability is weak, at best, based on large-scale drive studies.  The vast majority of drives give up the ghost without any warning and without any predictive algorithm being able to predict it.  This is why we replace drives prophylactically as part of PM.

 

Finally, I would not save the disk order on any data disk.  That would be poor planning, as a future direction of unRAID could make partition-less disks desirable.  Other changes in partitioning schemes could also break it.  Basic rule is that no array info should be stored on a data disk.  DP disk order only needs to be stored on flash with the array config.  I do agree, however, that there should be redundancy of the array config on the flash, but that is independent of the DP discussion.

I would be really happy to see dual parity, but i have a feeling this is not a priority in Tom's list so i don't think we will be getting it any time soon.

 

Also, I am wondering what a second parity disk entail for parity checking...

I would be really happy to see dual parity, but i have a feeling this is not a priority in Tom's list so i don't think we will be getting it any time soon.

 

I would love to see protection against a 2-disk failure.  When Q-parity was first mentioned, using R/S ECC, I was hopeful, but not optimistic due to the drawbacks of using R/S on live data (it works great on DVDs and other static data).

 

DP, on the other hand, has none of the negatives of R/S, and algorithmically it is nearly identical to the existing parity, so the existing codebase can likely be adapted to do DP rather than having to write it from scratch.

 

Tom has consistently placed data integrity and data protection at the top of the feature criteria along with performance, and I think a RAID-6 (dual parity) implementation fits that paradigm.  The shortcomings of doing it with R/S held it back.

 

The last major kernel upgrade doubled write performance, so now it is approaching (over 80% of theoretical for most people) theoretical maximums.  So there is not a lot that could be done write w/r/t performance.

 

I am now I am more optimistic that dual parity is doable, and that will move it higher on the to-do list.

As I said before, this predictive ability is weak, at best, based on large-scale drive studies.  The vast majority of drives give up the ghost without any warning and without any predictive algorithm being able to predict it.  This is why we replace drives prophylactically as part of PM.

 

Finally, I would not save the disk order on any data disk.  That would be poor planning, as a future direction of unRAID could make partition-less disks desirable.  Other changes in partitioning schemes could also break it.  Basic rule is that no array info should be stored on a data disk.  DP disk order only needs to be stored on flash with the array config.  I do agree, however, that there should be redundancy of the array config on the flash, but that is independent of the DP discussion.

 

If you're referring to the Google study on drive reliability, keep in mind that is just 1 study, and the Google environment is highly controlled. Field sitations in homes are different (no reliable air conditioning, users poking and prodding with the OS, etc).

 

The reliability I wish to protect against is more than simple disk failure. I am also referring to protection against OS, filesystem, and user error or failure. For example, suppose the user makes an error using the 'dd' command (not uncommon!). Or suppose a bug in the Samba+Windows combination results in a mangled filename, and when passed to the filesystem causes corruption of data in the filesystem usermemory, resulting in bad disk writes to random blocks unrelated to the file at hand. Or suppose a user tries a new SSD trim utility that mucks up their hard drive. These are the reasons why RAID is not a backup. Adding additional protection, in the form of DP and RP, can help make RAID closer to the reliability of a backup. RP simply offers a few more options than DP.

 

Yes, some of the "new features" of putting RP on data disk partitions would take more coding than DP. DP could provide a similar feature, but would need to power up all disk. To provide the online data checking of RP, DP must read all disks in parallel but RP only needs to read 2 disks.

 

Disk order MUST be stored on the disks with DP. Yes, this breaks an advantage of unRAID. If you lose your flash drive, you lose the disk order; you can't store backup copies on the same flash drive. Where else would you store it? You must be able to store it on the disks themselves. That way, if you have complete motherboard+flash meltdown, you can still bring up your unRAID array. RP does not rely upon disk order (it can be quickly detected automatically).

If you're referring to the Google study on drive reliability

 

I'm not.  The earlier Russian study is more real-world.  Google's is one of several, and all have more or less the same results about predictive algorithms.

 

I am also referring to protection against OS, filesystem, and user error or failure. For example, suppose the user makes an error using the 'dd' command (not uncommon!).

 

First, I do not see protection for user's intentional acts to be within the feature set of a NAS.

 

Second, the dd command is not in the feature set of most NAS users.  A NAS is an appliance.  unRAID is very power-user friendly and hackable, but don't hang anchors around its neck that have use to only 0.01% of purchasers.

 

Third, you keep adding to the feature set of what you "can" do with RP, but all of those features take code to implement them, and a user interface to use them.  That ain't gonna happen.

 

As for powering up all disks, that is no different than what we have now with standard parity.  When the array is in a degraded state, all the drives spin.... and you are supposed to replace the failed devices ASAP.  It is called fault-tolerance for a reason.

 

Disk order MUST be stored on the disks with DP.

 

No, it doesn't.  Today, if you lose your flash, you have to reassign all your disks, and know which was parity.  You can simply discard DP when you lose the flash and have to reconstruct your array -- that's the simple solution.  To improve on that, with DP, you just have to know which was DP, and a simple snippet of code can determine the right disk order for DP.  Or you can enforce disk order to always be sorted by serial number.  That makes additions of new disks to an array a little more complex, but it is doable.

 

The reliability I wish to protect against is more than simple disk failure. I am also referring to protection against OS, filesystem, and user error or failure. For example, suppose the user makes an error using the 'dd' command (not uncommon!). Or suppose a bug in the Samba+Windows combination results in a mangled filename, and when passed to the filesystem causes corruption of data in the filesystem usermemory, resulting in bad disk writes to random blocks unrelated to the file at hand. Or suppose a user tries a new SSD trim utility that mucks up their hard drive. These are the reasons why RAID is not a backup. Adding additional protection, in the form of DP and RP, can help make RAID closer to the reliability of a backup. RP simply offers a few more options than DP.

The majority of those types of file loss are not going to be helped by any kind of "raid" array, regardless of how many parity drives it has.  Those are only handled by backup copies of your files stored elsewhere.     RAID is not a backup of your files.  All it takes is one "aw-shit" moment, and your files are erased.   Ask me for the horror stories...  I have them from personal experiences.

 

Joe L.

A few more thoughts on ease of implementation of DP.

 

Creating DP is not hard.  Updating DP as the array is used is not hard.  Rebuilding two failed disks is not that hard.  The hardest part of doing DP today would be keeping the array up while simulating two failed data disks. 

 

Perhaps as the first step to DP, is that with 2 simultaneous data disk failures, or a data disk failure along with the parity disk, the array remains offline until at least one of the two failed drives is replaced and rebuilt.

 

After all, having the array offline after 2 simultaneous failures is no worse off then we are now.

Rebuilding two failed disks is not that hard.  The hardest part of doing DP today would be keeping the array up while simulating two failed data disks.

 

Now, I may not completely understand RDP, so please correct me where I'm wrong, but:

 

To rebuild the failed disks, you'll still need code that will recover data for the failed disk using parity and data from the other drives.  At that point, you've already made it most of the way to simulating two failed data disks.  If rebuild code is completed, simulated reads are basically done.  The extra code to allow simulated writes to two failed "disks" with RDP shouldn't be much more difficult than one.

 

Comparisons of writes to single vs dual drive failures if using RDP:

 

1) Data disk (DD) failures:

1a) To simulate writes to a single failed DD, you need to recover existing data for the failed disk using parity and data from the other DDs, then update both DP and RP based on the new data written.

1b) Not much changes for a two DD failure.  You still need to recover existing data for the failed disk using parity and data from the other DDs, then update both DP and RP based on the new data written (no new code here).

 

2) Parity disk failures:

2a) If only DP parity failed, then you still keep RP up-to-date on writes.

2b) If only RP failed, then you still keep DP up-to-date on writes.

2c) If both DP and RP failed, then you don't write parity (no new code here).

 

3)  1 DD, and 1 parity drive failure:

3a) If a DD and DP failed, see 1a, but only update RP (combination of 1a and 2a, very little new code here)

3b) If a DD and RP failed, see:1a, but only update DP (combination of 1a and 2b, very little new code here)

 

Unless I'm missing something (very possible), it seems that simulating writes with two disk failures is very close to simulating writes with only one.

 

I didn't mean it was hard, per se, just hardER than the other parts.

 

unRAID rebuilding works at the driver level.  At that level, there is no concept of files... it is just sectors.  A request for sector "X" can have 3 results:

 

#1  - sector X is on a valid disk.

#2a- sector X is on a missing disk, but all other sectors needed to calc it are present using regular parity.

#2b- sector X is on a missing disk, but all other sectors needed to calc it are present using DP.

#3  - sector X is on a missing disk, and both regular parity and DP for that sector use the other missing disk (z)

 

Only 2b and 3 are new.

 

The solution for 2a is simply rebuild the missing sector regular parity. 

The solution for 2b is simply rebuild the missing sector using DP.  

 

The solution for 3 is to rebuild the missing necessary sector (either regular parity or DP) on z (which can always be done with #2a or #2b).  Then you have a second #2a or #2b to solve.

 

In a n-drive system, all the sectors will be within n+1 sectors of each other so you will have some potential thrashing when crossing a cylinder boundary, but the track buffering should handle that.

If you mean "harder" as is harder to code, I still don't think that's the case.  Yes, you may hit the "harder" code path (using both RP and DP) twice to rebuild a missing sector, but the code to rebuild failed sectors from parity is already there -- why not use it?

 

If you mean "harder" as in performance would suffer badly -- I agree, and see your point.

 

Speed could be a bear... instead of having a algorithm to determine the parity participants, I'm thinking that having n+1 abstraction arrays for a system with n data drives would be fastest.  That paired with an abstraction array for the disk order would be fast. Then you can select the proper abstraction array based on mod(n+1) of the sector number (of course accounting for the first n+1 and last n+1 sectors of a disk).

 

It won't be as fast as regular parity, but it would take much fewer cycles than an algorithmic solution, unless you could do it with some bitmaps and shift through the bits.

Anyone have an idea what the cost of the modulo function is, compared to other arithmetic ops using current flavors of gcc on x86?

I am also referring to protection against OS, filesystem, and user error or failure. For example, suppose the user makes an error using the 'dd' command (not uncommon!).

 

Disk order MUST be stored on the disks with DP.

 

No, it doesn't.  Today, if you lose your flash, you have to reassign all your disks, and know which was parity.  You can simply discard DP when you lose the flash and have to reconstruct your array -- that's the simple solution.  To improve on that, with DP, you just have to know which was DP, and a simple snippet of code can determine the right disk order for DP.  Or you can enforce disk order to always be sorted by serial number.  That makes additions of new disks to an array a little more complex, but it is doable.

 

 

I'll start by correcting myself: you can't protect the data from filesystem bugs with RP or DP. I wish, but unfortunately it won't happen because the filesystem talks to the md driver, which does all corresponding checksum calculations automagically. So an errant filesystem will also corrupt the checksum sectors. What I was thinking is when a power user uses dd on the underlying drive device (eg, /dev/sda instead of /dev/md0). That said, I wish filesystems had better data integrity checking, something along the lines of WeeboTech's par2 idea.

 

Next, it is impossible to automatically detect the correct order of drives with DP for a large array. For N data disks, there are N! possible orderings (N! = 1*2*3*4*5...*(N-1)*N. With DP, to automatically detect the ordering, you must first assume a particular order, do a DP calculation, and then see if the DP disk agrees. If the DP disk disagrees, you must try a different order. For small arrays, N=6 leads to 720 possible orderings, which is small enough to automatically try all orderings and determine the correct one. For N=10, there are 3.6 million orderings, which might also be do-able. For N=15, there are 1.3 trillion orderings. A single DP calculation on a 512 byte block, for 15 disks, using 128-bit SSE instructions involves more than 480 XOR instructions (not counting instructions to load the blocks from cache to a register, which would probably double or triple the run-time). On a 3GHz CPU, this would take at least 58 hours (possibly triple). Add a 16th drive, and you're looking at 16-times longer run-time (about 30 days). Fortunately, you could reduce this run-time by asking all unRAID@Home users in the world to compute a small subset of the orderings for you :-)

 

With unRAID today, it is easy to auto-detect the parity disk. All you have to do is inspect the data disks for valid filesystems -- in most cases, the parity will be the disk without a valid / consistent filesystem. If unRAID does not parity-check the MBR, you could simply identify the parity drive via special pattern of bits in the MBR (eg, a custom partition type, or an invalid partition where the end-of-partition appears before the beginning). Of course, unRAID doesn't care what order your disks are in because the calculations are all independent of order.  This is not true with DP, so if anyone is to ever implement DP, I strongly suggest encoding the data disk order somehow. Otherwise, people with large arrays (anything over 10 drives) will have a difficult time reconstructing their data after a catastrophe.

 

Guy

For N data disks, there are N! possible orderings

 

A single DP calculation on a 512 byte block, for 15 disks, using 128-bit SSE instructions involves more than 480 XOR instructions (not counting instructions to load the blocks from cache to a register, which would probably double or triple the run-time).

 

True, but there is a shortcut you are missing.  If you have 15 disks, once you calc the parity, you don't have to redo all of them for changing one disk.  You can save the intermediate result, subtract one sector, and add another.  That's 2 sectors worth of XOR calcs, to try the next permutation, not 15.  The larger the number of disks, the more you save (percentage wise).  I believe there is a matrix formula that also creates a vast computational savings, but I can't remember the specifics... its been many years since I've coded using matrix calcs.

 

I admittedly have not speced it out to see how long it would take, but even it it was prohibitively costly, the solution is not to totally screw with the underlying core of unRAID by putting any array state information on a disk.  That could also affect the disaster recoverability of removing drives from unRAID and mounting them in another system.

 

The proper place is on the flash, or enforce a self-defined order, such as sorted by serial number.  While you "wish filesystems had better data integrity checking" the same goes for more redundancy in the flash.  But neither of those are the bailiwick of dual parity.

True, but there is a shortcut you are missing.  If you have 15 disks, once you calc the parity, you don't have to redo all of them for changing one disk. 

 

...the solution is not to totally screw with the underlying core of unRAID by putting any array state information on a disk.  That could also affect the disaster recoverability of removing drives from unRAID and mounting them in another system.

 

The proper place is on the flash, or enforce a self-defined order, such as sorted by serial number.

 

Incremental XOR calculation lets you add one more disk, so you don't have a huge problem until N=17.

 

Sorted by serial number works like a charm -- I like that! The main disadvantage is the need to recompute DP from scratch whenever you add a new disk (since the disk ordering changed). The other disadvantage is that after a rebuild (remove dead disk, insert new disk, rebuild data on new disk), the old DP data that was used to rebuild the data disk is now useless and you need to rebuild DP again. This lengthens (doubles) recovery time, which shouldn't be a huge deal.

 

There is plenty of free space on a disk: all you need is ~32 bytes. A single sector can store the ordering without affecting ability to pop a drive into another system. With advanced format drives and SSDs, you need to align partition boundaries, thus creating many gaps between the start-of-disk and start of the first partition. The start of partition 1 with unRAID is currently sector 63, but this needs to move to 64 for the new Advanced Format drives. On many SSDs, you should bump the starting sector to 1024 or 2048. There is also that last sector on the hard disk which nobody every uses :-)

 

I just looked at the unraid md.c and unraid.c code. It's pretty cleanly written. I can probably code something up over the weekend, ... if I get a spare wkend this winter I'll give it a try. The starting point is to simply add a 2nd parity disk which contains the exact same data as the 1st parity disk. Once that works, we can get back to discussing writing either RP vs DP data to the 2nd parity disk. After that is implemented, it's a matter of allowing degraded RAID operation, and rebuilding missing disks, ...

So, you it is possible to integrate double parity in current unraid versions without any further action from Tom?

So, you it is possible to integrate double parity in current unraid versions without any further action from Tom?

 

No. The management console (emhttp) definitely needs to be changed too. Only Limetech can change the management console. Only the Linux device drivers are open-sourced.

 

So, you it is possible to integrate double parity in current unraid versions without any further action from Tom?

 

No. The management console (emhttp) definitely needs to be changed too. Only Limetech can change the management console. Only the Linux device drivers are open-sourced.

 

 

Yes, right, I got carried away.

It was too good to be true...

That space between the partition table and the start of the first partition is a minefield.  Over the years, several (very bad) pieces of software have used it for various things, such as copy protection, to disastrous results.  unRAID should not join that list of shame-ware.

 

Sorted by serial number works like a charm -- I like that! The main disadvantage is the need to recompute DP from scratch whenever you add a new disk

 

And you then lose 1 degree of protection every time you add a drive, until that drive has been folded into DP. Everyone adds drives.  Few people lose a flash.  So I still prefer storing it on the flash. (and beefing up recovery abilities for a lost/corrupted flash, such as automatic backing up the flash to a data disk (in a real file, not sme hidden sector) or automatic e-mailing it to you whenever there is a change.

 

I would store the disk order on the flash with the superblock for the array.  If you lose the flash, you do not lose data, and you still have regular parity, so you lose DP until it is rebuilt -- the exact same result (1 degree of protection lost until DP is recalced) as when you add a new drive if you are using serial number order.  But you only hit that state  (1 degree of protection lost until DP is recalced)  when you lose the flash, and have no backup or printout to reassign the disks to the same order.

 

Look at it this way.  If you lose a flash, you should reassign the drives to the same slots in order to maintain both standard parity (SP) and DP.  Or you should have a backup of the array config from the flash.  Otherwsie, you get to keep SP, but you have to rebuild DP.

 

OTOH, I just realized that you can not add a zeroed-out drive to the array and maintain DP. When you go from N to N+1 disks, you go from N+1 diagonal stripes to N+2, and not just an additional sector added to the existing N+1 stripes.  I'd like to be proven wrong here.....

 

So since you have to discard DP when you add a disk to the array, you don't have  to maintain an arbitrary order, and you can just 1) sort by serial number or 2) use the unRAID slot-order.

 

But I would still use the arbitrary order or disk-slot order and store it on the flash rather than using serial number order, because it allows retaining DP when replacing a failed drive. 

 

But either method is still a hell of a lot better than any other RAID flavor.