How does dual parity work, actually?


Recommended Posts

That is the most reasonable way of doing it, imho.

I could buy another license and put up a 2nd unraid, but it's more convenient to have all the stuff in one place. Although, the 2nd one could be a backup, that's always online also. Hmm, gonna think about it some more...

Link to comment
14 minutes ago, shEiD said:

gonna think about it some more...

You can probably find the beginnings for a backup server at a garage sale.  The MB and CPU requirements are rather light for a strictly NAS server.  (If you want dual parity and fast parity check speeds, you need a CPU no later than ~2015.  But slow parity checks on a backup server are not near the issue that they can be an production server...)

Edited by Frank1940
Link to comment

I have an old i7 rig (about year 2010, IIRC) I could use. How would that cope with dual parity?

So unraid's attraction was, it was always running smoothly on old an not power-full hardware. Did that change with dual parity? I mean, forget docker and VMs - just using dual parity...?

Link to comment

Might want to look here:

https://www.backblaze.com/blog/hard-drive-benchmark-stats-2016/

https://www.backblaze.com/blog/hard-drive-failure-rates-q1-2017/

 

The 6T REDS don't have the best stats, at nearly 3% failure rate. The Seagate 8Ts are looking pretty good, and look at how many they bought! (That's usually a sign of a good value drive). They are not the archive version (at first I thought they were but they are not). Seagate was sued for quality issues and since their quality has improved. 

 

The HGST 8Ts are stellar, but they only bought a few because they were the Helium drives and high dollar.

 

There is nothing out on the WD 8T or 10T REDS.

 

I can tell you as a brand name the HGST have been the highest for some time now, and I tend to buy them. But Seagates were 3 for the price 2 HGSTs (24T for the price of 16T). So it was hard to resist. May pay for it in reliability, though, but it was a calculated risk.

 

The archives are good performers. 5900 RPM and high density. There is a possibility of it slowing down due to excessive random writes, but with media data it is normally sequential and the slowdown should never happen. Many users here use them successfully, even for parity, and report they work great. These are not recommended for RAID, but unRAID is not RAID.

 

Not trying to sell you on anything, just providing info.

Link to comment

@bjp999 @johnnie.black Thanks for info, guys. And I will look into those Seagates, if their reliability has become better. Because in my own experience - Seagates were the worst, actually more than half of mine are dead already. I really can't remember exactly, but I think I have 10+ 1.5TB and 10+ 3TB dead Seagate drives. At the same time - I have like ~30 old as hell 500GB and 750GB Samsung drives - most of them still work perfectly. My go to testing drives :) Maybe 3-5 dead from ~30, some I have stopped using just because SMART gave warning about reallocated sectors.

Link to comment

Haswell and newer Intel CPUs contain support for the AVX2 instruction set.  The unRAID dual parity calculations leverage this for faster performance, but your 2010 i7 should be able to handle dual parity without issue.  The AVX2 instruction set probably benefits low end processors the most since they have less raw horsepower to work with.

Link to comment
  • 2 years later...
On 7/14/2017 at 11:30 AM, shEiD said:

I am still kinda confused, how does dual parity work. Does it really mean, that having unRAID use dual parity I could recover from ANY 2 drives failing?

 

Like they say, a picture is worth a thousand words...

 

 

2017-07-14_05-20-29__chrome.png

 

2017-07-14_05-23-45__chrome.png

 

2017-07-14_05-24-55__chrome.png

Greetings,

 

Sorry for bumping this old thread but after quite intensive research and carefull reading of this thread, I still could not find documentation explaning how does *dual* parity work and how it can manage to recover data from *any* two drives (i.e. data and/or parity drive).

 

Many thanks for your replies.

 

G

Edited by Opawesome
typo
Link to comment
20 minutes ago, Opawesome said:

Greetings,

 

Sorry for bumping this old thread but after quite intensive research and carefull reading of this thread, I still could not find documentation explaning how does *dual* parity work and how it can manage to recover data from *any* two drives (i.e. data and/or parity drive).

 

Many thanks for your replies.

 

G

The parity2 calculation is different and independent of the parity calculation. The exact algorithm is a bit more complicated and I haven't really tried to understand it in detail, but the fact that these calculations are different and independent is what allows them to recover 2 drives. The contents of parity and parity2 are different, so these disks are not interchangeable and cannot be swapped without rebuilding parity.

 

Here is a thread I found on the forum that discusses it a bit more. If you google "Q parity" you can probably find more technical discussions.

 

https://forums.unraid.net/topic/51362-pq-or-dual-parity-in-detail/

 

Link to comment
43 minutes ago, trurl said:

The parity2 calculation is different and independent of the parity calculation. The exact algorithm is a bit more complicated and I haven't really tried to understand it in detail, but the fact that these calculations are different and independent is what allows them to recover 2 drives. The contents of parity and parity2 are different, so these disks are not interchangeable and cannot be swapped without rebuilding parity.

 

Here is a thread I found on the forum that discusses it a bit more. If you google "Q parity" you can probably find more technical discussions.

 

https://forums.unraid.net/topic/51362-pq-or-dual-parity-in-detail/

 

Hi trurl,

 

Many thanks. I will investigate towards that direction.

 

Have nice day,

 

G

Edited by Opawesome
typo
Link to comment
  • 7 months later...
On 7/14/2017 at 11:30 AM, shEiD said:

I am still kinda confused, how does dual parity work. Does it really mean, that having unRAID use dual parity I could recover from ANY 2 drives failing?

 

Like they say, a picture is worth a thousand words...

 

 

2017-07-14_05-20-29__chrome.png

 

2017-07-14_05-23-45__chrome.png

 

2017-07-14_05-24-55__chrome.png

Sorry for bumping this thread again, but I think, it's not answered corretly.

Tis question was answered with a "Yes" to "having unRAID use dual parity I could recover from ANY 2 drives failing?" and the following link was provided:

Quote

....but....

behind this link at the bottom of the page I found the following information:

Quote

For large arrays, ‘dual parity’ – or, the facility to have a second parity disc that is not simply a mirror of the first – would be useful. This would permit two simultaneous drive failures without losing data. unRAID does not have dual parity at present, but ‘P + Q redundancy’ is part of the future roadmap.(Dead Link)

In a P + Q redundancy system (as in a RAID-6 system), there would be two redundancy disks: ‘P’, which is the ordinary XOR parity, and ‘Q’, which is a Reed-Solomon code. This would allow unRAID to recover from any 2 disk errors, with minimal impact on performance.

Further discussion: [1], [2]

This means:

- the second parity disc is a mirror of the first one

- there is no possibility to recover from ANY 2 drives failing

- from the image above it's possible to recover from the 1st and 2nd scenario, but the third one sums up in data-loss

 

Or do I miss something?

cheers, d.

Edited by dingsda
Link to comment
12 minutes ago, dingsda said:

the second parity disc is a mirror of the first one

That's not correct 2nd parity uses a different calculation and isn't interchangeable with parity1, some info here.

 

13 minutes ago, dingsda said:

there is no possibility to recover from ANY 2 drives failing

There is, you can recover any two failed drives.

Link to comment
13 minutes ago, dingsda said:

This means:

- the second parity disc is a mirror of the first one

- there is no possibility to recover from ANY 2 drives failing

- from the image above it's possible to recover from the 1st and 2nd scenario, but the third one sums up in data-loss

 

Or do I miss something?

Parity2 uses a different algorithm to parity1 to calculate its contents so they are NOT mirrors.  Unlike parity1, the algorithm for parity2 includes disk slot number as one of its inputs.  I am not sure if Limetech have ever given details of the exact calculations that are used other than to say they are more demanding on the CPU than the ones used for parity1.

Link to comment

If you google    P+Q parity   you will find numerous articles on it.  The following link will take to to one of them

 

      https://igoro.com/archive/how-raid-6-dual-parity-calculation-works/

 

This post also discusses the Parity2 (or Q) calculation as it applies to Unraid. 

 

      https://forums.unraid.net/topic/49598-unraid-server-release-620-rc4-available/#comment-487759

 

This has come up before and I usually tell folks that to fully understand what it going on, you will have to had a graduate-level course in Matrix Algebra.  The problem with any proposed Dual Parity solution is that you must have a rigorous mathematical proof that the method being implemented actually works 100.000000% of the time.  Without that proof, no one would commit their data to being protected by it.  Imagine that you are the IT manager of Chase Manhattan bank and you had to explain to your Board of Directors that all of their customer account information had been lost because the disk failures 'happened' to be one of .00001% of the cases where data recovery fails because of a flaw in the parity scheme.

 

The proof for single parity is almost trivial in comparison!

Link to comment
  • 1 month later...

Hello, 
I have read through some of the posts here and got lost in some of the conversations. Thought to reach out on this topic as I am in the middle of an upgrade. I will try to keep this short and simple

Current array size in disk - 11 disk total
Current chassis physical drive bays - 12 disk total

Parity was upgraded successfully from 6TB to 10TB
Disk 3 was upgraded (currently rebuilding) 4TB to 10TB

I currently have 1X physical drive bay OPEN for one of the following purposes that I was considering:
01. Dual parity (add another 10TB disk in the future)
02. Data disk (add more storage space)
**If i go with option 01, does the second parity disk need to match @ 10TB or can it be 8,6,4,etc?**


As there are many experienced users in the community, and what you would recommend? Option 01 or Option 02?
Being my array is not large in physical disk (currently 11 disk), would dual party would be worth wild?
As disk size in the world today has improved, I am remaining with this chassis at the 12X drive bay limit.

Suggestions (if you were me)?

Thanks,

Edited by bombz
Link to comment
18 minutes ago, bombz said:

Disk 3 was upgraded (currently rebuilding) 4TB to 10TB

18 minutes ago, bombz said:

does the second parity disk need to match @ 10TB or can it be 8,6,4,etc?

Each parity disk must be at least as large as the largest single data disk. Since disk3 is 10TB parity2 must be at least that large.

 

Link to comment
6 minutes ago, trurl said:

Each parity disk must be at least as large as the largest single data disk. Since disk3 is 10TB parity2 must be at least that large.

 

OK.
Good to know when running dual parity, both parity disks need to be as large or larger then the largest, so same rules apply.
I thought I could get away with a smaller disk as parity2, now thinking about it, that wouldn't make sense.

Now the decision time. Do I run single parity, or dual parity, hmm hmm

Link to comment
21 minutes ago, bombz said:

As there are many experienced users in the community, and what you would recommend? Option 01 or Option 02?

Depends on your adversion (or tolerance) to Risk!   Single parity will permit you to recover from a single disk failure.  That part is obvious.

 

Dual parity allows you to recover from a two disk failure.  While it is possible for a prudent person to monitor his server so that a two disk failure 'would' not occur because at the first sign of a problem with a disk, he would replace it with a new disk.  But what would happen if a second disk were to start acting up while the first disk is being rebuilt.  One could lose the contents of both disks in a single parity situation.  With dual parity, the rebuild would simply continue and the second disk would be flagged as having a problem.  (Which of course a risk- adverse person would address immediately!) 

 

For more information, see this thread:

 

    https://forums.unraid.net/topic/50504-dual-or-single-parity-its-your-choice/

 

By the way, it appears that failure rates of hard disks are about the same for low capacity disks as for high capacity disks.  So you should always opt to replace a small capacity drive with a larger rather than adding another small capacity disk to the array when adding capacity to the array.  It may cost more initially but that older small capacity drive is nearer to its end-of-life and you will be doing it anyway at some time in the future.  Always be looking at the sweet spot of cost/TB when buying disks...

 

Link to comment
7 minutes ago, Frank1940 said:

Depends on your adversion (or tolerance) to Risk!   Single parity will permit you to recover from a single disk failure.  That part is obvious.

 

Dual parity allows you to recover from a two disk failure.  While it is possible for a prudent person to monitor his server so that a two disk failure 'would' not occur because at the first sign of a problem with a disk, he would replace it with a new disk.  But what would happen if a second disk were to start acting up while the first disk is being rebuilt.  One could lose the contents of both disks in a single parity situation.  With dual parity, the rebuild would simply continue and the second disk would be flagged as having a problem.  (Which of course a risk- adverse person would address immediately!) 

 

For more information, see this thread:

 

    https://forums.unraid.net/topic/50504-dual-or-single-parity-its-your-choice/

 

By the way, it appears that failure rates of hard disks are about the same for low capacity disks as for high capacity disks.  So you should always opt to replace a small capacity drive with a larger rather than adding another small capacity disk to the array when adding capacity to the array.  It may cost more initially but that older small capacity drive is nearer to its end-of-life and you will be doing it anyway at some time in the future.  Always be looking at the sweet spot of cost/TB when buying disks...

 

Thanks for your input.
I am juggling the thought if I do or don't. Dual parity may be a year out still, to match the 10TB disk.
I am often monitoring my server daily if not, weekly, and review SMART, and if there are any error counts within the 'main' tab menu. Generally if I see SMART or errors start to populate (as it has happened in the past) I immediately swap the disk as soon as I can.
Trying to think if I burn another 10TB on dual parity setup for the satisfaction and protection.

I have some thinking to do on that empty drive bay, as I do have some 4TBs sitting around after this upgrade that are not being used. Never a bad idea to have hot swap available for the current 4s in the array currently.

I will think on it. Thanks again!

cosmic.JPG

Edited by bombz
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.