Jump to content
jonp

Dual Parity

19 posts in this topic Last Reply

Recommended Posts

Implementation of RAID6-like P+Q redundancy.

Share this post


Link to post

Some great ideas are being discussed in this section, but for me this is the most important. 

 

In other threads about adding a second parity drive some "helpful person" always chimes in about how having a good backup solution is more important than the addition of this feature would be, and then it's another year before someone brings it up again.  If you don't want unraid to get this feature, then when it's finally implemented you don't have to use it. 

Share this post


Link to post

Indeed, this is an important one. Larger drives will mean smaller arrays for a lot of people and the potential for significant data loss is much greater. Many other NAS solutions: Freenas, snapraid probably others I don't know of already support such a feature. Its really a must have.

 

Backing up 30 TB of movies/tv shows is obviously not a feasible option for most people. Nor is it really a solution. A backup is for if you have a flood or a fire or some other natural disaster which is unforseen and otherwise impossible to recover from. A NAS should be able to recover from a few failed disks..

Share this post


Link to post

Anything important should be backed up and offsite. Having lived through Sandy, I can't tell you how important it is.

Scanned insurance documents, tax records and other forms of important documentation.

Then there is your families priceless pictures.

Dual parity will not help you in a disaster.

 

That being said, with the larger disks and arrays in our future, I'm all for Dual parity.

Having been bit by the double disk failure during a rebuild, it sure would have saved me some headache.

 

Share this post


Link to post

I agree with TSM.  I do backup my very important files off site but losing non critical data because of a dual drive failure is still a major loss for some people.  Now a days ISP data limits are becoming more and more common making people like me to be more selective of how we backup our data online.  A lot of people here use these servers to serve media files, I for instance have a 15TB array with 5 data drives, I can build the same size array with just 3 6TB drives.  If I loose 2 drives on my current array I can lose up to 6TB of data but on arrays that have larger drives like the 6TB ones you can lose 12TB of data.

 

I don't know about you but I would not like to have to go through my blu rays and have to rip all of them again.

 

Having dual parity is no substitute for backing up your data off site.  But we shouldn't also have to rely solely on off site backups in the event of dual drive failure.

 

In the industry I work in we have standby equipment and backup equipment.  Our standby equipment can be equivalent to having extra parity to keep us up and running and our backup equipment is like having off site data.  We would prefer to never even have to use the backup equipment but it is there in the event that it is needed.

Share this post


Link to post

...Larger drives will mean smaller arrays for a lot of people and the potential for significant data loss is much greater...

But smaller arrays decreases the risk of multiple drive failures, so a single parity drive is less of a problem for smaller arrays.

Share this post


Link to post

...Larger drives will mean smaller arrays for a lot of people and the potential for significant data loss is much greater...

But smaller arrays decreases the risk of multiple drive failures, so a single parity drive is less of a problem for smaller arrays.

 

Not sure I'd call it "less of a problem" ... although I agree the likelihood of a failure during a drive rebuild is lower, since there are fewer drives involved.    But with modern high-capacity drives, even an error rate on the order of 1 x 10^(-14) starts to not be very reassuring when drives have over 10^12th bits and many arrays have well over 10^13th bits.  A 2nd parity drive to protect against these is a very reassuring extra bit of "insurance" for our data.

 

 

Share this post


Link to post

Having a backup is essential to keeping your data safe but should not be you first and only option in the event of dual drive failure.  More than one drive can go bad from reasons other than hard drive hardware failure.  For instance back in the dual up days I had lighting it my phone line in my back yard, travel through my dad's modem, MB, NIC, hub, my NIC.  It destroyed the modem and my NIC only.

 

Electricity can do crazy things like travel thought components destroying some and not even hurting others.  Having that extra layer of protection could only be a benefit.

Share this post


Link to post

Anything important should be backed up and offsite.

 

True, but what's "important" is likely under 1TB out of 30TB for a typical user.  DP *is* the disaster recovery solution for most users because backing up 30TB is just not realistically doable for the average user.  Plus a proper backup needs versioning.  Even  then, offsite backups are not 100% bulletproof either.  Plus who wants to spend the time to restore 30TB from an offsite backup?  DP is about fault tolerance, and fault tolerance is all about staying in operation.

 

Plus DP is critical to bridge you while a first failed drive is rebuilt.

 

My vote is for diagonal parity, rather than P+Q.

 

 

 

 

Share this post


Link to post

... what's "important" is likely under 1TB out of 30TB for a typical user ...

 

So you think about 97% of what folks spend the $$ to build a fault-tolerant server for isn't important to them??    Interesting assumption.

 

 

... backing up 30TB is just not realistically doable for the average user.

 

It's may be more costly than some like to spend, but it's hardly difficult.    If you've kept good backups "along the way" it's not all that much incrementally -- a lot of the backup drives can simply be smaller drives you've replaced -- with a few dedicated backup drives purchased as needed.  A backup is just insurance against losing data you don't want to lose ... in some cases it's irreplaceable (records, pictures, etc.);  in other cases it would just be a major hassle to recreate (re-ripping hundreds/thousands of DVDs/BluRays;  re-rendering them to the format you want; cataloging; indexing; etc.).    Just a matter of what it's worth to never have to do that again.  The basic rule is simple:  If you don't want to lose it, back it up;  if you don't care if it's lost; then you don't have to bother ... but if that's the case, why bother to spend the $$ for a fault-tolerant server?

 

 

Plus DP is critical to bridge you while a first failed drive is rebuilt.

 

Certainly agree the biggest benefit to dual fault-tolerance is the ability to sustain a 2nd drive fault while you're rebuilding an initial failed drive.

 

 

 

 

 

Share this post


Link to post

Anything important should be backed up and offsite.

 

True, but what's "important" is likely under 1TB out of 30TB for a typical user.

 

And that was the point of my 'anything important' If your important stuff is 30TB, then DP is not a viable disaster recovery solution. People need to think wisely of what is crucial to their life and what is not. If' it's only 1TB, then back that up elsewhere and utilize DP or P+Q to alleviate having to reconstruct data for a possible dual drive failure.

 

I've been in both cases.

There was data lost because of a dual drive failure while recovering and there was data I needed just to get disaster relief and get my life back on track when shit hit the fan.

 

I'm not opposing dual parity or whatever the method is going to be, just be wise because it's not the answer for critical files.

 

We know Tom is working towards a multiple parity solution, it's only a matter of when.

Share this post


Link to post

 

So you think about 97% of what folks spend the $$ to build a fault-tolerant server for isn't important to them??    Interesting assumption.

 

Yes. Most of that data will be movies and music that can be restored from the physical copies they own. The fault tolerance is to save the time required from going from physical media back onto the server. Lets face it, if they don't own the physical materials they can simply steal it from the internet again like they did in the first place. After all, that is how most users are filling their media servers.

 

Share this post


Link to post

... I'm not opposing dual parity or whatever the method is going to be, just be wise because it's not the answer for critical files.

 

Absolutely.  NO fault-tolerant system is a substitute for backing up your important data.  Fault-tolerance is to keep your system running ... not to back it up.

 

... and if you only have 1TB of "important" data, then it's simple to just back it up on another system on your network;  to the cloud (Carbonite, etc.);  to an external drive;  etc.    My "important" data (financial records; scanned archival records; pictures; etc.) is always stored in at least 3 different places at home -- all UPS-protected and one a fault-tolerant UnRAID server -- AND on the cloud (Carbonite).    My UnRAID servers are all fully replicated -- but admittedly not backed up off-site [Although the backup disks are stored in a waterproof, fireproof safe].

 

 

Share this post


Link to post

I really dislike the fact that every time, EVERY TIME!!! Dual parity drives comes up as a topic, backups always become part of the conversation.  The necessity for dual or more parity drives, and the best methodology for backing up your unraid box are really 2 different discussions.  Both discussions have merit, and are somewhat related to each other, but they are 2 different discussions. 

 

 

Share this post


Link to post

...Larger drives will mean smaller arrays for a lot of people and the potential for significant data loss is much greater...

But smaller arrays decreases the risk of multiple drive failures, so a single parity drive is less of a problem for smaller arrays.

 

Not sure I'd call it "less of a problem" ... although I agree the likelihood of a failure during a drive rebuild is lower, since there are fewer drives involved.    But with modern high-capacity drives, even an error rate on the order of 1 x 10^(-14) starts to not be very reassuring when drives have over 10^12th bits and many arrays have well over 10^13th bits.  A 2nd parity drive to protect against these is a very reassuring extra bit of "insurance" for our data.

 

Yes its a bit counterintuitive, longer rebuild time definitely increases the likelihood of multiple failures and especially since the drive is so large, the potential for errors is greater as you say. This is precisely the reason why many people do not use large disks for RAID.

 

It's may be more costly than some like to spend, but it's hardly difficult.    If you've kept good backups "along the way" it's not all that much incrementally -- a lot of the backup drives can simply be smaller drives you've replaced -- with a few dedicated backup drives purchased as needed.  A backup is just insurance against losing data you don't want to lose ... in some cases it's irreplaceable (records, pictures, etc.);  in other cases it would just be a major hassle to recreate (re-ripping hundreds/thousands of DVDs/BluRays;  re-rendering them to the format you want; cataloging; indexing; etc.).    Just a matter of what it's worth to never have to do that again.  The basic rule is simple:  If you don't want to lose it, back it up;  if you don't care if it's lost; then you don't have to bother ... but if that's the case, why bother to spend the $$ for a fault-tolerant server?

 

Yep, of course you back up all the important data. But like I said, I cannot justify backing up 30TB of tvshows...sure I could recover it all from Bluray but its a huge hassle like you say.

 

I think its just a basic feature nowadays that a NAS should be able to recover from multiple failures...

Share this post


Link to post

I really dislike the fact that every time, EVERY TIME!!! Dual parity drives comes up as a topic, backups always become part of the conversation.  The necessity for dual or more parity drives, and the best methodology for backing up your unraid box are really 2 different discussions.  Both discussions have merit, and are somewhat related to each other, but they are 2 different discussions.

 

100% agreed and the exact sentiment that I think that garycase's statement reflects that sentiment as well:

 

Absolutely.  NO fault-tolerant system is a substitute for backing up your important data.  Fault-tolerance is to keep your system running ... not to back it up.

Share this post


Link to post

I really dislike the fact that every time, EVERY TIME!!! Dual parity drives comes up as a topic, backups always become part of the conversation.  The necessity for dual or more parity drives, and the best methodology for backing up your unraid box are really 2 different discussions.  Both discussions have merit, and are somewhat related to each other, but they are 2 different discussions.

 

I can see how that might be annoying, but consider that almost every time dual parity (I refuse to call it DP because I can't stop giggling) comes up someone says something that almost necessitates remind them, and everyone else that unraid is not a backup solution.  It literally cannot be said enough; it is that important.

 

...Larger drives will mean smaller arrays for a lot of people and the potential for significant data loss is much greater...

But smaller arrays decreases the risk of multiple drive failures, so a single parity drive is less of a problem for smaller arrays.

 

Not sure I'd call it "less of a problem" ... although I agree the likelihood of a failure during a drive rebuild is lower, since there are fewer drives involved.    But with modern high-capacity drives, even an error rate on the order of 1 x 10^(-14) starts to not be very reassuring when drives have over 10^12th bits and many arrays have well over 10^13th bits.  A 2nd parity drive to protect against these is a very reassuring extra bit of "insurance" for our data.

 

Exactly.  Without going into the actual math that could quantify it, generally consider that Risk = probability x consequence.  Bigger drives mean bigger consequence due to a single failure.  Without a drop in probability I start to get nervous thinking about housing more and more of my data on a any given single drive.  If that single failure probability was the same or lower as a smaller drive then it might not be such an issue. 

 

Consider a thought experiment: would you be willing to run a server with only one 30TB parity and one 30TB data drive.  Would you trust a parity rebuild to complete while reading/writing ~1/3 the amount of data expected before a bit error (3x10^13 vs 1x10^14)?  Remember BER is based on probabilities.  You could easily hit that first bad bit on the first bit read, the last, anything in the middle and multiples at that.

 

I mean sure you could go about recovering everything else using the right tools, but you'll have failed the parity rebuilt and your arse will have ingested your chair during the pucker!

 

The devil is in the math but there is no debating that there is a point at which risk becomes unaccaptable ... oh yeah that is what backups are for ;-)

Share this post


Link to post

I'm locking the thread.  Everyone knows what P+Q parity is.  Implementation is going to be a complex task which we cannot undertake until some other features are addressed (or unless someone comes to us and says, "hey we'll order 1000 servers but only if you have P+Q parity support."

Share this post


Link to post

...  we'll order 1000 servers but only if you have P+Q parity support.

 

:) :)

Share this post


Link to post
Guest
This topic is now closed to further replies.