http://www.enterprisestorageforum.com/technology/features/article.php/3839636

 

Unfortunately, many RAID implementations do exactly that -- a single read/write error and the drive is marked bad and taken offline, and a rebuild is required.  unRAID does it too with a write error. That can be avoided.  Taking the example from the article at face value, you get a second disk failure while rebuilding one out of 234 rebuilds, then you need to REDUCE the frequency of rebuilds.  If you have 1 rebuild a year, that means you're good for 200 years.  I'm OK with that.

 

But still, by using intelligent remapping with a single error rather than taking the whole drive offline and requiring a rebuild, you can drastically reduce the number of rebuilds.

 

This is another reason to have an intelligent drive removal (i.e. zero it out and remove it).

 

While I do have some issues with the article, it is better than the previous one on this topic.  The second is a significant flaw in the article itself is that it does not account for the drastic reduction of price in storage.  As the article itself states, "Yes, you could mirror to get out of the disk reliability penalty box, but that does not address the cost issue."  Mirroring solves the "problem" and the cost issue is a red herring as a modern array, mirrored to avoid the problem the article claims exists, will still cost less than a system of the same size a few years ago.

 

But one thing is clear:  RAID developers need to be adopting strategies to AVOID rebuilds unless absolutely necessary. 

 

Taking the example from the article at face value, you get a second disk failure while rebuilding one out of 234 rebuilds, then you need to REDUCE the frequency of rebuilds.  If you have 1 rebuild a year, that means you're good for 200 years.  I'm OK with that.

The article says:

A single 8+2 rebuild with 1.5 TB drives reads and writes about 28.5 TB (read 1.5 TB*9 drives, write 1.5 TB*10 drives)

When rebuilding a drive we do NOT write 1.5TB * 10 drives...  unRAID would write 1.5TB on one drive.

 

Yes, we are still at risk of a read error, but the estimate of 28.5TB is probably twice what unRAID would experience.  That would seem to indicate I'm good for about 400 years.      Now, personally, I'm guessing my old IDE based array will probably be retired by then... and me too.

 

Joe L.

I don't think the math in that article is quite right.  Here's the issue as I understand it:

 

1. you have a drive failure, hence you need to rebuild to a replacement drive

2. while you are rebuilding you are no longer fault tolerant

3. because you are no longer fault tolerant, if you have an "uncorrectable read error" (URE) on any of the other (still good) drives or the parity drive you cannot replace this block by using the other drives to simulate it.

4. so any URE (while the array is not fault tolerant) will cause data loss

 

Now lets assume you have an unRAID array with 10 disks in it, if they are seagates then the URE is less than one bit in error out of 1x10^14 bits read.  If these are all 1TB drives then while rebuilding on failed disk we have to read the complete contents of the other 9 drives, so we read 9TB of data, do the parity calcs and write 1TB of simulated data to the replacement disk.  9TB of data read is 72Tb (bits) which is 72x10^12, so dividing: 1x10^14 / 72x10^12 we get 1.39, so there is only a safety factor of 1.4, which sounds pretty slim.

 

Note some sources report that actual URE rates about a factor of 3 less than what the manufacturers are stating, which would bring us up to a factor of 5. Still that's not much.  Stated another way there is something like a 20% to 70% chance that any time you rebuild a 1TB disk in a 10 drive array that you are going to get a 1-bit read error that you can't do anything about.

 

If you switch to western digital drives then you get a factor of 10 improvement as they quote a 1x10^15 URE level.

 

Now the practice of periodically running a parity check may help reduce the URE rate, but I can't help but wonder how many of the errors that people get from time to time while doing parity checks are being caused by this URE effect. My experience seems to be that out of a few parity checks, I get one which reports an error or two that were corrected - this would seem to be about the right level for a seagate based system or on the high side for a western digital system (I've got 8 drives and about half are from WD).

 

Based on this I'm going to keep buying western digital drives instead of seagates.

 

Also, what really happens on an unRAID system if it encounters a URE while rebuilding a failed drive? From other comments it sounds like it does not abort the rebuild and in the end you will have a repaired array but somewhere on your rebuilt drive there is a block of data with a 1 bit error on it.  Does this get logged? Is there any way of discovering what file (if any) contains the flipped bit?

 

Comments?

 

so we read 9TB of data, do the parity calcs and write 1TB of simulated data to the replacement disk.  9TB of data read is 72Tb (bits) which is 72x10^12, so dividing: 1x10^14 / 72x10^12 we get 1.39,

 

No.  That is no how you do the math.

 

You do the math for 1 drive, take the inverse, and take that to the n power where n is the number of drives, then invert it back again.

 

You take the NON-error rate for 1 drive (the inverse of the error rate) and raise it to the n power, and that gives you the likelihood of all drives NOT having an error... then invert that to get the rate of failure.

 

Inverting here means 1-x, not 1/x.

 

So the result is 1 out of 139.3 rebuilds, you would get an error while rebuilding 9x1TB array... or one error per 1.25 pentabytes of rebuilding activity.

 

 

Also, what really happens on an unRAID system if it encounters a URE while rebuilding a failed drive? From other comments it sounds like it does not abort the rebuild and in the end you will have a repaired array but somewhere on your rebuilt drive there is a block of data with a 1 bit error on it.  Does this get logged? Is there any way of discovering what file (if any) contains the flipped bit?

 

Has anyone reviewed the MD driver enough to know that a "unrecoverable" read error will be skipped and the md driver will keep going.

In any case with our drives and arrays getting bigger and bigger it seems that checking with MD5 and protecting with PAR will be a worthwhile investment.

Now the practice of periodically running a parity check may help reduce the URE rate, but I can't help but wonder how many of the errors that people get from time to time while doing parity checks are being caused by this URE effect. My experience seems to be that out of a few parity checks, I get one which reports an error or two that were corrected - this would seem to be about the right level for a seagate based system or on the high side for a western digital system (I've got 8 drives and about half are from WD).

 

Don't think your experience is typical.  I NEVER get sync errors unless working through some problem.  I have never seen a user post -" I just ran a parity check and got 1 sync error - what should I do?"  Seems reasonable you'd have one in a blue moon - just hasn't happened unless others like you are keeping it to themselves.  The specs are the specs, but in the collective experience here, bit read errors are much less likely than true drive failures.

So the result is 1 out of 139.3 rebuilds, you would get an error while rebuilding 9x1TB array... or one error per 1.25 pentabytes of rebuilding activity.

Actually, that is one out of 139.3 parity rebuilds, or parity checks.

 

Now, it will take any one of us a while to perform 139 checks... but collectively on this forum, between all the users of unRAID posting, it will not take that long... especially if they do monthly checks as we suggest.  The occasional statistical "read" error will occur, and in an array of 10 disks it 9 times as likely to occur on a read of a data drive than the parity drive.  The parity drive would be "corrected" based on the incorrectly read "bit"

 

A subsequent parity check should fix parity once more, assuming the data bit can be properly read and is not a permanent failure. It too would show a error as it updates parity once more.

 

For that reason, the new "Verify-but-do-not-correct" feature in release 4.5beta2 onward of unRAID will be even more important in the future.

 

Anybody currently running a monthly check via cron, and using the "mdcmd check" command can change their script to one that does not update parity, but reports the errors by using

mdcmd check NOCORRECT

as the command in their script. 

 

It looks identical on the unRAID management console, it will still show errors, and it currently does not tell you the addresses of the errors (that would be nice so analysis would be easier)

 

The NOCORRECT version of the parity Verify might still show a rare parity mismatch as a result of a "read" error, but a subsequent verify would show all is OK if the bit can properly be read.

 

Joe L.

 

 

The occasional statistical "read" error will occur, and in an array of 10 disks it 9 times as likely to occur on a read of a data drive than the parity drive.  The parity drive would be "corrected" based on the incorrectly read "bit"

 

I don't believe that is correct.  You are assuming that the read error would be passed to the OS undetected, so the OS thought the data was "good" and thus mismatch the parity and cause parity to be rewritten (using the data with an error).

 

My understanding that the read error would be passed to the controller, which itself would retry the sector some number of times, and then the OS driver would have an opportunity to retry.  If an error-free read was not accomplished, unRAID would reconstruct the data from parity, and rewrite it to the data drive, to allow the drive remapping to take care of it.

If you switch to western digital drives then you get a factor of 10 improvement as they quote a 1x10^15 URE level.

In terms of the math and calculating probabilities of failures....How is the URE rate calculated by these HDD companies?  Is this the average URE rate?  Is it the 95% confidence interval rate, etc? 

 

Numbers usually only mean something if you know how they are stated.  A bag of M&M's says it has some number of ounces of candy in it, say 3 ounces.  very few bags will have *exactly* 3 ounces, but they set up the packaging systems such that some percentage of M&M bags will have 3 or more ounces (95%, 98%, 99%, etc).  The URE rate could be a 95% confidence rate as well, meaning 95% of hard drives will not see this error in, say, 1x10^15 bit reads, but the typical rate could be several orders of magnitude higher.  And of course that would mean 5% would be lower.

 

Fun with statistics.

A single 8+2 rebuild with 1.5 TB drives reads and writes about 28.5 TB (read 1.5 TB*9 drives, write 1.5 TB*10 drives)

 

When rebuilding a drive we do NOT write 1.5TB * 10 drives...  unRAID would write 1.5TB on one drive.

 

Raid 1, 5 or 6 wouldn't write 1.5TB*10 drives either. The only time that would happen is a complete restore, not a rebuild!

 

Sure raid isnt perfect but most of the problems laid out in this article arent raid issues, some of the proposed solutions have the same issues. Distributed or cloud fs are still fundamentally raid 1 or multiple raid 1 file systems stored on hdds somewhere.

 

Raid is not and never was a backup solution. Its a means of cheap fast storage. Sans and nas devices make storage more available and affordable than ever.

 

A lot of companies have been setup and a lot of money spent solving this"global storage issue", funnily enough none of them have made any money...

 

The big boys like google simply did their own thing, not only that they told the whole world how-to just for good measure. In case you missed it, same way google does everthing, lots of cheap nodes meshed to form bigger nodes, middleware to duplicate/manage data globally. Think backblaze storage pods.

 

The next tier is running big ass solaris/zfs solutions. The enterprise tier is running san based raid solutions. Everyone else, nas based raid or das based raid.

 

Seems the global storage problem didnt materialize. These same people are now claiming raid is dead.

Just a thought: isn't it better prior starting rebulding the existing parity drive, to pul it out and have it on side as a spare, put in a new (precleared and tested) drive and rebuild parity from scratch on this one? Comments?

Just a thought: isn't it better prior starting rebulding the existing parity drive, to pul it out and have it on side as a spare, put in a new (precleared and tested) drive and rebuild parity from scratch on this one? Comments?

 

I'm not sure I understand the logic in this segment. 

I think with this assumption, the drive which has failed is the parity drive.

 

What if one of the other drives has issues?

 

Also, if you have an unRAID array and a data drive dies, Pulling the parity drive and rebuilding parity will not reconstruct the failed data drive.

It will rebuild the parity drive without the failed drive, thus loosing all the failed drive's data.

No, if someone needs or wants to rebuild the parity drive (no data drives has failed), isn't it better to pull the actual parity drive to have as a backup and insert a new one and rebuild parity on that one? I gues this is only relevant if parity ever needs be rebuild whithout a data drive failing?!

I think there's some confusion as to what the parity drive does.  There's not data, just a number that says "All the other drives should add up to this".  Taking it out as "a backup" doesn't mean anything without all the other drives.  As soon as you write a file to one of the other drives the parity drive isn't right anymore - and in fact will UNDO the write if the old parity was used to rebuild the data drive.

 

It's not like a copy of all the data on all the other drives is stored on the parity drive.