One of our key ?ndings has been the lack of a consistent pattern of higher failure rates for higher temperature drives or for those drives at higher utilization levels.

So their findings indicate that high utilization is not correlated to higher failure rates.  Doesn't this disagree with your point KYThrill?  I believe you are trying to make the point that the more you use (i.e. read from) a drive the sooner you are likely to encounter an error which will eventually lead to a failure.

 

My experience, from more than 30 years working in the IT industry, is that most drive failures are age-related, not use-related.

 

This concentration on read errors seems to be ignoring any write activity.  If a failure really was going to depend on the number of reads, being, in some way, wear related (This itself is illogical - can I read a single sector 10^15 times before I would expect it to exhibit a hard error, as long as I don't read any other sectors???  If I spread those reads over two sectors, does a failure become more, or less, likely???)) do the number of writes have no effect on the likelihood of a read error?

 

Oh yes.  There are write errors to consider.  I didn't even include those, because they are statistically shown to occur 10 to 100X less frequently.  I didn't included miscorrection either, because it occurs even less frequently than write errors.

 

The exact cause (if there is just one cause) of URE's is undefined.  It could be caused by bad head spacing, a defect in the platter (warped, missed surface defect, bad surface coating, etc), a tiny drop of lubricant somehow getting on a platter, wear from multiple reads or writes, etc., etc.  Some of these things, like wear, may take 100's of cycles to occur.  Other things, like a lube leak, might not happen until the first time the drive hits 40C, which might be a day or week, or year later.

 

It's just that after having tested a number of drives using a test method unknown to us, the aggregate odds of a URE occurring (whatever the cause) are published.  You can only assess your risk management with the data you have, not what you speculate might be the case.  Unless you have superior empirical data, then the theoretical data must take priority.

 

Now, if you are someplace like Google, and you run a test program with 10X the number of drives that HDD manufacturers use, and you come up with different results, then you are probably safe going with what you have learned.  But at this time, the most knowledgeable source for URE is the drive manufacturers and what they publish.

 

The safest thing you can do as an end user is not use 2TB 10^14 drives in your unRAID.  It is an unnecessary source of risk.

Also, according to that paper, Google specifically did NOT include in the statistics those disks that did not pass their initial burn-in tests.  They burn-in their drives before adding them to their servers because of the high failure rate detected during the burn-in process.

 

Before being put into production, all disk drives go

through a short burn-in process, which consists of a

combination of read/write stress tests designed to catch

many of the most common assembly, con?guration, or

component-level problems. The data shown here do not

include the fall-out from this phase, but instead begin

when the systems are of?cially commissioned for use.

Therefore our data should be consistent with what a reg-

ular end-user should see, since most equipment manu-

facturers put their systems through similar tests before

shipment.

 

Although manufacturers of PCs may put their units through a burn-in phase, unRAID users with OEM and RETAIL drives must do it themselves using something similar to the pre-clear script in order to get failure rates similar to the google statistics.

 

 

So KYThrill, what 1 TB drives do you buy?  The EADS models?  Because there are 1 TB EARS out now as well.  I think there are even 500 GB EARS.

 

While the price per GB of a 1 TB may be cheaper or equivalent to a 2 TB, the overhead cost is still higher since you will require twice as many SATA ports, drive bays, etc and you will use twice as much power.

 

Well, one thing to keep in mind is that with 1TB and smaller discs, 10^14 URE ratings become more acceptable (10^15 or better would still be preferred).  So the field is open.

 

But I use a mixture of 7200 and 5400 rpm discs from a variety of manufacturers.  I don't have any two drives that are the same.  And don't forget about the 1TB EAVS model.  You can get an EAVS or a EADS model if you want to avoid advance format, get 10^15 URE rating, and go WD.

 

Also, overhead costs are only more is you assume you will never grow your unRAID beyond a set point.  For example, lets say I have to buy a SATA card and two 4 n 3's to get a seven drive (6TB array).  With a 4 drive array (6TB), you would at least have to get a 4 n 3 icy dock.  So initially the extra overhead is the cost of a supermicro and a icy dock.

 

However, your 6TB system is now full and can't be expanded without buying a supermicro and another icy dock.  So you overhead is only less, if you assume you never go beyond 6TB.  If you need to go beyond 6TB, as you can in the first system, your overhead becomes the same.  Pay now or pay later, but you pay just the same...

 

Power use could be higher, but how much depends on your usage.  If my three extra drives are always spun down, the extra power used will be minimal.  If the typical usage of both systems is one data and parity disc spun up, then the difference in idle power consumption of 2 drives vs 5 is probably very small (depends on the drives).  Drives with 5V stand-by vs drives with 12V standby could make power consumption in that case equal, or even less for the system with more drives.

 

It isn't as clear cut as you make it out to be.

Well, I was working with your '12 drive max' philosophy.  Your 12 drive server of 1 TB drives gives you 11 TB storage capacity.  If I built a similar server with 2 TB drives instead, it would take 7 drives to match your 11 TB capacity (and I would actually have 12 TB at that point).

 

So in your server you need 12 drive slots (say 6 on the motherboard, 8 on a SuperMicro card and two of your SATA ports are unused.  You'll also need two breakout cables, and 3 x IcyDock 4-in-3s).  Roughly, that's about $600 (I'm saying $70 for the motherboard, $100 for the SuperMicro card, $20 each for the breakout cables, and $130 each on the IcyDocks).

 

In my server I would need 7 drive slots (6 on the motherboard, 2 on a PCIe x1 card and one of my SATA ports would be unused.  Also 2 x IcyDock 4-in-3s, one bay being unused).  That's roughly $350 ($70 for the mobo, $20 for the PCIe x1 card, and $130 each for the IcyDocks).

 

If we assume that the hard drives are about the same price per GB (even though 2 TB drives are generally a bit cheaper), you can see that my server is a full $250 cheaper, and it actually has an extra TB of capacity and a slot for one more drive if I choose to use them.  Obviously this discounts other factors, such as the case (my case could be smaller and cheaper as it only needs to have 6 5.25" bays instead of 9).

 

A rough example, but I think it works.

The Google paper doesn't include their definition of high utilization

Yes it does.  See paragraph 3.3 - Utilization for their definition of utilization.

It is dif?cult for us to arrive at a meaningful numerical utilization metric given that our measurements do not provide enough detail to derive what 100% utilization might be for any given disk model. We choose instead to measure utilization in terms of weekly averages of read/write bandwidth per drive. We categorize utilization in three levels: low, medium and high, corresponding respectively to the lowest 25th percentile, 50-75th percentiles and top 75th percentile. This categorization is performed for each drive model, since the maximum bandwidths have signi?cant variability across drive families.

So according to their definition preclearing a disk would be high utilization since it is using 100% of the disk's bandwidth throughout the process.  (Correct me if I'm wrong Joe L.)

 

The Google data is compiled from the hard drives used in Office PC's, their web servers, etc.

Wrong again.  These hard drives were employed only in rack-mounted servers.  The disk population is described in paragraph 2.2 - Deployment Details.

The data in this study are collected from a large number of disk drives, deployed in several types of systems across all of Google’s services. More than one hundred thousand disk drives were used for all the results presented here. The disks are a combination of serial and parallel ATA consumer-grade hard disk drives, ranging in speed from 5400 to 7200 rpm, and in size from 80 to 400 GB. All units in this study were put into production in or after 2001. The population contains several models from many of the largest disk drive manufacturers and from at least nine different models. The data used for this study were collected between December 2005 and August 2006.

As is common in server-class deployments, the disks were powered on, spinning, and generally in service for essentially all of their recorded life. They were deployed in rack-mounted servers and housed in professionally-managed datacenter facilities.

 

Did you read the article that you cited?

Well, I was working with your '12 drive max' philosophy.  Your 12 drive server of 1 TB drives gives you 11 TB storage capacity.  If I built a similar server with 2 TB drives instead, it would take 7 drives to match your 11 TB capacity (and I would actually have 12 TB at that point).

 

So in your server you need 12 drive slots (say 6 on the motherboard, 8 on a SuperMicro card and two of your SATA ports are unused.  You'll also need two breakout cables, and 3 x IcyDock 4-in-3s).  Roughly, that's about $600 (I'm saying $70 for the motherboard, $100 for the SuperMicro card, $20 each for the breakout cables, and $130 each on the IcyDocks).

 

In my server I would need 7 drive slots (6 on the motherboard, 2 on a PCIe x1 card and one of my SATA ports would be unused.  Also 2 x IcyDock 4-in-3s, one bay being unused).  That's roughly $350 ($70 for the mobo, $20 for the PCIe x1 card, and $130 each for the IcyDocks).

 

If we assume that the hard drives are about the same price per GB (even though 2 TB drives are generally a bit cheaper), you can see that my server is a full $250 cheaper, and it actually has an extra TB of capacity and a slot for one more drive if I choose to use them.  Obviously this discounts other factors, such as the case (my case could be smaller and cheaper as it only needs to have 6 5.25" bays instead of 9).

 

A rough example, but I think it works.

 

Yes, but by the time I fill 11TB, all 2TB are 10^15 and unRAID supports 4k drives and all the bugs are hashed out.  I've spent $250 more, but then I'm fully ready to go to 22TB, which you can't touch without dumping an equal amount of money into the other system.  Except parts have been discontinued and you can't put together matching docks, etc, so mine looks cleaner too.

 

Yes, it will be more money, but that is how the trade-off works.  You can trade a little money for less risk, or trade more risk for less money.  There are many ways to build a system.  You can design it around cost, around risk, around performance, are capacity, etc.  To me, $250 isn't that much money.  My last weekly trip to the grocery was $198.  Sadly, when I was a kid, $20 was a fortune.  After college, I had to carry $20's like I used to carry ones.  Now it seems like $200 is the new $20.  It's $50 to fill my gas tank, which I do three times a week.  When I first started driving it was $15.  So in the scheme of things, $250 on a one time purchase that will last a couple of years isn't something I worry about.  That is just marginally more.

 

For a college student, that may me a whole months worth of disposable income.  It's all relative...

The Google paper doesn't include their definition of high utilization

Yes it does.  See paragraph 3.3 - Utilization for their definition of utilization.

It is dif?cult for us to arrive at a meaningful numerical utilization metric given that our measurements do not provide enough detail to derive what 100% utilization might be for any given disk model. We choose instead to measure utilization in terms of weekly averages of read/write bandwidth per drive. We categorize utilization in three levels: low, medium and high, corresponding respectively to the lowest 25th percentile, 50-75th percentiles and top 75th percentile. This categorization is performed for each drive model, since the maximum bandwidths have signi?cant variability across drive families.

So according to their definition preclearing a disk would be high utilization since it is using 100% of the disk's bandwidth throughout the process.  (Correct me if I'm wrong Joe L.)

 

The Google data is compiled from the hard drives used in Office PC's, their web servers, etc.

Wrong again.  These hard drives were employed only in rack-mounted servers.  The disk population is described in paragraph 2.2 - Deployment Details.

The data in this study are collected from a large number of disk drives, deployed in several types of systems across all of Google’s services. More than one hundred thousand disk drives were used for all the results presented here. The disks are a combination of serial and parallel ATA consumer-grade hard disk drives, ranging in speed from 5400 to 7200 rpm, and in size from 80 to 400 GB. All units in this study were put into production in or after 2001. The population contains several models from many of the largest disk drive manufacturers and from at least nine different models. The data used for this study were collected between December 2005 and August 2006.

As is common in server-class deployments, the disks were powered on, spinning, and generally in service for essentially all of their recorded life. They were deployed in rack-mounted servers and housed in professionally-managed datacenter facilities.

 

Did you read the article that you cited?

 

It doesn't say anything about 100% bandwidth being defined as high utilization.  Did you read the article?  It says high utilization drives were those that fell in the top 75 percentile of that model #'s weekly utilization.  A one time preclear (even if it is 100% utilization) does not mean the weekly average is 100%.  Unless you precleared 24 hours a day/ 7 days a week, you would not be at a weekly average of 100%.

 

By their own definition, there could have been 40 Seagate drives who each transferred 50MB per week, but 10 drives transferred 100MB, so they get categorized as high utilization.  Seems like a pretty poor way to define it.

 

I also doubt unRAID would be "high utilization", because as you quoted, all drives were powered on and spinning for essentially their entire life.  unRAID spins down, so unless you use your unRAID so heavily that it never spins down any drives, then you will fall outside of the scope of the definitions used in the paper.

 

And sorry I didn't see that line about rack mount servers, all I remember is that it said they were deployed in several different types of systems.  To me if they are all in rack mount servers in data centers, then that wouldn't be several different types of systems.

It doesn't say anything about 100% bandwidth being defined as high utilization.  Did you read the article?  It says high utilization drives were those that fell in the top 75 percentile of that model #'s weekly utilization.  A one time preclear (even if it is 100% utilization) does not mean the weekly average is 100%.  Unless you precleared 24 hours a day/ 7 days a week, you would not be at a weekly average of 100%.

For most 2TB drives the preclear & parity sync process might take about 35-40 hours to do.  Now although it's not a week like the study used but for that period of time the drive would have to be classified as high utilization.  How could it not, it's using 100% of it's bandwith?  Granted it's not for an entire week but I was speaking about the period of time that it was preclearing. This is really irrelevant since this study of over 100,000 drives already concluded that drive utilization had no correlation to failure rate.

 

Now maybe higher utilization would lead to a higher URE rate and potential data loss but as the MS study we previously discussed showed there was no correlation there either.

Please be careful with how you say things like this.  These URE's happen based on a probability.  If one occurs, anther could occur the same day or 20 years from now - anytime.  You can't say "you will have another 6-7 months before you potentially encounter another URE".  I enjoy debates like this one, but you lose credibiilty with this type of inaccuracy.

 

It is not inaccurate.  It is inaccurate to assume that in more cases than not, the statistics will not play out as defined (unless their is a fault in determining the statistics).  If I say that 50% of the time, a coin toss will come up heads.  It is inaccurate to assume that every time I flip it, it will be tails.

 

But, your saying "you will have another 6-7 months before you potentially encounter another URE", is like saying I've just tossed the coin and it came up heads therefore, because we get 50% heads and 50% tails potentially the next toss will come up tails.

 

This is clearly untrue, because each toss has a 50% chance and the result is totally unaffected by what came up before.

 

In the same way, "because I've just had a URE, it's going to be another 7 months before I get another" is a complete fallacy.  The way the specs are presented, each read has a 1 in 10^15 chance of failing, irrespective of what happened on the last read.

Interesting side discussion on UREs, even if it includes a lot of blatantly false information and perspectives.

Please be careful with how you say things like this.  These URE's happen based on a probability.  If one occurs, anther could occur the same day or 20 years from now - anytime.  You can't say "you will have another 6-7 months before you potentially encounter another URE".  I enjoy debates like this one, but you lose credibiilty with this type of inaccuracy.

 

It is not inaccurate.  It is inaccurate to assume that in more cases than not, the statistics will not play out as defined (unless their is a fault in determining the statistics).  If I say that 50% of the time, a coin toss will come up heads.  It is inaccurate to assume that every time I flip it, it will be tails.

 

But, your saying "you will have another 6-7 months before you potentially encounter another URE", is like saying I've just tossed the coin and it came up heads therefore, because we get 50% heads and 50% tails potentially the next toss will come up tails.

 

This is clearly untrue, because each toss has a 50% chance and the result is totally unaffected by what came up before.

 

In the same way, "because I've just had a URE, it's going to be another 7 months before I get another" is a complete fallacy.  The way the specs are presented, each read has a 1 in 10^15 chance of failing, irrespective of what happened on the last read.

 

I don't think you have a full grasp of probability and statistics.  I'm talking about a theoretical sequence of events that have not started yet.  From this reference point, it is correct of me to say that the odds of any individual bit read resulting in a URE is 10^14 (on a 10^14 rated drive).  AND that the odds of me having back to back URE errors is 10^28 (it is not 10^14 as you claim).  As you do the math, it is more likely for the URE's to be spaced out that it is for them to come close in frequency.  It is not a fallacy for me to make that observation.  Nor do I loose any credibility among those who understand probability.

 

However, if your reference point is from a real life URE has just occurred, then the odds of the next bit being read having a URE is 10^14, if you assume that the two events are independent of each other.  I'm not sure it would be correct to assume that.  Can you show that the occurrence of URE's is an independent event?  And if they are dependent, then a different set of odds calculations would occur, depending on what the actual cause of the URE was.  The odds could either increase or decrease from 10^14.

 

What you are stuck on is the Gambler's Fallacy, but you are not applying it correctly.  If I say that I am going to flip a coin four times in a row, before the first coin is ever flipped, I can accurately state that the odds of the theoretical sequence of heads being flipped four times in a row is 1/16.  However, after the first coin has been flipped, it is no longer 1/16.  Each flip is independent so after each flip, the odds of heads or tails is 50/50 on the next flip.  Just because I flipped heads three times in a row does not make it more or less likely that the next flip will be heads.

 

To the best of my knowledge, all I've proposed here is a theoretical sequence of events.