I just wanted to chime in requesting dual parity support.

 

I had two samsung HD15UIs just die this weekend as I was adding 3 new drives. After shutdown they'd never spin up again. Both drives where about 80% full. And I had no backup, it's hard to backup 9TB of stuff. Just wanted to put out a warning on these drives. I've got the array rebuilding, and as soon as I'm able, i'm yanking the 3rd drive of that era out of the machine... 2 out of 3 failing at the same time is a little too scary for me.

 

Both drives where purchased in early 2010, both made 2-10.

 

Dual parity would have saved my bacon.

While I agree it's possible that both could die essentially simultaneously, it's such a small possibility that you really need to examine each of them one more time.  It's especially suspicious when you say they died while you were *adding* additional drives (perhaps overloading the PSU).  I would try reconnecting each, one at a time, preferably to a different controller and definitely different power cables, and even better to a different power supply.  If that is too inconvenient, try unhooking the power to all other drives before retesting these drives.

While I agree it's possible that both could die essentially simultaneously, it's such a small possibility that you really need to examine each of them one more time.  It's especially suspicious when you say they died while you were *adding* additional drives (perhaps overloading the PSU).  I would try reconnecting each, one at a time, preferably to a different controller and definitely different power cables, and even better to a different power supply.  If that is too inconvenient, try unhooking the power to all other drives before retesting these drives.

 

Rob, both drives have been hooked up independently to my desktop (500 watt antec PSU supporting 1 2500k w/ 1 SSD) ... neither of them powered up. Either I had the weirdest case of BIOS locking on the drives (ever) ... or I had a simultaneous case of motor controller failure ... considering both drives are from the same maker during the same date? I blame the failure on poor electronics. AKA motor controller failure. My server has had a 680 watt Seasonic PSU for the past year, I'd assume this is adequete for powering 10 drives. If I'm wrong, well, color me surprised.

 

Don't worry, I plan on continuing the investigation. It's 2.4 TB of data I'm losing. I don't take this lightly ....

 

But personally? If I had any other samsung drives in my system I'd run, not walk, away from them.

 

***Correction, it was a 620 watt seasonic PSU.  http://www.newegg.com/Product/Product.aspx?Item=N82E16817151096 to be specific

Also, more info on my dead drives @ http://lime-technology.com/forum/index.php?topic=28787.0 ... I'm not giving up. I'm going to sacrifice the last of the three drives in a circuit board swap to try to reacquire the data.

 

The loss of two drives *really* ticks me off. Serious, makes me hulk smash angry.

 

**** Ok, so I've modified this post mightily... Ultimately, what this has taught me is to not buy drives in batches. Vary the maker and date of make. Mix it up. It'll make it that much more random to have a double drive failure. Put 3 drives of one make, and one date in a system? That's asking for failure.

 

Unraid superbly supports drives of different makes and models. I should have been utilizing this from day one vs buying 3 or 4 of the cheapest drives available.

 

 

Ultimately, what this has taught me is to not buy drives in batches. Vary the maker and date of make. Mix it up. It'll make it that much more random to have a double drive failure. Put 3 drives of one make, and one date in a system? That's asking for failure.

 

 

I've been saying this for years. This happened allot in our hosting company where we would buy drives in batches from vendors.

Last ditch thing to try before giving up the ghost: Drop it from 8'.

Do you have any working hard drives around of the same make/model? If so, maybe you can try this (assuming your data is valuable enough):

 

1) make an immediate backup of any working drives

2) try replacing the controller circuit board of a dead drive with a working drive

3) see if you can backup any data off of the modified drive

 

of course, you'll need a few spare hard drives to do all these backups. you may need to buy them from different sources if you are trying to avoid the same batch.

 

guy

 

 

Ultimately, what this has taught me is to not buy drives in batches. Vary the maker and date of make. Mix it up. It'll make it that much more random to have a double drive failure. Put 3 drives of one make, and one date in a system? That's asking for failure.

 

 

I've been saying this for years. This happened allot in our hosting company where we would buy drives in batches from vendors.

I'm inclined to believe the batch problem is a more complex thing than just a manufacturing problem. I think the majority of batch failures are caused by some shipper in the supply chain dropping or otherwise abusing a master shipping box, containing sometimes 20 or more drives. That doesn't change how you deal with the issue, I'm just not sure it's the manufacturers direct fault. The assembly lines run hundreds or thousands of parts in a batch, so having only a few fail with the same issue makes me think the problem happens later in the supply chain.

Ultimately, what this has taught me is to not buy drives in batches. Vary the maker and date of make. Mix it up. It'll make it that much more random to have a double drive failure. Put 3 drives of one make, and one date in a system? That's asking for failure.

 

 

I've been saying this for years. This happened allot in our hosting company where we would buy drives in batches from vendors.

I'm inclined to believe the batch problem is a more complex thing than just a manufacturing problem. I think the majority of batch failures are caused by some shipper in the supply chain dropping or otherwise abusing a master shipping box, containing sometimes 20 or more drives. That doesn't change how you deal with the issue, I'm just not sure it's the manufacturers direct fault. The assembly lines run hundreds or thousands of parts in a batch, so having only a few fail with the same issue makes me think the problem happens later in the supply chain.

 

 

In my company we had problems with specific models of specific manufactures and specific batch numbers.

 

 

While what is said, makes sense, in my experience when we had these one after another failures, it was usually acknowledged by the vendor. I remember there being specific trouble in the early Hitach 1TB days and some WD 250s.  The Hitachi's were bad due to a design problem, the WD's had some kind of firmware issue and because we were doing RAID10 across 8 drives the issue was very visible to us.

I'm going to sacrifice the last of the three drives in a circuit board swap to try to reacquire the data.

 

Keep us posted, I'm interested in the result of this effort!

I will.

 

Right now I've got a spare unraid box doing parity so I can start moving data off of the old one. After parity completes, (around 2 am tonite) I'll start the move process. Once I'm happy with that, I'll sacrifice the old drives.

 

 

Be a few days/week (or two) until I get this finished. Oddly enough I find I keep working weekends to migrate ESX 3.5 servers to ESXi 4.1u3 servers.

 

As far as avoiding same batch issues I now vary my purchase dates, make and model. I'll buy a WD green, a WD Red, a toshiba (shouldn't done that one I don't think) some random seagates, etc. Just so I can different electronics.

 

 

Jon, even if it's not just a manufacturing problem, but is also tied into shipping and or other environmental concerns - wouldn't buying drives from different manufactures eliminate the issue?

 

I know one can't blame a single manufacturer for failures. I've had drives from seagate, conner, western digital, and quantum fail. Every drive dies, it's not a matter of if, it's a matter of when. Spreading it out would help ... just purely from a statistical value.

 

In my experiences? 2 out of 7 samsung drives died. That's ok. They died at the same time, that's *not* ok. If it had even been a *week* apart I'd be fine. I keep cold spares on hand for just this issue.

 

Ultimately, what this has taught me is to not buy drives in batches. Vary the maker and date of make. Mix it up. It'll make it that much more random to have a double drive failure. Put 3 drives of one make, and one date in a system? That's asking for failure.

 

 

I've been saying this for years. This happened allot in our hosting company where we would buy drives in batches from vendors.

I'm inclined to believe the batch problem is a more complex thing than just a manufacturing problem. I think the majority of batch failures are caused by some shipper in the supply chain dropping or otherwise abusing a master shipping box, containing sometimes 20 or more drives. That doesn't change how you deal with the issue, I'm just not sure it's the manufacturers direct fault. The assembly lines run hundreds or thousands of parts in a batch, so having only a few fail with the same issue makes me think the problem happens later in the supply chain.

Jon, even if it's not just a manufacturing problem, but is also tied into shipping and or other environmental concerns - wouldn't buying drives from different manufactures eliminate the issue?

As long as they weren't shipped in the same box at some point in time. Buying several different models from the same vendor simultaneously can bite you if they aren't packed well, or the shipper plays football with it.

 

I'd be inclined to mix it up when ordering as well. I've received single hard drives flopping around loose,but double boxed, and from the same vendor in another shipment everything was securely bubble wrapped and taped. It seems to be the luck of the draw as to which minimum wage worker packed which box.

Different makes, different models tend to ship on different pallets.

 

Manufactures tend to ship out X of the same product at once. Generally these products fill one to multiple pallets. They don't ship out a days worth of production on one pallet, unless it's one pallet of the same thing.

 

That's just common sense manufacturing process. One (big) vendor will get a pallet or two of the same product. Saves the vendor money, saves the manufacturer money.

 

 

I had no backup

 

... RAID (whether UnRAID, RAID-6, or whatever) is NOT a Backup.  Regardless of how "hard" it is, you should always have backups of anything you don't want to lose.

 

 

Dual parity would have saved my bacon.

 

As would a backup.    And a backup would have saved your "bacon" even if you had 3 drives fail at once ... or, for that matter, the entire array !!

 

I had no backup

 

... RAID (whether UnRAID, RAID-6, or whatever) is NOT a Backup.  Regardless of how "hard" it is, you should always have backups of anything you don't want to lose.

 

 

Dual parity would have saved my bacon.

 

As would a backup.    And a backup would have saved your "bacon" even if you had 3 drives fail at once ... or, for that matter, the entire array !!

 

Of course you should have a backup. Dual parity is a different solution for a different problem. As drives get bigger the cases of a second failure during a rebuild increase. Dual parity ensures that you don't lose your array and that doesn't go offline if this happens. Its a super important feature especially since we are going to 5TB by next year...no need to downplay it.

When was the last time you backed up 12 TERABYTES?

 

Even at work, this doesn't happen. Got a 12 TB array that's full? It has a double drive failure, you're boned.

 

Seriously 12 TB.

 

I had no backup

 

... RAID (whether UnRAID, RAID-6, or whatever) is NOT a Backup.  Regardless of how "hard" it is, you should always have backups of anything you don't want to lose.

 

 

Dual parity would have saved my bacon.

 

As would a backup.    And a backup would have saved your "bacon" even if you had 3 drives fail at once ... or, for that matter, the entire array !!

If you have a cost effective method of backing up 12TB, please by all means, let me know. I'd *LOVE* to know about it.

 

 

 

I had no backup

 

... RAID (whether UnRAID, RAID-6, or whatever) is NOT a Backup.  Regardless of how "hard" it is, you should always have backups of anything you don't want to lose.

 

 

Dual parity would have saved my bacon.

 

As would a backup.    And a backup would have saved your "bacon" even if you had 3 drives fail at once ... or, for that matter, the entire array !!

 

Of course you should have a backup. Dual parity is a different solution for a different problem. As drives get bigger the cases of a second failure during a rebuild increase. Dual parity ensures that you don't lose your array and that doesn't go offline if this happens. Its a super important feature especially since we are going to 5TB by next year...no need to downplay it.

I am with kc on this one. Every time I see/ask about issues with data storage, I get response of need backup, raid is not a backup well. Duh

 

Sent from my SGH-T889 using Tapatalk 4

 

 

You're absolutely correct. I think double drive parity should be the #1 priority in the next version of Unraid.

 

Yes, having backups is essentially a requirement for keeping data that is important, however once we hit the 6+ tb range of data size? Any kind of backup becomes improbable, actually impossible without a doubling of the price of the storage array. not to mention the time involved in synchronizing the backups over the associated link. This isn't including off-site replication.

 

I could *never* do an offsite replication of 12tb in 30 days. Not unless I had a fiber GB connection between the two sites, *OR* I devoted 5 days a week to swapping disks and one day a week to driving. Not possible.

 

Double Parity Protection may be seen by some who don't have large arrays as important, to those of us that have 10TB or greater arrays? (Which isn't that hard to do nowadays with 3tb drives becoming the norm) ... backups are impossible.

 

Hell if I was offered a triple parity protection? I'd attempt to do handstands, and *happily* buy 2 extra drives for the protection (of course, one seagate drive, one western digital drive)

 

Well you can always look into btrfs on opensuse. They just msrked it as stable. It offers most of the good stuff of zfs plus raid6 functionality.

 

Sent from my SGH-T889 using Tapatalk 4

 

 

Backing up that data is possible given a few circumstances.

 

1. You do the initial backups locally somehow, disk to disk or over a network with rsync.

2. Once the disks are backed up,  they do not change all that often.  Do we change all 12tb that often?

3. You use rsync to do a disk# to disk# backup over some kind of connection, local or remote.

 

I was able to rsync specific disks from one machine to a backup machine on a daily basis since they didn't change all that much.  With unraid it's a bit easier since each filesystem is a self contained entity capable of synchronization pretty easily.

 

The real issue then becomes cost.

Yup, cost *is* the issue.

 

Would you spend 2300 on a backup server, or perhaps another 100 to 300 on a double parity protected array?

 

Myself, I'll go the software route. Software is always cheaper than hardware. Unless of course you're buying an Oracle DB. If you *are* buying oracle databases, I doubt you'd be in this forum.

 

Backing up that data is possible given a few circumstances.

 

1. You do the initial backups locally somehow, disk to disk or over a network with rsync.

2. Once the disks are backed up,  they do not change all that often.  Do we change all 12tb that often?

3. You use rsync to do a disk# to disk# backup over some kind of connection, local or remote.

 

I was able to rsync specific disks from one machine to a backup machine on a daily basis since they didn't change all that much.  With unraid it's a bit easier since each filesystem is a self contained entity capable of synchronization pretty easily.

 

The real issue then becomes cost.

What checksum function does Btrfs use?

 

Currently Btrfs uses crc32c for data and metadata. The disk format has room for 256bits of checksum for metadata and up to a full leaf block (roughly 4k or more) for data blocks. Over time we'll add support for more checksum alternatives.

 

...

 

Really?

 

Well you can always look into btrfs ... It offers most of the good stuff of zfs

What ZFS has and BTRFS hasn't is builtin checksuming for the data.  That's gold!

Anyway, I tried BTRFS for awhile, and I ran into some serious problems:  Once I fully filled up one disk, I was no longer able to delete (or modify) any files on it, no matter what tricks I tried.  I googled around for quite awhile, but never found a solution, although found many people having the same problem.  I just gave up on BTRFS for now.

Yup, cost *is* the issue.

 

Would you spend 2300 on a backup server, or perhaps another 100 to 300 on a double parity protected array?

 

The answer is obvious, the only issue is the cheaper choice is not available with unRAID.

unRAID still has to go 64bit.

Given how long it's taken to get this far.

I would not hold my breath for dual parity support.

 

Do something else for your most critical files. Dual parity is not the answer for your most critical files.

unRAID is not a backup.

 

There is no piece of software that will save you from a natural disaster. I lived through 2 of them in my lifetime.

 

I've seen how fast life can change in the blink of an eye.

I've learned how critical some documents are and was fortunate I sent them offsite.

The only thing that saved my critical data was an offsite mirror.

 

Would I pay $2300 to backup my data. Yes. However, I would also be wise about what really needs to be backed up.

It would not cost me $2300.

Well you can always look into btrfs ... It offers most of the good stuff of zfs

What ZFS has and BTRFS hasn't is builtin checksuming for the data.  That's gold!

Anyway, I tried BTRFS for awhile, and I ran into some serious problems:  Once I fully filled up one disk, I was no longer able to delete (or modify) any files on it, no matter what tricks I tried.  I googled around for quite awhile, but never found a solution, although found many people having the same problem.  I just gave up on BTRFS for now.

 

You could not delete because of the snapshot feature,  you have to use snaper I believe to go in and delete some of the snapshots to clear space for metadata

 

 

Sent from my SGH-T889 using Tapatalk 4