After years of being an unRAID user, and running ESXi for a while with a standard tower case, I finally made the transition to a Norco 4224.  Oh how excited I was when I put all my drives in, fired it up, and everything was running perfectly on the first go.  I even love how the backplanes only require one molex power adapter now!  No question on trying to squeeze 12 of those bad boys in that small area.  Anyhoo...

 

Unfortunately, my SAS breakout cable got misrouted by DHL, so I had my ESXi datastore drives loose in the case (can't have a cable holding up my re-build, now can I?).  Yesterday the cable arrived, and it was time to get everything in their nice drive bays.

 

I decided to put the datastore drives on the bottom backplane, to keep things tidy.  One SSD, and 2 2TB spinners.  Pressed the power button and... POP.  Hmm... That was weird.  Everything shut down then restarted on its own.  Fans spinning, drives spinning.  Everything sounds good.

 

When ESXi loaded up, 3 of my VMs were missing.  The datastore drive was not listed, but my SSD was (the other spinner was for expansion).  Maybe a bad backplane?  Let's move it all up one.  Sure enough, still none of the 2 TB drives were showing up.  So I pop into the BIOS.  Not there either.  Pull out the drives, remove them from the caddies, and the smell was unmistakable... burnt electronics.  So I do some googling and come across this: How a Norco case killed 13TB of our data.

 

I removed the controller boards from the hard drives and saw the same problem--a burnt hole in one of the chips.  These drives are toast.  Miraculously the SSD is working fine, but I think its because of a different electronic architecture perhaps?  Complete guess on my part.

 

Norco has been great about sending out a replacement backplane (emailed them last night, went back and forth today, and already have a tracking number for a backplane), but they seem to think it's my PSU.  No idea why my Seasonic Platinum PSU would fry two hard drives that were confirmed working before, all in the same backplane, and it not be the backplane, but who knows.

 

Already requested an RMA for those drives from WD, but who knows if they're going to end up charging my card when they see the fried chips.  I also tried using a controller board from the same model drive, but its definitely dead.  Sad.  And I didn't have a backup procedure in place for the ESXi datastores.  (On an aside... suggestions moving forward??)

 

TL;DR: I think a bad backplane fried 2 hard drives.  In the future I'm going to put in an old drive into a slot in each backplane before I populate it with my data-filled drives.  You never can be too careful.

I had a similar issue where I lost 5 drives once stacked into a 4224. Thankfully 2 of them ended up being recoverable, but I ended up losing 6TB of data. I ended up RMAing the 4224 and got a replacement case, plus I have 2 extra backplanes from the RMA with Norco.

 

It's an awesome case and somewhat awe-inspiring at home, but there definitely seem to be a few issues which can have huge ramifications.

 

I know there are several others on the forums that use the same case and have had no issues at all. It's all luck of the draw, but your odds seem somewhat worse with the 4224 than most standard cases.

Sorry to hear man. I had a backplane go bad but no killed disks.

 

As for making your datastore fault tolerant I'd suggest setting up something like FreeNAS or OmniOS/napp-it and mount the ZFS array over NFS to use as your datastore. There's a few of us on here that do it. I have a RAID-Z2 (Z2 allows for 2 simultaneous failed disks) pool using five 500GB disks. Gives me about 1.4TB of usable space so it's more than enough for my various VMs. The disks have to be spun up all the time like any other RAID array so I'm using laptop drives to cut down on energy consumption.

 

I'm assuming this must be a problem with the Norcor 4220 as well, then? Yikes! I just ordered this drive and I have a 22 TB server now filled with precious data. I should probably test the backplanes before I use it. Thanks for the post!

I have only had it happen once in all the servers I have built using the 4220 and 4224.  It was my own fault as I had a splitter in there that fried first and cause the backplane to go out, taking one drive with it.

 

I no longer use splitters...

I have only had it happen once in all the servers I have built using the 4220 and 4224.  It was my own fault as I had a splitter in there that fried first and cause the backplane to go out, taking one drive with it.

 

I no longer use splitters...

 

 

This.

 

I think a splitter was what killed my backplane.

My PSU only has 5 molex adapters, and I don't have enough drives to populate the entire 4224, so I left one of the backplanes unplugged.  No splitter here, so I don't believe that was the problem.  But I did just order another modular cable from Seasonic for when the need arises.

 

I didn't post this to knock on Norco.  I really like my case so far, despite my two fried drives.  I just wanted to warn others to maybe test each backplane first with a drive you wouldn't worry about losing, just to be safe.

 

My two WD RMA drives just arrived this morning (that was fast).  Recertified, but should be fine.  Replacement backplane also arrived.  Need to track down a junk disk to test.

As for making your datastore fault tolerant I'd suggest setting up something like FreeNAS or OmniOS/napp-it and mount the ZFS array over NFS to use as your datastore. There's a few of us on here that do it. I have a RAID-Z2 (Z2 allows for 2 simultaneous failed disks) pool using five 500GB disks. Gives me about 1.4TB of usable space so it's more than enough for my various VMs. The disks have to be spun up all the time like any other RAID array so I'm using laptop drives to cut down on energy consumption.

I'm having trouble conceptualizing how this would be done.  Would this be FreeNAS (or OmniOS) in a VM or in a standalone box?

 

Edit: After doing some reading (and not to hijack my own thread), I think its simpler and cheaper for me to just automate VM backup with ghettoVCB.  I figure (and someone correct me if I'm wrong) I would just add a free 2TB disk to my unRAID array, make that an NFS-only disk share, and automate the ghettoVCB backup to that NFS share.  That way I have redundancy on the backup of my VMs (as long as they all fit on that 2 TB drive).  Last time I tried configuring NFS years ago on unRAID I gave up.  Hopefully it'll be simpler if I avoid using NFS with user shares...

Yes, it would be a VM. As I have it now, the ZFS VM boots off an SSD. I have it then set to not try to load any other VMs for 240 seconds. By the time that 4 minutes is up the ZFS pool has initialized and the rest of my VMs are available and boot up.

 

It's your call but you're in this situation because of 2 fried disks. What you're proposing now would still leave you with no fault tolerance for more than 1 failed disk.

 

The other upside to using ZFS over NFS as your datastore is very fast read/write speeds to your VMs.