How to lose 100tb of data: A guide to my $2000 mistake

[Aerodb]

This is the story of how I managed to fry all 12 of my HDDs in moment of stupidity. (5 min read)

 

To start, we should talk about my setup. To simplify, I'll post my old signature to cut to the chase. But in short, it was built up over time. Slowly adding drives as needed to satiate my data hording. Much of it was Plex media but I had setup things like a Pi-Hole, Nextcloud, personal VPN (for when I wanted to use public Wi-Fi on my phone in a safer manner). Much of this I think is pretty common in our community so my server wasn't really that out-of-the-ordinary. 

 

Config - Pre-incident

Unraid Plus

MB: ASRock X370 Taichi | CPU: AMD Ryzen 7 1700 | RAM: 32GB DDR4 | Cooler: Cooler Master MasterLiquid Lite 240 | Case: Thermaltake Core X9 | Cache: CORSAIR FORCE Series MP510 (960GB) | Parity: 2 X WD Red 10TB | Array: 3 X WD Red 8TB, 4 X Seagate 8TB, 2 X WD Red 10TB | GPU: XFX Radeon RX 560 | PSU: EVGA SuperNOVA G2, 80+ GOLD 650W

 

(Side note- never went full enterprise/rack mounted due to some confusion around backplanes and concerns over fan noise)

 

So this all started back when I resolved to add more streaming capability to the plex server. I simply wanted to be able to more comfortably encode/direct-stream more concurrent streams. Toying with the Idea of adding a Nvidia GPU rather than my Radeon which doesn't support hardware acceleration. But thanks to some great advice on the unraid discord, I would only add 3 or so streams, not a meaningful improvement for the cost. So was still debating a full motherboard and CPU upgrade. But one thing was clear, I would likely need to upgrade my power supply to support any major changes, GPU, CPU or otherwise.

 

Christmas came and my wife, the incredible Rockstar that she is, blew passed our gift giving budget (for each other) to buy me a Seasonic 1300 gold. (before anyone says it, platinum or titanium would be nice, but the power savings would be nominal and the return on investment was negligible especially at the premium). Needless to say, I was elated. 

 

I planned to turn down the server and and carry out the upgrade in one day. If you have ever upgraded a PSU, its not that big a deal usually. 

 

For anyone who doesn't know. Power supply units convert my American 120v AC power into DC and deliver multiple voltages to the various components. The connectors on the PSU side of the power cables are often keyed or shaped in a unique manner to ensure that they only fit/plug into the correct voltage ports. Square peg, round hole type of thing.

 

Skilled hardware builders out there, you know what's coming. 

 

I checked the preexisting power cables. The motherboard and GPU cables didn't fit. Easy enough, new unit has cables. I swapped those out. I double checked my work, plugged it all in, flipped the switch to turn on and the PSU clicked. A Common sign of a power fault or fault protection. Something was wrong. Perhaps something wasn't fully connected or maybe a new cable was faulty. After all, even Seasonic for all their track record, isn't infallible. But nothing looks wrong and I don't NEED the new PSU today, so I'll reinstall the old unit PSU and have the new PSU exchanged as faulty and try again in a week or two. No big. 

 

Old PSU reinstalled, turned on, BIOS shows no SATA disks....started trouble shooting and checking. Pulled HDD and attempted to connect directly to a windows machine to just see if the device is detected.... nothing. None of them. 

 

If you didn't connect the dots to what has transpired. Though a square peg doesn't fit into a round hole, many other shapes do fit into a square. Including the wrong voltages being unintentionally sent to my disks, frying the circuitry to all 12 disks including the parity drives, not that 2 of those could repair this level of dumassary. 

 

I can only explain the emotion I felt as mourning the loss of a loved one. I mean, you must understand, this was my biggest hobby and pleasure. Learning the endless capabilities of what I could do with Unraid. 

 

I did check with well known data recovery services but with the sheer volume of data... the quote was $9,000-21,000 dollars. Though they did say that if no data was recovered, It would cost me zero. A testament to the undoubtable quality of work they would be able to offer. But this isn't enterprise data. the financial risk to me was nil. For more reasons that the obvious, this was not a viable path forward. 

 

I resolved to rebuild. The sweet spot for storage size, cost, and physical drive slots for me was going with 7 new 14tb drives. It would yield only 70tb of usable storage (rather than my previous 86tb) but with drives of this size, I was definitely going to need/want 2 parity disks again. Also that would help with the prior issue I had of only being able to fit 12 disks in my case. Now I would have 5 more slots to grow into. All in all the new disks would cost me nearly $1900 (pardon the dramatic title but its nearly 2K and the same goes for my 86tb of usable space out of the 100tb worth of drives).

 

Plus with the advent of things like Unmaniac, I will be able to reduce my storage needs considerably, re-encoding as I regrow my data. Shout out to the great Youtube creators teaching me. Linus tech tips, Spaceinvador One, the OGs. The new ones too, Capt. Chaz, The Mysticle, WhiteWitch Craft. I suppose I even owe a bit of thanks to bite my bits, garage escapades and all. 

 

The lesson learned that I must have forgotten since my computer building classes in high school...

 

Swap the power cables when swapping the PSU unless you're more than certain they are the same.

 

So that's where I am today. Preclearing the new 7 disks and soon I'll start installing dockers and seeing what I can get going. I tried to tell the story as best I could but feel free to ask anything you like that I may have left out. 

 

Edited January 18, 2021 by Aerodb

[JonathanM]

12 minutes ago, Aerodb said:

 

I did check with well known data recovery services but with the sheer volume of data... the quote was $9,000-21,000 dollars.

Once you have come to grips with how much the data is actually worth to you, I suggest shopping around. There are services that will handle circuit board issues for much less, since no clean room work is required.

 

I'm not saying it's a $50 repair, but like I said, pick a reasonable number in your head for what you expect and then go shopping. If it's still too much, fine, but don't write the old data off yet.

 

There are multiple threads discussing this exact scenario on this forum.

 

Also, there are multiple levels of OCD for data management, I like to think I'm on the more sane side of things, but I still keep track of which content is on which disks, and out of double digit number of disks in my hoard, I would only feel obligated to recover 3 physical drives, and those are duplicated elsewhere. So, would you need to recover all the dead disks, or could you live with recovering a small subset?

[Vr2Io]

Sorry for the story, lost so much.

 

27 minutes ago, Aerodb said:

Skilled hardware builders

I also build so much, I always take account of incident cause by PSU, so avoid put all in one.

 

Few days ago, I plan on swap a new mainboard, there are God sound talk to me ....

 

"Now is Winter, did you notice static electricity always around your body ..... "

 

I suddenly thinking I need ESD discharge, it really important, it will kill old / new component, in fact I seldom concern this part, now with different.

Edited January 18, 2021 by Vr2Io

[Aerodb]

4 minutes ago, jonathanm said:

Once you have come to grips with how much the data is actually worth to you, I suggest shopping around. There are services that will handle circuit board issues for much less, since no clean room work is required.

 

I'm not saying it's a $50 repair, but like I said, pick a reasonable number in your head for what you expect and then go shopping. If it's still too much, fine, but don't write the old data off yet.

 

There are multiple threads discussing this exact scenario on this forum.

 

Also, there are multiple levels of OCD for data management, I like to think I'm on the more sane side of things, but I still keep track of which content is on which disks, and out of double digit number of disks in my hoard, I would only feel obligated to recover 3 physical drives, and those are duplicated elsewhere. So, would you need to recover all the dead disks, or could you live with recovering a small subset?

The truth is, well let's just say most of my data can be downloaded. The only backups I have were of the nextcloud files from August 2020 and what was synced locally to my laptop. So the most valuable lost data was my wife's nextcloud data (so for her she lost everything after August 2020). 

 

And determining the value would be maybe 150 bucks to her. Seeing as that is a stupidly small budget and she doesn't care much, I could say that they would be worth more to me as new array drives, but not the data per se. So maybe that's worth digging into more. No idea what companies do that, but I'll search a bit.

 

Thank you for the wise input. 

[Kaldek]

I'm sorry, I'm scratching my head here a bit.  

You say you replaced the PSU, and before turning it on you also used the new power cables that came with the new PSU, and then it nuked everything even when using the right cables?

[Aerodb]

8 minutes ago, Kaldek said:

I'm sorry, I'm scratching my head here a bit.  

You say you replaced the PSU, and before turning it on you also used the new power cables that came with the new PSU, and then it nuked everything even when using the right cables?

My wording may be a bit confusing. The motherboard and GPU cables didn't fit, so I only replaced those. The HDD cables were reused. Which cause the whole issue. 

[Kaldek]

2 minutes ago, Aerodb said:

My wording may be a bit confusing. The motherboard and GPU cables didn't fit, so I only replaced those. The HDD cables were reused. Which cause the whole issue. 

Ohhhhhhh.   Yeah that will mess stuff up.  Ouch.  

I'm actually really curious what the polarity/voltage differences were between the two cables to find out whether it was a reverse polarity issue or a 12v into 5v issue.

[lionelhutz]

Ouch, but thanks for sharing as a reminder to others about that.

 

I never thought about those cables much, but it's really a dick move on the power supply companies side, not standardizing on the pinout for both ends of the modular cables.

[hawihoney]

13 hours ago, jonathanm said:

but I still keep track of which content is on which disks

This! I'm running tree for all disks every night.

 

[PSYCHOPATHiO]

I'm sorry you lost all your data, I know the feeling as something similar happened to me once. In my case I was hit by ransomware but I had a second unraid server with a copy of all my dataoffline.

As geek  & a data hoarder I like to keep a minimal of 1 extra backup offline of all my data

[Aerodb]

38 minutes ago, PSYCHOPATHiO said:

I'm sorry you lost all your data, I know the feeling as something similar happened to me once. In my case I was hit by ransomware but I had a second unraid server with a copy of all my dataoffline.

As geek  & a data hoarder I like to keep a minimal of 1 extra backup offline of all my data

I have always been a fan of the rule of three copies (local, off-site, off-line) but with the volume of data I never found a trustworthy method that didn't cost a TON. I mean for a while there was that Cayman islands cloud storage thing... that died quickly. for now the best option I see is a mirrored server offsite. but replicating that amount of data across the internet... not fun even if I invested in WAN-X.

[Aerodb]

9 hours ago, hawihoney said:

This! I'm running tree for all disks every night.

 

running Tree? is that a plugin of sorts?

[kizer]

https://www.tecmint.com/linux-tree-command-examples/

 

I know from Experience Corsair has a bad habit of changing power on their connectors and me/you having the burden of figuring that out at our expense. So I always make it a habit of never re-using power cables from the PSU to the device. I know you already learned this and know this and I'm sorry for your loss. Man that's rather devastating actually. 

[hawihoney]

2 hours ago, Aerodb said:

running Tree? is that a plugin of sorts?

tree is a standard tool. Just try the following to see what it does:

 

tree /mnt/disk1

 

I'm running dual parity, but in case three or more disks fail at once, I want to know what is lost and where it was stored.

 

[PSYCHOPATHiO]

14 hours ago, Aerodb said:

I have always been a fan of the rule of three copies (local, off-site, off-line)

This is the best rule but at least keep a minimal even if local offline backup of the most important data you have just in case of something going wrong. Good luck with the new build.