Amazing unRaid.

 

I cannot overstate what a nightmare I lived through for the past two weeks ever since a stupid mistake cost me 6 years worth of movies, TV shows and family photos. Thanks to a few well meaning friends who gave me some solid advice, and thanks to the guidance of some of the amazing people here on this forum such as garycase and DaleWilliams, and thanks to an amazing company named Donor Drives, I was able to salvage all of my data. I wanted to recount my story here so that it serves as a guide and a resource for any other newbies who may have a similar experience. Mostly, though, I'm telling my story because I am so relieved and happy and I wanted to share it with all of you.

 

The Disaster

I've always believed that unRaid is a solid OS for a media server, and this is why I used unraid to build my first media server, nearly five years ago and have since slowly expanded. As of this writing, I was running a 22 TB media server with literally close to 2000 movies and TV shows that I had accumulated over the years. My family photos and family videos (vacations, kids growing up) were all stored on the server since I had assumed that the server will protect it due to parity and fault tolerance. My first mistake, as Garycase pointed out in this awesome review, was confusing fault-tolerance of unraid = no need for additional backup. unRaid may protect data when a hard drive fails, but it cannot protect data when the house burns down, or when the entire array fails at the same time (and for those of you who think this never happens, well, this is exactly what happened to me).

 

The problem came when I decided that my server needed an upgrade and I bought myself a fantastic Norco 4220 server chassis with its ability to power 20 hard drives. It's a great case, but without testing the case or familiarizing myself with the case, I made a stupid rookie mistake. I plugged in all my drives into the array and turned on the server... and fried pretty much most of my hard drives almost instantaneously.

 

There are a lot of places an electrical surge can come from. It can come from a simple problem with your home's wiring or due to a generic electrical surge that can happen (this is what surge protectors and UPS devices are for). Always have your unraid connected to one of these. I had mine connected to a UPS, so this was not the source.

 

There can be a short-circuit in the case, or in one of the components causing the electrical surge. There can also be a problem with your power supply unit (PSU) due to a faulty regulator, for example, and this can deliver an electrical surge to the delicate circuitry in the motherboard or hard drives.

 

In my case, however, the problem was just plain old stupidity. I did not appreciate that 4-pin molex connectors were keyed and had polarity to it. It's obvious when you look at it closely, but with enough force, you can force a 4-pin molex into its connector in the wrong orientation. This is exactly what happened to me. And in doing so, I delivered +12V into a hard drive circuit board that was rated for +5V.

 

Once this happened and I realized that all of my hard drives had fried, I didn't have much choice. I posted my story here and thanks to the help of many people, I was able to figure out where I went wrong (plugging the molex connectors in the reverse orientation).

 

The good news is that, about a month ago, one of my good friends told me to consider using Crashplan. As many of you know, unraid has a Crashplan plugin (which I read can theoretically backup your NAS to the cloud). But there are other companies such as Blackblaze, Carbonite etc, that provide a similar service.

 

I must say that I was skeptical about using Crashplan and I felt that $5 /month for their unlimited option was a waste of money, but for the first time I am soooo incredibly glad that I used it. See, about two weeks prior to this event, I had backed up all my family pictures, videos and memories all to crash plan. Thank God!

 

In under 10 minutes, I was able to restore all that information into my hard drive of my local computer. Now even if I lost my hard drives on my server, it would just be movies and TV shows that I lose, and not any of my family memories.

 

My first advice to anyone who hasn't done this yet, is BACK UP IMPORTANT STUFF ON YOUR SERVER to AN OFFSITE LOCATION. Use Crashplan, BlackBlaze, Carbonite..... Whatever! The advantage of this system is that it will protect you even if your house burns down. Believe me, I will never question the importance of backing up after this day. I am so glad that I salvaged at least this bit of data.

 

The Recovery

 

In the aftermath of my server dying, I contacted Norco since I didn't know if this happened due to a defective backplane in the chassis. I've heard that this had happened to a few people but other folks here on this forum such as Prostuff1 had reassured me that these events were few and far between. The good news is that Norco customer service was fantastic and they promptly provided me with some replacement backplanes.

 

Due to the possibility that this was a defective PSU, I went ahead and ordered a new PSU as well.

 

Now the problem was the hard drives. I was advised by some amazing people here such as garycase that there was a chance that the PCBs (printed circuit board) on the hard drive maybe fried due to the increased voltage, but if that is the case, that there maybe a chance that I could salvage the data by doing a PCB swap + ROM chip transfer (adaptation).  There are some youtube videos on how to accomplish this, but this is a bit complicated and although I can swap out of the PCB, I didn't feel comfortable swapping out the ROM chip (heat gun, soldering iron and a pair of delicate hands).

 

There are some companies that actually will take the PCB from you, and send you a new PCB with the ROM chip transferred. These, on average cost about $50-60 per PCB. I then found a company named Donor Drives. The guy who is the office manager at Donor Drives, has a nice youtube video where he shows how to do the Adaptation process yourself. But I decided to give them a call.

 

My Experience with Donor Drives.

 

 

Donor Drives is a company that specializes in Data Recovery, based out of Cleveland OH. When I called, I spoke to Alex Didek, who I believe is one of the Office Manager's there. He was very helpful over the phone and suggested that I send my drives over for a free diagnostic. The adaptation cost (PCB replacement + ROM chip transfer) would cost about $60 per drive.

 

I have to say that at this point, I was a bit nervous. There are some bad reviews about this company (which no doubt you will find if you just google them), but I felt reassured after speaking to Alex, worst case scenario I would lose my movies/TV shows, while although devastating, was not nearly as bad as losing all my families memories (which I salvaged thanks to Crashplan).

 

I packed up my hard drives (all four of them that failed) and sent them over to DonorDrives by UPS. They sent me an email 2 days later confirming that they received them and advising me that the diagnostic process could take 2-3 days.

 

In the meantime, I continued my communication with Alex, who advised me that even with the situation that I had that about 50% of drives can be fixed with a simple PCB swap/Adaptation but about half of these drives have more severe damage including head failure which requires more extensive data recovery (more $$$). I told him that it would not be worth it for me to spend $$$ on recovering the data since these were just movies/TV shows, albeit, 6 years worth of it.

 

Four days since the delivery, Alex sent me an email notifying me that the drives were ready. I called him, and I was told that the PCB adaptation process worked for 3 of the 4 drives. One of my hard drives, had drive head failure and required more extensive recovery. I abandoned that drive.

 

Within 2 days, all four of my drives arrived. They were impressively packed.

 

 

Each drives was enclosed in a sealed anti-static wrap, with a label indicating whether this drive was fixed or not.

There was an impressive amount of bubble wrap protecting each of the drives that I am confident that there would be no way short of a massive collision en route that those drives would be damaged.

 

I plugged in the drives to my server (now that I have a new PSU, new backplanes and plugged in the molex connectors correctly).

 

In no time, unraid powered up and before I knew it, data rebuilding had commenced for the drive that was missing (which could not be recovered).

 

 

I continue to be amazed by unraid. I was left in awe as I watched as the server began rebuilding the data into a new hard drive that I placed into the array. It's been a crazy experience going through the heartache of losing these drives, but with the help of companies like DonorDrives, folks like Garycase and by virtue of the amazing features of unraid I have got ALL of my data back.

 

Lesson learned. Always backup your data. NOW!

 

Thank you everyone.

 

Great to hear that all worked out so nicely => you were (as I'm sure you know) very lucky that all but one drive could be restored.    There's a BIG difference between "restored" and "recovered" -- if they had only been able to recover the data onto another hard drive (which is a common scenario with data recovery), then the drives wouldn't have been identical to their original state and the rebuild of the missing drive wouldn't have been possible.

 

Clearly you are now a believer in what I've said for many years -- BACKUP !! 

http://lime-technology.com/forum/index.php?topic=31020.0

... By the way, I'm VERY impressed at your experience with Donor Drives.  They are definitely now among my bookmarks -- and I'll be sure to recommend them to others who may need their services.  I doubt I'll ever need their services, but it's nice to know about folks like this when you're helping others.

 

I'm very happy for you rwickra...that's a happy ending to a heart wrenching story.

 

I'd also 'second' your comment about the forums being a very helpful and friendly place...its an amazingly good community.

I also +1 your amazement with UnRaid..  I've got a parallel topic on my similar, 18tb loss, but mine was due to fire...  I also had SOME stuff backed up to crashplan, but really wanted to get these drives back..  Long story short, the drives still work, even as stinky and charred up as they are, and I am just now starting to copy the data to fresh new external Seagate 4tb USB drives, which I'll use until I can rebuild the UnRaid (a hotel room is no place to build a small datacenter!)

 

Anyway..  I had read your story earlier today, and I'm glad you got yours up, and happy to report that I'm another happy customer today!!

 

-Steve

WeeboTech's unRecover - Another interesting disaster recovery tale

A few posts related to data recovery from my early days with unRAID. Most still relevant.

 

Data Recovery Experience

 

Top 10 ways to Lose Data with unRAID

 

I have never heard of anyone plugging in a molex upside down! Even plugging them in the right way can be a PITA. You definitely get the square peg in round hole brute force award!

Agreed.

 

I can be quite obtuse when it comes to brute force. Funny thing is that because there was a fan plate with 4 x 80 mm fans separating the backplanes of the chassis from the rest of the case, I couldn't really see where I was plugging in the molex connectors. I admit it was quite hard to push them in, but I persevered on. 

 

I am amazed at how frequently these sorts of catastrophes happen to people, and I am not oblivious to the fact that I am EXTREMELY lucky. When Donor Drives worked on my drives, had more than 1 drive failed permanently I would have been S.O.L.

 

It was just very fortunate that only one drive was not salvageable and all of the other drives in the array were operational.

 

I'm waiting for the build process to be over (for a 4TB drive it's taking close to 24 hrs), and then I'm going to be looking at installing the crashplan plugin and see if I could start backing up the rest of system to Crashplan.

... By the way, I'm VERY impressed at your experience with Donor Drives.  They are definitely now among my bookmarks -- and I'll be sure to recommend them to others who may need their services.  I doubt I'll ever need their services, but it's nice to know about folks like this when you're helping others.

 

@Garycase -- First of all, THANK YOU! I cannot thank you enough. Of all the people who posted to help with my woes you were tremendously helpful. Thank you so much for that.  About Donor Drives, I would definitely recommend them now with my positive experience. They were honest and did not charge me for the drive that they could not salvage. They told me that they didn't even attempt adaptation because they found from their diagnostic that the drive head had failed. So the recovery process cost me a grand total of like $196 ($180 + $16 for 2-day shipping). I'm very impressed with their work.

 

You're most welcome.  By the way, although I've never done it operationally where it would matter, I HAVE seen some molex plugs that could be fairly easily forced in backwards.  Usually it's because the female end on the device (typically a cheap optical drive) isn't well made with regards to the keying.    But it still takes a pretty solid amount of force -- and if you just look carefully for the rounded corners you're not likely to do it wrong.

 

In any event, it's probably safe to say that the likelihood of YOU ever doing it wrong again is something very close to ZERO 

A question for those using CrashPlan.

 

Are you using it to backup all 20TB of data? Is this feasible? Since it seems to be unlimited fro $5/month I am thinking this is a good investment (for reasons such as this thread), but wanted to confirm that people are backing up their entire UnRAID servers to this without issue.

 

 

No cloud-based backup is realistic for most people using UnRAID.    Just consider the math ... a typical uplink connection is perhaps 1-2Mb/s.  At 2Mb/s it takes roughly an hour to upload a GB.  At 1GB/hr uploaded it takes ~ 42 days to upload a TB ... close to 2 1/2 YEARS to upload 20TB.    And that's assuming NO data changes over time and needs to be re-uploaded.

 

Faster upload speeds would of course reduce that time, but it will still likely be unrealistic.

 

You could, of course, use Crashplan backing up to a local server ... perhaps another UnRAID box 

We've just started using Backblaze to back up our photos. We've got around 500GB of RAW photos, uplink of 1MB speeds. Going for a month, got another month to go 

 

Worth it if the place burns down, or it gets nicked.

Okay, you may have a point.

 

However my internet provider is offering  fibre-hybrid solution that is 350MB up & down. Unfortunately it's $225/month and not available in my area.

 

Until then I guess I will be a bit more selective.

 

Does the unRAID crashplan plugin allow you to select specific shares to sync via the GUI?

 

Yes, for enough money you can get enough bandwidth 

 

But it's far better to simply build a backup server and/or establish a process where you back everything up to a set of bare external drives you can connect via an eSATA or USB3 dock.

 

You may want to read this:  http://lime-technology.com/forum/index.php?topic=31020.0