HELP! Harvey bit me


MacDaddy

Recommended Posts

I'm a poster child for Murphy.  I have my personal data backed up to the cloud.  I had been rotating my media library to another friends house.  However we both took a hit and my offsite backup was destroyed.  I still have all my DVDs, but they are slimed by the sewer water that was Harvey.  If I have to, I can try to rebuild it by feeding each DVD back through.  The drives were in water minimal time so I'm going to attempt a transfer.

 

I am attempting to bring up a temp unRaid server.  My intent is to load the minimum plugins to allow me to preclear a new disk, and hopefully rclone my old disk to a new disk.  I have to transfer 5 WD 4TB red drives to fresh drives.  I don't intend to transfer the parity drive.

 

- I let my old server depart in a trash truck without pulling out the USB (if it even survived).  I'm assuming I can get the trial version going without a hitch.

- I don't plan to bring the actual array online.

- I'm hoping rclone is native on the USB stick or I can somehow get it there. If there are wiser minds out there that can suggest the parameters to use (or suggest a different approach altogether) I would really appreciate your help.

- I plan to go disk by disk.

- There is an existing hard drive on the surplus equipment I'm using.  It's Win10, but now that I have the network working I don't have a problem converting it to Linux.

- All my drives were XFS.  

 

Please let me know if I can provide any other details that would help in trying to get a disaster recovery workbench going.  I'm going to start putting the surplus equipment together and get a basic unRaid going.  Thanks for any advice or direction that you can offer.

Link to comment

I'd have certainly made an attempt to grab the server and let the drives dry out and see if they are accessible. They are airtight. If they were powered on at the time of the flooding they might have been fried. But if the power failed first, I would expect a reasonable chance of at least some of them yielding data. And if you had the primary and the backup, I'd certainly give extremely good odds of recovering your data.

 

Why did you just throw them away?

 

DVDs should be pretty impervious to water - unless there were nasty chemicals, which I would doubt. Re-ripping is always an option. 20T of space x ~3G per disk ... that's about 5000 DVDs? Averaging 8 hours a day, and doing a disk every 10 minutes, you'll be through in ~100 days. :|

Link to comment

@ bjp999 -I did power all the equipment down.  I was working on raising other items when I realized the water was at the level where my unrRaid system was located in the rack.  I didn't pull the full server at that time, but I was able to pull all the hard drives and place them on the highest shelf in the laundry.  The muck destroyed the UPS and the server hardware.  The equipment was moved out while I was distracted with the 1000 other things that happen simultaneously while trying to muck out a house before mold takes over.  

 

@ trurl - Thanks for the helpful info on WeeboTech.  I'll look into.  I'm stalled anyway tonight with the equipment I have.  I was told that it would "absolutely" boot from USB.  About as accurate as the weather forecast.  

Link to comment

Sorry misunderstood. Glad you have the drives.

 

I would suggest rinsing them in clean water, and getting any slime off of them. The goal then would be to dry them out thoroughly.

 

Two ideas. One it putting them in a warm oven. The drives run about 46C (115F), so you might preheat and let them bake at that temp. Should evaporate any water. Problem might be that oven may not maintain perfect temperature and could overheat. You might therefore try a little lower temp (105F). If you have a manual thermometer, you could experiment with your oven and see if you can get it to maintain a temp without ever going over 118F.

 

Other idea is put them in uncooked rice. The rice will absorb water over time. I once had a nano that went through the washing machine, and it sat in rice for a few days and still didn't work. I left it in the rice and sort of forgot about it, thinking it was a lost cause. Then several weeks (or longer) later tried it and it worked. So rice method can take a long time.

 

I'd recommend some googling on drive drying techniques and find the best method you are comfortable with.

 

Here is an example.  

http://tscrestoration.com/blog/how-to-repair-hard-drive-water-damage/

 

He recommends professional help, which is going to be pricey - but is your best chance of full recovery.

 

If going it alone, I personally think overnight with a paper towel is not going to give the drive enough time/opportunity to dry out, but I am approaching this from a logical perspective, having no practical experience.

 

Once you have a disk clean and dry, take one of the drives and install it in your fresh server (just one). First you want to see if it is recognized during POST and doesn't exhibit mechanical issues (loud noises, clicking, grinding, etc.). If it does any of them, I'd hit the big red switch immediately and let it dry more. If disk is quiet and runs, assign it to disk1 of the "array" with no parity, and assign a fresh disk as "disk2". Enable disk shares.  Start the "array".  See if disk1 looks good. If so, you should copy the data to disk2 (might have to format disk2, make sure to pick XFS file system).

 

If it fails you still have the other drives to experiment with.  That disk may yet work if it dries more. You might find the previously wet disks work perfectly fine, but I'd have a complete backup just in case. Likely they'll be fine or die quickly IMO.

 

 

Link to comment

You could use the Krusader Docker if you get it setup.

 

I'd probably run in a screen session or from the console...

 

Something like (if disk1 is the recovered disk, and disk2 is the new disk)

 

cp -rpv /mnt/disk1/* /mnt/disk2

 

But before I did that I could copy a single file over and make sure it is accurate.

 

If there are certain files that you consider most valuable, you might also copy those before the big copy - if you turn on disk shares you can access it via SMB and copy the files to your workstation for safe keeping.

 

I would copy but not move / delete from the source disk. No need to perform writes to it, and it is just another opportunity for the disk to crap out.

Link to comment

An old tip use by underwater photographers when cameras get flooded is to wash them out with distilled water then let the dry. You want to get any salts and impurities off the electronics before you try to power them up. You can use warm , not hot, air to speed up the process.

Edited by Russ Uno
Link to comment
3 hours ago, tdallen said:

I wonder if the new vacuum/heat based process to dry out cell phones would be appropriate for hard drives or other computer electronics?  I understand at least one of these companies was trying to setup a center in Houston to help with the aftermath of Harvey.

Sounds reasonable.. The trick is How Much Heat..

You need to get the contaminates off with the distilled water, which has no impurities, ASAP then dry completely as quick as you can.

I've used an oven at very low temps, a convection oven works well.. I'd say maybe 90-100 deg.F.  to start.

Old recording tape is cured at 180 deg or so for around 8 hours in convection ovens to get the moisture out of the coatings to make it playable again.

Link to comment

Here's where I stand so far. 

 

- I've got an unRaid workbench up and running with a trial license. 

- running preclear on a new WD red 4TB drive. 

- meeting a friend this afternoon to run drives through a CFC bath (she works in medical equipment field and apparently this stuff can clean even energized equipment). 

 

 

I planning to try the first test case tonight. Someone posted a recommended cp command to copy disks, but I'm leaning to using rsync in case I need a restart. I would appreciate any suggestions for command line options from knowledgeable members. 

 

Is is there a way to retrieve my license credentials from my old setup for use in the workbench?

Link to comment

Just a heads up that not all hard drives are sealed.  They usually have some note on the drive with an arrow pointing to a hole saying not to block it.  It provides filtered ventilation to/from atmosphere.  Only newer helium drives would be sealed.  Not sure how well the filters keep out water so might want to do some research before powering them up and potentially trashing what could be a recoverable disk with a recovery service, if needed.

Link to comment
2 hours ago, unevent said:

arrow pointing to a hole saying not to block it

Yeppers the hole was there as described.  Thankfully the disk orientation had it on the bottom.  

 

The CFC bath was a strange thing to behold.  At first I thought there was a reaction like acid on metal.  It was strange to bring it out "dry".  First copy is going on now.  Fingers crossed but so far so good.

Link to comment

I've got two data disks recovered.  One disk was slow to power up so I stopped in mid boot.  Then I noticed a small water drop on the breather hole.  For that disk I'm starting the process all over again.  I've got one more data disk and the parity remaining.  

 

I'm using the following rsync options with good results:

rsync -avhW --no-compress --progress /mnt/disk1/ /mnt/disk2/
-a is for archive, which preserves ownership, permissions etc.
-v is for verbose, so I can see what's happening (optional)
-h is for human-readable, so the transfer rate and file sizes are easier to read (optional)
-W is for copying whole files only, without delta-xfer algorithm which should reduce CPU load
--no-compress as there's no lack of bandwidth between local devices
--progress so I can see the progress of large files (optional)

I have a couple of questions:

 

1 - Do I need to recover the parity disk to a fresh disk?  Is the parity still valid if I rsync data to a set of fresh data disks?

 

2 - Thinking ahead to the possibility that one data disk may not make it.  Is it possible to put in the fresh data disks, the original parity disk and cross my fingers I can rebuild on a fresh disk the dead drive?  Or should I put all the original drives in a workbench machine and try to rebuild the failed disk with a fresh drive.  I have the disk layout saved in a file on the server.  I'm hoping that I can retrieve that should the appropriate directory be in one of the resurrected disk drives.  If not, is there a way to determine which disk should be mounted on which /sdx mount point?  

 

Thanks much for the help so far.  Harvey has really been crushing me on virtually every front.  It would be nice to have a win here.

Link to comment
1 minute ago, johnnie.black said:

About your second question, it's possible to rebuild one failed disk (assuming single parity) using all other original data disks plus parity.

 

Any ideas on assignment?  A perfect world would let me make each diskX assignment to the correct /sdX mount.  If I don't luck out with my machine config showing up on the recovered drives then I'll be in uncharted territory (for me).

Link to comment

Again, assuming single parity, only important assignment is the parity disk, other disks slots don't matter, this is what you'd do:

 

-Tools -> New Config -> Apply
-assign all original disks (except the failed disk) + parity + a spare of the same size of the failed disk
-check both "parity is already valid" and "maintenance mode" before starting the array
-start the array
-stop array, unassign disk to rebuild
-start array, check emulated disk mounts and contents look correct
-stop array, reassign disk
-start array to begin rebuild

Edited by johnnie.black
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.