RIPLInux and DDRESCUE saved my data.

[WeeboTech]

I'm posting this, not as a dig to unRAID, but as a possible last resort save for someone who may have a drive that is unrecoverable and where unRAID parity was not valid enough to recover the drive.

 

What occured.

1. I've been running 4.5.4 for ages. I never changed it because nothing in the newer versions appealed enough.

2. I do a parity check once a month. I do a daily find down the whole tree of all files for indexed searching.

3. This weekend before moving the machine from the bench to it's normal place I decided to upgrade it to 4.7.

4. I upgraded, rebooted, and saw the screen saying upgrading parity. Apparently the newer version detected more space on the parity drive then prior version. I remember Tom mentioning this once before.

5. Unrecoverable errors on Disk1.  Disk1 went offline. In fact it got kicked out of the machine.

6. Parity was being re-written while disk1 was kicked out.

 

No amount of switching versions or trust my parity would work.

There was no way I could get the system to trust parity and rebuild the failed disk on a new drive.

 

So there I was. My most important drive of hundreds of thousands of small files with a bad sector on it.

It was so bad that a reiserfsck would fail and the drive would stop responding to the system.

Bad blocks failed and the drive was kicked out of the system.

Even the SMART short test would go through an endless loop trying to recover.

 

Yet I knew if I could get past the bad blocks, I would recover some of the data.

Reiserfs seems to be quite resilient in recovery.

 

So I stumbled upon ddrescue and Riplinux. Booted and tried to do a dd from one drive to another.

 

I have to say this tool really did provide a last ditch effort to recover data.

When the drive got kicked out. I would stop the tool. Disconnect and reconnect the drive to reset it and start the tool up again.

With the log file it remembered good and bad sectors and continued past the point of failure.

 

I thought for sure I would loose data. It was predicting the loss of 1GB out of 1TB.

Small percentage I know, but then again, you don't know where the loss is.

 

After 12 hours of sweating over it, it finished.  There was 1GB loss about 5000 sectors.

Then I tried it in reverse mode. with a try again option.

This starts the reads from the end of the drive for all failed sectors.

 

I had to reset the drive a number of times over the next 3-4 hours. But in the end it recovered all but 1 sector.

On the new drive I did a reiserfsck --check. Only 1 corruption found and fixed.

 

So I'm back in business now.

This was a seagate 1TB drive. I've had it for a long time with no issues,

 

It surely made me realize a few things.

Even though unRAID protects your data, a hiccup could render it useless.

I may start swapping drives out based on age rather then failure.

I really need to finish that locate database with MD5sum so you can double check files for integrity.

I may need to do some kind of badblocks check on an automated fashion.

I'll have to set up a parallel server for rsyncing over critical disks.

 

In any case, ddrescue really saved my data and I thought it was important enough to mention it for others in the future. /me waves to the future.

 

It could be an important tool to put on the distro.

 

I'll review how to set syslinux.cfg up to boot riplinux directly from your flash as it contained all sorts of tools to help recover drives.

 

http://rip.7bf.de/current/

http://www.gnu.org/s/ddrescue/ddrescue.html

http://www.gnu.org/s/ddrescue/manual/ddrescue_manual.html

http://www.forensicswiki.org/wiki/Ddrescue

http://www.cyberciti.biz/tips/repairing-reiserfs-file-system-with-reiserfsck.html

 

 

 

 

 

[Lantesh]

This is good info.  I'm bookmarking for future reference.  I'm glad to hear you were able to recover your data, and by posting this you probably have saved someone else from a future loss of data.

[WeeboTech]

I'm trying to remember and gather the procedure I used.

I had to do it in a bunch of steps.

 

First was a forward recovery with the -n option, (no split).

Then was a forward recover without the -n option.

 

Then was a reverse recovery with the -R(reverse) -r3(retry each sector three times).

 

It was amazing to see the reverse recovery work and get the data from going backwards.

It stopped finally on the 1 bad sector that prevented any sequential reads.

 

 

I'll be posting a slackware package for ddrescue and md5deep soon.

[WeeboTech]

Here's how I configured syslinux.cfg to boot riplinux from the UNRAID flash drive.

 

I created a CD-RW bootable ISO, then copied the kernel.32 and rootfs.cgz from the BOOT folder to the flash drivel

I used periods to help keep the files together. It could have easily been done in a directory.

I may change it one day so that the grub boot is accessible, but for now this is a quick way.

 

label RIPLinuX
  kernel riplinux.kernel.32 
  append initrd=riplinux.rootfs.cgz nokeymap root=/dev/ram0 rw vga=normal video=640x400

 

 

fwiw, I was able to get it going with PXE boot too.

It was pretty much the same in the pxeboot DEFAULT file on the tftp server.

 

LABEL riplinux
    KERNEL images/riplinux-kernel32 nokeymap root=/dev/ram0 rw vga=normal video=640x400
    APPEND initrd=images/riplinux-rootfs.cgz    

[WeeboTech]

Uploaded ddrescue 1.14 to google code.

md5deep 4.0 beta compiled there too.

 

Good tools to have around.

 

http://code.google.com/p/unraid-weebotech/downloads/list

[fitbrit]

This looks great, and something I could use right now. Unfortunately, it seems rather complicated and not something I could use right now.

I'll have to do a lot of learning about this.