[SOLVED] HELP! I think I just lost an entire disk worth of content

[betaman]

So I was trying to use the procedure documented here to remove a 1TB drive from my array that's running 4.7:

 

http://lime-technology.com/forum/index.php?topic=2591.0

 

The write was taking an extremely long time and I realized I left the k off the block size.  When I was retyping the dd command I accidentally put /md2 instead of /md6 (note to self, don't undertake projects of this nature hung over!).  I realized it almost immediately (about 190MB of zeros was copied) but now the disk is completely blank.  I must have executed the umount command on /md6 but the dd command on /md2 because the drive is still showing formatted in UnMenu.

 

I noticed this in my syslog after aborting the dd command:

 

Nov 24 11:57:54 Tower kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 0 does not match to the expected one -1 (Minor Issues)
Nov 24 11:57:54 Tower kernel: REISERFS error (device md2): vs-5150 search_by_key: invalid format found in block 0. Fsck? (Errors)
Nov 24 11:57:54 Tower kernel: REISERFS (device md2): Remounting filesystem read-only (Drive related)
Nov 24 11:57:54 Tower kernel: REISERFS error (device md2): zam-7001 reiserfs_find_entry: io error (Errors)
Nov 24 11:57:54 Tower kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 0 does not match to the expected one -1 (Minor Issues)
Nov 24 11:57:54 Tower kernel: REISERFS error (device md2): vs-5150 search_by_key: invalid format found in block 0. Fsck? (Errors)
Nov 24 11:57:54 Tower kernel: REISERFS error (device md2): zam-7001 reiserfs_find_entry: io error (Errors)
Nov 24 11:57:54 Tower kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 0 does not match to the expected one -1 (Minor Issues)
Nov 24 11:57:54 Tower kernel: REISERFS error (device md2): vs-5150 search_by_key: invalid format found in block 0. Fsck? (Errors)
Nov 24 11:57:54 Tower kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 0 does not match to the expected one -1 (Minor Issues)
Nov 24 11:57:54 Tower kernel: REISERFS error (device md2): vs-5150 search_by_key: invalid format found in block 0. Fsck? (Errors)
Nov 24 11:57:54 Tower kernel: REISERFS error (device md2): zam-7001 reiserfs_find_entry: io error (Errors)
Nov 24 11:57:54 Tower kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 0 does not match to the expected one -1 (Minor Issues)
Nov 24 11:57:54 Tower kernel: REISERFS error (device md2): vs-5150 search_by_key: invalid format found in block 0. Fsck? (Errors)
Nov 24 11:57:54 Tower kernel: REISERFS error (device md2): zam-7001 reiserfs_find_entry: io error (Errors)
Nov 24 11:57:54 Tower kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 0 does not match to the expected one -1 (Minor Issues)
Nov 24 11:57:54 Tower kernel: REISERFS error (device md2): vs-5150 search_by_key: invalid format found in block 0. Fsck? (Errors)
Nov 24 11:57:54 Tower kernel: REISERFS error (device md2): zam-7001 reiserfs_find_entry: io error (Errors)
Nov 24 11:57:54 Tower kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 0 does not match to the expected one -1 (Minor Issues)
Nov 24 11:57:54 Tower kernel: REISERFS error (device md2): vs-5150 search_by_key: invalid format found in block 0. Fsck? (Errors)
Nov 24 11:57:54 Tower kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 0 does not match to the expected one -1 (Minor Issues)
Nov 24 11:57:54 Tower kernel: REISERFS error (device md2): vs-5150 search_by_key: invalid format found in block 0. Fsck? (Errors)
Nov 24 11:57:54 Tower kernel: REISERFS error (device md2): zam-7001 reiserfs_find_entry: io error (Errors)
Nov 24 11:57:54 Tower kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 0 does not match to the expected one -1 (Minor Issues)
Nov 24 11:57:54 Tower kernel: REISERFS error (device md2): vs-5150 search_by_key: invalid format found in block 0. Fsck? (Errors)
Nov 24 11:57:54 Tower kernel: REISERFS error (device md2): zam-7001 reiserfs_find_entry: io error (Errors)
Nov 24 11:57:54 Tower kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 0 does not match to the expected one -1 (Minor Issues)
Nov 24 11:57:54 Tower kernel: REISERFS error (device md2): vs-5150 search_by_key: invalid format found in block 0. Fsck? (Errors)
Nov 24 11:57:54 Tower kernel: REISERFS error (device md2): zam-7001 reiserfs_find_entry: io error (Errors)
Nov 24 11:57:54 Tower kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 0 does not match to the expected one -1 (Minor Issues)
Nov 24 11:57:54 Tower kernel: REISERFS error (device md2): vs-5150 search_by_key: invalid format found in block 0. Fsck? (Errors)
Nov 24 11:57:54 Tower kernel: REISERFS error (device md2): zam-7001 reiserfs_find_entry: io error (Errors)
Nov 24 11:57:54 Tower kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 0 does not match to the expected one -1 (Minor Issues)
Nov 24 11:57:54 Tower kernel: REISERFS error (device md2): vs-5150 search_by_key: invalid format found in block 0. Fsck? (Errors)
Nov 24 11:57:54 Tower kernel: REISERFS error (device md2): zam-7001 reiserfs_find_entry: io error (Errors)
Nov 24 11:57:54 Tower kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 0 does not match to the expected one -1 (Minor Issues)
Nov 24 11:57:54 Tower kernel: REISERFS error (device md2): vs-5150 search_by_key: invalid format found in block 0. Fsck? (Errors)

 

Is it still possible to restore this drive (disk 2) from parity?  Keep in mind that Disk 6 is still part of the array and showing unformatted.

[vca]

As I understand it, the mdX disks are all in the protected array, so when you did the dd to the wrong disk the zeroes it wrote to that disk were seen by the array and the parity was updated to match.  So you probably cannot undo the damage by just rebuilding to another disk.

 

I'm not 100% certain this is the case, so maybe someone else will add a comment.

 

As for the data loss, you might not be totally hosed, there is a reiserfs utility command (reiserfsck) that can search a drive for all the content and try to rebuild the file system based on what it finds.  Take a look at the wiki here:

 

http://lime-technology.com/wiki/index.php?title=FAQ#How_do_I_recover_data_from_an_unRAID_disk.3F

 

Before starting on something like this it would be prudent to copy (using that dd command) the disk you are trying to fix to a blank work disk that is outside the array and then run the reiser commands on the copy, then if you have some success you can choose how to proceed.

 

Good luck,

 

Stephen

 

[betaman]

Thanks for the info.  Just to be clear, are you saying I should complete the removal of disk 6 and get it out of the array before doing anything with the disk that lost data?

[Joe L.]

You cannot restore from parity.

All you can do is recreate the MBR and the format the disk and then try to rebuild the file tree scanning the entire partition.

 

It might recover most of what was on the disk... or might not. But you have nothing to lose.  The disk is effectively erased otherwise.

[Joe L.]

Thanks for the info.  Just to be clear, are you saying I should complete the removal of disk 6 and get it out of the array before doing anything with the disk that lost data?

no, recover first.

[betaman]

Joe,

 

Thanks for the info.  Do I follow the steps below or do I need to do something different since my disk is now showing unformatted and doesn't mount when the server boots?  I'm not exactly sure what you meant by recreate the MBR "then" format?  What are the steps to recreate the MBR?  I'll wait for a response as I don't want to screw this up any further but I'm wondering if this is the right approach (i.e. recreate MBR, reformat then reiserfsck)?

 

I have managed to accidently delete a whole bunch of files. The disk hasn't been written to since. Is there a way/program to help me recover them?

Were they on your unRAId server?

 

If yes, then see this entry in the wiki:  http://lime-technology.com/wiki/index.php?title=FAQ#How_can_I_undelete_files_from_an_unRAID_disk.3F

 

In essence

 

stop samba  [all your shares will disappear from network]

killall smbd nmbd

 

un-mount the disk  (md1=disk1, md2=disk2, md10-disk10,  etc.  Use the correct disk for your needs. this example is for disk1)

umount /dev/md1

note the above command is umount, not unmount.

 

run reiserfsck on the disk to rebuild the file tree.  Any files that can be recovered will.  This process could take many hours for a large disk as the entire disk will need to be scanned.

reiserfsck --rebuild-tree -S /dev/md1

It will ask you to confirm the rebuild of the file tree. Make certain you answer Yes (three letters, Capital "Y", lower case "es")

Anything else will be considered as "no"

 

re-mount the disk  (again, use the correct "md" and "disk" for your situation)

mount /dev/md1 /mnt/disk1

re-start samba

/usr/sbin/smbd -D

/usr/sbin/nmbd -D

 

your deleted files (any that could be recovered) will be in a lost+found folder on the disk.

 

See here for a more general description of the process

http://antrix.net/journal/techtalk/reiserfs_data_recovery_howto.comments

 

 

[Joe L.]

If the disk is showing as "unformatted" it might already be partitioned.  Do you see a "Format" button?  If so, and only that disk is unformatted, press it to format the disk.  Then follow the directions in the wiki to recover files.

[betaman]

Well, it would've been the 2nd disk to be unformatted in the array but I went ahead and removed the disk 6 that I was trying to remove when this whole thing started.  Now it is the only disk showing unformatted so I'll go ahead and proceed as you indicated.

[betaman]

Wow...thanks for all the help!  I think I got about 99% of the drive back so I really appreciate the info!

[Joe L.]

99% recovery is way better than nothing.

.

 

I've often thought that giving instructions on how to use "dd" directly on a disk is exactly like giving matches and dynamite to a teenager. 

Sooner or later, somebody is going to get hurt.

 

Joe L.

[Joe L.]

Make sure you do a full parity check now.