So I had a disk fail..... although it turns out the disk probably didn't fail.

 

I replaced the disk with a different one,  started rebuilding from parity overnight, in the morning all of the disks plugged into a one of my sata backplanes have died.  Switch the box off. Startup again after removing the suspect backplane....  now I have 2 red disks....  disk 4 was the original one I started rebuilding.  disk 6 is one i suspect is still fine.

 

The log shows it wont mount for some reason, I've tried running reiserfsck and that comes back without any errors so how do I get the thing to mount?

 

This is the error I see :-

 

Sep 22 18:40:19 Tower kernel: ReiserFS: md6: warning: sh-2006: read_super_block: bread failed (dev md6, block 2, size 4096)

Sep 22 18:40:19 Tower kernel: ReiserFS: md6: warning: sh-2006: read_super_block: bread failed (dev md6, block 16, size 4096)

Sep 22 18:40:19 Tower kernel: ReiserFS: md6: warning: sh-2021: reiserfs_fill_super: can not find reiserfs on md6

 

but the same thing is showing for disk 4 too, the one i started to rebuild.

 

No idea what to do,  get the feeling I've now lost 2 disks worth of data

 

TIA for any suggestions on getting disk6 to mount so I can at least salvage the data from that.

 

 

 

Since both disk4 and disk6 seem to be corrupted somehow, what reiserfs command did you run that came back without any errors?  I don't see how that is possible unless you did it on a different disk?

 

Since you just removed backplanes, did you make certain the disks are assigned to the correct slots in the array?  (Verify on the devices page you did not accidentally assign a data drive as the parity drive.)

 

Oh yes... in case you did not know better.  DO NOT, DO NOT, DO NOT press the button labeled "restore" as it will start a new parity calculation without either of the failed disks, and completely eliminate any chance of data recovery from parity as you will have asked it be calculated new, but without those two drives.

 

Joe L.

Here is the output from reiserfsck :-

root@Tower:~# reiserfsck /dev/sdd1
reiserfsck 3.6.19 (2003 www.namesys.com)

*************************************************************
** If you are using the latest reiserfsprogs and  it fails **
** please  email bug reports to [email protected], **
** providing  as  much  information  as  possible --  your **
** hardware,  kernel,  patches,  settings,  all reiserfsck **
** messages  (including version),  the reiserfsck logfile, **
** check  the  syslog file  for  any  related information. **
** If you would like advice on using this program, support **
** is available  for $25 at  www.namesys.com/support.html. **
*************************************************************

Will read-only check consistency of the filesystem on /dev/sdd1
Will put log info to 'stdout'

Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes
###########
reiserfsck --check started at Tue Sep 22 19:08:41 2009
###########
Replaying journal..
Reiserfs journal '/dev/sdd1' in blocks [18..8211]: 0 transactions replayed
Checking internal tree..finished                               
Comparing bitmaps..finished
Checking Semantic tree:
finished                                                                       
No corruptions found
There are on the filesystem:
Leaves 32702
Internal nodes 201
Directories 29
Other files 125
Data block pointers 33078162 (0 of them are zero)
Safe links 0
###########
reiserfsck finished at Tue Sep 22 19:13:38 2009
###########

 

Maybe I'm just running it on the wrong thing.  I'm a bit fuzzy about how exactly I'm supposed to figure out what to run it on.

 

I'm sure all the disks are in the right place and I haven't touched the restore button.

According to your syslog:

Sep 22 18:40:18 Tower kernel: md: import disk0: [22,0] (hdc) SAMSUNG HD154UI S1Y6J1KS516318 offset: 63 size: 1465138552

Sep 22 18:40:18 Tower kernel: md: import disk1: [3,0] (hda) WDC WD1200JB-00EVA0 WD-WMAEK2829749 offset: 63 size: 117220792

Sep 22 18:40:18 Tower kernel: md: import disk2: [3,64] (hdb) WDC WD800BB-60CJA1 WD-WMAAC1145603 offset: 63 size: 78150712

Sep 22 18:40:18 Tower kernel: md: import disk3: [8,0] (sda) SAMSUNG HD154UI   S1Y6J1KS616605       offset: 63 size: 1465137496

Sep 22 18:40:18 Tower kernel: md: import disk4: [8,16] (sdb) SAMSUNG HD154UI   S1Y6J1KS516537       offset: 63 size: 1465138552

Sep 22 18:40:18 Tower kernel: md: import disk5: [8,32] (sdc) SAMSUNG HD154UI    S1Y6J1KS516320       offset: 63 size: 1465138552

Sep 22 18:40:18 Tower kernel: md: import disk6: [8,48] (sdd) SAMSUNG HD154UI    S1Y6J1KS516326       offset: 63 size: 1465138552

Sep 22 18:40:18 Tower kernel: md: import disk7: [8,64] (sde) WDC WD10EACS-00ZJB0    WD-WCASJ1828278 offset: 63 size: 976762552

Sep 22 18:40:18 Tower kernel: md: import disk8: [8,96] (sdg) WDC WD10EACS-00ZJB0    WD-WCASJ1775088 offset: 63 size: 976762552

Sep 22 18:40:18 Tower kernel: md: import disk9: [8,112] (sdh) WDC WD10EACS-00ZJB0     WD-WCASJ1728696 offset: 63 size: 976762552

Sep 22 18:40:18 Tower kernel: md: import disk10: [8,128] (sdi) WDC WD10EACS-00ZJB0     WD-WCASJ1804275 offset: 63 size: 976762552

/dev/sdd1 would have the partition for disk6... however, you should never run reiserfsck on the raw partition to correct anything as it would not correct parity.

 

Most of the time, you would run reiserfsck on the /dev/mdX device.   (in this case, /dev/md6 ) but with the disk un-mounted, and the array still running. That way, any fixes also get applied to the parity drive and it stays in sync.

 

It does not seem that is possible with two disks failed, so I'm a bit confused too...  Now would be a good time to make a copy of the "config" folder on your flash drive, just in case you need to get the super.dat file back to where it is now.

 

If you think that disk6 is good, and know that disk4 is bad, you could try starting the rebuild process all over again by using a modified "Trust my parity" process.

 

The modification would be that you set disk4 as invalid so it gets rebuilt.

 

First... though... is there a new disk4 in place?

 

Then following the procedure in the wiki, but typing

mdcmd set invalidslot 4

between pressing the two buttons.

 

Joe L.

Thanks Joe,  yes disk 4 has been replaced, there was probably nothing wrong with it however.

 

Just to confirm, doing the trust parity procedure with the mdcmd set invalidslot 4 will result in disk 6 being marked as good and starting a rebuild on disk 4?

 

Is this likely to destroy anything? (other than disk 4!)

Thanks Joe,  yes disk 4 has been replaced, there was probably nothing wrong with it however.

 

Just to confirm, doing the trust parity procedure with the mdcmd set invalidslot 4 will result in disk 6 being marked as good and starting a rebuild on disk 4?

 

Is this likely to destroy anything? (other than disk 4!)

It should start writing disk4, and reading all the others....

If you see anything else happening... press "Stop" as soon as your fingers can reach for the mouse. 

 

I'd also keep an eye on the syslog while doing this... you should not see disk errors.

 

Open up a telnet window, log in as root, and type

tail -f /var/log/syslog

Type control-C to get out of it when your re-construct is done.

 

You must not have any disk "busy" when doing all this, so log off of any others, and

type

cd /root

before typing the mdcmd command. (it lives in /root)

 

To trust the parity disk in this way, all of the disks must be in place that were when you created the parity data.  If it complains it cannot mount disk6, (or any disk other than disk4) don't go further... DO NOT start the array... 

 

Joe L.

Thanks again Joe,  OK looks like its rebuilding disk 4..... see what happens next

 

fingers crossed.

It sounded like your backplane lost power the last time your tried this...  I'll keep my fingers crossed.

 

You should see writes to disk4, and reads from all the other disks incrementing on the management console.

 

Joe L.

OK Good news it seems.  The rebuild from the trust my parity procedure was successful.

 

Unfortunately it seems during the last rebuild the mover process decided to move a few files for me before the power failed to the backplane.

 

Obviously this has resulted in parity being not quite right   So despite a successful rebuild, disk4 was still showing as unformatted.  At this point I wasn't hopeful to say the least....

 

I unmounted disk 4 with the array running as per the instructions in the wiki and ran reiserfsck, it reported the disk needed a tree rebuild.  Ran reiserfsck with the --rebuild-tree option overnight and in the morning is seems to have recovered most of the data,  I haven't had time to check what's actually missing but the important stuff is there.

 

Lessons learned..... 

 

1) Cancel the mover schedule if I'm rebuilding a disk. 

2) If a disk "fails" wait for the rebuild before doing destructive testing in another PC  

 

All of this pain is attributed to an Icy Dock 4 in 3 backplane. (and my own stupidity  )

 

Many thanks to Joe L for your excellent advice.

 

I unmounted disk 4 with the array running as per the instructions in the wiki and ran reiserfsck, it reported the disk needed a tree rebuild.  Ran reiserfsck with the --rebuild-tree option overnight and in the morning is seems to have recovered most of the data,  I haven't had time to check what's actually missing but the important stuff is there.

 

This is where I think we need some sort of MD5sum verification tool (so we know what files may be corrupt).

Or a par2 tool to verify and/or rebuild the corrupt segments.

 

I'm working on an md5sum tool.

They key issue I have is tracking it in some form of database.

I'm exploring a miniversion of perl (which would be pretty easy) or an updated modified enhanced bash that includes builtins so we can use gdbm files right from the shell.

 

I have thought it out fully, but this is an addon I think we could all use "just in case".

 

I unmounted disk 4 with the array running as per the instructions in the wiki and ran reiserfsck, it reported the disk needed a tree rebuild.  Ran reiserfsck with the --rebuild-tree option overnight and in the morning is seems to have recovered most of the data,  I haven't had time to check what's actually missing but the important stuff is there.

 

This is where I think we need some sort of MD5sum verification tool (so we know what files may be corrupt).

Or a par2 tool to verify and/or rebuild the corrupt segments.

 

I'm working on an md5sum tool.

They key issue I have is tracking it in some form of database.

I'm exploring a miniversion of perl (which would be pretty easy) or an updated modified enhanced bash that includes builtins so we can use gdbm files right from the shell.

 

I have thought it out fully, but this is an addon I think we could all use "just in case".

 

 

After the last disaster I just had I would be very interested in a easy to use & reliable md5sum tool. I hope you can get it figured out..

Or a par2 tool to verify and/or rebuild the corrupt segments.

 

Last summer, I spoke with a dev for a hard drive manufacturer that is looking at implementing this in hardware as a form of enhanced ECC... it can add several orders of magnitude of improvement to URE rates.

Glad to hear the data is back, or at least most is back.

 

I don't think the unformatted problem was necessarily because of the mover. Besides, the mover should have kept parity updated even if a rebuild was in process.

 

The same thing happened another time when someone had to rebuild a drive which is documented in this thread.

 

http://lime-technology.com/forum/index.php?topic=3716.0

 

It appears that manually using the mdcmd command to do a rebuild onto a new drive results in the drive not being partitioned and appearing as unformatted.

 

I think Joe will remember that one.

 

Peter

 

Yeah I'm not sure why the drive was garbage after the rebuild, its possible the mover was still running when the backplane went kaputt which would certainly not help matters.