I have two boxes here; one running unraid Pro and the other just has the free license. The latter was where I was temporarily storing _ALL_ my data until the new box was up and running the way I wanted.

 

Things started to go downhill when a chunk of data was accidentally deleted. No problem, I scoured the forums and found that running reiserfsck --rebuild-tree -S -l should help find the missing files. Unfortunately, through a little disregard for the security of the rest of the files on the disk and the lack of space to back them up, I went for it have no other copies of this data elsewhere. For shame, I know.

 

It appears that after 2 times, reiserfsk has been unable to finish. Even starting up the server yields these errors: REISERFS error ... invalid format found in block 0. I've read that this can be fixed by remapping but I haven't found any clear instructions on how to accomplish that. I can't seem to re-mount the file system because of it which is likely why it's showing as unformatted in the unraid menu. It also has another error saying reiserfs_read_locked_inode: i/o failure occurred trying to find stat data [of 1 2 0x0 SD]

 

I've been pouring through a lot of threads where people have managed to get data back so I'm hopeful... but just don't know what to try next. I've attached a syslog captured yesterday.

 

Pertinent info:

Running 4.5.3

P4 2.8 box w/ 512MB DDR2

Few month old Samsung 1.5TB data disk involved with rather important files on it.

Just disabled parity (same drive as above) as to not make things worse

 

Any help / suggestions on how to proceed would be appreciated!

Edit: Currently running memtest until I/someone comes up with a new idea. 

syslog-2010-06-11.txt

 

You may have complicated things by un-assigning parity.  We'll have to see.

 

Step 1.

 

Perform a file system check on the drive with the problems.  NOT a rebuild-tree, but

reiserfsck /dev/md1

 

Follow exactly the instructions as shown in the wiki.  You'll need to stop SAMBA and un-mount the drive as described in the wiki here:

http://lime-technology.com/wiki/index.php?title=Check_Disk_Filesystems

 

Do not go further until you post the output of the reiserfsck and get further instructions.

 

Unfortunately, now that you un-assigned parity it cannot be used as it will be invalid once you correct the file-system corruption.

If you had left it in place, it would have had been corrected along with the file-system repair.

 

Also... what else (what other commands) did you try running on the /dev/md1 disk?  In fact, what disk did you type when you attempted to run the rebuild-tree?    I know it is embarrassing, but we need all the clues in order to best figure out how to get to your data.  Did you run a file-system check first? or did you just assume it was good?

 

When was the last time you ran a parity check?

 

In the future, once any change is made to data, parity is immediately in sync.  Do NOT assign or un-Assign any disks upon an error.  Post on the forum asking for assistance.  You only made it more difficult now as it is likely you'll need to do a full parity calculation after the disk file-system is healthy.

 

Joe L.

Ok, should I re-assign? I literally just did that a few minutes ago and haven't done anything since except re-boot and start memtest. This morning when I started it up it looked like it was trying to re-calculate parity which I didn't think was a good thing? Also thought I read somewhere on the forums that it was a good idea haha. Maybe I took it out of context.

Complicating things is that you are running the buggy version of unRAID that may show all your disks as un-formatted when you first start the array.

 

Keep your fingers away from the "Format" button.   Keep your fingers away from the button labeled as "Restore"  It has NOTHING to do with data restoral, and in fact the button is now removed from the 4.5.4 version of unRAID because some pressed it thinking it would re-construct a failed disk.  It is actually a "Delete Disk Configuration and Immediately Invalidate Parity" button.  Not what you want to do when you have a disk failure.

 

If you do see all your disks as un-formatted, just Stop the array and then press "Start" once more.

 

Joe L.

Nope,  too late to re-assign it now...  It has already been marked as invalid and by starting and stopping the array you have written to all the disks just to get them mounted and un-mounted.

 

Joe L.

Step 1.

Perform a file system check on the drive with the problems.  NOT a rebuild-tree, but

reiserfsck /dev/md1

 

Do not go further until you post the output of the reiserfsck and get further instructions.

 

Ok, I have "Replaying journal..

Reiserfs journal '/dev/md1' in blocks 18..8211]: 0 transactions replayed

Zero bit found in on-disk bitmap after the last valid bit.

Checking internal tree.

 

Bad root block 0. (--rebuild-tree did not complete)

 

Aborted"

Sitting at the prompt now.

 

Crap. I wasn't thinking that unassigning parity invalidated it. Makes sense but...  

 

Also... what else (what other commands) did you try running on the /dev/md1 disk?   In fact, what disk did you type when you attempted to run the rebuild-tree?     I know it is embarrassing, but we need all the clues in order to best figure out how to get to your data.  Did you run a file-system check first? or did you just assume it was good?

[/Quote]

 

So a friend and I were working  on this yesterday. I'm not sure of everything tried. (For the record, that's Brass42 in this thread: http://lime-technology.com/forum/index.php?topic=2381.0.)  Definitely --rebuild-tree -S -l and the fix-fixable command.  

 

When was the last time you ran a parity check?

 

Not overly recently. Probably a month or so ago ie just after I got the system up and was transferring first batches of data over.

 

reiserfsck did not suggest the command to run next?  Usually it does.

 

If it did not, then I'll have to assume the rebuild-tree is next.  But... not with any special commands

Just

reiserfsck --rebuild-tree /dev/md1

 

Once we get the file-system sane, we can re-run it with the option to scan the entire disk in the attempt to get back the files you deleted.

 

Joe L.

Even though after running the latter it said "--rebuild-tree did not complete" ?

 

The nice disclaimer states this shouldn't be run without copying data off first and suggests dd_rescue first especially if there are bad sectors. I don't think there should be bad sectors given the new-ness of the drives but how could I be certain? I my original post I said I didn't have room to make a backup but I've since picked up another drive. If this is at all an option at this point I'd like to try and exercise it, especially since I went ahead and invalidated the parity.

 

Thoughts? Or just go ahead and attempt the rebuild tree once again?

 

Thanks for your time btw, Joe.

Even though after running the latter it said "--rebuild-tree did not complete" ?

 

The nice disclaimer states this shouldn't be run without copying data off first and suggests dd_rescue first especially if there are bad sectors. I don't think there should be bad sectors given the new-ness of the drives but how could I be certain? I my original post I said I didn't have room to make a backup but I've since picked up another drive. If this is at all an option at this point I'd like to try and exercise it, especially since I went ahead and invalidated the parity.

 

Thoughts? Or just go ahead and attempt the rebuild tree once again?

 

Thanks for your time btw, Joe.

Is the new drive installed? Do you know its device name?  We could certainly easily make an image of the existing drive as a fallback position.

 

Joe L.

Not in the problematic system. I was referring to some space I have available in my other box (running unraid pro) which is sitting right next to it. Making an image sounds like a good move right about now.

 

I have a new WD Green 2TB drive that has been precleared and is sitting in the other unraid box. Can we leave it there and do the image from one system to the next or would it be easier to move the new drive to the problematic system?

Far far easier if they are in the same system.

 

And Waaaaaaaaaaaaaaayyyyyyyyy faster.

 

Joe L.

ok, seeing as the parity drive is now invalidated, I have removed it from the system and put the 2TB EARS (pre-cleared, jumpered) in its place. Only it and the troubled drive are in the system now.

Faster is better!

 

So should I use dd_rescue? Any tips on getting the image going?

Do you know the device name of the new 2TB drive... this is critical, since if you get it wrong you will be overwriting one of your data disks.

 

I don't think you have "dd_rescue"  but it probably just uses "dd" as you will on the command line.

 

The basic syntax is

dd bs=16384 conv=noerror if=/dev/md1 of=/dev/sdX1

 

Note the "1" on the end of sdX1 and you will need to know the correct letter to substitute for the "X"

 

Depending on how large the source disk is, this could take 4 or more hours to make the copy.

 

I just checked via hdparm -i and the 2TB green is sda and the 1.5TB Samsung is sdb.

 

so to clarify, I'm typing:

dd bs=16384 conv=noerror if=/dev/md1 of=/dev/sda1

 

Alright. I'll get this underway and hope it completes in that time or less.

 

Update: should it give some sort of indication of what it's doing? I decided to telnet in to run this command and so far it's just sitting there.

Screenshot attached from 5 mins in. I guess it's working; hdd led is going nuts. Guess I'm too used to cushy windows where you get progress bars with everything haha.

I just checked via hdparm -i and the 2TB green is sda and the 1.5TB Samsung is sdb.

 

so to clarify, I'm typing:

dd bs=16384 conv=noerror if=/dev/md1 of=/dev/sda1

 

Alright. I'll get this underway and hope it completes in that time or less.

Once it is under way for a 10 minutes or so, you can log in via a second telnet session and type:

killall -USR1 dd

 

In the first telnet session the "dd" command will output how far it has gotten thus far.  You can send it the USR1 "signal" as you like to learn of its progress. 

 

I'll be at a dinner-dance when the copy completes...  It will just end.  At that point you have an identical copy of the failed disk on sda1.  We'll just leave that as is and not touch it.  You'll re-run the --rebuild-tree on /dev/md1.

 

Once it is complete, and there are no more errors when you run

reiserfsck /dev/md1

then you can re-try the rebuild-tree to include all the deleted files too.

The command is:

reiserfsck --rebuild-tree --scan-whole-partition /dev/md1

 

I'll be back later after the dance.  (Ballroom-dancing is another hobby... can't live as a geek entirely  )

 

Joe L.

Haha nice, I'm impressed

 

Thanks for all the help thus far. I'll feel much safer with a copy of whatever's left before trying too much else. I'll let you know how it goes.

 

Update: 1/2 hour in we've copied 172GB of data @ ~90MB/s.

Update: going on 4 hours with 1.2TB copied, still @ ~86MB/s

And.. 5 hours 15 min & 1.5TB later, we're done! On to the rest you suggested now...

 

So just as before, reiserfsck /dev/md1 returned the attached error which kept creeping up while trying to rebuild the tree & fix-fixable yesterday.

Ok, it is time to re-run the --rebuild-tree but as I said earlier, without the scan whole partition option.

 

reiserfsck --rebuild-tree /dev/md1

Got it underway. So far, same first line as the last command: "Reiserfs journal '/dev/md1' inblocks [18..8211]: 0 transactions replayed."

 

I'll leave it be for a bit and see what happens.

 

Ahh here we go. Lots of stuff is getting corrected, so it seems.

Don't just leave it for a bit, and do not disconnect the terminal session if doing it via telnet.  Let it run to completion... and that could take hours if your disk is full or has lots of files.

 

Joe L.

Good to know, yes I'm doing it via telnet. Looks like pass 0 is going to take 45 min if my math is correct. Does it keep doing passes until there are no more errors or?

Good to know, yes I'm doing it via telnet. Looks like pass 0 is going to take 45 min if my math is correct. Does it keep doing passes until there are no more errors or?

I've never seen it do more than one pass.  It might suggest the next option to use if it can't fix things with the current.

And, we're done! Now what should my next step be?

I'm going to go back and try the general reiserfsck /dev/md1.

 

...which yielded the exact same results as before :( (attached). This isn't good and the weekend is quickly slipping away! That being said, so has the night. Thanks again for the help Joe. I think I'm calling it a night to tackle this with a fresh mind tomorrow.

And, we're done! Now what should my next step be?

I'm going to go back and try the general reiserfsck /dev/md1.

 

...which yielded the exact same results as before :( (attached). This isn't good and the weekend is quickly slipping away! That being said, so has the night. Thanks again for the help Joe. I think I'm calling it a night to tackle this with a fresh mind tomorrow.

First, disable any spin-down timers, just in case they are causing the disk to spin down and terminate the file-system check.

I don;t think they would if you are accessing through /dev/mdX but who knows.

 

Second, before you go to sleep, reboot and start a memory test.  Let it run overnight while you sleep.  Expect it to find no memory errors.

 

Joe L.

Not sure about any spin-down timers. None were added that's for sure unless you were referring to timers built into unraid? Even so, I wouldn't think so given that it takes maybe 5 minutes for the check to fail.

 

So it felt like a short night but there are no memory errors detected so far.