Urgent help needed please:

 

Running unRAID Server Pro version: 5.0-rc5 (rc8 didn't work for me, gave me lots of problems, not tried new v10 yet). Full syslog included.

 

Problem: Copying a folder to my server crashed the server and I couldn't access it from my Win7 PC. I had to hard reboot it.

 

After the reboot it took a lot longer than normal to be responsive and allow me access to \\tower and unmenu through my browser.

 

When it finally worked Disk9 is saying unformatted and the server has started a parity check and found 6 errors so far:

 

Total size: 3 TB

Current position: 33.11 GB (1%)

Estimated speed: 16.04 MB/sec

Estimated finish: 3083 minutes

Sync errors corrected: 6

 

Is it safe for it to be doing a parity check with Disk9 missing? I hope it's not overwriting all Disk9 info from the parity making it impossible to rebuild the data once I swop the disk with a new one.

 

Looking at the syslog I noticed lots of the following lines repeated over and over:

 

Jan 14 20:46:12 Tower kernel: handle_stripe read error: 103544/9, count: 1 (Errors)

Jan 14 20:46:12 Tower kernel: md: disk9 read error (Errors)

 

Checking smart for disk 9 shows:

 

» reallocated_sector_ct=1

» reallocated_event_count=4

» current_pending_sector=2

» ata_error_count=289

 

I noticed back in October that same drive went from 0 to ata_error_count=107 but remained stable since then.

 

It looks like it's time to replace that drive before any more problems occur.

 

My concern is the data that's on it and making sure I don't lose anything.

 

I'll wait until the parity check is complete and then reboot the server as I've had a disk show up as unformatted before but after a reboot it mysteriously went back to normal and didn't need formatting and rebuilding.

 

Any advice would be greatly welcomed.

 

I'm slightly concerned by the fact that the GUI says "Format will create a file system in all Unformatted disks, discarding all data currently on those disks." but doesn't mention anything about rebuilding the data and I cannot browse any files that are on Disk9.

 

Thanks,

Ben

syslog-2013-01-14.zip

First, whatever you do - DO NOT FORMAT DISK 9!!!

 

You have bad sectors, media errors in the early part of Disk 9, and I strongly believe that they are in the Reiser file system itself.  That would explain why it was unable to load the superblock, and without the superblock, the system thinks there is no file system installed, and that results in it currently marking it as Unformatted.  Because the drive is there and readable, it is able to access the rest of the sectors (except the bad ones) and run a parity check.  Personally, I'd cancel the parity check though, until this issue is fixed.

 

To be honest, I'm not quite sure what to advise you yet, perhaps others will have better ideas...  This unfortunately looks quite serious, and while I believe you will be able to recover all or almost all of the files on Disk 9, it is going to take some work and time.  I'd pause for a bit to allow everyone to come up with the best strategy.  I'm not sure we have seen an issue quite like this where the file system is trashed, including the superblock, AND there are bad sectors in the file system area.  The badblocks command may be the first step...

One possibility would be to pull Disk 9 and install a new drive and rebuild Disk 9 onto it, but unfortunately we cannot guarantee an intact superblock.  But since that is a problem no matter which way we go, it seems like a good choice because it skips having to deal with the bad sectors.

I agree. Rebuild the disk. Use a spare disk or preclear the current one and rebuild. Once the disk has zero pending sectors the file system issue can be resolved.

Thanks for the advice.

 

After leaving it parity checking when I went to bed last night it's now at:

 

Total size: 3 TB

Current position: 282.89 GB (9%)

Estimated speed: 7 MB/sec

Estimated finish: 6472 minutes

Sync errors corrected: 73

 

A scary amount of errors reported in \\tower\main menu: 21516!

 

I've cancelled the parity check. I'll leave the server off until my new disk arrives tomorrow and I'll preclear it in my second unRAID server, then use it to rebuild Disk9 and see what's been recovered.

 

WOW:

» reallocated_sector_ct=14

» reallocated_event_count=17

» current_pending_sector=10

» ata_error_count=2154

 

New syslog attached.

syslog-2013-01-15.zip

More help desperately needed! After swopping the failed 2TB with a brand new and precleared 3TB and rebuilding Disk9 it still shows up as empty 

 

After rebuilding Disk9 reports Reads: 185, Writes: 9,079,999 Size: 3T % Used: Empty Free: 3T

 

It looks like all the data is there but unRAID cannot read it.

 

Is there a way to make unRAID see all my data or is it all lost for good?

 

Is it possible to insert the old Disk9 into another PC to try salvage any data still readable or will that appear blank now too?

 

Thanks for any help, It'd be such a shame to lose nearly 2TB of films.

 

New syslog after rebuild included.

syslog-2013-01-18.zip

It's OK, this was expected.  Syslog showed a few oddities, in that it actually tried to mount the virtual but corrupted Disk 9, before rebuilding it.  It IS a little odd that it tried to format Disk 9, before it began the rebuild, but the rebuild overwrote it with the true Disk 9 image.

 

This is what we talked about before, where you had both bad sectors and a corrupted file system.  Now we have no bad sectors, but we still have a very corrupted ReiserFS, so we start the process of rebuilding the file system and recovering the files.  It takes awhile, with no guarantees for perfect recovery.  Usually, most of the files are recovered, but sometimes with changed names, changed locations (in a lost&found folder), and possibly with some corruption within the file.  So some files may be recovered, but if corrupted in the wrong place, may not be usable.

 

For some help with the recovery process, please see the Check Disk File systems wiki page.  You will definitely have to rebuild the superblock, then other steps as dictated.  Read and try to understand the page first, before you try anything.  Read also the linked stories (near the bottom) of others who have gone through the process.  There are a number of other users with experience much more recent than mine, that can chime in here and help.  Good luck!

Great stuff. Thank you very much for your guidance 

OK, here is my output from running the command reiserfsck --check /dev/md9

 

Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes

###########

reiserfsck --check started at Fri Jan 18 19:04:54 2013

###########

Replaying journal: Done.

Reiserfs journal '/dev/md9' in blocks [18..8211]: 0 transactions replayed

Checking internal tree.. finished

Comparing bitmaps..finished

Checking Semantic tree:

finished

No corruptions found

There are on the filesystem:

        Leaves 1

        Internal nodes 0

        Directories 2

        Other files 0

        Data block pointers 0 (0 of them are zero)

        Safe links 0

###########

reiserfsck finished at Fri Jan 18 19:07:23 2013

###########

As it didn't recommend any next steps would I be safe in assuming I should do the following next?

If your file system has only minor issues, then running reiserfsck --fix-fixable should be all that is necessary.

How long roughly would this command take to execute because I'm wondering whether it's worth running the command directly from the server by hooking it up to my TV or from the telnet session I currently have open?

 

If it takes many hours then I'd rather not have to keep my main PC on overnight and also risk anything crashing on my PC that might close the telnet session and cause issues or mean I have to start the reiserfsck command again from the beginning.

You were hoping it would be that easy?    I'm sorry, but you will have to treat this as a worst case recovery, and rebuild the superblock.  It looks like it repeated what it did before the rebuild, when it saw the lack of superblock and decided it needed to be formatted.  So what you just checked/tested is a fresh format, a totally empty file system, no nodes at all.  Running fix-fixable won't find anything to fix either.

 

You need to go directly to the rebuild-sb command, please follow the Important Notes, especially #2.  Make sure you are in Maintenance mode, and do not use Telnet for this.  Hook up a monitor, so there is no chance of it being interrupted.  As I recall, after the rebuild-sb, you will need to do the rebuild-tree, and that will take a long time.  Sorry, there are no shortcuts.

Thanks again. For some reason I'm not being asked all the questions like the people in the previous posts about using reiserfsck --rebuild-sb.

 

I've used telnet here just to copy/paste then cancelled out before actually running the command.

 

It only asked me one question: Did you use resizer(y/n)[n]: n

root@Tower:~# reiserfsck --rebuild-sb /dev/md9

reiserfsck 3.6.21 (2009 www.namesys.com)

 

*************************************************************

** If you are using the latest reiserfsprogs and  it fails **

** please  email bug reports to [email protected], **

** providing  as  much  information  as  possible --  your **

** hardware,  kernel,  patches,  settings,  all reiserfsck **

** messages  (including version),  the reiserfsck logfile, **

** check  the  syslog file  for  any  related information. **

** If you would like advice on using this program, support **

** is available  for $25 at  www.namesys.com/support.html. **

*************************************************************

 

Will check superblock and rebuild it if needed

Will put log info to 'stdout'

 

Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes

 

Did you use resizer(y/n)[n]: n

rebuild-sb: wrong block count occured (732566633), fixed (732566624)

Reiserfs super block in block 16 on 0x909 of format 3.6 with standard journal

Count of blocks on the device: 732566624

Number of bitmaps: 22357

Blocksize: 4096

Free blocks (count of blocks - used [journal, bitmaps, data, reserved] blocks): 732536065

Root block: 8211

Filesystem is clean

Tree height: 2

Hash function used to sort names: "r5"

Objectid map size 2, max 972

Journal parameters:

        Device [0x0]

        Magic [0x28e6e97b]

        Size 8193 blocks (including 1 for journal header) (first block 18)

        Max transaction length 1024 blocks

        Max batch size 900 blocks

        Max commit age 30

Blocks reserved by journal: 0

Fs state field: 0x1:

        some corruptions exist.

sb_version: 2

inode generation number: 0

UUID: 5eb12d98-e8f3-46d9-afca-037ea4dd19ec

LABEL:

Set flags in SB:

        ATTRIBUTES CLEAN

Mount count: 3

Maximum mount count: 30

Last fsck run: Fri Jan 18 20:59:59 2013

Check interval in days: 180

Is this ok ? (y/n)[n]:

Does this look OK to proceed?

After reading more threads on the topic I went ahead and executed the reiserfsck --rebuild-sb /dev/md9 command followed by another reiserfsck --check /dev/md9 and then reiserfsck --rebuild-tree /dev/md9 and nothing seems to have happened except it created an empty lost+found folder that windows wouldn't let me access.

 

The 4 passes of the --rebuild tree only took about 15 minutes so it obviously didn't run properly.

 

Not sure what to try next.

I was stumped at first, and hoping that others with more recent experience would jump in, but did a little research, and found we had forgotten one more parameter - the --scan-whole-partition extra parameter for --rebuild-tree!  Try again with:  reiserfsck --scan-whole-partition --rebuild-tree /dev/md9

Thanks, I'll try that now. I had seen Joe L. say the following in the thread linked below so left that part out:

scan-whole-partition is used typically when you've deleted files, or re-formatted the disk and want to attempt to recover the deleted files.

 

It is not normally needed to fix a file-system.

http://lime-technology.com/forum/index.php?topic=19041.0

I agree with Joe on that, but in your case the disk was reformatted, plus there were bad sectors in various places.  I believe the scan-whole-partition option to be correct for you.

After running reiserfsck --scan-whole-partition --rebuild-tree /dev/md9 followed by reiserfsck --check /dev/md9 my drive no longer says it's empty and a huge lost+found folder has been created. How do I access this folder as when I try browse it from windows explorer it says "You do not have permission to access \\tower\disk9\lost+found"

 

I can browse the folder through firefox http://tower/Shares/Share?name=lost%2Bfound and the folder is set to Security: Public. I've tried changing it to Secure and giving myself read/write access the same as my other shares but that doesn't work either.

 

A funny thing just happened. I noticed there was no number next to Split Level so I entered '2' then Apply and it said "share lost+found has been deleted" which nearly gave me a heart attack haha but luckily it hasn't been deleted!

 

I've seen someone on another thread say they used telnet to move the lost+found into another share. Would this let me access the files?

The files are probably set to root as the user.  If you run

newperms path-to-lost+found

from a telnet/console session then this should set the permissions to allow access via the share.

Thanks itimpi.

 

After a reboot I cannot access my main share now either from the mapped drive or through TOWER in network places. It's asking for the root password and when I enter it it says "Access is denied".

Sounds as if might still have permission issues so you might want to run the newperms utility with no parameters - that will reset all files on the share.

Thanks again. Running http://tower/Utils/NewPerms 

Great news, it's all been a success! 

 

After NewPerms I had to chose log in as other user and enter my Windows logon name as it was asking for the root password whenever I tried to access folders on my server.

 

The lost+found folder is very neatly organised.

 

All folders named something like 4_24537 are my root folders (eg ebooks, Full BluRays, etc) and all subfolder names and files are intact and not renamed.

 

All folders named something like 23662_45278 were folders that I had previously deleted which it has recovered so are safe to remove again to free up the disk space.

 

There were also 6161 loose files (148GB) with names like 22767_23298 and no file extension which I'm hoping are fragments of previously deleted files that it found.

 

Are there any further steps required or am I OK now I've moved my recovered folders from lost+found to my main share?

 

Thanks again for all the help. Hopefully this thread will come in handy for any one else who has this problem in the future.