June 13, 201016 yr Not sure about any spin-down timers. None were added that's for sure unless you were referring to timers built into unraid? Even so, I wouldn't think so given that it takes maybe 5 minutes for the check to fail. So it felt like a short night but there are no memory errors detected so far. I had no way of knowing how long it was before the "Killed" error message at the end occurred. Yes, I was referring to the built-in spin-down timers in unRAID set on the "Settings" page. For now, set the spin-down to "Never" just so it won't be tempted. are there any messages in the syslog when it aborts the --rebuild-tree ??
June 13, 201016 yr Author Ok, spin-down set to "never". After running the reiserfsck again, I paid more attention to the time that passes. About 10 seconds in is when the "0 transactions replayed message appears. About 3 minutes later it follows with the "Zero bit found in on-disk bitmap after the last valid bit and spits out the Bad root block 0" once again, aborting the check. Ok, so here's this morning's syslog after only rebooting and running the command above. syslog-2010-06-13.txt
June 13, 201016 yr Author I decided to get a smart report on the drive to see if there were any clues on the hardware side of things. To my untrained eye, it looks not too bad in there. The only thing that stands out is the "Hardware_ECC_Recovered being a 7 digit number... Running a short test now. Hrm, I've never run an actual smart test via telnet before. Should I expect to see results like the smart report? It just came back to the prompt; no results displayed anywhere that I can tell? smart.txt
June 13, 201016 yr Ok, spin-down set to "never". After running the reiserfsck again, I paid more attention to the time that passes. About 10 seconds in is when the "0 transactions replayed message appears. Normal message... There may, or may not be transactions to be replayed. In your case, there are none. About 3 minutes later it follows with the "Zero bit found in on-disk bitmap after the last valid bit and spits out the Bad root block 0" once again, aborting the check. I'd see if the following command can fix the bad root block reiserfsck --fix-fixable /dev/md1 [check]Ok, so here's this morning's syslog after only rebooting and running the command above. I'll take a look.
June 13, 201016 yr I decided to get a smart report on the drive to see if there were any clues on the hardware side of things. To my untrained eye, it looks not too bad in there. The only thing that stands out is the "Hardware_ECC_Recovered being a 7 digit number... Running a short test now. Hrm, I've never run an actual smart test via telnet before. Should I expect to see results like the smart report? It just came back to the prompt; no results displayed anywhere that I can tell? The raw numbers are meaningful ONLY to the manufacturers.. All disks have hardware error correcting... some disks show the raw number, some do not, the value is meaningless. There is nothing wrong in the smart data report. The "short" test and "long" tests are requests for the disks to run the internal test. They print nothing. After waiting the appropriate length of time, you can request another status report and see their results near the bottom of the report. They are referred to as "offline" tests. For what we are doing, they'll tell us nothing.
June 13, 201016 yr Author I'd see if the following command can fix the bad root block reiserfsck --fix-fixable /dev/md1 It says fixed but then still reports a bad root block when running the rebuild tree command. syslog-2010-06-13-2.txt
June 13, 201016 yr Yes, but it still aborts... I did some searching on google and found that the version of reiserfsck we have on unRAID has a bug that causes the abort. I've found the source for a newer version, but not yet an installable compiled version. Perhaps I'll try compiling it. We are looking for a version of reiserfsck later than 3.6.19 The bug was apparently fixed in 3.6.20, and the source code I found is for 3.6.21. Joe L.
June 13, 201016 yr Download from here: http://carroll.cac.psu.edu/pub/linux/distributions/slackware/slackware-current/slackware/a/reiserfsprogs-3.6.21-i486-1.txz Install with installpkg reiserfsprogs-3.6.21-i486-1.txz Then try the fix-fixable once more, and then the --rebuild-tree Joe L.
June 13, 201016 yr Author Most interesting discovery. This gives me hope again. I've installed the new version and running now. Update: No such luck. Still aborts. I have never installed packages before. It appears to have installed... should I have maybe tried restarting before issuing the command again?
June 13, 201016 yr Author Tried rebooting for the fun of it and running again. Same result. At this point I'm less concerned about recovering the deleted images and much more concerned with getting all of the "once good" data off there. I wish I was more familiar with linux / unraid to help troubleshoot this road block. What if I made a copy of the flash drive's contents then upgraded to 4.5.4? Edit: The other thread in which Brass posted, it was suggested to run reiserfsck --rebuild-sb but from what I've seen in the wiki it can be rather dangerous. I'm wondering if this is last-resort time or if running this in the current state would only yield worse results. Edit 2: So when I restart the system, I get two Reiserfs errors: invalid format found in block 0 and i/o failure occurred trying to find stat data of [1 2 0x0 SD]. Edit 3: The latter part of that last error: data of [1 2 0x0 SD] and references to Block 0 both seem to indicate an issues at the beginning of the drive, no? I mean searching for the 2nd error yields a ton of search results with "stat data of [xxxxxx xxxxxx ....] really large numbers. I haven't found any quite like mine. I've seen recovery strategies in certain software that work "from both sides" of a drive. Is there any way to come at this "from the back end" so to speak, if the errors are happening right from the get-go? In searching for those errors I've come up with people running ubuntu and other linux distros panicking and somehow backing up their data. I don't have the foggiest idea of how this is possible. What are they backing up if there's an i/o error and issues with the file system? Is any of my data (aside from the accidentally deleted stuff) accessible somehow at this point? Edit 4: One other time I tried using the windows XP driver (YAReG) to pull data off an unraid drive. Could that work in this case or not seeing as the fs is acting up? Nope. It was crazy slow but... Ok, tried YAReg, didn't detect anything (not surprisingly). I tried the drive we made the image to; doubt the original would behave any differently. What exactly did dd do? The 2TB drive showed up in Windows XP's disk management as unallocated while the 1.5TB drive shows up as healthy with an unknown file system. The 2TB did not show up in YAReg but the 1.5 did, with errors. (No files were accessible). Also interesting to note that when I pulled this drive out of the case I was able to hear it start up much more clearly. It actually doesn't sound that healthy. There's a lot of thrashing / churning upon start-up, then it settles down. I started up the parity drive to compare and surprisingly it goes through a similar sequence. Just not what I'm used to hearing from drives... Edit 5: Here was that quote: "A long while back I had to run reiserfsck with rebuild tree and it failed (don't remember if it was a seg fault or not). Anyway - problem turned out to be a bad sector on the disk. I had to get that sector to remap before I was able to get reiserfsck to complete." from this thread. Could the same be happening here?
June 14, 201016 yr Tried rebooting for the fun of it and running again. Same result. At this point I'm less concerned about recovering the deleted images and much more concerned with getting all of the "once good" data off there. I wish I was more familiar with linux / unraid to help troubleshoot this road block. What if I made a copy of the flash drive's contents then upgraded to 4.5.4? It would make no difference at all. You are complicating your life by doing it now. Edit: The other thread in which Brass posted, it was suggested to run reiserfsck --rebuild-sb but from what I've seen in the wiki it can be rather dangerous. I'm wondering if this is last-resort time or if running this in the current state would only yield worse results. Edit 2: So when I restart the system, I get two Reiserfs errors: invalid format found in block 0 and i/o failure occurred trying to find stat data of [1 2 0x0 SD]. Edit 3: The latter part of that last error: data of [1 2 0x0 SD] and references to Block 0 both seem to indicate an issues at the beginning of the drive, no? I mean searching for the 2nd error yields a ton of search results with "stat data of [xxxxxx xxxxxx ....] really large numbers. I haven't found any quite like mine. I've seen recovery strategies in certain software that work "from both sides" of a drive. Is there any way to come at this "from the back end" so to speak, if the errors are happening right from the get-go? In searching for those errors I've come up with people running ubuntu and other linux distros panicking and somehow backing up their data. I don't have the foggiest idea of how this is possible. What are they backing up if there's an i/o error and issues with the file system? Is any of my data (aside from the accidentally deleted stuff) accessible somehow at this point? Edit 4: One other time I tried using the windows XP driver (YAReG) to pull data off an unraid drive. Could that work in this case or not seeing as the fs is acting up? Nope. It was crazy slow but... Ok, tried YAReg, didn't detect anything (not surprisingly). I tried the drive we made the image to; doubt the original would behave any differently. What exactly did dd do? The 2TB drive showed up in Windows XP's disk management as unallocated while the 1.5TB drive shows up as healthy with an unknown file system. The 2TB did not show up in YAReg but the 1.5 did, with errors. (No files were accessible). Also interesting to note that when I pulled this drive out of the case I was able to hear it start up much more clearly. It actually doesn't sound that healthy. There's a lot of thrashing / churning upon start-up, then it settles down. I started up the parity drive to compare and surprisingly it goes through a similar sequence. Just not what I'm used to hearing from drives... Edit 5: Here was that quote: "A long while back I had to run reiserfsck with rebuild tree and it failed (don't remember if it was a seg fault or not). Anyway - problem turned out to be a bad sector on the disk. I had to get that sector to remap before I was able to get reiserfsck to complete." from this thread. Could the same be happening here? You showed me the output of the smart report. It did not indicste any sectors pending re-allocation. Your original rebuild-tree was interrupted. Probably by the bug in reiserfsck when it aborted. It did not finish the fix of the file-tree in the file-system. You said you did not know how to install the package. You also said you rebooted. You would need to install it again after every reboot. To install it you change directoy to where you downloaded it. If you downloaded it to your PC and then moved it to \\tower\flash you would log in via telnet and type cd /boot Then type installpkg reiserfsprogs-3.6.21-i486-1.txz It will say something like this: installpkg reiserfsprogs-3.6.21-i486-1.txz Verifying package reiserfsprogs-3.6.21-i486-1.txz. Installing package reiserfsprogs-3.6.21-i486-1.txz: PACKAGE DESCRIPTION: # reiserfsprogs (Reiserfs filesystem utilities) # # These utilities are used for Reiserfs. Reiserfs is a file system # based on balanced tree algorithms. Learn more at the home site: # http://www.namesys.com # # Reiserfs is the work of Hans Reiser and many others. See the file # /usr/doc/reiserfs*/README for full credits. # Executing install script for reiserfsprogs-3.6.21-i486-1.txz. Package reiserfsprogs-3.6.21-i486-1.txz installed. You can see the version number by typing reiserfsck -V As far as the file system we copied to the other disk. We did not set the partition type byte. It is still a zero from the pre-clearing process. We did not need to worry about the OS knowing the partition had data since it was not going to be mounted. That is why your other PC does not see a file-system. (we did not set the byte telling it it had a specific type, we just populated the partition with data from the other drive.) Please do not write to the 2TB drive. We have it as a fall-back. We can be adventurous with the bad disk since we can get back to how it looked before any attempted repair with a single command. That command would read from the 2TB disk and write it back to the defective disk... still defective, but exactly the same corruption as befor any attempted repair. If you wish to rebuild the superblock be aware you will be asked a series of prompts. The correct responses are NOT the default prompts reiserfsck offers. The correct responses are those pointed to by this post: http://lime-technology.com/forum/index.php?topic=1483 You can then try the reiserfsck --check /dev/md1 It may still complain, but if not try reiserfsck --fix-fixable /dev/md1 and if not corrected reiserfsck --rebuild-tree /dev/md1 and if it then works try adding the option to scan the entire disk. If you want to get the disk back to the same exact state as it was when we made the image copy to the 2TB disk type: dd bs=16384 if=/dev/sda1 of=/dev/md1 That will copy from the 2TB drive to the /dev/md1 disk and we can try a different approach. The danger of the rebuild-sb option is minimized because we basically have an "undo" Joe L
June 14, 201016 yr Author You would need to install it again after every reboot. I did manage to get it to install the first time but did not realize I needed to install it every re-boot. I'm pretty sure I attempted everything listed after the first install (before rebooting) but it won't hurt to try again just to make sure. This time I'll verify the version number as well. I'll post back shortly. Update: Installed the package, verified the version as .21 but just as last time, running reiserfsck --check just spewed out bad root block 0 and aborted. I'm reading through that thread and going to try the rebuild command.
June 14, 201016 yr Author Curious? I followed the thread you pointed to as well as the one linked in the wiki but it didn't ask any of the questions. It just went ahead and did this: root@ihsserver04:~# cd root@ihsserver04:~# samba stop root@ihsserver04:~# umount /dev/md1 umount: /dev/md1: not mounted root@ihsserver04:~# reiserfsck --rebuild-sb /dev/md1 reiserfsck 3.6.21 (2009 www.namesys.com) ************************************************************* ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will check superblock and rebuild it if needed Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes rebuild-sb: wrong tree height occured (65535), zeroed Reiserfs super block in block 16 on 0x901 of format 3.6 with standard journal Count of blocks on the device: 366284624 Number of bitmaps: 11179 Blocksize: 4096 Free blocks (count of blocks - used [journal, bitmaps, data, reserved] blocks): 366284624 Root block: 0 Filesystem is clean Tree height: 0 Hash function used to sort names: "r5" Objectid map size 544, max 972 Journal parameters: Device [0x0] Magic [0x61bfef6a] Size 8193 blocks (including 1 for journal header) (first block 18) Max transaction length 1024 blocks Max batch size 900 blocks Max commit age 30 Blocks reserved by journal: 0 Fs state field: 0xfa03: FATAL corruptions exist. some corruptions exist. sb_version: 2 inode generation number: 13620 UUID: 31f35c71-89df-45a0-bb9c-0bb9236a068e LABEL: Set flags in SB: ATTRIBUTES CLEAN Mount count: 12 Maximum mount count: Disabled. Run fsck.reiserfs( or use tunefs.reiserfs( to enable. Last fsck run: Never with a version that supports this feature. Check interval in days: Disabled. Run fsck.reiserfs( or use tunefs.reiserfs( to enable. Is this ok ? (y/n)[n]: y The fs may still be unconsistent. Run reiserfsck --check.
June 14, 201016 yr [quote name="TyantA" post="64924" timestamp="1276476693"] The fs may still be unconsistent. Run reiserfsck --check. OK, do as it says... run the --check and see what it says to do next.
June 14, 201016 yr Author Yup, already on it. Ok, got the exact same results we've been getting: root@ihsserver04:~# reiserfsck --check /dev/md1 reiserfsck 3.6.21 (2009 www.namesys.com) ************************************************************* ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will read-only check consistency of the filesystem on /dev/md1 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes ########### reiserfsck --check started at Sun Jun 13 19:54:44 2010 ########### Replaying journal: Done. Reiserfs journal '/dev/md1' in blocks [18..8211]: 0 transactions replayed Zero bit found in on-disk bitmap after the last valid bit. Checking internal tree.. Bad root block 0. (--rebuild-tree did not complete) Aborted root@ihsserver04:~# Stuff like this makes me nervous: Fs state field: 0xfa03: FATAL corruptions exist. some corruptions exist.
June 14, 201016 yr Author So apparently it's on its way to trying to rebuild the file system tree again. So far I have: 0%block 10689002: The number of items ( is incorrect, should be (-) - corrected. The list is growing as it was last night when we ran this; back and forth between free space and number of items being incorrect and "corrected". I believe the other night it took 45 min or so to run. Will report back once it's done. You said reiserfsck --rebuild-tree /dev/md1 and if it then works try adding the option to scan the entire disk. I know I've seen the option around but my brain is mush after working on all this today. What is it again? Or better yet, I should learn how to get options displayed. Would it just be -? or something (guess).
June 14, 201016 yr On some commands you can do that... I usually type man reiserfsck in the google search box and you'll get the manual pages for the command. In any case, type: reiserfsck --rebuild-tree --scan-whole-partition /dev/md1 and don't disconnect the terminal session until it completes. (or have power failures like I did today... nearly an hour outage after a summer storm. Luckily, after 3 minutes my server shut itself down cleanly when the UPS batteries started to loose their charge. An hour later, when power returned, the server booted itself back up with no problems at all.) Joe L.
June 14, 201016 yr Author Thanks for the tips. We'll see how this current command does then I'll try the whole partition switch. Yeah.... power failures. Sigh. This box actually took one or two hits from power outages within the last month or so. Ran out of UPSes. Have one hooked up to my other box and other workstations in the house... It had seemed fine at first glance so I think the current issues are unrelated. Never know.
June 14, 201016 yr Author Ok, here's what we've got: Will rebuild the filesystem (/dev/md1) tree Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Replaying journal: Done. Reiserfs journal '/dev/md1' in blocks [18..8211]: 0 transac Zero bit found in on-disk bitmap after the last valid bit. ########### reiserfsck --rebuild-tree started at Sun Jun 13 20:06:34 20 ########### Pass 0: ####### Pass 0 ####### Loading on-disk bitmap .. ok, 79130755 blocks marked used Skipping 19389 blocks (super block, journal, bitmaps) 79111 ad 0%block 10689002: The number of items ( is incorrect, sho d block 10689002: The free space (7) is incorrect, should be .block 21949326: The number of items (1) is incorrect, shou block 21949326: The free space ( is incorrect, should be block 21950313: The number of items (1) is incorrect, shoul block 21950313: The free space ( is incorrect, should be ...20%.block 107354135: The number of items (1) is incorrec rrected block 107354135: The free space ( is incorrect, should be block 107355122: The number of items (1) is incorrect, shou block 107355122: The free space ( is incorrect, should be .block 138667788: The number of items (2) is incorrect, sho d block 138667788: The free space (12) is incorrect, should b .block 144050718: The number of items (2) is incorrect, sho d block 144050718: The free space (12) is incorrect, should b .block 166769764: The number of items (3328) is incorrect, cted block 166769764: The free space (768) is incorrect, should pass0: vpf-10110: block 166769764, item (0): Unknown item t 469762129 0x0 (15)] - deleted 40%..block 210881533: The free space (0) is incorrect, should be (40 48) - corrected pass0: vpf-10110: block 210881533, item (0): Unknown item type found [1208877058 3820628 0x690072 (15)] - deleted .block 216625319: The number of items (6679) is incorrect, should be (1) - corrected block 216625319: The free space (6696) is incorrect, should be (4048 ) - corrected pass0: vpf-10200: block 216625319, item 0: The item [439162081 64422 870 0xa2f00011a1d1a2e IND (1)] with wrong offset is deleted block 217345312: The number of items (8392) is incorrect, should be (1) - corrected block 217345312: The free space (8406) is incorrect, should be (4048 ) - corrected pass0: vpf-10110: block 217345312, item (0): Unknown item type found [551231668 81928412 0x20ca20c2 (15)] - deleted block 347410342: The number of items (256) is incorrect, should be (1) - correctedsecc block 347410342: The free space (256) is incorrect, should be (4048) - corrected pass0: vpf-10110: block 347410342, item (0): Unknown item type found [131075 1075052544 0x4f400000000006 (4)] - deleted left 0, 19877 /secc 136322 directory entries were hashed with "r5" hash. "r5" hash is selected Flushing..finished Read blocks (but not data blocks) 79111366 Leaves among those 97712 - leaves all contents of which could not be saved and deleted 18 Objectids found 136324 Pass 1 (will try to insert 97694 leaves): ####### Pass 1 ####### Looking for allocable blocks .. finished Killed Now I'll try it with the whole partition switch. My math indicates it's going to take close to 6 hours so I'll report back in the morning. Any suggestions for next step(s) to take after the whole partition tree rebuild finishes?
June 14, 201016 yr Author Well, that's done: root@ihsserver04:~# reiserfsck --rebuild-tree --scan-whole- reiserfsck 3.6.21 (2009 www.namesys.com) *********************************************************** ** Do not run the program with --rebuild-tree unless ** something is broken and MAKE A BACKUP before using it. ** If you have bad sectors on a drive it is usually a bad ** idea to continue using it. Then you probably should get ** a working hard drive, copy the file system from the bad ** drive to the good one -- dd_rescue is a good tool for ** that -- and only then run this program. ** If you are using the latest reiserfsprogs and it fails ** please email bug reports to [email protected], ** providing as much information as possible -- your ** hardware, kernel, patches, settings, all reiserfsck ** messages (including version), the reiserfsck logfile, ** check the syslog file for any related information. ** If you would like advice on using this program, support ** is available for $25 at www.namesys.com/support.html. *********************************************************** Will rebuild the filesystem (/dev/md1) tree Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Replaying journal: Done. Reiserfs journal '/dev/md1' in blocks [18..8211]: 0 transactions replayed Zero bit found in on-disk bitmap after the last valid bit. Fixed. ########### reiserfsck --rebuild-tree started at Sun Jun 13 21:22:03 2010 ########### Pass 0: ####### Pass 0 ####### The whole partition (366284624 blocks) is to be scanned Skipping 19389 blocks (super block, journal, bitmaps) 366265235 blocks wi ll be read 0%block 10689002: The number of items ( is incorrect, should be (0) - c orrected block 10689002: The free space (7) is incorrect, should be (4072) - corre cted .block 21949326: The number of items (1) is incorrect, should be (0) - co rrected block 21949326: The free space ( is incorrect, should be (4072) - corre cted block 21950313: The number of items (1) is incorrect, should be (0) - cor rected block 21950313: The free space ( is incorrect, should be (4072) - corre cted .block 107354135: The number of items (1) is incorrect, should be (0) - corrected block 107354135: The free space ( is incorrect, should be (4072) - corrected block 107355122: The number of items (1) is incorrect, should be (0) - corrected block 107355122: The free space ( is incorrect, should be (4072) - corrected block 138667788: The number of items (2) is incorrect, should be (0) - corrected block 138667788: The free space (12) is incorrect, should be (4072) - corrected block 144050718: The number of items (2) is incorrect, should be (0) - corrected block 144050718: The free space (12) is incorrect, should be (4072) - corrected block 166769764: The number of items (3328) is incorrect, should be (1) - corrected block 166769764: The free space (768) is incorrect, should be (720) - corrected pass0: vpf-10110: block 166769764, item (0): Unknown item type found [3204449536 469762129 0x0 (15)] - deleted block 210881533: The free space (0) is incorrect, should be (4048) - corrected pass0: vpf-10110: block 210881533, item (0): Unknown item type found [1208877058 3820628 0x690072 (15)] - deleted block 216625319: The number of items (6679) is incorrect, should be (1) - corrected block 216625319: The free space (6696) is incorrect, should be (4048) - corrected pass0: vpf-10200: block 216625319, item 0: The item [439162081 64422870 0xa2f00011a1d1a2e IND (1)] with wrong offset is deleted block 217345312: The number of items (8392) is incorrect, should be (1) - corrected block 217345312: The free space (8406) is incorrect, should be (4048) - corrected pass0: vpf-10110: block 217345312, item (0): Unknown item type found [551231668 81928412 0x20ca20c2 (15)] - deleted block 347410342: The number of items (256) is incorrect, should be (1) - corrected block 347410342: The free space (256) is incorrect, should be (4048) - corrected pass0: vpf-10110: block 347410342, item (0): Unknown item type found [131075 1075052544 0x4f400000000006 (4)] - deleted left 0, 21024 /secc 140001 directory entries were hashed with "r5" hash. "r5" hash is selected Flushing..finished Read blocks (but not data blocks) 366265235 Leaves among those 101471 - leaves all contents of which could not be saved and deleted 18 Objectids found 136927 Pass 1 (will try to insert 101453 leaves): ####### Pass 1 ####### Looking for allocable blocks .. finished Killed root@ihsserver04:~#
June 14, 201016 yr Since we can re-group by copying back the original contents from the 2TB drive, let's try something a bit more unusual. First, I am puzzled by the "Killed" line at the end of the reiserfsck output. Can you type: tail -100 /var/log/syslog and see if you see any out-of-memory error messages and reiserfsck being killed?
June 14, 201016 yr What I'm going to suggest is if we do not have enough memory to fix the file-system we simply re-format the drive and THEN run the command to rebuild the file-tree scanning the entire drive. I'm going to try it here on a drive not in my array first to see how it goes. I've got an old 8Gig drive in my array I keep for just such tests. i'll create a file-system on it, fill it with files and directories, re-format it, and then run the --rebuild-tree scanning the entire disk. I can then see what it was able to recover. This should not take too long since it is a small disk. Joe L.
June 14, 201016 yr Author Since we can re-group by copying back the original contents from the 2TB drive, let's try something a bit more unusual. First, I am puzzled by the "Killed" line at the end of the reiserfsck output. Can you type: tail -100 /var/log/syslog and see if you see any out-of-memory error messages and reiserfsck being killed? Back from my morning run to clear my head! As you suspected; talks about lowmem_reserve then eventually Out of Memory: kill process... root@ihsserver04:~# tail -100 /var/log/syslog Jun 13 19:22:56 ihsserver04 ntpd[1247]: kernel time sync status 0040 Jun 13 19:22:56 ihsserver04 ntpd[1247]: frequency initialized -25.395 PPM from /etc/ntp/drift Jun 13 19:22:57 ihsserver04 ifplugd(eth0)[1086]: Program executed successfully. Jun 13 19:23:07 ihsserver04 ntpd[1247]: synchronized to 216.129.104.26, stratum 2 Jun 13 19:23:06 ihsserver04 ntpd[1247]: time reset -0.224826 s Jun 13 19:25:38 ihsserver04 in.telnetd[1250]: connect from 192.168.1.105 (192.168.1.105) Jun 13 19:25:44 ihsserver04 login[1251]: ROOT LOGIN on `pts/0' from `Shuttle' Jun 13 19:30:39 ihsserver04 ntpd[1247]: synchronized to 216.129.104.26, stratum 2 Jun 13 21:14:39 ihsserver04 kernel: reiserfsck invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0 Jun 13 21:14:39 ihsserver04 kernel: Pid: 1450, comm: reiserfsck Not tainted 2.6.32.9-unRAID #1 Jun 13 21:14:39 ihsserver04 kernel: Call Trace: Jun 13 21:14:39 ihsserver04 kernel: [<c104aae9>] oom_kill_process+0x59/0x1cd Jun 13 21:14:39 ihsserver04 kernel: [<c104af41>] __out_of_memory+0xef/0x102 Jun 13 21:14:39 ihsserver04 kernel: [<c104afb2>] out_of_memory+0x5e/0x83 Jun 13 21:14:39 ihsserver04 kernel: [<c104cf71>] __alloc_pages_nodemask+0x375/0x42f Jun 13 21:14:39 ihsserver04 kernel: [<c105960e>] handle_mm_fault+0x254/0x8f1 Jun 13 21:14:39 ihsserver04 kernel: [<c1028255>] ? __do_softirq+0xf0/0xf8 Jun 13 21:14:39 ihsserver04 kernel: [<c10284de>] ? irq_exit+0x29/0x2b Jun 13 21:14:39 ihsserver04 kernel: [<c101ab55>] ? smp_invalidate_interrupt+0x70/0x83 Jun 13 21:14:39 ihsserver04 kernel: [<c1003046>] ? invalidate_interrupt0+0x2a/0x30 Jun 13 21:14:39 ihsserver04 kernel: [<c101700c>] do_page_fault+0x17c/0x1e4 Jun 13 21:14:39 ihsserver04 kernel: [<c1016e90>] ? do_page_fault+0x0/0x1e4 Jun 13 21:14:39 ihsserver04 kernel: [<c129cd0e>] error_code+0x66/0x6c Jun 13 21:14:39 ihsserver04 kernel: [<c1016e90>] ? do_page_fault+0x0/0x1e4 Jun 13 21:14:39 ihsserver04 kernel: Mem-Info: Jun 13 21:14:39 ihsserver04 kernel: DMA per-cpu: Jun 13 21:14:39 ihsserver04 kernel: CPU 0: hi: 0, btch: 1 usd: 0 Jun 13 21:14:39 ihsserver04 kernel: CPU 1: hi: 0, btch: 1 usd: 0 Jun 13 21:14:39 ihsserver04 kernel: Normal per-cpu: Jun 13 21:14:39 ihsserver04 kernel: CPU 0: hi: 186, btch: 31 usd: 168 Jun 13 21:14:39 ihsserver04 kernel: CPU 1: hi: 186, btch: 31 usd: 200 Jun 13 21:14:39 ihsserver04 kernel: active_anon:59054 inactive_anon:25903 isolated_anon:0 Jun 13 21:14:39 ihsserver04 kernel: active_file:0 inactive_file:0 isolated_file:0 Jun 13 21:14:39 ihsserver04 kernel: unevictable:36428 dirty:0 writeback:0 unstable:0 Jun 13 21:14:39 ihsserver04 kernel: free:1195 slab_reclaimable:549 slab_unreclaimable:1630 Jun 13 21:14:39 ihsserver04 kernel: mapped:736 shmem:14 pagetables:254 bounce:0 Jun 13 21:14:39 ihsserver04 kernel: DMA free:2032kB min:88kB low:108kB high:132kB active_anon:13772kB inactive_anon:128kB active_file:0kB inactive_file:0kB unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:4kB slab_unreclaimable:0kB kernel_stack:0kB pagetables:8kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? yes Jun 13 21:14:39 ihsserver04 kernel: lowmem_reserve[]: 0 490 490 490 Jun 13 21:14:39 ihsserver04 kernel: Normal free:2748kB min:2784kB low:3480kB high:4176kB active_anon:222444kB inactive_anon:103484kB active_file:0kB inactive_file:0kB unevictable:145712kB isolated(anon):0kB isolated(file):0kB present:502448kB mlocked:0kB dirty:0kB writeback:0kB mapped:2944kB shmem:56kB slab_reclaimable:2192kB slab_unreclaimable:6520kB kernel_stack:512kB pagetables:1008kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:128 all_unreclaimable? yes Jun 13 21:14:39 ihsserver04 kernel: lowmem_reserve[]: 0 0 0 0 Jun 13 21:14:39 ihsserver04 kernel: DMA: 0*4kB 0*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2032kB Jun 13 21:14:39 ihsserver04 kernel: Normal: 7*4kB 0*8kB 0*16kB 1*32kB 0*64kB 1*128kB 0*256kB 1*512kB 0*1024kB 1*2048kB 0*4096kB = 2748kB Jun 13 21:14:39 ihsserver04 kernel: 36449 total pagecache pages Jun 13 21:14:39 ihsserver04 kernel: 0 pages in swap cache Jun 13 21:14:39 ihsserver04 kernel: Swap cache stats: add 0, delete 0, find 0/0 Jun 13 21:14:39 ihsserver04 kernel: Free swap = 0kB Jun 13 21:14:39 ihsserver04 kernel: Total swap = 0kB Jun 13 21:14:39 ihsserver04 kernel: 130698 pages RAM Jun 13 21:14:39 ihsserver04 kernel: 0 pages HighMem Jun 13 21:14:39 ihsserver04 kernel: 2363 pages reserved Jun 13 21:14:39 ihsserver04 kernel: 3004 pages shared Jun 13 21:14:39 ihsserver04 kernel: 125961 pages non-shared Jun 13 21:14:39 ihsserver04 kernel: Out of memory: kill process 1251 (bash) score 2867 or a child Jun 13 21:14:39 ihsserver04 kernel: Killed process 1450 (reiserfsck) Jun 14 02:14:09 ihsserver04 kernel: reiserfsck invoked oom-killer: gfp_mask=0x280da, order=0, oom_adj=0 Jun 14 02:14:09 ihsserver04 kernel: Pid: 1455, comm: reiserfsck Not tainted 2.6.32.9-unRAID #1 Jun 14 02:14:09 ihsserver04 kernel: Call Trace: Jun 14 02:14:09 ihsserver04 kernel: [<c104aae9>] oom_kill_process+0x59/0x1cd Jun 14 02:14:09 ihsserver04 kernel: [<c104af41>] __out_of_memory+0xef/0x102 Jun 14 02:14:09 ihsserver04 kernel: [<c104afb2>] out_of_memory+0x5e/0x83 Jun 14 02:14:09 ihsserver04 kernel: [<c104cf71>] __alloc_pages_nodemask+0x375/0x42f Jun 14 02:14:09 ihsserver04 kernel: [<c105960e>] handle_mm_fault+0x254/0x8f1 Jun 14 02:14:09 ihsserver04 kernel: [<c1028255>] ? __do_softirq+0xf0/0xf8 Jun 14 02:14:09 ihsserver04 kernel: [<c10284de>] ? irq_exit+0x29/0x2b Jun 14 02:14:09 ihsserver04 kernel: [<c10118ac>] ? smp_apic_timer_interrupt+0x6f/0x7d Jun 14 02:14:09 ihsserver04 kernel: [<c10031f6>] ? apic_timer_interrupt+0x2a/0x30 Jun 14 02:14:09 ihsserver04 kernel: [<c101700c>] do_page_fault+0x17c/0x1e4 Jun 14 02:14:09 ihsserver04 kernel: [<c1016e90>] ? do_page_fault+0x0/0x1e4 Jun 14 02:14:09 ihsserver04 kernel: [<c129cd0e>] error_code+0x66/0x6c Jun 14 02:14:09 ihsserver04 kernel: [<c1016e90>] ? do_page_fault+0x0/0x1e4 Jun 14 02:14:09 ihsserver04 kernel: Mem-Info: Jun 14 02:14:09 ihsserver04 kernel: DMA per-cpu: Jun 14 02:14:09 ihsserver04 kernel: CPU 0: hi: 0, btch: 1 usd: 0 Jun 14 02:14:09 ihsserver04 kernel: CPU 1: hi: 0, btch: 1 usd: 0 Jun 14 02:14:09 ihsserver04 kernel: Normal per-cpu: Jun 14 02:14:09 ihsserver04 kernel: CPU 0: hi: 186, btch: 31 usd: 182 Jun 14 02:14:09 ihsserver04 kernel: CPU 1: hi: 186, btch: 31 usd: 186 Jun 14 02:14:09 ihsserver04 kernel: active_anon:56323 inactive_anon:28607 isolated_anon:0 Jun 14 02:14:09 ihsserver04 kernel: active_file:0 inactive_file:4 isolated_file:0 Jun 14 02:14:09 ihsserver04 kernel: unevictable:36429 dirty:0 writeback:0 unstable:0 Jun 14 02:14:09 ihsserver04 kernel: free:1183 slab_reclaimable:577 slab_unreclaimable:1630 Jun 14 02:14:09 ihsserver04 kernel: mapped:736 shmem:14 pagetables:243 bounce:0 Jun 14 02:14:09 ihsserver04 kernel: DMA free:2048kB min:88kB low:108kB high:132kB active_anon:13584kB inactive_anon:256kB active_file:0kB inactive_file:16kB unevictable:4kB isolated(anon):0kB isolated(file):0kB present:15872kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB slab_reclaimable:36kB slab_unreclaimable:0kB kernel_stack:8kB pagetables:20kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jun 14 02:14:09 ihsserver04 kernel: lowmem_reserve[]: 0 490 490 490 Jun 14 02:14:09 ihsserver04 kernel: Normal free:2684kB min:2784kB low:3480kB high:4176kB active_anon:211708kB inactive_anon:114172kB active_file:0kB inactive_file:0kB unevictable:145712kB isolated(anon):0kB isolated(file):0kB present:502448kB mlocked:0kB dirty:0kB writeback:0kB mapped:2944kB shmem:56kB slab_reclaimable:2272kB slab_unreclaimable:6520kB kernel_stack:504kB pagetables:952kB unstable:0kB bounce:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no Jun 14 02:14:09 ihsserver04 kernel: lowmem_reserve[]: 0 0 0 0 Jun 14 02:14:09 ihsserver04 kernel: DMA: 1*4kB 2*8kB 1*16kB 1*32kB 1*64kB 1*128kB 1*256kB 1*512kB 1*1024kB 0*2048kB 0*4096kB = 2052kB Jun 14 02:14:09 ihsserver04 kernel: Normal: 671*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 2684kB Jun 14 02:14:09 ihsserver04 kernel: 36445 total pagecache pages Jun 14 02:14:09 ihsserver04 kernel: 0 pages in swap cache Jun 14 02:14:09 ihsserver04 kernel: Swap cache stats: add 0, delete 0, find 0/0 Jun 14 02:14:09 ihsserver04 kernel: Free swap = 0kB Jun 14 02:14:09 ihsserver04 kernel: Total swap = 0kB Jun 14 02:14:09 ihsserver04 kernel: 130698 pages RAM Jun 14 02:14:09 ihsserver04 kernel: 0 pages HighMem Jun 14 02:14:09 ihsserver04 kernel: 2363 pages reserved Jun 14 02:14:09 ihsserver04 kernel: 3013 pages shared Jun 14 02:14:09 ihsserver04 kernel: 125967 pages non-shared Jun 14 02:14:09 ihsserver04 kernel: Out of memory: kill process 1251 (bash) score 1433 or a child Jun 14 02:14:09 ihsserver04 kernel: Killed process 1455 (reiserfsck)
June 14, 201016 yr Author What I'm going to suggest is if we do not have enough memory to fix the file-system we simply re-format the drive and THEN run the command to rebuild the file-tree scanning the entire drive. I'm going to try it here on a drive not in my array first to see how it goes. I've got an old 8Gig drive in my array I keep for just such tests. i'll create a file-system on it, fill it with files and directories, re-format it, and then run the --rebuild-tree scanning the entire disk. I can then see what it was able to recover. This should not take too long since it is a small disk. Joe L. Ok, so if you wanted to try that; interesting thought. Being a memory issue though, I can try throwing more memory at it. Right now I have 2x 256MB sticks in there but if I find say 2x 512MB, I wonder if that would be enough? (That's likely all I will be able to find). Obviously that will mean shutting down the system. Maybe I'll capture the syslog before doing so for reference. Anything else I should do? Once I have the new mem in, should I start from the top with everything except the rebuild-sb?
Archived
This topic is now archived and is closed to further replies.