April 12, 201016 yr Hello I really hope somebody can help me. I am trying to --rebuild-tree one one of my disks using reiserfsck and it crashes out at around 20% eevry time, with a segmentation fault. My last parity check was about a day ago, so should be valid, if I need to rebuild. The stdout is as follows: root@Tower:~# reiserfsck --rebuild-tree /dev/md3 reiserfsck 3.6.19 (2003 www.namesys.com) ************************************************************* ** Do not run the program with --rebuild-tree unless ** ** something is broken and MAKE A BACKUP before using it. ** ** If you have bad sectors on a drive it is usually a bad ** ** idea to continue using it. Then you probably should get ** ** a working hard drive, copy the file system from the bad ** ** drive to the good one -- dd_rescue is a good tool for ** ** that -- and only then run this program. ** ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will rebuild the filesystem (/dev/md3) tree Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes Replaying journal.. Reiserfs journal '/dev/md3' in blocks [18..8211]: 0 transactions replayed ########### reiserfsck --rebuild-tree started at Mon Apr 12 19:19:16 2010 ########### Pass 0: ####### Pass 0 ####### Loading on-disk bitmap .. ok, 122274929 blocks marked used Skipping 15663 blocks (super block, journal, bitmaps) 122259266 blocks will be r ead 0%....20%... left 79301556, 1 left 69970736, 12793 /secc Message from syslogd@Tower at Mon Apr 12 20:27:23 2010 ... Tower kernel: ------------[ cut here ]------------ Message from syslogd@Tower at Mon Apr 12 20:27:23 2010 ... Tower kernel: invalid opcode: 0000 [#3] SMP Message from syslogd@Tower at Mon Apr 12 20:27:23 2010 ... Tower kernel: last sysfs file: /sys/devices/pci0000:00/0000:00:08.0/0000:01:07.0 /host3/target3:0:0/3:0:0:0/block/sdc/removable Message from syslogd@Tower at Mon Apr 12 20:27:23 2010 ... Tower kernel: Process reiserfsck (pid: 459, ti=d8f32000 task=f6b20dc0 task.ti=d8 f32000) Message from syslogd@Tower at Mon Apr 12 20:27:23 2010 ... Tower kernel: Stack: Message from syslogd@Tower at Mon Apr 12 20:27:23 2010 ... Tower kernel: Call Trace: Message from syslogd@Tower at Mon Apr 12 20:27:23 2010 ... Tower kernel: Code: 8d 44 07 30 89 45 dc 39 18 74 05 42 39 ca 7c ea 83 7d dc 00 75 04 0f 0b eb fe 8d 47 28 e8 73 3e 01 c9 8b 45 dc 83 78 0c 00 74 04 <0f> 0b eb fe 8b 55 dc 83 7a 10 00 74 04 0f 0b eb fe 83 7d c8 00 Message from syslogd@Tower at Mon Apr 12 20:27:23 2010 ... Tower kernel: EIP: [<f8288c74>] unraid_make_request+0x1d4/0x342 [md_mod] SS:ESP 0068:d8f33c4c Segmentation fault root@Tower:~#
April 12, 201016 yr Hello I really hope somebody can help me. I am trying to --rebuild-tree one one of my disks using reiserfsck and it crashes out at around 20% eevry time, with a segmentation fault. My last parity check was about a day ago, so should be valid, if I need to rebuild. The stdout is as follows: root@Tower:~# reiserfsck --rebuild-tree /dev/md3 reiserfsck 3.6.19 (2003 www.namesys.com) ************************************************************* ** Do not run the program with --rebuild-tree unless ** ** something is broken and MAKE A BACKUP before using it. ** ** If you have bad sectors on a drive it is usually a bad ** ** idea to continue using it. Then you probably should get ** ** a working hard drive, copy the file system from the bad ** ** drive to the good one -- dd_rescue is a good tool for ** ** that -- and only then run this program. ** ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will rebuild the filesystem (/dev/md3) tree Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes Replaying journal.. Reiserfs journal '/dev/md3' in blocks [18..8211]: 0 transactions replayed ########### reiserfsck --rebuild-tree started at Mon Apr 12 19:19:16 2010 ########### Pass 0: ####### Pass 0 ####### Loading on-disk bitmap .. ok, 122274929 blocks marked used Skipping 15663 blocks (super block, journal, bitmaps) 122259266 blocks will be r ead 0%....20%... left 79301556, 1 left 69970736, 12793 /secc Message from syslogd@Tower at Mon Apr 12 20:27:23 2010 ... Tower kernel: ------------[ cut here ]------------ Message from syslogd@Tower at Mon Apr 12 20:27:23 2010 ... Tower kernel: invalid opcode: 0000 [#3] SMP Message from syslogd@Tower at Mon Apr 12 20:27:23 2010 ... Tower kernel: last sysfs file: /sys/devices/pci0000:00/0000:00:08.0/0000:01:07.0 /host3/target3:0:0/3:0:0:0/block/sdc/removable Message from syslogd@Tower at Mon Apr 12 20:27:23 2010 ... Tower kernel: Process reiserfsck (pid: 459, ti=d8f32000 task=f6b20dc0 task.ti=d8 f32000) Message from syslogd@Tower at Mon Apr 12 20:27:23 2010 ... Tower kernel: Stack: Message from syslogd@Tower at Mon Apr 12 20:27:23 2010 ... Tower kernel: Call Trace: Message from syslogd@Tower at Mon Apr 12 20:27:23 2010 ... Tower kernel: Code: 8d 44 07 30 89 45 dc 39 18 74 05 42 39 ca 7c ea 83 7d dc 00 75 04 0f 0b eb fe 8d 47 28 e8 73 3e 01 c9 8b 45 dc 83 78 0c 00 74 04 <0f> 0b eb fe 8b 55 dc 83 7a 10 00 74 04 0f 0b eb fe 83 7d c8 00 Message from syslogd@Tower at Mon Apr 12 20:27:23 2010 ... Tower kernel: EIP: [<f8288c74>] unraid_make_request+0x1d4/0x342 [md_mod] SS:ESP 0068:d8f33c4c Segmentation fault root@Tower:~# Did the prior reiserfsck fix-fixable suggest the rebuild-tree? Have you run a fix-fixable at all? If not, it might be needed first. Have you run a memtest? Bad memory could be one possible cause of both the file-system corruption AND the segmentation fault. How full is the file-system? If the disk is corrupted, unless you wrote to the raw data drive, parity is equally corrupted. You do not have an option to rebuild from parity (or rather, odds are very high you 'll just have the exact same corruption of the file-tree once the rebuild is complete) Worst case, if memory is tested, you might need to run reiserfsck on the raw device. (if the error seems to be in the md_device code. We really can't tell for sure from what you posted, as I don't recognize it as a full stack trace) If you do that, you are guaranteed that parity will be out of sync. You must expect parity errors when you next calculate parity.
April 12, 201016 yr Author Did the prior reiserfsck fix-fixable suggest the rebuild-tree? Yes it did. Have you run a fix-fixable at all? If not, it might be needed first. No I havent - I will try this Have you run a memtest? Bad memory could be one possible cause of both the file-system corruption AND the segmentation fault. No - I can try this. I assume this is the one offered at the boot screen. Is it ok to reboot the server and pick up from where I left off, or will unraid try and strat the parity or soemthing that may make things worse? How full is the file-system? This disk is pretty much full up If the disk is corrupted, unless you wrote to the raw data drive, parity is equally corrupted. You do not have an option to rebuild from parity (or rather, odds are very high you 'll just have the exact same corruption of the file-tree once the rebuild is complete) Worst case, if memory is tested, you might need to run reiserfsck on the raw device. (if the error seems to be in the md_device code. We really can't tell for sure from what you posted, as I don't recognize it as a full stack trace) If you do that, you are guaranteed that parity will be out of sync. You must expect parity errors when you next calculate parity. Oh dear!
April 12, 201016 yr Author I just tried fix-fixable and it fails as follows: root@Tower:~# reiserfsck --fix-fixable /dev/md3 reiserfsck 3.6.19 (2003 www.namesys.com) ************************************************************* ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will check consistency of the filesystem on /dev/md3 and will fix what can be fixed without --rebuild-tree Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes ########### reiserfsck --fix-fixable started at Mon Apr 12 21:16:02 2010 ########### Replaying journal.. Reiserfs journal '/dev/md3' in blocks [18..8211]: 0 transactions replayed Checking internal tree.. Bad root block 0. (--rebuild-tree did not complete) Aborted root@Tower:~#
April 12, 201016 yr Author If the disk is corrupted, unless you wrote to the raw data drive, parity is equally corrupted. You do not have an option to rebuild from parity (or rather, odds are very high you 'll just have the exact same corruption of the file-tree once the rebuild is complete) Worst case, if memory is tested, you might need to run reiserfsck on the raw device. (if the error seems to be in the md_device code. We really can't tell for sure from what you posted, as I don't recognize it as a full stack trace) If you do that, you are guaranteed that parity will be out of sync. You must expect parity errors when you next calculate parity. Just reading your advice a few times more. If I could get back to the state I was in before I started these re-builds I would be a happy man. I am concenred I have lost all my data now. The last parity calculation may have some bad areas, but it was mounting fine in read-only, and would allow me to retrieve some of my files. I am unsure how to run on the raw device. Again I would be fine with some lost data and extremly happy to be running a parity check again, with the bulk of my data present! As you can see I am pretty nervous about this one. Many thanks for your advice so far.
April 12, 201016 yr If the disk is corrupted, unless you wrote to the raw data drive, parity is equally corrupted. You do not have an option to rebuild from parity (or rather, odds are very high you 'll just have the exact same corruption of the file-tree once the rebuild is complete) Worst case, if memory is tested, you might need to run reiserfsck on the raw device. (if the error seems to be in the md_device code. We really can't tell for sure from what you posted, as I don't recognize it as a full stack trace) If you do that, you are guaranteed that parity will be out of sync. You must expect parity errors when you next calculate parity. Just reading your advice a few times more. If I could get back to the state I was in before I started these re-builds I would be a happy man. I am concenred I have lost all my data now. The last parity calculation may have some bad areas, but it was mounting fine in read-only, and would allow me to retrieve some of my files. I am unsure how to run on the raw device. Again I would be fine with some lost data and extremly happy to be running a parity check again, with the bulk of my data present! As you can see I am pretty nervous about this one. Many thanks for your advice so far. The odds of you losing all your data is fairly low. Is a parity check running now? (Somehow I don't think so, but I figured I'd ask.) My advice, before you go further, Stop the array, reboot and run the memory test from the main menu for at least several passes. Let's make sure it is working before anything else. If you can run several passes, or overnight with NO memory errors, then we can proceed. Do you have any other processes or add-ons running? Post a syslog. If you want to see what you could get be a restore from parity, try this Stop the array make a copy of the config folder ... save it elsewhere (not on your array) You can name it old_config and keep it on the flash drive. This will let you get back to the exact same config easily. Then, go to the devices page and un-assign the disk with the corrupt file-system. Then go back to the main page and start the array. You will then see whatever can be rebuilt from the other drives. If the disk does not mount, you might try mounting it as read only to see if it shows your files as it did in the past. mkdir /mnt/disk3 mount -r -t reiserfs /dev/md3 /mnt/disk3 To go back to where you were, stop the array once more, copy the old config folder back (it will still have the disk assigned), and reboot. Joe L.
April 13, 201016 yr Author Thanks for the re-assurance, and your help, it is very much appreciated. The array is stopped and all drives unmounted at present so parity check is not running. I was concerned as my current parity was successful and very recent therefore I did not want it kicking off again now I have messed up the reiserfsck tree build. I had an issue that was caused by my music software locking whilst tagging some updates (after my last parity check, hence my hope that it is a useful fall back). This caused problems on my md4 and md3 drives. I ran --check and it suggested a --rebuild-tree for md4. this successfully rebuilt, but has a lot less data. The --check also suggested --rebuild-tree for md3 - its the md3 drive that is causing the problems, with --rebuild-tree failing. I am away with work now, but will be tackling this when i return tomorrow and will run memory test, and post a syslog I have all the usual add-ons, so will disable these in the go script before my reboot.
April 15, 201016 yr Author Ok so I am back up and running with minimal damage rebooted. Tried running memtest - this wouldnt even kick off, just seemed to hang on the start screen (problem for another day) Booted back into unraid. Sure enough my md3 disk appeared unformatted. So i tried --rebuild-tree again, this time it ran through fine and repaired the bad blocks. Retrieved some lost files in lost+found share. So reiserfsck wouldn't run without falling over with segmentation fault until I rebooted Thanks for all the help, panic over. I think I may set my cache drive up on my music share, so tagging goes there first rather than directly to the parity, as this caused the whole problem in the first place.
April 15, 201016 yr Tried running memtest - this wouldnt even kick off, just seemed to hang on the start screen (problem for another day) Wait, why did that happen? You either have a bad flash drive or bad ram. Aren't you the least bit worried that this is a problem?
April 15, 201016 yr Tried running memtest - this wouldnt even kick off, just seemed to hang on the start screen (problem for another day) Yes, you should solve this issue first, as this is probably the root of your problem..
April 15, 201016 yr Tried running memtest - this wouldnt even kick off, just seemed to hang on the start screen (problem for another day) Yes, you should solve this issue first, as this is probably the root of your problem.. Bad memory will cause file-system corruption AND data corruption. You cannot ignore the fact that the memory test would not run. As already stated, it is likely the root cause of the file-system damage.
April 16, 201016 yr Author It was remiss of me to ignore that. The problem was that memtest was not in fact launching at all. I have an ASUS motherboard, as such after a little more investigation, I needed to disable USB Mouse support (have kept this off for the time being) in the BIOS (but kept USB Keyboard & Legacy Support enabled - otherwise I cannot boot from the flash) Anyway the memtest has run through several times with no errors and 100% pass, so I guess that all is well with the memory.
April 16, 201016 yr A long while back I had to run reiserfsck with rebuild tree and it failed (don't remember if it was a seg fault or not). Anyway - problem turned out to be a bad sector on the disk. I had to get that sector to remap before I was able to get reiserfsck to complete.
Archived
This topic is now archived and is closed to further replies.