zack84a Posted December 31, 2012 Share Posted December 31, 2012 Was running into problems with the mover script, and a drive appeared to be Read Only. So through research looks like a reiserfsck --check was needed. I did so and it suggested rebuild tree. Ran that. Then checked again. Suggested rebuild tree still. Ran one more time. Same error. Will read-only check consistency of the filesystem on /dev/md1 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes ########### reiserfsck --check started at Sun Dec 30 23:54:59 2012 ########### Replaying journal: Done. Reiserfs journal '/dev/md1' in blocks [18..8211]: 0 transactions replayed Checking internal tree.. \block 126746674: The level of the node (2) is not correct, (4) expected the problem in the internal node occured (126746674), whole subtree is skipped finished Comparing bitmaps..vpf-10640: The on-disk and the correct bitmaps differs. Bad nodes were found, Semantic pass skipped 1 found corruptions can be fixed only when running with --rebuild-tree ########### reiserfsck finished at Sun Dec 30 23:57:04 2012 ########### root@Tower:~# Now if I start the array, it shows the drive as unformatted. So looking what I should try next Attached is my syslog as well. syslog-2012-12-31.zip Quote Link to comment
zack84a Posted December 31, 2012 Author Share Posted December 31, 2012 If its any help, short SMART test didn't show much, running long right now and will post when it when it finishes. Quote Link to comment
dgaschk Posted January 1, 2013 Share Posted January 1, 2013 Show the output with --rebuild-tree Quote Link to comment
Joe L. Posted January 1, 2013 Share Posted January 1, 2013 unformatted simply means "not able to be mounted as a reiserfs file system" Once you fix the corruption, it should mount. DO NOT FORMAT THE DRIVE UNLESS YOU WANT TO ERASE ALL THE CONTENTS!!!!!!! Just run reiserfsck --rebuild-tree /dev/md1 If you are running a "long" test, make sure you disable spin-down, otherwise the test will abort when the disk spins down and never complete. Quote Link to comment
zack84a Posted January 1, 2013 Author Share Posted January 1, 2013 Here is my smart report. Looks like I have a crap ton of relocated sectors, which I would assume this drive might be on its way out. Might be why 2 --rebuild-tree runs aren't actually fixing the drive. I have a drive that I just added to the array that is the same size and doesn't have much on it. What steps should I take (assuming I remove whats on the drive) to use it to replace the failing drive. smart.txt Quote Link to comment
zack84a Posted January 2, 2013 Author Share Posted January 2, 2013 Show the output with --rebuild-tree Output is attached. Note this is the third time to run --rebuild-tree. I didn't export the command to a text file, and was unable to get all of the output through telnet :\ However I think from below you can see the significant part or where --rebuild-tree is doing something for better or worse. Should be noted, that my drive is still showing as unformatted rebuild-tree.txt Quote Link to comment
Joe L. Posted January 2, 2013 Share Posted January 2, 2013 It appears as if the --rebuild-tree completed just fine. Why did you run it three times? Did it not complete the previous attempts? You should not be able to run reiserfsck --check on the disk and have it show no errors. Once you get to that point, just reboot the server, it should mount. Quote Link to comment
Joe L. Posted January 2, 2013 Share Posted January 2, 2013 Should be noted, that my drive is still showing as unformatted to unRAID, any disk it cannot mount, for any reason, is labeled as un-formatted. That label will not go away until the disk is mounted. (and that can most easily be done by rebooting and have unRAID attempt to mount the disk again.) Quote Link to comment
zack84a Posted January 2, 2013 Author Share Posted January 2, 2013 It appears as if the --rebuild-tree completed just fine. Why did you run it three times? Did it not complete the previous attempts? You should not be able to run reiserfsck --check on the disk and have it show no errors. Once you get to that point, just reboot the server, it should mount. Did it once, ran --check to see if it was good. Wasn't good, so figured it maybe needed another pass to correct what was left. Didn't work. Then dgaschk wanted to know what it reported back, so ran it a third time. Check previous post of mine as to the error I get when I run check.. Still (even after 3 times) tells me to rebuild tree. I haven't rebooted it after this last --rebuild-tree so I will do so now. I have stop and restarted the array and it did not mount the drive. Thanks for your help. UPDATE: Still didn't mount (unformatted) after reboot. Quote Link to comment
Joe L. Posted January 2, 2013 Share Posted January 2, 2013 It appears as if the --rebuild-tree completed just fine. Why did you run it three times? Did it not complete the previous attempts? You should not be able to run reiserfsck --check on the disk and have it show no errors. Once you get to that point, just reboot the server, it should mount. Did it once, ran --check to see if it was good. Wasn't good, so figured it maybe needed another pass to correct what was left. Didn't work. Then dgaschk wanted to know what it reported back, so ran it a third time. Check previous post of mine as to the error I get when I run check.. Still (even after 3 times) tells me to rebuild tree. I haven't rebooted it after this last --rebuild-tree so I will do so now. I have stop and restarted the array and it did not mount the drive. Thanks for your help. UPDATE: Still didn't mount (unformatted) after reboot. I'd perform a memory test on your server. If the --rebuild-tree keeps saying it was successful, and if the SMART report on the disk shows it is not experiencing un-readable sectors, then possibly something else is involved (memory, mb disk controller, etc) Easiest to rule out is memory. Reboot server, choose the memory test, let it run for several cycles. Joe L. Quote Link to comment
lionelhutz Posted January 2, 2013 Share Posted January 2, 2013 Ouch, that's too bad you didn't see the bad sector issues before you went any further. You had a chance to simply replace the drive but you buggered the parity by messing with the failed drive. At this point, I would get another drive to replace this failed drive and rebuild onto it. It will likely still be unformatted but reiserfsck might be able to fix a healthy drive. Quote Link to comment
zack84a Posted January 3, 2013 Author Share Posted January 3, 2013 I'd perform a memory test on your server. If the --rebuild-tree keeps saying it was successful, and if the SMART report on the disk shows it is not experiencing un-readable sectors, then possibly something else is involved (memory, mb disk controller, etc) Easiest to rule out is memory. Reboot server, choose the memory test, let it run for several cycles. Joe L. Ran memory test while I was out last night, 7 Passes clear with no errors :\ I have a replacement drive that should arrive today. Just need to get it cleared. Joe do you recommend as lionelhutz pointed out to just rebuild it and try to reiserfsck it again? Definitely willing to try anything else though. Quote Link to comment
Joe L. Posted January 3, 2013 Share Posted January 3, 2013 I'd perform a memory test on your server. If the --rebuild-tree keeps saying it was successful, and if the SMART report on the disk shows it is not experiencing un-readable sectors, then possibly something else is involved (memory, mb disk controller, etc) Easiest to rule out is memory. Reboot server, choose the memory test, let it run for several cycles. Joe L. Ran memory test while I was out last night, 7 Passes clear with no errors :\ I have a replacement drive that should arrive today. Just need to get it cleared. Joe do you recommend as lionelhutz pointed out to just rebuild it and try to reiserfsck it again? Definitely willing to try anything else though. Not likely to be memory then. Just to be clear, if you run a --rebuild-tree, followed by a --check, the --check recommends another --rebuild tree? Since the only disk involved in the --rebuild-tree is the one being rebuilt, you can give a try of re re-constructing onto a replacement disk... Be aware however that result will depend upon all the other disks and hardware in the server. If you trust them, then go for it. Joe L. Quote Link to comment
zack84a Posted January 3, 2013 Author Share Posted January 3, 2013 I'd perform a memory test on your server. If the --rebuild-tree keeps saying it was successful, and if the SMART report on the disk shows it is not experiencing un-readable sectors, then possibly something else is involved (memory, mb disk controller, etc) Easiest to rule out is memory. Reboot server, choose the memory test, let it run for several cycles. Joe L. Ran memory test while I was out last night, 7 Passes clear with no errors :\ I have a replacement drive that should arrive today. Just need to get it cleared. Joe do you recommend as lionelhutz pointed out to just rebuild it and try to reiserfsck it again? Definitely willing to try anything else though. Not likely to be memory then. Just to be clear, if you run a --rebuild-tree, followed by a --check, the --check recommends another --rebuild tree? Since the only disk involved in the --rebuild-tree is the one being rebuilt, you can give a try of re re-constructing onto a replacement disk... Be aware however that result will depend upon all the other disks and hardware in the server. If you trust them, then go for it. Joe L. Correct. Each --rebuild-tree has been followed with a check which asks for another --rebuild-tree.. Almost in the exact fashion. I'll run checks on the rest of the drives just to be safe before proceeding.. It will be a a day or so since I will need to preclear, so if anyone does come up with an idea to try let me know. Else I'll report back if I am able to reconstruct the disk... Joe do I just need to bring array down, and replace disk in config with new disk and bring the array back up? (never actually replaced a disk before, but have read about quite a bit) Quote Link to comment
zack84a Posted January 8, 2013 Author Share Posted January 8, 2013 Update... Replaced the drive, and ran the --rebuild-tree again and it appeared to fix it. Currently running a parity check, but the drive came back as formatted so I am hoping we are good. Thanks all for the help! Hopefully this will one day help someone else. Quote Link to comment
dgaschk Posted January 8, 2013 Share Posted January 8, 2013 Update... Replaced the drive, and ran the --rebuild-tree again and it appeared to fix it. Currently running a parity check, but the drive came back as formatted so I am hoping we are good. Thanks all for the help! Hopefully this will one day help someone else. Run with --check again to confirm success. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.