dwoods99 Posted January 21, 2017 Share Posted January 21, 2017 New problem on server running 5.0rc8. Searching didn't provide new ideas to try. I had a failed drive on disk 7. Replaced it with a larger drive, rebuilt from parity. It all seemed fine except it later came up in read only mode after a reboot. I ran reiserfsck --check /dev/md7 and it reported corruption and suggested using --rebuild-tree After doing that, it finished ok but another --check reported more errors and in read only mode. I even attempted a umount /dev/md7 followed by mount /dev/mnt7 /mnt/disk7 to force it. Now I can't seem to be able to access the contents of the sub-folder Movies/1960-1979/ I can cd into it but trying any ls command just hangs there (even a simple ls A*) Therefore I am not able to copy those movies onto other disks for backup before more aggressive rebuild attempts. What else can I try besides a --rebuild-sb as a last resort? root@Moat:/boot/config# reiserfsck --check /dev/md7 reiserfsck 3.6.21 (2009 www.namesys.com) ... Will read-only check consistency of the filesystem on /dev/md7 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes ########### reiserfsck --check started at Thu Jan 19 23:00:07 2017 ########### Filesystem seems mounted read-only. Skipping journal replay. Checking internal tree.. \/ 1 (of 18|/ 1 (of 143// 2 (of 168\bad_path: block 155484162, pointer 1: The used space (4048) of the child block (256545669) is not equal to the (blocksize (4096) - free space (96) - header size (24)) / 24 (of 168/bad_path: The left delimiting key [2 5 0x46109280 DIR (3)] of the node (32810) must be equal to the first element's key [2 167 0x21c001 IND (1)] within the node. / 25 (of 168-bad_path: The left delimiting key [2 5 0x477cab80 DIR (3)] of the node (131088563) must be equal to the first element's key [2 5 0x46109280 DIR (3)] within the node. / 55 (of 168|bad_path: block 155484162, pointer 54: The used space (4068) of the child block (32776) is not equal to the (blocksize (4096) - free space (0) - header size (24)) /106 (of 168\bad_path: block 155484162, pointer 105: The used space (2712) of the child block (32846) is not equal to the (blocksize (4096) - free space (1220) - header size (24)) / 7 (of 18\/118 (of 170\block 149955: The level of the node (3) is not correct, (2) expected the problem in the internal node occured (149955), whole subtree is skipped finished Comparing bitmaps..vpf-10640: The on-disk and the correct bitmaps differs. Bad nodes were found, Semantic pass skipped 3 found corruptions can be fixed only when running with --rebuild-tree ########### reiserfsck finished at Thu Jan 19 23:05:02 2017 ########### root@Moat:/boot/config# reiserfsck --rebuild-tree /dev/md7 reiserfsck 3.6.21 (2009 www.namesys.com) ... Will rebuild the filesystem (/dev/md7) tree Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes Filesystem seems mounted read-only. Skipping journal replay. ########### reiserfsck --rebuild-tree started at Fri Jan 20 01:28:15 2017 ########### Pass 0: ####### Pass 0 ####### Loading on-disk bitmap .. ok, 485191348 blocks marked used Skipping 30567 blocks (super block, journal, bitmaps) 485160781 blocks will be read 0%.. left 0, 32372 /sec 1875 directory entries were hashed with "r5" hash. "r5" hash is selected Flushing..finished Read blocks (but not data blocks) 485160781 Leaves among those 480167 Objectids found 1878 Pass 1 (will try to insert 480167 leaves): ####### Pass 1 ####### Looking for allocable blocks .. finished 0%....20%....40%....60%....80%....100% left 0, 2652 /sec Flushing..finished 480167 leaves read 480166 inserted 1 not inserted ####### Pass 2 ####### Pass 2: 0%....20%....40%....60%....80%....100% left 0, 1 /sec Flushing..finished Leaves inserted item by item 1 Pass 3 (semantic): ####### Pass 3 ######### /Movies/1960-1979vpf-10650: The directory [2 5] has the wrong size in the StatData (139016) - corrected to (138944) Flushing..finished Files found: 1870 Directories found: 7 Pass 3a (looking for lost dir/files): ####### Pass 3a (lost+found pass) ######### Looking for lost directories: Flushing..finished5, 747 /sec Empty lost dirs removed 1 Pass 4 - finisheddone 463088, 506 /sec Deleted unreachable items 2 Flushing..finished The partition is mounted ro. It is better to umount and mount it again. Syncing..finished ########### reiserfsck finished at Fri Jan 20 06:06:09 2017 ########### root@Moat:/# umount /dev/md7 root@Moat:/# mount /dev/md7 /mnt/disk7 root@Moat:~# reiserfsck --check /dev/md7 reiserfsck 3.6.21 (2009 www.namesys.com) ... Will read-only check consistency of the filesystem on /dev/md7 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes ########### reiserfsck --check started at Fri Jan 20 15:22:01 2017 ########### Filesystem seems mounted read-only. Skipping journal replay. Checking internal tree.. \/ 1 (of 18|/ 1 (of 143// 2 (of 168\bad_path: block 155484162, pointer 1: The used space (4048) of the child block (256545669) is not equal to the (blocksize (4096) - free space (96) - header size (24)) / 24 (of 168/bad_path: The left delimiting key [2 5 0x46109280 DIR (3)] of the node (32810) must be equal to the first element's key [2 167 0x21c001 IND (1)] within the node. / 25 (of 168-bad_path: The left delimiting key [2 5 0x477cab80 DIR (3)] of the node (131088563) must be equal to the first element's key [2 5 0x46109280 DIR (3)] within the node. / 55 (of 168|bad_path: block 155484162, pointer 54: The used space (4068) of the child block (32776) is not equal to the (blocksize (4096) - free space (0) - header size (24)) /106 (of 168\bad_stat_data: The objectid (1437) is marked free, but used by an object [2 1437 0x0 SD (0)] bad_path: block 155484162, pointer 105: The used space (2712) of the child block (32846) is not equal to the (blocksize (4096) - free space (1220) - header size (24)) / 7 (of 18\/118 (of 170\block 149955: The level of the node (3) is not correct, (2) expected the problem in the internal node occured (149955), whole subtree is skipped finished Comparing bitmaps..vpf-10640: The on-disk and the correct bitmaps differs. Bad nodes were found, Semantic pass skipped 3 found corruptions can be fixed only when running with --rebuild-tree ########### reiserfsck finished at Fri Jan 20 15:26:24 2017 ########### root@Moat:~# root@Moat:~# tail -333 /var/log/syslog Jan 20 04:40:02 Moat syslogd 1.4.1: restart. Jan 20 07:06:13 Moat kernel: mdcmd (169): spindown 0 Jan 20 07:06:13 Moat kernel: mdcmd (170): spindown 7 Jan 20 15:19:34 Moat kernel: REISERFS (device md7): found reiserfs format "3.6" with standard journal Jan 20 15:19:34 Moat kernel: REISERFS (device md7): using ordered data mode Jan 20 15:19:34 Moat kernel: reiserfs: using flush barriers Jan 20 15:19:34 Moat kernel: REISERFS (device md7): journal params: device md7, size 8192, journal first block 18, max trans len 1024, max batch 900, max commit age 30, max trans age 30 Jan 20 15:19:34 Moat kernel: REISERFS (device md7): checking transaction log (md7) Jan 20 15:19:34 Moat kernel: REISERFS (device md7): replayed 3 transactions in 0 seconds Jan 20 15:19:34 Moat kernel: REISERFS (device md7): Using r5 hash to sort names Jan 20 15:19:54 Moat kernel: REISERFS error (device md7): vs-7000 search_by_entry_key: search_by_key returned item position == 0 Jan 20 15:19:54 Moat kernel: REISERFS (device md7): Remounting filesystem read-only Jan 20 15:19:54 Moat kernel: REISERFS error (device md7): vs-7000 search_by_entry_key: search_by_key returned item position == 0 Jan 20 15:20:25 Moat last message repeated 192379 times Jan 20 15:21:24 Moat last message repeated 378685 times root@Moat:~# root@Moat:~# reiserfsck --check /dev/md7 reiserfsck 3.6.21 (2009 www.namesys.com) ... Will read-only check consistency of the filesystem on /dev/md7 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes ########### reiserfsck --check started at Fri Jan 20 15:22:01 2017 ########### Filesystem seems mounted read-only. Skipping journal replay. Checking internal tree.. \/ 1 (of 18|/ 1 (of 143// 2 (of 168\bad_path: block 155484162, pointer 1: The used space (4048) of the child block (256545669) is not equal to the (blocksize (4096) - free space (96) - header size (24)) / 24 (of 168/bad_path: The left delimiting key [2 5 0x46109280 DIR (3)] of the node (32810) must be equal to the first element's key [2 167 0x21c001 IND (1)] within the node. / 25 (of 168-bad_path: The left delimiting key [2 5 0x477cab80 DIR (3)] of the node (131088563) must be equal to the first element's key [2 5 0x46109280 DIR (3)] within the node. / 55 (of 168|bad_path: block 155484162, pointer 54: The used space (4068) of the child block (32776) is not equal to the (blocksize (4096) - free space (0) - header size (24)) /106 (of 168\bad_stat_data: The objectid (1437) is marked free, but used by an object [2 1437 0x0 SD (0)] bad_path: block 155484162, pointer 105: The used space (2712) of the child block (32846) is not equal to the (blocksize (4096) - free space (1220) - header size (24)) / 7 (of 18\/118 (of 170\block 149955: The level of the node (3) is not correct, (2) expected the problem in the internal node occured (149955), whole subtree is skipped finished Comparing bitmaps..vpf-10640: The on-disk and the correct bitmaps differs. Bad nodes were found, Semantic pass skipped 3 found corruptions can be fixed only when running with --rebuild-tree ########### reiserfsck finished at Fri Jan 20 15:26:24 2017 ########### root@Moat:~# 1 Quote Link to comment
JorgeB Posted January 21, 2017 Share Posted January 21, 2017 Rebuilding superblock won't help, run with rebuild tree again, may be worth running memtest before just to make sure there are no RAM issues. Quote Link to comment
dwoods99 Posted January 22, 2017 Author Share Posted January 22, 2017 I updated the file memtest on the flash drive by using To update: 1. download the "Pre-Compiled Bootable Binary" (either .zip or .gz) from here: http://www.memtest.org/#downiso 2. extract / rename / overwrite the file 'memtest' that is in the root directory of the UnRAID USB stick. Version 5.01 is the same as the one on v6.2.4 install package. Ran the test for 6 hours and 0 errors after 4 cycles/passes. System came back up and tried a parity check, which was very slow. I stopped the process by stopping the array. However this means nothing mounted and so I could not use reiserfsck. I started array but now reiserfsck won't work since they are all mounted with write permissions. Do I have to stop the array and mount the drive manually as read only? What's the best way if I want to do a --check on each of the 11 drives? Quote Link to comment
JorgeB Posted January 22, 2017 Share Posted January 22, 2017 You need to start the array in maintenance mode: https://lime-technology.com/wiki/index.php/Check_Disk_Filesystems#Drives_formatted_with_ReiserFS_using_unRAID_v5_or_later Quote Link to comment
trurl Posted January 22, 2017 Share Posted January 22, 2017 System came back up and tried a parity check, which was very slow.Parity check cannot be affected by filesystem corruption so maybe you have additional problems. Quote Link to comment
JorgeB Posted January 22, 2017 Share Posted January 22, 2017 System came back up and tried a parity check, which was very slow.Parity check cannot be affected by filesystem corruption so maybe you have additional problems. Good point, I missed that line, you should check your syslog and SMART reports or post them if you don't know how. Quote Link to comment
dwoods99 Posted January 23, 2017 Author Share Posted January 23, 2017 I spent the whole day yesterday running reiserfsck --check on each of the drives while in maintenance mode. All reported no corruption except for disk7 as expected. Ran reiserfsck --rebuild-tree and then reiserfsck --check again. Now it reports fine. I stopped the array and restarted in non-maintenance mode. Drive is no longer in read only mode, and I can finally access and delete files, as well as get a listing of the movie folder (which failed due to corruptions). Lesson learned... use maintenance mode when using reiserfsck Thanks for the help, it's appreciated. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.