bfeist Posted February 24, 2011 Share Posted February 24, 2011 Hi all, I've been running unRAID for almost two years with no issues until this week. I ran a parity check earlier in the week and it reported that it found "384 errors". I did some reading and learned that these kinds of errors can mark the beginning of read errors of a drive that was about to fail. Yesterday my disk5 red balled, it was an older 1tb seagate. I replaced it with a 1.5TB seagate that I had as spare. The rebuild took about 8 hours. At the end of the rebuild it reported that it finished with "384 errors". I checked the syslog and the errors were on the new disk5. I've attached a portion of the syslog with some of the errors below. Can someone tell me how I should proceed in order to repair these errors? Feb 24 07:50:07 Tower ntpd[1601]: synchronized to 173.203.122.111, stratum 3 Feb 24 08:05:50 Tower ntpd[1601]: time reset +1.833650 s Feb 24 08:06:50 Tower ntpd[1601]: synchronized to 173.203.122.111, stratum 3 Feb 24 08:08:04 Tower kernel: REISERFS warning: reiserfs-5081 is_leaf: nr_item seems wrong: level=1, nr_items=47621, free_space=9209 rdkey Feb 24 08:08:04 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 181829665. Fsck? Feb 24 08:08:04 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2688 150 0x0 SD] Feb 24 08:08:04 Tower kernel: REISERFS warning: reiserfs-5081 is_leaf: nr_item seems wrong: level=1, nr_items=47621, free_space=9209 rdkey Feb 24 08:08:04 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 181829665. Fsck? Feb 24 08:08:04 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2688 150 0x0 SD] Feb 24 08:08:11 Tower kernel: REISERFS warning: reiserfs-5081 is_leaf: nr_item seems wrong: level=1, nr_items=47621, free_space=9209 rdkey Feb 24 08:08:11 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 181829665. Fsck? Feb 24 08:08:11 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2688 150 0x0 SD] Feb 24 08:08:11 Tower kernel: REISERFS warning: reiserfs-5081 is_leaf: nr_item seems wrong: level=1, nr_items=47621, free_space=9209 rdkey Feb 24 08:08:11 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 181829665. Fsck? Feb 24 08:08:11 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2688 150 0x0 SD] Feb 24 08:08:11 Tower kernel: REISERFS warning: reiserfs-5088 is_internal: number of key seems wrong: level=2, nr_items=47784, free_space=49997 rdkey Feb 24 08:08:11 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 169592216. Fsck? Feb 24 08:08:11 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [18 2716 0x0 SD] Feb 24 08:08:11 Tower kernel: REISERFS warning: reiserfs-5088 is_internal: number of key seems wrong: level=2, nr_items=47784, free_space=49997 rdkey Feb 24 08:08:11 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 169592216. Fsck? Feb 24 08:08:11 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [18 2716 0x0 SD] Feb 24 08:08:11 Tower kernel: REISERFS warning: reiserfs-5088 is_internal: number of key seems wrong: level=2, nr_items=47784, free_space=35148 rdkey Feb 24 08:08:11 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 166430395. Fsck? Feb 24 08:08:11 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [18 2714 0x0 SD] Feb 24 08:08:11 Tower kernel: REISERFS warning: reiserfs-5088 is_internal: number of key seems wrong: level=2, nr_items=47784, free_space=35148 rdkey Feb 24 08:08:11 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 166430395. Fsck? Feb 24 08:08:11 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [18 2714 0x0 SD] Feb 24 08:08:12 Tower kernel: REISERFS warning: reiserfs-5081 is_leaf: nr_item seems wrong: level=1, nr_items=47618, free_space=324 rdkey Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 18700689. Fsck? Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2567 2592 0x0 SD] Feb 24 08:08:12 Tower kernel: REISERFS warning: reiserfs-5081 is_leaf: nr_item seems wrong: level=1, nr_items=47618, free_space=324 rdkey Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 18700689. Fsck? Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2567 2592 0x0 SD] Feb 24 08:08:12 Tower kernel: REISERFS warning: reiserfs-5088 is_internal: number of key seems wrong: level=2, nr_items=47784, free_space=6220 rdkey Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 18794884. Fsck? Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2567 2597 0x0 SD] Feb 24 08:08:12 Tower kernel: REISERFS warning: reiserfs-5088 is_internal: number of key seems wrong: level=2, nr_items=47784, free_space=6220 rdkey Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 18794884. Fsck? Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2567 2597 0x0 SD] Feb 24 08:08:12 Tower kernel: REISERFS warning: reiserfs-5088 is_internal: number of key seems wrong: level=2, nr_items=47784, free_space=46156 rdkey Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 19311503. Fsck? Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2567 2605 0x0 SD] Feb 24 08:08:12 Tower kernel: REISERFS warning: reiserfs-5088 is_internal: number of key seems wrong: level=2, nr_items=47784, free_space=46156 rdkey Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 19311503. Fsck? Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2567 2605 0x0 SD] Feb 24 08:08:12 Tower kernel: REISERFS warning: reiserfs-5081 is_leaf: nr_item seems wrong: level=1, nr_items=47619, free_space=14404 rdkey Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 181272595. Fsck? Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2450 2452 0x0 SD] Feb 24 08:08:12 Tower kernel: REISERFS warning: reiserfs-5081 is_leaf: nr_item seems wrong: level=1, nr_items=47619, free_space=14404 rdkey Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 181272595. Fsck? Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2450 2452 0x0 SD] Feb 24 08:08:12 Tower kernel: REISERFS warning: reiserfs-5081 is_leaf: nr_item seems wrong: level=1, nr_items=47618, free_space=19524 rdkey Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 181317016. Fsck? Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2450 2453 0x0 SD] Feb 24 08:08:12 Tower kernel: REISERFS warning: reiserfs-5081 is_leaf: nr_item seems wrong: level=1, nr_items=47618, free_space=19524 rdkey Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 181317016. Fsck? Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2450 2453 0x0 SD] Feb 24 08:08:12 Tower kernel: REISERFS warning: reiserfs-5081 is_leaf: nr_item seems wrong: level=1, nr_items=47618, free_space=23364 rdkey Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 181338269. Fsck? Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2450 2454 0x0 SD] Feb 24 08:08:12 Tower kernel: REISERFS warning: reiserfs-5081 is_leaf: nr_item seems wrong: level=1, nr_items=47618, free_space=23364 rdkey Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 181338269. Fsck? Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2450 2454 0x0 SD] Feb 24 08:08:12 Tower kernel: REISERFS warning: reiserfs-5081 is_leaf: nr_item seems wrong: level=1, nr_items=47618, free_space=24132 rdkey Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 181342298. Fsck? Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2450 2455 0x0 SD] Feb 24 08:08:12 Tower kernel: REISERFS warning: reiserfs-5081 is_leaf: nr_item seems wrong: level=1, nr_items=47618, free_space=24132 rdkey Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 181342298. Fsck? Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2450 2455 0x0 SD] Feb 24 08:08:12 Tower kernel: REISERFS warning: reiserfs-5088 is_internal: number of key seems wrong: level=2, nr_items=47784, free_space=24652 rdkey Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 181345320. Fsck? Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2450 2456 0x0 SD] Feb 24 08:08:12 Tower kernel: REISERFS warning: reiserfs-5088 is_internal: number of key seems wrong: level=2, nr_items=47784, free_space=24652 rdkey Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 181345320. Fsck? Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2450 2456 0x0 SD] Feb 24 08:08:12 Tower kernel: REISERFS warning: reiserfs-5088 is_internal: number of key seems wrong: level=2, nr_items=47784, free_space=24652 rdkey Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 181345320. Fsck? Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2450 2457 0x0 SD] Feb 24 08:08:12 Tower kernel: REISERFS warning: reiserfs-5088 is_internal: number of key seems wrong: level=2, nr_items=47784, free_space=24652 rdkey Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 181345320. Fsck? Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2450 2457 0x0 SD] Feb 24 08:08:12 Tower kernel: REISERFS warning: reiserfs-5088 is_internal: number of key seems wrong: level=2, nr_items=47784, free_space=24652 rdkey Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 181345320. Fsck? Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2450 2458 0x0 SD] Feb 24 08:08:12 Tower kernel: REISERFS warning: reiserfs-5088 is_internal: number of key seems wrong: level=2, nr_items=47784, free_space=24652 rdkey Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 181345320. Fsck? Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2450 2458 0x0 SD] Feb 24 08:08:12 Tower kernel: REISERFS warning: reiserfs-5088 is_internal: number of key seems wrong: level=2, nr_items=47784, free_space=24652 rdkey Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 181345320. Fsck? Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2450 2459 0x0 SD] Feb 24 08:08:12 Tower kernel: REISERFS warning: reiserfs-5088 is_internal: number of key seems wrong: level=2, nr_items=47784, free_space=24652 rdkey Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 181345320. Fsck? Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2450 2459 0x0 SD] Feb 24 08:08:12 Tower kernel: REISERFS warning: reiserfs-5088 is_internal: number of key seems wrong: level=2, nr_items=47784, free_space=24652 rdkey Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 181345320. Fsck? Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2450 2460 0x0 SD] Feb 24 08:08:12 Tower kernel: REISERFS warning: reiserfs-5088 is_internal: number of key seems wrong: level=2, nr_items=47784, free_space=24652 rdkey Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 181345320. Fsck? Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2450 2460 0x0 SD] Feb 24 08:08:12 Tower kernel: REISERFS warning: reiserfs-5088 is_internal: number of key seems wrong: level=2, nr_items=47784, free_space=43340 rdkey Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 181520453. Fsck? Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2450 2461 0x0 SD] Feb 24 08:08:12 Tower kernel: REISERFS warning: reiserfs-5088 is_internal: number of key seems wrong: level=2, nr_items=47784, free_space=43340 rdkey Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 181520453. Fsck? Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2450 2461 0x0 SD] Feb 24 08:08:12 Tower kernel: REISERFS warning: reiserfs-5088 is_internal: number of key seems wrong: level=2, nr_items=47784, free_space=43340 rdkey Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 181520453. Fsck? Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2462 2463 0x0 SD] Feb 24 08:08:12 Tower kernel: REISERFS warning: reiserfs-5088 is_internal: number of key seems wrong: level=2, nr_items=47784, free_space=43340 rdkey Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 181520453. Fsck? Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2462 2463 0x0 SD] Feb 24 08:08:12 Tower kernel: REISERFS warning: reiserfs-5088 is_internal: number of key seems wrong: level=2, nr_items=47784, free_space=43340 rdkey Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 181520453. Fsck? Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2462 2464 0x0 SD] Feb 24 08:08:12 Tower kernel: REISERFS warning: reiserfs-5088 is_internal: number of key seems wrong: level=2, nr_items=47784, free_space=43340 rdkey Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 181520453. Fsck? Feb 24 08:08:12 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2462 2464 0x0 SD] Feb 24 08:08:16 Tower kernel: REISERFS warning: reiserfs-5081 is_leaf: nr_item seems wrong: level=1, nr_items=47618, free_space=4164 rdkey Feb 24 08:08:16 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 16170478. Fsck? Feb 24 08:08:16 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2358 2367 0x0 SD] Feb 24 08:08:16 Tower kernel: REISERFS warning: reiserfs-5081 is_leaf: nr_item seems wrong: level=1, nr_items=47618, free_space=4164 rdkey Feb 24 08:08:16 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 16170478. Fsck? Feb 24 08:08:16 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2358 2367 0x0 SD] Feb 24 08:08:16 Tower kernel: REISERFS warning: reiserfs-5081 is_leaf: nr_item seems wrong: level=1, nr_items=47618, free_space=9028 rdkey Feb 24 08:08:16 Tower kernel: REISERFS error (device md5): vs-5150 search_by_key: invalid format found in block 16223142. Fsck? Feb 24 08:08:16 Tower kernel: REISERFS error (device md5): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [2358 2368 0x0 SD] Feb 24 08:08:16 Tower kernel: REISERFS warning: reiserfs-5081 is_leaf: nr_item seems wrong: level=1, nr_items=47618, free_space=9028 rdkey Link to comment
bfeist Posted February 24, 2011 Author Share Posted February 24, 2011 Some more info. I ran a disk check on disk5 following the wiki's instructions. The results are below. My guess is that the old disk suffered some data corruption which was replicated into parity when I redid a parity check (at some point in the past). This corruption was restored to the new disk when I rebuild disk5. My question is: how do I repair the filesystem on disk5 correctly? Thanks, Ben root@Tower:~# samba stop root@Tower:~# umount /dev/md5 root@Tower:~# reiserfsck --check /dev/md5 reiserfsck 3.6.21 (2009 www.namesys.com) ************************************************************* ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will read-only check consistency of the filesystem on /dev/md5 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes ########### reiserfsck --check started at Thu Feb 24 11:15:45 2011 ########### Replaying journal: Trans replayed: mountid 4294967286, transid 4294966495, desc 867, len 1, commit 869, next trans offset 852 Trans replayed: mountid 4294967286, transid 4294966496, desc 870, len 1, commit 872, next trans offset 855 Trans replayed: mountid 4294967286, transid 4294966497, desc 873, len 1, commit 875, next trans offset 858 Trans replayed: mountid 4294967286, transid 4294966498, desc 876, len 1, commit 878, next trans offset 861 Trans replayed: mountid 4294967286, transid 4294966499, desc 879, len 1, commit 881, next trans offset 864 Trans replayed: mountid 4294967286, transid 4294966500, desc 882, len 1, commit 884, next trans offset 867 Trans replayed: mountid 4294967286, transid 4294966501, desc 885, len 1, commit 887, next trans offset 870 Trans replayed: mountid 4294967286, transid 4294966502, desc 888, len 1, commit 890, next trans offset 873 Trans replayed: mountid 4294967286, transid 4294966503, desc 891, len 1, commit 893, next trans offset 876 Replaying journal: Done. Reiserfs journal '/dev/md5' in blocks [18..8211]: 9 transactions replayed Checking internal tree.. \/ 1 (of 9|/ 17 (of 129\/ 5 (of 170|block 51445763: The level of the node (3) is not correct, (1) expected the problem in the internal node occured (51445763), whole subtree is skipped / 28 (of 129\/ 55 (of 170- block 53389811: The number of items (47616) is incorrect, should be (0) the problem in the internal node occured (53389811), whole subtree is skipped / 29 (of 129\block 5351014 6: The level of the node (3) is not correct, (2) expected the problem in the internal node occured (53510146), whole subtree is skipped / 2 (of 9|/ 15 (of 149\ / 62 (of 170/block 166721138: The number of items (47616) is incorrect, should be (0) the problem in the internal node occured (166721138), whole subtree is skipped / 38 (of 149|/158 (of 170- block 198136927: The number of items (47616) is incorrect, should be (0) the problem in the internal node occured (198136927), whole subtree is skipped / 39 (of 149\/ 1 (of 170| block 198150091: The number of items (47616) is incorrect, should be (0) the problem in the internal node occured (198150091), whole subtree is skipped / 40 (of 149/bad_internal: vpf-10320: block 198233168, items 0 and 1: The wrong order of items: [2475639765 2215658100 0x2a5491d4ff54111 (10)], [144378126 3232 724046 0x89de45c3779001 (14)] the problem in the internal node occured (198233168), whole subtree is skipped / 3 (of 9-/ 9 (of 16/ 13 (of 162\/111 (of 170\block 166721139: The number of items (47616) is incorrect, should be (0) the problem in the internal node occured (166721139), whole subtree is skipped / 93 (ofblock 166721199: The number of items (47616) is incorrect, should be (0) the problem in the internal node occured (166721199), whole subtree is skipped / 94 (of vpf-10320: block 166721282, items 0 and 1: The wrong order of items: [4054060951 3394013161 0x70701515641243d (13)], [3304685315 3127079367 0xb (6)] the problem in the internal node occured (166721282), whole subtree is skipped / 4 (of/118 (of 170|bad_path: The left delimiting key [18 2713 0xcbdf3001 IND (1)] of the node (165838849) must be equal to the first element's key [1000 1(1)] within the node. bad_path: The left delimiting key [18 2713 0xcc1e7001 IND (1)] of the node (165838850) must be equal to the first element's key [1000 1001 0xace4001he node. bad_path: The left delimiting key [18 2713 0xcc5db001 IND (1)] of the node (165838851) must be equal to the first element's key [1000 1001 0xb0d8001he node. bad_path: The left delimiting key [18 2713 0xcc9cf001 IND (1)] of the node (165838852) must be equal to the first element's key [1000 1001 0xb4cc001he node. block 165841832: The level of the node (44870) is not correct, (1) expected the problem in the internal node occured (165841832), whole subtree is skipped / 92 (ofblock 165889436: The number of items (47616) is incorrect, should be (0) the problem in the internal node occured (165889436), whole subtree is skipped / 93 (of vpf-10320: block 165913748, items 0 and 1: The wrong order of items: [3679432907 4063185254 0x69a85a8fe480129 DIR (3)], [2958692052 2677212795 0xe2(4)] the problem in the internal node occured (165913748), whole subtree is skipped / 5 (of/ 41 (of 85-bad_node: vpf-10350: The block (165841720) is used more than once in the tree. the problem in the internal node occured (165841720), whole subtree is skipped / 6 (of 9// 55 (ofblock 16121857: The level of the node (0) is not correct, (1) expected the problem in the internal node occured (16121857), whole subtree is skipped / 56 (offault root@Tower:~# Link to comment
Joe L. Posted February 24, 2011 Share Posted February 24, 2011 Some more info. I ran a disk check on disk5 following the wiki's instructions. The results are below. My guess is that the old disk suffered some data corruption which was replicated into parity when I redid a parity check (at some point in the past). This corruption was restored to the new disk when I rebuild disk5. My question is: how do I repair the filesystem on disk5 correctly? Thanks, Ben root@Tower:~# samba stop root@Tower:~# umount /dev/md5 root@Tower:~# reiserfsck --check /dev/md5 reiserfsck 3.6.21 (2009 www.namesys.com) ************************************************************* ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will read-only check consistency of the filesystem on /dev/md5 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes ########### reiserfsck --check started at Thu Feb 24 11:15:45 2011 ########### Replaying journal: Trans replayed: mountid 4294967286, transid 4294966495, desc 867, len 1, commit 869, next trans offset 852 Trans replayed: mountid 4294967286, transid 4294966496, desc 870, len 1, commit 872, next trans offset 855 Trans replayed: mountid 4294967286, transid 4294966497, desc 873, len 1, commit 875, next trans offset 858 Trans replayed: mountid 4294967286, transid 4294966498, desc 876, len 1, commit 878, next trans offset 861 Trans replayed: mountid 4294967286, transid 4294966499, desc 879, len 1, commit 881, next trans offset 864 Trans replayed: mountid 4294967286, transid 4294966500, desc 882, len 1, commit 884, next trans offset 867 Trans replayed: mountid 4294967286, transid 4294966501, desc 885, len 1, commit 887, next trans offset 870 Trans replayed: mountid 4294967286, transid 4294966502, desc 888, len 1, commit 890, next trans offset 873 Trans replayed: mountid 4294967286, transid 4294966503, desc 891, len 1, commit 893, next trans offset 876 Replaying journal: Done. Reiserfs journal '/dev/md5' in blocks [18..8211]: 9 transactions replayed Checking internal tree.. \/ 1 (of 9|/ 17 (of 129\/ 5 (of 170|block 51445763: The level of the node (3) is not correct, (1) expected the problem in the internal node occured (51445763), whole subtree is skipped / 28 (of 129\/ 55 (of 170- block 53389811: The number of items (47616) is incorrect, should be (0) the problem in the internal node occured (53389811), whole subtree is skipped / 29 (of 129\block 5351014 6: The level of the node (3) is not correct, (2) expected the problem in the internal node occured (53510146), whole subtree is skipped / 2 (of 9|/ 15 (of 149\ / 62 (of 170/block 166721138: The number of items (47616) is incorrect, should be (0) the problem in the internal node occured (166721138), whole subtree is skipped / 38 (of 149|/158 (of 170- block 198136927: The number of items (47616) is incorrect, should be (0) the problem in the internal node occured (198136927), whole subtree is skipped / 39 (of 149\/ 1 (of 170| block 198150091: The number of items (47616) is incorrect, should be (0) the problem in the internal node occured (198150091), whole subtree is skipped / 40 (of 149/bad_internal: vpf-10320: block 198233168, items 0 and 1: The wrong order of items: [2475639765 2215658100 0x2a5491d4ff54111 (10)], [144378126 3232 724046 0x89de45c3779001 (14)] the problem in the internal node occured (198233168), whole subtree is skipped / 3 (of 9-/ 9 (of 16/ 13 (of 162\/111 (of 170\block 166721139: The number of items (47616) is incorrect, should be (0) the problem in the internal node occured (166721139), whole subtree is skipped / 93 (ofblock 166721199: The number of items (47616) is incorrect, should be (0) the problem in the internal node occured (166721199), whole subtree is skipped / 94 (of vpf-10320: block 166721282, items 0 and 1: The wrong order of items: [4054060951 3394013161 0x70701515641243d (13)], [3304685315 3127079367 0xb (6)] the problem in the internal node occured (166721282), whole subtree is skipped / 4 (of/118 (of 170|bad_path: The left delimiting key [18 2713 0xcbdf3001 IND (1)] of the node (165838849) must be equal to the first element's key [1000 1(1)] within the node. bad_path: The left delimiting key [18 2713 0xcc1e7001 IND (1)] of the node (165838850) must be equal to the first element's key [1000 1001 0xace4001he node. bad_path: The left delimiting key [18 2713 0xcc5db001 IND (1)] of the node (165838851) must be equal to the first element's key [1000 1001 0xb0d8001he node. bad_path: The left delimiting key [18 2713 0xcc9cf001 IND (1)] of the node (165838852) must be equal to the first element's key [1000 1001 0xb4cc001he node. block 165841832: The level of the node (44870) is not correct, (1) expected the problem in the internal node occured (165841832), whole subtree is skipped / 92 (ofblock 165889436: The number of items (47616) is incorrect, should be (0) the problem in the internal node occured (165889436), whole subtree is skipped / 93 (of vpf-10320: block 165913748, items 0 and 1: The wrong order of items: [3679432907 4063185254 0x69a85a8fe480129 DIR (3)], [2958692052 2677212795 0xe2(4)] the problem in the internal node occured (165913748), whole subtree is skipped / 5 (of/ 41 (of 85-bad_node: vpf-10350: The block (165841720) is used more than once in the tree. the problem in the internal node occured (165841720), whole subtree is skipped / 6 (of 9// 55 (ofblock 16121857: The level of the node (0) is not correct, (1) expected the problem in the internal node occured (16121857), whole subtree is skipped / 56 (offault root@Tower:~# You follow almost the same steps, but instead of "--check", you use "--fix-fixable" root@Tower:~# samba stop root@Tower:~# umount /dev/md5 root@Tower:~# reiserfsck --fix-fixable /dev/md5 Link to comment
bfeist Posted February 24, 2011 Author Share Posted February 24, 2011 Thanks for the help Joe. I did the --fix-fixable and it resulted in a segmentation fault in reiserfsck. I noticed that the write count of disk5 did not go up in the unraid gui. The reiserfsck output is below. Any ideas on what I can try next? Ben root@Tower:~# reiserfsck --fix-fixable /dev/md5 reiserfsck 3.6.21 (2009 www.namesys.com) ************************************************************* ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will check consistency of the filesystem on /dev/md5 and will fix what can be fixed without --rebuild-tree Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes ########### reiserfsck --fix-fixable started at Thu Feb 24 11:58:47 2011 ########### Replaying journal: Done. Reiserfs journal '/dev/md5' in blocks [18..8211]: 0 transactions replayed Checking internal tree.. \/ 1 (of 9|/ 17 (of 129\/ 5 (of 170|block 51445763: The level of the node (3) is not correct, (1) expected the problem in the internal node occured (51445763), whole subtree is skipped / 28 (of 129\/ 55 (of 170-block 53389811: The number of items (47616) is incorrect, should be (0) the problem in the internal node occured (53389811), whole subtree is skipped / 29 (of 129\block 53510146: The level of the node (3) is not correct, (2) expected the problem in the internal node occured (53510146), whole subtree is skipped / 2 (of 9|/ 15 (of 149\/ 62 (of 170/block 166721138: The number of items (47616) is incorrect, should be (0) the problem in the internal node occured (166721138), whole subtree is skipped / 38 (of 149|/158 (of 170-block 198136927: The number of items (47616) is incorrect, should be (0) the problem in the internal node occured (198136927), whole subtree is skipped / 39 (of 149\/ 1 (of 170|block 198150091: The number of items (47616) is incorrect, should be (0) the problem in the internal node occured (198150091), whole subtree is skipped / 40 (of 149/bad_internal: vpf-10320: block 198233168, items 0 and 1: The wrong order of items: [2475639765 2215658100 0x2a5491d4ff54111 (10)], [144378126 3232724046 0x89de45c3779001 (14)] the problem in the internal node occured (198233168), whole subtree is skipped / 3 (of 9-/ 92 (of 162|/111 (of 170\block 166721139: The number of items (47616) is incorrect, should be (0) the problem in the internal node occured (166721139), whole subtree is skipped / 93 (of 162|/ 1 (of 170/block 166721199: The number of items (47616) is incorrect, should be (0) the problem in the internal node occured (166721199), whole subtree is skipped / 94 (of 162-bad_internal: vpf-10320: block 166721282, items 0 and 1: The wrong order of items: [4054060951 3394013161 0x70701515641243d (13)], [3304685315 3127079367 0xb5042fd21c6c2a5 (6)] the problem in the internal node occured (166721282), whole subtree is skipped / 4 (of 9\/ 91 (of 170-/118 (of 170|bad_path: The left delimiting key [18 2713 0xcbdf3001 IND (1)] of the node (165838849) must be equal to the first element's key [1000 1001 0xa8f0001 IND (1)] within the node. /119 (of 170/bad_path: The left delimiting key [18 2713 0xcc1e7001 IND (1)] of the node (165838850) must be equal to the first element's key [1000 1001 0xace4001 IND (1)] within the node. /120 (of 170-bad_path: The left delimiting key [18 2713 0xcc5db001 IND (1)] of the node (165838851) must be equal to the first element's key [1000 1001 0xb0d8001 IND (1)] within the node. /121 (of 170\bad_path: The left delimiting key [18 2713 0xcc9cf001 IND (1)] of the node (165838852) must be equal to the first element's key [1000 1001 0xb4cc001 IND (1)] within the node. /124 (of 170-block 165841832: The level of the node (44870) is not correct, (1) expected the problem in the internal node occured (165841832), whole subtree is skipped / 92 (of 170\/ 1 (of 170|block 165889436: The number of items (47616) is incorrect, should be (0) the problem in the internal node occured (165889436), whole subtree is skipped / 93 (of 170/bad_internal: vpf-10320: block 165913748, items 0 and 1: The wrong order of items: [3679432907 4063185254 0x69a85a8fe480129 DIR (3)], [2958692052 2677212795 0xe2b410947f24a29 (4)] the problem in the internal node occured (165913748), whole subtree is skipped / 5 (of 9-/ 79 (of 128// 41 (of 85-bad_node: vpf-10350: The block (165841720) is used more than once in the tree. the problem in the internal node occured (165841720), whole subtree is skipped / 6 (of 9// 55 (of 130\/ 36 (of 170\block 16121857: The level of the node (0) is not correct, (1) expected the problem in the internal node occured (16121857), whole subtree is skipped / 56 (of 130|Segmentation fault root@Tower:~# Link to comment
dgaschk Posted February 24, 2011 Share Posted February 24, 2011 Apart from the Segmentation fault, this means that your original may still be good. Check its SMART report. Link to comment
Joe L. Posted February 24, 2011 Share Posted February 24, 2011 Apart from the Segmentation fault, this means that your original may still be good. Check its SMART report. In my previous experience it was insufficient memory that caused the segmentation error. Link to comment
dgaschk Posted February 24, 2011 Share Posted February 24, 2011 Apart from the Segmentation fault, this means that your original may still be good. Check its SMART report. In my previous experience it was insufficient memory that caused the segmentation error. Yes, I agree. The segmentation fault is a memory issue. I meant to say that the original disk may still be good, since the problem is in the file system. This does not indicate a failing HD. Link to comment
bfeist Posted February 24, 2011 Author Share Posted February 24, 2011 Hmm, well I have 2GB of RAM. I'll try shutting down unraid completely and running the --fix-fixable on the old drive. Link to comment
dgaschk Posted February 24, 2011 Share Posted February 24, 2011 If it segmentation faults again you should run a memory test. Run it for a long time, preferably overnight. Link to comment
bfeist Posted February 24, 2011 Author Share Posted February 24, 2011 Ok, strange, I just ran reiserfsck --fix-fixable on the disk that had originally redballed (1TB seagate). It ran without error (results below). I'm starting to wonder if I have a cabling issue or something else that might be causing it. The new 1.5TB drive is not on the same cable as the original drive, but I did switch the cabling on the 1TB drive with another channel when I added the new one. Perhaps the switch fixed the problem on the 1TB drive which means that I have a SATA problem with my SAS card or something. I'll switch it back and run reiserfsck on the 1TB drive again to test. I'll also do a memory test overnight to make sure I'm on solid ground from that aspect. ########### reiserfsck --fix-fixable started at Thu Feb 24 17:01:31 2011 ########### Replaying journal: Trans replayed: mountid 64, transid 25018, desc 1310, len 1, commit 1312, next trans offset 1295 Trans replayed: mountid 64, transid 25019, desc 1313, len 1, commit 1315, next trans offset 1298 Replaying journal: Done. Reiserfs journal '/dev/sdk1' in blocks [18..8211]: 2 transactions replayed Checking internal tree.. finished Comparing bitmaps..finished Checking Semantic tree: finished No corruptions found There are on the filesystem: Leaves 203154 Internal nodes 1222 Directories 108 Other files 604 Data block pointers 205545636 (0 of them are zero) Safe links 0 ########### reiserfsck finished at Thu Feb 24 17:34:41 2011 ########### root@Tower:/boot# Link to comment
dgaschk Posted February 24, 2011 Share Posted February 24, 2011 Yes, those things you mention could be the cause of the segmentation fault. Link to comment
bfeist Posted February 25, 2011 Author Share Posted February 25, 2011 I just changed the cabling of the drive back and rebooted to run the reiserfsck check on the drive again. It's running now but I caught this in the log (below). It sure seems like a lot of random things are going wrong. Feb 24 20:27:59 Tower kernel: ------------[ cut here ]------------ Feb 24 20:27:59 Tower kernel: WARNING: at drivers/ata/libata-core.c:5186 ata_qc_issue+0x10b/0x308() Feb 24 20:27:59 Tower kernel: Hardware name: GA-MA74GM-S2 Feb 24 20:27:59 Tower kernel: Modules linked in: md_mod xor sata_promise atiixp ahci r8169 mvsas libsas scst scsi_transpo rt_sas [last unloaded: xor] Feb 24 20:27:59 Tower kernel: Pid: 2517, comm: hdparm Not tainted 2.6.32.9-unRAID #8 Feb 24 20:27:59 Tower kernel: Call Trace: Feb 24 20:27:59 Tower kernel: [<c102449e>] warn_slowpath_common+0x60/0x77 Feb 24 20:27:59 Tower kernel: [<c10244c2>] warn_slowpath_null+0xd/0x10 Feb 24 20:27:59 Tower kernel: [<c11b624d>] ata_qc_issue+0x10b/0x308 Feb 24 20:27:59 Tower kernel: [<c11ba260>] ata_scsi_translate+0xd1/0xff Feb 24 20:27:59 Tower kernel: [<c11a816c>] ? scsi_done+0x0/0xd Feb 24 20:27:59 Tower kernel: [<c11a816c>] ? scsi_done+0x0/0xd Feb 24 20:27:59 Tower kernel: [<c11baa40>] ata_sas_queuecmd+0x120/0x1d7 Feb 24 20:27:59 Tower kernel: [<c11bc6df>] ? ata_scsi_pass_thru+0x0/0x21d Feb 24 20:27:59 Tower kernel: [<f842569a>] sas_queuecommand+0x65/0x20d [libsas] Feb 24 20:27:59 Tower kernel: [<c11a816c>] ? scsi_done+0x0/0xd Feb 24 20:27:59 Tower kernel: [<c11a82c0>] scsi_dispatch_cmd+0x147/0x181 Feb 24 20:27:59 Tower kernel: [<c11ace4d>] scsi_request_fn+0x351/0x376 Feb 24 20:27:59 Tower kernel: [<c1126798>] __blk_run_queue+0x78/0x10c Feb 24 20:27:59 Tower kernel: [<c1124446>] elv_insert+0x67/0x153 Feb 24 20:27:59 Tower kernel: [<c11245b8>] __elv_add_request+0x86/0x8b Feb 24 20:27:59 Tower kernel: [<c1129343>] blk_execute_rq_nowait+0x4f/0x73 Feb 24 20:27:59 Tower kernel: [<c11293dc>] blk_execute_rq+0x75/0x91 Feb 24 20:27:59 Tower kernel: [<c11292cc>] ? blk_end_sync_rq+0x0/0x28 Feb 24 20:27:59 Tower kernel: [<c112636f>] ? get_request+0x204/0x28d Feb 24 20:27:59 Tower kernel: [<c11269d6>] ? get_request_wait+0x2b/0xd9 Feb 24 20:27:59 Tower kernel: [<c112c2bf>] sg_io+0x22d/0x30a Feb 24 20:27:59 Tower kernel: [<c112c5a8>] scsi_cmd_ioctl+0x20c/0x3bc Feb 24 20:27:59 Tower kernel: [<c11b3257>] sd_ioctl+0x6a/0x8c Feb 24 20:27:59 Tower kernel: [<c112a420>] __blkdev_driver_ioctl+0x50/0x62 Feb 24 20:27:59 Tower kernel: [<c112ad1c>] blkdev_ioctl+0x8b0/0x8dc Feb 24 20:27:59 Tower kernel: [<c1131e2d>] ? kobject_get+0x12/0x17 Feb 24 20:27:59 Tower kernel: [<c112b0f8>] ? get_disk+0x4a/0x61 Feb 24 20:27:59 Tower kernel: [<c101b028>] ? kmap_atomic+0x14/0x16 Feb 24 20:27:59 Tower kernel: [<c11334a5>] ? radix_tree_lookup_slot+0xd/0xf Feb 24 20:27:59 Tower kernel: [<c104a179>] ? filemap_fault+0xb8/0x305 Feb 24 20:27:59 Tower kernel: [<c1048c43>] ? unlock_page+0x18/0x1b Feb 24 20:27:59 Tower kernel: [<c1057c63>] ? __do_fault+0x3a7/0x3da Feb 24 20:27:59 Tower kernel: [<c105985f>] ? handle_mm_fault+0x42d/0x8f1 Feb 24 20:27:59 Tower kernel: [<c108b6c6>] block_ioctl+0x2a/0x32 Feb 24 20:27:59 Tower kernel: [<c108b69c>] ? block_ioctl+0x0/0x32 Feb 24 20:27:59 Tower kernel: [<c10769d5>] vfs_ioctl+0x22/0x67 Feb 24 20:27:59 Tower kernel: [<c1076f33>] do_vfs_ioctl+0x478/0x4ac Feb 24 20:27:59 Tower kernel: [<c105dcdd>] ? do_mmap_pgoff+0x232/0x294 Feb 24 20:27:59 Tower kernel: [<c1076f93>] sys_ioctl+0x2c/0x45 Feb 24 20:27:59 Tower kernel: [<c1002935>] syscall_call+0x7/0xb Feb 24 20:27:59 Tower kernel: ---[ end trace 63108e687c99d1bd ]--- root@Tower:~# Link to comment
bfeist Posted February 25, 2011 Author Share Posted February 25, 2011 I just had a thought. Could this be the dreaded HPA problem that i've been reading about. I have a GA-MA74GM-S2 motherboard. Maybe when I switched the cabling from my SAS card to the motherboard the Gigabyte motherboard wrote the HPA data to the drive. It could be possible that a drive has never been connected directly to the motherboard before, it may have always been connected to either my SAS card or my internal PCI SATA card that I used to use. I did some searching and it looks like that kernel dump has been attributed to the Gigabyte HPA problem before. Link to comment
bfeist Posted February 25, 2011 Author Share Posted February 25, 2011 An update: I put the original 1TB drive back into the array and invalidated parity. I rebuilt parity overnight with 0 errors. I will now preclear the 1.5TB disk that was throwing the segmentation fault. Link to comment
bfeist Posted February 25, 2011 Author Share Posted February 25, 2011 Well, everything seems to be back to normal and all I did were two things: I opened the case and double checked that all SATA cables were secure. I didn't notice any loose but who knows what I might have inadvertently fixed. I downgraded to 4.6 after reading all the threads about Gigabyte motherboards and HPA problems that happened under 4.7. I don't really know if this has helped but I'm now mentally stuck against upgrading to 4.7 or 5.x without risking everything. Link to comment
Recommended Posts
Archived
This topic is now archived and is closed to further replies.