May 25, 201214 yr Hello, Looking for direction on how to correct this problem on unRaid v5.0b3. I know I need to upgrade, but need to get it working and then backup 16TB as I do not trust the unRaid upgrades as I have lost stuff in the past. This unRaid has been running fine with no problems for over a year. The system has 10 2TB drives in it. A couple weeks ago, I had a 2TB drive fail and replaced it with a RMA from Western Digital. I followed the correct procedures and it rebuild successfully. Then a day later another drive failed. I replaced this one and it looked to rebuild successfully. Before I could verify, the system was hung. I had to reboot and then a parity-check started automatically and it showed this 9th drive as green. After various amounts of time, the system hangs and the syslog fills up until the server does not even respond to pings. I was able to grab enough of the syslog before the no response. Here are the lines that begin to be recorded, along with the syslog attached. Thanks for any direction/assistance. Peace, Michael May 25 12:15:58 Tower kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 37926 does not match to the expected one 1 May 25 12:15:58 Tower kernel: REISERFS error (device md9): vs-5150 search_by_key: invalid format found in block 463011841. Fsck? May 25 12:15:58 Tower kernel: REISERFS (device md9): Remounting filesystem read-only May 25 12:15:58 Tower kernel: REISERFS error (device md9): vs-13070 reiserfs_read_locked_inode: i/o failure occurred trying to find stat data of [643 1256 0x0 SD] May 25 12:15:58 Tower kernel: REISERFS warning: reiserfs-5090 is_tree_node: node level 37926 does not match to the expected one 1 May 25 12:15:58 Tower kernel: REISERFS error (device md9): vs-5150 search_by_key: invalid format found in block 463011841. Fsck? Update - this was resolved by upgrading to 5.0-rc8a. syslog3.zip
July 13, 201213 yr Author Hello Folks, Still fighting this problem for a couple months and unable to bring up my disk 9. Ran reiserfsck --check /dev/md9 and received the following: segfault at 81b8920 ip 080636ba sp bf815470 error 4 in reiserfsck Attached is the syslog. Any ideas on what to try and run next? Many thanks. Sincerel, Michael syslog1.txt
July 13, 201213 yr Author Hello Folks, Just to add, I have tried 2 brand new drives for md9 and rebuilding fails. I run into recursive errors and the whole unRaid locks up. Wondering if I need to run: reiserfsck with the --rebuild-sb or --rebuild-tree switches Looking for that secret special sauce command that fixes everything. Many thanks for anyone's time and efforts and hopefully I can post a resolution soon for people to see if they run into the same issue. Peace, Michael
July 13, 201213 yr Try again. This time use telnet (putty) to connect to the server and then paste the terminal session into a post. Show everything from login to segfault.
July 14, 201213 yr Author Here she goes. This is after a reboot. Stubborn little problem. Thanks dgaschk. Tower login: root Password: Linux 2.6.32.9-unRAID. root@Tower:~# cd root@Tower:~# samba stop root@Tower:~# umount /dev/md9 root@Tower:~# reiserfsck --check /dev/md9 reiserfsck 3.6.21 (2009 www.namesys.com) ************************************************************* ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will read-only check consistency of the filesystem on /dev/md9 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes ########### reiserfsck --check started at Fri Jul 13 13:22:35 2012 ########### Replaying journal: Done. Reiserfs journal '/dev/md9' in blocks [18..8211]: 0 transactions replayed Zero bit found in on-disk bitmap after the last valid bit. Checking internal tree.. \/ 1 (of 9|/123 (of 170// 78 (of 86\block 485851137: The level of the node (46687) is not correct, (1) expected the problem in the internal node occured (485851137),/124 (of 170|/ 37 (of 85/block 463011841: The level of the node (37926) is not correct, (1) expected the problem in the internal node occured/ 2 (of 9// 11 (of 170|/ 60 (of 86|block 455049235: The number of items (47616) is incorrect, should be (0) the problem in the internal node occured (455049235),/ 12 (of 170/bad_internal: vpf-10320: block 455049237, items 0 and 1: The wrong order of items: [1378396848 40768891 0x4a891642e75a5 (4)], [512771986 612778150 0xaf79de67c2fcd9b (9)] the problem in the internal node occured (455049237),/ 3 (of 9-/ 46 (of 170-/ 63 (of 89/block 473006081: The level of the node (0) is not correct, (1) expected the problem in the internal node occured (473006081),/103 (of 170// 81 (of 86-block 485523457: The level of the node (37008) is not correct, (1) expected the problem in the internal node occured/ 4 (of 9|/ 45 (of 170//132 (of 170/block 455639041: The number of items (47637) is incorrect, should be (0) the problem in the internal node occured (455639041),/108 (of 170|/157 (of 170/block 487751681: The level of the node (55350) is not correct, (1) expected the problem in the internal node occured (487751681),/109 (of 170-block 487751696: The level of the node (56373) is not correct, (2) expected the problem in the internal node occured (487751696),/ 5 (of 9\/ 33 (of 170|/ 23 (of 170\block 482902017: The level of the node (3349) is not correct, (1) expected the problem in the internal node occured (482902017),/ 85 (of 170// 83 (of 87|block 476610561: The level of the node (5326) is not correct, (1) expected the problem in the internal node occured (476610561),/ 86 (of 170/block 476610567: The level of the node (6757) is not correct, (2) expected the problem in the internal node occured (476610567),/ 6 (of 9-/ 13 (of 170// 99 (of 139|block 461799425: The level of the node (64809) is not correct, (1) expected the problem in the internal node occured (461799425),/ 60 (of 170\/ 25 (of 86|block 468811777: The level of the node (42368) is not correct, (1) expected the problem in the internal node occured (468811777),/128 (of 170//138 (of 170\block 484580406: The level of the node (53415) is not correct, (1) expected the problem in the internal node occured (484580406),/129 (of 170|block 463601679: The level of the node (65533) is not correct, (2) expected the problem in the internal node occured (463601679),/ 7 (of 9// 21 (of 170-/132 (of 133-block 473825281: The level of the node (8311) is not correct, (1) expected the problem in the internal node occured (473825281),/ 94 (of 170// 91 (of 170|block 469204993: The level of the node (33597) is not correct, (1) expected the problem in the internal node occured (469204993),/ 95 (of 170// 18 (of 170\block 463718855: The level of the node (795) is not correct, (1) expected the problem in the internal node occured (463718855),/148 (of 170// 6 (of 91\block 484580643: The level of the node (62387) is not correct, (1) expected the problem in the internal node occured/ 8 (of 9-/ 31 (of 128// 33 (of 170-block 455049238: The number of items (47621) is incorrect, should be (0) the problem in the internal node occured (455049238),/ 32 (of 128\/ 23 (of 86-block 468156417: The level of the node (0) is not correct, (1) expected the problem in the internal node occured (468156417),/ 44 (of 128\/162 (of 170/block 447958099: The number of items (47616) is incorrect, should be (0) the problem in the internal node occured (447958099),/ 45 (of 128-Segmentation fault root@Tower:~#
July 15, 201213 yr Here she goes. This is after a reboot. Stubborn little problem. Thanks dgaschk. Tower login: root Password: Linux 2.6.32.9-unRAID. root@Tower:~# cd root@Tower:~# samba stop root@Tower:~# umount /dev/md9 root@Tower:~# reiserfsck --check /dev/md9 reiserfsck 3.6.21 (2009 www.namesys.com) ************************************************************* ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will read-only check consistency of the filesystem on /dev/md9 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes ########### reiserfsck --check started at Fri Jul 13 13:22:35 2012 ########### Replaying journal: Done. Reiserfs journal '/dev/md9' in blocks [18..8211]: 0 transactions replayed Zero bit found in on-disk bitmap after the last valid bit. Checking internal tree.. \/ 1 (of 9|/123 (of 170// 78 (of 86\block 485851137: The level of the node (46687) is not correct, (1) expected the problem in the internal node occured (485851137),/124 (of 170|/ 37 (of 85/block 463011841: The level of the node (37926) is not correct, (1) expected the problem in the internal node occured/ 2 (of 9// 11 (of 170|/ 60 (of 86|block 455049235: The number of items (47616) is incorrect, should be (0) the problem in the internal node occured (455049235),/ 12 (of 170/bad_internal: vpf-10320: block 455049237, items 0 and 1: The wrong order of items: [1378396848 40768891 0x4a891642e75a5 (4)], [512771986 612778150 0xaf79de67c2fcd9b (9)] the problem in the internal node occured (455049237),/ 3 (of 9-/ 46 (of 170-/ 63 (of 89/block 473006081: The level of the node (0) is not correct, (1) expected the problem in the internal node occured (473006081),/103 (of 170// 81 (of 86-block 485523457: The level of the node (37008) is not correct, (1) expected the problem in the internal node occured/ 4 (of 9|/ 45 (of 170//132 (of 170/block 455639041: The number of items (47637) is incorrect, should be (0) the problem in the internal node occured (455639041),/108 (of 170|/157 (of 170/block 487751681: The level of the node (55350) is not correct, (1) expected the problem in the internal node occured (487751681),/109 (of 170-block 487751696: The level of the node (56373) is not correct, (2) expected the problem in the internal node occured (487751696),/ 5 (of 9\/ 33 (of 170|/ 23 (of 170\block 482902017: The level of the node (3349) is not correct, (1) expected the problem in the internal node occured (482902017),/ 85 (of 170// 83 (of 87|block 476610561: The level of the node (5326) is not correct, (1) expected the problem in the internal node occured (476610561),/ 86 (of 170/block 476610567: The level of the node (6757) is not correct, (2) expected the problem in the internal node occured (476610567),/ 6 (of 9-/ 13 (of 170// 99 (of 139|block 461799425: The level of the node (64809) is not correct, (1) expected the problem in the internal node occured (461799425),/ 60 (of 170\/ 25 (of 86|block 468811777: The level of the node (42368) is not correct, (1) expected the problem in the internal node occured (468811777),/128 (of 170//138 (of 170\block 484580406: The level of the node (53415) is not correct, (1) expected the problem in the internal node occured (484580406),/129 (of 170|block 463601679: The level of the node (65533) is not correct, (2) expected the problem in the internal node occured (463601679),/ 7 (of 9// 21 (of 170-/132 (of 133-block 473825281: The level of the node (8311) is not correct, (1) expected the problem in the internal node occured (473825281),/ 94 (of 170// 91 (of 170|block 469204993: The level of the node (33597) is not correct, (1) expected the problem in the internal node occured (469204993),/ 95 (of 170// 18 (of 170\block 463718855: The level of the node (795) is not correct, (1) expected the problem in the internal node occured (463718855),/148 (of 170// 6 (of 91\block 484580643: The level of the node (62387) is not correct, (1) expected the problem in the internal node occured/ 8 (of 9-/ 31 (of 128// 33 (of 170-block 455049238: The number of items (47621) is incorrect, should be (0) the problem in the internal node occured (455049238),/ 32 (of 128\/ 23 (of 86-block 468156417: The level of the node (0) is not correct, (1) expected the problem in the internal node occured (468156417),/ 44 (of 128\/162 (of 170/block 447958099: The number of items (47616) is incorrect, should be (0) the problem in the internal node occured (447958099),/ 45 (of 128-Segmentation fault root@Tower:~# How much memory do you have? We've seen reiserfsck seg-fault when there was not enough RAM.
July 15, 201213 yr Author Hello Joe, Thanks for the reply. The system has 4GB's in it. I can upgrade it if needed? 8GB or 16GB? Thanks.
July 17, 201213 yr Author Hello Folks, Would anyone elese have any other ideas? Upgraded the system to 8GB's and same issue. Tried new 4GB sticks and same issue. Try --rebuild-sb or --rebuild-tree options?? Thanks for any additional steps to try. Here is the latest output from a telnet session: Tower login: root Password: Linux 2.6.32.9-unRAID. root@Tower:~# cd root@Tower:~# samba stop root@Tower:~# umount /dev/md9 root@Tower:~# reiserfsck --check /dev/md9 reiserfsck 3.6.21 (2009 www.namesys.com) ************************************************************* ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will read-only check consistency of the filesystem on /dev/md9 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes ########### reiserfsck --check started at Tue Jul 17 15:38:30 2012 ########### Replaying journal: Done. Reiserfs journal '/dev/md9' in blocks [18..8211]: 0 transactions replayed Zero bit found in on-disk bitmap after the last valid bit. Checking internal tree.. \/ 1 (of 9|/123 (of 170// 78 (of 86\block 485851137: The level of the node (46687) is not correct, (1) expected the problem in the internal node occured (485851137),/124 (of 170|/ 37 (of 85/block 463011841: The level of the node (37926) is not correct, (1) expected the problem in the internal node occured/ 2 (of 9// 11 (of 170|/ 60 (of 86|block 455049235: The number of items (47616) is incorrect, should be (0) the problem in the internal node occured (455049235),/ 12 (of 170/bad_internal: vpf-10320: block 455049237, items 0 and 1: The wrong order of items: [1378396848 40768891 0x4a891642e75a5 (4)], [512771986 612778150 0xaf79de67c2fcd9b (9)] the problem in the internal node occured (455049237),/ 3 (of 9-/ 46 (of 170-/ 63 (of 89/block 473006081: The level of the node (0) is not correct, (1) expected the problem in the internal node occured (473006081),/103 (of 170// 81 (of 86-block 485523457: The level of the node (37008) is not correct, (1) expected the problem in the internal node occured (48/ 4 (of 9|/ 45 (of 170//132 (of 170/block 455639041: The number of items (47637) is incorrect, should be (0) the problem in the internal node occured (455639041),/108 (of 170|/157 (of 170/block 487751681: The level of the node (55350) is not correct, (1) expected the problem in the internal node occured (487751681),/109 (of 170-block 487751696: The level of the node (56373) is not correct, (2) expected the problem in the internal node occured (487751696),/ 5 (of 9\/ 33 (of 170|/ 23 (of 170\block 482902017: The level of the node (3349) is not correct, (1) expected the problem in the internal node occured (482902017),/ 85 (of 170// 83 (of 87|block 476610561: The level of the node (5326) is not correct, (1) expected the problem in the internal node occured (476610561),/ 86 (of 170/block 476610567: The level of the node (6757) is not correct, (2) expected the problem in the internal node occured (476610567),/ 6 (of 9-/ 13 (of 170// 99 (of 139|block 461799425: The level of the node (64809) is not correct, (1) expected the problem in the internal node occured (461799425),/ 60 (of 170\/ 25 (of 86|block 468811777: The level of the node (42368) is not correct, (1) expected the problem in the internal node occured (468811777),/128 (of 170//138 (of 170\block 484580406: The level of the node (53415) is not correct, (1) expected the problem in the internal node occured (484580406),/129 (of 170|block 463601679: The level of the node (65533) is not correct, (2) expected the problem in the internal node occured (463601679),/ 7 (of 9// 21 (of 170-/132 (of 133-block 473825281: The level of the node (8311) is not correct, (1) expected the problem in the internal node occured (473825281),/ 94 (of 170// 91 (of 170|block 469204993: The level of the node (33597) is not correct, (1) expected the problem in the internal node occured (469204993),/ 95 (of 170// 18 (of 170\block 463718855: The level of the node (795) is not correct, (1) expected the problem in the internal node occured (463718855),/148 (of 170// 6 (of 91\block 484580643: The level of the node (62387) is not correct, (1) expected the problem in the internal node occured/ 8 (of 9-/ 31 (of 128// 33 (of 170-block 455049238: The number of items (47621) is incorrect, should be (0) the problem in the internal node occured (455049238),/ 32 (of 128\/ 23 (of 86-block 468156417: The level of the node (0) is not correct, (1) expected the problem in the internal node occured (468156417),/ 44 (of 128\/162 (of 170/block 447958099: The number of items (47616) is incorrect, should be (0) the problem in the internal node occured (447958099),/ 45 (of 128-Segmentation fault root@Tower:~#
July 18, 201213 yr Author Hello Support community, Still looking for any additional ideas on this seg fault? I have 2 possible solutions: 1. Try to upgrade to the latest 5.0 release and see if the reiserfsck program works any better. 2. Remove drive 9. Remove all content from that drive. Remove drive 9 from unRaid. Then add it back into the unRaid and have the system reconfigure with this new 9th drive. I have 3 different 2TB drives I can try. Would anyone suggest either of these or any other possibility with fixing reiserfsck? Many thanks as this has been a grueling battle. Sincerely Michael
July 18, 201213 yr Michael, It looks like you have SM X7SB4/E board and two SM SASLP controllers and they use the same IRQ16 - I will suggest to remove one card and to use the onboard SATA ports (they are superior to the used SM add-on cards)- especially for the disks you want to check. Keep in mind that some Unraid versions do not play nice with two of these SM cards (and perhaps even with a single one). After you fix the disk problems and upgrade to a newer version I suggest to check your motherboard BIOS and see if it is the latest version. Good luck
Archived
This topic is now archived and is closed to further replies.