June 26, 201016 yr Hello, Disk 1 on my system was coming up as read-only in the syslog. I ran reiserfsck --check and it instructed a --rebuild-tree. However rebuild-tree keeps failing at different places on Pass 3. Pass 1 and Pass 2 complete successfully (the initial runs corrected errors) Am am continuing to run --rebuild-tree and it would appear I am getting different errors after which resierfsck aborts. Any advice - do I just keep running again until I possibly get success? Thanks Matt Below are the last 2 failed runs: Tower login: root Linux 2.6.32.9-unRAID. root@Tower:~# cd root@Tower:~# samba stop root@Tower:~# umount /dev/md1 umount: /dev/md1: not mounted root@Tower:~# reiserfsck --rebuild-tree /dev/md1 reiserfsck 3.6.19 (2003 www.namesys.com) ************************************************************* ** Do not run the program with --rebuild-tree unless ** ** something is broken and MAKE A BACKUP before using it. ** ** If you have bad sectors on a drive it is usually a bad ** ** idea to continue using it. Then you probably should get ** ** a working hard drive, copy the file system from the bad ** ** drive to the good one -- dd_rescue is a good tool for ** ** that -- and only then run this program. ** ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will rebuild the filesystem (/dev/md1) tree Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes Replaying journal.. Reiserfs journal '/dev/md1' in blocks [18..8211]: 0 transactions replayed ########### reiserfsck --rebuild-tree started at Sat Jun 26 05:51:38 2010 ########### Pass 0: ####### Pass 0 ####### Loading on-disk bitmap .. ok, 195451388 blocks marked used Skipping 15663 blocks (super block, journal, bitmaps) 195435725 blocks will be r ead 0%....20% left 152474706, 1438 block 52844665: The number of items (28) is incorrect, should be (10) - correcte d block 52844665: The free space (236) is incorrect, should be (2064) - corrected block 161903153: The number of items (23) is incorrect, should be (10) - correct ed block 161903153: The free space (12) is incorrect, should be (1552) - corrected verify_directory_item: block 161903153, item 3559 3568 0x1 DIR (3), len 784, loc ation 3312 entry count 25, fsck need 1, format old: Found a name "VTS_04_0.VOB" for 21-th entry matching to the hash 1297977984. pass0: block 161903153, item 3559 3568 0x1 DIR (3), len 784, location 3312 entry count 25, fsck need 1, format old: 2 entries were deleted verify_directory_item: block 207474886, item 7579 7580 0x1 DIR (3), len 80, loca tion 640 entry count 3, fsck need 1, format old: All entries were deleted from t he directory left 0, 13668 /secccc 1 directory entries were hashed with not set hash. 11814 directory entries were hashed with "r5" hash. "r5" hash is selected Flushing..finished Read blocks (but not data blocks) 195435725 Leaves among those 194985 - corrected leaves 906 pointers in indirect items to wrong area 3437 (zeroed) Objectids found 11990 Pass 1 (will try to insert 194985 leaves): ####### Pass 1 ####### Looking for allocable blocks .. finished 0%....20%....40%....60%....80%....100% left 0, 624 /sec Flushing..finished 194985 leaves read 194977 inserted 8 not inserted non-unique pointers in indirect items (zeroed) 7181 ####### Pass 2 ####### Pass 2: 0%....20%....40%....60%....80%....100% left 0, 0 /sec Flushing..finished Leaves inserted item by item 8 Pass 3 (semantic): ####### Pass 3 ######### /Pictures/Oman 2008/DSC00466.JPGvpf-10680: The file [9451 9549] has the wrong bl ock count in the StatData (8304) - corrected to (8248) /Movies/Blow/VIDEO_TS/VTS_01_1.VOBvpf-10680: The file [356 363] has the wrong bl ock count in the StatData (2095624) - corrected to (2095480) /VTS_01_2.VOBvpf-10680: The file [356 364] has the wrong block count in the Stat Data (2096160) - corrected to (2096104) /VTS_01_3.VOBvpf-10680: The file [356 365] has the wrong block count in the Stat Data (2096600) - corrected to (2095440) /VTS_01_4.VOBvpf-10680: The file [356 366] has the wrong block count in the Stat Data (2096488) - corrected to (2096352) /VTS_01_5.VOBvpf-10680: The file [356 367] has the wrong block count in the Stat Data (365248) - corrected to (365192) /VTS_02_1.VOBvpf-10680: The file [356 371] has the wrong block count in the Stat Data (1101856) - corrected to (1101752) /VTS_05_1.VOBvpf-10680: The file [356 384] has the wrong block count in the Stat Data (1649192) - corrected to (1648320) /VTS_06_1.VOBvpf-10680: The file [356 388] has the wrong block count in the Stat Data (255632) - corrected to (255440) /Movies/Push/PUSH/VIDEO_TS/VTS_04_4.VOBvpf-10680: The file [5344 5365] has the w rong block count in the StatData (1940336) - corrected to (2097152) ... /Gremlins??_??Gremlins??2????The??New??Batch?? (1)/VIDEO_TS/VTS_02_1.VOBv pf-10680: The file [1685 1696] has the wrong block count in the StatData (209665 6) - corrected to (2095720) ... /Gremlins??_??Gremlins??2????The??New??Batch?? (1)/VIDEO_TS/VTS_02_2.VOBv pf-10680: The file [1685 1697] has the wrong block count in the StatData (209628 0) - corrected to (2095808) ... /Gremlins??_??Gremlins??2????The??New??Batch?? (1)/VIDEO_TS/VTS_02_5.VOBv pf-10680: The file [1685 1700] has the wrong block count in the StatData (209682 4) - corrected to (305488) ... /Gremlins??_??Gremlins??2????The??New??Batch?? (1)/VIDEO_TS/VTS_02_6.VOBv pf-10680: The file [1685 1701] has the wrong block count in the StatData (209658 4) - corrected to (240432) ... /Gremlins??_??Gremlins??2????The??New??Batch?? (1)/VIDEO_TS/VTS_03_1.VOBv pf-10680: The file [1685 1706] has the wrong block count in the StatData (999632 ) - corrected to (999328) /Movies/Meet??The??Parents/VIDEO_TS/VTS_01_1.VOBvpf-10680: The file [2571 2578] has the wrong block count in the StatData (32032) - corrected to (32064) semantic_rebuild.c 563 get_next_directory_item get_next_directory_item: item [2562 2571 0x28325a00 DIR (3)]: hidden entry 22 '' Aborted root@Tower:~# Fail number 2..... Tower login: root Linux 2.6.32.9-unRAID. root@Tower:~# cd root@Tower:~# samba stop root@Tower:~# umount /dev/md1 umount: /dev/md1: not mounted root@Tower:~# reiserfsck --rebuild-tree /dev/md1 reiserfsck 3.6.19 (2003 www.namesys.com) ************************************************************* ** Do not run the program with --rebuild-tree unless ** ** something is broken and MAKE A BACKUP before using it. ** ** If you have bad sectors on a drive it is usually a bad ** ** idea to continue using it. Then you probably should get ** ** a working hard drive, copy the file system from the bad ** ** drive to the good one -- dd_rescue is a good tool for ** ** that -- and only then run this program. ** ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will rebuild the filesystem (/dev/md1) tree Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes Replaying journal.. Reiserfs journal '/dev/md1' in blocks [18..8211]: 0 transactions replayed ########### reiserfsck --rebuild-tree started at Sat Jun 26 05:51:38 2010 ########### Pass 0: ####### Pass 0 ####### Loading on-disk bitmap .. ok, 195451388 blocks marked used Skipping 15663 blocks (super block, journal, bitmaps) 195435725 blocks will be r ead 0%....20% left 152474706, 1438 block 52844665: The number of items (28) is incorrect, should be (10) - correcte d block 52844665: The free space (236) is incorrect, should be (2064) - corrected block 161903153: The number of items (23) is incorrect, should be (10) - correct ed block 161903153: The free space (12) is incorrect, should be (1552) - corrected verify_directory_item: block 161903153, item 3559 3568 0x1 DIR (3), len 784, loc ation 3312 entry count 25, fsck need 1, format old: Found a name "VTS_04_0.VOB" for 21-th entry matching to the hash 1297977984. pass0: block 161903153, item 3559 3568 0x1 DIR (3), len 784, location 3312 entry count 25, fsck need 1, format old: 2 entries were deleted verify_directory_item: block 207474886, item 7579 7580 0x1 DIR (3), len 80, loca tion 640 entry count 3, fsck need 1, format old: All entries were deleted from t he directory left 0, 13668 /secccc 1 directory entries were hashed with not set hash. 11814 directory entries were hashed with "r5" hash. "r5" hash is selected Flushing..finished Read blocks (but not data blocks) 195435725 Leaves among those 194985 - corrected leaves 906 pointers in indirect items to wrong area 3437 (zeroed) Objectids found 11990 Pass 1 (will try to insert 194985 leaves): ####### Pass 1 ####### Looking for allocable blocks .. finished 0%....20%....40%....60%....80%....100% left 0, 624 /sec Flushing..finished 194985 leaves read 194977 inserted 8 not inserted non-unique pointers in indirect items (zeroed) 7181 ####### Pass 2 ####### Pass 2: 0%....20%....40%....60%....80%....100% left 0, 0 /sec Flushing..finished Leaves inserted item by item 8 Pass 3 (semantic): ####### Pass 3 ######### /Pictures/Oman 2008/DSC00466.JPGvpf-10680: The file [9451 9549] has the wrong bl ock count in the StatData (8304) - corrected to (8248) /Movies/Blow/VIDEO_TS/VTS_01_1.VOBvpf-10680: The file [356 363] has the wrong bl ock count in the StatData (2095624) - corrected to (2095480) /VTS_01_2.VOBvpf-10680: The file [356 364] has the wrong block count in the Stat Data (2096160) - corrected to (2096104) /VTS_01_3.VOBvpf-10680: The file [356 365] has the wrong block count in the Stat Data (2096600) - corrected to (2095440) /VTS_01_4.VOBvpf-10680: The file [356 366] has the wrong block count in the Stat Data (2096488) - corrected to (2096352) /VTS_01_5.VOBvpf-10680: The file [356 367] has the wrong block count in the Stat Data (365248) - corrected to (365192) /VTS_02_1.VOBvpf-10680: The file [356 371] has the wrong block count in the Stat Data (1101856) - corrected to (1101752) /VTS_05_1.VOBvpf-10680: The file [356 384] has the wrong block count in the Stat Data (1649192) - corrected to (1648320) /VTS_06_1.VOBvpf-10680: The file [356 388] has the wrong block count in the Stat Data (255632) - corrected to (255440) /Movies/Push/PUSH/VIDEO_TS/VTS_04_4.VOBvpf-10680: The file [5344 5365] has the w rong block count in the StatData (1940336) - corrected to (2097152) ... /Gremlins??_??Gremlins??2????The??New??Batch?? (1)/VIDEO_TS/VTS_02_1.VOBv pf-10680: The file [1685 1696] has the wrong block count in the StatData (209665 6) - corrected to (2095720) ... /Gremlins??_??Gremlins??2????The??New??Batch?? (1)/VIDEO_TS/VTS_02_2.VOBv pf-10680: The file [1685 1697] has the wrong block count in the StatData (209628 0) - corrected to (2095808) ... /Gremlins??_??Gremlins??2????The??New??Batch?? (1)/VIDEO_TS/VTS_02_5.VOBv pf-10680: The file [1685 1700] has the wrong block count in the StatData (209682 4) - corrected to (305488) ... /Gremlins??_??Gremlins??2????The??New??Batch?? (1)/VIDEO_TS/VTS_02_6.VOBv pf-10680: The file [1685 1701] has the wrong block count in the StatData (209658 4) - corrected to (240432) ... /Gremlins??_??Gremlins??2????The??New??Batch?? (1)/VIDEO_TS/VTS_03_1.VOBv pf-10680: The file [1685 1706] has the wrong block count in the StatData (999632 ) - corrected to (999328) /Movies/Meet??The??Parents/VIDEO_TS/VTS_01_1.VOBvpf-10680: The file [2571 2578] has the wrong block count in the StatData (32032) - corrected to (32064) semantic_rebuild.c 563 get_next_directory_item get_next_directory_item: item [2562 2571 0x28325a00 DIR (3)]: hidden entry 22 '' Aborted root@Tower:~# reiserfsck --rebuild-tree /dev/md1 reiserfsck 3.6.19 (2003 www.namesys.com) ************************************************************* ** Do not run the program with --rebuild-tree unless ** ** something is broken and MAKE A BACKUP before using it. ** ** If you have bad sectors on a drive it is usually a bad ** ** idea to continue using it. Then you probably should get ** ** a working hard drive, copy the file system from the bad ** ** drive to the good one -- dd_rescue is a good tool for ** ** that -- and only then run this program. ** ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will rebuild the filesystem (/dev/md1) tree Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes Replaying journal.. Reiserfs journal '/dev/md1' in blocks [18..8211]: 0 transactions replayed ########### reiserfsck --rebuild-tree started at Sat Jun 26 10:12:40 2010 ########### Pass 0: ####### Pass 0 ####### Loading on-disk bitmap .. ok, 195440275 blocks marked used Skipping 15663 blocks (super block, journal, bitmaps) 195424612 blocks will be r ead 0%....20% left 152462527, 1356 block 82149377: The number of items (14) is incorrect, should be (10) - correcte d block 82149377: The free space (52) is incorrect, should be (892) - corrected block 130148042: The number of items (19) is incorrect, should be (10) - correct ed block 130148042: The free space (0) is incorrect, should be (2832) - corrected pass0: block 132170643, item 2: The file [10964 11430] has the wrong mode (p---- -----), corrected to (----------) verify_directory_item: block 138674177, item 2562 2571 0x28325a00 DIR (3), len 1 376, location 2720 entry count 43, fsck need 1, format old: Found a name "VIDEO_ TS.IFO" for 24-th entry matching to the hash 1548490496. pass0: block 138674177, item 2562 2571 0x28325a00 DIR (3), len 1376, location 27 20 entry count 43, fsck need 1, format old: 2 entries were deleted verify_directory_item: block 153878541, item 3257 3494 0x1 DIR (3), len 784, loc ation 3312 entry count 25, fsck need 1, format old: Found a name "00099.mpls" fo r 23-th entry matching to the hash 1779085952. pass0: block 153878541, item 3257 3494 0x1 DIR (3), len 784, location 3312 entry count 25, fsck need 1, format old: 2 entries were deleted left 0, 13246 /secccc 2 directory entries were hashed with not set hash. 11810 directory entries were hashed with "r5" hash. "r5" hash is selected Flushing..finished Read blocks (but not data blocks) 195424612 Leaves among those 194984 - corrected leaves 891 pointers in indirect items to wrong area 2026 (zeroed) Objectids found 11987 Pass 1 (will try to insert 194984 leaves): ####### Pass 1 ####### Looking for allocable blocks .. finished 0%....20%....40%....60%....80%....100% left 0, 626 /sec Flushing..finished 194984 leaves read 194975 inserted - pointers in indirect items pointing to metadata 5 (zer oed) 9 not inserted non-unique pointers in indirect items (zeroed) 10192 ####### Pass 2 ####### Pass 2: 0%....20%....40%....60%....80%....100% left 0, 18 /sec Flushing..finished Leaves inserted item by item 9 Pass 3 (semantic): ####### Pass 3 ######### /Movies/Blow/VIDEO_TS/VTS_01_1.VOBufile.c 391 are_file_items_correct are_file_items_correct: Position (offset == 743387137) in the middle ofthe file [356 363] was not found. Aborted root@Tower:~#
June 26, 201016 yr Hello, Disk 1 on my system was coming up as read-only in the syslog. I ran reiserfsck --check and it instructed a --rebuild-tree. However rebuild-tree keeps failing at different places on Pass 3. Pass 1 and Pass 2 complete successfully (the initial runs corrected errors) Am am continuing to run --rebuild-tree and it would appear I am getting different errors after which resierfsck aborts. Any advice - do I just keep running again until I possibly get success? Thanks Matt Either you are running out of available memory, or you are running a version of reiserfsck that has a bug that causes the abort: You can get the newer version of reiserfsck from the link in this thread: http://lime-technology.com/forum/index.php?topic=6701.msg64894#msg64894 The other unRAID owner was unable to get the file-system check with a rebuild-tree with only 512 Meg of memory. He upgraded the memory to 2Gig and was able to get reiserfsck to complete without an abort. (we do not know if 1Gig would be enough, he did not try it with that amount) He did download, install, and try the newer version of reiserfsck, but it too ran out of memory and aborted. I have a feeling this will happen with disks with lots of files and larger 2TB drives frequently. How much memory are you running in your server? Joe L.
June 27, 201016 yr Joe, Do you think adding swap space via local temporary swapfile would help? unknown. It is worth a try. To create a swapfile 1. Create a file to hold the "swapfile" dd if=/dev/zero of=/mnt/disk1/unraid_swapfile bs=1M count=2000 The above command will create a 2Gig file on disk1. If disk 1 does not have the space, or, if it is the one with the file system damage, use a different disk in the command. The disk you use must be a working disk, not the one you are attempting to repair. 2. Initialize the swapfile. (modify the next commands to be the correct disk, wherever you created the swapfile) mkswap -L SWAP /mnt/disk1/unraid_swapfile chmod 600 /mnt/disk1/unraid_swapfile 3. Enable the use of the swapfile /sbin/swapon -v /mnt/disk1/unraid_swapfile Now you can try the reiserfsck --rebuild-tree once more Once you are through, and BEFORE you can stop the array to reboot you'll need to discontinue the use of the swap-file. (More accurately, before you can un-mount disk1 and subsequently stop the unRAID array you'll need to disable the swapfile.) To do that you will need to type /sbin/swapoff -a -v When you are done, you can remove the swap-file by typing: rm /mnt/disk1/unraid_swapfile Joe L.
June 27, 201016 yr Author Hi Joe, Thanks for getting back to me. I upgraded reiserfsck as you suggested. I did have 1GB of memory, just added an additional 512MB in the last slot, to bring it up to 1.5GB. I am still failing part way through Pass 3, the first 2 running fine. I feel like I see the finish line with this reiserfsck, but fall just short and it aborts after finding a number of inconsistencies. Is 2GB needed to run this successfully. The disk is 1TB and is pretty much full, hence the issues probably gone unnoticed fro so long (although no syslog issues reported). I notice there is a full partition switch from your other posts - is this an option? Many thanks Matt
June 27, 201016 yr Author Ok - just seen your recent post on swap file - will give this a go Thanks Matt
June 27, 201016 yr Hi Joe, Thanks for getting back to me. I upgraded reiserfsck as you suggested. I did have 1GB of memory, just added an additional 512MB in the last slot, to bring it up to 1.5GB. I am still failing part way through Pass 3, the first 2 running fine. I feel like I see the finish line with this reiserfsck, but fall just short and it aborts after finding a number of inconsistencies. Is 2GB needed to run this successfully. We honestly do not know. Did you run reiserfsck with the --fix-fixable option first? If it suggested the --rebuild-tree, good, if not, you might try it first. The disk is 1TB and is pretty much full, hence the issues probably gone unnoticed fro so long (although no syslog issues reported). I notice there is a full partition switch from your other posts - is this an option? Only if you want to recover any deleted files. In those cases the suers had accidentally deleted their own files and were trying to recover them. Many thanks Matt You are welcome. We do not know if the swap partition will help. we are assuming the server is running out of memory when attempting to fix the file-tree, since that is consistent with the "Abort" you are seeing. Let us know if the swap file does the trick. Joe L.
June 28, 201016 yr Author Ok, so with the swap file in place on a good disk, the following just happened.....(see below) Did you run reiserfsck with the --fix-fixable option first? If it suggested the --rebuild-tree, good, if not, you might try it first. It asked for --rebuild-tree straight away, which is what I went for. As the --rebuld-tree failed I dont have the option of doing this now. So I guess that there there is something very wrong with the data on this disk - is multiple runs normal? I would have thought that it should correct everything first time, block 217677829: The number of items (22) is incorrect, should be (9) - correcte d block 217677829: The free space (20) is incorrect, should be (2136) - corrected left 0, 16618 /secccc 5 directory entries were hashed with not set hash. 11770 directory entries were hashed with "r5" hash. "r5" hash is selected Flushing..finished Read blocks (but not data blocks) 195411637 Leaves among those 194984 - corrected leaves 960 pointers in indirect items to wrong area 2151 (zeroed) Objectids found 11987 Pass 1 (will try to insert 194984 leaves): ####### Pass 1 ####### Looking for allocable blocks .. finished 0%....20%....40%....60%....80%....100% left 0, 633 /sec Flushing..finished 194984 leaves read 194976 inserted - pointers in indirect items pointing to metadata 2 (zer oed) 8 not inserted non-unique pointers in indirect items (zeroed) 10573 ####### Pass 2 ####### Pass 2: 0%....20%....40%....60%....80%....100% left 0, 16 /sec Flushing..finished Leaves inserted item by item 8 Pass 3 (semantic): ####### Pass 3 ######### /Pictures/Oman 2008/DSC00763.JPGvpf-10680: The file [9451 10164] has the wrong b lock count in the StatData (4456) - corrected to (4400) /DSC00535.JPGvpf-10680: The file [9451 9703] has the wrong block count in the St atData (4008) - corrected to (3960) /Movies/Blow/VIDEO_TS/VTS_01_1.VOBvpf-10680: The file [356 363] has the wrong bl ock count in the StatData (2095480) - corrected to (2095160) /VTS_01_2.VOBvpf-10680: The file [356 364] has the wrong block count in the Stat Data (2096104) - corrected to (2095392) /VTS_01_3.VOBvpf-10680: The file [356 365] has the wrong block count in the Stat Data (2095440) - corrected to (2095288) /VTS_01_5.VOBvpf-10680: The file [356 367] has the wrong block count in the Stat Data (365192) - corrected to (364984) /VTS_08_1.VOBvpf-10680: The file [356 396] has the wrong block count in the Stat Data (543120) - corrected to (543064) /VTS_03_1.VOBvpf-10680: The file [356 375] has the wrong block count in the Stat Data (2096976) - corrected to (2096440) /VTS_05_1.VOBvpf-10680: The file [356 384] has the wrong block count in the Stat Data (1648320) - corrected to (1647880) /VTS_06_1.VOBvpf-10680: The file [356 388] has the wrong block count in the Stat Data (255440) - corrected to (255240) /Movies/Push/PUSH/VIDEO_TS/VTS_04_2.VOBvpf-10680: The file [5344 5363] has the w rong block count in the StatData (2096768) - corrected to (2096176) /VTS_04_4.VOBvpf-10680: The file [5344 5365] has the wrong block count in the St atData (2097152) - corrected to (2096832) /VTS_04_5.VOBvpf-10680: The file [5344 5366] has the wrong block count in the St atData (2097096) - corrected to (1802024) /VTS_05_1.VOBvpf-10680: The file [5344 5371] has the wrong block count in the St atData (852264) - corrected to (852208) ... /Gremlins??_??Gremlins??2????The??New??Batch?? (1)/VIDEO_TS/VTS_02_3.VOBv pf-10680: The file [1685 1698] has the wrong block count in the StatData (106964 - corrected to (1069448) ... /Gremlins??_??Gremlins??2????The??New??Batch?? (1)/VIDEO_TS/VTS_02_4.VOBu file.c 391 are_file_items_correct are_file_items_correct: Position (offset == 492503041) in the middle ofthe file [1685 1699] was not found. Aborted root@Tower:~# wget http://carroll.cac.psu.edu/pub/linux/distributions/slackware/ slackware-current/slackware/a/reiserfsprogs-3.6.21-i486-1.txz --2010-06-27 12:38:02-- http://carroll.cac.psu.edu/pub/linux/distributions/slac kware/slackware-current/slackware/a/reiserfsprogs-3.6.21-i486-1.txz Resolving carroll.cac.psu.edu... 128.118.2.96 Connecting to carroll.cac.psu.edu|128.118.2.96|:80... connected. HTTP request sent, awaiting response... 200 OK Length: 213908 (209K) [text/plain] Saving to: `reiserfsprogs-3.6.21-i486-1.txz' 100%[======================================>] 213,908 305K/s in 0.7s 2010-06-27 12:38:04 (305 KB/s) - `reiserfsprogs-3.6.21-i486-1.txz' saved [213908 /213908] root@Tower:~# installpkg reiserfsprogs-3.6.21-i486-1.txz Verifying package reiserfsprogs-3.6.21-i486-1.txz. Installing package reiserfsprogs-3.6.21-i486-1.txz: PACKAGE DESCRIPTION: # reiserfsprogs (Reiserfs filesystem utilities) # # These utilities are used for Reiserfs. Reiserfs is a file system # based on balanced tree algorithms. Learn more at the home site: # http://www.namesys.com # # Reiserfs is the work of Hans Reiser and many others. See the file # /usr/doc/reiserfs*/README for full credits. # Executing install script for reiserfsprogs-3.6.21-i486-1.txz. Package reiserfsprogs-3.6.21-i486-1.txz installed. root@Tower:~# reiserfsck --rebuild-tree /dev/md1 reiserfsck 3.6.21 (2009 www.namesys.com) ************************************************************* ** Do not run the program with --rebuild-tree unless ** ** something is broken and MAKE A BACKUP before using it. ** ** If you have bad sectors on a drive it is usually a bad ** ** idea to continue using it. Then you probably should get ** ** a working hard drive, copy the file system from the bad ** ** drive to the good one -- dd_rescue is a good tool for ** ** that -- and only then run this program. ** ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will rebuild the filesystem (/dev/md1) tree Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes Replaying journal: Done. Reiserfs journal '/dev/md1' in blocks [18..8211]: 0 transactions replayed ########### reiserfsck --rebuild-tree started at Sun Jun 27 12:41:07 2010 ########### Pass 0: ####### Pass 0 ####### Loading on-disk bitmap .. ok, 195414572 blocks marked used Skipping 15663 blocks (super block, journal, bitmaps) 195398909 blocks will be r ead 0%..pass0: block 28409868, item 4118 4316 0x344bac00 DIR (3), len 2656, location 1440 entry count 83, fsck need 1, format old: 2 entries were deleted ..20% left 152431935, 17667 /s pass0: block 94174846, item 2: The file [6084 6086] has the wrong mode (?-w--wx- w-), corrected to (--w--wx-w-) block 107855583: The number of items (12) is incorrect, should be (9) - correcte d block 107855583: The free space (1544) is incorrect, should be (1712) - correcte d block 132170142: The number of items (11) is incorrect, should be (10) - correct ed block 132170142: The free space (168) is incorrect, should be (504) - corrected left 0, 16231 /secccc 2 directory entries were hashed with not set hash. 11768 directory entries were hashed with "r5" hash. "r5" hash is selected Flushing..finished Read blocks (but not data blocks) 195398909 Leaves among those 194983 - corrected leaves 965 pointers in indirect items to wrong area 2151 (zeroed) Objectids found 11987 Pass 1 (will try to insert 194983 leaves): ####### Pass 1 ####### Looking for allocable blocks .. finished 0%....20%....40%....60%....80%....100% left 0, 626 /sec Flushing..finished 194983 leaves read 194975 inserted - pointers in indirect items pointing to metadata 1 (zer oed) 8 not inserted non-unique pointers in indirect items (zeroed) 11148 ####### Pass 2 ####### Pass 2: 0%....20%....40%....60%....80%....100% left 0, 16 /sec Flushing..finished Leaves inserted item by item 8 Pass 3 (semantic): ####### Pass 3 ######### /Pictures/General/MOV00085.MPGvpf-10680: The file [9561 10201] has the wrong blo ck count in the StatData (93152) - corrected to (93112) /Movies/Push/PUSH/VIDEO_TS/VTS_04_5.VOBvpf-10680: The file [5344 5366] has the w rong block count in the StatData (1802024) - corrected to (2096944) ... /Gremlins??_??Gremlins??2????The??New??Batch?? (1)/VIDEO_TS/VTS_02_4.VOBv pf-10680: The file [1685 1699] has the wrong block count in the StatData (209588 - corrected to (2095384) ... /Gremlins??_??Gremlins??2????The??New??Batch?? (1)/VIDEO_TS/VTS_02_5.VOBv pf-10680: The file [1685 1700] has the wrong block count in the StatData (305488 ) - corrected to (2095808) ... /Gremlins??_??Gremlins??2????The??New??Batch?? (1)/VIDEO_TS/VTS_02_6.VOBv pf-10680: The file [1685 1701] has the wrong block count in the StatData (240432 ) - corrected to (2096104) ... /Gremlins??_??Gremlins??2????The??New??Batch?? (1)/VIDEO_TS/VTS_02_7.VOBv pf-10680: The file [1685 1702] has the wrong block count in the StatData (778824 ) - corrected to (778688) ... /Gremlins??_??Gremlins??2????The??New??Batch?? (1)/VIDEO_TS/VTS_03_1.VOBv pf-10680: The file [1685 1706] has the wrong block count in the StatData (999328 ) - corrected to (999256) /Movies/Meet??The??Parents/VIDEO_TS/VTS_01_1.VOBvpf-10680: The file [2571 2578] has the wrong block count in the StatData (32064) - corrected to (31968) /VTS_14_1.VOBvpf-10680: The file [2571 2636] has the wrong block count in the St atData (1014384) - corrected to (1014368) /VTS_03_1.VOBvpf-10680: The file [2571 2586] has the wrong block count in the St atData (2096528) - corrected to (2096072) /VTS_03_2.VOBvpf-10680: The file [2571 2587] has the wrong block count in the St atData (2096808) - corrected to (2096336) /VTS_03_3.VOBvpf-10680: The file [2571 2588] has the wrong block count in the St atData (2096696) - corrected to (2095984) /VTS_03_4.VOBvpf-10680: The file [2571 2589] has the wrong block count in the St atData (2096808) - corrected to (2094720) /VTS_03_5.VOBvpf-10680: The file [2571 2590] has the wrong block count in the St atData (2096808) - corrected to (2096752) /VTS_12_1.VOBvpf-10680: The file [2571 2627] has the wrong block count in the St atData (2096488) - corrected to (2096144) vpf-10650: The directory [2562 2571] has the wrong size in the StatData (2128) - corrected to (2064) /Good Luck Chuck/GOOD_LUCK_CHUCK/VIDEO_TS/VTS_01_1.VOBvpf-10680: The file [3862 3883] has the wrong block count in the StatData (591392) - corrected to (591232) /VTS_02_1.VOBvpf-10680: The file [3876 3887] has the wrong block count in the St atData (663120) - corrected to (662512) /VTS_03_1.VOBvpf-10680: The file [3876 3891] has the wrong block count in the St atData (2096080) - corrected to (2095032) /VTS_03_2.VOBufile.c 391 are_file_items_correct are_file_items_correct: Position (offset == 859164673) in the middle ofthe file [3876 3892] was not found. Aborted root@Tower:~# dd if=/dev/zero of=/mnt/disk2/unraid_swapfile bs=1M count=2000 2000+0 records in 2000+0 records out 2097152000 bytes (2.1 GB) copied, 115.367 s, 18.2 MB/s root@Tower:~# mkswap -L SWAP /mnt/disk2/unraid_swapfile Setting up swapspace version 1, size = 2097147 kB LABEL=SWAP, UUID=356c585f-7181-44f6-9c78-0734ac9bfcaa root@Tower:~# chmod 600 /mnt/disk2/unraid_swapfile root@Tower:~# /sbin/swapon -v /mnt/disk2/unraid_swapfile swapon on /mnt/disk2/unraid_swapfile root@Tower:~# reiserfsck --rebuild-tree /dev/md1 reiserfsck 3.6.21 (2009 www.namesys.com) ************************************************************* ** Do not run the program with --rebuild-tree unless ** ** something is broken and MAKE A BACKUP before using it. ** ** If you have bad sectors on a drive it is usually a bad ** ** idea to continue using it. Then you probably should get ** ** a working hard drive, copy the file system from the bad ** ** drive to the good one -- dd_rescue is a good tool for ** ** that -- and only then run this program. ** ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will rebuild the filesystem (/dev/md1) tree Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes Replaying journal: Done. Reiserfs journal '/dev/md1' in blocks [18..8211]: 0 transactions replayed ########### reiserfsck --rebuild-tree started at Sun Jun 27 19:44:07 2010 ########### Pass 0: ####### Pass 0 ####### Loading on-disk bitmap .. ok, 195401192 blocks marked used Skipping 15663 blocks (super block, journal, bitmaps) 195385529 blocks will be r ead 0%..block 28409908: The number of items (19) is incorrect, should be (15) - corr ected block 28409908: The free space (604) is incorrect, should be (1356) - corrected ..20% left 152423688, 13046 /s pass0: block 65405119, item 6: The file [2521 2544] has the wrong mode (?-w-r-xr -x), corrected to (--w-r-xr-x) pass0: block 194576385, item 2: The file [3196 3202] has the wrong mode (?r--r-x --x), corrected to (-r--r-x--x) pass0: block 198901763, item 5745 5918 0x1 DIR (3), len 104, location 3472 entry count 3, fsck need 1, format old: 1 entries were deleted left 0, 12845 /secccc 1 directory entries were hashed with not set hash. 11767 directory entries were hashed with "r5" hash. "r5" hash is selected Flushing..finished Read blocks (but not data blocks) 195385529 Leaves among those 194983 - corrected leaves 976 pointers in indirect items to wrong area 2185 (zeroed) Objectids found 11987 Pass 1 (will try to insert 194983 leaves): ####### Pass 1 ####### Looking for allocable blocks .. finished 0%....20%....40%....60%...pass1.c 424 pass1_correct_leaf left 51056, 615 /sec pass1_correct_leaf: block 187728997, item 0, pointer 57: The wrong pointer (2147 487744) in the file [3760 3764]. Must be fixed on pass0. Aborted root@Tower:~#
June 28, 201016 yr I do not have any better suggestions. Contact limetech on this one, maybe he has more experience with issues like this. Check the smartlogs of this drive. Perhaps run a memtest if you do reboot at any point. Perhaps borrow more memory temporarily from another machine.
June 28, 201016 yr Author Thanks Weebotech, I appreciate yours and Joe L's help on this. I will email this thread to limetech, to see if he has any other suggestions. I suppose I always have the options to format the disk and build from parity, although it would seem that this is a bit drastic as I can clearly see from the reiserfsck that files still exist on this disk, albeit stuck in a --rebuild-tree aborted no mans land. The following is the smart status for this disk: Statistics for /dev/sde SAMSUNG_HD103UJ_S13PJ1LS704586 smartctl -a -d ata /dev/sde smartctl version 5.38 [i486-slackware-linux-gnu] Copyright (C) 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/ === START OF INFORMATION SECTION === Device Model: SAMSUNG HD103UJ Serial Number: S13PJ1LS704586 Firmware Version: 1AA01118 User Capacity: 1,000,204,886,016 bytes Device is: In smartctl database [for details use: -P show] ATA Version is: 8 ATA Standard is: ATA-8-ACS revision 3b Local Time is: Mon Jun 28 07:11:04 2010 GMT ==> WARNING: May need -F samsung or -F samsung2 enabled; see manual for details. SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (13533) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 226) minutes. Conveyance self-test routine recommended polling time: ( 24) minutes. SCT capabilities: (0x003f) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 100 100 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0007 067 067 011 Pre-fail Always - 10560 4 Start_Stop_Count 0x0032 099 099 000 Old_age Always - 845 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 100 100 051 Pre-fail Always - 0 8 Seek_Time_Performance 0x0025 100 100 015 Pre-fail Offline - 10421 9 Power_On_Hours 0x0032 099 099 000 Old_age Always - 6957 10 Spin_Retry_Count 0x0033 100 100 051 Pre-fail Always - 0 11 Calibration_Retry_Count 0x0012 100 100 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 170 13 Read_Soft_Error_Rate 0x000e 100 100 000 Old_age Always - 0 183 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 184 Unknown_Attribute 0x0033 100 100 000 Pre-fail Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 073 045 000 Old_age Always - 27 (Lifetime Min/Max 22/31) 194 Temperature_Celsius 0x0022 072 044 000 Old_age Always - 28 (0 21 33 22) 195 Hardware_ECC_Recovered 0x001a 100 100 000 Old_age Always - 38220686 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 0 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 100 100 000 Old_age Always - 281 200 Multi_Zone_Error_Rate 0x000a 100 100 000 Old_age Always - 0 201 Soft_Read_Error_Rate 0x000a 100 100 000 Old_age Always - 0 SMART Error Log Version: 1 ATA Error Count: 14 (device log contains only the most recent five errors) CR = Command Register [HEX] FR = Features Register [HEX] SC = Sector Count Register [HEX] SN = Sector Number Register [HEX] CL = Cylinder Low Register [HEX] CH = Cylinder High Register [HEX] DH = Device/Head Register [HEX] DC = Device Command Register [HEX] ER = Error register [HEX] ST = Status register [HEX] Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days. Error 14 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 51 2f 31 00 00 e0 Error: ICRC, ABRT 47 sectors at LBA = 0x00000031 = 49 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 40 20 00 00 e0 00 16:34:48.900 READ DMA c8 00 20 00 00 00 e0 00 16:34:48.870 READ DMA c8 00 08 00 00 00 e0 00 16:34:46.250 READ DMA e0 00 00 00 00 00 a0 00 16:34:45.220 STANDBY IMMEDIATE ec 00 00 00 00 00 a0 02 16:34:45.200 IDENTIFY DEVICE Error 13 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 51 bf 18 83 de ee Error: ICRC, ABRT 191 sectors at LBA = 0x0ede8318 = 249463576 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 00 d7 82 de ee 00 02:28:43.880 READ DMA c8 00 00 d7 75 de ee 00 02:28:43.780 READ DMA c8 00 00 d7 63 de ee 00 02:28:43.700 READ DMA c8 00 00 d7 5a de ee 00 02:28:43.660 READ DMA c8 00 f0 d7 52 de ee 00 02:28:43.600 READ DMA Error 12 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 51 8f 08 bd eb ed Error: ICRC, ABRT 143 sectors at LBA = 0x0debbd08 = 233553160 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 a0 f7 bc eb ed 00 02:19:20.650 READ DMA c8 00 60 97 b8 eb ed 00 02:19:20.610 READ DMA c8 00 a0 f7 b3 eb ed 00 02:19:20.600 READ DMA c8 00 60 97 af eb ed 00 02:19:20.560 READ DMA c8 00 a0 f7 aa eb ed 00 02:19:20.550 READ DMA Error 11 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 51 9f 88 99 86 ec Error: ICRC, ABRT 159 sectors at LBA = 0x0c869988 = 210147720 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 00 27 99 86 ec 00 02:08:20.870 READ DMA c8 00 00 27 90 86 ec 00 02:08:20.800 READ DMA c8 00 00 27 87 86 ec 00 02:08:20.770 READ DMA c8 00 00 27 7e 86 ec 00 02:08:20.720 READ DMA c8 00 00 27 75 86 ec 00 02:08:20.680 READ DMA Error 10 occurred at disk power-on lifetime: 0 hours (0 days + 0 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 51 2f a0 69 f8 e9 Error: ICRC, ABRT 47 sectors at LBA = 0x09f869a0 = 167274912 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 50 7f 69 f8 e9 00 01:42:39.950 READ DMA c8 00 b0 cf 64 f8 e9 00 01:42:39.920 READ DMA c8 00 50 7f 60 f8 e9 00 01:42:39.910 READ DMA c8 00 00 7f 49 f8 e9 00 01:42:39.780 READ DMA c8 00 00 7f 40 f8 e9 00 01:42:39.740 READ DMA SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 6956 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.
June 28, 201016 yr I haven't heard of reiserfsck requiring multiple --rebuild-tree passes. Rebuilding from parity won't do you any good, it will just rebuild the same corrupted file system! Do you have any idea how it got corrupted in the first place? Disk errors? There are some known issues with reiserfsck, for example, if you have a "loopback" file, which itself is formatted with reiserfs, on the disk you are running reiserfsck on, then reiserfsck will be very confused when it hits the loopback file. This is probably not your case. If you really really need the data, I suggest you go to 2GB RAM or more.
June 28, 201016 yr Can you check you syslog to see if there are out-of-memory errors at the time the reiserfsck aborts? If there are, and the kernel out-of-memory process is killing reiserfsck, then we can try to minimize the use of the disk cache hogging the RAM. Did you try setting the cache_pressure to 200? sysctl vm.vfs_cache_pressure=200 Did you clear the existing buffer cache to free up as much memory as possible? sync echo 3 > /proc/sys/vm/drop_caches If so, I suggest also trying: 1. Disable user shares temporally. (They use up memory which is in short supply.) You can re-enable them once the disk is repaired. 2. Creating a swapfile in memory, bypassing the disk cache entirely. I know this sounds strange, but if we can keep the out-of-memory kernel process from killing the reiserfsck, it might just finish. This will reserve some amount of memory just for "swapping" This will reserve 100Meg of your ram as a file-system to hold the "swapfile" echo 10 >/proc/sys/vm/swappiness sysctl vm.vfs_cache_pressure=200 sync echo 3 > /proc/sys/vm/drop_caches mkdir /mnt/swap_fs mount -t ramfs -o size=100m ramfs /mnt/swap_fs dd if=/dev/zero of=/mnt/swap_fs/unraid_swapfile bs=1M count=100 The above commands will create a 100 Meg file on the RAM file system in memory. This space is "reserved" and will not be used for disk cache, and will not be "swapped" out. Next, Initialize the swapfile. mkswap -L SWAP /mnt/swap_fs/unraid_swapfile chmod 600 /mnt/swap_fs/unraid_swapfile Then, Enable the use of the swapfile /sbin/swapon -v /mnt/swap_fs/unraid_swapfile To free up some more RAM, stop SAMBA and even the unRAID web-interface. killall emhttp /root/samba stop Then, try the reiserfsck once more... To re-start the web-interface so we can stop the array and reboot, type: cd / nohup /usr/local/sbin/emhttp & Good luck. Joe L.
June 28, 201016 yr Author Do you have any idea how it got corrupted in the first place? Disk errors? Not really the disk was pretty full and so most new stuff has been going to disks 3 and 4, which have space. Although I do have MySQL running on this drive and untorrent. It was when I tried to delete some files I got read-only filesystem in the syslog. Before this it had been clear in syslog - no reported errors. If you really really need the data, I suggest you go to 2GB RAM or more. Just been out and bought 2GB so this will take me up to 2.5GB I am currently running Joe Ls in memory swap file suggestion. If this fails I will power down and plug in extra ram. Again thanks for staying with me on this one.
June 28, 201016 yr I'm not so keen on the in memory swap. I would try the swap on disk and just set echo 100 >/proc/sys/vm/swappiness See http://unixfoo.blogspot.com/2007/11/linux-performance-tuning.html But by all means, do what ever helps with this situation. If I would have known you were purchasing ram I would have suggested going right to 4GB. Please do a memtest also. 1 pass should be good. Have you looked at the syslog to see if the reiserfsck was being killed by an OOM condition? The aborted may be an internal issue. If it is still possible to mount the filesystem readonly, I would try that and grab the most important files off it, but this may not be possible. Keep us updated.
June 28, 201016 yr I'm not so keen on the in memory swap. I would try the swap on disk and just set... I was just trying to eliminate the use of the disk buffer cache as the swapfile was being accessed. It is a way to eliminate its use as disk-cache, and ensure it can be used to for data page swapping if needed. Joe L. It's all just things to try at this point..., especially since 512 Meg is still what many of us (myself included) are running in our servers. If there is a solution with just a few commands to get the reiserfsck to be able to complete, it is worth a try.
June 28, 201016 yr Although I do have MySQL running on this drive and untorrent. Make sure you terminate those extra processes when you do the reiserfsck, they are using up RAM you probably need right now. To be sure, the extra RAM you purchased sure won't hurt if you plan on running them on an ongoing basis.
June 28, 201016 yr I'm not so keen on the in memory swap. I would try the swap on disk and just set... I was just trying to eliminate the use of the disk buffer cache as the swapfile was being accessed. It is a way to eliminate its use as disk-cache, and ensure it can be used to for data page swapping if needed. The swappiness value is what is needed here. There are certain pages that would never get swapped out. Something is weird in unRAID's tuning. Even with 2G of swap my rsync's were failing with OOM condition. Dropping the cache frequently helped with this. grep Cached /proc/meminfo Cached: 3571940 kB echo 3 > /proc/sys/vm/drop_caches grep Cached /proc/meminfo Cached: 177948 kB So monitoring that in another session and dropping it when it gets full will surely free up ram. But I dunno if it's the application starved for ram (reiserfsck) or kernel space causing an OOM condition (if that is still the case).
June 28, 201016 yr The swappiness value is what is needed here. And that is why in the previous post I added a line: echo 10 >/proc/sys/vm/swappiness Just in case it might help... Joe L.
June 28, 201016 yr Needs to be 100 for maximum swappiness. Perhaps some monitor that if the cached level exceeds a certain threshold it is dropped. I dunno this is an odd case.
June 28, 201016 yr Needs to be 100 for maximum swappiness. From my reading, the closer to 0, the more likely to free a disk and inode buffer cache to be used when memory is requested, and the least likely to use the "swapfile" to swap out running processes. The closer to 100, the more likely to swap out processes to the swapfile, and less likely to free disk and inode buffer cache. Since reiserfsck is likely to be going through lots of disk blocks, and likely to not need them in the cache, I elected to leave as much RAM as possible for processes, so the out-of-memory kernel process does not get involved. So, I added a swapfile (in RAM, to specifically reserve it if needed) and also asked to free disk buffers more readily by choosing swappiness=10. Joe L.
June 28, 201016 yr Perhaps i have this wrong, I would think maximum swappiness would be helpful as the kernel will page out whatever processes that are not running and do not need ram at that moment. The kernel has a choice when ram is needed. Kill it, free some ram from a cache, swap it out. Putting swap on a tmpfs does not make sense to me. The kernel will adjust ram usage as it needs it if the tunables are where they need to be. swapiness at 100. It makes your computer run more efficiently, but at a sacrifice of possibly having latency in your desktop applications. Others have resorted to pushing this to 0 so the kernel never swaps applications out of memory so there is little latency in recovering desktop applications from swap. Just my guess: Desktop users will find that a lower number is more satisfactory. Servers admins will find that a higher number makes the server more memory efficient because it will swap out all the stuff that runs once during bootup and is never used again. I would think maximizing ram that could be used by processes and reiserfsck is what is wanted. Even if they need to swap to disk for the virtual ram. vm.vfs_cache_pressure=N : Controls the tendency of the kernel to reclaim the memory which is used for caching of directory and inode objects. At the default value of vfs_cache_pressure=100 the kernel will attempt to reclaim dentries and inodes at a "fair" rate with respect to pagecache and swapcache reclaim. Decreasing vfs_cache_pressure causes the kernel to prefer to retain dentry and inode caches. Increasing vfs_cache_pressure beyond 100 causes the kernel to prefer to reclaim dentries and inodes. I noticed that by changing swappiness to 100 and cache_pressure to a higher value, my system no longer has OOM and the rsync's run without crashing (Even multiple massive rsyncs).
June 28, 201016 yr The following script does not solve the problem, but it provides a nifty visual tool that "may" provide a little more interactive control to dropping the cache and freeing memory in a pinch. Run this in another session. If you start to see the Free number get too low and the cached number get too high, press one of the choices 1 2 3 to drop the cache. I built this since I'm going to have to track down another OOM condition with my locate/updatedb issue. I'm sure Joe can think up a good way of automating this with a mathematical algorithm. In the meantime it provides an interesting way to use the shell, monitor the cache and flush it in a pinch. It was a fun shell to build and play with. It may or may not help. cache_monitor.sh #!/bin/bash [ ${DEBUG:=0} -gt 0 ] && set -x -v declare -a FIELDS export FIELDS export LINE="" while true do LINE="" while read -a FIELDS do case "${FIELDS[0]}" in Cached: ) LINE="${LINE} ${FIELDS[0]} ${FIELDS[1]}";; MemTotal: ) LINE="${LINE} ${FIELDS[0]} ${FIELDS[1]}";; MemFree: ) LINE="${LINE} ${FIELDS[0]} ${FIELDS[1]}";; esac done < /proc/meminfo echo -e "`date +%T` ${LINE}" read -t3 -n1 -p"[Q]uit, [1]pagecache, [2]dentries and inodes, [3]all: " echo -e "\r\c" case "${REPLY}" in 3|2|1 ) echo ${REPLY} > /proc/sys/vm/drop_caches;; Q*|q*|E*|e* ) echo;exit;; " "|"" ) ;; * ) echo -e "\rInvalid Keystroke \r";; esac done
June 28, 201016 yr Author Some progress. After upgrading to 2 GB memory and shutting everything off, I did not get an abort, just a "Segmentation Fault", which I have been on these forums for before, last time a reboot and re-run of --rebuild-tree sorted this, it was suggested memeory to be at fault Since last time I have swapped all memory for new Kingston, with successful memtest runs. Currently running --rebuild-tree again.
June 28, 201016 yr Author Back to an abort on Pass 3 again. Upgraded to latest reiserfsck, giving it another run overnight. Starting to think that this 1TB is gone. Any advice on how to mount read only to grab some data off this drive?
June 28, 201016 yr Author Apologies guys just seen your recent comments and suggested script to try I will look to try this and stop re-running on the basis that I now have 2GB, as it seems it is still not enough for reiserfsck. I will let you know how I get on.
Archived
This topic is now archived and is closed to further replies.