Chris Posted January 8, 2011 Share Posted January 8, 2011 4.5.6 before, 4.6 AIO after, same results: I noticed I cannot delete several folders using my Win7 box, so I SSH'd into the server as root and was still unable to rm the directories or files within them. I wondered if there was a permission issue, and look what the permissions/ownership is set to: root@Media:/mnt/disk3/Music/Sade# ls -al /bin/ls: cannot access Flow #2.mp3: Permission denied total 0 drwx--x--x 2 root root 80 Jan 6 15:23 ./ drwx------ 7 root root 192 Jan 7 08:37 ../ ???? ? ? ? ? ? Flow\ #2.mp3 root@Media:/mnt/disk3/Music/Sade# rm -f Flow\ #2.mp3 rm: cannot remove `Flow #2.mp3': Permission denied I can't chmod or chown them, and there are no errors in the log. This has me baffled. Quote Link to comment
Chuck Posted January 8, 2011 Share Posted January 8, 2011 Just a guess, but it looks like file system corruption. I would try taking the array offline and online, assuming it will run fsck or similar (I know nothing about the reiserfs). Quote Link to comment
Chris Posted January 8, 2011 Author Share Posted January 8, 2011 I'm worried you're right. I see that the Music share (which I've been cleaning out - deleting) is now limited to /mnt/disk3 out of 4 disks. I suspect that drive has issues. I now see this in the logs - ugh: Jan 7 19:05:21 Media kernel: REISERFS warning: green-16003 errcatch_is_left_mergeable: Invalid item type observed, run fsck ASAP Jan 7 19:05:21 Media last message repeated 5 times Jan 7 19:05:21 Media kernel: REISERFS warning (device md3): vs-5355 reiserfs_delete_solid_item: [3631 3632 0x0 SD] not found Jan 7 19:05:21 Media kernel: REISERFS warning: green-16003 errcatch_is_left_mergeable: Invalid item type observed, run fsck ASAP Jan 7 19:05:21 Media kernel: REISERFS warning: green-16003 errcatch_is_left_mergeable: Invalid item type observed, run fsck ASAP Jan 7 19:05:21 Media kernel: BUG: unable to handle kernel NULL pointer dereference at 0000000f Jan 7 19:05:21 Media kernel: IP: [] prepare_error_buf+0x68/0x421 Jan 7 19:05:21 Media kernel: *pdpt = 000000000287c001 *pde = 0000000000000000 Jan 7 19:05:21 Media kernel: Oops: 0000 [#1] SMP Jan 7 19:05:21 Media kernel: last sysfs file: /sys/devices/pci0000:00/0000:00:14.1/ide1/1.0/block/hdc/stat Jan 7 19:05:21 Media kernel: Modules linked in: md_mod xor ide_gd_mod sata_sil atiixp ahci r8169 Jan 7 19:05:21 Media kernel: Jan 7 19:05:21 Media kernel: Pid: 1767, comm: mc Not tainted (2.6.32.9-unRAID # GA-MA74GM-S2 Jan 7 19:05:21 Media kernel: EIP: 0060:[] EFLAGS: 00210286 CPU: 1 Jan 7 19:05:21 Media kernel: EIP is at prepare_error_buf+0x68/0x421 Jan 7 19:05:21 Media kernel: EAX: c14726a6 EBX: c290be24 ECX: 00000000 EDX: c14726a6 Jan 7 19:05:21 Media kernel: ESI: ffffffff EDI: c290be20 EBP: c290bdf0 ESP: c290bd80 Jan 7 19:05:21 Media kernel: DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 Jan 7 19:05:21 Media kernel: Process mc (pid: 1767, ti=c290a000 task=f762ecb0 task.ti=c290a000) Jan 7 19:05:21 Media kernel: Stack: Jan 7 19:05:21 Media kernel: c290be20 00000f4c c290bdc4 c1472aa6 c1472688 f7714c00 c24663cc c290be10 Jan 7 19:05:21 Media kernel: <0> c10bb947 00000001 c290be50 c290be94 00001000 c10b4f41 c290be24 f7714c00 Jan 7 19:05:21 Media kernel: <0> c14726a6 00000000 c290bef0 f82e1000 00000000 00000000 00000000 00000000 Jan 7 19:05:21 Media kernel: Call Trace: Jan 7 19:05:21 Media kernel: [] ? search_for_position_by_key+0x37/0x27f Jan 7 19:05:21 Media kernel: [] ? add_save_link+0x16f/0x1b6 Jan 7 19:05:21 Media kernel: [] ? __reiserfs_error+0x20/0xac Jan 7 19:05:21 Media kernel: [] ? reiserfs_do_truncate+0x17a/0x487 Jan 7 19:05:21 Media kernel: [] ? reiserfs_for_each_xattr+0x51/0x1ca Jan 7 19:05:21 Media kernel: [] ? reiserfs_delete_object+0x2b/0x5c Jan 7 19:05:21 Media kernel: [] ? reiserfs_delete_inode+0x67/0xae Jan 7 19:05:21 Media kernel: [] ? reiserfs_delete_inode+0x0/0xae Jan 7 19:05:21 Media kernel: [] ? generic_delete_inode+0x75/0xdb Jan 7 19:05:21 Media kernel: [] ? generic_drop_inode+0x11/0x4c Jan 7 19:05:21 Media kernel: [] ? iput+0x4b/0x4e Jan 7 19:05:21 Media kernel: [] ? do_unlinkat+0xb4/0xf9 Jan 7 19:05:21 Media kernel: [] ? sys_unlink+0x10/0x12 Jan 7 19:05:21 Media kernel: [] ? syscall_call+0x7/0xb Jan 7 19:05:21 Media kernel: Code: 90 8b 45 d0 8b 5d 90 8b 11 83 c3 04 e8 8b fc ff ff 89 5d 90 e9 a6 02 00 00 8b 5d 90 8b 7d 90 83 c3 04 89 5d c8 8b 37 85 f6 74 6a <8a> 46 10 bb 53 f3 37 c1 84 c0 74 20 3c 03 0f 84 58 03 00 00 3c Jan 7 19:05:21 Media kernel: EIP: [] prepare_error_buf+0x68/0x421 SS:ESP 0068:c290bd80 Jan 7 19:05:21 Media kernel: CR2: 000000000000000f Jan 7 19:05:21 Media kernel: ---[ end trace 65ec803ea86daeff ]--- Jan 7 19:05:21 Media kernel: ------------[ cut here ]------------ Jan 7 19:05:21 Media kernel: WARNING: at kernel/exit.c:895 do_exit+0x2b/0x508() Jan 7 19:05:21 Media kernel: Hardware name: GA-MA74GM-S2 Jan 7 19:05:21 Media kernel: Modules linked in: md_mod xor ide_gd_mod sata_sil atiixp ahci r8169 Jan 7 19:05:21 Media kernel: Pid: 1767, comm: mc Tainted: G D 2.6.32.9-unRAID #8 Jan 7 19:05:21 Media kernel: Call Trace: Jan 7 19:05:21 Media kernel: [] warn_slowpath_common+0x60/0x77 Jan 7 19:05:21 Media kernel: [] warn_slowpath_null+0xd/0x10 Jan 7 19:05:21 Media kernel: [] do_exit+0x2b/0x508 Jan 7 19:05:21 Media kernel: [] ? print_oops_end_marker+0x1e/0x23 Jan 7 19:05:21 Media kernel: [] oops_end+0x75/0x7c Jan 7 19:05:21 Media kernel: [] no_context+0x14b/0x155 Jan 7 19:05:21 Media kernel: [] __bad_area_nosemaphore+0xe0/0xe8 Jan 7 19:05:21 Media kernel: [] __bad_area+0x2e/0x37 Jan 7 19:05:21 Media kernel: [] bad_area+0xd/0x10 Jan 7 19:05:21 Media kernel: [] do_page_fault+0x135/0x1e4 Jan 7 19:05:21 Media kernel: [] ? do_page_fault+0x0/0x1e4 Jan 7 19:05:21 Media kernel: [] error_code+0x66/0x6c Jan 7 19:05:21 Media kernel: [] ? do_page_fault+0x0/0x1e4 Jan 7 19:05:21 Media kernel: [] ? prepare_error_buf+0x68/0x421 Jan 7 19:05:21 Media kernel: [] ? search_for_position_by_key+0x37/0x27f Jan 7 19:05:21 Media kernel: [] ? add_save_link+0x16f/0x1b6 Jan 7 19:05:21 Media kernel: [] __reiserfs_error+0x20/0xac Jan 7 19:05:21 Media kernel: [] reiserfs_do_truncate+0x17a/0x487 Jan 7 19:05:21 Media kernel: [] ? reiserfs_for_each_xattr+0x51/0x1ca Jan 7 19:05:21 Media kernel: [] reiserfs_delete_object+0x2b/0x5c Jan 7 19:05:21 Media kernel: [] reiserfs_delete_inode+0x67/0xae Jan 7 19:05:21 Media kernel: [] ? reiserfs_delete_inode+0x0/0xae Jan 7 19:05:21 Media kernel: [] generic_delete_inode+0x75/0xdb Jan 7 19:05:21 Media kernel: [] generic_drop_inode+0x11/0x4c Jan 7 19:05:21 Media kernel: [] iput+0x4b/0x4e Jan 7 19:05:21 Media kernel: [] do_unlinkat+0xb4/0xf9 Jan 7 19:05:21 Media kernel: [] sys_unlink+0x10/0x12 Jan 7 19:05:21 Media kernel: [] syscall_call+0x7/0xb Jan 7 19:05:21 Media kernel: ---[ end trace 65ec803ea86daf00 ]--- Quote Link to comment
Joe L. Posted January 8, 2011 Share Posted January 8, 2011 I'm worried you're right. I see that the Music share (which I've been cleaning out - deleting) is now limited to /mnt/disk3 out of 4 disks. I suspect that drive has issues. I now see this in the logs - ugh: Jan 7 19:05:21 Media kernel: REISERFS warning: green-16003 errcatch_is_left_mergeable: Invalid item type observed, run fsck ASAP Jan 7 19:05:21 Media last message repeated 5 times Jan 7 19:05:21 Media kernel: REISERFS warning (device md3): vs-5355 reiserfs_delete_solid_item: [3631 3632 0x0 SD] not found Jan 7 19:05:21 Media kernel: REISERFS warning: green-16003 errcatch_is_left_mergeable: Invalid item type observed, run fsck ASAP Jan 7 19:05:21 Media kernel: REISERFS warning: green-16003 errcatch_is_left_mergeable: Invalid item type observed, run fsck ASAP Jan 7 19:05:21 Media kernel: BUG: unable to handle kernel NULL pointer dereference at 0000000f Jan 7 19:05:21 Media kernel: IP: [] prepare_error_buf+0x68/0x421 Jan 7 19:05:21 Media kernel: *pdpt = 000000000287c001 *pde = 0000000000000000 Jan 7 19:05:21 Media kernel: Oops: 0000 [#1] SMP Jan 7 19:05:21 Media kernel: last sysfs file: /sys/devices/pci0000:00/0000:00:14.1/ide1/1.0/block/hdc/stat Jan 7 19:05:21 Media kernel: Modules linked in: md_mod xor ide_gd_mod sata_sil atiixp ahci r8169 Jan 7 19:05:21 Media kernel: Jan 7 19:05:21 Media kernel: Pid: 1767, comm: mc Not tainted (2.6.32.9-unRAID # GA-MA74GM-S2 Jan 7 19:05:21 Media kernel: EIP: 0060:[] EFLAGS: 00210286 CPU: 1 Jan 7 19:05:21 Media kernel: EIP is at prepare_error_buf+0x68/0x421 Jan 7 19:05:21 Media kernel: EAX: c14726a6 EBX: c290be24 ECX: 00000000 EDX: c14726a6 Jan 7 19:05:21 Media kernel: ESI: ffffffff EDI: c290be20 EBP: c290bdf0 ESP: c290bd80 Jan 7 19:05:21 Media kernel: DS: 007b ES: 007b FS: 00d8 GS: 0033 SS: 0068 Jan 7 19:05:21 Media kernel: Process mc (pid: 1767, ti=c290a000 task=f762ecb0 task.ti=c290a000) Jan 7 19:05:21 Media kernel: Stack: Jan 7 19:05:21 Media kernel: c290be20 00000f4c c290bdc4 c1472aa6 c1472688 f7714c00 c24663cc c290be10 Jan 7 19:05:21 Media kernel: <0> c10bb947 00000001 c290be50 c290be94 00001000 c10b4f41 c290be24 f7714c00 Jan 7 19:05:21 Media kernel: <0> c14726a6 00000000 c290bef0 f82e1000 00000000 00000000 00000000 00000000 Jan 7 19:05:21 Media kernel: Call Trace: Jan 7 19:05:21 Media kernel: [] ? search_for_position_by_key+0x37/0x27f Jan 7 19:05:21 Media kernel: [] ? add_save_link+0x16f/0x1b6 Jan 7 19:05:21 Media kernel: [] ? __reiserfs_error+0x20/0xac Jan 7 19:05:21 Media kernel: [] ? reiserfs_do_truncate+0x17a/0x487 Jan 7 19:05:21 Media kernel: [] ? reiserfs_for_each_xattr+0x51/0x1ca Jan 7 19:05:21 Media kernel: [] ? reiserfs_delete_object+0x2b/0x5c Jan 7 19:05:21 Media kernel: [] ? reiserfs_delete_inode+0x67/0xae Jan 7 19:05:21 Media kernel: [] ? reiserfs_delete_inode+0x0/0xae Jan 7 19:05:21 Media kernel: [] ? generic_delete_inode+0x75/0xdb Jan 7 19:05:21 Media kernel: [] ? generic_drop_inode+0x11/0x4c Jan 7 19:05:21 Media kernel: [] ? iput+0x4b/0x4e Jan 7 19:05:21 Media kernel: [] ? do_unlinkat+0xb4/0xf9 Jan 7 19:05:21 Media kernel: [] ? sys_unlink+0x10/0x12 Jan 7 19:05:21 Media kernel: [] ? syscall_call+0x7/0xb Jan 7 19:05:21 Media kernel: Code: 90 8b 45 d0 8b 5d 90 8b 11 83 c3 04 e8 8b fc ff ff 89 5d 90 e9 a6 02 00 00 8b 5d 90 8b 7d 90 83 c3 04 89 5d c8 8b 37 85 f6 74 6a <8a> 46 10 bb 53 f3 37 c1 84 c0 74 20 3c 03 0f 84 58 03 00 00 3c Jan 7 19:05:21 Media kernel: EIP: [] prepare_error_buf+0x68/0x421 SS:ESP 0068:c290bd80 Jan 7 19:05:21 Media kernel: CR2: 000000000000000f Jan 7 19:05:21 Media kernel: ---[ end trace 65ec803ea86daeff ]--- Jan 7 19:05:21 Media kernel: ------------[ cut here ]------------ Jan 7 19:05:21 Media kernel: WARNING: at kernel/exit.c:895 do_exit+0x2b/0x508() Jan 7 19:05:21 Media kernel: Hardware name: GA-MA74GM-S2 Jan 7 19:05:21 Media kernel: Modules linked in: md_mod xor ide_gd_mod sata_sil atiixp ahci r8169 Jan 7 19:05:21 Media kernel: Pid: 1767, comm: mc Tainted: G D 2.6.32.9-unRAID #8 Jan 7 19:05:21 Media kernel: Call Trace: Jan 7 19:05:21 Media kernel: [] warn_slowpath_common+0x60/0x77 Jan 7 19:05:21 Media kernel: [] warn_slowpath_null+0xd/0x10 Jan 7 19:05:21 Media kernel: [] do_exit+0x2b/0x508 Jan 7 19:05:21 Media kernel: [] ? print_oops_end_marker+0x1e/0x23 Jan 7 19:05:21 Media kernel: [] oops_end+0x75/0x7c Jan 7 19:05:21 Media kernel: [] no_context+0x14b/0x155 Jan 7 19:05:21 Media kernel: [] __bad_area_nosemaphore+0xe0/0xe8 Jan 7 19:05:21 Media kernel: [] __bad_area+0x2e/0x37 Jan 7 19:05:21 Media kernel: [] bad_area+0xd/0x10 Jan 7 19:05:21 Media kernel: [] do_page_fault+0x135/0x1e4 Jan 7 19:05:21 Media kernel: [] ? do_page_fault+0x0/0x1e4 Jan 7 19:05:21 Media kernel: [] error_code+0x66/0x6c Jan 7 19:05:21 Media kernel: [] ? do_page_fault+0x0/0x1e4 Jan 7 19:05:21 Media kernel: [] ? prepare_error_buf+0x68/0x421 Jan 7 19:05:21 Media kernel: [] ? search_for_position_by_key+0x37/0x27f Jan 7 19:05:21 Media kernel: [] ? add_save_link+0x16f/0x1b6 Jan 7 19:05:21 Media kernel: [] __reiserfs_error+0x20/0xac Jan 7 19:05:21 Media kernel: [] reiserfs_do_truncate+0x17a/0x487 Jan 7 19:05:21 Media kernel: [] ? reiserfs_for_each_xattr+0x51/0x1ca Jan 7 19:05:21 Media kernel: [] reiserfs_delete_object+0x2b/0x5c Jan 7 19:05:21 Media kernel: [] reiserfs_delete_inode+0x67/0xae Jan 7 19:05:21 Media kernel: [] ? reiserfs_delete_inode+0x0/0xae Jan 7 19:05:21 Media kernel: [] generic_delete_inode+0x75/0xdb Jan 7 19:05:21 Media kernel: [] generic_drop_inode+0x11/0x4c Jan 7 19:05:21 Media kernel: [] iput+0x4b/0x4e Jan 7 19:05:21 Media kernel: [] do_unlinkat+0xb4/0xf9 Jan 7 19:05:21 Media kernel: [] sys_unlink+0x10/0x12 Jan 7 19:05:21 Media kernel: [] syscall_call+0x7/0xb Jan 7 19:05:21 Media kernel: ---[ end trace 65ec803ea86daf00 ]--- You will need to follow the steps outlined here in the wiki: http://lime-technology.com/wiki/index.php?title=Check_Disk_Filesystems Joe L. Quote Link to comment
Chris Posted January 8, 2011 Author Share Posted January 8, 2011 Thanks for that link, Joe. If that fails, can't I simply pull that drive out and replace it with a new one, allowing it to rebuild? Chris Quote Link to comment
Joe L. Posted January 8, 2011 Share Posted January 8, 2011 Thanks for that link, Joe. If that fails, can't I simply pull that drive out and replace it with a new one, allowing it to rebuild? Chris Unfortunately, no, you cannot. The drive is probably fine, the data on it, AND on the simulated drive that would be used reconstructing from parity and other disks are both corrupted on the exact same way. You'll need to repair the file system while the array is online as it instructs in the wiki. That procedure will repair the file system and also keep parity in sync so it too is repaired. Joe L. Quote Link to comment
Chris Posted January 9, 2011 Author Share Posted January 9, 2011 Joe, that did the trick, at least to get the array back without undeletable files. I have a lost+found folder with 33 files in it which I'll sort through to attempt to figure out what they are and name them correctly. They are all mp3 songs which play, and appear to at least have tags. Thanks so much for pointing me in the right direction! Chris parity device: pci-0000:00:11.0-scsi-1:0:0:0 host2 (sdb) ########### reiserfsck --check started at Fri Jan 7 21:51:08 2011 ########### Replaying journal: Done. Reiserfs journal '/dev/md3' in blocks [18..8211]: 0 transactions replayed Checking internal tree.. \/ 18 (of 18// 23 (of 137|bad_internal: vpf-10320: block 88279193, items 35 and 36: The wrong order of items: [41131063 41131064 0x2739c39 (15)], [2781 2783 0x83001 IND (1)] the problem in the internal node occured (88279193), whole subtree is skipped finished Comparing bitmaps..vpf-10640: The on-disk and the correct bitmaps differs. Bad nodes were found, Semantic pass skipped 1 found corruptions can be fixed only when running with --rebuild-tree ########### reiserfsck finished at Fri Jan 7 23:20:10 2011 ########### ########### reiserfsck --rebuild-tree started at Sat Jan 8 06:54:39 2011 ########### Pass 0: ####### Pass 0 ####### Loading on-disk bitmap .. ok, 447968730 blocks marked used Skipping 23115 blocks (super block, journal, bitmaps) 447945615 blocks will be read 0%.pass0: vpf-10180: block 37939817: item 11: The item [2777 2778 0x1 IND (1)], which follows non StatData item [215 15738 0x1 DIR (3)], is deleted . left 0, 21126 /secc 354 directory entries were hashed with "r5" hash. "r5" hash is selected Flushing..finished Read blocks (but not data blocks) 447945615 Leaves among those 442223 Objectids found 357 Pass 1 (will try to insert 442223 leaves): ####### Pass 1 ####### Looking for allocable blocks .. finished 0%....20%....40%....60%....80%....100% left 0, 67 /sec Flushing..finished 442223 leaves read 442221 inserted 2 not inserted ####### Pass 2 ####### Pass 2: 0%....20%....40%....60%....80%....100% left 0, 0 /sec Flushing..finished Leaves inserted item by item 2 Pass 3 (semantic): ####### Pass 3 ######### /Music/VASTrebuild_semantic_pass: The entry [2779 2780] ("Pretty When You Cry #2.mp3") in directory [215 2779] points to nowhere - is removed vpf-10650: The directory [215 2779] has the wrong size in the StatData (96) - corrected to (48) /Sagarebuild_semantic_pass: The entry [2859 2860] ("On The Loose #2.mp3") in directory [215 2859] points to nowhere - is removed vpf-10650: The directory [215 2859] has the wrong size in the StatData (88) - corrected to (48) /Movies/HavFlushing..finished Files found: 297 Directories found: 18 Names pointing to nowhere (removed): 2 Pass 3a (looking for lost dir/files): ####### Pass 3a (lost+found pass) ######### Looking for lost directories: ^[[A^[[bLooking for lost files:2 /sec vpf-10680: The file [2777 2778] has the wrong block count in the StatData (5344) - corrected to (2280) vpf-10680: The file [2781 40545] has the wrong block count in the StatData (11096) - corrected to (7960) vpf-10680: The file [2781 40573] has the wrong block count in the StatData (9952) - corrected to (7808) vpf-10680: The file [2781 40578] has the wrong block count in the StatData (12576) - corrected to (3168) Flushing..finished Objects without names 33 Files linked to /lost+found 33 Pass 4 - finished done 442196, 23 /sec Deleted unreachable items 5 Flushing..finished Syncing..finished ########### reiserfsck finished at Sat Jan 8 19:53:57 2011 ########### Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.