Tybio Posted January 12, 2011 Share Posted January 12, 2011 After an issue with a disk last night, I swapped it out for a backup disk and it rebuilt cleanly (From all that I can see at this point). However, even after a reboot the "Free" and "Avaliable" space are both clearly wrong. Can anyone give me insight onto what might be going on here? Syslog attached, output from cli below: root@Storage:/var/log# mount fusectl on /sys/fs/fuse/connections type fusectl (rw) usbfs on /proc/bus/usb type usbfs (rw) /dev/sdg1 on /boot type vfat (rw,noatime,nodiratime,umask=0,shortname=mixed) /dev/md7 on /mnt/disk7 type reiserfs (rw,noatime,nodiratime,noacl,nouser_xattr) /dev/md4 on /mnt/disk4 type reiserfs (rw,noatime,nodiratime,noacl,nouser_xattr) /dev/md1 on /mnt/disk1 type reiserfs (rw,noatime,nodiratime,noacl,nouser_xattr) /dev/md3 on /mnt/disk3 type reiserfs (rw,noatime,nodiratime,noacl,nouser_xattr) /dev/md6 on /mnt/disk6 type reiserfs (rw,noatime,nodiratime,noacl,nouser_xattr) /dev/md2 on /mnt/disk2 type reiserfs (rw,noatime,nodiratime,noacl,nouser_xattr) /dev/md5 on /mnt/disk5 type reiserfs (rw,noatime,nodiratime,noacl,nouser_xattr) /dev/md9 on /mnt/disk9 type reiserfs (rw,noatime,nodiratime,noacl,nouser_xattr) /dev/md8 on /mnt/disk8 type reiserfs (rw,noatime,nodiratime,noacl,nouser_xattr) shfs on /mnt/user type fuse.shfs (rw,nosuid,nodev,noatime,allow_other,default_permissions) root@Storage:/var/log# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sdg1 1003488 94320 909168 10% /boot /dev/md7 732552188 257745516 474806672 36% /mnt/disk7 /dev/md4 488371640 448063056 40308584 92% /mnt/disk4 /dev/md1 1953454928 1772297552 181157376 91% /mnt/disk1 /dev/md3 1953454928 -1166915984 3120370912 - /mnt/disk3 /dev/md6 488371640 472507012 15864628 97% /mnt/disk6 /dev/md2 1953454928 1004594508 948860420 52% /mnt/disk2 /dev/md5 488371640 444802028 43569612 92% /mnt/disk5 /dev/md9 488371640 231472808 256898832 48% /mnt/disk9 /dev/md8 488371640 276332004 212039636 57% /mnt/disk8 shfs 9034775172 3740898500 5293876672 42% /mnt/user syslog.bak.zip Quote Link to comment
Tybio Posted January 12, 2011 Author Share Posted January 12, 2011 Interesting, I was trying to do a 'du -hs' on the drive to see what it should be and found a file that root didn't have permission to stat: root@Storage:/mnt/disk3/Media# du -hs du: cannot access `./TV/V 2009/Season 02/V (2009) - 02x02 - Serpent\'s Tooth.avi': Permission denied 934G . root@Storage:/mnt/disk3/Media# cd /mnt/disk3/Media/TV/V 2009/Season 02 root@Storage:/mnt/disk3/Media/TV/V 2009/Season 02# ls -alh /bin/ls: cannot access V (2009) - 02x02 - Serpent's Tooth.avi: Permission denied total 351M drwx--x--x 2 root root 200 Jan 12 03:03 ./ drwx--x--x 3 root root 80 Jan 4 22:42 ../ -rwx------ 1 root root 351M Jan 4 22:43 V\ (2009)\ -\ 02x01\ -\ Red\ Rain.avi* -rwx------ 1 root root 2.6K Jan 6 10:05 V\ (2009)\ -\ 02x01\ -\ Red\ Rain.nfo* ???? ? ? ? ? ? V\ (2009)\ -\ 02x02\ -\ Serpent's\ Tooth.avi root@Storage:/mnt/disk3/Media/TV/V 2009/Season 02# Quote Link to comment
Joe L. Posted January 12, 2011 Share Posted January 12, 2011 After an issue with a disk last night, I swapped it out for a backup disk and it rebuilt cleanly (From all that I can see at this point). However, even after a reboot the "Free" and "Avaliable" space are both clearly wrong. Can anyone give me insight onto what might be going on here? Syslog attached, output from cli below: root@Storage:/var/log# mount fusectl on /sys/fs/fuse/connections type fusectl (rw) usbfs on /proc/bus/usb type usbfs (rw) /dev/sdg1 on /boot type vfat (rw,noatime,nodiratime,umask=0,shortname=mixed) /dev/md7 on /mnt/disk7 type reiserfs (rw,noatime,nodiratime,noacl,nouser_xattr) /dev/md4 on /mnt/disk4 type reiserfs (rw,noatime,nodiratime,noacl,nouser_xattr) /dev/md1 on /mnt/disk1 type reiserfs (rw,noatime,nodiratime,noacl,nouser_xattr) /dev/md3 on /mnt/disk3 type reiserfs (rw,noatime,nodiratime,noacl,nouser_xattr) /dev/md6 on /mnt/disk6 type reiserfs (rw,noatime,nodiratime,noacl,nouser_xattr) /dev/md2 on /mnt/disk2 type reiserfs (rw,noatime,nodiratime,noacl,nouser_xattr) /dev/md5 on /mnt/disk5 type reiserfs (rw,noatime,nodiratime,noacl,nouser_xattr) /dev/md9 on /mnt/disk9 type reiserfs (rw,noatime,nodiratime,noacl,nouser_xattr) /dev/md8 on /mnt/disk8 type reiserfs (rw,noatime,nodiratime,noacl,nouser_xattr) shfs on /mnt/user type fuse.shfs (rw,nosuid,nodev,noatime,allow_other,default_permissions) root@Storage:/var/log# df Filesystem 1K-blocks Used Available Use% Mounted on /dev/sdg1 1003488 94320 909168 10% /boot /dev/md7 732552188 257745516 474806672 36% /mnt/disk7 /dev/md4 488371640 448063056 40308584 92% /mnt/disk4 /dev/md1 1953454928 1772297552 181157376 91% /mnt/disk1 /dev/md3 1953454928 -1166915984 3120370912 - /mnt/disk3 /dev/md6 488371640 472507012 15864628 97% /mnt/disk6 /dev/md2 1953454928 1004594508 948860420 52% /mnt/disk2 /dev/md5 488371640 444802028 43569612 92% /mnt/disk5 /dev/md9 488371640 231472808 256898832 48% /mnt/disk9 /dev/md8 488371640 276332004 212039636 57% /mnt/disk8 shfs 9034775172 3740898500 5293876672 42% /mnt/user That would usually indicate some kid of file system corruption. The issue is that it thinks you have 3120370912 bytes available. (and your disk is not that big) Use the procedure described here in the wiki to check and repair your file system on /dev/md3. http://lime-technology.com/wiki/index.php?title=Check_Disk_Filesystems Quote Link to comment
Tybio Posted January 12, 2011 Author Share Posted January 12, 2011 Joe, Thanks for the quick response, is there a risk with -rebuild-tree? I see that it is noted in the red section on the wiki, but from my reading I don't see that it has the risk of -rebuild-sb so I should be able to run it. Output for reference: root@Storage:~# reiserfsck --check /dev/md3 reiserfsck 3.6.19 (2003 www.namesys.com) ************************************************************* ** If you are using the latest reiserfsprogs and it fails ** ** please email bug reports to [email protected], ** ** providing as much information as possible -- your ** ** hardware, kernel, patches, settings, all reiserfsck ** ** messages (including version), the reiserfsck logfile, ** ** check the syslog file for any related information. ** ** If you would like advice on using this program, support ** ** is available for $25 at www.namesys.com/support.html. ** ************************************************************* Will read-only check consistency of the filesystem on /dev/md3 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes ########### reiserfsck --check started at Wed Jan 12 09:56:11 2011 ########### Replaying journal.. Reiserfs journal '/dev/md3' in blocks [18..8211]: 0 transactions replayed Checking internal tree../ 6 (of 11)/ 19 (of 140)/137 (of 170)bad_indirect_item: block 118740499: The item (1504 37 0xbf573001 IND (1), len 4048, location 48 entry count 0, fsck need 0, format new) has the bad pointer (761) to the block (85927183), which is in tree already / 11 (of 11)/ 79 (of 93)/ 26 (of 128)bad_path: The left delimiting key [3917 3919 0x14001 IND (1)] of the node (200310785) must be equal to the first element's key [3917 3956 0x182001 IND (1)] within the node. / 27 (of 128)block 200310786: The level of the node (2) is not correct, (1) expected the problem in the internal node occured (200310786), whole subtree is skipped finished Comparing bitmaps..vpf-10640: The on-disk and the correct bitmaps differs. Bad nodes were found, Semantic pass skipped 2 found corruptions can be fixed only when running with --rebuild-tree ########### reiserfsck finished at Wed Jan 12 10:18:25 2011 ########### root@Storage:~# Quote Link to comment
Joe L. Posted January 12, 2011 Share Posted January 12, 2011 If the file system is corrupted, there is more risk. The warnings are there so you know there is always some risk. Go ahead with the rebuild-tree. reiserfsck has been very good at repairing and recovering what it can. Joe L. Quote Link to comment
Tybio Posted January 12, 2011 Author Share Posted January 12, 2011 If the file system is corrupted, there is more risk. The warnings are there so you know there is always some risk. Go ahead with the rebuild-tree. reiserfsck has been very good at repairing and recovering what it can. Joe L. Granted, it's already broken...so recovering any of the files at this point is on the positive side. Losing all 900G of movies/tv shows would...well, hurt . I'll run the rebuild-tree and move this thread to solved if things come out clean at the end. Quote Link to comment
Tybio Posted January 12, 2011 Author Share Posted January 12, 2011 Once the check is finished, how should I move forward? Right now the webUI shows the disk as unformatted, because it was umounted. When the check finishes should I just mount it back to disk3, or should I stop the array and reboot? I don't want unraid to think this disk is "new" so I'm a bit confused as to the path forward. Quote Link to comment
Joe L. Posted January 12, 2011 Share Posted January 12, 2011 Once the check is finished, how should I move forward? Right now the webUI shows the disk as unformatted, because it was umounted. When the check finishes should I just mount it back to disk3, or should I stop the array and reboot? I don't want unraid to think this disk is "new" so I'm a bit confused as to the path forward. Easiest is to just stop the array and re-start it. It should not see the disk as un-formatted. (and if by some chance it does, DO NOT PRESS THE FORMAT BUTTON) Joe L. Quote Link to comment
Tybio Posted January 12, 2011 Author Share Posted January 12, 2011 Lost about half my data...not...good really. Now I need to figure out how to know what data was lost...going to try to get the old bad disk working enough to get a file list off of it I suppose . But that's another issue, this one is resolved. Thanks Joe! Quote Link to comment
Joe L. Posted January 12, 2011 Share Posted January 12, 2011 Lost about half my data...not...good really. Now I need to figure out how to know what data was lost...going to try to get the old bad disk working enough to get a file list off of it I suppose . But that's another issue, this one is resolved. Thanks Joe! Did you look in the lost+found directory? You'll probably find most of it there, even if under different names. You'll need to rename the files. Joe L. Quote Link to comment
Tybio Posted January 12, 2011 Author Share Posted January 12, 2011 I'm looking at it, but there are only 93G of files in lost+found and disk usage went down by ~450G So I'm going to see if I can get valid data off that bad drive before I RMA it. Quote Link to comment
Joe L. Posted January 12, 2011 Share Posted January 12, 2011 I'm looking at it, but there are only 93G of files in lost+found and disk usage went down by ~450G So I'm going to see if I can get valid data off that bad drive before I RMA it. Do you trust your previous "free" space numbers? remember, they are what tipped you off when you discovered the corruption. Yes. any attempt to get the data off the old drives would help. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.