February 20, 201412 yr I was in the middle of doing some disk swaps, trying to get my fastest drive as the parity drive and upgrading some other drives. Parity was valid. I noticed that one of dirs on disk 5 gave a permission denied error, even though the permissions were wide open. smartctl reported "overall success", but I went ahead and ran "reiserfsck /mnt/md5". It recommended that I run with the --rebuild-tree option, so I did. The rebuild succeeded, and a subsequent "reiserfsck /mnt/md5" didn't reveal any errors. But at that time I could hear a high-pitched whine coming from the drive. So I went ahead and replaced it. A data rebuild is in progress now. However, the original directory with the problem is now empty! (Using the array-reconstructed directory.) I'm afraid that maybe the problems with disk5 caused unraid to think that the directory was empty. I guess it's possible that I fat-fingered some rm command and wiped the folder, but I doubt it. I'm thinking at this point my only option is to mount the original failing disk outside the array and run a "reiserfsck --rebuild-tree --scan-whole-partition /mnt/sdX". Does anyone have a better idea?
February 20, 201412 yr When I trashed a FULL cache drive by trying to rebuild another drive on it - that is what I did. But I also used a dd command to copy the drive to my cold spare on another computer. I found the dd command I used here on the forums and/or wiki. It has been long enough that I don't remember what I used - sorry. After I used the DD command to copy the drive I used reiserfsck on the copy and got back all but about 200GB of the 1.8TB of data I had copied to the cache drive. I stuck it back in my server and copied the remaining data to the array like I originally intended. So long story short if you have another drive available you might try copying it with DD first then use reiserfsck on the copy. If that doesn't work and the original is still functional you can still try the reiserfsck on that if necessary. Edit Opps - I believe the command is 'dd' not DD since linux is case sensitive.
February 20, 201412 yr Author Thanks for the confirmation. I may try that soon. Meanwhile, disk7 is now mounted read-only, and reiserfsck wants me to rebuild the superblock. Before doing that, I'm hoping some experts can look at my attached syslog. Here's what I was doing. This is looking a lot like what happened on my other drive (minor edits for clarity): # dd if=/dev/zero of=/mnt/disk7/test.dd bs=1M count=256 conv=fdatasync dd: error writing â/mnt/disk7/test.ddâ: Cannot allocate memory 1+0 records in 0+0 records out 0 bytes (0 B) copied, 0.00187236 s, 0.0 kB/s # ls /mnt/disk7 Backups/ Media/ test.dd # dd if=/dev/zero of=/mnt/disk7/test.dd bs=1M count=256 conv=fdatasync dd: failed to open â/mnt/disk7/test.ddâ: Read-only file system # ls /mnt/disk7/Backups/ /bin/ls: cannot access /mnt/disk7/Backups/Juggernaut Windows 7: Permission denied Juggernaut\ Windows\ 7 # reiserfsck --check /mnt/disk7 reiserfsck 3.6.24 Will read-only check consistency of the filesystem on /mnt/disk7 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes bread: Cannot read the block (2): (Is a directory). reiserfs_open: bread failed reading block 2 bread: Cannot read the block (16): (Is a directory). reiserfs_open: bread failed reading block 16 reiserfs_open: the reiserfs superblock cannot be found on /mnt/disk7. Failed to open the filesystem. If the partition table has not been changed, and the partition is valid and it really contains a reiserfs partition, then the superblock is corrupted and you need to run this utility with --rebuild-sb. The memory error might have been due to cache_dirs. I've disabled that for now. What I don't understand is why unraid's main page shows disk7 as being just fine, when clearly it is not... syslog.txt
February 20, 201412 yr You are well beyond my knowledge level now. I'm sure an expert will step in with advice.
February 20, 201412 yr Use the technique described here to examine the disk: http://lime-technology.com/forum/index.php?topic=5072.msg47037#msg47037
February 20, 201412 yr I just realized that your running reiserfsck on the wrong device. See Check Disk Filesystems in my sig.
February 21, 201412 yr Author Thanks for the help! Below is the reiserfsck output. Should I run --rebuild-tree? I'm concerned that this is the 2nd drive in 2 days where this has happened. Any ideas on the cause? Unraid 6.0 beta3? Bad controller card? My SFF->SATA breakout cables are kinda cheaply made. Could those be the cause? # cd # umount /dev/md7 # reiserfsck --check /dev/md7 reiserfsck 3.6.24 Will read-only check consistency of the filesystem on /dev/md7 Will put log info to 'stdout' Do you want to run this program?[N/Yes] (note need to type Yes if you do):Yes ########### reiserfsck --check started at Thu Feb 20 20:13:03 2014 ########### Replaying journal: Trans replayed: mountid 269, transid 853448, desc 296, len 1, commit 298, next trans offset 281 Trans replayed: mountid 269, transid 853449, desc 299, len 1, commit 301, next trans offset 284 Replaying journal: Done. Reiserfs journal '/dev/md7' in blocks [18..8211]: 2 transactions replayed Checking internal tree.. \/ 1 (of 27|/ 1 (of 166// 2 (of 150\block 231427306: The level of the node (44399) is not correct, (1) expected the problem in the internal node occured (231427306), whole subtree is skipped / 2 (of 166|block 311578333: The level of the node (16421) is not correct, (2) expected the problem in the internal node occured (311578333), whole subtree is skipped / 3 (of 27// 10 (of 136// 45 (of 170-block 320405777: The level of the node (22022) is not correct, (1) expected the problem in the internal node occured (320405777), whole subtree is skipped / 11 (of 136\/ 1 (of 163|block 320405946: The level of the node (45648) is not correct, (1) expected the problem in the internal node occured (320405946), whole subtree is skipped / 12 (of 136/block 320406291: The level of the node (8254) is not correct, (2) expected the problem in the internal node occured (320406291), whole subtree is skipped / 4 (of 27-/ 1 (of 170\block 219072050: The level of the node (2166) is not correct, (2) expected the problem in the internal node occured (219072050), whole subtree is skipped / 5 (of 27|/ 19 (of 86//162 (of 170\block 241132669: The level of the node (50878) is not correct, (1) expected the problem in the internal node occured (241132669), whole subtree is skipped / 20 (of 86|/ 1 (of 86/block 242472160: The level of the node (31124) is not correct, (1) expected the problem in the internal node occured (242472160), whole subtree is skipped / 21 (of 86-/ 1 (of 156\block 261752421: The level of the node (55760) is not correct, (1) expected the problem in the internal node occured (261752421), whole subtree is skipped / 22 (of 86|block 243602000: The level of the node (57068) is not correct, (2) expected the problem in the internal node occured (243602000), whole subtree is skipped / 6 (of 27// 1 (of 85-block 284811275: The level of the node (12510) is not correct, (2) expected the problem in the internal node occured (284811275), whole subtree is skipped / 7 (of 27\/ 18 (of 170-/109 (of 170\block 318061551: The level of the node (49063) is not correct, (1) expected the problem in the internal node occured (318061551), whole subtree is skipped / 19 (of 170|/ 1 (of 170/block 318121124: The level of the node (5172) is not correct, (1) expected the problem in the internal node occured (318121124), whole subtree is skipped / 20 (of 170-/ 1 (of 170\block 318293366: The level of the node (36357) is not correct, (1) expected the problem in the internal node occured (318293366), whole subtree is skipped / 21 (of 170|/ 1 (of 170/block 333744823: The level of the node (42739) is not correct, (1) expected the problem in the internal node occured (333744823), whole subtree is skipped / 22 (of 170-block 333845119: The level of the node (24980) is not correct, (2) expected the problem in the internal node occured (333845119), whole subtree is skipped / 8 (of 27\/ 4 (of 169// 90 (of 160\block 288523196: The level of the node (32409) is not correct, (1) expected the problem in the internal node occured (288523196), whole subtree is skipped / 5 (of 169|/ 1 (of 161/block 288592949: The level of the node (52181) is not correct, (1) expected the problem in the internal node occured (288592949), whole subtree is skipped / 9 (of 169\/ 97 (of 170|block 288614748: The level of the node (58969) is not correct, (1) expected the problem in the internal node occured (288614748), whole subtree is skipped / 10 (of 169/block 288614749: The level of the node (20093) is not correct, (2) expected the problem in the internal node occured (288614749), whole subtree is skipped / 9 (of 27-/ 50 (of 170-block 240043665: The level of the node (14298) is not correct, (2) expected the problem in the internal node occured (240043665), whole subtree is skipped / 10 (of 27\/ 1 (of 86|block 261362584: The level of the node (59299) is not correct, (2) expected the problem in the internal node occured (261362584), whole subtree is skipped / 11 (of 27// 1 (of 106-block 261379815: The level of the node (52143) is not correct, (2) expected the problem in the internal node occured (261379815), whole subtree is skipped / 12 (of 27\/ 95 (of 170//162 (of 170\block 212233928: The level of the node (12966) is not correct, (1) expected the problem in the internal node occured (212233928), whole subtree is skipped / 96 (of 170|/ 1 (of 170/block 212240092: The level of the node (58188) is not correct, (1) expected the problem in the internal node occured (212240092), whole subtree is skipped / 97 (of 170-/ 1 (of 170\block 241335172: The level of the node (29677) is not correct, (1) expected the problem in the internal node occured (241335172), whole subtree is skipped / 99 (of 170\/155 (of 170-block 282055460: The level of the node (2698) is not correct, (1) expected the problem in the internal node occured (282055460), whole subtree is skipped /100 (of 170\/ 1 (of 170|block 282079237: The level of the node (23712) is not correct, (1) expected the problem in the internal node occured (282079237), whole subtree is skipped /101 (of 170/block 282264595: The level of the node (12261) is not correct, (2) expected the problem in the internal node occured (282264595), whole subtree is skipped / 13 (of 27-/ 1 (of 156\/ 17 (of 170|block 289445792: The level of the node (41374) is not correct, (1) expected the problem in the internal node occured (289445792), whole subtree is skipped / 5 (of 156|/ 42 (of 170-block 366204056: The level of the node (29173) is not correct, (1) expected the problem in the internal node occured (366204056), whole subtree is skipped / 6 (of 156\block 366204057: The level of the node (31054) is not correct, (2) expected the problem in the internal node occured (366204057), whole subtree is skipped / 14 (of 27|/ 50 (of 154|/ 89 (of 170/block 324501507: The number of items (23) is incorrect, should be (1) the problem in the internal node occured (324501507), whole subtree is skipped / 65 (of 154|/166 (of 170-block 243272466: The level of the node (3511) is not correct, (1) expected the problem in the internal node occured (243272466), whole subtree is skipped / 66 (of 154\/ 2 (of 170/block 256205512: The level of the node (53408) is not correct, (1) expected the problem in the internal node occured (256205512), whole subtree is skipped / 67 (of 154-block 342392921: The level of the node (13536) is not correct, (2) expected the problem in the internal node occured (342392921), whole subtree is skipped / 15 (of 27\block 342394803: The level of the node (98) is not correct, (3) expected the problem in the internal node occured (342394803), whole subtree is skipped finished Comparing bitmaps..vpf-10640: The on-disk and the correct bitmaps differs. Bad nodes were found, Semantic pass skipped 37 found corruptions can be fixed only when running with --rebuild-tree ########### reiserfsck finished at Thu Feb 20 20:20:51 2014 ###########
February 21, 201412 yr Author smartctl output is below. Replaced SAS cables, and running repair now. # smartctl -a /dev/sdj smartctl 5.43 2012-06-30 r3573 [x86_64-linux-3.10.24p-unRAID] (local build) Copyright (C) 2002-12 by Bruce Allen, http://smartmontools.sourceforge.net === START OF INFORMATION SECTION === Model Family: Western Digital Caviar Green (Adv. Format) Device Model: WDC WD20EARS-00MVWB0 Serial Number: WD-WMAZA3209264 LU WWN Device Id: 5 0014ee 65617cc0d Firmware Version: 51.0AB51 User Capacity: 2,000,398,934,016 bytes [2.00 TB] Sector Size: 512 bytes logical/physical Device is: In smartctl database [for details use: -P show] ATA Version is: 8 ATA Standard is: Exact ATA specification draft version not indicated Local Time is: Fri Feb 21 16:02:19 2014 EST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x84) Offline data collection activity was suspended by an interrupting command from host. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (38580) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 372) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x3035) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 0 3 Spin_Up_Time 0x0027 253 168 021 Pre-fail Always - 1875 4 Start_Stop_Count 0x0032 099 099 000 Old_age Always - 1882 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 067 067 000 Old_age Always - 24331 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 122 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 66 193 Load_Cycle_Count 0x0032 194 194 000 Old_age Always - 19553 194 Temperature_Celsius 0x0022 124 099 000 Old_age Always - 26 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 No self-tests have been logged. [To run self-tests, use: smartctl -t] SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.
Archived
This topic is now archived and is closed to further replies.