January 2, 201016 yr Hi all, Today I received an email that one of my drives was disabled: This message is a status update for unRAID unraid ----------------------------------------------------------------- Server Name: unraid Status: The unRaid array needs attention. One or more disks are disabled or invalid. Date: Fri Jan 1 15:27:05 GMT+7 2010 Disk Temperature Status ----------------------------------------------------------------- Parity Disk [sda]: 25°C (DiskId: ata-WDC_WD20EADS-00R6B0_WD-WCAVY1212066) Disk 1 [sdd]: 22°C (DiskId: ata-WDC_WD20EADS-00S2B0_WD-WCAVY1279748) Disk 2 [sdc]: 23°C (DiskId: ata-WDC_WD10EACS-00C7B0_WD-WCASJ1431222) Disk 3 [sdb]: 17°C (DiskId: ata-SAMSUNG_HD103UJ_S13PJDWS117041) Disk 4 [sde]: 22°C (DiskId: ata-WDC_WD10EADS-00L5B1_WD-WCAU49710758) Disk 5 [sdg]: Not-Reported (DiskId: ata-WDC_WD10EACS-00ZJB0_WD-WCASJ0579788) Disk SMART Health Status ----------------------------------------------------------------- Parity Disk PASSED (DiskId: ata-WDC_WD20EADS-00R6B0_WD-WCAVY1212066) Disk 1 PASSED (DiskId: ata-WDC_WD20EADS-00S2B0_WD-WCAVY1279748) Disk 2 PASSED (DiskId: ata-WDC_WD10EACS-00C7B0_WD-WCASJ1431222) Disk 3 PASSED (DiskId: ata-SAMSUNG_HD103UJ_S13PJDWS117041) Disk 4 PASSED (DiskId: ata-WDC_WD10EADS-00L5B1_WD-WCAU49710758) Disk 5 Not-Reported (DiskId: ata-WDC_WD10EACS-00ZJB0_WD-WCASJ0579788) Output of /proc/mdcmd: ----------------------------------------------------------------- cmdOper=status cmdResult=ok sbName=/boot/config/super.dat sbVersion=0.95.0 sbCreated=1247152851 sbUpdated=1262384815 sbEvents=37 sbState=0 sbNumDisks=6 sbSynced=1262366896 sbSyncErrs=0 mdVersion=0.95.3 mdState=STARTED mdNumProtected=6 mdNumDisabled=1 mdDisabledDisk=5 mdNumInvalid=1 mdInvalidDisk=5 mdNumMissing=0 mdMissingDisk=0 mdNumNew=0 mdResync=0 <snip> diskNumber.5=5 diskName.5=md5 diskSize.5=976762552 diskState.5=4 diskModel.5=WDC WD10EACS-00ZJB0 diskSerial.5= WD-WCASJ0579788 diskId.5=ata-WDC_WD10EACS-00ZJB0_WD-WCASJ0579788 rdevNumber.5=5 rdevStatus.5=DISK_DSBL rdevName.5=sdg rdevSize.5=976762552 rdevModel.5=WDC WD10EACS-00ZJB0 rdevSerial.5= WD-WCASJ0579788 rdevId.5=ata-WDC_WD10EACS-00ZJB0_WD-WCASJ0579788 rdevNumErrors.5=4 rdevLastIO.5=1262384815 rdevSpinupGroup.5=0 Looking at the syslog there were a bunch of errors like this: Dec 30 18:40:25 unraid shfs: shfs_setxattr: setxattr: /mnt/disk2/Pics/Slideshow/Trips/~MG_1837.tmp (95) Operation not supported Then last night it started the monthly parity check and the poop looks to have hit the fan, although the parity check seems to have passed. See firsterrors.zip. I then rebooted the system and disk 5 showed up with a good temp and passed SMART health. I used unmenu to run a short SMART test on the drive in question and it passed with no problems. So I followed the wiki instructions on how to re-enable the drive. Its now doing the parity test and I notice that there are sync errors and errors on the parity drive (see screenshot). I then looked at the syslog and see that its filling with errors. I've also attached it. I'd be grateful for any input on what might be going on and if there's anything I can do. I'm worried I might have lost data...
January 2, 201016 yr Author The check finished with 74 sync errors and the parity drive shows 1487 errors. It says the parity is valid. SMART shows no errors on any of the drives. What gives?
January 8, 201016 yr Author Help!! Can someone look at the logs above as its failed again in exactly the same manner: This message is a status update for unRAID unraid ----------------------------------------------------------------- Server Name: unraid Status: The unRaid array needs attention. One or more disks are disabled or invalid. Date: Thu Jan 7 16:08:04 GMT+7 2010 Disk Temperature Status ----------------------------------------------------------------- Parity Disk [sda]: Spun-Down (DiskId: ata-WDC_WD20EADS-00R6B0_WD-WCAVY1212066) Disk 1 [sdd]: Spun-Down (DiskId: ata-WDC_WD20EADS-00S2B0_WD-WCAVY1279748) Disk 2 [sdc]: Spun-Down (DiskId: ata-WDC_WD10EACS-00C7B0_WD-WCASJ1431222) Disk 3 [sdb]: Spun-Down (DiskId: ata-SAMSUNG_HD103UJ_S13PJDWS117041) Disk 4 [sde]: Spun-Down (DiskId: ata-WDC_WD10EADS-00L5B1_WD-WCAU49710758) Disk 5 [sdg]: Not-Reported (DiskId: ata-WDC_WD10EACS-00ZJB0_WD-WCASJ0579788) Disk SMART Health Status ----------------------------------------------------------------- Parity Disk Spun-Down (DiskId: ata-WDC_WD20EADS-00R6B0_WD-WCAVY1212066) Disk 1 Spun-Down (DiskId: ata-WDC_WD20EADS-00S2B0_WD-WCAVY1279748) Disk 2 Spun-Down (DiskId: ata-WDC_WD10EACS-00C7B0_WD-WCASJ1431222) Disk 3 Spun-Down (DiskId: ata-SAMSUNG_HD103UJ_S13PJDWS117041) Disk 4 Spun-Down (DiskId: ata-WDC_WD10EADS-00L5B1_WD-WCAU49710758) Disk 5 Not-Reported (DiskId: ata-WDC_WD10EACS-00ZJB0_WD-WCASJ0579788) Output of /proc/mdcmd: ----------------------------------------------------------------- cmdOper=status cmdResult=ok sbName=/boot/config/super.dat sbVersion=0.95.3 sbCreated=1262388571 sbUpdated=1262819235 sbEvents=3 sbState=0 sbNumDisks=6 sbSynced=1262423086 sbSyncErrs=74 mdVersion=0.95.3 mdState=STARTED mdNumProtected=6 mdNumDisabled=1 mdDisabledDisk=5 mdNumInvalid=1 mdInvalidDisk=5 mdNumMissing=0 mdMissingDisk=0 mdNumNew=0 mdResync=0 <snip> diskNumber.5=5 diskName.5=md5 diskSize.5=976762552 diskState.5=4 diskModel.5=WDC WD10EACS-00ZJB0 diskSerial.5= WD-WCASJ0579788 diskId.5=ata-WDC_WD10EACS-00ZJB0_WD-WCASJ0579788 rdevNumber.5=5 rdevStatus.5=DISK_DSBL rdevName.5=sdg rdevSize.5=976762552 rdevModel.5=WDC WD10EACS-00ZJB0 rdevSerial.5= WD-WCASJ0579788 rdevId.5=ata-WDC_WD10EACS-00ZJB0_WD-WCASJ0579788 rdevNumErrors.5=630 rdevLastIO.5=0 rdevSpinupGroup.5=0
January 8, 201016 yr My first recommendation, since the drive cut out and then worked again and then was lost again, is to check or better yet change the SATA and power cables for that drive. If the drive actually died then it likely wouldn't wake up long enough to work for a while or pass the smart tests. Peter
January 8, 201016 yr any input on the error he posted? Dec 30 18:40:25 unraid shfs: shfs_setxattr: setxattr: /mnt/disk2/Pics/Slideshow/Trips/~MG_1837.tmp (95) Operation not supported I see this regularly on my array, though no errors on any disks. I'll be running a parity check on the system tonight as well and then adding 5 more drives for a total of 8. (work toward acheiving level 2 testing on my mobo) The only errors I've ever seen though are the one the OP stated above.
January 10, 201016 yr Seems like garbling of signal on the SATA cable. I'd try to reseat both ends and if that doesn't work replace the cable. If you are using a dock of some type, that might also be the problem. Try to connect directly to the port using a locking SATA cable and see if the problem resolves. With these errors, I would not feel confident about my ability to recover. The fact that the problem is on the parity drive means that the data is likely not corrupted.
January 10, 201016 yr Author Thanks for the help guys! I just ran a short test on the Parity drive and its reporting read errors so now I'm running a long test. Here's what SMART says about the parity drive: === START OF INFORMATION SECTION === Device Model: WDC WD20EADS-00R6B0 Serial Number: WD-WCAVY1212066 Firmware Version: 01.00A01 User Capacity: 2,000,398,934,016 bytes Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 8 ATA Standard is: Exact ATA specification draft version not indicated Local Time is: Sun Jan 10 10:36:06 2010 GMT+7 SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x84) Offline data collection activity was suspended by an interrupting command from host. Auto Offline Data Collection: Enabled. Self-test execution status: ( 121) The previous self-test completed having the read element of the test failed. Total time to complete Offline data collection: (41580) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 255) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x303f) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 198 198 051 Pre-fail Always - 5203 3 Spin_Up_Time 0x0027 180 152 021 Pre-fail Always - 7966 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 141 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0 9 Power_On_Hours 0x0032 099 099 000 Old_age Always - 1108 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 22 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 14 193 Load_Cycle_Count 0x0032 198 198 000 Old_age Always - 6512 194 Temperature_Celsius 0x0022 123 117 000 Old_age Always - 29 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 199 198 000 Old_age Always - 634 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed: read failure 90% 1108 547631370 # 2 Short offline Completed: read failure 90% 1108 1286625255 # 3 Short offline Completed: read failure 90% 1108 1286625270 # 4 Short offline Completed: read failure 80% 1096 1286625391 SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. Hmmm. Looks like the long test finished right away with a read error also. Those pending's can't be good. Next I'll power down and check the parity drives power and SATA cable. I'll definitely be putting an order into Monoprice for a bunch of locking SATA cables.
January 10, 201016 yr Thanks for the help guys! I just ran a short test on the Parity drive and its reporting read errors so now I'm running a long test. Here's what SMART says about the parity drive: === START OF INFORMATION SECTION === Device Model: WDC WD20EADS-00R6B0 Serial Number: WD-WCAVY1212066 Firmware Version: 01.00A01 User Capacity: 2,000,398,934,016 bytes Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 8 ATA Standard is: Exact ATA specification draft version not indicated Local Time is: Sun Jan 10 10:36:06 2010 GMT+7 SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x84) Offline data collection activity was suspended by an interrupting command from host. Auto Offline Data Collection: Enabled. Self-test execution status: ( 121) The previous self-test completed having the read element of the test failed. Total time to complete Offline data collection: (41580) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 255) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x303f) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 198 198 051 Pre-fail Always - 5203 3 Spin_Up_Time 0x0027 180 152 021 Pre-fail Always - 7966 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 141 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0 9 Power_On_Hours 0x0032 099 099 000 Old_age Always - 1108 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 22 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 14 193 Load_Cycle_Count 0x0032 198 198 000 Old_age Always - 6512 194 Temperature_Celsius 0x0022 123 117 000 Old_age Always - 29 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 199 198 000 Old_age Always - 634 198 Offline_Uncorrectable 0x0030 200 200 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 200 200 000 Old_age Offline - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed: read failure 90% 1108 547631370 # 2 Short offline Completed: read failure 90% 1108 1286625255 # 3 Short offline Completed: read failure 90% 1108 1286625270 # 4 Short offline Completed: read failure 80% 1096 1286625391 SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. Hmmm. Looks like the long test finished right away with a read error also. Those pending's can't be good. Next I'll power down and check the parity drives power and SATA cable. I'll definitely be putting an order into Monoprice for a bunch of locking SATA cables. The "current sectors pending re-allocation" have nothing to do with cables, locking or otherwise. They are an indication of a drive that is failing. They are sectors on the disk that could not be read, so the drive has marked then to be re-written in a new location on the disk when they are next written. (it is why they are "pending" re-allocation) I'd put in an order for a new drive to go along with those cables you've ordered. Joe L.
January 10, 201016 yr Author Yep, swapped power and SATA cables with another drive and errors stayed on parity disk. Only bought this 2TB drive a few months ago so hopefully I'll be able to get a replacement.
January 19, 201016 yr You have had good advice here. Just to summarize, there are 2 drives with problems, but the problems are completely different. The parity drive has a lot of bad sectors, and may be going bad. Disk 5 has serious interface issues, either data or power related, I don't know which. Disk 5 is probably good, but data on it *may* have been affected by the interface issues, in particular the inability to complete some of the writes, resulting in write errors. That may have resulted in partially written files and partial file system updates. Once it is reliable again, I would run reiserfsck on it. Since I don't think there were any CRC errors, the problem does not look like poor quality cables. It looks more like loose or defective SATA or power connectors or splitters, defective backplane, or possibly something wrong with its disk controller or the SATA port itself. In separate syslogs, Disk 5 was disabled when contact was completely lost, but each time prior to being disabled, there were serious episodes of interface errors. One other very remote possibility, something is getting too hot. In other words, I agree with the other posters!
Archived
This topic is now archived and is closed to further replies.