alphazo Posted November 25, 2014 Share Posted November 25, 2014 Hello, I have been running unRaid 5.0 for quite some time now have 13 various drives in the system. Yesterday in order to expand space I decided to replace disk1 (1.5B) by a brand new 4TB Hitachi drive. Here is what I did: Ran a parity check and rebuild Precleared new 4TB Replaced disk1 (1.5TB) by the 4TB one Rebooted and assigned the new drive Data rebuild then started for disk1 with an estimated time to completion of 1 day. Following moring I checked the status and ETA jumped to 400 days. I then looked at dmesg and it was full of read error messages from disk12 (another 4TB that was added to the system many months ago). md: disk12 read error, sector=586096 md: disk12 read error, sector=586104 ata7.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 ata7.00: edma_err_cause=00000084 pp_flags=00000001, dev error, EDMA self-disable ata7.00: failed command: READ DMA ata7.00: cmd c8/00:00:c0:f1:08/00:00:00:00:00/e0 tag 0 dma 131072 in res 51/40:00:c0:f1:08/40:04:00:00:00/e0 Emask 0x9 (media error) ata7.00: status: { DRDY ERR } ata7.00: error: { UNC } ata7: hard resetting link ata7: SATA link up 3.0 Gbps (SStatus 123 SControl 300) ata7.00: configured for UDMA/133 sd 6:0:0:0: [sdh] Unhandled sense code sd 6:0:0:0: [sdh] Result: hostbyte=0x00 driverbyte=0x08 sd 6:0:0:0: [sdh] Sense Key : 0x3 [current] [descriptor] Descriptor sense data with sense descriptors (in hex): 72 03 11 04 00 00 00 0c 00 0a 80 00 00 00 00 00 00 08 f1 c0 sd 6:0:0:0: [sdh] ASC=0x11 ASCQ=0x4 sd 6:0:0:0: [sdh] CDB: cdb[0]=0x88: 88 00 00 00 00 00 00 08 f1 c0 00 00 01 00 00 00 end_request: I/O error, dev sdh, sector 586176 md: disk12 read error, sector=586112 md: disk12 read error, sector=586120 ata7: EH complete md: disk12 read error, sector=586128 md: disk12 read error, sector=586136 md: disk12 read error, sector=586144 md: disk12 read error, sector=586152 md: disk12 read error, sector=586160 I shutdown the sytem and replaced sata cable and power cable for disk12 but issue keeps coming back. I don't exactly what to do at this point. Here is what I have: 4TB disk1 drive that needs to be rebuilt Original 1.5B disk1 drive with valid data and that is working fine Valid 4TB Parity drive (I guess) since I haven't started a parity check since the data rebuild Spare (precleared) 4TB drive I was tempted to put the 1.5TB back and go the "Make unRAID Trust the Parity Drive, Avoid Rebuilding Parity Unnecessarily" way but since the configuration doesn't watch because of the size mismatch I'm not sure it will work. What would you recommend? Thanks Alphazo PS: here is the smartctl for disk12 smartctl -a -d ata /dev/sdh smartctl 6.2 2013-07-26 r3841 [i686-linux-3.9.11p-unRAID] (local build) Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Device Model: WDC WD40EFRX-68WT0N0 Serial Number: WD-WCC4E0083446 LU WWN Device Id: 5 0014ee 2b3ba9249 Firmware Version: 80.00A80 User Capacity: 4,000,787,030,016 bytes [4.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 5400 rpm Device is: Not in smartctl database [for details use: -P showall] ATA Version is: ACS-2 (minor revision not indicated) SATA Version is: SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s) Local Time is: Tue Nov 25 23:12:18 2014 CET SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: (52080) seconds. Offline data collection capabilities: (0x7b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 521) minutes. Conveyance self-test routine recommended polling time: ( 5) minutes. SCT capabilities: (0x703d) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 5737 3 Spin_Up_Time 0x0027 177 177 021 Pre-fail Always - 8125 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 509 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 200 200 000 Old_age Always - 0 9 Power_On_Hours 0x0032 090 090 000 Old_age Always - 7571 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 29 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 4 193 Load_Cycle_Count 0x0032 198 198 000 Old_age Always - 8770 194 Temperature_Celsius 0x0022 130 111 000 Old_age Always - 22 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 197 197 000 Old_age Always - 2163 198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age Offline - 0 SMART Error Log Version: 1 ATA Error Count: 108 (device log contains only the most recent five errors) CR = Command Register [HEX] FR = Features Register [HEX] SC = Sector Count Register [HEX] SN = Sector Number Register [HEX] CL = Cylinder Low Register [HEX] CH = Cylinder High Register [HEX] DH = Device/Head Register [HEX] DC = Device Command Register [HEX] ER = Error register [HEX] ST = Status register [HEX] Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days. Error 108 occurred at disk power-on lifetime: 7571 hours (315 days + 11 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 40 38 f9 09 e0 Error: UNC 64 sectors at LBA = 0x0009f938 = 653624 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 40 38 f9 09 e0 08 00:10:51.916 READ DMA b0 d1 01 01 4f c2 00 08 00:10:51.896 SMART READ ATTRIBUTE THRESHOLDS [OBS-4] b0 d5 01 06 4f c2 00 08 00:10:51.876 SMART READ LOG b0 d0 01 00 4f c2 00 08 00:10:51.856 SMART READ DATA Error 107 occurred at disk power-on lifetime: 7571 hours (315 days + 11 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 c0 78 f8 09 e0 Error: UNC 192 sectors at LBA = 0x0009f878 = 653432 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 c0 78 f8 09 e0 08 00:10:47.104 READ DMA ec 00 00 00 00 00 00 08 00:10:47.101 IDENTIFY DEVICE ec 00 00 00 00 00 a0 08 00:10:47.081 IDENTIFY DEVICE ef 03 46 00 00 00 a0 08 00:10:47.081 SET FEATURES [set transfer mode] Error 106 occurred at disk power-on lifetime: 7571 hours (315 days + 11 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 40 38 f0 09 e0 Error: UNC 64 sectors at LBA = 0x0009f038 = 651320 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 40 38 f0 09 e0 08 00:10:33.633 READ DMA b0 d5 01 01 4f c2 00 08 00:10:33.613 SMART READ LOG ec 00 00 00 00 00 a0 08 00:10:33.593 IDENTIFY DEVICE ef 03 46 00 00 00 a0 08 00:10:33.593 SET FEATURES [set transfer mode] Error 105 occurred at disk power-on lifetime: 7571 hours (315 days + 11 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 c0 78 ef 09 e0 Error: UNC 192 sectors at LBA = 0x0009ef78 = 651128 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 c0 78 ef 09 e0 08 00:10:28.858 READ DMA ec 00 00 00 00 00 a0 08 00:10:28.838 IDENTIFY DEVICE ef 03 46 00 00 00 a0 08 00:10:28.838 SET FEATURES [set transfer mode] Error 104 occurred at disk power-on lifetime: 7571 hours (315 days + 11 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 40 38 e7 09 e0 Error: UNC 64 sectors at LBA = 0x0009e738 = 649016 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 40 38 e7 09 e0 08 00:10:15.350 READ DMA b0 d5 01 00 4f c2 00 08 00:10:15.330 SMART READ LOG ec 00 00 00 00 00 a0 08 00:10:15.310 IDENTIFY DEVICE ef 03 46 00 00 00 a0 08 00:10:15.310 SET FEATURES [set transfer mode] SMART Self-test log structure revision number 1 No self-tests have been logged. [To run self-tests, use: smartctl -t] SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. I have an old backup of super.dat that seems to have the configuation with the working 1.5TB disk1 drive. I looked at the file using an hex editor and all the serial numbers were there. Since disk12 seems in bad shape I need the parity not to be overwritten so I can restore disk12 on a new blank drive. [Few hours later] I went ahead and replaced my super.dat by an older backup of it that had the 1.5TB registered. Started back the machine with the old working 1.5TB on disk1 and a brand new precleared 4TB on disk12. Disk12 is now reconstructing. Will investigate later the status of the faulty drive. Lets's see what comes out of this rebuild. Quote Link to comment
itimpi Posted November 26, 2014 Share Posted November 26, 2014 The SMART report you posted looks rather like a sick disk with 2162 pending sectors being reported. It is possible they might all disappear with running a pre-clear cycle but I would be reluctant to use that disk even if that happened. Having a disk with ANY pending sectors is not advisable as they are likely to stop any other disk that fails from being successfully rebuilt without data loss. Quote Link to comment
alphazo Posted November 26, 2014 Author Share Posted November 26, 2014 Thanks for the advice. That particular drive is not going back to the array. Next time I expand the array I will 1) backup super.dat just to make sure 2) Go through the SMART reports of all drives to check for possible issues. What's strange on this one is that the parity check did run fine few weeks ago. Funny that an almost new drive from the WD RED serie (dedicated to NAS) is dying so fast Quote Link to comment
itimpi Posted November 26, 2014 Share Posted November 26, 2014 Thanks for the advice. That particular drive is not going back to the array. Next time I expand the array I will 1) backup super.dat just to make sure 2) Go through the SMART reports of all drives to check for possible issues. What's strange on this one is that the parity check did run fine few weeks ago. Funny that an almost new drive from the WD RED serie (dedicated to NAS) is dying so fast Did you put the drive through pre-clear cycles? A significant number of drives fail very early in their lifetime, so an early stress test is highly advisable. Quote Link to comment
Traxxus Posted November 26, 2014 Share Posted November 26, 2014 You should always run a parity check just before making any changes like that, smart tests aren't a bad idea either. If you were rebuilding to replace a failed drive you would likely be facing the loss of two drives worth of data right now. That drive has been in use for almost a year and not a single test on it. I try to remember to get smart reports before every parity check (monthly), and also run at least short smart tests before then. That way if I do get errors I can immediately pin down whether it's parity or data drive. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.