October 7, 201411 yr Had to reboot my server a couple of times... this time, it came up and now i've got one redballed drive. This drive has had errors before. I'm seeing a power up error...but not sure what to make of the rest of this: smartctl 6.2 2013-07-26 r3841 [i686-linux-3.9.11p-unRAID] (local build) Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: SAMSUNG SpinPoint F3 Device Model: SAMSUNG HD103SJ Serial Number: S2AEJ1BZ600843 LU WWN Device Id: 5 0024e9 0039bf7b9 Firmware Version: 1AJ10001 User Capacity: 1,000,204,886,016 bytes [1.00 TB] Sector Size: 512 bytes logical/physical Rotation Rate: 7200 rpm Device is: In smartctl database [for details use: -P show] ATA Version is: ATA8-ACS T13/1699-D revision 6 SATA Version is: SATA 2.6, 3.0 Gb/s Local Time is: Mon Oct 6 22:29:09 2014 EDT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 9240) seconds. Offline data collection capabilities: (0x5b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 154) minutes. SCT capabilities: (0x003f) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 100 100 051 Pre-fail Always - 46 2 Throughput_Performance 0x0026 252 252 000 Old_age Always - 0 3 Spin_Up_Time 0x0023 071 069 025 Pre-fail Always - 9004 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 125 5 Reallocated_Sector_Ct 0x0033 252 252 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 252 252 051 Old_age Always - 0 8 Seek_Time_Performance 0x0024 252 252 015 Old_age Offline - 0 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 30883 10 Spin_Retry_Count 0x0032 252 252 051 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 252 252 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 152 191 G-Sense_Error_Rate 0x0022 100 100 000 Old_age Always - 1 192 Power-Off_Retract_Count 0x0022 252 252 000 Old_age Always - 0 194 Temperature_Celsius 0x0002 064 049 000 Old_age Always - 35 (Min/Max 16/51) 195 Hardware_ECC_Recovered 0x003a 100 100 000 Old_age Always - 0 196 Reallocated_Event_Count 0x0032 252 252 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 252 252 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0030 252 252 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0036 100 100 000 Old_age Always - 29 200 Multi_Zone_Error_Rate 0x002a 100 100 000 Old_age Always - 149 223 Load_Retry_Count 0x0032 252 252 000 Old_age Always - 0 225 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 183 SMART Error Log Version: 1 ATA Error Count: 9 (device log contains only the most recent five errors) CR = Command Register [HEX] FR = Features Register [HEX] SC = Sector Count Register [HEX] SN = Sector Number Register [HEX] CL = Cylinder Low Register [HEX] CH = Cylinder High Register [HEX] DH = Device/Head Register [HEX] DC = Device Command Register [HEX] ER = Error register [HEX] ST = Status register [HEX] Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days. Error 9 occurred at disk power-on lifetime: 3296 hours (137 days + 8 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 51 5f 00 00 00 40 Error: ICRC, ABRT 95 sectors at LBA = 0x00000000 = 0 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 20 3f 00 00 40 00 00:00:00.045 READ DMA c8 00 20 3f 00 00 40 00 00:00:00.044 READ DMA c8 00 20 3f 00 00 40 00 00:00:00.043 READ DMA c8 00 20 3f 00 00 40 00 00:00:00.042 READ DMA c8 00 20 3f 00 00 40 00 00:00:00.041 READ DMA Error 8 occurred at disk power-on lifetime: 3296 hours (137 days + 8 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 51 5f 00 00 00 40 Error: ICRC, ABRT 95 sectors at LBA = 0x00000000 = 0 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 20 3f 00 00 40 00 00:00:00.044 READ DMA c8 00 20 3f 00 00 40 00 00:00:00.043 READ DMA c8 00 20 3f 00 00 40 00 00:00:00.042 READ DMA c8 00 20 3f 00 00 40 00 00:00:00.041 READ DMA c8 00 20 3f 00 00 40 00 00:00:00.040 READ DMA Error 7 occurred at disk power-on lifetime: 3296 hours (137 days + 8 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 51 5f 00 00 00 40 Error: ICRC, ABRT 95 sectors at LBA = 0x00000000 = 0 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 20 3f 00 00 40 00 00:00:00.043 READ DMA c8 00 20 3f 00 00 40 00 00:00:00.042 READ DMA c8 00 20 3f 00 00 40 00 00:00:00.041 READ DMA c8 00 20 3f 00 00 40 00 00:00:00.040 READ DMA c8 00 20 3f 00 00 40 00 00:00:00.039 READ DMA Error 6 occurred at disk power-on lifetime: 3296 hours (137 days + 8 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 51 5f 00 00 00 40 Error: ICRC, ABRT 95 sectors at LBA = 0x00000000 = 0 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 20 3f 00 00 40 00 00:00:00.042 READ DMA c8 00 20 3f 00 00 40 00 00:00:00.041 READ DMA c8 00 20 3f 00 00 40 00 00:00:00.040 READ DMA c8 00 20 3f 00 00 40 00 00:00:00.039 READ DMA c8 00 20 3f 00 00 40 00 00:00:00.038 READ DMA Error 5 occurred at disk power-on lifetime: 3296 hours (137 days + 8 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 84 51 5f 00 00 00 40 Error: ICRC, ABRT 95 sectors at LBA = 0x00000000 = 0 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 20 3f 00 00 40 00 00:00:00.041 READ DMA c8 00 20 3f 00 00 40 00 00:00:00.040 READ DMA c8 00 20 3f 00 00 40 00 00:00:00.039 READ DMA c8 00 20 3f 00 00 40 00 00:00:00.038 READ DMA c8 00 20 3f 00 00 40 00 00:00:00.037 READ DMA SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 25819 - # 2 Short offline Completed without error 00% 18679 - SMART Selective self-test log data structure revision number 0 Note: revision number not 1 implies that no selective self-test has ever been run SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Completed [00% left] (0-65535) 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. /code]
October 7, 201411 yr Author tail from Log Oct 6 22:27:10 Rigel kernel: mpt2sas0: log_info(0x31120303): originator(PL), code(0x12), sub_code(0x0303) Oct 6 22:27:41 Rigel last message repeated 24 times Oct 6 22:28:05 Rigel last message repeated 18 times Oct 6 22:28:06 Rigel kernel: mpt2sas0: log_info(0x3112010c): originator(PL), code(0x12), sub_code(0x010c) Oct 6 22:28:08 Rigel kernel: mpt2sas0: log_info(0x31120303): originator(PL), code(0x12), sub_code(0x0303) Oct 6 22:28:39 Rigel last message repeated 27 times Oct 6 22:29:40 Rigel last message repeated 47 times Oct 6 22:30:08 Rigel last message repeated 20 times Oct 6 22:30:09 Rigel kernel: mpt2sas0: log_info(0x3112010c): originator(PL), code(0x12), sub_code(0x010c) Oct 6 22:30:11 Rigel kernel: mpt2sas0: log_info(0x31120303): originator(PL), code(0x12), sub_code(0x0303) Oct 6 22:30:18 Rigel last message repeated 5 times Oct 6 22:30:19 Rigel kernel: mpt2sas0: log_info(0x3112010c): originator(PL), code(0x12), sub_code(0x010c) Oct 6 22:30:20 Rigel kernel: mpt2sas0: log_info(0x31120303): originator(PL), code(0x12), sub_code(0x0303) Oct 6 22:30:21 Rigel kernel: mpt2sas0: log_info(0x3112010c): originator(PL), code(0x12), sub_code(0x010c) Oct 6 22:30:22 Rigel kernel: mpt2sas0: log_info(0x31120303): originator(PL), code(0x12), sub_code(0x0303) Oct 6 22:30:53 Rigel last message repeated 22 times Oct 6 22:31:12 Rigel last message repeated 14 times Oct 6 22:31:13 Rigel kernel: mpt2sas0: log_info(0x3112010c): originator(PL), code(0x12), sub_code(0x010c) Oct 6 22:31:13 Rigel kernel: mpt2sas0: log_info(0x31120303): originator(PL), code(0x12), sub_code(0x0303) Oct 6 22:31:31 Rigel last message repeated 14 times Oct 6 22:31:40 Rigel kernel: mpt2sas0: log_info(0x3112010c): originator(PL), code(0x12), sub_code(0x010c) Oct 6 22:32:02 Rigel kernel: mpt2sas0: log_info(0x31120303): originator(PL), code(0x12), sub_code(0x0303) Oct 6 22:32:06 Rigel kernel: mpt2sas0: log_info(0x3112010c): originator(PL), code(0x12), sub_code(0x010c) Oct 6 22:32:08 Rigel kernel: mpt2sas0: log_info(0x3112010c): originator(PL), code(0x12), sub_code(0x010c) Oct 6 22:32:32 Rigel kernel: mpt2sas0: log_info(0x31120303): originator(PL), code(0x12), sub_code(0x0303) Oct 6 22:32:34 Rigel kernel: mpt2sas0: log_info(0x3112010c): originator(PL), code(0x12), sub_code(0x010c) Oct 6 22:32:42 Rigel kernel: mpt2sas0: log_info(0x31120303): originator(PL), code(0x12), sub_code(0x0303) Oct 6 22:33:07 Rigel kernel: mpt2sas0: log_info(0x3112010c): originator(PL), code(0x12), sub_code(0x010c) Oct 6 22:33:43 Rigel last message repeated 3 times Oct 6 22:34:10 Rigel kernel: mpt2sas0: log_info(0x31120303): originator(PL), code(0x12), sub_code(0x0303) Oct 6 22:35:01 Rigel kernel: mpt2sas0: log_info(0x31120303): originator(PL), code(0x12), sub_code(0x0303) Oct 6 22:36:04 Rigel last message repeated 3 times Oct 6 22:37:22 Rigel last message repeated 2 times Oct 6 22:38:31 Rigel last message repeated 3 times Oct 6 22:40:12 Rigel kernel: mpt2sas0: log_info(0x31120303): originator(PL), code(0x12), sub_code(0x0303) Oct 6 22:41:50 Rigel last message repeated 2 times
October 8, 201411 yr Check for SATA card firmware and MB BIOS updates. Also see here: http://lime-technology.com/wiki/index.php/Troubleshooting#What_do_I_do_if_I_get_a_red_ball_next_to_a_hard_disk.3F
October 8, 201411 yr Author Check for SATA card firmware and MB BIOS updates. Also see here: http://lime-technology.com/wiki/index.php/Troubleshooting#What_do_I_do_if_I_get_a_red_ball_next_to_a_hard_disk.3F thanks - i've seen errors on this drive before... 205 the last time. around 35 this time... so I started moving data unto a free drive. One thing that i learned from this ordeal is to be even better prepared. Moving data from a simulated disk to a disk that is part of the array is painfully slow. moving 700GB from a 1TB failed drive to a 3TB empty (same array) drive is taking almost 48hrs. i'll see if there are any updates. Thanks again.
November 9, 201411 yr Author So I followed the "Srink array" procedure - everything was fine until today, when I got another failed drive. Smart test shows no errors; the drives were using different bays and different cables (internally), so ruling that out. However, they are connected via external SAS with an expander, so it's posbbile one of those are flukey. not able to rule anything out - I am going to try swapping out some of the cabling in the next few months. Meanwhile - should I go ahead and let UnRaid reconstruct from parity? my SMART is attached, nothing specatuclar though WD3TB_Disk9_SDK_smart.txt
November 25, 201411 yr Author another drive - more errors... this is now the third one.. they are on different cable paths, and different slots in backplane. Nov 24 22:14:05 Rigel kernel: sd 0:0:7:0: [sdi] Synchronizing SCSI cache Nov 24 22:14:06 Rigel kernel: mpt2sas0: log_info(0x31120303): originator(PL), code(0x12), sub_code(0x0303) Nov 24 22:14:06 Rigel kernel: sd 0:0:7:0: [sdi] Unhandled error code Nov 24 22:14:06 Rigel kernel: sd 0:0:7:0: [sdi] Nov 24 22:14:06 Rigel kernel: Result: hostbyte=0x01 driverbyte=0x00 Nov 24 22:14:06 Rigel kernel: sd 0:0:7:0: [sdi] CDB: Nov 24 22:14:06 Rigel kernel: cdb[0]=0x8a: 8a 00 00 00 00 00 4a 6c da 48 00 00 00 b8 00 00 Nov 24 22:14:06 Rigel kernel: end_request: I/O error, dev sdi, sector 1248647752 Nov 24 22:14:06 Rigel kernel: sd 0:0:7:0: [sdi] Nov 24 22:14:06 Rigel kernel: Result: hostbyte=0x01 driverbyte=0x00 Nov 24 22:14:06 Rigel kernel: scsi 0:0:7:0: [sdi] Unhandled error code Nov 24 22:14:06 Rigel kernel: scsi 0:0:7:0: [sdi] Nov 24 22:14:06 Rigel kernel: Result: hostbyte=0x01 driverbyte=0x00 Nov 24 22:14:06 Rigel kernel: scsi 0:0:7:0: [sdi] CDB: Nov 24 22:14:06 Rigel kernel: cdb[0]=0x8a: 8a 00 00 00 00 00 4a 6c f3 00 00 00 04 00 00 00 Nov 24 22:14:06 Rigel kernel: end_request: I/O error, dev sdi, sector 1248654080 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248647688 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248647696 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248647704 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248647712 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248647720 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248647728 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248647736 <snip> Nov 24 22:14:06 Rigel kernel: md: recovery thread woken up ... Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655192 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655200 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655208 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655216 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655224 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655232 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655240 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655248 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655256 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655264 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655272 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655280 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655288 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655296 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655304 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655312 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655320 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655328 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655336 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655344 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655352 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655360 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655368 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655376 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655384 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655392 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655400 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655408 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655416 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655424 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655432 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655440 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655448 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655456 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655464 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655704 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655712 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655720 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655728 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655736 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655744 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655752 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655760 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655768 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655776 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655784 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655792 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655800 Nov 24 22:14:06 Rigel kernel: md: disk8 write error, sector=1248655808 <snip> Nov 24 22:14:06 Rigel kernel: mpt2sas0: removing handle(0x0011), sas_addr(0x5001e676b5923feb) Nov 24 22:14:06 Rigel kernel: md: recovery thread has nothing to resync First failure was a TRUE failure - drive is no longer recognized by unraid; Second failure seemed to be loose cabling? I shut down, removed from array, re-added to array and allowed rebuild from parity. Succeeded. This third error - seems to have almost 800 failed writes. I'm guessing this will be a REAL error too. i'm going to shutdown and reset cables again. we'll see. Also - my smartctl report is not working: [issued: smartctl -a -A /dev/sdi | todos >boot/smart_sdi.txt] Smartctl open device: /dev/sdi failed: No such device
November 25, 201411 yr I wonder did the additional drive failures occur due to the additional stress put on the other drives in your system from the initial drive failure? If all the drives were similar ages then they might be reaching their life cycle?
November 25, 201411 yr Author but that's the thing - they don't seem to be real failures SMART reports seem fine - power up OK run OK etc. Even a rebuild off parity back to the drives in quesion works fine. This has to be a backplane/power/interface issue.
Archived
This topic is now archived and is closed to further replies.