October 10, 201510 yr Hi everyone, I got a red ball on one of my disks today. I couldn't access the SMART data, so I wondered if it was my SATA port / controller. The red ball was parity. I moved it to the other bay / SATA, and now my config looks like: Before, it looked like... And before I had rebooted (yielding the above message), I had a red ball with "53 errors". For the parity disk, the SMART status is provided below: root@Tower:~# smartctl -a -A /dev/sdd smartctl 6.2 2013-07-26 r3841 [i686-linux-3.9.11p-unRAID] (local build) Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Seagate Barracuda 7200.14 (AF) Device Model: ST3000DM001-1CH166 Serial Number: Z1F40YWL LU WWN Device Id: 5 000c50 065298f6e Firmware Version: CC27 User Capacity: 3,000,592,982,016 bytes [3.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b SATA Version is: SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s) Local Time is: Sat Oct 10 22:15:03 2015 EDT ==> WARNING: A firmware update for this drive may be available, see the following Seagate web pages: http://knowledge.seagate.com/articles/en_US/FAQ/207931en http://knowledge.seagate.com/articles/en_US/FAQ/223651en SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 584) seconds. Offline data collection capabilities: (0x73) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. No Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 320) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x3085) SCT Status supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 117 099 006 Pre-fail Always - 134475576 3 Spin_Up_Time 0x0003 095 094 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 832 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 052 052 030 Pre-fail Always - 2499758075130 9 Power_On_Hours 0x0032 082 082 000 Old_age Always - 16078 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 44 183 Runtime_Bad_Block 0x0032 099 099 000 Old_age Always - 1 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 0 0 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 068 058 045 Old_age Always - 32 (Min/Max 31/32) 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 44 193 Load_Cycle_Count 0x0032 078 078 000 Old_age Always - 44181 194 Temperature_Celsius 0x0022 032 042 000 Old_age Always - 32 (0 20 0 0 0) 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 5569h+38m+36.074s 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 39811969952 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 204216837578 SMART Error Log Version: 1 ATA Error Count: 1 CR = Command Register [HEX] FR = Features Register [HEX] SC = Sector Count Register [HEX] SN = Sector Number Register [HEX] CL = Cylinder Low Register [HEX] CH = Cylinder High Register [HEX] DH = Device/Head Register [HEX] DC = Device Command Register [HEX] ER = Error register [HEX] ST = Status register [HEX] Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days. Error 1 occurred at disk power-on lifetime: 15853 hours (660 days + 13 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 04 71 00 ff ff ff 0f Device Fault; Error: ABRT at LBA = 0x0fffffff = 268435455 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 35 00 08 ff ff ff ef 00 11d+16:23:19.261 WRITE DMA EXT ca 00 08 b0 c0 00 e0 00 11d+16:22:45.377 WRITE DMA c8 00 08 b0 c0 00 e0 00 11d+16:22:45.370 READ DMA ca 00 08 a8 c0 00 e0 00 11d+16:22:45.345 WRITE DMA ca 00 08 a0 c0 00 e0 00 11d+16:22:45.320 WRITE DMA SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 8989 - # 2 Short offline Completed without error 00% 5631 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. It looks like one other of my drives also had a problem last year: root@Tower:~# smartctl -a -A /dev/sdb smartctl 6.2 2013-07-26 r3841 [i686-linux-3.9.11p-unRAID] (local build) Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Seagate Barracuda 7200.14 (AF) Device Model: ST3000DM001-1CH166 Serial Number: Z1F37YED LU WWN Device Id: 5 000c50 063af85f3 Firmware Version: CC27 User Capacity: 3,000,592,982,016 bytes [3.00 TB] Sector Sizes: 512 bytes logical, 4096 bytes physical Rotation Rate: 7200 rpm Device is: In smartctl database [for details use: -P show] ATA Version is: ACS-2, ACS-3 T13/2161-D revision 3b SATA Version is: SATA 3.1, 6.0 Gb/s (current: 3.0 Gb/s) Local Time is: Sat Oct 10 22:23:39 2015 EDT ==> WARNING: A firmware update for this drive may be available, see the following Seagate web pages: http://knowledge.seagate.com/articles/en_US/FAQ/207931en http://knowledge.seagate.com/articles/en_US/FAQ/223651en SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED See vendor-specific Attribute list for marginal Attributes. General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 584) seconds. Offline data collection capabilities: (0x73) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. No Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 343) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x3085) SCT Status supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 117 099 006 Pre-fail Always - 131530240 3 Spin_Up_Time 0x0003 094 094 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 098 098 020 Old_age Always - 2097 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 047 045 030 Pre-fail Always - 4857658421861 9 Power_On_Hours 0x0032 082 082 000 Old_age Always - 16100 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 45 183 Runtime_Bad_Block 0x0032 099 099 000 Old_age Always - 1 184 End-to-End_Error 0x0032 099 099 099 Old_age Always FAILING_NOW 1 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 099 000 Old_age Always - 1 1 1 189 High_Fly_Writes 0x003a 096 096 000 Old_age Always - 4 190 Airflow_Temperature_Cel 0x0022 067 057 045 Old_age Always - 33 (Min/Max 33/34) 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 45 193 Load_Cycle_Count 0x0032 082 082 000 Old_age Always - 37157 194 Temperature_Celsius 0x0022 033 043 000 Old_age Always - 33 (0 20 0 0 0) 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 3281h+52m+33.615s 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 18547403840 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 215095371776 SMART Error Log Version: 1 ATA Error Count: 1 CR = Command Register [HEX] FR = Features Register [HEX] SC = Sector Count Register [HEX] SN = Sector Number Register [HEX] CL = Cylinder Low Register [HEX] CH = Cylinder High Register [HEX] DH = Device/Head Register [HEX] DC = Device Command Register [HEX] ER = Error register [HEX] ST = Status register [HEX] Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days. Error 1 occurred at disk power-on lifetime: 5380 hours (224 days + 4 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 00 00 00 00 00 Error: UNC at LBA = 0x00000000 = 0 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- c8 00 08 c0 00 00 e0 00 2d+17:33:23.675 READ DMA ca 00 08 10 57 00 e0 00 2d+17:32:50.848 WRITE DMA c8 00 08 10 57 00 e0 00 2d+17:32:50.848 READ DMA ca 00 18 f8 56 00 e0 00 2d+17:32:50.817 WRITE DMA ca 00 08 f0 56 00 e0 00 2d+17:32:50.792 WRITE DMA SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 9012 - # 2 Short offline Completed without error 00% 5675 - # 3 Short offline Completed without error 00% 5649 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. root@Tower:~# The blue ball is weird and before rebooting would not pick up the disk. Swapping the SATA port made the disk come back "online", alebit unRAID thinks it's new. And yes, I know I should be moving to unRAID 6. The migration plan is for Feb. of next year, I plan on moving to 4TB drives. Thanks!
October 10, 201510 yr Well, there isn't really much difference between a parity drive that needs to be rebuilt, and a new parity drive. In both cases, you need to build parity.
October 10, 201510 yr Author Thanks for the reply. My confusion is... 1) Shouldn't it remember the serial? It's showing as a new disk; I figured it should remember the old parity data. 2) Is it safe to rebuild the parity?
October 10, 201510 yr Thanks for the reply. My confusion is... 1) Shouldn't it remember the serial? It's showing as a new disk; I figured it should remember the old parity data. 2) Is it safe to rebuild the parity? Since it wound up being set (by you?) as no device, as far as unRaid is concerned, it is a new disk, hence why it didn't remember it Yes its safe to rebuild.
October 13, 201510 yr As I'm sure you have heard a number of times, you HAVE to grab the syslog in order to know what actually happened, too late now. All we know is that the parity drive became inaccessible and was therefore disabled by the kernel at some point, and a reboot would make it reappear. It may or may not have been anything wrong with the drive, all indications are the drive was fine, so it's more likely a problem with the port or controller or power or a loose or vibrated connection. The actual cause should have been evident in the syslog for that session, before the reboot. Second SMART report (Disk 2) looks worrying, and I suspect that drive will not live very long. Its media surface has reported only one issue, but there are indications of general mechanical issues with the drive. There's a "FAILING NOW" for one attribute, but it's not a critical one. The one that is most concerning is how low the Seek_Error_Rate dropped, once to 45, and it IS a critical attribute. I'd keep a close eye on this drive, and a replacement ready. Hopefully you can get another year out of it. You don't want to upgrade the software AND the drives at the same time. Probably best to upgrade to v6 as soon as convenient for you (and gain the better SMART monitoring and notifications for your current drives), then upgrade the drives in February.
October 14, 201510 yr Author /usr/bin/tail -f /var/log/syslog Oct 14 18:11:11 Tower kernel: sd 2:0:0:0: [sdc] Unhandled error code Oct 14 18:11:11 Tower kernel: sd 2:0:0:0: [sdc] Oct 14 18:11:11 Tower kernel: Result: hostbyte=0x04 driverbyte=0x00 Oct 14 18:11:11 Tower kernel: sd 2:0:0:0: [sdc] CDB: Oct 14 18:11:11 Tower kernel: cdb[0]=0x8a: 8a 00 00 00 00 00 b2 57 99 a8 00 00 00 08 00 00 Oct 14 18:11:11 Tower kernel: end_request: I/O error, dev sdc, sector 2992085416 Oct 14 18:11:11 Tower kernel: md: disk1 write error, sector=2992085352 Oct 14 21:54:20 Tower dhcpcd[982]: eth0: renewing lease of 192.168.1.217 Oct 14 21:54:20 Tower dhcpcd[982]: eth0: acknowledged 192.168.1.217 from 192.168.1.1 Oct 14 21:54:20 Tower dhcpcd[982]: eth0: leased 192.168.1.217 for 86400 seconds This is on Disk #1. Am I having my disk go bad? It seems unlikely all 3 would be going bad at once but I have heard terrible things about these drives...
October 15, 201510 yr Possibly connection problem caused by disturbing cables, etc. We really need a complete syslog. And SMART reports.
October 15, 201510 yr Author Syslog is attached, smart reports for drives that are booted right now (one is red balled): root@Tower:~# smartctl -A /dev/sdb smartctl 6.2 2013-07-26 r3841 [i686-linux-3.9.11p-unRAID] (local build) Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org === START OF READ SMART DATA SECTION === SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 109 099 006 Pre-fail Always - 24294136 3 Spin_Up_Time 0x0003 094 094 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 098 098 020 Old_age Always - 2103 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 047 045 030 Pre-fail Always - 4857659502476 9 Power_On_Hours 0x0032 082 082 000 Old_age Always - 16201 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 46 183 Runtime_Bad_Block 0x0032 099 099 000 Old_age Always - 1 184 End-to-End_Error 0x0032 099 099 099 Old_age Always FAILING_NOW 1 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 099 000 Old_age Always - 1 1 1 189 High_Fly_Writes 0x003a 096 096 000 Old_age Always - 4 190 Airflow_Temperature_Cel 0x0022 065 057 045 Old_age Always - 35 (Min/Max 25/38) 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 46 193 Load_Cycle_Count 0x0032 081 081 000 Old_age Always - 38314 194 Temperature_Celsius 0x0022 035 043 000 Old_age Always - 35 (0 20 0 0 0) 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 3352h+32m+45.793s 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 18883194096 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 221884624187 root@Tower:~# root@Tower:~# smartctl -A /dev/sdd smartctl 6.2 2013-07-26 r3841 [i686-linux-3.9.11p-unRAID] (local build) Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org === START OF READ SMART DATA SECTION === SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 108 099 006 Pre-fail Always - 18712744 3 Spin_Up_Time 0x0003 095 094 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 833 5 Reallocated_Sector_Ct 0x0033 100 100 010 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 052 052 030 Pre-fail Always - 2499759130609 9 Power_On_Hours 0x0032 082 082 000 Old_age Always - 16179 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 45 183 Runtime_Bad_Block 0x0032 099 099 000 Old_age Always - 1 184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Command_Timeout 0x0032 100 100 000 Old_age Always - 0 0 0 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 065 058 045 Old_age Always - 35 (Min/Max 32/38) 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 45 193 Load_Cycle_Count 0x0032 078 078 000 Old_age Always - 44545 194 Temperature_Celsius 0x0022 035 042 000 Old_age Always - 35 (0 20 0 0 0) 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 5662h+53m+08.962s 241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 46247931720 242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 204267771471 root@Tower:~# Thanks for your help! syslog.txt
October 20, 201510 yr Disk1 was disabled due to write error (obviously). It will have to be rebuilt. Not clear to me exactly what the problem is though. Do you have backups? Technically, you are installing some of your user plugins from the system plugins folder, but that isn't the cause of your issues. You might consider booting in safe mode until you get things straight again though. You might also do a memtest. Probably not the issue either, but won't hurt. Since you don't have many drives I wouldn't think the power supply was the issue, but maybe something about the power connections. Are all drives on motherboard ports? Maybe check all SATA and power connections, both ends. Maybe try changing cables.
Archived
This topic is now archived and is closed to further replies.