December 12, 201411 yr My system is doing a parity check, and I happened to notice the last three lines of the syslog are: Dec 11 20:33:10 NAS shfs/user: shfs_write: write: (28) No space left on device Dec 11 20:48:03 NAS last message repeated 24 times Dec 11 20:56:41 NAS last message repeated 4 times Anything to worry about? Full log attached syslog-2014-12-11.zip
December 12, 201411 yr Not quite sure what that error means, but it does look like you've got some problems on disk 5... Looks like a cabling issue. A whack of disk 5 read errors followed by this: Dec 11 12:21:59 NAS kernel: md: disk5 read error, sector=1940380280 Dec 11 12:24:28 NAS kernel: ata6.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 Dec 11 12:24:28 NAS kernel: ata6.00: irq_stat 0x40000001 Dec 11 12:24:28 NAS kernel: ata6.00: failed command: READ DMA EXT Dec 11 12:24:28 NAS kernel: ata6.00: cmd 25/00:00:f8:43:a8/00:04:73:00:00/e0 tag 0 dma 524288 in Dec 11 12:24:28 NAS kernel: res 51/40:4c:ac:45:a8/00:00:73:00:00/00 Emask 0x9 (media error) Dec 11 12:24:28 NAS kernel: ata6.00: status: { DRDY ERR } Dec 11 12:24:28 NAS kernel: ata6.00: error: { UNC } Dec 11 12:24:28 NAS kernel: ata6.00: configured for UDMA/133
December 12, 201411 yr Author Yeah, I've been having issues with disk 5. In the 17.5 hours since a reboot, running basically nothing on the box but a parity check, it's already got > 400 errors. Before the reboot, it had more than 900 errors. I managed to get all the files transferred off the drive, so nothing should be in jeopardy, and I've got a new 4TB drive on its way, so I'll be replacing it shortly. Thanks for digging far enough through to find those errors. I'll reseat the cable at both ends (and check power while I'm in there), and see if it becomes salvageable. If not, it's a 1TB drive that's about 5 years old, and spent a couple of years running 24/7 in a WinXP box before making the jump to unRAID, so it's served well. Still open to any other ideas if anyone has 'em.
December 12, 201411 yr Not quite sure what that error means, but it does look like you've got some problems on disk 5... Looks like a cabling issue. A whack of disk 5 read errors followed by this: Dec 11 12:21:59 NAS kernel: md: disk5 read error, sector=1940380280 Dec 11 12:24:28 NAS kernel: ata6.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 Dec 11 12:24:28 NAS kernel: ata6.00: irq_stat 0x40000001 Dec 11 12:24:28 NAS kernel: ata6.00: failed command: READ DMA EXT Dec 11 12:24:28 NAS kernel: ata6.00: cmd 25/00:00:f8:43:a8/00:04:73:00:00/e0 tag 0 dma 524288 in Dec 11 12:24:28 NAS kernel: res 51/40:4c:ac:45:a8/00:00:73:00:00/00 Emask 0x9 (media error) Dec 11 12:24:28 NAS kernel: ata6.00: status: { DRDY ERR } Dec 11 12:24:28 NAS kernel: ata6.00: error: { UNC } Dec 11 12:24:28 NAS kernel: ata6.00: configured for UDMA/133 I thought (media error) meant the disk itself, not the cable. A smart report would be interesting.
December 12, 201411 yr Not quite sure what that error means, but it does look like you've got some problems on disk 5... Looks like a cabling issue. A whack of disk 5 read errors followed by this: Dec 11 12:21:59 NAS kernel: md: disk5 read error, sector=1940380280 Dec 11 12:24:28 NAS kernel: ata6.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x0 Dec 11 12:24:28 NAS kernel: ata6.00: irq_stat 0x40000001 Dec 11 12:24:28 NAS kernel: ata6.00: failed command: READ DMA EXT Dec 11 12:24:28 NAS kernel: ata6.00: cmd 25/00:00:f8:43:a8/00:04:73:00:00/e0 tag 0 dma 524288 in Dec 11 12:24:28 NAS kernel: res 51/40:4c:ac:45:a8/00:00:73:00:00/00 Emask 0x9 (media error) Dec 11 12:24:28 NAS kernel: ata6.00: status: { DRDY ERR } Dec 11 12:24:28 NAS kernel: ata6.00: error: { UNC } Dec 11 12:24:28 NAS kernel: ata6.00: configured for UDMA/133 I thought (media error) meant the disk itself, not the cable. A smart report would be interesting. Definately... I only saw the READ DMA EXT which tends to be cabling. Media error would mean drive itself
December 14, 201411 yr Author Thanks, guys. I ran a long SMART report from Dynamix, but don't remember where it puts the file, so here's the smart report as shown from unMenu. Point me in the right direction, and I'll post the full report (if this isn't everything). smartctl -a -d ata /dev/sdg (disk5) smartctl 6.2 2013-07-26 r3841 [i686-linux-3.9.11p-unRAID] (local build) Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org === START OF INFORMATION SECTION === Model Family: Hitachi Deskstar 7K1000.C Device Model: Hitachi HDS721010CLA332 Serial Number: JP2921HQ03M4KA LU WWN Device Id: 5 000cca 35dc1a4fd Firmware Version: JP4OA25C User Capacity: 1,000,204,886,016 bytes [1.00 TB] Sector Size: 512 bytes logical/physical Rotation Rate: 7200 rpm Device is: In smartctl database [for details use: -P show] ATA Version is: ATA8-ACS T13/1699-D revision 4 SATA Version is: SATA 2.6, 3.0 Gb/s Local Time is: Sun Dec 14 00:54:11 2014 EST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x84) Offline data collection activity was suspended by an interrupting command from host. Auto Offline Data Collection: Enabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 9694) seconds. Offline data collection capabilities: (0x5b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 162) minutes. SCT capabilities: (0x003d) SCT Status supported. SCT Error Recovery Control supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 16 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000b 093 093 016 Pre-fail Always - 1376270 2 Throughput_Performance 0x0005 136 136 054 Pre-fail Offline - 94 3 Spin_Up_Time 0x0007 124 124 024 Pre-fail Always - 313 (Average 299) 4 Start_Stop_Count 0x0012 100 100 000 Old_age Always - 2606 5 Reallocated_Sector_Ct 0x0033 100 100 005 Pre-fail Always - 146 7 Seek_Error_Rate 0x000b 100 100 067 Pre-fail Always - 0 8 Seek_Time_Performance 0x0005 130 130 020 Pre-fail Offline - 35 9 Power_On_Hours 0x0012 094 094 000 Old_age Always - 43173 10 Spin_Retry_Count 0x0013 100 100 060 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 97 192 Power-Off_Retract_Count 0x0032 098 098 000 Old_age Always - 2860 193 Load_Cycle_Count 0x0012 098 098 000 Old_age Always - 2860 194 Temperature_Celsius 0x0002 214 214 000 Old_age Always - 28 (Min/Max 18/56) 196 Reallocated_Event_Count 0x0032 100 100 000 Old_age Always - 146 197 Current_Pending_Sector 0x0022 100 100 000 Old_age Always - 8 198 Offline_Uncorrectable 0x0008 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x000a 200 200 000 Old_age Always - 0 SMART Error Log Version: 1 ATA Error Count: 176 (device log contains only the most recent five errors) CR = Command Register [HEX] FR = Features Register [HEX] SC = Sector Count Register [HEX] SN = Sector Number Register [HEX] CL = Cylinder Low Register [HEX] CH = Cylinder High Register [HEX] DH = Device/Head Register [HEX] DC = Device Command Register [HEX] ER = Error register [HEX] ST = Status register [HEX] Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days. Error 176 occurred at disk power-on lifetime: 43112 hours (1796 days + 8 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 4c ac 45 a8 00 Error: UNC 76 sectors at LBA = 0x00a845ac = 11027884 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 25 00 00 f8 43 a8 e3 08 27d+21:46:07.155 READ DMA EXT 25 00 60 98 43 a8 e3 08 27d+21:46:07.151 READ DMA EXT 25 00 00 98 3f a8 e0 08 27d+21:45:33.689 READ DMA EXT 25 00 00 98 3b a8 e0 08 27d+21:44:58.296 READ DMA EXT 25 00 00 98 37 a8 e0 08 27d+21:44:31.340 READ DMA EXT Error 175 occurred at disk power-on lifetime: 43112 hours (1796 days + 8 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 3d 83 d7 a7 00 Error: UNC 61 sectors at LBA = 0x00a7d783 = 10999683 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 25 00 00 c0 d6 a7 e0 08 27d+21:31:06.105 READ DMA EXT 25 00 00 c0 d2 a7 e0 08 27d+21:30:37.476 READ DMA EXT 25 00 40 80 d2 a7 e0 08 27d+21:30:35.031 READ DMA EXT 25 00 00 80 ce a7 e0 08 27d+21:30:13.470 READ DMA EXT 25 00 00 80 ca a7 e0 08 27d+21:29:42.255 READ DMA EXT Error 174 occurred at disk power-on lifetime: 43112 hours (1796 days + 8 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 2e 12 e9 a6 00 Error: UNC 46 sectors at LBA = 0x00a6e912 = 10938642 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 25 00 00 40 e8 a6 e0 08 27d+20:56:42.513 READ DMA EXT 25 00 00 40 e4 a6 e0 08 27d+20:56:06.101 READ DMA EXT 25 00 00 40 e0 a6 e0 08 27d+20:55:28.198 READ DMA EXT b0 da 00 00 4f c2 00 08 27d+20:55:27.552 SMART RETURN STATUS 25 00 00 40 dc a6 e0 08 27d+20:54:54.963 READ DMA EXT Error 173 occurred at disk power-on lifetime: 43112 hours (1796 days + 8 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 c3 8d fb 9b 00 Error: UNC 195 sectors at LBA = 0x009bfb8d = 10222477 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 25 00 00 50 fa 9b e0 08 27d+20:10:24.977 READ DMA EXT b0 d0 01 00 4f c2 00 08 27d+20:10:23.898 SMART READ DATA 25 00 00 50 f6 9b e0 08 27d+20:09:51.047 READ DMA EXT 25 00 00 50 f2 9b e0 08 27d+20:09:21.603 READ DMA EXT 25 00 00 50 ee 9b e0 08 27d+20:09:00.506 READ DMA EXT Error 172 occurred at disk power-on lifetime: 43112 hours (1796 days + 8 hours) When the command that caused the error occurred, the device was active or idle. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 40 51 86 8a 68 9b 00 Error: UNC 134 sectors at LBA = 0x009b688a = 10184842 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- 25 00 00 10 66 9b e0 08 27d+19:55:50.894 READ DMA EXT 25 00 00 10 62 9b e3 08 27d+19:55:40.785 READ DMA EXT 25 00 00 10 5e 9b e0 08 27d+19:55:22.130 READ DMA EXT b0 d1 01 01 4f c2 00 08 27d+19:55:21.461 SMART READ ATTRIBUTE THRESHOLDS [OBS-4] 25 00 00 10 5a 9b e0 08 27d+19:54:59.444 READ DMA EXT SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Extended offline Completed: read failure 10% 43097 1936133003 # 2 Extended offline Completed without error 00% 42088 - # 3 Short offline Completed without error 00% 42067 - # 4 Short offline Completed without error 00% 39655 - # 5 Extended offline Completed without error 00% 37130 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. As I said, I've managed to get all the files off it and I've excluded it from all my shares, so nothing new should be written to it. I ordered a new drive last Tuesday, so it'll be here as soon as Amazon gets around to shipping the darn thing out the door... EDIT 1: As a side note, the drive has been spinning for 4.92 years, so I'd say it's lived a pretty full life! EDIT 2: I guess that was the long SMART test results... Should I pull the drive from the array, logically, at least?
December 14, 201411 yr See here: http://lime-technology.com/wiki/index.php/Troubleshooting#Resolving_a_Pending_Sector
December 14, 201411 yr Author Sounds like a drive replacement is the best course of action, then. Once I've got that taken care of, I'll pre-clear this one again, and see how things look before deciding whether or not to put new data on it. Thanks, all, for your help.
Archived
This topic is now archived and is closed to further replies.