June 2, 200917 yr Hi Guys, I think i might have a failed disk... but it might be the way i've used unRAID. 4.4.2 with vmserver installed i setup a 4 disk array (no parity) then created a VM volume on each, joined them in the guest Server OS. and started to move data onto them over the network. all seemed fine. now i have the entire of my data on them, i tried to add a parity disk. Parity being generated fine working fine 50MB/s or so. I started my VM back up and noticed LOADS of errors form one of the disks. the VM does use this disk for media storage but it's not the VM's OS disk disk so i'd not expect much IO on that virtual volume. anyway, i've not stopped my VM. but i've an error rate of in the +1000 range. looking at the syslog Jun 2 05:24:11 Tower kernel: md: disk2 read error Jun 2 05:24:11 Tower kernel: handle_stripe read error: 16578272/2, count: 1 Jun 2 05:24:11 Tower kernel: md: disk2 read error Jun 2 05:24:11 Tower kernel: handle_stripe read error: 16578280/2, count: 1 Jun 2 05:24:11 Tower kernel: md: disk2 read error Jun 2 05:24:11 Tower kernel: handle_stripe read error: 16578288/2, count: 1 Jun 2 05:24:11 Tower kernel: md: disk2 read error Jun 2 05:24:11 Tower kernel: handle_stripe read error: 16578296/2, count: 1 Jun 2 05:24:11 Tower kernel: md: disk2 read error Jun 2 05:24:11 Tower kernel: handle_stripe read error: 16578304/2, count: 1 Jun 2 05:24:11 Tower kernel: md: disk2 read error Jun 2 05:24:11 Tower kernel: handle_stripe read error: 16578312/2, count: 1 Appears over and over. should i panic, or or assume that it's down to me having a VM with some active disk IO on whilst it was trying to create a parity disk (i dind't think this was a problem for unraid... ) I've now set it off again with no VM running to see what happens this time round. thanks for any advice.
June 2, 200917 yr Hi Guys, I think i might have a failed disk... but it might be the way i've used unRAID. 4.4.2 with vmserver installed i setup a 4 disk array (no parity) then created a VM volume on each, joined them in the guest Server OS. and started to move data onto them over the network. all seemed fine. now i have the entire of my data on them, i tried to add a parity disk. Parity being generated fine working fine 50MB/s or so. I started my VM back up and noticed LOADS of errors form one of the disks. the VM does use this disk for media storage but it's not the VM's OS disk disk so i'd not expect much IO on that virtual volume. anyway, i've not stopped my VM. but i've an error rate of in the +1000 range. looking at the syslog Jun 2 05:24:11 Tower kernel: md: disk2 read error Jun 2 05:24:11 Tower kernel: handle_stripe read error: 16578272/2, count: 1 Jun 2 05:24:11 Tower kernel: md: disk2 read error Jun 2 05:24:11 Tower kernel: handle_stripe read error: 16578280/2, count: 1 Jun 2 05:24:11 Tower kernel: md: disk2 read error Jun 2 05:24:11 Tower kernel: handle_stripe read error: 16578288/2, count: 1 Jun 2 05:24:11 Tower kernel: md: disk2 read error Jun 2 05:24:11 Tower kernel: handle_stripe read error: 16578296/2, count: 1 Jun 2 05:24:11 Tower kernel: md: disk2 read error Jun 2 05:24:11 Tower kernel: handle_stripe read error: 16578304/2, count: 1 Jun 2 05:24:11 Tower kernel: md: disk2 read error Jun 2 05:24:11 Tower kernel: handle_stripe read error: 16578312/2, count: 1 Appears over and over. should i panic, or or assume that it's down to me having a VM with some active disk IO on whilst it was trying to create a parity disk (i dind't think this was a problem for unraid... ) I've now set it off again with no VM running to see what happens this time round. thanks for any advice. Did you create the VM volume on the "/dev/md?" devices, or the /dev/sd?" devices? If you did it on the /dev/sd? devices you are bypassing the raid array and writing directly to the physical disks. This is guaranteed to cause parity errors... However... you are getting "read" errors..., not parity errors. That indicates to me a defective disk as it is being read in the attempt to calculate parity. Since you did not have a parity disk, the data disks would not have been "cleared" (all bits set to zero) and therefore, you would not have known about any bad sectors until now that you are attempting to read them. What does a smartctl report show for disk2? It will tell you if the disk hardware is having problems with bad sectors. Joe L.
June 2, 200917 yr Author VMs are created under on the data stores which are made \mnt\disk1 \mnt\disk2 etc is this protected through unraid? I have since stopped the VM, stopped the array and restarted the array - it reset it's error count when it did this, it again was fine for about 10 minutes then loads of errors - reached 3425 so i suspect there are some bad sections of the disk as the errors come in waves (it's been clean of errors for 20 minutes and the parity wirte speed it back up to 50MBs. i definatly susspect a dodgy disk purhaps the disk got moved whilst spinning and the head had damaged some of the platter hence the area of bad reads and then being ok again.
June 2, 200917 yr Author Joe, this is the report from that disk... smartctl version 5.38 [i486-slackware-linux-gnu] Copyright © 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/ === START OF INFORMATION SECTION === Model Family: Maxtor DiamondMax 10 family (SATA/300) Device Model: Maxtor 6V250F0 Serial Number: V59401YG Firmware Version: VA111680 User Capacity: 250,059,350,016 bytes Device is: In smartctl database [for details use: -P show] ATA Version is: 7 ATA Standard is: ATA/ATAPI-7 T13 1532D revision 0 Local Time is: Tue Jun 2 06:44:27 2009 GMT SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x80) Offline data collection activity was never started. Auto Offline Data Collection: Enabled. Self-test execution status: ( 118) The previous self-test completed having the read element of the test failed. Total time to complete Offline data collection: (2102) seconds. Offline data collection capabilities: (0x5b) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. Offline surface scan supported. Self-test supported. No Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 2) minutes. Extended self-test routine recommended polling time: ( 113) minutes. SCT capabilities: (0x0021) SCT Status supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 32 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 3 Spin_Up_Time 0x0027 192 187 063 Pre-fail Always - 20141 4 Start_Stop_Count 0x0032 253 253 000 Old_age Always - 283 5 Reallocated_Sector_Ct 0x0033 233 233 063 Pre-fail Always - 259 7 Seek_Error_Rate 0x000a 253 252 000 Old_age Always - 0 8 Seek_Time_Performance 0x0027 253 230 187 Pre-fail Always - 50774 9 Power_On_Hours 0x0032 236 236 000 Old_age Always - 6033 10 Spin_Retry_Count 0x002b 253 252 157 Pre-fail Always - 0 11 Calibration_Retry_Count 0x002b 253 252 223 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 253 253 000 Old_age Always - 309 189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0 190 Airflow_Temperature_Cel 0x0022 074 047 000 Old_age Always - 26 (Lifetime Min/Max 21/26) 192 Power-Off_Retract_Count 0x0032 253 253 000 Old_age Always - 0 193 Load_Cycle_Count 0x0032 253 253 000 Old_age Always - 0 194 Temperature_Celsius 0x0032 042 253 000 Old_age Always - 26 195 Hardware_ECC_Recovered 0x000a 253 252 000 Old_age Always - 110 196 Reallocated_Event_Count 0x0008 253 253 000 Old_age Offline - 0 197 Current_Pending_Sector 0x0008 249 237 000 Old_age Offline - 62 198 Offline_Uncorrectable 0x0008 252 252 000 Old_age Offline - 1 199 UDMA_CRC_Error_Count 0x0008 199 199 000 Old_age Offline - 0 200 Multi_Zone_Error_Rate 0x000a 253 252 000 Old_age Always - 0 201 Soft_Read_Error_Rate 0x000a 253 252 000 Old_age Always - 0 202 TA_Increase_Count 0x000a 253 250 000 Old_age Always - 0 203 Run_Out_Cancel 0x000b 253 252 180 Pre-fail Always - 0 204 Shock_Count_Write_Opern 0x000a 253 252 000 Old_age Always - 0 205 Shock_Rate_Write_Opern 0x000a 253 252 000 Old_age Always - 0 207 Spin_High_Current 0x002a 253 252 000 Old_age Always - 0 208 Spin_Buzz 0x002a 253 252 000 Old_age Always - 0 210 Unknown_Attribute 0x0032 253 252 000 Old_age Always - 0 211 Unknown_Attribute 0x0032 253 252 000 Old_age Always - 0 212 Unknown_Attribute 0x0032 253 252 000 Old_age Always - 0 SMART Error Log Version: 1 ATA Error Count: 16218 (device log contains only the most recent five errors) CR = Command Register [HEX] FR = Features Register [HEX] SC = Sector Count Register [HEX] SN = Sector Number Register [HEX] CL = Cylinder Low Register [HEX] CH = Cylinder High Register [HEX] DH = Device/Head Register [HEX] DC = Device Command Register [HEX] ER = Error register [HEX] ST = Status register [HEX] Powered_Up_Time is measured from power on, and printed as DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes, SS=sec, and sss=millisec. It "wraps" after 49.710 days. Error 16218 occurred at disk power-on lifetime: 5656 hours (235 days + 16 hours) When the command that caused the error occurred, the device was in an unknown state. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 00 50 00 00 00 00 a0 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- ec 00 00 00 00 00 a0 00 00:54:42.437 IDENTIFY DEVICE 25 00 00 5f 50 10 e0 00 00:54:39.477 READ DMA EXT ec 00 00 00 00 00 a0 00 00:54:39.457 IDENTIFY DEVICE ef 03 42 00 00 00 a0 00 00:54:39.437 SET FEATURES [set transfer mode] ec 00 00 00 00 00 a0 00 00:54:39.320 IDENTIFY DEVICE Error 16217 occurred at disk power-on lifetime: 5656 hours (235 days + 16 hours) When the command that caused the error occurred, the device was in an unknown state. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 00 50 00 00 00 00 a0 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- ec 00 00 00 00 00 a0 00 00:54:39.320 IDENTIFY DEVICE 25 00 00 5f 50 10 e0 00 00:54:36.406 READ DMA EXT ec 00 00 00 00 00 a0 00 00:54:36.386 IDENTIFY DEVICE ef 03 42 00 00 00 a0 00 00:54:36.366 SET FEATURES [set transfer mode] ec 00 00 00 00 00 a0 00 00:54:36.252 IDENTIFY DEVICE Error 16216 occurred at disk power-on lifetime: 5656 hours (235 days + 16 hours) When the command that caused the error occurred, the device was in an unknown state. After command completion occurred, registers were: ER ST SC SN CL CH DH -- -- -- -- -- -- -- 00 50 00 00 00 00 a0 Commands leading to the command that caused the error were: CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name -- -- -- -- -- -- -- -- ---------------- -------------------- ec 00 00 00 00 00 a0 00 00:54:36.252 IDENTIFY DEVICE 25 00 00 5f 50 10 e0 00 00:54:33.295 READ DMA EXT ec 00 00 00 00 00 a0 00 00:54:33.275 IDENTIFY DEVICE ef 03 42 00 00 00 a0 00 00:54:33.255 SET FEATURES [set transfer mode] ec 00 00 00 00 00 a0 00 00:54:33.168 IDENTIFY DEVICE SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed: read failure 60% 6024 16555602 # 2 Selective offline Completed without error 00% 1 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 39062500 Not_testing 2 449334667 488397167 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay.
June 2, 200917 yr VMs are created under on the data stores which are made \mnt\disk1 \mnt\disk2 etc is this protected through unraid? Yes, it is, you are fine as far as parity goes. I have since stopped the VM, stopped the array and restarted the array - it reset it's error count when it did this, it again was fine for about 10 minutes then loads of errors - reached 3425 so i suspect there are some bad sections of the disk as the errors come in waves (it's been clean of errors for 20 minutes and the parity wirte speed it back up to 50MBs. i definatly susspect a dodgy disk purhaps the disk got moved whilst spinning and the head had damaged some of the platter hence the area of bad reads and then being ok again. Your SMART report shows a sick disk. It has already reallocated 259 sectors 5 Reallocated_Sector_Ct 0x0033 233 233 063 Pre-fail Always - 259 And it has another 62 pending relocation once they are eventually written to Current_Pending_Sector 0x0008 249 237 000 Old_age Offline - 62 And the short smart test aborted at the 60% point on errors # 1 Short offline Completed: read failure 60% 6024 16555602 I'd RMA it.
June 3, 200917 yr Author Joe, As always thankyou so much for you help! I'm going to throw that HDD in the bin. It's too old to RMA. I've run tests on each of my other HDDs. I see Old_age on all of them - should i be worried about this? none of the "Reallocated_Sector_Ct" values are anything other than 0. none of the Current_Pending_Secotrs" also nothing other than 0. I've also had no more errors since i took the damaged disk out of the array this AM. Thanks very much Joe.
June 3, 200917 yr I've run tests on each of my other HDDs. I see Old_age on all of them - should i be worried about this? Just indicates they've been spinning for a while. If no other errors, I'd say leave them. At some point it will make sense to retire them, larger single drives become cheep. none of the "Reallocated_Sector_Ct" values are anything other than 0. none of the Current_Pending_Secotrs" also nothing other than 0. That's good... I've also had no more errors since i took the damaged disk out of the array this AM. Thanks very much Joe. Also good. Did you replace the drive with another? And rebuilt onto it? or Did you just un--assign it and the indicator is now "red" or did you un-assign it and then press "Restore" to calculate new parity without it? If the indicator is "red" on that drive, even if un-assigned, then you do not have parity protection... not until you either replace it, or pres the button labeled "restore" to save a new configuration and calculate parity on the new configuration. Joe L.
June 3, 200917 yr Author stopped the array, un-assigned disk 2 (the dead one) ticked the restore button to bring the arraay back online and then i think checked check parity. can't be 100% sure on the last step.... i read the info on the side so i htink i got it right. it took about 4 or more hours to re-calculate the parity so i think im covered. (thanks for checking!)
Archived
This topic is now archived and is closed to further replies.