June 29, 201214 yr I'm running an array using 2 Supermicro AOC-SAT2-MV8 8-port SATA Controllers, a Corsair TX 750 PSU, 8 WD green drives on Unraid version 4.7 I recently had a hard drive fail and replaced successfully: http://lime-technology.com/forum/index.php?topic=21042.msg187040#msg187040 I also purchased an additional backup drive but have had nothing but problems getting one to successfully preclear. 4 of 5 drives I have purchased have produced similar errors in my syslog, rawread error rates and pending sector counts. I'm starting to think this is not an issue with the drives and something wrong with my hardware. (I did however get a clear syslog and smart report from the drive I used to replace my failed disk) I have tried using different sata ports (both directly from the board and off a controller) and checked all the connections. Smart report and syslog below taken about 15 hours into a preclear: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 36 3 Spin_Up_Time 0x0027 253 192 021 Pre-fail Always - 1116 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 37 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 15 10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 35 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 34 193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 39 194 Temperature_Celsius 0x0022 120 117 000 Old_age Always - 30 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 1 198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age Offline - 0 Does this look like a bad drive or some kind of hardware issue? If hardware what should my next steps be? syslog-2012-06-29.txt
June 29, 201214 yr Drives purchased at the same time from the same vendor had many things in common throughout their life, and easily can be all bad. Most of the time they were probably all right when they left the manufacturer, but had trauma during shipment. Fedups has a bad habit of drop kicking the odd box if it happens to be stacked too high, pushed to the side, etc. Some drive shipments I receive from my supplier I have to test every single drive, because they just weren't packed correctly. For your sanity, I suggest temporarily setting up another PC to use as a preclear station, and test the drives in that prior to putting them in your main rig. That way if there is something up with your main unraid box, you can start troubleshooting there instead of beating your head against a bunch of possibly bad (or not) drives.
June 29, 201214 yr For your sanity, I suggest temporarily setting up another PC to use as a preclear station, and test the drives in that prior to putting them in your main rig. That way if there is something up with your main unraid box, you can start troubleshooting there instead of beating your head against a bunch of possibly bad (or not) drives. Good advice from jonathanm. Also, preclear is a good tool, but badblocks is a better tool to exercise your drive. It does 4 pattern tests of 0xff, 0x0a, 0xa0, 0x00 Here is my badblocks.sh script. #!/bin/bash mkdir /boot/logs OP="/boot/logs/${1#/dev/disk/by-id/}.badblocks" badblocks -vs -w -o ${OP} $1 It will do a 4 pass write test on the drive. example use; ./badblocks.sh /dev/disk/by-id/scsi-SATA_WDC_WD10EADS-00_WD-WCAU4C985230 Creates a log report of what blocks are bad in /boot/logs/scsi-SATA_WDC_WD10EADS-00_WD-WCAU4C985230 Note change scsi-SATA_WDC_WD10EADS-00_WD-WCAU4C985230 to the path of your drive without the part segment. Then compare the starting and ending smart reports. Note, this will take a long time, but it will fully exercise the drive.
June 29, 201214 yr Author Drives purchased at the same time from the same vendor had many things in common throughout their life, and easily can be all bad. Most of the time they were probably all right when they left the manufacturer, but had trauma during shipment. Fedups has a bad habit of drop kicking the odd box if it happens to be stacked too high, pushed to the side, etc. Some drive shipments I receive from my supplier I have to test every single drive, because they just weren't packed correctly. For your sanity, I suggest temporarily setting up another PC to use as a preclear station, and test the drives in that prior to putting them in your main rig. That way if there is something up with your main unraid box, you can start troubleshooting there instead of beating your head against a bunch of possibly bad (or not) drives. I've tried setting up my other PCs for unraid but none of them see to want to boot from the flash drive. Just reading the FAQ it seems like a mammoth set of troubleshooting to figure out why not. Anyone know of a disk checker I can run from windows?
June 29, 201214 yr if they are wd drives, you can download the data lifeguard utilities from wd. that will run tests against the drive. not sure its as detailed or as strenuous as a preclear though.
June 30, 201214 yr 4 of 5 drives I have purchased have produced similar errors in my syslog, rawread error rates and pending sector counts. I'm starting to think this is not an issue with the drives and something wrong with my hardware. Please ignore completely the RAW values for Raw_Read_Error_Rate and Seek_Error_Rate, look only at the VALUE number. They should normally be 100 or a little higher, or 200 or a little higher (unless Seagates, then Seek_Error_Rate is typically in the 60's). As you can see, yours are perfect. The ONLY issue apparent is the Current_Pending_Sector count of one, which just means it has found one questionable sector. When written to, it will be tested and if bad, remapped to a spare sector. Wait and check the final Preclear report for what happened to it. I have tried using different sata ports (both directly from the board and off a controller) and checked all the connections. Smart report and syslog below taken about 15 hours into a preclear: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 36 3 Spin_Up_Time 0x0027 253 192 021 Pre-fail Always - 1116 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 37 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 15 10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 35 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 34 193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 39 194 Temperature_Celsius 0x0022 120 117 000 Old_age Always - 30 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 1 198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age Offline - 0 Does this look like a bad drive or some kind of hardware issue? If hardware what should my next steps be? Not a bad hard drive so far. Your syslog shows the system is having trouble with the interrupt assigned to this port, constantly indicating "lost interrupt", which is rather strange. I recommend checking your BIOS settings for the onboard SATA ports on the next boot, and changing them to a native SATA mode, especially AHCI if available, anything but IDE Emulation. At the end of your syslog, media errors are being reported on the new drive, which probably correspond to the Current Pending sector. Again, let's see what the final Preclear report looks like.
June 30, 201214 yr You can try booting riplinux which has other rescue tools in it. Use badblocks which can do write/read tests on your drive. I've had a number of drives come back to a useable state after the 4 pass write/read test.
June 30, 201214 yr Author Final smart report and preclear attached ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 200 200 051 Pre-fail Always - 36 3 Spin_Up_Time 0x0027 253 192 021 Pre-fail Always - 1108 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 56 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 38 10 Spin_Retry_Count 0x0032 100 253 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 253 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 54 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 53 193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 210 194 Temperature_Celsius 0x0022 123 117 000 Old_age Always - 27 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 2 198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age Offline - 0
June 30, 201214 yr Although success is reported, I'd not consider that a successful pre-clear. Repeat pre clear until there are no pending sectors and the reallocated sector count is stable.
June 30, 201214 yr Author Not a bad hard drive so far. Your syslog shows the system is having trouble with the interrupt assigned to this port, constantly indicating "lost interrupt", which is rather strange. I recommend checking your BIOS settings for the onboard SATA ports on the next boot, and changing them to a native SATA mode, especially AHCI if available, anything but IDE Emulation. These are the options I have in BIOS settings: Native Mode Operation: AUTO or serial ATA (set to auto) Sata RAID Enable: DISABLED SATA AHCI Enable: ENABLED (Was previously disabled) SATA AHCI Legacy: DISABLED (Enabling turns off 2 of my sata ports) Made the change and will attempt another preclear? EDIT: With AHCI enabled the server doesnt boot. After it detects the drives the system hangs. I had to change it back. Any other ideas?
June 30, 201214 yr Author Set Native Mode to serial ATA. Changing that allowed me to boot with AHCI enabled as well. Running preclear now
July 1, 201214 yr Some additional info about lost interrupts, think I'll keep it handy for others too. ata lost interrupts (Status 0x50) While I suggested looking for a better SATA mode, hoping for AHCI, because it is considered optimal for SATA drives, the main reason I hoped you could change that was so a different driver would be used, and therefore probably a different IRQ assigned. But I studied your syslog again today, and I don't see any conflicts with the IRQ used by that new drive, so I still do not know why it is missing interrupts. Hoping that using AHCI support will improve that... I was curious about "SATA AHCI Legacy", don't remember seeing it before. I found a link here, basically confirming your discovery that if enabled, it disables SATA ports 5 and 6. Obviously something we will want to warn users about - leave this setting Disabled!
July 1, 201214 yr Author Update: I swapped the drive back to a sata port on my AOC-SAT2-MV8 controller just to see what would happen for the preclear. During the night a parity check was also scheduled and unfortunately my system crashed. I did pull a syslog before I left for the evening and it appears my interrupt errors are still at large. Attached is the most recent syslog as well as the smart report from the drive (its ugly but im assuming most of this is caused from the AOC-SAT2-MV8) I am now letting the server finish up the parity check and I will begin another preclear on this disk from a motherboard sata port. ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x002f 199 199 051 Pre-fail Always - 2163 3 Spin_Up_Time 0x0027 253 176 021 Pre-fail Always - 1950 4 Start_Stop_Count 0x0032 100 100 000 Old_age Always - 189 5 Reallocated_Sector_Ct 0x0033 200 200 140 Pre-fail Always - 0 7 Seek_Error_Rate 0x002e 100 253 000 Old_age Always - 0 9 Power_On_Hours 0x0032 100 100 000 Old_age Always - 62 10 Spin_Retry_Count 0x0032 100 100 000 Old_age Always - 0 11 Calibration_Retry_Count 0x0032 100 100 000 Old_age Always - 0 12 Power_Cycle_Count 0x0032 100 100 000 Old_age Always - 187 192 Power-Off_Retract_Count 0x0032 200 200 000 Old_age Always - 180 193 Load_Cycle_Count 0x0032 200 200 000 Old_age Always - 253 194 Temperature_Celsius 0x0022 122 109 000 Old_age Always - 28 196 Reallocated_Event_Count 0x0032 200 200 000 Old_age Always - 0 197 Current_Pending_Sector 0x0032 200 200 000 Old_age Always - 16 198 Offline_Uncorrectable 0x0030 100 253 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x0032 200 200 000 Old_age Always - 0 200 Multi_Zone_Error_Rate 0x0008 100 253 000 Old_age Offline - 0 syslog-2012-07-01.zip
July 2, 201214 yr Just a quick response, no more lost interrupts, now that it is using AHCI. That disk is having trouble though, looks like some bad sectors, possibly other issues. More later ... (if I or anyone else does not get back to you soon, I would Preclear it) And something I forgot to mention earlier, your router (or whatever is serving as your DHCP server) is configured for renewals every 30 minutes. If you have reasons to force renewal that quickly, then no problem, but I suspect you may not have realized that. I would change it to once a week or once a day. That will cut some of the repetition from your syslog.
July 2, 201214 yr OT but I found this in your syslog (really amazing! outstanding work from NTP people!) : Jun 30 18:00:00 Tower kernel: Clock: inserting leap second 23:59:60 UTC
July 2, 201214 yr Author Just a quick response, no more lost interrupts, now that it is using AHCI. That disk is having trouble though, looks like some bad sectors, possibly other issues. More later ... (if I or anyone else does not get back to you soon, I would Preclear it) And something I forgot to mention earlier, your router (or whatever is serving as your DHCP server) is configured for renewals every 30 minutes. If you have reasons to force renewal that quickly, then no problem, but I suspect you may not have realized that. I would change it to once a week or once a day. That will cut some of the repetition from your syslog. Ok well no surprises there. I suspect most of my problems are due to this batch of drives being junk. I started running another preclear on it but it was already heading for error city crashing. I'll swap this drive again, possibly with another brand and hope that clears things up. I'm using a Airport Express as my router and forgive my newbieness but I have no idea what that is. Any advice on how I might go about changing it would be very appreciated
July 3, 201214 yr Author Traded in the 2TB WD drive for a 1TB and successfully ran a preclear with no errors in my syslog and no other problems. Issue appears to be solved. Thanks so much everyone for all the help! (Also managed to change my DHCP renewal time)
October 17, 201213 yr For your sanity, I suggest temporarily setting up another PC to use as a preclear station, and test the drives in that prior to putting them in your main rig. That way if there is something up with your main unraid box, you can start troubleshooting there instead of beating your head against a bunch of possibly bad (or not) drives. Good advice from jonathanm. Also, preclear is a good tool, but badblocks is a better tool to exercise your drive. It does 4 pattern tests of 0xff, 0x0a, 0xa0, 0x00 Here is my badblocks.sh script. #!/bin/bash mkdir /boot/logs OP="/boot/logs/${1#/dev/disk/by-id/}.badblocks" badblocks -vs -w -o ${OP} $1 It will do a 4 pass write test on the drive. example use; ./badblocks.sh /dev/disk/by-id/scsi-SATA_WDC_WD10EADS-00_WD-WCAU4C985230 Creates a log report of what blocks are bad in /boot/logs/scsi-SATA_WDC_WD10EADS-00_WD-WCAU4C985230 Note change scsi-SATA_WDC_WD10EADS-00_WD-WCAU4C985230 to the path of your drive without the part segment. Then compare the starting and ending smart reports. Note, this will take a long time, but it will fully exercise the drive. Can someone post a badblocks.sh script or point me into the right direction of how to create one myself.
Archived
This topic is now archived and is closed to further replies.