January 7, 201313 yr Hey guys, i've been having a wierd issue with my unraid box, where (seemingly at random) a drive (always model Seagate 3TB SV35.5 "ST3000VX000") will start issuing errors. Syslog will start to show something like... kernel: md: disk1: ATA_OP e0 ioctl error: -5 mdcmd: write: Input/output error kernel: mdcmd (121) at this point, the drive is still Green Balled and reporting no issues in the web interface. if i stop the array, the drive will redball and say that its missing (and its missing from the drop down menu) Rebooting the system will detect the missing drive (Shows in the drop down); but the array is stopped saying that disk is DISK_DSBL. This has happened twice before, each time with a different Seagate ST3000vx000 (3tb) drive (each drive has been connected to a different controller/cable/power connector), i have replaced the drive each time, and the raid has rebuilt it, and i've never thought too much of it. but this is the third time, so i figured i need to look into it. My configuration is.. Unraid Pro ver: 5.0-rc8a, installed on a Kingston DT_101 8gb USB drive motherboard: Asus p8z77-v LK CPU: intel i3-2120 Memory: 16GB (4x4gb) SAS Controllers: 3x Supermicro AOC-SAS2LP-MV8 Power supply: 850watt PSU i have a mix of drives in my machine, mostly Seagates, only a handful of 3TB SV35.5 though. The smart status is showing the following. smartctl version 5.38 [i486-slackware-linux-gnu] Copyright © 2002-8 Bruce Allen Home page is http://smartmontools.sourceforge.net/ === START OF INFORMATION SECTION === Device Model: ST3000VX000-9YW166 Serial Number: [cut] Firmware Version: CV13 User Capacity: 3,000,592,982,016 bytes Device is: Not in smartctl database [for details use: -P showall] ATA Version is: 8 ATA Standard is: ATA-8-ACS revision 4 Local Time is: Mon Jan 7 16:46:10 2013 EST SMART support is: Available - device has SMART capability. SMART support is: Enabled === START OF READ SMART DATA SECTION === SMART overall-health self-assessment test result: PASSED General SMART Values: Offline data collection status: (0x00) Offline data collection activity was never started. Auto Offline Data Collection: Disabled. Self-test execution status: ( 0) The previous self-test routine completed without error or no self-test has ever been run. Total time to complete Offline data collection: ( 592) seconds. Offline data collection capabilities: (0x73) SMART execute Offline immediate. Auto Offline data collection on/off support. Suspend Offline collection upon new command. No Offline surface scan supported. Self-test supported. Conveyance Self-test supported. Selective Self-test supported. SMART capabilities: (0x0003) Saves SMART data before entering power-saving mode. Supports SMART auto save timer. Error logging capability: (0x01) Error logging supported. General Purpose Logging supported. Short self-test routine recommended polling time: ( 1) minutes. Extended self-test routine recommended polling time: ( 255) minutes. Conveyance self-test routine recommended polling time: ( 2) minutes. SCT capabilities: (0x10b9) SCT Status supported. SCT Feature Control supported. SCT Data Table supported. SMART Attributes Data Structure revision number: 10 Vendor Specific SMART Attributes with Thresholds: ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE 1 Raw_Read_Error_Rate 0x000f 116 099 006 Pre-fail Always - 115549888 3 Spin_Up_Time 0x0003 094 093 000 Pre-fail Always - 0 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 278 5 Reallocated_Sector_Ct 0x0033 100 100 036 Pre-fail Always - 0 7 Seek_Error_Rate 0x000f 072 060 030 Pre-fail Always - 15450467 9 Power_On_Hours 0x0032 099 099 000 Old_age Always - 1169 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 25 184 Unknown_Attribute 0x0032 100 100 099 Old_age Always - 0 187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0 188 Unknown_Attribute 0x0032 100 100 000 Old_age Always - 0 189 High_Fly_Writes 0x003a 001 001 000 Old_age Always - 159 190 Airflow_Temperature_Cel 0x0022 067 056 045 Old_age Always - 33 (Lifetime Min/Max 25/43) 191 G-Sense_Error_Rate 0x0032 100 100 000 Old_age Always - 0 192 Power-Off_Retract_Count 0x0032 100 100 000 Old_age Always - 17 193 Load_Cycle_Count 0x0032 100 100 000 Old_age Always - 1919 194 Temperature_Celsius 0x0022 033 044 000 Old_age Always - 33 (0 21 0 0) 197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0 198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0 199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0 SMART Error Log Version: 1 No Errors Logged SMART Self-test log structure revision number 1 Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error # 1 Short offline Completed without error 00% 1169 - SMART Selective self-test log data structure revision number 1 SPAN MIN_LBA MAX_LBA CURRENT_TEST_STATUS 1 0 0 Not_testing 2 0 0 Not_testing 3 0 0 Not_testing 4 0 0 Not_testing 5 0 0 Not_testing Selective self-test flags (0x0): After scanning selected spans, do NOT read-scan remainder of disk. If Selective self-test is pending on power-up, resume after 0 minute delay. Does anyone have any ideas? smart.log.txt
January 10, 201313 yr Author so i've caught it doing it again; i checked the server today, everything was green balled (but the drives were sleeping), i checked the syslog, and it was reporting the same errors before Jan 9 08:47:12 STORE emhttp: mdcmd: write: Input/output error Jan 9 08:47:12 STORE kernel: mdcmd (276): spindown 17 Jan 9 08:47:12 STORE kernel: md: disk17: ATA_OP e0 ioctl error: -5 Jan 9 08:47:22 STORE emhttp: mdcmd: write: Input/output error Jan 9 08:47:22 STORE kernel: mdcmd (277): spindown 17 Jan 9 08:47:22 STORE kernel: md: disk17: ATA_OP e0 ioctl error: -5 Jan 9 08:47:32 STORE emhttp: mdcmd: write: Input/output error Jan 9 08:47:32 STORE kernel: mdcmd (278): spindown 17 Jan 9 08:47:32 STORE kernel: md: disk17: ATA_OP e0 ioctl error: -5 Jan 9 08:47:42 STORE emhttp: mdcmd: write: Input/output error Jan 9 08:47:42 STORE kernel: mdcmd (279): spindown 17 Jan 9 08:47:42 STORE kernel: md: disk17: ATA_OP e0 ioctl error: -5 these were the same messages it was spamming before when the drive failed, previously i stopped the array, and that caused the drive to redball, this time i decided to 'Spinup all disks' to see what it would do. Then it displayed (sdy is disk17) Jan 9 20:08:19 STORE kernel: md: disk17: ATA_OP e3 ioctl error: -5 Jan 9 20:08:19 STORE kernel: mdcmd (4331): spinup 20 Jan 9 20:08:19 STORE ata_id[20757]: HDIO_GET_IDENTITY failed for '/dev/sdy' Jan 9 20:08:19 STORE kernel: program smartctl is using a deprecated SCSI ioctl, please convert it to SG_IO All drive spun up and showed a solid green ball, shortly after that, Disk17 redballed and then Disk8 also redballed. Jan 9 20:30:16 STORE kernel: mdcmd (4390): spinup 20 Jan 9 20:30:25 STORE kernel: md: disk8 read error Jan 9 20:30:25 STORE kernel: handle_stripe read error: 36112/8, count: 1 Jan 9 20:30:25 STORE kernel: Buffer I/O error on device md8, logical block 4514 Jan 9 20:30:25 STORE kernel: lost page write due to I/O error on md8 Jan 9 20:30:25 STORE kernel: md: disk8 read error Jan 9 20:30:25 STORE kernel: handle_stripe read error: 36120/8, count: 1 Jan 9 20:30:25 STORE kernel: Buffer I/O error on device md8, logical block 4515 Jan 9 20:30:25 STORE kernel: lost page write due to I/O error on md8 I have powered the unraid box down, and powered it back up, DISK8 was detected (and shows green), and so was DISK17 (and shows RED), i have replaced disk17 with a fresh 3TB Seagate SV35.5 drive and the array is rebuilding. any suggestions? or ideas? i went through my notes and this has actually happened 5 times so far each on an Seagate 3TB SV35.5 drive. i have posted the syslog, and i cut out a bunch of the repeated error messages just to keep the log small for posting. thanks! syslog.20130109.txt
January 10, 201313 yr Is there anything in common between the drives that exhibit this problem, for example on the same controller? I guess you're using all the PCI-E slots on that motherboard for disk controllers, correct? It's possible there is an issue with a disk controller being plugged into a certain slot.
January 10, 201313 yr Author Hi there, the power supply is an 850watt Silverstone; which was brand new back in september 2011 (along with most of the other hardware).
January 10, 201313 yr Author Is there anything in common between the drives that exhibit this problem, for example on the same controller? I guess you're using all the PCI-E slots on that motherboard for disk controllers, correct? It's possible there is an issue with a disk controller being plugged into a certain slot. i have 21 disks, all on the SuperMicro PCIE Controller cards, Disk1 (the original disk in the post), was on Controller1(port0), Disk17 is on Controller3(port1), Disk8 is on Controller1(port1). the 5 failures i've had on these SV35.5 drives have happened on almost all controllers, and all ports. Nothing is following a pattern; except that all of the drives that have exhibited this behavior are the same model. I'm wondering if these drives issue strange SMART responses, and are being falsely detected as failing...
January 10, 201313 yr The relevant and significant error messages are: Jan 9 08:28:50 STORE kernel: ata30.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen ... (timeout on sdy Disk 17 ) Jan 9 08:29:11 STORE kernel: ata30.00: disabled Jan 9 20:28:22 STORE kernel: ata7.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen ... (timeout on sdb Disk 8 ) Jan 9 20:28:43 STORE kernel: ata7.00: disabled All subsequent error messages, including the ones you quoted above can be ignored, once the drive is disabled at a low level. It takes a while for the higher level functions to recognize that the drive is no longer there. Once the kernel disables a drive, it is essentially gone until the next boot, so UnRAID will mark it Disabled also. In my experience, almost all such situations are NOT the fault of the drive, and in fact there is probably nothing at all wrong with the drive. And I don't believe SMART was directly involved, although may have been the first command not responded to, which triggered the drive recovery process. Rather shockingly I think, both drives were disabled in only 21 seconds(!), after the kernel detected that each drive had stopped responding. My memory is somewhat vague, but I believe we had a couple of rare cases where certain drive model and controller or driver combinations would occasionally fail to communicate or other weird behavior. The only solution I think was to wait for newer drivers or newer firmwares. I checked and so far there is no new firmware for those drives, so I believe your only recourse may be to try each of the latest Linux kernels (in the latest UnRAID releases). Sometimes a newer driver will include a workaround for a specific drive or controller. Backup first of course so you can revert, if necessary. Just a note, please do not truncate or edit the syslog file. We prefer it intact, but zipped, as these syslog files compress very small. I know you were trying to be helpful, and thank you for that. Edit: the SMART report above looks fine, drive looks good.
January 10, 201313 yr Edit: the SMART report above looks fine, drive looks good. You can not trust SMART to tell you anything believable about hard drives.
January 11, 201313 yr I was diagnosing with Tom a disconnecting disk issue for 2-weeks. This may not be the same issues, but what we do have in common is the use of Supermicro AOC-SAS2LP-MV8 and rc8a and disconnecting drives. I could reproduce the communication disconnect by running a parity check, at which time a random drive on any controller (supermicro or mobo or stlabs) would red-ball and the drive would be gone when I stopped the array, but a reboot would connect the drive an let me do a rebuild. My solution was to put the supermicro into a different PCI-E slot and voila, parity sync successful no more disconnecting drives. It could also have been that my supermicro card wasn't seated quite properly, so first I would try unplugging, cleaning, then re-inserting all your Supermicro AOC-SAS2LP-MV8 cards. If you can avoid writing to your server for a while, and if you trust that your drives are not failing, then you can save a copy of your super.dat from your flash, and when a drive red-balls you can replace the super.dat with the backup and it will restore the drive to original - green. Hopefully you can reproduce the disk disconnection, so backup your whole flash (in the future you can do just super.dat but always good to have full flash too) then run a non-correcting parity check in maintenance mode. See if a disk red-balls. To try and see if you have a problem with a supermicro card, can you go down to 2 super micro cards and use the SATA on your mobo? That way you could free up a PCI-E slot, then keep running parity checks leaving each slot open, and hopefully leaving one of the 3 slots open will yield a successful parity. This is of course assuming you can reproduce your drive disconnect during a parity sync. If you cannot, then you will just have to leave the server running for a period of time with each PCI-E open until it fails, or you are satisfied that it is working. My issue ended up being the particular AOC-SAS2LP-MV8 in the 1 particular slot, I have now picked up a second AOC-SAS2LP-MV8 for testing, and I can use it in the previously "bad" slot without error. If you cannot free up a PCI-E slot for this testing and you still want to test this path, then you will need a new clean, pre-cleared parity drive. (if you don't have one, you should always have a pre-cleared drive of equal or greater size than your parity on hand anyways, so get one). Pull your existing parity drive and set it aside, then use the new parity drive with a sub-set of your data drives using just 1 or 2 of the PCI-E cards (1 might make testing faster) and pull the other cards out of the PCI-E slots. Then you can build new parity and then run parity checks or just use the server with just a single Supermicro AOC-SAS2LP-MV8 in use, and see if the errors return, and if they do, switch the PCI-E slot the Supermicro AOC-SAS2LP-MV8 is in, and repeat.
January 12, 201313 yr Author What is really strange, is that my parity drive, which is also the same SV3.5 model that is causing me problems, has never disconnected on its own (like all of the other drives), i think almost ALL of the SV35.5 drives i have, i have had to replace (except the parity drive).
January 12, 201313 yr It might not be the drive model causing the issue. I never had my parity drive disconnect out of 10+ disconnects either. Rory
Archived
This topic is now archived and is closed to further replies.