August 15, 201213 yr After 1,5 months of happy running I got a failure message on my email telling that one of the disks have had errors and has been disabled. Below is the syslog snippet containing starting from the first indication of problems. I will post the full syslog if necessary (it was checked 1,5 months ago when I had problems with Sil3132 card and switched to AOC-SASLP-MV8). The error regarding spindown problem repeated in the log every 10 seconds until server was shutdown. Is this a certain case of drive failure or should I do standard "check all connections" operation? Any advice on further action is appreciated. Aug 8 10:01:16 Viper_UnRAID1 kernel: mdcmd (44): spindown 5 Aug 14 14:38:05 Viper_UnRAID1 kernel: ata5.00: exception Emask 0x0 SAct 0x0 SErr 0x0 action 0x6 frozen Aug 14 14:38:05 Viper_UnRAID1 kernel: ata5.00: failed command: READ DMA EXT Aug 14 14:38:05 Viper_UnRAID1 kernel: ata5.00: cmd 25/00:08:87:f9:59/00:00:2d:00:00/e0 tag 0 dma 4096 in Aug 14 14:38:05 Viper_UnRAID1 kernel: res 40/00:00:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) Aug 14 14:38:05 Viper_UnRAID1 kernel: ata5.00: status: { DRDY } Aug 14 14:38:05 Viper_UnRAID1 kernel: ata5: hard resetting link Aug 14 14:38:15 Viper_UnRAID1 kernel: ata5: softreset failed (timeout) Aug 14 14:38:15 Viper_UnRAID1 kernel: ata5: hard resetting link Aug 14 14:38:25 Viper_UnRAID1 kernel: ata5: softreset failed (timeout) Aug 14 14:38:25 Viper_UnRAID1 kernel: ata5: hard resetting link Aug 14 14:39:00 Viper_UnRAID1 kernel: ata5: softreset failed (timeout) Aug 14 14:39:00 Viper_UnRAID1 kernel: ata5: limiting SATA link speed to 1.5 Gbps Aug 14 14:39:00 Viper_UnRAID1 kernel: ata5: hard resetting link Aug 14 14:39:05 Viper_UnRAID1 kernel: ata5: softreset failed (timeout) Aug 14 14:39:05 Viper_UnRAID1 kernel: ata5: reset failed, giving up Aug 14 14:39:05 Viper_UnRAID1 kernel: ata5.00: disabled Aug 14 14:39:05 Viper_UnRAID1 kernel: ata5.00: device reported invalid CHS sector 0 Aug 14 14:39:05 Viper_UnRAID1 kernel: ata5: EH complete Aug 14 14:39:05 Viper_UnRAID1 kernel: sd 5:0:0:0: [sde] Unhandled error code Aug 14 14:39:05 Viper_UnRAID1 kernel: sd 5:0:0:0: [sde] Result: hostbyte=0x04 driverbyte=0x00 Aug 14 14:39:05 Viper_UnRAID1 kernel: sd 5:0:0:0: [sde] CDB: cdb[0]=0x28: 28 00 2d 59 f9 87 00 00 08 00 Aug 14 14:39:05 Viper_UnRAID1 kernel: end_request: I/O error, dev sde, sector 760871303 Aug 14 14:39:05 Viper_UnRAID1 kernel: md: disk9 read error Aug 14 14:39:05 Viper_UnRAID1 kernel: handle_stripe read error: 760871240/9, count: 1 Aug 14 14:39:17 Viper_UnRAID1 kernel: sd 5:0:0:0: [sde] Unhandled error code Aug 14 14:39:17 Viper_UnRAID1 kernel: sd 5:0:0:0: [sde] Result: hostbyte=0x04 driverbyte=0x00 Aug 14 14:39:17 Viper_UnRAID1 kernel: sd 5:0:0:0: [sde] CDB: cdb[0]=0x2a: 2a 00 2d 59 f9 87 00 00 08 00 Aug 14 14:39:17 Viper_UnRAID1 kernel: end_request: I/O error, dev sde, sector 760871303 Aug 14 14:39:17 Viper_UnRAID1 kernel: sd 5:0:0:0: [sde] Unhandled error code Aug 14 14:39:17 Viper_UnRAID1 kernel: sd 5:0:0:0: [sde] Result: hostbyte=0x04 driverbyte=0x00 Aug 14 14:39:17 Viper_UnRAID1 kernel: sd 5:0:0:0: [sde] CDB: cdb[0]=0x28: 28 00 41 3e 99 97 00 00 08 00 Aug 14 14:39:17 Viper_UnRAID1 kernel: end_request: I/O error, dev sde, sector 1094621591 Aug 14 14:39:17 Viper_UnRAID1 kernel: md: disk9 write error Aug 14 14:39:17 Viper_UnRAID1 kernel: handle_stripe write error: 760871240/9, count: 1 Aug 14 14:39:17 Viper_UnRAID1 kernel: md: recovery thread woken up ... Aug 14 14:39:17 Viper_UnRAID1 kernel: md: disk9 read error Aug 14 14:39:17 Viper_UnRAID1 kernel: handle_stripe read error: 1094621528/9, count: 1 Aug 14 14:39:17 Viper_UnRAID1 kernel: sd 5:0:0:0: [sde] Unhandled error code Aug 14 14:39:17 Viper_UnRAID1 kernel: sd 5:0:0:0: [sde] Result: hostbyte=0x04 driverbyte=0x00 Aug 14 14:39:17 Viper_UnRAID1 kernel: sd 5:0:0:0: [sde] CDB: cdb[0]=0x2a: 2a 00 41 3e 99 97 00 00 08 00 Aug 14 14:39:17 Viper_UnRAID1 kernel: end_request: I/O error, dev sde, sector 1094621591 Aug 14 14:39:17 Viper_UnRAID1 kernel: md: disk9 write error Aug 14 14:39:17 Viper_UnRAID1 kernel: handle_stripe write error: 1094621528/9, count: 1 Aug 14 14:39:18 Viper_UnRAID1 kernel: md: recovery thread has nothing to resync Aug 14 15:39:23 Viper_UnRAID1 emhttp: mdcmd: write: Input/output error Aug 14 15:39:23 Viper_UnRAID1 kernel: mdcmd (45): spindown 9 Aug 14 15:39:23 Viper_UnRAID1 kernel: md: disk9: ATA_OP_STANDBYNOW1 ioctl error: -5 Aug 14 15:39:33 Viper_UnRAID1 emhttp: mdcmd: write: Input/output error Aug 14 15:39:33 Viper_UnRAID1 kernel: mdcmd (46): spindown 9 Aug 14 15:39:33 Viper_UnRAID1 kernel: md: disk9: ATA_OP_STANDBYNOW1 ioctl error: -5 Aug 14 15:39:43 Viper_UnRAID1 emhttp: mdcmd: write: Input/output error Aug 14 15:39:43 Viper_UnRAID1 kernel: mdcmd (47): spindown 9 Aug 14 15:39:43 Viper_UnRAID1 kernel: md: disk9: ATA_OP_STANDBYNOW1 ioctl error: -5 Aug 14 15:39:53 Viper_UnRAID1 emhttp: mdcmd: write: Input/output error Aug 14 15:39:53 Viper_UnRAID1 kernel: mdcmd (48): spindown 9 Aug 14 15:39:53 Viper_UnRAID1 kernel: md: disk9: ATA_OP_STANDBYNOW1 ioctl error: -5 Aug 14 15:40:03 Viper_UnRAID1 emhttp: mdcmd: write: Input/output error Aug 14 15:40:03 Viper_UnRAID1 kernel: mdcmd (49): spindown 9 Aug 14 15:40:03 Viper_UnRAID1 kernel: md: disk9: ATA_OP_STANDBYNOW1 ioctl error: -5 Aug 14 15:40:13 Viper_UnRAID1 emhttp: mdcmd: write: Input/output error Aug 14 15:40:13 Viper_UnRAID1 kernel: mdcmd (50): spindown 9 Aug 14 15:40:13 Viper_UnRAID1 kernel: md: disk9: ATA_OP_STANDBYNOW1 ioctl error: -5 Aug 14 15:40:23 Viper_UnRAID1 emhttp: mdcmd: write: Input/output error Aug 14 15:40:23 Viper_UnRAID1 kernel: mdcmd (51): spindown 9
August 17, 201213 yr Author Finally had some time to check this more thoroughly. I re-seated all the cables and started the server and it came up without errors in the syslog but the sde-drive being disabled (as it should have been). I stopped the array just to be safe and ran "smartctl -A /sdd/sde" which did not indicate changes on any meaningful values. I also have smarthistory running and it did not report any value changes. Also running short smartctl on the drive did not indicate any problems. So the drive seems "ok". I then re-read the wiki chapter about re-enabling a failed drive (see http://lime-technology.com/wiki/index.php/Troubleshooting#How_do_I_re-enable_the_drive.3F). I seem to have two options: 1. Use the "Trust my array" procedure 2. Reconstruct the drive Reading the description and especially all the warnings on the "Trust my array" procedure, reconstructing seems to be the safer choice. But should I preclear the failed drive before reconstruct?
August 17, 201213 yr Finally had some time to check this more thoroughly. I re-seated all the cables and started the server and it came up without errors in the syslog but the sde-drive being disabled (as it should have been). I stopped the array just to be safe and ran "smartctl -A /sdd/sde" which did not indicate changes on any meaningful values. I also have smarthistory running and it did not report any value changes. Also running short smartctl on the drive did not indicate any problems. So the drive seems "ok". I then re-read the wiki chapter about re-enabling a failed drive (see http://lime-technology.com/wiki/index.php/Troubleshooting#How_do_I_re-enable_the_drive.3F). I seem to have two options: 1. Use the "Trust my array" procedure 2. Reconstruct the drive Reading the description and especially all the warnings on the "Trust my array" procedure, reconstructing seems to be the safer choice. But should I preclear the failed drive before reconstruct? no need, unless you never pre-cleared it previously, as it will be entirely written.(and then you'll want to perform a NOCORRECT parity check after, to confirm it was written and can be read)
August 18, 201213 yr Author Ok, thanx for the confirm and advice to use NOCORRECT parity sync afterwards. The disk is now re-building itself on the same drive. No errors in the syslog so far. I'll report back once it's rebuilt itself and have run parity check couple of times. Edit: Proper syntax for read-only parity check found here: http://lime-technology.com/wiki/index.php/FAQ#How_can_I_do_a_read-only_parity_check.3F http://lime-technology.com/forum/index.php?topic=3430.msg29619#msg29619 And just to repeat it: mdcmd check NOCORRECT Rebuild finished without any errors. Read-only parity check now running...
August 19, 201213 yr Author First parity check finished without errors and also there was nothing alarming in the syslog. The question remains what caused the problem. Personally I suspect Sil3132 card. Though it's been working fine for quite some time I recently had problems with another similar card. If the problem ever repeats I will definitely ditch the Sil3132 card. So problem solved, it seemed not to be related to the drive but some weird glitch.
Archived
This topic is now archived and is closed to further replies.