I selected parity check from the web interface while all my drives spun down. After about 5 minutes the web interface came back with the parity drive red-balled. I stopped the array, and the web interface reported that there was "no device" in the parity slot. Attached is the syslog, and the entries at the time of it happening.

 

This has happened before, and not only with the parity drive, it will happen randomly with the other data drives, and always after spin up. I usually power down the server, on power up the device is present again, and I can start a parity re-build or data rebuild.

 

Jun 24 05:38:49 Tower kernel: mdcmd (93): check CORRECT
Jun 24 05:38:49 Tower kernel: md: recovery thread woken up ...
Jun 24 05:38:49 Tower kernel: md: recovery thread checking parity...
Jun 24 05:38:49 Tower kernel: md: using 1536k window, over a total of 2930266532 blocks.
Jun 24 05:39:01 Tower kernel: ata1.00: exception Emask 0x10 SAct 0x0 SErr 0x90202 action 0xe frozen
Jun 24 05:39:01 Tower kernel: ata1.00: irq_stat 0x00400000, PHY RDY changed
Jun 24 05:39:01 Tower kernel: ata1: SError: { RecovComm Persist PHYRdyChg 10B8B }
Jun 24 05:39:01 Tower kernel: ata1.00: failed command: SMART
Jun 24 05:39:01 Tower kernel: ata1.00: cmd b0/d0:01:00:4f:c2/00:00:00:00:00/00 tag 0 pio 512 in
Jun 24 05:39:01 Tower kernel:          res 50/00:ff:00:00:00/00:00:00:00:00/40 Emask 0x10 (ATA bus error)
Jun 24 05:39:01 Tower kernel: ata1.00: status: { DRDY }
Jun 24 05:39:01 Tower kernel: ata1: hard resetting link
Jun 24 05:39:01 Tower kernel: md: parity incorrect: 128
Jun 24 05:39:11 Tower kernel: ata1: softreset failed (1st FIS failed)
Jun 24 05:39:11 Tower kernel: ata1: hard resetting link
Jun 24 05:39:21 Tower kernel: ata1: softreset failed (1st FIS failed)
Jun 24 05:39:21 Tower kernel: ata1: hard resetting link
Jun 24 05:39:56 Tower kernel: ata1: softreset failed (1st FIS failed)
Jun 24 05:39:56 Tower kernel: ata1: limiting SATA link speed to 1.5 Gbps
Jun 24 05:39:56 Tower kernel: ata1: hard resetting link
Jun 24 05:40:01 Tower kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
Jun 24 05:40:01 Tower kernel: ata1.00: link online but device misclassifed
Jun 24 05:40:06 Tower kernel: ata1.00: qc timeout (cmd 0xec)
Jun 24 05:40:06 Tower kernel: ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4)
Jun 24 05:40:06 Tower kernel: ata1.00: revalidation failed (errno=-5)
Jun 24 05:40:06 Tower kernel: ata1: hard resetting link
Jun 24 05:40:16 Tower kernel: ata1: softreset failed (1st FIS failed)
Jun 24 05:40:16 Tower kernel: ata1: hard resetting link
Jun 24 05:40:26 Tower kernel: ata1: softreset failed (1st FIS failed)
Jun 24 05:40:26 Tower kernel: ata1: hard resetting link
Jun 24 05:41:01 Tower kernel: ata1: softreset failed (1st FIS failed)
Jun 24 05:41:01 Tower kernel: ata1: hard resetting link
Jun 24 05:41:06 Tower kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
Jun 24 05:41:06 Tower kernel: ata1.00: link online but device misclassifed
Jun 24 05:41:16 Tower kernel: ata1.00: qc timeout (cmd 0xec)
Jun 24 05:41:16 Tower kernel: ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4)
Jun 24 05:41:16 Tower kernel: ata1.00: revalidation failed (errno=-5)
Jun 24 05:41:16 Tower kernel: ata1: hard resetting link
Jun 24 05:41:26 Tower kernel: ata1: softreset failed (1st FIS failed)
Jun 24 05:41:26 Tower kernel: ata1: hard resetting link
Jun 24 05:41:36 Tower kernel: ata1: softreset failed (1st FIS failed)
Jun 24 05:41:36 Tower kernel: ata1: hard resetting link
Jun 24 05:42:11 Tower kernel: ata1: softreset failed (1st FIS failed)
Jun 24 05:42:11 Tower kernel: ata1: hard resetting link
Jun 24 05:42:17 Tower kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
Jun 24 05:42:17 Tower kernel: ata1.00: link online but device misclassifed
Jun 24 05:42:47 Tower kernel: ata1.00: qc timeout (cmd 0xec)
Jun 24 05:42:47 Tower kernel: ata1.00: failed to IDENTIFY (I/O error, err_mask=0x4)
Jun 24 05:42:47 Tower kernel: ata1.00: revalidation failed (errno=-5)
Jun 24 05:42:47 Tower kernel: ata1.00: disabled
Jun 24 05:42:47 Tower kernel: ata1: hard resetting link
Jun 24 05:42:57 Tower kernel: ata1: softreset failed (1st FIS failed)
Jun 24 05:42:57 Tower kernel: ata1: hard resetting link
Jun 24 05:43:07 Tower kernel: ata1: softreset failed (1st FIS failed)
Jun 24 05:43:07 Tower kernel: ata1: hard resetting link
Jun 24 05:43:42 Tower kernel: ata1: softreset failed (1st FIS failed)
Jun 24 05:43:42 Tower kernel: ata1: limiting SATA link speed to 1.5 Gbps
Jun 24 05:43:42 Tower kernel: ata1: hard resetting link
Jun 24 05:43:47 Tower kernel: ata1: SATA link up 1.5 Gbps (SStatus 113 SControl 310)
Jun 24 05:43:47 Tower kernel: ata1.00: link online but device misclassifed
Jun 24 05:43:47 Tower kernel: ata1: link online but 1 devices misclassified, device detection might fail
Jun 24 05:43:47 Tower kernel: sd 0:0:0:0: rejecting I/O to offline device
Jun 24 05:43:47 Tower kernel: sd 0:0:0:0: [sda] killing request
Jun 24 05:43:47 Tower kernel: sd 0:0:0:0: rejecting I/O to offline device
Jun 24 05:43:47 Tower last message repeated 3 times
Jun 24 05:43:47 Tower kernel: ata1: EH complete
Jun 24 05:43:47 Tower kernel: sd 0:0:0:0: [sda] Unhandled error code
Jun 24 05:43:47 Tower kernel: sd 0:0:0:0: [sda]  Result: hostbyte=0x01 driverbyte=0x00
Jun 24 05:43:47 Tower kernel: sd 0:0:0:0: [sda] CDB: cdb[0]=0x28: 28 00 00 00 0c 40 00 00 20 00
Jun 24 05:43:47 Tower kernel: end_request: I/O error, dev sda, sector 3136
Jun 24 05:43:47 Tower kernel: ata1.00: detaching (SCSI 0:0:0:0)
Jun 24 05:43:47 Tower kernel: md: disk0 write error
Jun 24 05:43:47 Tower kernel: handle_stripe write error: 128/0, count: 1
Jun 24 05:43:47 Tower kernel: md: disk0 read error
Jun 24 05:43:47 Tower kernel: handle_stripe read error: 3104/0, count: 1
Jun 24 05:43:47 Tower kernel: md: disk0 read error

syslog.zip

Those errors indicate a bad SATA cable.

Those errors indicate a bad SATA cable.

or a bad disk?

Check the power connections, especially any power splitter, to the parity drive.  Also obtain a SMART report for it, after rebooting (if the drive shows up). 

 

I usually associate bad SATA cables with CRC errors, and none appear here.  The lack of them does not mean it's a good cable, but it does not implicate the SATA cable as the problem cause here.  The 2 possibilities I see are either a power problem, or a rather sudden drive failure.  There are only 4 drives, so even a poor PSU should be OK, but the power cabling could be a problem.  We had a similar case recently like this, where the drive was suddenly unresponsive except for the SATA link level, and it turned out to be a bad connector on the power cable.

 

The other possibility, in my opinion, is the drive itself failed.  We know it is still electrically connected, because the low level SATA link is at least partially operable, but there is no response at all at the higher communication levels.  The SMART report, obtained after a reboot (and only if the drive shows up), should indicate the drive's health.

 

Chronology:

- Jun 20 16:52:00 - system booted with no issues

- (~ 60 hours) - normal operation with no issues, includes spin ups and spin downs over next 2 and a half days with no issues

- Jun 23 04:41:23 - final spindown

- (~ 25 hours) - no activity, stays spun down until parity check requested

- Jun 24 05:38:49 - parity check started

- Jun 24 05:39:01 - errors begin, timeouts with no response, 12 second delay is about 6 second spin up plus 5 second exception timeout

- (numerous resets, attempts to communicate, resulting in 5 second, 10 second, and 30 second waits, all timed out)

- Jun 24 05:42:47 - drive disabled (you can ignore all subsequent errors), drive lost until next boot

 

I'm starting to associate 'PHY RDY CHG' with a power interruption, brief or otherwise.  Not sure I'm right but seems to make sense.  The first exception here indicates this error flag.  A related cause, could also be a vibration induced separation or loose connection in a back frame.