Repeat drive failures - but only during midnight housekeeping

[stultus]

I'm having the same drive flake out in the middle of the night, once every couple months. Yesterday I reseated the SATA cabling on drive and motherboard, and it died again in the night. Usually I just reboot and rebuild from parity since it's a newer drive. Can someone take a peek at the attached logs and give me any indication on the cause/solution? TIA

 

Some snippets:

Jun 21 00:00:51 Sophos kernel: ata9.00: exception Emask 0x0 SAct 0xffffffff SErr 0x0 action 0x6 frozen
Jun 21 00:00:51 Sophos kernel: ata9.00: failed command: WRITE FPDMA QUEUED
Jun 21 00:00:51 Sophos kernel: ata9.00: cmd 61/20:00:20:3c:a4/00:00:00:00:00/40 tag 0 ncq dma 16384 out
Jun 21 00:00:51 Sophos kernel:         res 40/00:01:01:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout)
Jun 21 00:00:51 Sophos kernel: ata9.00: status: { DRDY }
Jun 21 00:00:51 Sophos kernel: ata9.00: failed command: WRITE FPDMA QUEUED
Jun 21 00:00:51 Sophos kernel: ata9.00: cmd 61/80:08:20:82:a4/00:00:00:00:00/40 tag 1 ncq dma 65536 out
Jun 21 00:00:51 Sophos kernel:         res 40/00:01:00:00:00/00:00:00:00:00/00 Emask 0x4 (timeout)

Jun 21 00:01:30 Sophos kernel: ata9: hard resetting link
Jun 21 00:01:30 Sophos kernel: ata9: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Jun 21 00:01:30 Sophos kernel: ata9.00: supports DRM functions and may not be fully accessible
Jun 21 00:01:45 Sophos kernel: ata9.00: qc timeout (cmd 0xef)
Jun 21 00:01:45 Sophos kernel: ata9.00: failed to set xfermode (err_mask=0x4)
Jun 21 00:01:45 Sophos kernel: ata9: hard resetting link
Jun 21 00:01:55 Sophos kernel: ata9: softreset failed (1st FIS failed)
Jun 21 00:01:55 Sophos kernel: ata9: hard resetting link
Jun 21 00:02:05 Sophos kernel: ata9: softreset failed (1st FIS failed)
Jun 21 00:02:05 Sophos kernel: ata9: hard resetting link
Jun 21 00:02:40 Sophos kernel: ata9: softreset failed (1st FIS failed)
Jun 21 00:02:40 Sophos kernel: ata9: limiting SATA link speed to 3.0 Gbps
Jun 21 00:02:40 Sophos kernel: ata9: hard resetting link
Jun 21 00:02:45 Sophos kernel: ata9: softreset failed (1st FIS failed)
Jun 21 00:02:45 Sophos kernel: ata9: reset failed, giving up
Jun 21 00:02:45 Sophos kernel: ata9.00: disabled
Jun 21 00:02:45 Sophos kernel: ata9: EH complete
Jun 21 00:02:45 Sophos kernel: sd 9:0:0:0: [sdj] tag#12 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 cmd_age=149s
Jun 21 00:02:45 Sophos kernel: sd 9:0:0:0: [sdj] tag#18 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 cmd_age=0s
Jun 21 00:02:45 Sophos kernel: sd 9:0:0:0: [sdj] tag#18 CDB: opcode=0x2a 2a 00 0b e0 15 10 00 00 30 00
Jun 21 00:02:45 Sophos kernel: blk_update_request: I/O error, dev sdj, sector 199234832 op 0x1:(WRITE) flags 0x8800 phys_seg 6 prio class 0
Jun 21 00:02:45 Sophos kernel: sd 9:0:0:0: [sdj] tag#12 CDB: opcode=0x93 93 08 00 00 00 00 00 00 30 00 00 00 00 20 00 00
Jun 21 00:02:45 Sophos kernel: blk_update_request: I/O error, dev sdj, sector 12288 op 0x3:(DISCARD) flags 0x800 phys_seg 1 prio class 0
Jun 21 00:02:45 Sophos kernel: BTRFS error (device sdj1): bdev /dev/sdj1 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
Jun 21 00:02:45 Sophos kernel: BTRFS warning (device sdj1): direct IO failed ino 265 rw 1,34817 sector 0xbe01528 len 0 err no 10
Jun 21 00:02:45 Sophos kernel: sd 9:0:0:0: [sdj] Read Capacity(16) failed: Result: hostbyte=0x04 driverbyte=0x00
Jun 21 00:02:45 Sophos kernel: BTRFS error (device sdj1): bdev /dev/sdj1 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0
Jun 21 00:02:45 Sophos kernel: BTRFS warning (device sdj1): direct IO failed ino 265 rw 1,34817 sector 0xbe01540 len 0 err no 10
Jun 21 00:02:45 Sophos kernel: sd 9:0:0:0: [sdj] Sense not available.
Jun 21 00:02:45 Sophos kernel: sd 9:0:0:0: [sdj] tag#19 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 cmd_age=0s
Jun 21 00:02:45 Sophos kernel: sd 9:0:0:0: [sdj] tag#19 CDB: opcode=0x2a 2a 00 0b e0 14 88 00 00 08 00
Jun 21 00:02:45 Sophos kernel: blk_update_request: I/O error, dev sdj, sector 199234696 op 0x1:(WRITE) flags 0x8800 phys_seg 1 prio class 0
Jun 21 00:02:45 Sophos kernel: BTRFS error (device sdj1): bdev /dev/sdj1 errs: wr 3, rd 0, flush 0, corrupt 0, gen 0
Jun 21 00:02:45 Sophos kernel: BTRFS warning (device sdj1): direct IO failed ino 265 rw 1,34817 sector 0xbe01490 len 0 err no 10
Jun 21 00:02:45 Sophos kernel: sd 9:0:0:0: [sdj] tag#20 UNKNOWN(0x2003) Result: hostbyte=0x04 driverbyte=0x00 cmd_age=106s
Jun 21 00:02:45 Sophos kernel: sd 9:0:0:0: [sdj] tag#20 CDB: opcode=0x2a 2a 00 0b e0 14 70 00 00 18 00

 

Jun 21 00:02:45 Sophos kernel: BTRFS warning (device sdj1): direct IO failed ino 265 rw 1,34817 sector 0xdfbdfc0 len 0 err no 10
Jun 21 00:02:45 Sophos kernel: sd 9:0:0:0: [sdj] tag#18 access beyond end of device
Jun 21 00:02:45 Sophos kernel: BTRFS warning (device sdj1): direct IO failed ino 265 rw 1,34817 sector 0xdfbdfe0 len 0 err no 10
Jun 21 00:02:45 Sophos kernel: sd 9:0:0:0: [sdj] tag#19 access beyond end of device
Jun 21 00:02:45 Sophos kernel: BTRFS warning (device sdj1): direct IO failed ino 265 rw 1,34817 sector 0xdfbe000 len 0 err no 10

sophos-diagnostics-20210621-0733 (1).zip

[JorgeB]

57 minutes ago, stultus said:

Jun 21 00:02:45 Sophos kernel: ata9: softreset failed (1st FIS failed)
Jun 21 00:02:45 Sophos kernel: ata9: reset failed, giving up
Jun 21 00:02:45 Sophos kernel: ata9.00: disabled

This means the drive is dropping offline, replace or swap cables, both SATA and power, or connect the SSD to a different controller.

[stultus]

Thanks, that was the 2nd error that happened last night - a first ever SSD drive dropping off. A reboot brought it right back up.

 

I'm running an extended SMART test on the spinning drive I originally posted about. That'll take a full day.