Drives falling out of array - print_req_error: I/O error

[RocketSLC]

Hello! I'm having issues with my array after an "upgrade" of the CPUs (2x Intel Xeon E5-2603) and motherboard (Supermicro X10DRL-i), and some of the SATA cables. I suspect that it's one of those things that is causing this issue and not the drives themselves. As shown below, two of my disks in my array are disabled. The 4th went down right before a reboot so I wasn't able to get the logs for that, but the 10th drive going down is shown in the diagnostic files. These also seem to be the drives with data actually on them, which is unfortunate. I'm running Unraid 6.8.3.

 

 

 

I've attached my diagnostic files, but here's the errors I can see in my syslog:

...
Apr  8 21:35:42 UNRAID kernel: ata6.00: exception Emask 0x50 SAct 0x100000 SErr 0x4090800 action 0xe frozen
Apr  8 21:35:42 UNRAID kernel: ata6.00: irq_stat 0x00400040, connection status changed
Apr  8 21:35:42 UNRAID kernel: ata2.00: exception Emask 0x10 SAct 0x4000 SErr 0x4090000 action 0xe frozen
Apr  8 21:35:42 UNRAID kernel: ata2.00: irq_stat 0x00400040, connection status changed
Apr  8 21:35:42 UNRAID kernel: ata6: SError: { HostInt PHYRdyChg 10B8B DevExch }
Apr  8 21:35:42 UNRAID kernel: ata2: SError: { PHYRdyChg 10B8B DevExch }
Apr  8 21:35:42 UNRAID kernel: ata6.00: failed command: READ FPDMA QUEUED
Apr  8 21:35:42 UNRAID kernel: ata2.00: failed command: READ FPDMA QUEUED
Apr  8 21:35:42 UNRAID kernel: ata2.00: cmd 60/00:70:d0:d3:ef/01:00:18:00:00/40 tag 14 ncq dma 131072 in
Apr  8 21:35:42 UNRAID kernel:         res 40/00:74:d0:d3:ef/00:00:18:00:00/40 Emask 0x10 (ATA bus error)
Apr  8 21:35:42 UNRAID kernel: ata6.00: cmd 60/00:a0:cf:d3:ef/01:00:18:00:00/40 tag 20 ncq dma 131072 in
Apr  8 21:35:42 UNRAID kernel:         res 40/00:a4:cf:d3:ef/00:00:18:00:00/40 Emask 0x50 (ATA bus error)
Apr  8 21:35:42 UNRAID kernel: ata6.00: status: { DRDY }
Apr  8 21:35:42 UNRAID kernel: ata2.00: status: { DRDY }
Apr  8 21:35:42 UNRAID kernel: ata6: hard resetting link
Apr  8 21:35:42 UNRAID kernel: ata2: hard resetting link
Apr  8 21:35:42 UNRAID kernel: ata8.00: exception Emask 0x50 SAct 0x2000000 SErr 0x4090800 action 0xe frozen
Apr  8 21:35:42 UNRAID kernel: ata8.00: irq_stat 0x00400040, connection status changed
Apr  8 21:35:42 UNRAID kernel: ata8: SError: { HostInt PHYRdyChg 10B8B DevExch }
Apr  8 21:35:42 UNRAID kernel: ata8.00: failed command: READ FPDMA QUEUED
Apr  8 21:35:42 UNRAID kernel: ata8.00: cmd 60/00:c8:d0:d3:ef/01:00:18:00:00/40 tag 25 ncq dma 131072 in
Apr  8 21:35:42 UNRAID kernel:         res 40/00:c8:d0:d3:ef/00:00:18:00:00/40 Emask 0x50 (ATA bus error)
Apr  8 21:35:42 UNRAID kernel: ata8.00: status: { DRDY }
Apr  8 21:35:42 UNRAID kernel: ata8: hard resetting link
Apr  8 21:35:43 UNRAID kernel: ata1: SATA link down (SStatus 0 SControl 310)
Apr  8 21:35:43 UNRAID kernel: ata1.00: link offline, clearing class 1 to NONE
Apr  8 21:35:44 UNRAID kernel: ata6: SATA link down (SStatus 0 SControl 300)
Apr  8 21:35:44 UNRAID kernel: ata6.00: link offline, clearing class 1 to NONE
Apr  8 21:35:44 UNRAID kernel: ata2: SATA link down (SStatus 0 SControl 300)
Apr  8 21:35:44 UNRAID kernel: ata2.00: link offline, clearing class 1 to NONE
Apr  8 21:35:44 UNRAID kernel: ata2: hard resetting link
Apr  8 21:35:44 UNRAID kernel: ata6: hard resetting link
Apr  8 21:35:45 UNRAID kernel: ata1: SATA link down (SStatus 0 SControl 310)
Apr  8 21:35:45 UNRAID kernel: ata1.00: link offline, clearing class 1 to NONE
Apr  8 21:35:45 UNRAID kernel: ata6: SATA link down (SStatus 0 SControl 300)
Apr  8 21:35:45 UNRAID kernel: ata6.00: link offline, clearing class 1 to NONE
Apr  8 21:35:45 UNRAID kernel: ata6: hard resetting link
Apr  8 21:35:46 UNRAID kernel: ata6: SATA link down (SStatus 0 SControl 300)
Apr  8 21:35:46 UNRAID kernel: ata6.00: link offline, clearing class 1 to NONE
Apr  8 21:35:46 UNRAID kernel: ata6.00: disabled
Apr  8 21:35:46 UNRAID kernel: sd 7:0:0:0: [sdm] tag#20 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
Apr  8 21:35:46 UNRAID kernel: sd 7:0:0:0: [sdm] tag#20 Sense Key : 0x5 [current] 
Apr  8 21:35:46 UNRAID kernel: sd 7:0:0:0: [sdm] tag#20 ASC=0x21 ASCQ=0x4 
Apr  8 21:35:46 UNRAID kernel: sd 7:0:0:0: [sdm] tag#20 CDB: opcode=0x28 28 00 18 ef d3 cf 00 01 00 00
Apr  8 21:35:46 UNRAID kernel: print_req_error: I/O error, dev sdm, sector 418370511
Apr  8 21:35:46 UNRAID kernel: md: disk10 read error, sector=418370448
Apr  8 21:35:46 UNRAID kernel: md: disk10 read error, sector=418370456
Apr  8 21:35:46 UNRAID kernel: md: disk10 read error, sector=418370464
Apr  8 21:35:46 UNRAID kernel: md: disk10 read error, sector=418370472
Apr  8 21:35:46 UNRAID kernel: md: disk10 read error, sector=418370480
Apr  8 21:35:46 UNRAID kernel: md: disk10 read error, sector=418370488
Apr  8 21:35:46 UNRAID kernel: md: disk10 read error, sector=418370496
Apr  8 21:35:46 UNRAID kernel: md: disk10 read error, sector=418370504
Apr  8 21:35:46 UNRAID kernel: md: disk10 read error, sector=418370512
Apr  8 21:35:46 UNRAID kernel: md: disk10 read error, sector=418370520
Apr  8 21:35:46 UNRAID kernel: md: disk10 read error, sector=418370528
Apr  8 21:35:46 UNRAID kernel: md: disk10 read error, sector=418370536
Apr  8 21:35:46 UNRAID kernel: md: disk10 read error, sector=418370544
Apr  8 21:35:46 UNRAID kernel: md: disk10 read error, sector=418370552
Apr  8 21:35:46 UNRAID kernel: md: disk10 read error, sector=418370560
Apr  8 21:35:46 UNRAID kernel: md: disk10 read error, sector=418370568
Apr  8 21:35:46 UNRAID kernel: md: disk10 read error, sector=418370576
Apr  8 21:35:46 UNRAID kernel: md: disk10 read error, sector=418370584
...

 

Installed plugins:

• Community Applications
• Dynamix Auto Fan Control (installed, but not used)
• Dynamix SDD Trim
• Dynamix System Temperature
• Fix Common Problems
• IPMI support
• Nerd Tools
• Unassigned Devices
• User Scripts

 

Any help or advice on what to do next would be helpful. Thanks!

 

 

 

Edited November 14, 2023 by RocketSLC
removed old diags

[JorgeB]

You're having issues with multiple drives on the onboard SATA controller, likely a connection or power problem.