Hello, out of nowhere I have a strange problem, some drives (pretty much random which ones) start to "fail" (they show as missing in unraid). But the thing is that they aren't failing, after a reboot everything works fine for a while until anoter drive randomly "fails. Looking into the logs it looks like a SATA problem? But how could I further investigate this problem? The cables are good (and it happens to all 12 possible drives (10 array, 2 cache)) so I don't suspect the cable or drive itself. I also have two different sata controller, the one on the board with 8 slots, and a 12 slot expansion card, 8 are connected on the mobo and 4 on the sata expansion card. The drives that "fail" do so on both controller.

 

If anyone can guide me into the right direction it would be really helpful, here are the relevant logs to my second cache ssd "failing"

 

```
Oct 30 02:37:14 Megumin monitor: Stop running nchan processes
Oct 30 02:54:47 Megumin monitor: Stop running nchan processes
Oct 30 04:17:50 Megumin monitor: Stop running nchan processes
Oct 30 06:28:59 Megumin kernel: ata22: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Oct 30 06:28:59 Megumin kernel: ata22.00: configured for UDMA/133
Oct 30 13:23:43 Megumin kernel: ata22: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Oct 30 13:23:43 Megumin kernel: ata22.00: configured for UDMA/133
Oct 30 13:23:44 Megumin kernel: ata21.00: exception Emask 0x10 SAct 0x0 SErr 0x10000 action 0xe frozen
Oct 30 13:23:44 Megumin kernel: ata21.00: irq_stat 0x80400000, PHY RDY changed
Oct 30 13:23:44 Megumin kernel: ata21: SError: { PHYRdyChg }
Oct 30 13:23:44 Megumin kernel: ata21.00: failed command: DATA SET MANAGEMENT
Oct 30 13:23:44 Megumin kernel: ata21.00: cmd 06/01:01:00:00:00/00:00:00:00:00/a0 tag 5 dma 512 out
Oct 30 13:23:44 Megumin kernel:         res 50/01:01:01:00:00/00:00:00:00:00/00 Emask 0x10 (ATA bus error)
Oct 30 13:23:44 Megumin kernel: ata21.00: status: { DRDY }
Oct 30 13:23:44 Megumin kernel: ata21.00: error: { AMNF }
Oct 30 13:23:44 Megumin kernel: ata21: hard resetting link
Oct 30 13:23:45 Megumin kernel: ata21: SATA link down (SStatus 0 SControl 300)
Oct 30 13:23:50 Megumin kernel: ata21: hard resetting link
Oct 30 13:23:50 Megumin kernel: ata21: SATA link down (SStatus 0 SControl 300)
Oct 30 13:23:50 Megumin kernel: ata21: limiting SATA link speed to <unknown>
Oct 30 13:23:55 Megumin kernel: ata21: hard resetting link
Oct 30 13:23:55 Megumin kernel: ata22: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Oct 30 13:23:55 Megumin kernel: ata22.00: configured for UDMA/133
Oct 30 13:23:56 Megumin kernel: ata21: SATA link down (SStatus 0 SControl 3F0)
Oct 30 13:23:56 Megumin kernel: ata21.00: disable device
Oct 30 13:23:56 Megumin kernel: sd 21:0:0:0: [sdb] tag#5 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=23s
Oct 30 13:23:56 Megumin kernel: sd 21:0:0:0: [sdb] tag#5 Sense Key : 0x3 [current]
Oct 30 13:23:56 Megumin kernel: sd 21:0:0:0: [sdb] tag#5 ASC=0x13 ASCQ=0x0
Oct 30 13:23:56 Megumin kernel: sd 21:0:0:0: [sdb] tag#5 CDB: opcode=0x93 93 08 00 00 00 00 01 64 03 70 00 00 04 d8 00 00
Oct 30 13:23:56 Megumin kernel: I/O error, dev sdb, sector 23331696 op 0x3:(DISCARD) flags 0x800 phys_seg 1 prio class 2
Oct 30 13:23:56 Megumin kernel: ata21: EH complete
Oct 30 13:23:56 Megumin kernel: sd 21:0:0:0: rejecting I/O to offline device
Oct 30 13:23:56 Megumin kernel: ata21.00: detaching (SCSI 21:0:0:0)
Oct 30 13:23:56 Megumin kernel: sd 21:0:0:0: [sdb] Synchronizing SCSI cache
Oct 30 13:23:56 Megumin kernel: sd 21:0:0:0: [sdb] Synchronize Cache(10) failed: Result: hostbyte=0x04 driverbyte=DRIVER_OK
Oct 30 13:23:56 Megumin kernel: sd 21:0:0:0: [sdb] Stopping disk
Oct 30 13:23:56 Megumin kernel: sd 21:0:0:0: [sdb] Start/Stop Unit failed: Result: hostbyte=0x04 driverbyte=DRIVER_OK
Oct 30 13:25:15 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0
Oct 30 13:25:15 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0
Oct 30 13:25:15 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 3, rd 0, flush 0, corrupt 0, gen 0
Oct 30 13:25:15 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 3, rd 0, flush 1, corrupt 0, gen 0
Oct 30 13:25:15 Megumin kernel: BTRFS warning (device sde1): lost page write due to IO error on /dev/sdb1 (-5)
Oct 30 13:25:15 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 4, rd 0, flush 1, corrupt 0, gen 0
Oct 30 13:25:15 Megumin kernel: BTRFS error (device sde1): error writing primary super block to device 2
Oct 30 13:25:47 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 5, rd 0, flush 1, corrupt 0, gen 0
Oct 30 13:25:47 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 6, rd 0, flush 1, corrupt 0, gen 0
Oct 30 13:25:47 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 6, rd 0, flush 2, corrupt 0, gen 0
Oct 30 13:25:47 Megumin kernel: BTRFS warning (device sde1): lost page write due to IO error on /dev/sdb1 (-5)
Oct 30 13:25:47 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 7, rd 0, flush 2, corrupt 0, gen 0
Oct 30 13:25:47 Megumin kernel: BTRFS warning (device sde1): lost page write due to IO error on /dev/sdb1 (-5)
Oct 30 13:25:47 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 8, rd 0, flush 2, corrupt 0, gen 0
Oct 30 13:25:47 Megumin kernel: BTRFS error (device sde1): error writing primary super block to device 2
Oct 30 13:28:25 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 9, rd 0, flush 2, corrupt 0, gen 0
Oct 30 13:28:25 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 10, rd 0, flush 2, corrupt 0, gen 0
Oct 30 13:28:25 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 11, rd 0, flush 2, corrupt 0, gen 0
Oct 30 13:28:25 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 11, rd 0, flush 3, corrupt 0, gen 0
Oct 30 13:28:25 Megumin kernel: BTRFS warning (device sde1): lost page write due to IO error on /dev/sdb1 (-5)
Oct 30 13:28:25 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 12, rd 0, flush 3, corrupt 0, gen 0
Oct 30 13:28:25 Megumin kernel: BTRFS error (device sde1): error writing primary super block to device 2
Oct 30 13:28:56 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 13, rd 0, flush 3, corrupt 0, gen 0
Oct 30 13:28:56 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 14, rd 0, flush 3, corrupt 0, gen 0
Oct 30 13:28:56 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 14, rd 0, flush 4, corrupt 0, gen 0
Oct 30 13:28:56 Megumin kernel: BTRFS warning (device sde1): lost page write due to IO error on /dev/sdb1 (-5)
Oct 30 13:28:56 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 15, rd 0, flush 4, corrupt 0, gen 0
Oct 30 13:28:56 Megumin kernel: BTRFS warning (device sde1): lost page write due to IO error on /dev/sdb1 (-5)
Oct 30 13:28:56 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 16, rd 0, flush 4, corrupt 0, gen 0
Oct 30 13:28:56 Megumin kernel: BTRFS error (device sde1): error writing primary super block to device 2
Oct 30 13:31:35 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 17, rd 0, flush 4, corrupt 0, gen 0
Oct 30 13:31:35 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 18, rd 0, flush 4, corrupt 0, gen 0
Oct 30 13:31:35 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 19, rd 0, flush 4, corrupt 0, gen 0
Oct 30 13:31:35 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 19, rd 0, flush 5, corrupt 0, gen 0
Oct 30 13:31:35 Megumin kernel: BTRFS warning (device sde1): lost page write due to IO error on /dev/sdb1 (-5)
Oct 30 13:31:35 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 20, rd 0, flush 5, corrupt 0, gen 0
Oct 30 13:31:35 Megumin kernel: BTRFS error (device sde1): error writing primary super block to device 2
Oct 30 13:32:06 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 21, rd 0, flush 5, corrupt 0, gen 0
Oct 30 13:32:06 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 22, rd 0, flush 5, corrupt 0, gen 0
Oct 30 13:32:06 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 22, rd 0, flush 6, corrupt 0, gen 0
Oct 30 13:32:06 Megumin kernel: BTRFS warning (device sde1): lost page write due to IO error on /dev/sdb1 (-5)
Oct 30 13:32:06 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 23, rd 0, flush 6, corrupt 0, gen 0
Oct 30 13:32:06 Megumin kernel: BTRFS warning (device sde1): lost page write due to IO error on /dev/sdb1 (-5)
Oct 30 13:32:06 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 24, rd 0, flush 6, corrupt 0, gen 0
Oct 30 13:32:06 Megumin kernel: BTRFS error (device sde1): error writing primary super block to device 2
```

Please post the diagnostics.

I hope it doesn't matter that I rebooted to regain the missing drive. Else I'll post another one later.

megumin-diagnostics-20231030-1425.zip

It matters but it still helps, ATA21 and 22 are two post of the same Asmedia controller, device dropping offline like that it's usually a power/connection issue, but see if it also happens with different controllers.

I swapped one of the power supplies and some sata cables and its running fine for now, I'll update if it should happen again. Thanks for the help!