stringent-spoke3323

I swapped one of the power supplies and some sata cables and its running fine for now, I'll update if it should happen again. Thanks for the help!

I hope it doesn't matter that I rebooted to regain the missing drive. Else I'll post another one later. megumin-diagnostics-20231030-1425.zip

Hello, out of nowhere I have a strange problem, some drives (pretty much random which ones) start to "fail" (they show as missing in unraid). But the thing is that they aren't failing, after a reboot everything works fine for a while until anoter drive randomly "fails. Looking into the logs it looks like a SATA problem? But how could I further investigate this problem? The cables are good (and it happens to all 12 possible drives (10 array, 2 cache)) so I don't suspect the cable or drive itself. I also have two different sata controller, the one on the board with 8 slots, and a 12 slot expansion card, 8 are connected on the mobo and 4 on the sata expansion card. The drives that "fail" do so on both controller. If anyone can guide me into the right direction it would be really helpful, here are the relevant logs to my second cache ssd "failing" ``` Oct 30 02:37:14 Megumin monitor: Stop running nchan processes Oct 30 02:54:47 Megumin monitor: Stop running nchan processes Oct 30 04:17:50 Megumin monitor: Stop running nchan processes Oct 30 06:28:59 Megumin kernel: ata22: SATA link up 6.0 Gbps (SStatus 133 SControl 300) Oct 30 06:28:59 Megumin kernel: ata22.00: configured for UDMA/133 Oct 30 13:23:43 Megumin kernel: ata22: SATA link up 6.0 Gbps (SStatus 133 SControl 300) Oct 30 13:23:43 Megumin kernel: ata22.00: configured for UDMA/133 Oct 30 13:23:44 Megumin kernel: ata21.00: exception Emask 0x10 SAct 0x0 SErr 0x10000 action 0xe frozen Oct 30 13:23:44 Megumin kernel: ata21.00: irq_stat 0x80400000, PHY RDY changed Oct 30 13:23:44 Megumin kernel: ata21: SError: { PHYRdyChg } Oct 30 13:23:44 Megumin kernel: ata21.00: failed command: DATA SET MANAGEMENT Oct 30 13:23:44 Megumin kernel: ata21.00: cmd 06/01:01:00:00:00/00:00:00:00:00/a0 tag 5 dma 512 out Oct 30 13:23:44 Megumin kernel: res 50/01:01:01:00:00/00:00:00:00:00/00 Emask 0x10 (ATA bus error) Oct 30 13:23:44 Megumin kernel: ata21.00: status: { DRDY } Oct 30 13:23:44 Megumin kernel: ata21.00: error: { AMNF } Oct 30 13:23:44 Megumin kernel: ata21: hard resetting link Oct 30 13:23:45 Megumin kernel: ata21: SATA link down (SStatus 0 SControl 300) Oct 30 13:23:50 Megumin kernel: ata21: hard resetting link Oct 30 13:23:50 Megumin kernel: ata21: SATA link down (SStatus 0 SControl 300) Oct 30 13:23:50 Megumin kernel: ata21: limiting SATA link speed to <unknown> Oct 30 13:23:55 Megumin kernel: ata21: hard resetting link Oct 30 13:23:55 Megumin kernel: ata22: SATA link up 6.0 Gbps (SStatus 133 SControl 300) Oct 30 13:23:55 Megumin kernel: ata22.00: configured for UDMA/133 Oct 30 13:23:56 Megumin kernel: ata21: SATA link down (SStatus 0 SControl 3F0) Oct 30 13:23:56 Megumin kernel: ata21.00: disable device Oct 30 13:23:56 Megumin kernel: sd 21:0:0:0: [sdb] tag#5 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=DRIVER_OK cmd_age=23s Oct 30 13:23:56 Megumin kernel: sd 21:0:0:0: [sdb] tag#5 Sense Key : 0x3 [current] Oct 30 13:23:56 Megumin kernel: sd 21:0:0:0: [sdb] tag#5 ASC=0x13 ASCQ=0x0 Oct 30 13:23:56 Megumin kernel: sd 21:0:0:0: [sdb] tag#5 CDB: opcode=0x93 93 08 00 00 00 00 01 64 03 70 00 00 04 d8 00 00 Oct 30 13:23:56 Megumin kernel: I/O error, dev sdb, sector 23331696 op 0x3:(DISCARD) flags 0x800 phys_seg 1 prio class 2 Oct 30 13:23:56 Megumin kernel: ata21: EH complete Oct 30 13:23:56 Megumin kernel: sd 21:0:0:0: rejecting I/O to offline device Oct 30 13:23:56 Megumin kernel: ata21.00: detaching (SCSI 21:0:0:0) Oct 30 13:23:56 Megumin kernel: sd 21:0:0:0: [sdb] Synchronizing SCSI cache Oct 30 13:23:56 Megumin kernel: sd 21:0:0:0: [sdb] Synchronize Cache(10) failed: Result: hostbyte=0x04 driverbyte=DRIVER_OK Oct 30 13:23:56 Megumin kernel: sd 21:0:0:0: [sdb] Stopping disk Oct 30 13:23:56 Megumin kernel: sd 21:0:0:0: [sdb] Start/Stop Unit failed: Result: hostbyte=0x04 driverbyte=DRIVER_OK Oct 30 13:25:15 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 1, rd 0, flush 0, corrupt 0, gen 0 Oct 30 13:25:15 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 2, rd 0, flush 0, corrupt 0, gen 0 Oct 30 13:25:15 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 3, rd 0, flush 0, corrupt 0, gen 0 Oct 30 13:25:15 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 3, rd 0, flush 1, corrupt 0, gen 0 Oct 30 13:25:15 Megumin kernel: BTRFS warning (device sde1): lost page write due to IO error on /dev/sdb1 (-5) Oct 30 13:25:15 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 4, rd 0, flush 1, corrupt 0, gen 0 Oct 30 13:25:15 Megumin kernel: BTRFS error (device sde1): error writing primary super block to device 2 Oct 30 13:25:47 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 5, rd 0, flush 1, corrupt 0, gen 0 Oct 30 13:25:47 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 6, rd 0, flush 1, corrupt 0, gen 0 Oct 30 13:25:47 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 6, rd 0, flush 2, corrupt 0, gen 0 Oct 30 13:25:47 Megumin kernel: BTRFS warning (device sde1): lost page write due to IO error on /dev/sdb1 (-5) Oct 30 13:25:47 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 7, rd 0, flush 2, corrupt 0, gen 0 Oct 30 13:25:47 Megumin kernel: BTRFS warning (device sde1): lost page write due to IO error on /dev/sdb1 (-5) Oct 30 13:25:47 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 8, rd 0, flush 2, corrupt 0, gen 0 Oct 30 13:25:47 Megumin kernel: BTRFS error (device sde1): error writing primary super block to device 2 Oct 30 13:28:25 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 9, rd 0, flush 2, corrupt 0, gen 0 Oct 30 13:28:25 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 10, rd 0, flush 2, corrupt 0, gen 0 Oct 30 13:28:25 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 11, rd 0, flush 2, corrupt 0, gen 0 Oct 30 13:28:25 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 11, rd 0, flush 3, corrupt 0, gen 0 Oct 30 13:28:25 Megumin kernel: BTRFS warning (device sde1): lost page write due to IO error on /dev/sdb1 (-5) Oct 30 13:28:25 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 12, rd 0, flush 3, corrupt 0, gen 0 Oct 30 13:28:25 Megumin kernel: BTRFS error (device sde1): error writing primary super block to device 2 Oct 30 13:28:56 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 13, rd 0, flush 3, corrupt 0, gen 0 Oct 30 13:28:56 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 14, rd 0, flush 3, corrupt 0, gen 0 Oct 30 13:28:56 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 14, rd 0, flush 4, corrupt 0, gen 0 Oct 30 13:28:56 Megumin kernel: BTRFS warning (device sde1): lost page write due to IO error on /dev/sdb1 (-5) Oct 30 13:28:56 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 15, rd 0, flush 4, corrupt 0, gen 0 Oct 30 13:28:56 Megumin kernel: BTRFS warning (device sde1): lost page write due to IO error on /dev/sdb1 (-5) Oct 30 13:28:56 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 16, rd 0, flush 4, corrupt 0, gen 0 Oct 30 13:28:56 Megumin kernel: BTRFS error (device sde1): error writing primary super block to device 2 Oct 30 13:31:35 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 17, rd 0, flush 4, corrupt 0, gen 0 Oct 30 13:31:35 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 18, rd 0, flush 4, corrupt 0, gen 0 Oct 30 13:31:35 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 19, rd 0, flush 4, corrupt 0, gen 0 Oct 30 13:31:35 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 19, rd 0, flush 5, corrupt 0, gen 0 Oct 30 13:31:35 Megumin kernel: BTRFS warning (device sde1): lost page write due to IO error on /dev/sdb1 (-5) Oct 30 13:31:35 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 20, rd 0, flush 5, corrupt 0, gen 0 Oct 30 13:31:35 Megumin kernel: BTRFS error (device sde1): error writing primary super block to device 2 Oct 30 13:32:06 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 21, rd 0, flush 5, corrupt 0, gen 0 Oct 30 13:32:06 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 22, rd 0, flush 5, corrupt 0, gen 0 Oct 30 13:32:06 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 22, rd 0, flush 6, corrupt 0, gen 0 Oct 30 13:32:06 Megumin kernel: BTRFS warning (device sde1): lost page write due to IO error on /dev/sdb1 (-5) Oct 30 13:32:06 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 23, rd 0, flush 6, corrupt 0, gen 0 Oct 30 13:32:06 Megumin kernel: BTRFS warning (device sde1): lost page write due to IO error on /dev/sdb1 (-5) Oct 30 13:32:06 Megumin kernel: BTRFS error (device sde1): bdev /dev/sdb1 errs: wr 24, rd 0, flush 6, corrupt 0, gen 0 Oct 30 13:32:06 Megumin kernel: BTRFS error (device sde1): error writing primary super block to device 2 ```