I've been experiencing some random shutdowns in the last few days. Starting up the machine is difficult, almost as if there isn't enough power to power up all devices, but the PSU should have more than enough capacity (6 HDDs, 2 SSDs, Quaddro K620, GB NIC, 16GB DDR4). One HDD is a parity, the SSDs are in their own cache pools.
I can only reliably startup when about half my disks are powered, and I power the rest one by one before bootup. With all of them connected, I can start up only about 10% of the time. I made the assumption that it was power related (it still might be), and managed to boot up successfully. While I had the server open, I used the opportunity to swap out a disk for increased capacity, however starting the array now that triggers a rebuild results in the system powering down immediately.
I managed to catch some logs:
Oct 15 17:37:58 Tower kernel: ata3.00: exception Emask 0x10 SAct 0x400 SErr 0x4890000 action 0xe frozen
Oct 15 17:37:58 Tower kernel: ata3.00: irq_stat 0x08400040, interface fatal error, connection status changed
Oct 15 17:37:58 Tower kernel: ata3: SError: { PHYRdyChg 10B8B LinkSeq DevExch }
Oct 15 17:37:58 Tower kernel: ata3.00: failed command: WRITE FPDMA QUEUED
Oct 15 17:37:58 Tower kernel: ata3.00: cmd 61/40:50:80:15:00/05:00:00:00:00/40 tag 10 ncq dma 688128 out
Oct 15 17:37:58 Tower kernel: res 40/00:50:80:15:00/00:00:00:00:00/40 Emask 0x10 (ATA bus error)
Oct 15 17:37:58 Tower kernel: ata3.00: status: { DRDY }
Oct 15 17:37:58 Tower kernel: ata3: hard resetting link
Oct 15 17:37:58 Tower kernel: ata2.00: exception Emask 0x10 SAct 0x207f8 SErr 0x4090000 action 0xe frozen
Oct 15 17:37:58 Tower kernel: ata2.00: irq_stat 0x00400040, connection status changed
Oct 15 17:37:58 Tower kernel: ata2: SError: { PHYRdyChg 10B8B DevExch }
Oct 15 17:37:58 Tower kernel: ata2.00: failed command: READ FPDMA QUEUED
Oct 15 17:37:58 Tower kernel: ata2.00: cmd 60/38:18:90:35:00/00:00:00:00:00/40 tag 3 ncq dma 28672 in
Oct 15 17:37:58 Tower kernel: res 40/00:50:d0:45:00/00:00:00:00:00/40 Emask 0x10 (ATA bus error)
Oct 15 17:37:58 Tower kernel: ata2.00: status: { DRDY }
Oct 15 17:37:58 Tower kernel: ata2.00: failed command: READ FPDMA QUEUED
Oct 15 17:37:58 Tower kernel: ata2.00: cmd 60/38:20:c8:35:00/05:00:00:00:00/40 tag 4 ncq dma 684032 in
Oct 15 17:37:58 Tower kernel: res 40/00:50:d0:45:00/00:00:00:00:00/40 Emask 0x10 (ATA bus error)
Oct 15 17:37:58 Tower kernel: ata2.00: status: { DRDY }
Oct 15 17:37:58 Tower kernel: ata2.00: failed command: READ FPDMA QUEUED
Oct 15 17:37:58 Tower kernel: ata2.00: cmd 60/08:28:00:3b:00/04:00:00:00:00/40 tag 5 ncq dma 528384 in
Oct 15 17:37:58 Tower kernel: res 40/00:50:d0:45:00/00:00:00:00:00/40 Emask 0x10 (ATA bus error)
Oct 15 17:37:58 Tower kernel: ata2.00: status: { DRDY }
Oct 15 17:37:58 Tower kernel: ata2.00: failed command: READ FPDMA QUEUED
Oct 15 17:37:58 Tower kernel: ata2.00: cmd 60/38:30:08:3f:00/01:00:00:00:00/40 tag 6 ncq dma 159744 in
Oct 15 17:37:58 Tower kernel: res 40/00:50:d0:45:00/00:00:00:00:00/40 Emask 0x10 (ATA bus error)
Oct 15 17:37:58 Tower kernel: ata2.00: status: { DRDY }
Oct 15 17:37:58 Tower kernel: ata2.00: failed command: READ FPDMA QUEUED
Oct 15 17:37:58 Tower kernel: ata2.00: cmd 60/c0:38:40:40:00/02:00:00:00:00/40 tag 7 ncq dma 360448 in
Oct 15 17:37:58 Tower kernel: res 40/00:50:d0:45:00/00:00:00:00:00/40 Emask 0x10 (ATA bus error)
Oct 15 17:37:58 Tower kernel: ata2.00: status: { DRDY }
Oct 15 17:37:58 Tower kernel: ata2.00: failed command: READ FPDMA QUEUED
Oct 15 17:37:58 Tower kernel: ata2.00: cmd 60/08:40:00:43:00/00:00:00:00:00/40 tag 8 ncq dma 4096 in
Oct 15 17:37:58 Tower kernel: res 40/00:50:d0:45:00/00:00:00:00:00/40 Emask 0x10 (ATA bus error)
Oct 15 17:37:58 Tower kernel: ata2.00: status: { DRDY }
Oct 15 17:37:58 Tower kernel: ata2.00: failed command: READ FPDMA QUEUED
Oct 15 17:37:58 Tower kernel: ata2.00: cmd 60/c8:48:08:43:00/02:00:00:00:00/40 tag 9 ncq dma 364544 in
Oct 15 17:37:58 Tower kernel: res 40/00:50:d0:45:00/00:00:00:00:00/40 Emask 0x10 (ATA bus error)
Oct 15 17:37:58 Tower kernel: ata2.00: status: { DRDY }
Oct 15 17:37:58 Tower kernel: ata2.00: failed command: READ FPDMA QUEUED
Oct 15 17:37:58 Tower kernel: ata2.00: cmd 60/40:50:d0:45:00/05:00:00:00:00/40 tag 10 ncq dma 688128 in
Oct 15 17:37:58 Tower kernel: res 40/00:50:d0:45:00/00:00:00:00:00/40 Emask 0x10 (ATA bus error)
Oct 15 17:37:58 Tower kernel: ata2.00: status: { DRDY }
Oct 15 17:37:58 Tower kernel: ata2.00: failed command: READ FPDMA QUEUED
Oct 15 17:37:58 Tower kernel: ata2.00: cmd 60/40:88:c0:1a:00/05:00:00:00:00/40 tag 17 ncq dma 688128 in
Oct 15 17:37:58 Tower kernel: res 40/00:50:d0:45:00/00:00:00:00:00/40 Emask 0x10 (ATA bus error)
Oct 15 17:37:58 Tower kernel: ata2.00: status: { DRDY }
Oct 15 17:37:58 Tower kernel: ata2: hard resetting link
And on another startup:
Oct 15 17:56:17 Tower kernel: ata4.00: exception Emask 0x50 SAct 0x400000 SErr 0x4090800 action 0xe frozen
Oct 15 17:56:17 Tower kernel: ata4.00: irq_stat 0x00400040, connection status changed
Oct 15 17:56:17 Tower kernel: ata4: SError: { HostInt PHYRdyChg 10B8B DevExch }
Oct 15 17:56:17 Tower kernel: ata4.00: failed command: READ FPDMA QUEUED
Oct 15 17:56:17 Tower kernel: ata4.00: cmd 60/d0:b0:80:58:00/02:00:00:00:00/40 tag 22 ncq dma 368640 in
Oct 15 17:56:17 Tower kernel: res 40/00:b0:80:58:00/00:00:00:00:00/40 Emask 0x50 (ATA bus error)
Oct 15 17:56:17 Tower kernel: ata4.00: status: { DRDY }
Oct 15 17:56:17 Tower kernel: ata4: hard resetting link
Oct 15 17:56:17 Tower kernel: ata3.00: exception Emask 0x50 SAct 0x3000 SErr 0x4890800 action 0xe frozen
Oct 15 17:56:17 Tower kernel: ata3.00: irq_stat 0x08400040, interface fatal error, connection status changed
Oct 15 17:56:17 Tower kernel: ata3: SError: { HostInt PHYRdyChg 10B8B LinkSeq DevExch }
Oct 15 17:56:17 Tower kernel: ata3.00: failed command: WRITE FPDMA QUEUED
Oct 15 17:56:17 Tower kernel: ata3.00: cmd 61/40:60:d0:45:00/05:00:00:00:00/40 tag 12 ncq dma 688128 out
Oct 15 17:56:17 Tower kernel: res 40/00:68:10:4b:00/00:00:00:00:00/40 Emask 0x50 (ATA bus error)
Oct 15 17:56:17 Tower kernel: ata3.00: status: { DRDY }
Oct 15 17:56:17 Tower kernel: ata3.00: failed command: WRITE FPDMA QUEUED
Oct 15 17:56:17 Tower kernel: ata3.00: cmd 61/38:68:10:4b:00/00:00:00:00:00/40 tag 13 ncq dma 28672 out
Oct 15 17:56:17 Tower kernel: res 40/00:68:10:4b:00/00:00:00:00:00/40 Emask 0x50 (ATA bus error)
Oct 15 17:56:17 Tower kernel: ata3.00: status: { DRDY }
Oct 15 17:56:17 Tower kernel: ata3: hard resetting link
Oct 15 17:56:17 Tower kernel: ata2.00: exception Emask 0x50 SAct 0x1e8060 SErr 0x4090800 action 0xe frozen
Oct 15 17:56:17 Tower kernel: ata2.00: irq_stat 0x00400040, connection status changed
Oct 15 17:56:17 Tower kernel: ata2: SError: { HostInt PHYRdyChg 10B8B DevExch }
Oct 15 17:56:17 Tower kernel: ata2.00: failed command: READ FPDMA QUEUED
Oct 15 17:56:17 Tower kernel: ata2.00: cmd 60/40:28:80:15:00/05:00:00:00:00/40 tag 5 ncq dma 688128 in
Oct 15 17:56:17 Tower kernel: res 40/00:a0:80:58:00/00:00:00:00:00/40 Emask 0x50 (ATA bus error)
Oct 15 17:56:17 Tower kernel: ata2.00: status: { DRDY }
Oct 15 17:56:17 Tower kernel: ata2.00: failed command: READ FPDMA QUEUED
Oct 15 17:56:17 Tower kernel: ata2.00: cmd 60/40:30:c0:1a:00/05:00:00:00:00/40 tag 6 ncq dma 688128 in
Oct 15 17:56:17 Tower kernel: res 40/00:a0:80:58:00/00:00:00:00:00/40 Emask 0x50 (ATA bus error)
Oct 15 17:56:17 Tower kernel: ata2.00: status: { DRDY }
Oct 15 17:56:17 Tower kernel: ata2.00: failed command: READ FPDMA QUEUED
Oct 15 17:56:17 Tower kernel: ata2.00: cmd 60/38:78:48:4b:00/05:00:00:00:00/40 tag 15 ncq dma 684032 in
Oct 15 17:56:17 Tower kernel: res 40/00:a0:80:58:00/00:00:00:00:00/40 Emask 0x50 (ATA bus error)
Oct 15 17:56:17 Tower kernel: ata2.00: status: { DRDY }
Oct 15 17:56:17 Tower kernel: ata2.00: failed command: READ FPDMA QUEUED
Oct 15 17:56:17 Tower kernel: ata2.00: cmd 60/08:88:80:50:00/04:00:00:00:00/40 tag 17 ncq dma 528384 in
Oct 15 17:56:17 Tower kernel: res 40/00:a0:80:58:00/00:00:00:00:00/40 Emask 0x50 (ATA bus error)
Oct 15 17:56:17 Tower kernel: ata2.00: status: { DRDY }
Oct 15 17:56:17 Tower kernel: ata2.00: failed command: READ FPDMA QUEUED
Oct 15 17:56:17 Tower kernel: ata2.00: cmd 60/38:90:88:54:00/01:00:00:00:00/40 tag 18 ncq dma 159744 in
Oct 15 17:56:17 Tower kernel: res 40/00:a0:80:58:00/00:00:00:00:00/40 Emask 0x50 (ATA bus error)
Oct 15 17:56:17 Tower kernel: ata2.00: status: { DRDY }
Oct 15 17:56:17 Tower kernel: ata2.00: failed command: READ FPDMA QUEUED
Oct 15 17:56:17 Tower kernel: ata2.00: cmd 60/c0:98:c0:55:00/02:00:00:00:00/40 tag 19 ncq dma 360448 in
Oct 15 17:56:17 Tower kernel: res 40/00:a0:80:58:00/00:00:00:00:00/40 Emask 0x50 (ATA bus error)
Oct 15 17:56:17 Tower kernel: ata2.00: status: { DRDY }
Oct 15 17:56:17 Tower kernel: ata2.00: failed command: READ FPDMA QUEUED
Oct 15 17:56:17 Tower kernel: ata2.00: cmd 60/d0:a0:80:58:00/02:00:00:00:00/40 tag 20 ncq dma 368640 in
Oct 15 17:56:17 Tower kernel: res 40/00:a0:80:58:00/00:00:00:00:00/40 Emask 0x50 (ATA bus error)
Oct 15 17:56:17 Tower kernel: ata2.00: status: { DRDY }
Oct 15 17:56:17 Tower kernel: ata2: hard resetting link
and again:
Oct 15 18:13:54 Tower kernel: r8169 0000:02:00.0: invalid VPD tag 0x00 (size 0) at offset 0; assume missing optional EEPROM
Oct 15 18:13:55 Tower kernel: ata4.00: exception Emask 0x10 SAct 0x1cc000 SErr 0x4090000 action 0xe frozen
Oct 15 18:13:55 Tower kernel: ata4.00: irq_stat 0x00400040, connection status changed
Oct 15 18:13:55 Tower kernel: ata4: SError: { PHYRdyChg 10B8B DevExch }
Oct 15 18:13:55 Tower kernel: ata4.00: failed command: READ FPDMA QUEUED
Oct 15 18:13:55 Tower kernel: ata4.00: cmd 60/38:70:c8:35:00/05:00:00:00:00/40 tag 14 ncq dma 684032 in
Oct 15 18:13:55 Tower kernel: res 40/00:a0:00:43:00/00:00:00:00:00/40 Emask 0x10 (ATA bus error)
Oct 15 18:13:55 Tower kernel: ata4.00: status: { DRDY }
Oct 15 18:13:55 Tower kernel: ata4.00: failed command: READ FPDMA QUEUED
Oct 15 18:13:55 Tower kernel: ata4.00: cmd 60/08:78:00:3b:00/04:00:00:00:00/40 tag 15 ncq dma 528384 in
Oct 15 18:13:55 Tower kernel: res 40/00:a0:00:43:00/00:00:00:00:00/40 Emask 0x10 (ATA bus error)
Oct 15 18:13:55 Tower kernel: ata4.00: status: { DRDY }
Oct 15 18:13:55 Tower kernel: ata4.00: failed command: READ FPDMA QUEUED
Oct 15 18:13:55 Tower kernel: ata4.00: cmd 60/38:90:08:3f:00/01:00:00:00:00/40 tag 18 ncq dma 159744 in
Oct 15 18:13:55 Tower kernel: res 40/00:a0:00:43:00/00:00:00:00:00/40 Emask 0x10 (ATA bus error)
Oct 15 18:13:55 Tower kernel: ata4.00: status: { DRDY }
Oct 15 18:13:55 Tower kernel: ata4.00: failed command: READ FPDMA QUEUED
Oct 15 18:13:55 Tower kernel: ata4.00: cmd 60/c0:98:40:40:00/02:00:00:00:00/40 tag 19 ncq dma 360448 in
Oct 15 18:13:55 Tower kernel: res 40/00:a0:00:43:00/00:00:00:00:00/40 Emask 0x10 (ATA bus error)
Oct 15 18:13:55 Tower kernel: ata4.00: status: { DRDY }
Oct 15 18:13:55 Tower kernel: ata4.00: failed command: READ FPDMA QUEUED
Oct 15 18:13:55 Tower kernel: ata4.00: cmd 60/d0:a0:00:43:00/02:00:00:00:00/40 tag 20 ncq dma 368640 in
Oct 15 18:13:55 Tower kernel: res 40/00:a0:00:43:00/00:00:00:00:00/40 Emask 0x10 (ATA bus error)
Oct 15 18:13:55 Tower kernel: ata4.00: status: { DRDY }
Oct 15 18:13:55 Tower kernel: ata4: hard resetting link
Oct 15 18:13:55 Tower kernel: ata2.00: exception Emask 0x50 SAct 0x3ec0 SErr 0x4090800 action 0xe frozen
Oct 15 18:13:55 Tower kernel: ata2.00: irq_stat 0x00400040, connection status changed
Oct 15 18:13:55 Tower kernel: ata2: SError: { HostInt PHYRdyChg 10B8B DevExch }
Oct 15 18:13:55 Tower kernel: ata2.00: failed command: READ FPDMA QUEUED
Oct 15 18:13:55 Tower kernel: ata2.00: cmd 60/40:30:80:15:00/05:00:00:00:00/40 tag 6 ncq dma 688128 in
Oct 15 18:13:55 Tower kernel: res 40/00:68:00:43:00/00:00:00:00:00/40 Emask 0x50 (ATA bus error)
Oct 15 18:13:55 Tower kernel: ata2.00: status: { DRDY }
Oct 15 18:13:55 Tower kernel: ata2.00: failed command: READ FPDMA QUEUED
Oct 15 18:13:55 Tower kernel: ata2.00: cmd 60/40:38:c0:1a:00/05:00:00:00:00/40 tag 7 ncq dma 688128 in
Oct 15 18:13:55 Tower kernel: res 40/00:68:00:43:00/00:00:00:00:00/40 Emask 0x50 (ATA bus error)
Oct 15 18:13:55 Tower kernel: ata2.00: status: { DRDY }
Oct 15 18:13:55 Tower kernel: ata2.00: failed command: READ FPDMA QUEUED
Oct 15 18:13:55 Tower kernel: ata2.00: cmd 60/38:48:c8:35:00/05:00:00:00:00/40 tag 9 ncq dma 684032 in
Oct 15 18:13:55 Tower kernel: res 40/00:68:00:43:00/00:00:00:00:00/40 Emask 0x50 (ATA bus error)
Oct 15 18:13:55 Tower kernel: ata2.00: status: { DRDY }
Oct 15 18:13:55 Tower kernel: ata2.00: failed command: READ FPDMA QUEUED
Oct 15 18:13:55 Tower kernel: ata2.00: cmd 60/08:50:00:3b:00/04:00:00:00:00/40 tag 10 ncq dma 528384 in
Oct 15 18:13:55 Tower kernel: res 40/00:68:00:43:00/00:00:00:00:00/40 Emask 0x50 (ATA bus error)
Oct 15 18:13:55 Tower kernel: ata2.00: status: { DRDY }
Oct 15 18:13:55 Tower kernel: ata2.00: failed command: READ FPDMA QUEUED
Oct 15 18:13:55 Tower kernel: ata2.00: cmd 60/38:58:08:3f:00/01:00:00:00:00/40 tag 11 ncq dma 159744 in
Oct 15 18:13:55 Tower kernel: res 40/00:68:00:43:00/00:00:00:00:00/40 Emask 0x50 (ATA bus error)
Oct 15 18:13:55 Tower kernel: ata2.00: status: { DRDY }
Oct 15 18:13:55 Tower kernel: ata2.00: failed command: READ FPDMA QUEUED
Oct 15 18:13:55 Tower kernel: ata2.00: cmd 60/c0:60:40:40:00/02:00:00:00:00/40 tag 12 ncq dma 360448 in
Oct 15 18:13:55 Tower kernel: res 40/00:68:00:43:00/00:00:00:00:00/40 Emask 0x50 (ATA bus error)
Oct 15 18:13:55 Tower kernel: ata2.00: status: { DRDY }
Oct 15 18:13:55 Tower kernel: ata2.00: failed command: READ FPDMA QUEUED
Oct 15 18:13:55 Tower kernel: ata2.00: cmd 60/d0:68:00:43:00/02:00:00:00:00/40 tag 13 ncq dma 368640 in
Oct 15 18:13:55 Tower kernel: res 40/00:68:00:43:00/00:00:00:00:00/40 Emask 0x50 (ATA bus error)
Oct 15 18:13:55 Tower kernel: ata2.00: status: { DRDY }
Oct 15 18:13:55 Tower kernel: ata2: hard resetting link
Oct 15 18:13:55 Tower kernel: ata3.00: exception Emask 0x50 SAct 0xc0 SErr 0x4090800 action 0xe frozen
Oct 15 18:13:55 Tower kernel: ata3.00: irq_stat 0x00400040, connection status changed
Oct 15 18:13:55 Tower kernel: ata3: SError: { HostInt PHYRdyChg 10B8B DevExch }
Oct 15 18:13:55 Tower kernel: ata3.00: failed command: WRITE FPDMA QUEUED
Oct 15 18:13:55 Tower kernel: ata3.00: cmd 61/40:30:50:30:00/05:00:00:00:00/40 tag 6 ncq dma 688128 out
Oct 15 18:13:55 Tower kernel: res 40/00:38:90:35:00/00:00:00:00:00/40 Emask 0x50 (ATA bus error)
Oct 15 18:13:55 Tower kernel: ata3.00: status: { DRDY }
Oct 15 18:13:55 Tower kernel: ata3.00: failed command: WRITE FPDMA QUEUED
Oct 15 18:13:55 Tower kernel: ata3.00: cmd 61/38:38:90:35:00/00:00:00:00:00/40 tag 7 ncq dma 28672 out
Oct 15 18:13:55 Tower kernel: res 40/00:38:90:35:00/00:00:00:00:00/40 Emask 0x50 (ATA bus error)
Oct 15 18:13:55 Tower kernel: ata3.00: status: { DRDY }
Oct 15 18:13:55 Tower kernel: ata3: hard resetting link
Any idea what could be going on here?
I am able to remove the new disk and put in the old, but Unraid refuses to accept the new disk into the array because it is 2TB and the new is 6TB. The array does start, but in degraded mode with one disk missing.
Would creating a new config trigger a rebuild?