JorgeB

rsync -X copies the extended attributes, resiserfs has issues with those.

You have, it's just not mounting, Please post the diagnostics.

I checked before and the parity swap procedure completed successfully without any errors, so all should be fine.

If it's doing weird noises it's likely the disk, assuming power cable/slot was already replaced/swapped.

Check filesystem on disk3.

That turned out to be a false issue, at least so far, and it's not the problem here: Oct 19 09:50:38 Tower shutdown[16774]: shutting down for system reboot ... Oct 19 09:51:45 Tower root: Waiting on VMs to hibernate............................................................ Oct 19 09:51:45 Tower root: Domain 86349a06-14bd-3361-c257-9064d80fe376 is being shutdown Oct 19 09:51:45 Tower root: Oct 19 09:51:45 Tower root: Domain 37dc3718-c315-0f42-39de-4a1ab8e25771 is being shutdown Oct 19 09:51:45 Tower root: VM timeout is set to 60 secs, and it wasn't enough, and looks like they didn't force shutdown either since libvirt was still in use after this.

This is usually caused by having NFS enable, if possible use SMB only or try disabling hard links.

There might be, but don't know how, someone else might have a suggestion.

Looks more like a power/connection problem, I would recommended replacing/swapping cables/slot to rule those out if it happens again to the same disk, then and if the emulated disk is mounting and contents look correct you can rebuild on top using the instructions above.

Oct 19 09:53:41 Tower emhttpd: shcmd (37287): umount /mnt/cache Oct 19 09:53:41 Tower root: umount: /mnt/cache: target is busy. Something was still using cache.

Yes, same issue as linked above, see that thread for some possible workarounds.

Not the same, diags you posted are after rebooting.

Could be, you can try swapping drives and cables with the onboard SATA.

You're still having multiple ATA errors on multiple disks: Oct 18 22:09:43 Cube kernel: ata10.00: status: { DRDY } Oct 18 22:09:43 Cube kernel: ata10.00: failed command: WRITE FPDMA QUEUED Oct 18 22:09:43 Cube kernel: ata10.00: cmd 61/40:88:08:58:00/05:00:00:00:00/40 tag 17 ncq dma 688128 out Oct 18 22:09:43 Cube kernel: res 40/00:00:00:4f:c2/00:00:00:00:00/00 Emask 0x4 (timeout) Oct 18 22:09:43 Cube kernel: ata10.00: status: { DRDY } Oct 18 22:09:43 Cube kernel: ata10.00: failed command: WRITE FPDMA QUEUED Oct 18 22:09:43 Cube kernel: ata10.00: cmd 61/f8:90:48:5d:00/04:00:00:00:00/40 tag 18 ncq dma 651264 out Oct 18 22:09:43 Cube kernel: res 40/00:01:01:00:00/00:00:00:00:00/00 Emask 0x4 (timeout) Oct 18 22:09:43 Cube kernel: ata10.00: status: { DRDY } Oct 18 22:09:43 Cube kernel: ata10: hard resetting link Oct 18 22:09:43 Cube ntpd[1794]: kernel reports TIME_ERROR: 0x41: Clock Unsynchronized Oct 18 22:09:48 Cube kernel: ata10: link is slow to respond, please be patient (ready=0) Oct 18 22:09:53 Cube kernel: ata10: COMRESET failed (errno=-16) Oct 18 22:09:53 Cube kernel: ata10: hard resetting link Oct 18 22:09:53 Cube kernel: ata10: SATA link up 6.0 Gbps (SStatus 133 SControl 310) Oct 18 22:09:53 Cube kernel: ata10.00: configured for UDMA/133 Oct 18 22:09:53 Cube kernel: ata10: EH complete Oct 18 22:10:11 Cube kernel: ata8.00: READ LOG DMA EXT failed, trying PIO Oct 18 22:10:11 Cube kernel: ata8: failed to read log page 10h (errno=-5) Oct 18 22:10:11 Cube kernel: ata8.00: exception Emask 0x1 SAct 0xc02000 SErr 0x0 action 0x0 Oct 18 22:10:11 Cube kernel: ata8.00: irq_stat 0x40000001 Oct 18 22:10:11 Cube kernel: ata8.00: failed command: READ FPDMA QUEUED Oct 18 22:10:11 Cube kernel: ata8.00: cmd 60/20:68:00:ee:4a/00:00:0a:00:00/40 tag 13 ncq dma 16384 in Oct 18 22:10:11 Cube kernel: res 51/04:b8:80:87:00/00:00:00:00:00/40 Emask These are usually a power/connection problem, and the data should come back after a reboot.

You're still getting an unclean shutdown, first thing you can do is to press array stop and see how long it takes to stop, and if it does, second thing is to post the diags saved on the flash drive after the latest unclean shutdown, they are generated automatically.

Could be a specific board issue, look for a BIOS update and/or try toggling any relevant BIOS options.

Yep, you should replace it ASAP.

It's logged as a disk problem, run an extended SMART test.

Nothing NVMe related is being detected by Linux, this is a hardware issue, but no idea how it's being detected in the BIOS.

Yes, that's why I said to look at the tests, depending on the devices used (and the number of them) it might be the same, and with 6 HDDs it will.

Please post diags.

Same, values on the actual diags are correct, looks like just a GUI issue, and of course the fact that it prevents spindown: [sde] => 0 0 22791858 13954498 [sdc] => 0 0 111481 174 3rd column is total reads, 4th is total writes.

Oct 18 19:19:02 Cube kernel: ata10: SATA link down (SStatus 0 SControl 300) Oct 18 19:19:02 Cube kernel: ata10.00: disabled Disk dropped offline, try replacing cables.

OK, if it's only that and it still keeps happening if might really be a visual bug, but it's very strange, you didn't post new diags now, but in the old ones the actual reads/writes values for both devices are completely different, not what the GUI was showing when the screenshot was taken.

Looks like there never was a valid xfs fileystem on that disk, most likely it was added to the array but never formatted.