JorgeB

Start by running memtest.

This means errors were limited to parity2, any unclean shutdowns since last check?

It's normal to still show a few MBs used after emptying a btrfs filesystem.

As long as you backup the data you can do it, in fact you should already have backups. No, I said you won't be able to replace a drive, unless you replace them all with larger capacity devices, starting with parity of course.

Not sure if they share the same BIOS, flash the first one and check. Yes, v5 is very old, hence the device ID issues.

Disk is showing some issue, run an extended SMART test.

Backups? Like mentioned you don't need to do this (though you'll run into trouble in the future if you need to replace one of the SSDs using a spare with the same size, it won't let you).

It would depend if the device was removed from the btrfs pool or not, i.e., if you started the array without it, only by seeing the diags could I be sure. Errors after the first scrub are expected, it's bringing that device up to date with the rest of the pool, as long as all errors were corrected you're fine, balance isn't really needed for this. Yep, as long as it was never removed from the pool you just need start the array with all devices present (all should be green) and run a scrub. Nothing special, always make sure backups are up do date, just in case. Always better to replace than remove, but remove should work in most cases (raid5/6 still has some corner issues). You should also reset the btrfs stats errors, take a look here, it also explains how to better monitor a pool.

IIRC they show up in the syslog with mcelog installed, but in any case it points to a hardware problem.

Yep, though using the beta should be perfectly safe.

You're welcome, but no way the bound controllers were the result of moving the server, why I didn't look at that first, though still strange about the 4TB disks.

AFAIK, not possible to flash it IT mode.

Before starting good idea to check that backups are up to date, then if enable disable array auto-start and reboot, check if the dropped device is already assigned to the pool and all devices have a green ball, if yes start the array and run a scrub, if not do this: If Docker/VM services are using the cache pool disable them, unassign all cache devices, start array to make Unraid "forget" current cache config, stop array, reassign all cache devices (there can't be an "All existing data on this device will be OVERWRITTEN when array is Started" warning for any cache device), re-enable Docker/VMs if needed, start array, run a scrub. P.S. next time please attache diags here instead.

Updating the LSI firmware should solve the missing array disks problem, as for the other missing devices see if they are detected by the LSI BIOS, if they aren't they also won't be detected by Unraid.

ECC RAM should rule that out, could be a board/CPU/controller issue or just a disk returning bad data, unfortunately no easy way to tell except to start swapping things around.

There's nothing in the logs that points to a cause, this together with the server being working in the past points to a hardware problem, difficult to diagnose remotely, if you can try with for example a different PSU.

It's for any flash based device. You don't "need" to use the new alignment, but it *should* improve performance and durability. Note that to re-partition them you'd need to blow up the whole array and start over, can't just do it one device at a time since the devices will be a little smaller with the new layout.

No NIC driver in v6.8 you need to use the beta or get an add-on NIC.

It should be clear if you turn on the GUI help, first set those shares to cache="yes", disable docker service (also VM if applicable), run the mover, when done set the shares to cache="prefer" and select the new cache pool, run the mover again.

You have multiple SATA controllers bound to vfio-pci: 09:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51) Subsystem: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] Kernel driver in use: vfio-pci Kernel modules: ahci 0a:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51) Subsystem: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] Kernel driver in use: vfio-pci Kernel modules: ahci Doesn't explain the 4TB disks, but probably does the other missing ones, unbind them and post new diags.

Fast NVMe device is enough for close to 10GbE line speed, another option is a multiple device SATA SSD raid pool.

Both 4TB drives are being detected but fail to initialize, e.g. : Dec 1 07:18:18 Smaugs-Trove kernel: scsi 6:0:0:0: Direct-Access ATA ST4000VN008-2DR1 SC60 PQ: 0 ANSI: 5 ... Dec 1 07:18:18 Smaugs-Trove kernel: sd 5:0:0:0: [sdb] Synchronize Cache(10) failed: Result: hostbyte=0x04 driverbyte=0x00 Dec 1 07:18:18 Smaugs-Trove kernel: sd 5:0:0:0: [sdb] Stopping disk Dec 1 07:18:18 Smaugs-Trove kernel: sd 5:0:0:0: [sdb] Start/Stop Unit failed: Result: hostbyte=0x04 driverbyte=0x00 Dec 1 07:18:18 Smaugs-Trove kernel: ata6.00: disabled Don't see any sign of the 10TB drives, this to me looks more like a power problem.

Please post the diagnostics: Tools -> Diagnostics

The NIC should auto-link at the max supported speed, if it's not the cable/port it could be a switch or NIC problem.

That's a lot for 10TB, it should be around 18 to 20 hours, unless there's some bottleneck or you have a lot of different size devices in the array.