JorgeB

If you have a backup of the pool you can try the repair, but with btrfs, this is risky; if there are no backups, see if you can mount it read-only to create some first. https://forums.unraid.net/topic/46802-faq-for-unraid-v6/page/2/#findComment-543490

Jun 7 10:00:21 Endurance kernel: nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10 Jun 7 10:00:21 Endurance kernel: nvme nvme0: Does your device have a faulty power saving mode enabled? Jun 7 10:00:21 Endurance kernel: nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off" and report a bug Jun 7 10:00:21 Endurance kernel: nvme0n1: I/O Cmd(0x2) @ LBA 2048, 8 blocks, I/O Error (sct 0x3 / sc 0x71) Jun 7 10:00:21 Endurance kernel: I/O error, dev nvme0n1, sector 2048 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 Jun 7 10:00:21 Endurance kernel: nvme 0000:03:00.0: enabling device (0000 -> 0002) Jun 7 10:00:21 Endurance kernel: nvme nvme0: Disabling device after reset failure: -19 Device is still dropping. Since you are already using the optional kernel parameters, my recommendation would be to try a different brand/model one, it may also be worth trying a different m.2 slot if available.

The server doesn't have internet connectivity; start by deleting/renaming /config/network.cfg on the flash drive to revert to default settings, then post new diags.

Do you want to keep the devices unassigned, or do you want to assign them in a pool? Also post the output from blkid

Server doesn't have internet connectivity; as a first step, rename/delete /config/network.cfg on the flash drive to revert to default DHCP and post new diagnostics.

You would need to test, for example, just having half of the container running; if it doesn't crash, try the other half, then keep drilling down.

You can now reset the pool stats, then monitor for any new errors.

There are still no controller-related errors, which is not normal. Reboot to clear the log and any weirdness that could be going on and post new diags if it happens again.

It should be the standard procedure, for that the 4Kn parity should not change anything.

Please create a new thread in the KVM section and add the diagnostics

You will need to find out which segment of the network is limiting the speed.

Python segfaulted, which is not normal, you can try booting in safe mode to rule out a plugin issue and/or running memtest in case it's a hardware problem.

Emulated disk is mounting, if contents look correct you can rebuild. https://docs.unraid.net/unraid-os/using-unraid-to/manage-storage/array/replacing-disks-in-array/#re-enabling-a-disabled-disk-rebuilding-onto-itself

I did a quick test and confirmed the 4Kn parity is indeed the issue. It still works if the replacement disk is also 4Kn, in case you have one available, the other option would be doing a parity swap. I'll raise this with LT, it should be a pretty rare occurrence, but if it's an easy fix, it would be worth doing it, IMO.

That means the test failed, and the device should be replaced.

Please post the diagnostics.

Unfortunately, there's nothing relevant logged, this can also be a hardware issue, one thing you can try is to boot the server in safe mode with all docker containers/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one, including the individual docker containers.

TPM is only supported for bare metal, not even passing through a TPM module can be used with a virtualized install, you can still use internal boot, but will need to keep the flash drive for licensing.

Iperf confirms it's a LAN problem; there's possibly some gigabit segment on the it, limiting the speed.

Enable the syslog server and post that together with fresh diagnostics if it happens again.

Jun 6 17:27:41 Rocinante php-fpm[8273]: [WARNING] [pool www] child 329491 exited on signal 9 (SIGKILL) after 476.489707 seconds from start Jun 6 17:35:04 Rocinante php-fpm[8273]: [WARNING] [pool www] child 333974 exited on signal 9 (SIGKILL) after 276.942640 seconds from start Jun 6 17:44:05 Rocinante php-fpm[8273]: [WARNING] [pool www] child 336161 exited on signal 9 (SIGKILL) after 644.283148 seconds from start Jun 6 17:50:42 Rocinante php-fpm[8273]: [WARNING] [pool www] child 341492 exited on signal 9 (SIGKILL) after 555.157480 seconds from start Jun 6 18:25:33 Rocinante php-fpm[8273]: [WARNING] [pool www] child 346740 exited on signal 9 (SIGKILL) after 589.301096 seconds from start In my experience, these errors can be the result of the server being close to exhausting the memory, GUI can become extremely slow, like 1 minute to open the dashboard, try limiting the memory for VMs/docker services, or adding a little more RAM. It could also be one or more containers hogging the CPU, try pinning only some cores to them, and leave cores 0/1 available for Unraid. Also, recommend trying a couple of other things, go to Settings - Global share settings and set the Number of fuse File Descriptors to the max, and enable this: https://docs.unraid.net/unraid-os/release-notes/7.0.0/#excessive-flash-drive-activity-slows-the-system-down

Normal syslog starts over after every boot. Enable the syslog server and post that after the next crash.

Try rebooting, if it still shows the same, post the diagnostics.

It's possible with mirrors only, not raidz, your pool has raidz vdevs, so not possible. While I never tested this, it may be possible, if you have extra SATA ports, to add a new vdev with the smaller devices to the existing special and dedup vdevs, then remove the other vdev (using the CLI), I can test this later to confirm if you like.

There's a known issue with the stock Syslinux UEFI bootloader and newer AGESA, you can use the Limine bootlçader if you want to boot from a flash drive; internal boot is not affected since it uses the Grub bootloader. https://forums.unraid.net/topic/196349-black-screen-after-usb-boot-msi-x870e-carbon-ryzen-9950x-rtx-4070-ti-super/#findComment-1607672