JorgeB

I assume these are the pool members? And they are all from the same pool? /dev/nvme0n1p1: UUID="299134d5-d2d5-41d9-9bfa-3e98d1e3f1d5" TYPE="crypto_LUKS" PARTUUID="e414749a-01" /dev/nvme2n1p1: UUID="8416d39f-e189-42e1-a1b8-956a23215ce8" TYPE="crypto_LUKS" PARTUUID="b660afb9-01" /dev/nvme3n1p1: UUID="2d3fcb77-83de-48a1-b4c2-11d4fe5ba331" TYPE="crypto_LUKS" PARTUUID="0262e045-35e3-48a5-8af5-47f862bab439" /dev/nvme4n1p1: UUID="b2d6b8d5-d4db-4938-a66b-907cd3ffa1ee" TYPE="crypto_LUKS" PARTUUID="d8fe2adb-7f6b-4fbc-9866-3daf82e56b31" If yes, on main, create a new pool with the name you like and 4 slots assign the pool devices, leave the filesystem set to auto It should ask for the encryption key/passphrase, then start the array to import the pool If it fails to mount, post the diags with the array started

The server is running out of RAM, you have a rogue process (starter.sh) using approximately 61GB That is likely running inside a container.

There are multiple disks dropping across different controllers, suggesting more of a power issue; it could be the PSU itself, or if there are any splitters being used, it could also be related to that

Just did a test and it's not possible. With RAIDZ vdevs in the pool, the zpool remove command is completely blocked, so even after adding a new vdev to special or dedup, it still doesn't allow you to remove the other one, since the new devices are smaller, and you cannot also do a replacement. I'm afraid the only way would be to back up and recreate the pool.

Yep, 8 mirrored vdevs. Should resume; it can go back just a little, but it should not restart. This is the scanned metadata, and the value should keep increasing as the resilver progresses, and it typically reaches the full capacity (1 TB in this case) before the pool is fully resilvered.

I don't see any relevant logged; post a new syslog after the next crash, and mention the aprox. crash timecode, together with the diags mostly to see the hardware.

Click the link If that fails, see if you can get the syslog cp /var/log/syslog /boot/syslog.txt

You can use the steps here: https://forums.unraid.net/topic/46802-faq-for-unraid-v6/#findComment-511923

If you have a backup of the pool you can try the repair, but with btrfs, this is risky; if there are no backups, see if you can mount it read-only to create some first. https://forums.unraid.net/topic/46802-faq-for-unraid-v6/page/2/#findComment-543490

Jun 7 10:00:21 Endurance kernel: nvme nvme0: controller is down; will reset: CSTS=0xffffffff, PCI_STATUS=0x10 Jun 7 10:00:21 Endurance kernel: nvme nvme0: Does your device have a faulty power saving mode enabled? Jun 7 10:00:21 Endurance kernel: nvme nvme0: Try "nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off" and report a bug Jun 7 10:00:21 Endurance kernel: nvme0n1: I/O Cmd(0x2) @ LBA 2048, 8 blocks, I/O Error (sct 0x3 / sc 0x71) Jun 7 10:00:21 Endurance kernel: I/O error, dev nvme0n1, sector 2048 op 0x0:(READ) flags 0x80700 phys_seg 1 prio class 2 Jun 7 10:00:21 Endurance kernel: nvme 0000:03:00.0: enabling device (0000 -> 0002) Jun 7 10:00:21 Endurance kernel: nvme nvme0: Disabling device after reset failure: -19 Device is still dropping. Since you are already using the optional kernel parameters, my recommendation would be to try a different brand/model one, it may also be worth trying a different m.2 slot if available.

The server doesn't have internet connectivity; start by deleting/renaming /config/network.cfg on the flash drive to revert to default settings, then post new diags.

Do you want to keep the devices unassigned, or do you want to assign them in a pool? Also post the output from blkid

Server doesn't have internet connectivity; as a first step, rename/delete /config/network.cfg on the flash drive to revert to default DHCP and post new diagnostics.

You would need to test, for example, just having half of the container running; if it doesn't crash, try the other half, then keep drilling down.

You can now reset the pool stats, then monitor for any new errors.

There are still no controller-related errors, which is not normal. Reboot to clear the log and any weirdness that could be going on and post new diags if it happens again.

It should be the standard procedure, for that the 4Kn parity should not change anything.

Please create a new thread in the KVM section and add the diagnostics

You will need to find out which segment of the network is limiting the speed.

Python segfaulted, which is not normal, you can try booting in safe mode to rule out a plugin issue and/or running memtest in case it's a hardware problem.

Emulated disk is mounting, if contents look correct you can rebuild. https://docs.unraid.net/unraid-os/using-unraid-to/manage-storage/array/replacing-disks-in-array/#re-enabling-a-disabled-disk-rebuilding-onto-itself

I did a quick test and confirmed the 4Kn parity is indeed the issue. It still works if the replacement disk is also 4Kn, in case you have one available, the other option would be doing a parity swap. I'll raise this with LT, it should be a pretty rare occurrence, but if it's an easy fix, it would be worth doing it, IMO.

That means the test failed, and the device should be replaced.

Please post the diagnostics.