JorgeB

Since there's nothing relevant logged, one thing you can try is to boot the server in safe mode with all docker containers/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one, including the individual docker containers.

Just that pool

If it's a kernel regression it will need to be fixed upstream in the kernel, then Unraid will inherit it

If you have the diags from the replacement would be interested in seeing them, but to fix the issue, open a terminal window and type: zpool detach flash 14843927469164253652

That suggests to me a CPU performance issue, I would not expect to see so much difference with those numbers. Make sure the CPU is throttling up; also check Settings - Power Mode is set to Best Performance.

Please post the diagsotics during a atransfer, also post the results of a single stream iperf test in one direction, then the other one (-R) to rule out a network problem.

That should not cause issues other than the spam.

Try copying to a disk share (or exclusive share)

Probably, but should be harmless.

That one should be OK

Jun 22 23:31:04 Tower webgui: /boot/config/plugins/dynamix/dynamix.cfg corrupted or missing Check if the file exists: cat /boot/config/plugins/dynamix/dynamix.cfg If it doesn't or looks corrupt, delete it and then make a dummy change to a setting that will recreate this file, for example, change the theme

Nothing relevant was logged that I can see in the last lines before the crash, except maybe the Docker Compose lines; they don't look normal, but it may be worth retesting with the Docker engine disabled, or trying to run half the containers you have; if it still crashes, try the other half. Jun 22 07:00:16 NakUnraid compose.manager: [INFO] [nchan] publisher starting Jun 22 07:05:44 NakUnraid compose.manager: [INFO] [nchan] publisher starting Jun 22 07:30:19 NakUnraid compose.manager: [INFO] [nchan] publisher starting Jun 22 07:33:49 NakUnraid compose.manager: [INFO] [nchan] publisher starting Jun 22 07:35:47 NakUnraid compose.manager: [INFO] [nchan] publisher starting Jun 22 08:00:22 NakUnraid compose.manager: [INFO] [nchan] publisher starting Jun 22 08:03:51 NakUnraid compose.manager: [INFO] [nchan] publisher starting Jun 22 08:04:05 NakUnraid emhttpd: spinning down /dev/sdb Jun 22 08:05:50 NakUnraid compose.manager: [INFO] [nchan] publisher starting Jun 22 08:30:25 NakUnraid compose.manager: [INFO] [nchan] publisher starting Jun 22 08:35:53 NakUnraid compose.manager: [INFO] [nchan] publisher starting Jun 22 08:40:48 NakUnraid compose.manager: [INFO] [nchan] publisher starting Jun 22 16:51:43 NakUnraid rc.rsyslogd: Syslog server daemon... Started.

I assume you mean disk4? Errors with multiple disks, and you are using a controller with SATA multipliers, which are not recommended; it should be replaced when possible: https://forums.unraid.net/topic/102010-recommended-controllers-for-unraid/ Jun 22 18:31:37 BattleStar kernel: ata8: found unknown device (class 0) ### [PREVIOUS LINE REPEATED 1 TIMES] ### Jun 22 18:31:37 BattleStar kernel: ata8: SATA link up 1.5 Gbps (SStatus 113 SControl 310) Jun 22 18:31:41 BattleStar kernel: ata8.00: failed to IDENTIFY (I/O error, err_mask=0x100) Jun 22 18:31:41 BattleStar kernel: ata8.00: revalidation failed (errno=-5) Jun 22 18:31:42 BattleStar kernel: ata8: hard resetting link Jun 22 18:31:42 BattleStar kernel: ata7.02: SATA link up 6.0 Gbps (SStatus 133 SControl 330) Jun 22 18:31:47 BattleStar kernel: ata8: link is slow to respond, please be patient (ready=0) Jun 22 18:31:48 BattleStar kernel: ata7.02: qc timeout after 5000 msecs (cmd 0xec) Jun 22 18:31:48 BattleStar kernel: ata7.02: failed to IDENTIFY (I/O error, err_mask=0x4) Jun 22 18:31:48 BattleStar kernel: ata7.02: revalidation failed (errno=-5) Jun 22 18:31:48 BattleStar kernel: ata7.15: hard resetting link Jun 22 18:31:49 BattleStar kernel: ata7.15: SATA link up 6.0 Gbps (SStatus 133 SControl 300) Jun 22 18:31:49 BattleStar kernel: ata7.00: hard resetting link Jun 22 18:31:49 BattleStar kernel: ata7.00: SATA link up 6.0 Gbps (SStatus 133 SControl 330) Jun 22 18:31:49 BattleStar kernel: ata7.01: hard resetting link Jun 22 18:31:50 BattleStar kernel: ata7.01: SATA link down (SStatus 0 SControl 330) Jun 22 18:31:51 BattleStar kernel: ata7.02: SATA link up 6.0 Gbps (SStatus 133 SControl 330) Jun 22 18:31:51 BattleStar kernel: ata7.00: configured for UDMA/133 Jun 22 18:31:51 BattleStar kernel: ata7.02: model number mismatch 'ST6000VX001-2BD186' != 'ST6000VX001' Jun 22 18:31:51 BattleStar kernel: ata7.02: revalidation failed (errno=-19) Jun 22 18:31:51 BattleStar kernel: ata7.02: limiting SATA link speed to 3.0 Gbps

I would suspect the change in stability were more caused by the other changes than the downgrade, but you can try reupgrading in the near future while leaving everything else intact

If it only mounted read only, yes, you will need to copy the data someplace else and then reformat.

Jun 22 15:05:05 Towerx2 kernel: mpt3sas_cm0: diag reset: FAILED Jun 22 15:05:05 Towerx2 kernel: mpt3sas_cm0: failure at drivers/scsi/mpt3sas/mpt3sas_scsih.c:12349/_scsih_probe()! HBA was failing to initialize; the BIOS update may have helped with that

Since the drive is encrypted, you need to decrypt it first, one of the reasons I don't recommend encryption unless absolutely necessary; it just complicates things, assuming you are using a keyfile and it's loaded: cryptsetup luksOpen /dev/sdf1 sdf1 --key-file=/root/keyfile

I don't think so, to rule out a FM issue, try using Mindnight COmmender (mc on the console).

If the NVMe device is dropping try this: On Main click on the flash drive, scroll down to "Syslinux Configuration", make sure it's set to "menu view" and add this to your default boot option, after "append initrd=/bzroot" nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off e.g.: append initrd=/bzroot nvme_core.default_ps_max_latency_us=0 pcie_aspm=off pcie_port_pm=off Reboot (or power cycle the server if just a reboot doesn't bring the device back) and then see if it makes a difference.

Stock Unraid can only spin down SATA drives, you can try the SAS spindown plugin, but not all SAS drives support spin down.

You can see here: https://forums.unraid.net/topic/102010-recommended-controllers-for-unraid/

Yes, already mentioned it above.

This would suggest a hardware problem, most often bad RAM, so I would recommend investigating this first. If you ahve enough space you can use the mover to empty an array drive, format it, then move on to the next one

Actual disk also has filesystem issues, you can try the recovery options here: https://forums.unraid.net/topic/46802-faq-for-unraid-v6/page/2/#findComment-543490 Since the problem is the same, they could be used with the actual disk or the emulated one

appdata is set to use cache_nvme pool, but no such pool exists, should it exist?