JorgeB

The new diagnostics are very useful. Now that the former parity disk (disk3) is on the ASMedia controller, it also gets corrupted after a fresh format, so it's not just the onboard SATA ports. On the previous boot the onboard SATA controller also logged "host bus error" on writes to disk2, which is where the 336 errors come from. That error means the controller failed to move data to or from system memory. It's not a cable or SATA link problem, which matches what you saw when swapping cables and ports. The disks themselves look healthy. Both SATA controllers on that board connect through the X570 chipset, and so do most USB ports, which could also explain the USB disconnects. The NVMe slot your Intel SSD is in connects directly to the CPU. The RAM test and Memtest both passed, but they don't test disk data being moved to and from memory, so for now I'd suspect the chipset or board first and RAM second. A quicker test than formatting and checking filesystems is to write known data straight to a disk and read it back. With the array stopped, run this, replacing sdX with the disk's current device letter from the Main page: dd if=/dev/urandom of=/tmp/test bs=1M count=4000 md5sum /tmp/test dd if=/tmp/test of=/dev/sdX bs=1M oflag=direct conv=fsync status=progress for i in 1 2 3; do dd if=/dev/sdX bs=1M count=4000 iflag=direct status=none | md5sum; done All four checksums should match. This overwrites the start of the disk, so that disk will need formatting again afterwards. That's fine here since there's no data. Please run it on: 1. A disk on the onboard SATA ports (disk1 was sde in these diagnostics). 2. The disk on the ASMedia ports (disk3, was sdc). 3. The Intel NVMe (/dev/nvme0n1), but only if there's nothing on it you need. If only the SATA disks fail and the NVMe passes every time, that points to the chipset or board. If the NVMe also fails, RAM or the CPU's memory controller are more likely, and then your one-DIMM-at-a-time rounds with this test are the next step. Also worth trying, retesting after each change: - Update the BIOS. A.N0 isn't the newest for this board as far as I know, so please check MSI's site. - In the BIOS, look under the PCIe settings for a chipset or PCIe "Gen" option and set it to Gen3 instead of Auto/Gen4. - Keep XMP disabled while testing. If the SATA disks still fail after all that while the NVMe passes, I'd consider the board faulty. Your earlier corrupted HDD files under Windows would fit that too. Please post the results and new diagnostics.

Good detailed testing, that was very well done. 1. Yes, with the PSU, RAM and board each replaced and the same sync flood still happening, the CPU is now the main suspect. The 5700X swap should confirm it. 2. Leave it on Auto for now, so the CPU swap is the only change. I would also undo the performance governor change before installing the new CPU, or keep it, but then keep it the same for the whole test. Just don't change both at the same time. After the new CPU has proven stable, you can set Typical Current Idle anyway, it's a recommended setting for Ryzen. 3. With the 5700X, check the syslog after boot for the amd64_edac driver loading, e.g. "EDAC MC0: Giving out device to module amd64_edac" and a line reporting that ECC is enabled, or run dmesg | grep -i edac. If there's nothing, check the BIOS for an ECC option (usually under AMD CBS, UMC/DDR options) and set it to Enabled. Also note that the DIMMs must be unbuffered ECC (ECC UDIMM). Registered DIMMs won't work, and non-ECC DIMMs obviously won't give you ECC. 4. It's possible that a software change, for example in power management, exposed a CPU that was already marginal, but there's no way to prove that from the logs. CPUs can also degrade over time. Either way, the CPU detects this error in hardware, and if the 5700X is stable on the same Unraid release, then the 5700G was the problem. 5. Unlikely, but it can't be completely excluded. I wouldn't test it now. If the 5700X still crashes, then try the NVMe in the other M.2 slot, or remove it temporarily. One more thing: if it crashes again, boot straight into Unraid and don't enter the BIOS first, because a reboot from the BIOS can overwrite the reset reason. Keep the syslog mirror enabled, and post new diagnostics if it crashes with the new CPU.

Thanks, that's useful. A freshly formatted disk2 failing the filesystem check right away, plus parity sync errors with no disk errors in the log, confirms that data is getting corrupted somewhere between the disks and RAM, not that one disk is bad. Since XMP was already off, RAM speed is not the cause. It's still between the RAM sticks, the memory slots or memory controller, and the board's SATA, USB or power side. The USB disconnects you mentioned also fit a board or power problem. Note that the dd command I posted doesn't read the whole disk. count=20000 with bs=1M reads only the first 20 GB, so it should take about 2 minutes per run. Run it 3 times on the same disk and compare the checksums. An even quicker test that doesn't use the disks at all, from the console: dd if=/dev/urandom of=/tmp/ramtest bs=1M count=4000 for i in 1 2 3 4 5 6 7 8 9 10; do md5sum /tmp/ramtest; done rm /tmp/ramtest /tmp is in RAM, so all 10 checksums should be the same. If any of them differ, the problem is RAM or the memory controller, and you can skip the disk tests for that configuration. Your test plan looks good. To keep each round short: - You don't need parity while testing. Unassign it (Tools > New Config, keep the current assignments, then unassign parity) so you don't need a parity sync after each change. - For the filesystem test, format the data disks, stop the array, run the filesystem check with -n (no repair), then start the array and reboot once and check again. On good hardware, a fresh filesystem passes every time. Don't repair at all. If it fails, just note it and format again in the next round. - Add one more round with a single known-good DIMM and the data disks on different SATA ports and cables. If you can, use the ASMedia ports, where parity and the SSD are now, instead of the chipset ports. Both data disks are on the onboard AMD SATA controller, and disk2 also had read errors there in your first setup. - It's also worth checking for a newer BIOS for the board. The parity check you ran is a correcting check, so the 282 sync errors were "corrected" into parity. That doesn't matter here because there's no data and parity will be rebuilt once the hardware is stable. Post the results of each round, and new diagnostics once you find a configuration that passes.

Disk2 isn't a problem. Since it was precleared, Unraid added it to the array without clearing it again, but it has no filesystem yet, so it will show as unmountable until you format it. Don't format anything yet, though. Disk1 is the real issue, and the log doesn't point to a failing disk. It points to data being corrupted somewhere between the disk and RAM: - On the first array start, disk1 failed to mount with a CRC error in one XFS allocation group header. - After the reboot, with nothing written to the disk in between, the same mount reported bad headers in five allocation groups. - After the filesystem repair, the next mount failed again on a header the repair had just rewritten. Reading the same unchanged disk and getting different results means the reads (or the repair's writes) are not reliable. There are no ATA errors, all disks have 0 UDMA CRC errors and SMART is clean, so the most likely suspect is RAM or the memory controller, then the SATA port, cable or power. A Memtest pass doesn't fully rule out RAM, especially with XMP enabled. I suggest: 1. Stop running filesystem repairs on disk1 for now. Repairing on unstable hardware can make the damage worse. 2. In the BIOS, load the defaults and make sure XMP (A-XMP on MSI) is disabled, so the RAM runs at its default speed. If possible, also test with one DIMM at a time. 3. Do a read test. With the array stopped, run this twice on disk1 (check its current device letter on the Main page, it was sdd in these diagnostics) and compare the two results: dd if=/dev/sdd bs=1M count=20000 status=progress | md5sum It only reads the disk. If the two checksums differ, the read path is unstable. Repeat the test after each change: RAM at defaults, a single DIMM, then disk1 on a different SATA port and cable. 4. Once the checksums match consistently, run the filesystem check on disk1 once more and see if it mounts and keeps mounting after a reboot. Also, is there any data on disk1 you need? If this is a fresh setup with nothing important on it, once the hardware is stable it is simpler to format both data disks and start clean. Leave the parity check you started as non-correcting. If it finds sync errors, that would also fit unstable reads. Please post new diagnostics after the tests.

The "mcelog does not support this processor" line is just Fix Common Problems failing to decode the event on this CPU family. The actual events are in the syslog, already decoded by the kernel. There are five of them since the last boot (Sep 28 to Nov 12), all on CPU 1. None of these look like real hardware errors. Every record has an IPID of zero and is in a bank the kernel reports as reserved, so it can't be tied to any part of the CPU. The status values also look like garbage, for example two of them are 0x9090909090909090, and others combine flags that don't make sense together. This pattern has been seen on other Zen 3 systems, also always on CPU 1, and looks like a platform or firmware quirk rather than a failing component. Nothing else in the diagnostics suggests a problem. There are no crashes, call traces, or filesystem checksum errors, the last parity sync finished with 0 errors, and your RAM is already running at stock speed. I suggest: - Check whether MSI has a newer BIOS than 2.90 for your board, and update if there is one. - Otherwise keep using the server normally. You can ignore this alert in Fix Common Problems. It will also clear after the next reboot if no new event is logged. - If you ever see crashes, lockups, filesystem checksum errors, or parity sync errors, run memtest and post new diagnostics.

Good to hear the VM is working well now. While the VM is running, the card is fully controlled by the NVIDIA driver inside Windows, and Unraid can't change the fan speed from the host. If the fan stays at max even with the Windows driver loaded and Afterburner can't lower it, that points to the card itself rather than the Unraid update. Please try these: 1. Shut the server down completely, unplug the power cable for a minute or two, then power it back on. A normal reboot doesn't fully reset the card. 2. When the server powers on, watch the fan during POST and the BIOS screen, before Unraid starts loading. Is it already at max there? 3. In Windows, check GPU-Z or Afterburner and note the fan speed (% and RPM) and the GPU temperature it reports, at idle and under load. If the fan is already at max during POST, or the fan RPM shows 0 or nothing while it is clearly spinning fast, the card's fan or its speed sensor is likely failing. In that case the card goes to full speed as a safety fallback, and no OS or driver setting will change it.

It may not be possible, most card readers don't have a unique GUID, so it may not work.

There are a few separate things here. The VM crash: the server ran out of RAM. When the Windows VM was running, the kernel killed it to free memory (Out of memory: Killed process ... qemu-system-x86, Windows11). The Windows VM was using about 20GB, the Home Assistant VM about 4 GB, and the Satisfactory server, Immich and Jellyfin also need memory. The ZFS cache can use up to about 6.4 GB too. With 32 GB there isn't enough RAM for all of that. Edit the Windows VM and lower its memory to 12 or 14 GB, then try again. nvidia-smi: this error is expected. The 2070 is bound to VFIO (Tools > System Devices) so it can be passed to the VM, which means the host NVIDIA driver can't use it. If the card is only for the VM, you don't need the Nvidia Driver plugin, and you can remove it. The GPU Statistics plugin is also trying to read the card, which adds the repeated NVRM messages to your log. The fan: when the card is bound to VFIO and the VM isn't running, no driver controls it, and some cards, especially blower types, spin their fan fast in that state. Please check two things: 1. Does the fan go back to normal once the Windows VM is running with the NVIDIA driver installed inside Windows? 2. Was the card already bound to VFIO before you updated to 7.4.0, or did you change anything else around the update? If the VM stops again after lowering its memory, please post new diagnostics right after it happens.

The diagnostics show the server finishes booting normally: all plugins install, the go file runs, the array starts, and the console login processes are running. So the login prompt is being started, it's just not being shown on the monitor. The RTX 3060 is the only video card in the server, so the local console runs on it, and the screen stops updating at about the point where the Nvidia driver plugin installs during boot. This looks like the console display freezing once the Nvidia driver takes over the card, rather than a boot problem. What exactly did you change? In particular, was there a different video card before, or did you move the 3060 to another slot or the monitor to another output? To confirm, please try two things: 1. With the server booted normally, press Alt+F2 on the console keyboard. If a login prompt shows up on that second console, the display is working and only the first console was affected. 2. Reboot and select Safe Mode in the boot menu. Safe Mode doesn't load plugins, so the Nvidia driver isn't loaded. If the login prompt appears in Safe Mode, that confirms the driver is involved. Also, the Fix Common Problems "Out Of Memory" error comes from the Nvidia driver reporting a GPU memory allocation failure. It is not the server running out of RAM, so you can ignore it for this.

Glad it's back. The diagnostics show what happened, and it's not related to the cache pool. The server's network connection keeps dropping. Since Nov 11 the network port lost its link >100 times, 80 of them today. Each time, the server loses its IP address for a few seconds and restarts the web server (nginx). At 04:07 this morning one of those restarts happened while the address was gone, so nginx couldn't start and stayed stopped. Plex and the other containers don't depend on nginx, so they came back by themselves after each drop. At 08:12 the link also came back at only 100 Mbps instead of 1 Gbps, and the network driver logged "Downshift occurred from negotiated speed 1Gbps to actual speed 100Mbps, check cabling!". That points to the cable, the router or switch port, or the server's network port. Please try this: 1. Replace the network cable with another one, ideally new or known good, and plug it into a different port on the router or switch. 2. Then check whether the drops stopped. In the terminal, run: grep -c "Link is Down" /var/log/syslog Note the number, run it again a day later, and post both numbers. The count should no longer go up. If the GUI stops responding again while the containers still work, run /etc/rc.d/rc.nginx start and it should come back. If the link keeps dropping with a different cable and port, let me know. Your board has a second network port (Intel), and we can move the server to that one. The good news: the parity check finished with 0 errors, so you can now do the last cache pool step: 1. Stop the array. 2. On Main, click the pool's name ("Cache"), then click "Remove Pool". 3. Click "Add Pool", name it cache again, with 1 slot. 4. Assign the NVMe ending in 647ACC to it, and leave the file system set to auto. 5. Start the array. The pool should now mount as a normal single-device pool, without the degraded warning. If anything looks different, post new diagnostics. Once everything looks right, you can turn array auto start back on (Settings > Disk Settings > Enable auto start).

Thanks, that output helps. The log space is fine, so that's not the problem. The web server (nginx) wasn't running at all, and "restart" only works when it's already running. That's why it just said "Not running". Please start it instead: /etc/rc.d/rc.nginx start It should say "Started". Then try the GUI again by IP address. If it says "Failed", run this and post the output: /etc/rc.d/rc.nginx check Either way, we need to see why it stopped. If the GUI comes back, download new diagnostics from Tools > Diagnostics and attach them (Edit post works, like last time). Don't reboot first. The log from before a reboot shows what happened. After that, once the parity check has finished, you can continue with the pool reimport.

Since Plex, Radarr and Sonarr still work, the server itself, the array and Docker are running fine. Only the web interface isn't answering. First, open the GUI using the server's IP address, the same one you use for Radarr or Sonarr, but without the port. For example, if Radarr is at http://192.168.1.50:7878, try http://192.168.1.50. Don't use the server name or a bookmark. If that still doesn't load, connect with SSH, or log in at a monitor and keyboard on the server as root, and run these one at a time: df -h /var/log /etc/rc.d/rc.nginx restart diagnostics The first command shows whether the log space is full. The second restarts the web server. The last one saves a diagnostics zip to the logs folder on the boot device. Then try the GUI again. If it comes back, download fresh diagnostics from Tools > Diagnostics and attach them here (Edit post works, like last time). If it doesn't, post the output of the first command. Please don't reboot for now. The parity check may still be running, and diagnostics taken before a reboot show what went wrong. Once the GUI is back and the parity check has finished, continue with the pool reimport from my last post.

The diagnostics confirm the BIOS changed from F54a to F54c. On the previous boot, with F54a, all 8 SATA devices were detected normally. Now SATA ports 1-4 and 7 (as Linux numbers them) see no device at all. These are the two SSDs and disks 1, 3 and 5. The other 3 ports on the same controller work. The "SATA link down" messages mean those ports don't see a drive connected at all, which usually points to power or cabling. It could also be a SATA setting the new BIOS or the optimized defaults changed. First, and most important: do not start the array, do not use New Config, and do not change any disk assignments while the disks are missing. Once all disks are detected again, the array should start normally with the current assignments. Then please check: 1. Power off and check the power and SATA cables of the missing drives. Pay special attention to whether those 5 drives share the same power cable or splitter, since it's easy to disturb something when working inside the case. 2. Enter the BIOS and check whether the missing drives are listed there. If the BIOS doesn't see them, Unraid won't either. 3. Look at the SATA and M.2 settings in the BIOS. Make sure all onboard SATA ports are enabled and in AHCI mode, and that no M.2 or SATA Express setting is disabling some of the ports. 4. Try connecting one of the missing drives to a port and cable that currently works. If it shows up there, the problem is the port or cable it was on. If it still doesn't show, it's the drive or its power. 5. If nothing else helps, flash back to F54a. For the USB stick that Q-Flash doesn't see, make sure it's FAT32 formatted and try a USB 2.0 port. Once all disks are back, post new diagnostics before starting the array. We can then look at the slow boot separately.

Possibly and likely, please post the current diagnostics.

You can also boot from an Unraid flash drive, then import the boot pool: zpool import -R /x flash zfs mount flash/boot The internal boot pool should then be mounted under /x/boot