JorgeB

The data share exists on two pools, cache and primus, and /mnt/user/data/media shows both of them merged. You checked the primus side, but not the cache side. Please post the output of: ls -la /mnt/cache/data/media ls -la /mnt/cache/data/media/anime zfs list -o name,mountpoint,mounted -r cache/data primus/data Look in particular for a name that exists as a file on one pool and as a folder on the other, or for any error when listing the cache side. If the cache copy only has some leftover files or folders, move them to the same place on primus and the user share should list normally again. Two other things from the diagnostics, not related to this problem: - During the last scrubs, the four primus disks connected to the onboard SATA ports had repeated link resets, all at the same time. SMART for all disks looks fine, so check the power cables and any splitters feeding those disks. That is likely why the scrubs keep finding a checksum error. Once that is fixed, run zpool clear primus and another scrub, and confirm that it finishes with 0 errors. - The docker pool reports 52 data errors, and it has no redundancy. Run zpool status -v docker to see the affected files. If they are only inside the Docker data, it is usually easiest to recreate it. A memtest would also be a good idea, since that pool and primus both have errors.

Yes, rsync is still running. Abort Script only stopped the script itself, not the rsync it started, and that's what is keeping the array from stopping. Sorry, I should have included this in my last post. Open a terminal (SSH or the web terminal) and type: pkill -x rsync Then check that it's gone. This command should show nothing: pgrep -a rsync The array should stop within a few seconds after that. Then continue with step 4 and the rest of the plan from my last post, nothing changes. If the array is still stopping a couple of minutes after rsync is gone, type this and post the output with new diagnostics: fuser -vm /mnt/user Don't force a reboot. Unrelated, but worth fixing once this is done: your server can't send email notifications, because Gmail has been rejecting its login since Nov 4. You'll want those alerts working, so check the SMTP settings in Settings > Notifications. Gmail normally needs an app password there.

That's not recursion, and nothing is wrong. Your pool only has 280 GB used, so 686G can't really be there. Your Docker data is about 125 GB. With a Docker folder on Btrfs, the image layers share the same data, but du counts it again for each layer. rsync does the same and copies each one in full, which is why the backup is taking so long. This has no effect on the pool steps, since they work with the real data, and it will easily fit on the 870 EVOs. So yes, abort the script. The array stop should then finish, and images and containers don't need a backup, because you can recreate them from Previous Apps. However, since rsync was stuck in docker, it may not have reached the folders after it, and any Docker named volumes are stored in that folder, not in appdata. Your compose stacks may use them, for example the superset database. 1. Use Abort Script in User Scripts and let the array stop. 2. Finish step 4. When the array is started and you've confirmed your data is there, keep Docker and VMs disabled and run this: rsync -a /mnt/cache/domains /mnt/cache/photos /mnt/cache/system /mnt/cache/tm-media /mnt/cache/docker/volumes /mnt/user/backup_arrayonly/ This creates a folder for each one inside backup_arrayonly. Use that share because it's array only, so the copy doesn't end up on the ADATA. 3. When that finishes, continue with step 5. You can delete the partial docker copy from the backup share later to get the space back, but only from the backup share. Don't delete anything from /mnt/cache/docker. Before the next backup run, exclude the docker folder from the script, but keep docker/volumes if it has anything you need. Once the pool is on the 870 EVOs and everything works, you may want to consider moving Docker to a vdisk (docker.img). It's a single file, so du shows its real size and this problem goes away. You'd need to recreate the containers, so that's optional and can wait. If the array still doesn't stop a few minutes after aborting the script, or if anything gives an error, stop there and post diagnostics. Don't force a reboot.

Good news, the balance finished without errors and the missing device is gone. The new diagnostics show the pool as a single device on the ADATA, as expected, so that part is done. The array stop is slow because it can't unmount /mnt/user, and that looks like your backup_cache user script. It started last night and its rsync was still running when you took the diagnostics. The stop will finish by itself once rsync ends. Check its log in User Scripts. Since that's your backup of a worn device, it's best to let it complete. If you need to stop it sooner, use Abort Script there, but then the backup will be incomplete. Don't force a reboot. About the SATA ports, the steps don't change, and sorry, I was wrong about 8 ports, your board has 6. The manual note only applies to a SATA M.2 SSD in M2_2. Both of your ADATA drives are NVMe, so SATA3 is not disabled and you have all 6 ports. It doesn't matter which M.2 slot the ADATA is in. You're using 5 now, the HFS500 and 4 array disks, so one is free. The first 870 EVO can use the HFS500's port and the second one the free port, with the ADATA still installed. So continue with step 4 once the array stops: - Step 5: when you shut down, replace the HFS500 with the first 870 EVO and connect the second 870 EVO to the free port. Check that both are detected. - Step 6: add the first 870 EVO and wait for the conversion to RAID1 to finish. - Step 7: assign the second 870 EVO to the ADATA's slot and wait for the replace to finish. Then shut down and remove the ADATA. Keep Docker and VMs disabled until this is all done. If the second 870 EVO isn't detected, or anything gives an error, stop there and post diagnostics. Don't format anything.

Thanks, but that output does not show what I expected, so I need to correct my last post. The block we were looking for is no longer at that address. The scrub found a checksum tree block there with a zeroed ID. Your output shows a different block, from the extent tree, with the correct ID and a generation newer than today's mount. Btrfs does not change a block in place. It writes a new copy somewhere else and frees the old spot, and here another block has since used that spot. I should have expected this with the pool in use. So this output says nothing about the old block, and the generation cannot show its age as I said it would. That means we cannot tell if this block dates from when XMP was on. It does not change the plan, because it still cannot explain the crashes. This is still a good sign. Btrfs had to read that block and verify its checksum before it could replace it. If your cache is still writable and the next diagnostics show no Btrfs errors, the block was valid on the SSD, which means the zeros were already there in RAM when it was written. So the SSD still does not look like the problem, and I do not suggest replacing it. About the files that failed: error 0x80070780 is not a disk read error. - ipc-socket and mongodb-27117.sock are sockets. Programs use them to talk to each other, they hold no data, and the app creates them again when it starts. Nothing is lost. - The qbittorrent log files are not sockets, so I cannot explain those from here, but they are only logs and you can ignore them. - The permission errors happen because Windows cannot read files owned by the containers. A Windows copy also loses the Linux owners and permissions, which apps like MongoDB need. You already have the Appdata Backup plugin installed, so please use that instead. It runs on the server, and it keeps owners and permissions. Keep your Windows copy as an extra. Next steps, in this order: 1. Run Appdata Backup, with the destination on the array, not on the cache. Copy anything else important on the cache to the array too. 2. Run a scrub on the cache again and post the result. If the same 8 errors appear at a different address, that is the same block, moved, because the zeroed ID is copied when Btrfs rewrites a block. Zero errors is also fine. If you get more than 8 errors, or any csum errors, tell us, because that would be new damage. 3. Post new diagnostics before you shut down. These will show if your backups caused any Btrfs errors, and the log is lost on shutdown. 4. Then start the test with two sticks at a time, and use the server normally for 3 or 4 days per pair. Post diagnostics after each run, or as soon as something crashes. 5. When the server is stable, reformat the cache and restore your backup. There is no rush unless the cache goes read-only again. Please still do not run btrfs check --repair.

Thanks, those photos are very useful, and they show this was not an nginx problem. The server lost the ability to read its own OS files. Exit status 135 means a program was killed because part of its file could not be read, and the same problem explains the "Input/output error" on rc.nginx and rc.sshd, and "command not found" for diagnostics. Unraid loads much of the OS on demand from the boot device while it runs, so if that device drops offline or starts returning read errors, anything new fails to start. Things already loaded in RAM, like Samba and your running containers, keep working, which is why SMB and Docker looked fine. The reboot stopped for the same reason, since it could not read the shutdown script, so the hard reset was the right call. The boot device is the main suspect, but there is no log from when it happened, so this is not confirmed yet. Yes, please post new diagnostics now. They won't show the failure itself, because the log from that boot was lost, but they will show the boot device and how it is connected, and whether anything is already logging errors. In the meantime: 1. If you plan to update Unraid itself, make a flash backup first (Main, click the boot device, then Flash Backup), since an OS update writes to that device. 2. If the boot device is a USB flash drive, make sure it is plugged directly into a motherboard port, not a hub or front panel header, preferably a USB 2.0 port if the board has one. Next time the server is off, check that drive on another computer for filesystem errors. 3. The hard reset counts as an unclean shutdown, so a parity check may run when the array starts. If it does, let it finish and tell us if it reports errors. 4. So that we can catch this if it happens again, go to Settings, Syslog Server, and send the syslog to another machine using Remote syslog server. If you don't have one, use Local syslog server with a folder on your array or a pool. Please don't rely on Mirror syslog to boot drive, because that device is the one we suspect, and the lines we need would be lost with it. If it happens again, a photo of the console is still useful, and the saved syslog should then show what the boot device did. Also let us know if this has happened before, and roughly how long the server had been up.

Thanks, these diagnostics answer most of this, and the news is better than I said in my last post. The 8 errors are all one 16 KiB block, stored twice, and it belongs to the Btrfs checksum tree. That tree stores the checksums of your data. It does not hold file names or folders. The only fault the log shows is that one 16-byte ID field in the block header is all zeros. Btrfs tests this field only during a scrub, not in normal use. The scrub then read your data, which needs this same tree, without errors. So this block most likely reads normally. That corrects my last post. If this holds, I do not expect I/O errors on folders from this block, or that it will make the pool go read-only when it changes. A pool with this history is still not one to trust long term, so the plan to reformat stays, but there is no rush. Does the SSD need to be replaced? Nothing points to that now. SMART is clean, there are no SATA errors, and the same fault is in both copies, 1 GiB apart on the SSD. A random SSD fault is very unlikely to do that. The command below can settle it. Is it failing RAM? It is still the main suspect, but it is not proven. The memory corruption is clear. The source could still be RAM, the CPU memory controller, the board, power, or a kernel module such as CoreFreq. The stick test will narrow this down. Could old XMP damage explain everything? Partly, but only for this block. It could be from when XMP was on. But it cannot cause the crashes, for two reasons: - On Nov 4, three days after you disabled XMP, Btrfs found a different block damaged in RAM. - Docker itself, PHP and smartctl all crashed, and these run from the Unraid OS, not from the cache. Damage on the cache cannot crash them. So something was still corrupting memory after XMP was off. The good sign is that there are no crashes since you loaded BIOS defaults and CoreFreq was removed, but that is less than 3 hours so far, which is too short to judge. Your server was logging a crash about every 8 hours. About the files that failed to copy: the log shows no Btrfs or disk error during your backup, so this block did not cause those failures. They could be files in use or permission problems. Please stop Docker and the VM service before you run the backup again. Then post the file names and the exact error for any file that still fails. Next recommended steps: 1. Run this now, and post the output. It only reads: btrfs inspect-internal dump-tree -b 588300288 /dev/sdc1 | head -n 5 If it shows the block with chunk uuid 00000000-..., the block checksum is valid. That means the zeros were already in RAM when the block was written, not caused by the SSD. If it shows checksum verify failed, tell us, because then the block cannot be read normally. The output also has a generation number. That shows if the block was last written before you disabled XMP. 2. Finish the backup, as above. 3. Continue the test with two sticks at a time, and use the server normally for 3 or 4 days per pair. Post diagnostics after each run, or as soon as something crashes. 4. When the server is stable, reformat the cache and restore your backup. Please still do not run btrfs check --repair.

That warning is normal, but it should still run.

Thanks for the scrub result. Unfortunately it shows the cache has some damage on the SSD. "verify" errors are metadata errors, which means the Btrfs internal structures are damaged, not your file data directly. Metadata on this pool is stored twice (DUP). "Uncorrectable" means both copies are bad, so scrub had no good copy to repair from. Most likely this is one damaged metadata block, counted once for each 4 KiB part on each copy, but the log will show that. So not all the damage was stopped in RAM. The check at write time catches only some types of damage, and this block got past it at some earlier time. The pool mounts and works now, but files or folders that depend on this block can give I/O errors when you read them. If Btrfs needs to change that block, the pool can go read-only again. Scrub cannot fix it, and please do not run btrfs check --repair. What I suggest: 1. Post new diagnostics now, before you reboot. The scrub wrote the location of each error to the syslog, and that shows which part of the filesystem is affected. 2. Back up everything important on the cache now, while it still mounts. If some files fail to copy with an I/O error, tell us which ones. 3. Continue the RAM test with two sticks at a time, as planned. 4. When you know which sticks are good, reformat the cache and restore your backup. If the cache goes read-only again before that, back up what you can and do it then. For app databases, a backup from before Nov 2 is the safest to restore. The new diagnostics will also confirm that CoreFreq is not loaded anymore.

The VM is clearly booting, since the recorder keeps working, so this looks like a display problem rather than Windows failing to start. The screen you see is from the VM firmware. Normally the Windows loading circle appears right after it on the same screen, so Windows doesn't seem to be using the VNC display at all since the update. Some things to check: 1. Does this VM have a GPU or iGPU passed through, for example for the recorder's hardware encoding? If so, connect a monitor to it. Windows may now be using it as the main display. 2. Try connecting with Remote Desktop to the VM IP address (if it's enabled in Windows). Once in, open Device Manager > Display adapters and check for a warning or error code on the virtual display adapter. If there is one, reinstall the display driver from the latest virtio-win ISO, or use Roll Back Driver if that option is available. 3. If you can't get in with RDP, shut down the VM, copy the current XML somewhere safe, and temporarily change the VNC Video Driver in the VM settings (for example from QXL to Virtio, or the other way round). Then start it again and see if the display comes back. Please also post the diagnostics (Tools > Diagnostics) with the VM running, and the VM XML (Edit VM, then toggle to XML view).

That's a useful finding, and it means the server itself is fine. If the WebGUI is instant by IP address and slow only through Nas.local, the extra time is spent on your computer before the request reaches the server, resolving the name or picking the address to use. The new hardware isn't the cause. Your server is set to IPv4+IPv6, so Nas.local can resolve to an IPv6 address as well as the IPv4 one. Some clients, especially Windows, can be slow to resolve .local names, or try IPv6 first and wait before falling back to IPv4. To see which one it is, from the computer you use for the WebGUI please post the output of: ping Nas.local ping -4 Nas.local Please also tell us which OS and browser you're using, and whether it's the same from another device. In the meantime, using the IP address works well. Your server has a static IP, so it won't change. If you prefer to use the name, you can add an entry to the hosts file on your PC that maps Nas to 192.168.1.222, or add it as a DNS entry on your router if it supports that.

Glad that you caught that line. Unraid has a built-in function for this, and it does not need a Parity check or any data rewrite. Change the encryption key 1. Make sure that you have a backup of your important data. If the new passphrase is lost, the encrypted data cannot be recovered. 2. Save the new passphrase in your password manager. The WebGUI accepts printable ASCII characters only, with a maximum of 512. 3. With the array started, go to Settings > Disk Settings > Change encryption key. 4. Enter the existing key if the page asks for it. If /root/keyfile exists, the page uses that file. 5. Select Passphrase, type the new passphrase twice, and click Apply. Unraid adds the new key to all open encrypted devices, then removes the old key. If the new key cannot be added to a device, Unraid removes it again and the old key stays valid. Only open devices are changed, so an encrypted disk that is not open at this time, for example in Unassigned Devices, keeps the old key. Check the result The success message does not show that only one key remains, so run this command: for d in $(ls /dev/disk/by-id | grep CRYPT-LUKS | awk -F- '{print $NF}'); do echo "$d: $(cryptsetup luksDump /dev/$d | grep -cE '^ +[0-9]+: luks2|ENABLED$')"; done You should see your 8 devices (4 array disks and 4 pool devices), each with the value 1. If a device shows 2 or more, it has another key. In that case, post the output before you change anything else. Then stop the array and start it again with the new passphrase, to make sure that all the devices open. Your keyfile script Your syslog shows a User Script named get-keyfile-before-stopping-array. I cannot see its contents, but update or disable it before you stop the array. If it puts the old key back in /root/keyfile, the array can fail to start with a wrong key error. After the array starts with the new passphrase, the old keyfile cannot unlock these disks. You can then delete all copies of it, the archive in your repository, and any older LUKS header backups, because those backups still accept the old key. Your performance plan cpuset is a good limit for container workloads like Immich. However, it does not limit image downloads and extraction, because those run in the Docker service on the host, not in the container. If updates still stall, this is a possible reason. The move of docker.img is a good test, and it does not need a reinstall. A single device is normally acceptable for docker.img, because you can download the images again and your templates are on the flash drive. The exception is Compose named volumes, which are stored inside docker.img unless you changed that. Run docker volume ls to check. Keep any important data in appdata, not in a named volume. Your NVMe is a Samsung 980 PRO with firmware 3B2QGXA7. This firmware has a known defect that can cause early failure. Your SMART data is clean, with 100% spare and no media errors, but update to 5B2QGXA7 or newer before you use it. The update must be done with Samsung's tools. Then: 1. Create the pool. Use zfs - encrypted if you want docker.img to stay encrypted. This erases anything that is on the NVMe. 2. Go to Settings > Docker, set Enable Docker to No, and click Apply. 3. Copy the image. This example uses a pool named fast: mkdir -p /mnt/fast/docker cp --sparse=always /mnt/cache-a/system/docker/docker-xfs.img /mnt/fast/docker/docker-xfs.img 4. Set Docker vDisk location to /mnt/fast/docker/docker-xfs.img. 5. Set Enable Docker to Yes and click Apply. 6. Make sure that your containers start. Then delete the old image on cache-a. Leave the system share on cache-a, because libvirt.img is also there. Keeping appdata on cache-a for now is fine, as this changes one thing at a time. If the problem happens again, please capture the top and pressure files from my earlier post, plus new diagnostics. These will show if the CPU or the disks are the limit.

If you already know, what is your question? Basically you can use a different flash drive or contact support to see if they can delete the old trial for you to use the same one.

Thanks for confirming. A small correction to my last reply: those fields are dropdowns, so there is nothing to type. Unraid adds the line break itself when two fields are selected, which is why selecting one field per dropdown fixes it. This looks like a bug in the Bark agent, I report it here: https://product.unraid.net/b/unraid-os-prerelease-bugs-feedback/p/7-4-0-rc-2-bark-and-slack-notifications-fail-with-http-400-when-two-fields-are-selected-or-the-text-contains-a-double-quote

Since XMP was already off on November 1, it does not explain the crashes from November 2 onward, so we can rule it out. The new diagnostics confirm the same problem. This time the Docker service itself crashed with "fatal error: invalid runtime symbol table". That error means Docker found that data it loads from its own program file was damaged in memory. This is a typical sign of memory corruption, not a Docker bug. A PHP process on the host also crashed on the 5th. The good news is that the cache mounted normally after the reboot, with no errors. About the RAM voltage: with XMP off, the board uses the standard DDR4 settings, which run at 1.2V. That is correct for the lower speed, so you do not need to set 1.35V manually. Just load the BIOS defaults. Removing CoreFreq does not fully take effect until you reboot, because its kernel module stays loaded until then. It was still loaded when both new crashes happened, so it is not ruled out yet. Next steps: 1. Load the BIOS defaults and reboot. 2. Confirm the cache is writable, then run a scrub on it and post the result. 3. Since you have 4 sticks, start with two of them, in the slots the board manual recommends for two sticks. Use the server normally for 3 or 4 days. Then replace them with the other two and repeat. 4. If one pair crashes, test those two sticks one at a time to find the bad one. 5. Post new diagnostics after each run, or as soon as something crashes again. Starting the RMA in parallel is a good idea. If a pair or a single stick crashes, that also gives you something concrete to report to G.Skill.