JorgeB

The data share exists on two pools, cache and primus, and /mnt/user/data/media shows both of them merged. You checked the primus side, but not the cache side. Please post the output of: ls -la /mnt/cache/data/media ls -la /mnt/cache/data/media/anime zfs list -o name,mountpoint,mounted -r cache/data primus/data Look in particular for a name that exists as a file on one pool and as a folder on the other, or for any error when listing the cache side. If the cache copy only has some leftover files or folders, move them to the same place on primus and the user share should list normally again. Two other things from the diagnostics, not related to this problem: - During the last scrubs, the four primus disks connected to the onboard SATA ports had repeated link resets, all at the same time. SMART for all disks looks fine, so check the power cables and any splitters feeding those disks. That is likely why the scrubs keep finding a checksum error. Once that is fixed, run zpool clear primus and another scrub, and confirm that it finishes with 0 errors. - The docker pool reports 52 data errors, and it has no redundancy. Run zpool status -v docker to see the affected files. If they are only inside the Docker data, it is usually easiest to recreate it. A memtest would also be a good idea, since that pool and primus both have errors.

Yes, rsync is still running. Abort Script only stopped the script itself, not the rsync it started, and that's what is keeping the array from stopping. Sorry, I should have included this in my last post. Open a terminal (SSH or the web terminal) and type: pkill -x rsync Then check that it's gone. This command should show nothing: pgrep -a rsync The array should stop within a few seconds after that. Then continue with step 4 and the rest of the plan from my last post, nothing changes. If the array is still stopping a couple of minutes after rsync is gone, type this and post the output with new diagnostics: fuser -vm /mnt/user Don't force a reboot. Unrelated, but worth fixing once this is done: your server can't send email notifications, because Gmail has been rejecting its login since Nov 4. You'll want those alerts working, so check the SMTP settings in Settings > Notifications. Gmail normally needs an app password there.

That's not recursion, and nothing is wrong. Your pool only has 280 GB used, so 686G can't really be there. Your Docker data is about 125 GB. With a Docker folder on Btrfs, the image layers share the same data, but du counts it again for each layer. rsync does the same and copies each one in full, which is why the backup is taking so long. This has no effect on the pool steps, since they work with the real data, and it will easily fit on the 870 EVOs. So yes, abort the script. The array stop should then finish, and images and containers don't need a backup, because you can recreate them from Previous Apps. However, since rsync was stuck in docker, it may not have reached the folders after it, and any Docker named volumes are stored in that folder, not in appdata. Your compose stacks may use them, for example the superset database. 1. Use Abort Script in User Scripts and let the array stop. 2. Finish step 4. When the array is started and you've confirmed your data is there, keep Docker and VMs disabled and run this: rsync -a /mnt/cache/domains /mnt/cache/photos /mnt/cache/system /mnt/cache/tm-media /mnt/cache/docker/volumes /mnt/user/backup_arrayonly/ This creates a folder for each one inside backup_arrayonly. Use that share because it's array only, so the copy doesn't end up on the ADATA. 3. When that finishes, continue with step 5. You can delete the partial docker copy from the backup share later to get the space back, but only from the backup share. Don't delete anything from /mnt/cache/docker. Before the next backup run, exclude the docker folder from the script, but keep docker/volumes if it has anything you need. Once the pool is on the 870 EVOs and everything works, you may want to consider moving Docker to a vdisk (docker.img). It's a single file, so du shows its real size and this problem goes away. You'd need to recreate the containers, so that's optional and can wait. If the array still doesn't stop a few minutes after aborting the script, or if anything gives an error, stop there and post diagnostics. Don't force a reboot.

Good news, the balance finished without errors and the missing device is gone. The new diagnostics show the pool as a single device on the ADATA, as expected, so that part is done. The array stop is slow because it can't unmount /mnt/user, and that looks like your backup_cache user script. It started last night and its rsync was still running when you took the diagnostics. The stop will finish by itself once rsync ends. Check its log in User Scripts. Since that's your backup of a worn device, it's best to let it complete. If you need to stop it sooner, use Abort Script there, but then the backup will be incomplete. Don't force a reboot. About the SATA ports, the steps don't change, and sorry, I was wrong about 8 ports, your board has 6. The manual note only applies to a SATA M.2 SSD in M2_2. Both of your ADATA drives are NVMe, so SATA3 is not disabled and you have all 6 ports. It doesn't matter which M.2 slot the ADATA is in. You're using 5 now, the HFS500 and 4 array disks, so one is free. The first 870 EVO can use the HFS500's port and the second one the free port, with the ADATA still installed. So continue with step 4 once the array stops: - Step 5: when you shut down, replace the HFS500 with the first 870 EVO and connect the second 870 EVO to the free port. Check that both are detected. - Step 6: add the first 870 EVO and wait for the conversion to RAID1 to finish. - Step 7: assign the second 870 EVO to the ADATA's slot and wait for the replace to finish. Then shut down and remove the ADATA. Keep Docker and VMs disabled until this is all done. If the second 870 EVO isn't detected, or anything gives an error, stop there and post diagnostics. Don't format anything.

Thanks, but that output does not show what I expected, so I need to correct my last post. The block we were looking for is no longer at that address. The scrub found a checksum tree block there with a zeroed ID. Your output shows a different block, from the extent tree, with the correct ID and a generation newer than today's mount. Btrfs does not change a block in place. It writes a new copy somewhere else and frees the old spot, and here another block has since used that spot. I should have expected this with the pool in use. So this output says nothing about the old block, and the generation cannot show its age as I said it would. That means we cannot tell if this block dates from when XMP was on. It does not change the plan, because it still cannot explain the crashes. This is still a good sign. Btrfs had to read that block and verify its checksum before it could replace it. If your cache is still writable and the next diagnostics show no Btrfs errors, the block was valid on the SSD, which means the zeros were already there in RAM when it was written. So the SSD still does not look like the problem, and I do not suggest replacing it. About the files that failed: error 0x80070780 is not a disk read error. - ipc-socket and mongodb-27117.sock are sockets. Programs use them to talk to each other, they hold no data, and the app creates them again when it starts. Nothing is lost. - The qbittorrent log files are not sockets, so I cannot explain those from here, but they are only logs and you can ignore them. - The permission errors happen because Windows cannot read files owned by the containers. A Windows copy also loses the Linux owners and permissions, which apps like MongoDB need. You already have the Appdata Backup plugin installed, so please use that instead. It runs on the server, and it keeps owners and permissions. Keep your Windows copy as an extra. Next steps, in this order: 1. Run Appdata Backup, with the destination on the array, not on the cache. Copy anything else important on the cache to the array too. 2. Run a scrub on the cache again and post the result. If the same 8 errors appear at a different address, that is the same block, moved, because the zeroed ID is copied when Btrfs rewrites a block. Zero errors is also fine. If you get more than 8 errors, or any csum errors, tell us, because that would be new damage. 3. Post new diagnostics before you shut down. These will show if your backups caused any Btrfs errors, and the log is lost on shutdown. 4. Then start the test with two sticks at a time, and use the server normally for 3 or 4 days per pair. Post diagnostics after each run, or as soon as something crashes. 5. When the server is stable, reformat the cache and restore your backup. There is no rush unless the cache goes read-only again. Please still do not run btrfs check --repair.

Thanks, those photos are very useful, and they show this was not an nginx problem. The server lost the ability to read its own OS files. Exit status 135 means a program was killed because part of its file could not be read, and the same problem explains the "Input/output error" on rc.nginx and rc.sshd, and "command not found" for diagnostics. Unraid loads much of the OS on demand from the boot device while it runs, so if that device drops offline or starts returning read errors, anything new fails to start. Things already loaded in RAM, like Samba and your running containers, keep working, which is why SMB and Docker looked fine. The reboot stopped for the same reason, since it could not read the shutdown script, so the hard reset was the right call. The boot device is the main suspect, but there is no log from when it happened, so this is not confirmed yet. Yes, please post new diagnostics now. They won't show the failure itself, because the log from that boot was lost, but they will show the boot device and how it is connected, and whether anything is already logging errors. In the meantime: 1. If you plan to update Unraid itself, make a flash backup first (Main, click the boot device, then Flash Backup), since an OS update writes to that device. 2. If the boot device is a USB flash drive, make sure it is plugged directly into a motherboard port, not a hub or front panel header, preferably a USB 2.0 port if the board has one. Next time the server is off, check that drive on another computer for filesystem errors. 3. The hard reset counts as an unclean shutdown, so a parity check may run when the array starts. If it does, let it finish and tell us if it reports errors. 4. So that we can catch this if it happens again, go to Settings, Syslog Server, and send the syslog to another machine using Remote syslog server. If you don't have one, use Local syslog server with a folder on your array or a pool. Please don't rely on Mirror syslog to boot drive, because that device is the one we suspect, and the lines we need would be lost with it. If it happens again, a photo of the console is still useful, and the saved syslog should then show what the boot device did. Also let us know if this has happened before, and roughly how long the server had been up.

Thanks, these diagnostics answer most of this, and the news is better than I said in my last post. The 8 errors are all one 16 KiB block, stored twice, and it belongs to the Btrfs checksum tree. That tree stores the checksums of your data. It does not hold file names or folders. The only fault the log shows is that one 16-byte ID field in the block header is all zeros. Btrfs tests this field only during a scrub, not in normal use. The scrub then read your data, which needs this same tree, without errors. So this block most likely reads normally. That corrects my last post. If this holds, I do not expect I/O errors on folders from this block, or that it will make the pool go read-only when it changes. A pool with this history is still not one to trust long term, so the plan to reformat stays, but there is no rush. Does the SSD need to be replaced? Nothing points to that now. SMART is clean, there are no SATA errors, and the same fault is in both copies, 1 GiB apart on the SSD. A random SSD fault is very unlikely to do that. The command below can settle it. Is it failing RAM? It is still the main suspect, but it is not proven. The memory corruption is clear. The source could still be RAM, the CPU memory controller, the board, power, or a kernel module such as CoreFreq. The stick test will narrow this down. Could old XMP damage explain everything? Partly, but only for this block. It could be from when XMP was on. But it cannot cause the crashes, for two reasons: - On Nov 4, three days after you disabled XMP, Btrfs found a different block damaged in RAM. - Docker itself, PHP and smartctl all crashed, and these run from the Unraid OS, not from the cache. Damage on the cache cannot crash them. So something was still corrupting memory after XMP was off. The good sign is that there are no crashes since you loaded BIOS defaults and CoreFreq was removed, but that is less than 3 hours so far, which is too short to judge. Your server was logging a crash about every 8 hours. About the files that failed to copy: the log shows no Btrfs or disk error during your backup, so this block did not cause those failures. They could be files in use or permission problems. Please stop Docker and the VM service before you run the backup again. Then post the file names and the exact error for any file that still fails. Next recommended steps: 1. Run this now, and post the output. It only reads: btrfs inspect-internal dump-tree -b 588300288 /dev/sdc1 | head -n 5 If it shows the block with chunk uuid 00000000-..., the block checksum is valid. That means the zeros were already in RAM when the block was written, not caused by the SSD. If it shows checksum verify failed, tell us, because then the block cannot be read normally. The output also has a generation number. That shows if the block was last written before you disabled XMP. 2. Finish the backup, as above. 3. Continue the test with two sticks at a time, and use the server normally for 3 or 4 days per pair. Post diagnostics after each run, or as soon as something crashes. 4. When the server is stable, reformat the cache and restore your backup. Please still do not run btrfs check --repair.

That warning is normal, but it should still run.

Thanks for the scrub result. Unfortunately it shows the cache has some damage on the SSD. "verify" errors are metadata errors, which means the Btrfs internal structures are damaged, not your file data directly. Metadata on this pool is stored twice (DUP). "Uncorrectable" means both copies are bad, so scrub had no good copy to repair from. Most likely this is one damaged metadata block, counted once for each 4 KiB part on each copy, but the log will show that. So not all the damage was stopped in RAM. The check at write time catches only some types of damage, and this block got past it at some earlier time. The pool mounts and works now, but files or folders that depend on this block can give I/O errors when you read them. If Btrfs needs to change that block, the pool can go read-only again. Scrub cannot fix it, and please do not run btrfs check --repair. What I suggest: 1. Post new diagnostics now, before you reboot. The scrub wrote the location of each error to the syslog, and that shows which part of the filesystem is affected. 2. Back up everything important on the cache now, while it still mounts. If some files fail to copy with an I/O error, tell us which ones. 3. Continue the RAM test with two sticks at a time, as planned. 4. When you know which sticks are good, reformat the cache and restore your backup. If the cache goes read-only again before that, back up what you can and do it then. For app databases, a backup from before Nov 2 is the safest to restore. The new diagnostics will also confirm that CoreFreq is not loaded anymore.

The VM is clearly booting, since the recorder keeps working, so this looks like a display problem rather than Windows failing to start. The screen you see is from the VM firmware. Normally the Windows loading circle appears right after it on the same screen, so Windows doesn't seem to be using the VNC display at all since the update. Some things to check: 1. Does this VM have a GPU or iGPU passed through, for example for the recorder's hardware encoding? If so, connect a monitor to it. Windows may now be using it as the main display. 2. Try connecting with Remote Desktop to the VM IP address (if it's enabled in Windows). Once in, open Device Manager > Display adapters and check for a warning or error code on the virtual display adapter. If there is one, reinstall the display driver from the latest virtio-win ISO, or use Roll Back Driver if that option is available. 3. If you can't get in with RDP, shut down the VM, copy the current XML somewhere safe, and temporarily change the VNC Video Driver in the VM settings (for example from QXL to Virtio, or the other way round). Then start it again and see if the display comes back. Please also post the diagnostics (Tools > Diagnostics) with the VM running, and the VM XML (Edit VM, then toggle to XML view).

That's a useful finding, and it means the server itself is fine. If the WebGUI is instant by IP address and slow only through Nas.local, the extra time is spent on your computer before the request reaches the server, resolving the name or picking the address to use. The new hardware isn't the cause. Your server is set to IPv4+IPv6, so Nas.local can resolve to an IPv6 address as well as the IPv4 one. Some clients, especially Windows, can be slow to resolve .local names, or try IPv6 first and wait before falling back to IPv4. To see which one it is, from the computer you use for the WebGUI please post the output of: ping Nas.local ping -4 Nas.local Please also tell us which OS and browser you're using, and whether it's the same from another device. In the meantime, using the IP address works well. Your server has a static IP, so it won't change. If you prefer to use the name, you can add an entry to the hosts file on your PC that maps Nas to 192.168.1.222, or add it as a DNS entry on your router if it supports that.

Glad that you caught that line. Unraid has a built-in function for this, and it does not need a Parity check or any data rewrite. Change the encryption key 1. Make sure that you have a backup of your important data. If the new passphrase is lost, the encrypted data cannot be recovered. 2. Save the new passphrase in your password manager. The WebGUI accepts printable ASCII characters only, with a maximum of 512. 3. With the array started, go to Settings > Disk Settings > Change encryption key. 4. Enter the existing key if the page asks for it. If /root/keyfile exists, the page uses that file. 5. Select Passphrase, type the new passphrase twice, and click Apply. Unraid adds the new key to all open encrypted devices, then removes the old key. If the new key cannot be added to a device, Unraid removes it again and the old key stays valid. Only open devices are changed, so an encrypted disk that is not open at this time, for example in Unassigned Devices, keeps the old key. Check the result The success message does not show that only one key remains, so run this command: for d in $(ls /dev/disk/by-id | grep CRYPT-LUKS | awk -F- '{print $NF}'); do echo "$d: $(cryptsetup luksDump /dev/$d | grep -cE '^ +[0-9]+: luks2|ENABLED$')"; done You should see your 8 devices (4 array disks and 4 pool devices), each with the value 1. If a device shows 2 or more, it has another key. In that case, post the output before you change anything else. Then stop the array and start it again with the new passphrase, to make sure that all the devices open. Your keyfile script Your syslog shows a User Script named get-keyfile-before-stopping-array. I cannot see its contents, but update or disable it before you stop the array. If it puts the old key back in /root/keyfile, the array can fail to start with a wrong key error. After the array starts with the new passphrase, the old keyfile cannot unlock these disks. You can then delete all copies of it, the archive in your repository, and any older LUKS header backups, because those backups still accept the old key. Your performance plan cpuset is a good limit for container workloads like Immich. However, it does not limit image downloads and extraction, because those run in the Docker service on the host, not in the container. If updates still stall, this is a possible reason. The move of docker.img is a good test, and it does not need a reinstall. A single device is normally acceptable for docker.img, because you can download the images again and your templates are on the flash drive. The exception is Compose named volumes, which are stored inside docker.img unless you changed that. Run docker volume ls to check. Keep any important data in appdata, not in a named volume. Your NVMe is a Samsung 980 PRO with firmware 3B2QGXA7. This firmware has a known defect that can cause early failure. Your SMART data is clean, with 100% spare and no media errors, but update to 5B2QGXA7 or newer before you use it. The update must be done with Samsung's tools. Then: 1. Create the pool. Use zfs - encrypted if you want docker.img to stay encrypted. This erases anything that is on the NVMe. 2. Go to Settings > Docker, set Enable Docker to No, and click Apply. 3. Copy the image. This example uses a pool named fast: mkdir -p /mnt/fast/docker cp --sparse=always /mnt/cache-a/system/docker/docker-xfs.img /mnt/fast/docker/docker-xfs.img 4. Set Docker vDisk location to /mnt/fast/docker/docker-xfs.img. 5. Set Enable Docker to Yes and click Apply. 6. Make sure that your containers start. Then delete the old image on cache-a. Leave the system share on cache-a, because libvirt.img is also there. Keeping appdata on cache-a for now is fine, as this changes one thing at a time. If the problem happens again, please capture the top and pressure files from my earlier post, plus new diagnostics. These will show if the CPU or the disks are the limit.

If you already know, what is your question? Basically you can use a different flash drive or contact support to see if they can delete the old trial for you to use the same one.

Thanks for confirming. A small correction to my last reply: those fields are dropdowns, so there is nothing to type. Unraid adds the line break itself when two fields are selected, which is why selecting one field per dropdown fixes it. This looks like a bug in the Bark agent, I report it here: https://product.unraid.net/b/unraid-os-prerelease-bugs-feedback/p/7-4-0-rc-2-bark-and-slack-notifications-fail-with-http-400-when-two-fields-are-selected-or-the-text-contains-a-double-quote

Since XMP was already off on November 1, it does not explain the crashes from November 2 onward, so we can rule it out. The new diagnostics confirm the same problem. This time the Docker service itself crashed with "fatal error: invalid runtime symbol table". That error means Docker found that data it loads from its own program file was damaged in memory. This is a typical sign of memory corruption, not a Docker bug. A PHP process on the host also crashed on the 5th. The good news is that the cache mounted normally after the reboot, with no errors. About the RAM voltage: with XMP off, the board uses the standard DDR4 settings, which run at 1.2V. That is correct for the lower speed, so you do not need to set 1.35V manually. Just load the BIOS defaults. Removing CoreFreq does not fully take effect until you reboot, because its kernel module stays loaded until then. It was still loaded when both new crashes happened, so it is not ruled out yet. Next steps: 1. Load the BIOS defaults and reboot. 2. Confirm the cache is writable, then run a scrub on it and post the result. 3. Since you have 4 sticks, start with two of them, in the slots the board manual recommends for two sticks. Use the server normally for 3 or 4 days. Then replace them with the other two and repeat. 4. If one pair crashes, test those two sticks one at a time to find the bad one. 5. Post new diagnostics after each run, or as soon as something crashes again. Starting the RMA in parallel is a good idea. If a pair or a single stick crashes, that also gives you something concrete to report to G.Skill.

The storage side is fine: your appdata is on the cache pool, which has plenty of free space and no filesystem errors. The Plex log shows the actual cause: database disk image is malformed This means the Plex library database file itself is corrupt. The damage was most likely already there before the update, and the new Plex version only found it when it tried to upgrade that part of the database. It isn't caused by Unraid or NVIDIA. The easiest fix is to restore one of Plex's automatic backups: 1. Stop the Plex container. 2. Go to your Plex appdata, then Plex Media Server/Plug-in Support/Databases, and copy the whole Databases folder somewhere safe first. 3. In that folder, delete (or rename) com.plexapp.plugins.library.db and any com.plexapp.plugins.library.db-wal / -shm files next to it. 4. Take the newest backup, named like com.plexapp.plugins.library.db-2026-MM-DD, and rename a copy of it to com.plexapp.plugins.library.db. If there is a com.plexapp.plugins.library.blobs.db backup with the same date, do the same for that file. 5. Start the container and check the log. If the newest backup also shows "malformed", stop the container and try the next older one. Plex's guide for this: https://support.plex.tv/articles/202485658-restore-a-database-backed-up-via-scheduled-tasks/ You will lose only the changes since that backup (watch status, recently added items). Plex adds those again on the next library scan. If none of the backups work, the next step is Plex's repair procedure: https://support.plex.tv/articles/repair-a-corrupted-database/. For help with that, the binhex Plex support thread or the Plex forums are the best place. Not related to this, but you still have the old "Unraid-Nvidia" plugin from 2019 installed next to the current "nvidia-driver" plugin. You can remove the old one from the Plugins page.

That message is not a CPU error, it's a warning the kernel logs once per boot when the Nvidia driver module loads. The proprietary Nvidia driver isn't built with one of the kernel's Spectre mitigation features, so the kernel prints that trace, but the GPU and the server work normally. The CPU number in it is just whichever core was loading the driver at that moment, which is why it changed. You can safely ignore it. You also didn't need to reinstall, Unraid detects the new hardware on boot, and nothing in the diagnostics points to a problem with the upgrade. The diagnostics look clean, but they were taken right after boot. If the server still feels sluggish, please describe what exactly is slow (the WebGUI, file transfers, a specific container, etc.) and when it happens, and post new diagnostics taken while it's happening.

Btrfs checks metadata before writing it. On Nov 4 at 09:39 it found a metadata block that was already damaged in memory, refused to write it, and made the pool read-only to protect it. That is why containers stopped and would not restart. The loop2 errors at that time are only a side effect, because loop2 is libvirt.img on the same cache. The SSD itself looks fine, as SMART is clean and there are no SATA errors. Does the cache need repair? Probably not. The bad block was stopped before it reached the SSD, so I expect the cache to mount normally after a reboot. Please do not run btrfs check --repair. 1. Reboot, because the read-only state does not clear without one, and confirm the cache is writable. 2. Back up appdata and anything else important on the cache. 3. Run a scrub on the pool and post the result. A clean scrub only proves the data matches its checksums. A file damaged in RAM before it was written would still pass, so if an application database misbehaves, restore it from an older backup. RAM is still the main suspect, even with those Memtest passes. In about 2 days, unrelated programs crashed with segfaults, a general protection fault and an invalid opcode, and they ran on different CPU threads. One of them was smartctl, which is not a container. That points to memory being corrupted while the server runs, not to one bad application. The cause could be RAM, the CPU memory controller, the board, power, or something in the kernel, and the logs cannot tell these apart. To answer your question, no number of passes proves it is not the RAM. Memtest does not reproduce a real workload, and I have seen a stick pass 48 hours of Memtest and then fail under real use. A clean Memtest result is useful, but it does not clear the hardware. What I suggest next: 1. Tell us when you disabled XMP, and whether it was already off on Nov 2 to 4. If it was still on, XMP is a good suspect. If it was already off, XMP cannot explain these crashes. 2. Load BIOS defaults, with no XMP and no CPU overclock or undervolt. 3. Uninstall the CoreFreq plugin for this test. It loads a low-level kernel module and the package dates from 2024, so I want it out of the picture. 4. Run the server with only one stick, or half of them if you have four, and use it normally for 3 or 4 days. Your server was logging a crash about every 8 hours, so that is enough time. Then swap to the other stick and repeat. A real workload is a better test here than more Memtest. 5. After each run, post new diagnostics, and tell us your exact RAM kit and how many sticks you have. If only one stick crashes, replace it. If both crash alone, the CPU, board or power supply become the next suspects. If neither crashes, try both together again with CoreFreq still removed. Once the server is stable, I also recommend going back to a btrfs Docker vDisk instead of the directory. The earlier vDisk errors were most likely another symptom of this same problem, not a fault of the vDisk.

That error isn't related to the NVIDIA transcoding. The container is starting and Plex itself is running. It's failing while upgrading its library database: the 503 "running database migrations" page together with N2DB9ExceptionE means the database migration step is failing, which usually happens because the Plex version changed (the container was likely updated at around the same time as Unraid). First, please post the diagnostics zip (Tools > Diagnostics) taken while the problem is happening, so we can confirm that the pool or disk holding appdata has free space and isn't read-only or showing filesystem errors. If the storage looks fine, the problem is most likely the Plex database itself: 1. Look at the last lines of Plex Media Server.log, in the Logs folder inside your Plex appdata. It should show the exact error behind the failed migration. 2. Plex keeps automatic database backups in Plug-in Support/Databases, named like com.plexapp.plugins.library.db-YYYY-MM-DD. With the container stopped, make a copy of the whole Databases folder first, then restore the most recent backup from before the update, following Plex's guide: https://support.plex.tv/articles/202485658-restore-a-database-backed-up-via-scheduled-tasks/ 3. Plex also has a repair guide: https://support.plex.tv/articles/repair-a-corrupted-database/ For questions about the container itself, the binhex support thread (or the Plex forums) is the best place, since they know the app much better.

Please post the diagnostics

Yes, that's the right idea, and you do start with the balance again. Sorry, the "few minutes" I gave you was too low. With this much data it can take considerebly more time, and it shows nothing while running, so it was likely working when you cancelled it. Nothing was lost, and it will skip whatever it already converted. One thing to check first: the ADATA is an NVMe device, so removing it won't free a SATA port. You'll need a port for each 870 EVO while the ADATA is still installed. Your October diagnostics show only 5 of the 8 onboard SATA ports in use, so you may already have free ones, unless you've added disks since or the board disables some when M.2 slots are used. If you can't connect both, let me know before you start. 1. Back up anything important from the pool to the array. The ADATA is also worn out, and the balance rewrites its data. 2. With the array running, start the balance in the background: btrfs balance start -f --bg -dconvert=single,soft -mconvert=dup,soft /mnt/cache Check progress with this command, it's done when it reports no balance found: btrfs balance status /mnt/cache 3. Remove the missing device: btrfs device remove missing /mnt/cache 4. Stop the array and re-import the pool with the ADATA only: - on Main, click on the first device for that pool and then "remove pool" - back on Main, create a new pool with the same name and 1 slot - assign the ADATA, leave the filesystem set to auto - start the array and confirm your data is there 5. Stop the array and shut down. Replace the HFS500 with the first 870 EVO, connect the second one too, and boot. 6. With the array stopped, change the pool to 2 slots and assign the first 870 EVO. Start the array. Unraid adds it and converts the pool to RAID1 automatically. Wait until this finishes, it can take a few hours. You can check by clicking on the pool device on Main, the balance status should show only RAID1. 7. Stop the array, assign the second 870 EVO to the ADATA's slot, and start the array. Unraid replaces the ADATA with it. Wait for this to finish, then the ADATA can be removed. The result will be a RAID1 pool on the two 870 EVOs with about 1 TB usable. Post new diagnostics when it's done so we can confirm all is good. If any step gives an error, or the pool shows as unmountable or offers to format, stop there and post diagnostics. Don't format anything.

Your screen shows the server is not stuck on the array disks. Those are all detected. It is stuck waiting for one device, sda, which is not responding: sd 10:0:0:0: [sda] tag#0 timing out command, waited 180s Each of these messages means one command failed after 3 minutes of trying, so boot can stall for a very long time. From its position after the SATA ports, sda is most likely a USB storage device. Since the only change was the script that turns off your USB drive, that drive is my first suspect. A normal reboot may not reset a USB drive, so it can stay unresponsive and block the boot. Please try this: 1. Turn the server off completely. If it does not respond to the keyboard, hold the power button. The array has not started at this point, so this is safe. 2. Unplug the USB drive from the server, and also unplug its power cable if it has one. 3. Turn the server on and see if it comes online. If it still stops on the same message, turn it off again, unplug any other USB storage, including a USB flash drive if you have one for the license, and try again. If you still see sda timing out with no USB storage attached, the problem is an internal device. In that case, please post a new photo of the screen. If it boots, please don't reconnect the USB drive yet. Post the script, tell us how you start it (go file, User Scripts plugin, or something else), and attach the diagnostics from Tools > Diagnostics. Then we can check how it turns the drive off. You don't need your backups for now. Nothing here points to a problem with your boot drive or data. The GPT messages on the screen for sdg and sdh are a separate thing and are not stopping the boot. Please don't try to fix them with Parted as the message suggests.

AFAIK you can't really route one container through another. The server container would need NET_ADMIN and NAT inside its image, and it would break on every update. There are two better ways to end up with one IP. Option 1: share the server's network The machine learning container joins the server container's network stack, so it uses the same IP and all its traffic leaves through br0.60: immich-machine-learning: container_name: immich_machine_learning # keep your image line here, it's missing from what you pasted network_mode: service:immich-server volumes: - model-cache:/cache env_file: - .env environment: - ORT_DISABLE_TELEMETRY=1 - IMMICH_HOST=127.0.0.1 restart: always healthcheck: disable: false - Remove its networks: section, it can't be combined with network_mode. - IMMICH_HOST=127.0.0.1 keeps port 3003 from being open on your VLAN IP. Set it here only, not in .env, because the server container reads the same variable. - The immich-machine-learning name no longer resolves, so go to Administration > Settings > Machine Learning and change the URL to http://localhost:3003. The downside is that this container depends on the server container. If immich_server is restarted or recreated on its own, machine learning loses its network until it's restarted too. docker compose up -d handles this, but restarting only the server from the Docker page doesn't. Also check that machine learning is working after a reboot. Option 2: no VLAN IP for machine learning It only needs the internet to download models that aren't in its cache yet, so it can stay on immich-net only. When it needs a new model: docker network connect br0.60 immich_machine_learning Run the job that needs the model (Smart Search or Face Detection, for example), then disconnect it once the download finishes: docker network disconnect br0.60 immich_machine_learning This is the tighter setup, since the container has no outside access at all. You'd just need to repeat it when you change models, or if an Immich update needs new ones. Until then, those jobs will fail with download errors in the machine learning container's log. I'd go with option 2 if you rarely change models, and option 1 if you'd rather not think about it. For both options: - Remove ports: - 2283:2283 from immich-server. It no longer does anything once immich-net is internal, so use http://192.168.60.xx:2283 and update anything that still points at the Unraid IP. - Recreate immich-net as internal with the stack stopped, as in my earlier post. Your existing br0.60 and immich-net entries under networks: are fine as they are.

Your diagnostics show that the AI share has "Use cache" set to No, so these downloads write directly to the array. At the time of the diagnostics, the array was writing at about 42 MB/s to disk2 and parity. This is a normal speed for parity writes, and it is much slower than a fast download or a 10GbE copy. When data arrives faster than that, the server holds it in RAM and then has to write it out at array speed. The client can see long pauses during that time. I must correct something from my last post. That write used only about 180 MB/s of the x1 link, so the link did not limit it. The disks did. The x1 link is still worth fixing, because everything on the array shares it, but the slot swap may not stop the stalls alone. Also, your diagnostics do not show a stall in progress, so I cannot confirm the cause yet. Please do these tests: 1. Swap the HBA and the GPU as described before, and confirm that the HBA links at x8. 2. Open a terminal on the server and run this command during a transfer: watch -n 5 "grep -E '^(Dirty|Writeback):' /proc/meminfo" 3. When a transfer stalls, look at that output and at the disk activity on the Main page. Then download new diagnostics while the stall continues, and post them with the values you saw. 4. Copy a 100-200 GB file to a share that uses the cache pool. This copy does not use the array until the mover runs, so tell us if it stalls too. 5. Copy a large file from the server to a local SSD on the client while nothing else is writing to the array. Plex must be idle during this test. Writes cannot explain a stall in this test, so a stall here points to a different cause. Also note these two items: - A file cannot span array disks. Only disks 9, 10, 11, and 14 have more than 1.6 TB free, and high-water can put a new file on a disk with less free space than that. A 1.6 TB file on such a disk will fail when the disk is full. Before you download files of that size, limit the share to disks with enough free space with "Included disk(s)" or set a higher minimum free space for the share. - After the HBA is out of the x1 slot, you can try "reconstruct write" for md_write_method in Settings > Disk Settings. It can increase array write speed, but all the disks must spin. Do not use it while the HBA is in the x1 slot, because all 15 disks would share that link. Please also tell us: - The exact error that LM Studio shows when a download fails. - Your internet download speed. - The direction of the rsync job to the second server. If it only reads from this server, the job does no parity writes here, so it does not rule out the array.

No problem about the filtering, I was wrong there. Plex just logs very little at this level. The full log has what we needed, and I think we found the trigger: Plex loudness analysis of one item, ID 43759. The job fails every night, so Plex tries it again in the next maintenance window. These lines show it: - Oct 28: started 02:01, killed by Plex at 05:02 ("job unable to complete in allotted time") - Oct 29: started 02:15, failed at 03:44:29, one second after the kernel killed the 14.3GB Plex Transcoder - Oct 30: started 02:15, killed by Plex at 05:15:18, and the stall ended at 05:15:27 So each stall ends when this job ends, either by the out of memory kill or by Plex's own 3 hour limit. Your Oct 28 reset was about 2 h 43 min after the job started again at 05:15, so the server may have recovered by itself if you had waited. That job's item is not in the log, so this one is likely, not certain. This also looks like the problem other users reported on the Plex forum, where loudness analysis uses a lot of RAM: https://forums.plex.tv/t/loudness-analysis-grows-to-10-gb-ram-and-triggers-host-oom-killer/943677/24 What I suggest: 1. Turn off loudness analysis in Plex, under Settings > Library, so the job stops running every night. You can turn it on again after a Plex update, or after you fix the file. 2. Find the file. The log only shows the ID. Open the Plex container console and run this (the path can differ on some Plex images): "/usr/lib/plexmediaserver/Plex SQLite" -readonly "/config/Library/Application Support/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db" "select mp.file from media_parts mp join media_items mi on mp.media_item_id=mi.id where mi.id=43759 or mi.metadata_item_id=43759;" It can return two files, because the ID may match in two ways. If so, check both. See if the file plays to the end, and replace it if it does not. 3. Keep the memory limits you set. That log ends on Oct 30, so it does not show them working yet. Has the server stalled since you added the limits on Nov 1? Please also check for new "job for item 43759 failed" lines since then. With the 8GB limit, the job should now be killed much sooner, without stalling the whole server. If it happens again, post new diagnostics. The other errors in the log (metadata matching, UltraBlur, credit detection, remote plex.direct connections, NAT-PMP) are not related to the stalls. The busy database errors appear only during this job, apart from a few during a short database optimization on Oct 30. About cache prefer: your reason makes sense, and the exclusive share is optional. It should help Plex database speed, but it does not fix the stalls. If you do make it exclusive, appdata can no longer overflow to the array, so set a Minimum free space on the cache pool. New files from your other cached shares will then go to the array before the pool fills up.