ijustwantunraidtowork

Latest run was 7 hours and 50 minutes. I turned off the remote sys logs but I will turn them back on and set it up properly. I confirmed that the Minecraft container retained its direct pool mappings. It seems like 30 minutes after my last reply the server died. The last lock up was 50 minutes and this one was almost 8 hours- I will need to do another run of Chunky to see if my cache pool changes actually changed something. Did it not change anything and after the big changes I made before the 5th lock up, is the server not locking up at a 3 hour interval anymore and is it just "random"? Did my changes to cache pool actually do something and now it takes longer to lock up? Was this lock up for the same reason, was there any evidence this time left behind? Not sure. Will do some more investigating when the server and KVM is back up. If I don't resolve this issue by the end of tonight, I will reset my Minecraft to remove the additional mods. I had a deadline to have an up and running server by today but pushed it to next Sunday, I cannot afford to wait longer. Edit (10:02pm EST): Chunky has been running for over 4 hours now, I am thinking that something related to the SSD's is causing these lock ups. I am going to keep investigating... I think if I get Chunky consistently running above 7 hours then the problem is manageable and I will keep investigating. I also attached the diagnostics right after the latest lock up. tower-diagnostics-20260920-1812.zip

It seems like the cache pool mappings was the issue- Chunky has been running for 8 hours now. I changed both Pelican Panel and Pelican Wings from using /mnt/user/appdata/... host mappings to the equivalent /mnt/cache/appdata/... mappings, and updated Wings’ config.yml so its root/data/log/archive/backup paths also use /mnt/cache. The files themselves did not move- they were already physically on the cache pool; this only changed how the containers reach them, bypassing Unraid’s shfs/FUSE user-share layer. A visual of the before and after mappings is below.. Before: Minecraft ↓ /home/container ↓ Docker bind mount /mnt/user/appdata/pelican-wing/data/... ↓ Unraid shfs/FUSE user-share layer ↓ /mnt/cache/appdata/pelican-wing/data/... ↓ Btrfs filesystem ↓ Samsung 990 Pro + Crucial P5 cache pool (RAID1) Now: Minecraft ↓ /home/container ↓ Docker bind mount /mnt/cache/appdata/pelican-wing/data/... ↓ Btrfs filesystem ↓ Samsung 990 Pro + Crucial P5 cache pool (RAID1) If Chunky makes it to about 8 more hours when I wake up, I'll mark this reply as the solution. Can't believe it took 6 days to find the solution to this..

On a different note, I think either the update to Unraid/Docker might've sped up the inevitable (the crash) or my changes to the cache pool caused the crash to be quicker. 4 out of 5 crashes were always between 3 - 3.5 hours and I don't think it is a coincidence that this latest crash barely made it to 50 minutes right after my changes. When I changed the boot parameters before either crash 2 or 3 and ran Chunky again, it made no changes to how long a crash would take- I doubt that removing the boot parameters did anything. I might be grasping at straws here but my only indicator is the cache pool system, so I am going to chase that lead (or shadow..). For context when I changed the boot from USB to internal boot I kinda messed something up so my other 2 SSD's appeared as internal boot drives as well. Apparently this might've been a glitch in Unraid 7.3.1, so I made some changes- after switching the 256 GB Samsung NVMe to Unraid Internal Boot, I removed it from the cache pool’s stale Btrfs membership with btrfs device remove 2 /mnt/cache, updated cache.cfg from a 3-device to a 2-device pool, and left the Samsung 990 Pro + Crucial P5 as the RAID1 cache pool. i also had to re-enable the cache pool’s user share assignment, because it had ended up disabled and /mnt/user/... paths stopped working. Some commands I am running now while investigating.. root@Tower:~# docker inspect 8ce6a07d-b752-4dd2-8c92-5435f1dfb537 \ --format '{{range .Mounts}}{{println .Source "->" .Destination}}{{end}}' /tmp/pelican/passwd -> /etc/passwd /mnt/user/appdata/pelican-wing/data/8ce6a07d-b752-4dd2-8c92-5435f1dfb537 -> /home/container /tmp/pelican/group -> /etc/group /tmp/pelican/machine-id/8ce6a07d-b752-4dd2-8c92-5435f1dfb537 -> /etc/machine-id root@Tower:~# findmnt -T /mnt/cache findmnt -T /mnt/user TARGET SOURCE FSTYPE OPTIONS /mnt/cache /dev/nvme1n1p1 btrfs rw,noatime,ssd,discard=async,space_cache=v2,subvolid=5,subvol=/ TARGET SOURCE FSTYPE OPTIONS /mnt/user shfs fuse.shfs rw,nosuid,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other root@Tower:~#

Hey. I got a new set of diagnostics for you to look at if you need it. I made some changes and ran some tests since I last replied, then ran the Minecraft server with Chunky. It only lasted 48 minutes and 12 seconds this time :( but the KVM was reachable. These diagnostics were immediately after the crash and the Chunky run was after all the changes listed below. Here are the list of changes I've made to Unraid: Updated Unraid from 7.3.1 → 7.3.2 Docker changed from 29.5.1 → 29.5.3 Migrated Unraid boot from the failing SanDisk USB to the 256 GB Samsung NVMe using Unraid Internal Boot. Removed the accidentally/stale third BTRFS cache-pool member created during the Internal Boot migration and restored the cache pool to the Samsung 990 Pro + Crucial P5 RAID1. Removed the boot parameters nvme_core.default_ps_max_latency_us=0 pcie_aspm=off Confirmed Magic SysRq is enabled (kernel.sysrq=1); tried executing that command via the KVM and terminal did not respond during crash. Tests performed on RAM and CPU: Memtest86+ 6 complete passes. 0 errors. Roughly 13+ hours. 32-worker stress-ng CPU test All 32 logical CPUs stressed. Reached ~94–100°C/TjMax, so test was stopped after 10m46s. 32/32 workers passed, 0 failed. No MCE/hardware errors afterward. 7-worker P-core stress-ng CPU test Restricted to Minecraft's CPU 1–15 pool. Full-speed CPU stress for 1 hour. Temperatures mostly ~78–85°C. 7/7 passed, 0 failed. 8-worker physical P-core stress-ng CPU test Allowed CPUs 1,2,4,6,8,10,12,14, giving access to one logical thread from each physical P-core. 1 hour. 8/8 passed, 0 failed. CPU cache/coherency stress-ng test --cache 4 --cacheline 4 --verify. 1 hour. 8/8 stressors passed, 0 failed. I think you asked for container mappings, so they are below: ===== 8ce6a07d-b752-4dd2-8c92-5435f1dfb537 ===== /tmp/pelican/group -> /etc/group /tmp/pelican/machine-id/8ce6a07d-b752-4dd2-8c92-5435f1dfb537 -> /etc/machine-id /tmp/pelican/passwd -> /etc/passwd /mnt/user/appdata/pelican-wing/data/8ce6a07d-b752-4dd2-8c92-5435f1dfb537 -> /home/container ===== immich ===== /mnt/cache/appdata/immich -> /config /mnt/user/Immich-Imports -> /libraries /usr/local/share/docker/tailscale_container_hook -> /opt/unraid/tailscale /mnt/user/Immich -> /photos ===== Mainfrezzer-Pelican-Panel ===== /mnt/user/appdata/pelican-panel/pelican-log -> /var/www/html/storage/logs /usr/local/share/docker/tailscale_container_hook -> /opt/unraid/tailscale /mnt/user/appdata/pelican-panel/pelican-data -> /pelican-data /mnt/user/appdata/pelican-panel/pelican-plugins -> /var/www/html/plugins ===== Mainfrezzer-Pelican-Wing ===== /mnt/user/appdata/pelican-wing/etc/pelican -> /etc/pelican /mnt/user/appdata/pelican-wing/data -> /mnt/user/appdata/pelican-wing/data /tmp/pelican -> /tmp/pelican /var/run/docker.sock -> /var/run/docker.sock ===== netdata ===== /mnt/user/appdata/netdata/cache -> /var/cache/netdata /mnt/user/appdata/netdata/lib -> /var/lib/netdata /var/run/docker.sock -> /var/run/docker.sock /mnt/user/appdata/netdata/config -> /etc/netdata /etc/os-release -> /host/etc/os-release /etc/passwd -> /host/etc/passwd /proc -> /host/proc /sys -> /host/sys /var/log -> /host/var/log /usr/local/share/docker/tailscale_container_hook -> /opt/unraid/tailscale /run/dbus -> /run/dbus /etc/localtime -> /etc/localtime /etc/group -> /host/etc/group ===== playitgg ===== /mnt/user/appdata/playitgg -> /root/.config/playit_gg ===== allthemods10 ===== /mnt/user/appdata/allthemods10 -> /data ===== syncthing ===== /mnt/cache/appdata/syncthing -> /config /mnt/user/files/raashad-files -> /data1 /mnt/user/all-recordings/imports -> /data2 /usr/local/share/docker/tailscale_container_hook -> /opt/unraid/tailscale ===== Phoronix-Test-Suite ===== /usr/local/share/docker/tailscale_container_hook -> /opt/unraid/tailscale /mnt/user/appdata/phoronix-test-suite -> /var/lib/phoronix-test-suite /mnt/user/appdata/phoronix-test-suite/test-results -> /var/lib/phoronix-test-suite/test-results ===== icloudpd ===== /mnt/cache/appdata/icloudpd -> /config /mnt/user/icloudpd-staging -> /home/user/iCloud ===== stremio ===== /usr/local/share/docker/tailscale_container_hook -> /opt/unraid/tailscale /mnt/user/appdata/stremio -> /root/.stremio-server ===== dupeGuru ===== /mnt/cache/appdata/dupeGuru -> /config /mnt/user/icloudpd-staging -> /data /usr/local/share/docker/tailscale_container_hook -> /opt/unraid/tailscale /mnt/user -> /storage /dev/null -> /trash ===== itzg-minecraft-server ===== /mnt/user/appdata/minecraft -> /data /mnt/user/appdata/minecraft/downloads -> /downloads ===== ProjectZomboid ===== /mnt/user/appdata/steamcmd -> /serverdata/steamcmd /mnt/cache/appdata/projectzomboid -> /serverdata/serverfiles ===== speedtest-tracker ===== /mnt/cache/appdata/speedtest-tracker -> /config /usr/local/share/docker/tailscale_container_hook -> /opt/unraid/tailscale ===== RUST ===== /mnt/cache/appdata/rust -> /serverdata/serverfiles /mnt/user/appdata/steamcmd -> /serverdata/steamcmd ===== Immich_Redis ===== /mnt/user/appdata/redis -> /bitnami/redis ===== PostgreSQL_Immich ===== /mnt/user/appdata/PostgreSQL_Immich -> /var/lib/postgresql/data 8ce6a07d-b752-4dd2-8c92-5435f1dfb537 is the ATM10 container, it was created by Pelican- sorry for the weird name. itzg-minecraft-server and allthemods10 are not the offending servers/docker containers and are unrelated to this issue. tower-diagnostics-20260919-1600.zip

Sorry forgot to quote you, the response is above this one.

Below is the fresh diagnostics, take a look when you can. My next step is to disable XMP and confirm Intel Default Settings are on- then I'll run a mem test and then run the same workload to see if a crash happens. Sometime between this I will swap the USB as I ended up ordering one on Doordash that will arrive in an hour or 2. The logs were saved on a pool on the server. This latest fail was at 11:25am today (EST). The comet uses its own power supply, at least I believe it does since its connected to an external power source. I will note: The server was turned back on at 3:02pm, was online on tailscale for about 2 minutes then went offline. Unfortunately the KVM was also offline and was unable to be successfully turned on so we were unable to see what happened. Then we hard reboot the server again and it was online. That's when I collected this batch of the latest diagnostics. Thanks again for your help. tower-diagnostics-20260918-1534.zip

The physical console does freeze. My KVM (the GL Comet) sometimes works and sometimes doesn't but when it does and I am able to access it during a freeze the physical console doesn't do anything (can't type anything in, no updates, etc). The KVM is on tailscale but I also have a seperate cloud service for the KVM- I am not sure how related the KVM issues are with the Unraid problem but sometimes both go offline at the same time (and when I say offline I don't just mean tailscale, although that is also offline- i mean someone on the local network cannot access the KVM's local address). During failure a mouse and keyboard does not respond. I have not checked if someone on the same LAN can ping or SSH into Unraid but I imagine not. Sorry I posted old details for Unraid version and BIOS version. This diagnostics was from the 3rd crash and before I made those changes you mentioned (this was maybe 2 days ago). I will post new diagnostics once someone local to the server can do a hard reboot, but I'd imagine the information would be the same. Yes I plan on replacing the USB soon, I'll order something on Amazon today. I've already backed up the files but I'll make a new back up. I discovered the USB was failing maybe 2-3 weeks ago (before I created this new docker container and while I was running a different Minecraft server) but I am pretty sure that USB has been failing for at least 6 months as I've always seen those errors but never knew what they were (lol). I enabled remote logs a few days ago and stored them in a place that was not the USB but the information was basically the same as what was in the diagnostics/what I could see on the KVM. I deleted those logs unfortunately because I found nothing from them. The BIOS is maybe 3 versions behind but no changes to the 13th gen Intel so I decided not to update as I've had a bad history with BIOS updates (bricked a motherboard or 2, love Gigabyte :D). I've tried Intel Default Settings but that did not fix the issue. XMP has been enabled since the server was born (and even when it used to be my gaming PC), but I'll try that. I'll also run memtest at some point soon (when the server is back online). Generation position doesn't seem to matter, just elapsed time since Chunky pregen has been running. Of course there is expensive terrain to generate and very easy ones (like oceans) but that doesn't matter in terms of the lock up. I'll provide the container mappings soon as I may have that information on hand, if not I'll provide it when the server is back online. Haha, I am well aware of the problems with 13th and 14th gen i9's- I am on my 3rd (gasps) replacement :D. I really hope its not the CPU.. i don't think so personally since like I said only with the addition of the 3 mods the issue occurs but I also acknowledge I don't have a clue as to what is going on LOL. Thank you so much for your help! This has been a tremendous 6 days of debugging..

I'm experiencing a reproducible complete Unraid host lockup when running Chunky (a minecraft mod) to pregenerate an ATM10 (minecraft modpack) world. Minecraft is running inside Docker through Pelican (the server itself is under Pelican-Wings). I have hosted Minecraft server with and without Unraid with a variety of configurations and never had this happen before. Long story short I added 3 additional mods to my minecraft server (and did a clean wipe of all the world files) and now after 3-3.5 hours of Chunky pregeneration, the entire Unraid host becomes inaccessible. Tailscale stops responding, my remote KVM becomes inaccessible, and the server requires a physical power-button reboot to recover. When it locks up the syslogs, docker logs, pelican logs and minecraft logs all have nothing of relevance- everything just stops reporting the second the lock up happens. Minecraft itself can remain running for many hours if Chunky pregeneration is not enabled. I also tested pausing Chunky for approximately 14 hours while leaving Minecraft and Unraid running, and the server remained completely stable. After resuming Chunky, the host eventually locked up again after approximately 2 hours 50 minutes. I've now had approximately four crashes with a similar general pattern, historically around 3–3.5 hours of continuous Chunky operation. The easy solution of course would be removing the 3 extra mods I added because I know that Chunky would be able to run infinitely without it however, I want to find the reason why Unraid is locking up. I've had Unraid set up on this server for a little over a year and something like this has never happened before. Sure, I've had my Minecraft docker container(s) crash many times, but never had Unraid lock up with no data in syslogs as to why its happening. Below is a list of things I've tried to fix the issue... Tests/troubleshooting already performed CPU temperature monitored under load — ~63–65°C during recent crashes. CPU/per-core utilization monitored — no sustained host-wide CPU saturation. CPU thermal-throttling counters checked. CPU power limits tested: 253/253W and 110/160W. CPU pinned to P-cores; other Unraid tasks pinned to E-cores. Unassigned Minecraft from P core 0 to insure there were no lock ups from collisions with Unraid System RAM monitored — no exhaustion; ~48 GiB available before crash. Swap monitored — 0 usage. Minecraft Java heap monitored — no evidence of a memory leak/OOM. Docker resource usage checked with docker stats. NVMe temperatures monitored. Samsung 990 Pro SMART checked — no media/data-integrity errors. Crucial P5 SMART checked — no media/data-integrity errors; self-test passed. NVMe I/O utilization, latency, queue depth, and system iowait monitored. BTRFS filesystem usage checked — large amount of free space. btrfs device stats checked — 0 read/write/flush/corruption/generation errors. vmstat monitored — no RAM, swap, CPU, or blocked-process exhaustion. mpstat monitored — no host-wide CPU saturation. iostat monitored — no obvious storage failure/latency event immediately before crash. syslog/dmesg checked for MCE, hardware errors, CPU errors, lockups, watchdog, thermal, and microcode errors — nothing useful found. /sys/fs/pstore checked — empty. MCE/NMI/EDAC/RAS error counters checked — no useful hardware-error evidence. NVMe APST disabled with nvme_core.default_ps_max_latency_us=0. PCIe ASPM disabled with pcie_aspm=off. Chunky paused for ~14 hours while Minecraft/Unraid remained running — stable. Chunky resumed afterward — host hard-locked again after ~2h50m. Reproduced several previous hard-locks after roughly 3–3.5 hours of continuous Chunky. Minecraft without continuous Chunky can run for many hours without a host lockup. Compared old worldgen setup (Terralith 2.5.8 + BOP + Tectonic) with current heavier worldgen setup. Current setup includes newer Terralith + BOP + Tectonic + Feature Recycler; the older setup did not produce these host lockups. Considered/tested the possibility of /mnt/user/FUSE storage overhead; direct /mnt/cache testing remains a potential next experiment. Considered BTRFS/CoW as a possible workload interaction; no BTRFS errors found. Changed Docker network driver from macvlan → ipvlan to test for Docker/network-driver kernel issues. The only redeeming finding from my tests has been pausing Chunky pregen and resuming it, which means the issue must be some resource that getting exhausted over time (as the lock up occurs always at around the same time interval). I'm at the end of the rope here, I do not want to delete the world again and downgrade the mods to the ones that were working before these locks up started happening. I believe that the issue can be found and is fixable so I'm looking for help identifying what could cause a complete kernel/host-level lockup under this workload. Below is some information about my system and the diagnostic logs from after one of the locks up is attached, although it isn't very helpful.. Note I am aware my USB boot device is failing and I plan to swap it soon however, this isn't the cause of the issue in this post. Hardware / SoftwareCPUIntel Core i9-13900K 24 cores / 32 threads 8 P-cores + 16 E-cores Minecraft/ATM10 Docker is pinned to all the P-cores (except core 0) MotherboardGigabyte Z790 AORUS Elite AX DDR4 BIOS: F13 RAM64 GB DDR4 4 × 16 GB Corsair modules Part number: CMW32GX4M2E3200C16 Configured speed: 3200 MT/s (XMP enabled) GPUEVGA RTX 3070 Ti FTW3 CPU coolingEK-Nucleus CR360 Lux D-RGB (360mm AIO) StorageBTRFS cache pool consisting of: Samsung 990 PRO 2 TB Serial: S73WNJ0W216739Z Firmware: 1B2QJXD7 Current mapping: /dev/nvme1n1 Crucial P5 2 TB Model: CT2000P5SSD8 Serial: 213431212F67 Firmware: P4CR311 Current mapping: /dev/nvme2n1 There is also: Samsung OEM 256 GB NVMe Model: MZVLB256HAHQ Current mapping: /dev/nvme0n1 Not currently being used as a BTRFS RAID1 member The cache pool is BTRFS RAID1. HDDsThe array also contains multiple 16 TB Toshiba/MG08-series HDDs. Unraid / KernelUnraid kernel: 7.3.1 Booted using the normal Unraid USB installation Minecraft / ATM10The server is running All The Mods 10 (ATM10). Minecraft is running inside Docker through Pelican (the server itself is under Pelican-Wings). All of my docker containers are in the Cache which are in the 2 ssds I mentioned previously. The Minecraft server uses: NeoForge 21.1.234 Custom JVM arguments (the default provided by ATM10) Approximately 18 GB maximum Java heap tower-diagnostics-20260917-0711.zip