I'm experiencing a reproducible complete Unraid host lockup when running Chunky (a minecraft mod) to pregenerate an ATM10 (minecraft modpack) world. Minecraft is running inside Docker through Pelican (the server itself is under Pelican-Wings). I have hosted Minecraft server with and without Unraid with a variety of configurations and never had this happen before. Long story short I added 3 additional mods to my minecraft server (and did a clean wipe of all the world files) and now after 3-3.5 hours of Chunky pregeneration, the entire Unraid host becomes inaccessible. Tailscale stops responding, my remote KVM becomes inaccessible, and the server requires a physical power-button reboot to recover. When it locks up the syslogs, docker logs, pelican logs and minecraft logs all have nothing of relevance- everything just stops reporting the second the lock up happens.

Minecraft itself can remain running for many hours if Chunky pregeneration is not enabled. I also tested pausing Chunky for approximately 14 hours while leaving Minecraft and Unraid running, and the server remained completely stable. After resuming Chunky, the host eventually locked up again after approximately 2 hours 50 minutes.

I've now had approximately four crashes with a similar general pattern, historically around 3–3.5 hours of continuous Chunky operation. The easy solution of course would be removing the 3 extra mods I added because I know that Chunky would be able to run infinitely without it however, I want to find the reason why Unraid is locking up. I've had Unraid set up on this server for a little over a year and something like this has never happened before. Sure, I've had my Minecraft docker container(s) crash many times, but never had Unraid lock up with no data in syslogs as to why its happening. Below is a list of things I've tried to fix the issue...

Tests/troubleshooting already performed

• CPU temperature monitored under load — ~63–65°C during recent crashes.

• CPU/per-core utilization monitored — no sustained host-wide CPU saturation.

• CPU thermal-throttling counters checked.

• CPU power limits tested: 253/253W and 110/160W.

• CPU pinned to P-cores; other Unraid tasks pinned to E-cores.

• Unassigned Minecraft from P core 0 to insure there were no lock ups from collisions with Unraid

• System RAM monitored — no exhaustion; ~48 GiB available before crash.

• Swap monitored — 0 usage.

• Minecraft Java heap monitored — no evidence of a memory leak/OOM.

• Docker resource usage checked with docker stats.

• NVMe temperatures monitored.

• Samsung 990 Pro SMART checked — no media/data-integrity errors.

• Crucial P5 SMART checked — no media/data-integrity errors; self-test passed.

• NVMe I/O utilization, latency, queue depth, and system iowait monitored.

• BTRFS filesystem usage checked — large amount of free space.

• btrfs device stats checked — 0 read/write/flush/corruption/generation errors.

• vmstat monitored — no RAM, swap, CPU, or blocked-process exhaustion.

• mpstat monitored — no host-wide CPU saturation.

• iostat monitored — no obvious storage failure/latency event immediately before crash.

• syslog/dmesg checked for MCE, hardware errors, CPU errors, lockups, watchdog, thermal, and microcode errors — nothing useful found.

• /sys/fs/pstore checked — empty.

• MCE/NMI/EDAC/RAS error counters checked — no useful hardware-error evidence.

• NVMe APST disabled with nvme_core.default_ps_max_latency_us=0.

• PCIe ASPM disabled with pcie_aspm=off.

• Chunky paused for ~14 hours while Minecraft/Unraid remained running — stable.

• Chunky resumed afterward — host hard-locked again after ~2h50m.

• Reproduced several previous hard-locks after roughly 3–3.5 hours of continuous Chunky.

• Minecraft without continuous Chunky can run for many hours without a host lockup.

• Compared old worldgen setup (Terralith 2.5.8 + BOP + Tectonic) with current heavier worldgen setup.

• Current setup includes newer Terralith + BOP + Tectonic + Feature Recycler; the older setup did not produce these host lockups.

• Considered/tested the possibility of /mnt/user/FUSE storage overhead; direct /mnt/cache testing remains a potential next experiment.

• Considered BTRFS/CoW as a possible workload interaction; no BTRFS errors found.

• Changed Docker network driver from macvlan → ipvlan to test for Docker/network-driver kernel issues.

The only redeeming finding from my tests has been pausing Chunky pregen and resuming it, which means the issue must be some resource that getting exhausted over time (as the lock up occurs always at around the same time interval).

I'm at the end of the rope here, I do not want to delete the world again and downgrade the mods to the ones that were working before these locks up started happening. I believe that the issue can be found and is fixable so I'm looking for help identifying what could cause a complete kernel/host-level lockup under this workload. Below is some information about my system and the diagnostic logs from after one of the locks up is attached, although it isn't very helpful..

Note I am aware my USB boot device is failing and I plan to swap it soon however, this isn't the cause of the issue in this post.

Hardware / Software

CPU

• Intel Core i9-13900K

• 24 cores / 32 threads

• 8 P-cores + 16 E-cores

• Minecraft/ATM10 Docker is pinned to all the P-cores (except core 0)

Motherboard

• Gigabyte Z790 AORUS Elite AX DDR4

• BIOS: F13

RAM

• 64 GB DDR4

• 4 × 16 GB Corsair modules

• Part number: CMW32GX4M2E3200C16

• Configured speed: 3200 MT/s (XMP enabled)

GPU

• EVGA RTX 3070 Ti FTW3

CPU cooling

• EK-Nucleus CR360 Lux D-RGB (360mm AIO)

Storage

BTRFS cache pool consisting of:

• Samsung 990 PRO 2 TB

  • Serial: S73WNJ0W216739Z

  • Firmware: 1B2QJXD7

  • Current mapping: /dev/nvme1n1

• Crucial P5 2 TB

  • Model: CT2000P5SSD8

  • Serial: 213431212F67

  • Firmware: P4CR311

  • Current mapping: /dev/nvme2n1

There is also:

• Samsung OEM 256 GB NVMe

• Model: MZVLB256HAHQ

• Current mapping: /dev/nvme0n1

• Not currently being used as a BTRFS RAID1 member

The cache pool is BTRFS RAID1.

HDDs

The array also contains multiple 16 TB Toshiba/MG08-series HDDs.

Unraid / Kernel

• Unraid kernel: 7.3.1

• Booted using the normal Unraid USB installation

Minecraft / ATM10

The server is running All The Mods 10 (ATM10). Minecraft is running inside Docker through Pelican (the server itself is under Pelican-Wings). All of my docker containers are in the Cache which are in the 2 ssds I mentioned previously.

The Minecraft server uses:

• NeoForge 21.1.234

• Custom JVM arguments (the default provided by ATM10)

• Approximately 18 GB maximum Java heap

tower-diagnostics-20260917-0711.zip

Edited September 18Sep 18 by ijustwantunraidtowork
more details added, fixed some information

Your pause/resume test links the failure to Chunky pregeneration, but it does not prove resource exhaustion. A workload-specific hardware or kernel problem can produce similar timing.

A few details in the diagnostics need clarification before further testing:

1. Does the physical console freeze? Please describe the remote KVM setup, including whether it depends on Tailscale or power from the server. During a failure, does a directly attached keyboard and monitor respond? Can another device on the same LAN ping or SSH into Unraid?

2. Please attach current diagnostics. This archive reports Unraid 7.3.1 and BIOS F13 dated January 7, 2025. Its boot command line contains neither nvme_core.default_ps_max_latency_us=0 nor pcie_aspm=off. Was it collected before those changes?

3. Replace the failing boot USB before more long reproduction tests. Back up its readable configuration first. The log contains repeated medium errors, failed FAT reads, and /boot I/O errors. These do not prove that the USB causes the lockup, but we cannot exclude it yet.

4. Capture logs on a separate machine. Configure forwarding under Settings → Syslog Server, then verify that the receiver gets messages. Include the exact failure time and the log from before the reboot. Avoid using the failing USB as the logging destination. Logging instructions (https://docs.unraid.net/unraid-os/troubleshooting/diagnostics/capture-diagnostics-and-logs/).

5. Establish a hardware baseline. Confirm current BIOS for your exact board revision, use Intel Default Settings, and disable XMP. Record each change and result. Run an offline memory test at default memory settings. Your loaded microcode is already 0x133, so this is not simply a missing runtime-microcode update.

If it still fails, please provide the actual Minecraft container’s host volume mappings. We can then assess a direct-cache comparison without accidentally exposing a different data set.

There is no need to delete the world to begin these checks. Also note whether failures track a particular generation position or only similar elapsed time.

P.S. The i9-13900K is also a significant suspect. These processors belong to the family affected by Intel’s acknowledged desktop CPU instability issue. I have seen several support cases involving stability problems with these CPUs that were onlyreoslved after replacing, in some cases users are on their 3rd and 4th CPU.

If the initial checks find no other cause, the CPU would be my main suspect. Having microcode 0x133 loaded does not rule out earlier degradation or prove that the CPU is healthy.

If failures continue with current BIOS, Intel Default Settings, and XMP disabled, a known-good CPU comparison or an Intel/vendor warranty assessment would be reasonable. The current logs do not confirm CPU failure.

1 minute ago, JorgeB said:

Your pause/resume test links the failure to Chunky pregeneration, but it does not prove resource exhaustion. A workload-specific hardware or kernel problem can produce similar timing.

A few details in the diagnostics need clarification before further testing:

1. Does the physical console freeze? Please describe the remote KVM setup, including whether it depends on Tailscale or power from the server. During a failure, does a directly attached keyboard and monitor respond? Can another device on the same LAN ping or SSH into Unraid?

2. Please attach current diagnostics. This archive reports Unraid 7.3.1 and BIOS F13 dated January 7, 2025. Its boot command line contains neither nvme_core.default_ps_max_latency_us=0 nor pcie_aspm=off. Was it collected before those changes?

3. Replace the failing boot USB before more long reproduction tests. Back up its readable configuration first. The log contains repeated medium errors, failed FAT reads, and /boot I/O errors. These do not prove that the USB causes the lockup, but we cannot exclude it yet.

4. Capture logs on a separate machine. Configure forwarding under Settings → Syslog Server, then verify that the receiver gets messages. Include the exact failure time and the log from before the reboot. Avoid using the failing USB as the logging destination. Logging instructions (https://docs.unraid.net/unraid-os/troubleshooting/diagnostics/capture-diagnostics-and-logs/).

5. Establish a hardware baseline. Confirm current BIOS for your exact board revision, use Intel Default Settings, and disable XMP. Record each change and result. Run an offline memory test at default memory settings. Your loaded microcode is already 0x133, so this is not simply a missing runtime-microcode update.

If it still fails, please provide the actual Minecraft container’s host volume mappings. We can then assess a direct-cache comparison without accidentally exposing a different data set.

There is no need to delete the world to begin these checks. Also note whether failures track a particular generation position or only similar elapsed time.

P.S. The i9-13900K is also a significant suspect. These processors belong to the family affected by Intel’s acknowledged desktop CPU instability issue. I have seen several support cases involving stability problems with these CPUs that were onlyreoslved after replacing, in some cases users are on their 3rd and 4th CPU.

If the initial checks find no other cause, the CPU would be my main suspect. Having microcode 0x133 loaded does not rule out earlier degradation or prove that the CPU is healthy.

If failures continue with current BIOS, Intel Default Settings, and XMP disabled, a known-good CPU comparison or an Intel/vendor warranty assessment would be reasonable. The current logs do not confirm CPU failure.

1. The physical console does freeze. My KVM (the GL Comet) sometimes works and sometimes doesn't but when it does and I am able to access it during a freeze the physical console doesn't do anything (can't type anything in, no updates, etc). The KVM is on tailscale but I also have a seperate cloud service for the KVM- I am not sure how related the KVM issues are with the Unraid problem but sometimes both go offline at the same time (and when I say offline I don't just mean tailscale, although that is also offline- i mean someone on the local network cannot access the KVM's local address). During failure a mouse and keyboard does not respond. I have not checked if someone on the same LAN can ping or SSH into Unraid but I imagine not.

2. Sorry I posted old details for Unraid version and BIOS version. This diagnostics was from the 3rd crash and before I made those changes you mentioned (this was maybe 2 days ago). I will post new diagnostics once someone local to the server can do a hard reboot, but I'd imagine the information would be the same.

3. Yes I plan on replacing the USB soon, I'll order something on Amazon today. I've already backed up the files but I'll make a new back up. I discovered the USB was failing maybe 2-3 weeks ago (before I created this new docker container and while I was running a different Minecraft server) but I am pretty sure that USB has been failing for at least 6 months as I've always seen those errors but never knew what they were (lol).

4. I enabled remote logs a few days ago and stored them in a place that was not the USB but the information was basically the same as what was in the diagnostics/what I could see on the KVM. I deleted those logs unfortunately because I found nothing from them.

5. The BIOS is maybe 3 versions behind but no changes to the 13th gen Intel so I decided not to update as I've had a bad history with BIOS updates (bricked a motherboard or 2, love Gigabyte :D). I've tried Intel Default Settings but that did not fix the issue. XMP has been enabled since the server was born (and even when it used to be my gaming PC), but I'll try that. I'll also run memtest at some point soon (when the server is back online).

Generation position doesn't seem to matter, just elapsed time since Chunky pregen has been running. Of course there is expensive terrain to generate and very easy ones (like oceans) but that doesn't matter in terms of the lock up. I'll provide the container mappings soon as I may have that information on hand, if not I'll provide it when the server is back online.

Haha, I am well aware of the problems with 13th and 14th gen i9's- I am on my 3rd (gasps) replacement :D. I really hope its not the CPU.. i don't think so personally since like I said only with the addition of the 3 mods the issue occurs but I also acknowledge I don't have a clue as to what is going on LOL.

Thank you so much for your help! This has been a tremendous 6 days of debugging..

That helps narrow it down. The frozen local console makes a Tailscale-only problem unlikely. The new mods identify a trigger, but they do not distinguish hardware instability from a kernel or driver problem.

The CPU remains a significant suspect, especially given the replacement history, but we do not have evidence that this CPU has failed.

Since Intel Default Settings did not help, disabling XMP is the next useful comparison. Previous stability with XMP does not exclude a problem under this workload. Run the offline memory test with XMP disabled. After replacing the failing USB, repeat the workload with XMP still disabled.

One correction on the BIOS: Gigabyte lists F14 with a 13th/14th-generation stability mitigation for this board. Your diagnostic already shows runtime microcode 0x133, so this is not simply a missing runtime-microcode update. However, newer BIOS releases are not necessarily irrelevant.

Please attach fresh diagnostics after recovery. Were the saved logs on a separate machine, or on a pool/share on this server? Keep the next log even if it ends without an error, and include the failure time.

Also, does the GL Comet use its own power supply or draw power from the server? During another freeze, please have someone test LAN access to Unraid and the Comet separately.

There is no need to delete the world. The XMP-disabled result, fresh diagnostics, and container mappings shoudl guide the next comparison.

Below is the fresh diagnostics, take a look when you can. My next step is to disable XMP and confirm Intel Default Settings are on- then I'll run a mem test and then run the same workload to see if a crash happens. Sometime between this I will swap the USB as I ended up ordering one on Doordash that will arrive in an hour or 2.

The logs were saved on a pool on the server. This latest fail was at 11:25am today (EST).

The comet uses its own power supply, at least I believe it does since its connected to an external power source.

I will note: The server was turned back on at 3:02pm, was online on tailscale for about 2 minutes then went offline. Unfortunately the KVM was also offline and was unable to be successfully turned on so we were unable to see what happened. Then we hard reboot the server again and it was online. That's when I collected this batch of the latest diagnostics.

Thanks again for your help.

tower-diagnostics-20260918-1534.zip

Edited September 18Sep 18 by ijustwantunraidtowork

1 hour ago, JorgeB said:

That helps narrow it down. The frozen local console makes a Tailscale-only problem unlikely. The new mods identify a trigger, but they do not distinguish hardware instability from a kernel or driver problem.

The CPU remains a significant suspect, especially given the replacement history, but we do not have evidence that this CPU has failed.

Since Intel Default Settings did not help, disabling XMP is the next useful comparison. Previous stability with XMP does not exclude a problem under this workload. Run the offline memory test with XMP disabled. After replacing the failing USB, repeat the workload with XMP still disabled.

One correction on the BIOS: Gigabyte lists F14 with a 13th/14th-generation stability mitigation for this board. Your diagnostic already shows runtime microcode 0x133, so this is not simply a missing runtime-microcode update. However, newer BIOS releases are not necessarily irrelevant.

Please attach fresh diagnostics after recovery. Were the saved logs on a separate machine, or on a pool/share on this server? Keep the next log even if it ends without an error, and include the failure time.

Also, does the GL Comet use its own power supply or draw power from the server? During another freeze, please have someone test LAN access to Unraid and the Comet separately.

There is no need to delete the world. The XMP-disabled result, fresh diagnostics, and container mappings shoudl guide the next comparison.

Sorry forgot to quote you, the response is above this one.

Edited September 18Sep 18 by ijustwantunraidtowork

The new diagnostics confirm that both boot arguments are now present. BIOS F13 and microcode 0x133 are unchanged. The USB still produces repeated read errors, so recommend to replace it before another long workload test.

This syslog only covers 15:28–15:34, after the second reboot. It contains neither the 11:25 failure nor the loss of access shortly after 15:02. There is no captured crash signature.

The logs saved on your pool still depend on the affected server. If possible configure syslog forwarding to a separate computer and verify that it receives messages before the next test. Keep the log even if it ends without an error. Any remaining pool logs need to be attached separately; standard diagnostics do not include them.

After replacing the USB, proceed with Intel Default Settings, XMP disabled, and the offline memory test. Please report the completed passes and any errors. If it passes, repeat the workload with XMP still disabled. A passing memory test does not rule out a CPU problem.

Had Minecraft and Chunky automatically restarted before the two-minute outage, or were they still stopped? That distinction matters because this event may differ from the earlier three-hour freezes.

If access fails again, have someone check the directly attached keyboard/display and LAN access to Unraid and the Comet separately. The Comet’s apparent external power supply makes its simultaneous outage worth investigating independently.

The CPU remains a significant suspect, but these diagnostics do not confirm CPU failure.

14 hours ago, ijustwantunraidtowork said:

1. Yes I plan on replacing the USB soon, I'll order something on Amazon today.

I advise against wasting money on low-quality drives sold on Amazon.

Instead get this:

https://forums.unraid.net/topic/199642-psa-on-industrial-mlc-usb-boot-drives-mlc-nand-end-of-life-and-last-accessible-stock/

13 hours ago, JorgeB said:

The new diagnostics confirm that both boot arguments are now present. BIOS F13 and microcode 0x133 are unchanged. The USB still produces repeated read errors, so recommend to replace it before another long workload test.

This syslog only covers 15:28–15:34, after the second reboot. It contains neither the 11:25 failure nor the loss of access shortly after 15:02. There is no captured crash signature.

The logs saved on your pool still depend on the affected server. If possible configure syslog forwarding to a separate computer and verify that it receives messages before the next test. Keep the log even if it ends without an error. Any remaining pool logs need to be attached separately; standard diagnostics do not include them.

After replacing the USB, proceed with Intel Default Settings, XMP disabled, and the offline memory test. Please report the completed passes and any errors. If it passes, repeat the workload with XMP still disabled. A passing memory test does not rule out a CPU problem.

Had Minecraft and Chunky automatically restarted before the two-minute outage, or were they still stopped? That distinction matters because this event may differ from the earlier three-hour freezes.

If access fails again, have someone check the directly attached keyboard/display and LAN access to Unraid and the Comet separately. The Comet’s apparent external power supply makes its simultaneous outage worth investigating independently.

The CPU remains a significant suspect, but these diagnostics do not confirm CPU failure.

Hey. I got a new set of diagnostics for you to look at if you need it. I made some changes and ran some tests since I last replied, then ran the Minecraft server with Chunky. It only lasted 48 minutes and 12 seconds this time :( but the KVM was reachable. These diagnostics were immediately after the crash and the Chunky run was after all the changes listed below.

Here are the list of changes I've made to Unraid:

• Updated Unraid from 7.3.1 → 7.3.2

• Docker changed from 29.5.1 → 29.5.3

• Migrated Unraid boot from the failing SanDisk USB to the 256 GB Samsung NVMe using Unraid Internal Boot.

• Removed the accidentally/stale third BTRFS cache-pool member created during the Internal Boot migration and restored the cache pool to the Samsung 990 Pro + Crucial P5 RAID1.

• Removed the boot parameters nvme_core.default_ps_max_latency_us=0 pcie_aspm=off

• Confirmed Magic SysRq is enabled (kernel.sysrq=1); tried executing that command via the KVM and terminal did not respond during crash.

Tests performed on RAM and CPU:

1. Memtest86+

   • 6 complete passes.

   • 0 errors.

   • Roughly 13+ hours.

2. 32-worker stress-ng CPU test

   • All 32 logical CPUs stressed.

   • Reached ~94–100°C/TjMax, so test was stopped after 10m46s.

   • 32/32 workers passed, 0 failed.

   • No MCE/hardware errors afterward.

3. 7-worker P-core stress-ng CPU test

   • Restricted to Minecraft's CPU 1–15 pool.

   • Full-speed CPU stress for 1 hour.

   • Temperatures mostly ~78–85°C.

   • 7/7 passed, 0 failed.

4. 8-worker physical P-core stress-ng CPU test

   • Allowed CPUs 1,2,4,6,8,10,12,14, giving access to one logical thread from each physical P-core.

   • 1 hour.

   • 8/8 passed, 0 failed.

5. CPU cache/coherency stress-ng test

   • --cache 4 --cacheline 4 --verify.

   • 1 hour.

   • 8/8 stressors passed, 0 failed.

I think you asked for container mappings, so they are below:
===== 8ce6a07d-b752-4dd2-8c92-5435f1dfb537 =====

/tmp/pelican/group -> /etc/group

/tmp/pelican/machine-id/8ce6a07d-b752-4dd2-8c92-5435f1dfb537 -> /etc/machine-id

/tmp/pelican/passwd -> /etc/passwd

/mnt/user/appdata/pelican-wing/data/8ce6a07d-b752-4dd2-8c92-5435f1dfb537 -> /home/container

===== immich =====

/mnt/cache/appdata/immich -> /config

/mnt/user/Immich-Imports -> /libraries

/usr/local/share/docker/tailscale_container_hook -> /opt/unraid/tailscale

/mnt/user/Immich -> /photos

===== Mainfrezzer-Pelican-Panel =====

/mnt/user/appdata/pelican-panel/pelican-log -> /var/www/html/storage/logs

/usr/local/share/docker/tailscale_container_hook -> /opt/unraid/tailscale

/mnt/user/appdata/pelican-panel/pelican-data -> /pelican-data

/mnt/user/appdata/pelican-panel/pelican-plugins -> /var/www/html/plugins

===== Mainfrezzer-Pelican-Wing =====

/mnt/user/appdata/pelican-wing/etc/pelican -> /etc/pelican

/mnt/user/appdata/pelican-wing/data -> /mnt/user/appdata/pelican-wing/data

/tmp/pelican -> /tmp/pelican

/var/run/docker.sock -> /var/run/docker.sock

===== netdata =====

/mnt/user/appdata/netdata/cache -> /var/cache/netdata

/mnt/user/appdata/netdata/lib -> /var/lib/netdata

/var/run/docker.sock -> /var/run/docker.sock

/mnt/user/appdata/netdata/config -> /etc/netdata

/etc/os-release -> /host/etc/os-release

/etc/passwd -> /host/etc/passwd

/proc -> /host/proc

/sys -> /host/sys

/var/log -> /host/var/log

/usr/local/share/docker/tailscale_container_hook -> /opt/unraid/tailscale

/run/dbus -> /run/dbus

/etc/localtime -> /etc/localtime

/etc/group -> /host/etc/group

===== playitgg =====

/mnt/user/appdata/playitgg -> /root/.config/playit_gg

===== allthemods10 =====

/mnt/user/appdata/allthemods10 -> /data

===== syncthing =====

/mnt/cache/appdata/syncthing -> /config

/mnt/user/files/raashad-files -> /data1

/mnt/user/all-recordings/imports -> /data2

/usr/local/share/docker/tailscale_container_hook -> /opt/unraid/tailscale

===== Phoronix-Test-Suite =====

/usr/local/share/docker/tailscale_container_hook -> /opt/unraid/tailscale

/mnt/user/appdata/phoronix-test-suite -> /var/lib/phoronix-test-suite

/mnt/user/appdata/phoronix-test-suite/test-results -> /var/lib/phoronix-test-suite/test-results

===== icloudpd =====

/mnt/cache/appdata/icloudpd -> /config

/mnt/user/icloudpd-staging -> /home/user/iCloud

===== stremio =====

/usr/local/share/docker/tailscale_container_hook -> /opt/unraid/tailscale

/mnt/user/appdata/stremio -> /root/.stremio-server

===== dupeGuru =====

/mnt/cache/appdata/dupeGuru -> /config

/mnt/user/icloudpd-staging -> /data

/usr/local/share/docker/tailscale_container_hook -> /opt/unraid/tailscale

/mnt/user -> /storage

/dev/null -> /trash

===== itzg-minecraft-server =====

/mnt/user/appdata/minecraft -> /data

/mnt/user/appdata/minecraft/downloads -> /downloads

===== ProjectZomboid =====

/mnt/user/appdata/steamcmd -> /serverdata/steamcmd

/mnt/cache/appdata/projectzomboid -> /serverdata/serverfiles

===== speedtest-tracker =====

/mnt/cache/appdata/speedtest-tracker -> /config

/usr/local/share/docker/tailscale_container_hook -> /opt/unraid/tailscale

===== RUST =====

/mnt/cache/appdata/rust -> /serverdata/serverfiles

/mnt/user/appdata/steamcmd -> /serverdata/steamcmd

===== Immich_Redis =====

/mnt/user/appdata/redis -> /bitnami/redis

===== PostgreSQL_Immich =====

/mnt/user/appdata/PostgreSQL_Immich -> /var/lib/postgresql/data

8ce6a07d-b752-4dd2-8c92-5435f1dfb537 is the ATM10 container, it was created by Pelican- sorry for the weird name. itzg-minecraft-server and allthemods10 are not the offending servers/docker containers and are unrelated to this issue.

tower-diagnostics-20260919-1600.zip

Edited September 19Sep 19 by ijustwantunraidtowork

On a different note, I think either the update to Unraid/Docker might've sped up the inevitable (the crash) or my changes to the cache pool caused the crash to be quicker. 4 out of 5 crashes were always between 3 - 3.5 hours and I don't think it is a coincidence that this latest crash barely made it to 50 minutes right after my changes. When I changed the boot parameters before either crash 2 or 3 and ran Chunky again, it made no changes to how long a crash would take- I doubt that removing the boot parameters did anything. I might be grasping at straws here but my only indicator is the cache pool system, so I am going to chase that lead (or shadow..).

For context when I changed the boot from USB to internal boot I kinda messed something up so my other 2 SSD's appeared as internal boot drives as well. Apparently this might've been a glitch in Unraid 7.3.1, so I made some changes- after switching the 256 GB Samsung NVMe to Unraid Internal Boot, I removed it from the cache pool’s stale Btrfs membership with btrfs device remove 2 /mnt/cache, updated cache.cfg from a 3-device to a 2-device pool, and left the Samsung 990 Pro + Crucial P5 as the RAID1 cache pool. i also had to re-enable the cache pool’s user share assignment, because it had ended up disabled and /mnt/user/... paths stopped working.

Some commands I am running now while investigating..
root@Tower:~# docker inspect 8ce6a07d-b752-4dd2-8c92-5435f1dfb537 \

--format '{{range .Mounts}}{{println .Source "->" .Destination}}{{end}}'

/tmp/pelican/passwd -> /etc/passwd

/mnt/user/appdata/pelican-wing/data/8ce6a07d-b752-4dd2-8c92-5435f1dfb537 -> /home/container

/tmp/pelican/group -> /etc/group

/tmp/pelican/machine-id/8ce6a07d-b752-4dd2-8c92-5435f1dfb537 -> /etc/machine-id

root@Tower:~# findmnt -T /mnt/cache

findmnt -T /mnt/user

TARGET SOURCE FSTYPE OPTIONS

/mnt/cache /dev/nvme1n1p1 btrfs rw,noatime,ssd,discard=async,space_cache=v2,subvolid=5,subvol=/

TARGET SOURCE FSTYPE OPTIONS

/mnt/user shfs fuse.shfs rw,nosuid,nodev,noatime,user_id=0,group_id=0,default_permissions,allow_other

root@Tower:~#

Edited September 19Sep 19 by ijustwantunraidtowork

It seems like the cache pool mappings was the issue- Chunky has been running for 8 hours now. I changed both Pelican Panel and Pelican Wings from using /mnt/user/appdata/... host mappings to the equivalent /mnt/cache/appdata/... mappings, and updated Wings’ config.yml so its root/data/log/archive/backup paths also use /mnt/cache. The files themselves did not move- they were already physically on the cache pool; this only changed how the containers reach them, bypassing Unraid’s shfs/FUSE user-share layer. A visual of the before and after mappings is below..

Before:

Minecraft

↓

/home/container

↓ Docker bind mount

/mnt/user/appdata/pelican-wing/data/...

↓

Unraid shfs/FUSE user-share layer

↓

/mnt/cache/appdata/pelican-wing/data/...

↓

Btrfs filesystem

↓

Samsung 990 Pro + Crucial P5 cache pool (RAID1)

Now:

Minecraft

↓

/home/container

↓ Docker bind mount

/mnt/cache/appdata/pelican-wing/data/...

↓

Btrfs filesystem

↓

Samsung 990 Pro + Crucial P5 cache pool (RAID1)

If Chunky makes it to about 8 more hours when I wake up, I'll mark this reply as the solution. Can't believe it took 6 days to find the solution to this..

Edited September 20Sep 20 by ijustwantunraidtowork
editted hierarchy to be more detailed for noobs (like me) who might go through the same issue

That is a useful result. Eight hours is well beyond the previous failure times. Since the files stayed on the same pool, this points toward access through /mnt/user and the user-share layer. It does not mean the cache pool itself was faulty or that your original mappings were wrong.

Keep the direct /mnt/cache mappings and let the same pregeneration run complete without further changes. Please report the final runtime and whether the server remains responsive.

For now, I would describe this as a promising workaround.

Your latest diagnostics already show appdata only on cache. Keep its Primary storage set to cache and Secondary storage set to None so those files remain available through the direct path. Also confirm that the Minecraft container retains its direct-pool mapping if Wings recreates it.

This changes my earlier assessment: the storage access path is now the stronger lead than the CPU. The memory and CPU tests are useful, although they cannot exclude every hardware fault.

The latest diagnostics cover only the reboot after the failure. They do not capture the stalled system, so we cannot yet identify a specific shfs problem.

8 hours ago, JorgeB said:

That is a useful result. Eight hours is well beyond the previous failure times. Since the files stayed on the same pool, this points toward access through /mnt/user and the user-share layer. It does not mean the cache pool itself was faulty or that your original mappings were wrong.

Keep the direct /mnt/cache mappings and let the same pregeneration run complete without further changes. Please report the final runtime and whether the server remains responsive.

For now, I would describe this as a promising workaround.

Your latest diagnostics already show appdata only on cache. Keep its Primary storage set to cache and Secondary storage set to None so those files remain available through the direct path. Also confirm that the Minecraft container retains its direct-pool mapping if Wings recreates it.

This changes my earlier assessment: the storage access path is now the stronger lead than the CPU. The memory and CPU tests are useful, although they cannot exclude every hardware fault.

The latest diagnostics cover only the reboot after the failure. They do not capture the stalled system, so we cannot yet identify a specific shfs problem.

Latest run was 7 hours and 50 minutes. I turned off the remote sys logs but I will turn them back on and set it up properly. I confirmed that the Minecraft container retained its direct pool mappings.

It seems like 30 minutes after my last reply the server died. The last lock up was 50 minutes and this one was almost 8 hours- I will need to do another run of Chunky to see if my cache pool changes actually changed something. Did it not change anything and after the big changes I made before the 5th lock up, is the server not locking up at a 3 hour interval anymore and is it just "random"? Did my changes to cache pool actually do something and now it takes longer to lock up? Was this lock up for the same reason, was there any evidence this time left behind? Not sure.

Will do some more investigating when the server and KVM is back up. If I don't resolve this issue by the end of tonight, I will reset my Minecraft to remove the additional mods. I had a deadline to have an up and running server by today but pushed it to next Sunday, I cannot afford to wait longer.

Edit (10:02pm EST): Chunky has been running for over 4 hours now, I am thinking that something related to the SSD's is causing these lock ups. I am going to keep investigating... I think if I get Chunky consistently running above 7 hours then the problem is manageable and I will keep investigating. I also attached the diagnostics right after the latest lock up.

tower-diagnostics-20260920-1812.zip

Edited September 21Sep 21 by ijustwantunraidtowork

That changes the earlier assessment: the direct /mnt/cache mappings did not prevent the lockup. The longer runtime alone does not show that the change helped or that an SSD is responsible.

The new diagnostics cover September 20, 17:08–18:12, after reboot. They record an unclean shutdown, but do not contain the log from the freeze. Both cache SSDs report zero media/data-integrity errors, and this session contains no NVMe timeout or Btrfs error. This does not exclude a transient storage problem, but there is no evidence here identifying a faulty SSD.

Before another planned run, configure syslog forwarding to a separate computer and confirm that it receives messages. The remote destination is still empty in this archive. Keep the mappings and other settings unchanged for that comparison. Please provide the failure time and the separate log, even if it ends without an error.

Was XMP disabled during the failed runs and the six-pass memory test? That remains unclear. Passing those tests does not exclude every CPU or memory-stability problem.

If you return to the previous mod setup to meet your deadline, preserve the current world and configuration first. Use a separate instance or a backup from before the mod changes. Avoid loading your only world copy with world-generation mods removed. A successful comparison would narrow the workload trigger, but would not establish why the host freezes.

Just for anyone reading this in the future, I removed the nvidia and IT87 plugins and my issue stopped.