Hello, I've been dealing with intermittent OS crashes for the last 2-3 weeks. Tailscale containers become inaccessible, webui doesn't load locally, sign in on terminal directly from the unraid pc times out etc. After the first couple times this happened, I was able to diagnose it to a potential issue with the USB boot stick due to several "critical medium error, dev sda" messages while booting up.

I ordered a new stick, transferred my config and license over and all was good for about 2 weeks.

Now I'm dealing with the same issue again. This time, I set up a userscript I found on here to collect diagnostics every hour so I would have something fairly recent in case the server crashed again. I'm attaching those diagnostics to this post. Is there anything in there that would explain the frequent crashes?

Happy to provide more info if needed.

ag-tower-diagnostics-20260710-2030.zip

Data Volume monitor spams logs so maybe remove it during troubleshooting.

Nothing striking outside of all of Unraid, plugins and apps being woefully out of date included some deprecated ones, so probably gonna need to work on that first.

Edited July 12Jul 12 by Kilrah

1. You're on a super old/outdated version of unraid. Consider updating. There have been loads of bugfixes from 7.0.0 RC to 7.3.2, I would start with the latest update your license supports.

2. Run unraid in safemode for an extended period with no plugins and see if crashes still occur.

3. Consider running a 24 hour minimum memtest from the boot menu to rule out bad ram

Edited July 13Jul 13 by MowMdown

Thanks, I updated unraid to 7.3.2 and most of my docker containers and all my plug-ins as well. Will wait to see if that fixes things and if not, will try the memtest and safe mode. Will update on this thread, thanks!

Updated unraid to the latest version and most containers and plugins to recent versions within the last 30 days.

Things ran fine without any crashes for a few weeks until today.

I noticed the cpu was at 100% usage and ram was at 97%. I ran df -h and didn't see anything that stood out (rootfs was at 22%). Luckily I'd set up a userscript I found on here to pull diagnostics every hour, so uploading the latest diagnostics here.

Let me know if anything stands out. Thanks!

ag-tower-diagnostics-20260812-1547.zip

This looks to me like it may be a container I/O operation that becomes blocked, post the following output while the problem is occurring:

docker ps --format 'table {{.Names}}\t{{.Status}}'

for c in $(docker ps --format '{{.Names}}'); do

printf '%s ' "$c"

docker top "$c" -eo comm= 2>/dev/null |

awk '$1=="python"{n++} END{print n+0}'

done | sort -k2 -nr | head -20

ps -eo pid,ppid,user,state,wchan:32,rss,etime,comm |

awk 'NR==1 || $4 ~ /^D/'

If a container with hundreds of Python processes is identified, type:

docker inspect --format '{{json .Config.Healthcheck}}' CONTAINER_NAME

docker logs --tail 300 CONTAINER_NAME

I would have them capture that first, then stop the offending container. After reboot, keep it disabled and confirm whether the server remains stable.

CPU usage spiked to 100% again today with RAM at 96%. I tried to run the docker command above but all docker commands were taking forever to complete because the system was resource-starved.

I was eventually able to use AI to diagnose and fix the issue mostly without docker commands. Figured I'd document here for anyone else who might need it.

I ran the following commands to diagnose the issue.

free - h

Which showed total of 31Gi of memory, 29Gi used, 364Mi free, 1.9Gi shared, 3.3Gi buff/cache, and 1.3Gi available. 0B across the board for swap.

Then I ran ps -e -o rss= | awk '{sum+=$1} END {printf "%.1f GiB total process RSS\n",sum/1048576}'

To see how much memory was being used by processes. This returned 7.9 GiB, so it wasn't the culprit.

Then I checked kernel and shared memory usage with

grep -E '^(AnonPages|Shmem|Slab|SReclaimable|SUnreclaim|PageTables|KernelStack|Unevictable|Mlocked):' /proc/meminfo

And saw AnonPages was using about 26.5 GiB of memory. Which apparently points to application/container allocations.

Still couldn't run any docker commands so directly read the per-process anonymous memory with

for p in /proc/[0-9]*; do awk '/^Name:/{n=$2}/^Pid:/{p=$2}/^RssAnon:/{print $2, p, n}' "$p/status" 2>/dev/null; done | sort -nr | head -15

Results didn't show any one container holding significant memory.

Checked the ZFS arc cache next with

awk '$1=="size" {printf "ZFS ARC: %.1f GiB\n",$3/1073741824}' /proc/spl/kstat/zfs/arcstats 2>/dev/null

And that returned 0.0 GiB used as well.

Finally I checked memory usage by control group

find /sys/fs/cgroup -name memory.current -type f -exec sh -c 'v=$(cat "$1"); printf "%12d %s\n" "$v" "$1"' _ {} \; 2>/dev/null | sort -nr | head -15

This finally revealed the PID of a docker container that was using about 21GiB of memory alone.

Figured out which container it was by running

docker ps -a --no-trunc --format '{{.ID}} {{.Names}}' | grep '^PID'

Returned Beaver-Habit-Tracker which I use to track certain habits and trends.

I stopped the container using

docker stop -t 20 Beaver-Habit-Tracker

And then ran free -h again. This time it said 9.2Gi was used, with 21Gi available.

I checked the logs of the offending container but wasn't able to find anything that would account for this memory leak. Either way I added --memory=2g under extra parameters for the container, which I probably should have had in the first place.

Will keep monitoring to make sure it respects the memory limit, but it should I think.

Thanks for the outputs; this identifies the responsible container and lines up closely with the previous diagnostics.

The earlier diagnostic contained 800+ Python processes owned by the same container account, using about 19 GiB in total. Your latest cgroup result attributed about 21 GiB to Beaver Habit Tracker, and stopping

that container immediately released approximately the same amount.

This therefore looks less like one Python process leaking 21 GiB and more like repeated Python healthcheck processes accumulating. Beaver Habit Tracker’s current image defines a Python healthcheck every 30

seconds. The previous Docker log also repeatedly reported that it could not drain container exec processes, which would explain why old healthchecks remained while new ones continued to start.

The 2 GiB limit is a good way to protect the rest of the server, but monitor whether the container becomes unhealthy, is OOM-killed, or repeatedly restarts. While the system is healthy, post:

docker inspect --format 'Image={{.Config.Image}} ImageID={{.Image}} Memory={{.HostConfig.Memory}} PidsLimit={{.HostConfig.PidsLimit}} Healthcheck={{json .Config.Healthcheck}}' Beaver-Habit-Tracker

docker inspect --format 'OOMKilled={{.State.OOMKilled}} RestartCount={{.RestartCount}} Health={{json .State.Health}}' Beaver-Habit-Tracker

docker top Beaver-Habit-Tracker -eo pid,ppid,state,rss,etime,comm,args

If it starts growing again, also capture the container cgroup’s memory.stat, memory.events, pids.current, and pids.max before stopping it.

As an A/B test, you could temporarily add --no-healthcheck to this container while leaving the application otherwise unchanged. If the process and memory growth stops, that would strongly confirm the

healthcheck-exec mechanism. A PID limit based on the container’s normal process count may also contain this failure more directly, but neither limit explains the initial application or I/O stall.