anubhavrb

CPU usage spiked to 100% again today with RAM at 96%. I tried to run the docker command above but all docker commands were taking forever to complete because the system was resource-starved. I was eventually able to use AI to diagnose and fix the issue mostly without docker commands. Figured I'd document here for anyone else who might need it. I ran the following commands to diagnose the issue. free - h Which showed total of 31Gi of memory, 29Gi used, 364Mi free, 1.9Gi shared, 3.3Gi buff/cache, and 1.3Gi available. 0B across the board for swap. Then I ran ps -e -o rss= | awk '{sum+=$1} END {printf "%.1f GiB total process RSS\n",sum/1048576}' To see how much memory was being used by processes. This returned 7.9 GiB, so it wasn't the culprit. Then I checked kernel and shared memory usage with grep -E '^(AnonPages|Shmem|Slab|SReclaimable|SUnreclaim|PageTables|KernelStack|Unevictable|Mlocked):' /proc/meminfo And saw AnonPages was using about 26.5 GiB of memory. Which apparently points to application/container allocations. Still couldn't run any docker commands so directly read the per-process anonymous memory with for p in /proc/[0-9]*; do awk '/^Name:/{n=$2}/^Pid:/{p=$2}/^RssAnon:/{print $2, p, n}' "$p/status" 2>/dev/null; done | sort -nr | head -15 Results didn't show any one container holding significant memory. Checked the ZFS arc cache next with awk '$1=="size" {printf "ZFS ARC: %.1f GiB\n",$3/1073741824}' /proc/spl/kstat/zfs/arcstats 2>/dev/null And that returned 0.0 GiB used as well. Finally I checked memory usage by control group find /sys/fs/cgroup -name memory.current -type f -exec sh -c 'v=$(cat "$1"); printf "%12d %s\n" "$v" "$1"' _ {} \; 2>/dev/null | sort -nr | head -15 This finally revealed the PID of a docker container that was using about 21GiB of memory alone. Figured out which container it was by running docker ps -a --no-trunc --format '{{.ID}} {{.Names}}' | grep '^PID' Returned Beaver-Habit-Tracker which I use to track certain habits and trends. I stopped the container using docker stop -t 20 Beaver-Habit-Tracker And then ran free -h again. This time it said 9.2Gi was used, with 21Gi available. I checked the logs of the offending container but wasn't able to find anything that would account for this memory leak. Either way I added --memory=2g under extra parameters for the container, which I probably should have had in the first place. Will keep monitoring to make sure it respects the memory limit, but it should I think.

Updated unraid to the latest version and most containers and plugins to recent versions within the last 30 days. Things ran fine without any crashes for a few weeks until today. I noticed the cpu was at 100% usage and ram was at 97%. I ran df -h and didn't see anything that stood out (rootfs was at 22%). Luckily I'd set up a userscript I found on here to pull diagnostics every hour, so uploading the latest diagnostics here. Let me know if anything stands out. Thanks! ag-tower-diagnostics-20260812-1547.zip

Thanks, I updated unraid to 7.3.2 and most of my docker containers and all my plug-ins as well. Will wait to see if that fixes things and if not, will try the memtest and safe mode. Will update on this thread, thanks!

Hello, I've been dealing with intermittent OS crashes for the last 2-3 weeks. Tailscale containers become inaccessible, webui doesn't load locally, sign in on terminal directly from the unraid pc times out etc. After the first couple times this happened, I was able to diagnose it to a potential issue with the USB boot stick due to several "critical medium error, dev sda" messages while booting up. I ordered a new stick, transferred my config and license over and all was good for about 2 weeks. Now I'm dealing with the same issue again. This time, I set up a userscript I found on here to collect diagnostics every hour so I would have something fairly recent in case the server crashed again. I'm attaching those diagnostics to this post. Is there anything in there that would explain the frequent crashes? Happy to provide more info if needed. ag-tower-diagnostics-20260710-2030.zip

Yep, I was able to get back into my server. I posted my docker mappings above, do any of those look like they may be writing to rootfs instead of to storage?

Was able to physically attach a keyboard to my server and use powerdown and restart. Now running df -h has the rootfs at 6% full, but I know it'll fill back up unless I find out what's causing the issue. Going through my docker mappings and these are the only ones that aren't a subdirectory of /mnt/user: Plex has a /dev/dri mapping for the iGPU Telegraf has a few suspect mappings( host/path -> containter/path ): /var/run/utmp -> /var/run/utmp /var/run/docker.sock -> /var/run/docker.sock / -> /rootfs /sys -> /rootfs/sys /etc -> /rootfs/etc /proc -> /rootfs/proc /run/udev -> /run/udev /rootfs/proc set as container variable HOST_PROC /rootfs/sys set as container variable HOST_SYS /rootfs/etc set as container variable HOST_ETC /rootfs set as container variable HOST_MOUNT_PREFIX UptimeKuma also has the /var/run/docker.sock mapping code-server has a couple: /config/workspace set as container variable DEFAULT_WORKSPACE /mnt/ -> /array TheLounge has /mnt/cache/appdata/thelounge -> /thelounge Do any of these look suspect to you?

Ah man, my server's web gui is now not working (500 Internal Server Error). I can still access the web ui for my docker containers so looks like things are still working, but I need to figure out how to restart the server without the web gui. And I can then look around my docker mappings to see what the root cause is.

Gotcha, I'll take a look at my mappings. I'm not too familiar with how to identify if something is writing to ram though. What kind of patterns or mappings should I be looking for?

Fix common problems flagged this issue recently. I haven't noticed any issues on my server, everything seems to be running smoothly, but thought I'd see if this is something to be concerned about / what I could do to mitigate it. I don't know enough to know whether this will eventually crash my server, but I suspect it might? I've attached my server diagnostics to this post, and this is the output when I run df -h: Filesystem Size Used Avail Use% Mounted on rootfs 16G 16G 162M 99% / tmpfs 128M 3.2M 125M 3% /run /dev/sda1 15G 1.7G 13G 12% /boot overlay 16G 16G 162M 99% /usr overlay 16G 16G 162M 99% /lib tmpfs 128M 14M 115M 11% /var/log devtmpfs 8.0M 0 8.0M 0% /dev tmpfs 16G 0 16G 0% /dev/shm efivarfs 192K 144K 44K 77% /sys/firmware/efi/efivars tmpfs 1.0M 0 1.0M 0% /mnt/disks tmpfs 1.0M 0 1.0M 0% /mnt/remotes tmpfs 1.0M 0 1.0M 0% /mnt/addons tmpfs 1.0M 0 1.0M 0% /mnt/rootshare /dev/md1p1 19T 16T 2.4T 87% /mnt/disk1 /dev/md2p1 15T 11T 4.6T 69% /mnt/disk2 /dev/md3p1 15T 11T 4.6T 69% /mnt/disk3 /dev/md4p1 19T 11T 7.3T 60% /mnt/disk4 /dev/nvme0n1p1 1.9T 89G 1.8T 5% /mnt/cache shfs 66T 47T 19T 72% /mnt/user0 shfs 66T 47T 19T 72% /mnt/user /dev/loop2 50G 30G 20G 60% /var/lib/docker /dev/loop3 1.0G 6.4M 905M 1% /etc/libvirt tmpfs 3.2G 0 3.2G 0% /run/user/0 tower-diagnostics-20250907-2313.zip