freddy0

I want to further document what i have found. and add some keywords so that anyone searching for a related topic might find this.
The issue I described is of the type "server out of memory" or short oom. It seems that some service on my host machine is allowed to suck up all of my good ram (64Gb filled). the CPU usage is also high because of the extremely high ram usage. There is a process for ram usage (kswapd0) that is constantly hammering on the cpu cores.
With that i had finally isolated the problem.
Cpu usage extremely high - ram is geting filled up because of heavy io - and the system is unresponsive.

SSH is the only thing that still works to some extend.
At this point i don't know what the source is though. Top sort by memory 
> top m shows that a process called /usr/local/bin/node or kswapd0 is using all of this memory. This unfortunately does not help any further, because i suspect another program to cause extremely heavy io load so that the server cannot keep up and has to store everything in ram.
The Problem is likely to be caused by either a docker container that is misconfigured in some way or a plug-in that does something special in the background.

To find out which docker might cause this problem i run
docker stats --no-stream To only show one final output that i need (it can take a very long time like 4 minutes to get a response, because the server is under heavy load)
Then i search for a container that is misbehaving in such a way, that is utilizes abnormal amounts of ram.
My output of
docker stats --format "table {{.Container}}\t{{.CPUPerc}}\t{{.MemUsage}}" --no-stream CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS d4e7fd9e1f9f dashy 7.24% 49.56GiB / 62.73GiB 79.00% 44.4GB / 73.3MB 0B / 0B 22 9d7aa6b28199 xxxxxxx 1.47% 52.42MiB / 62.73GiB 0.08% 82.1MB / 8.88MB 0B / 0B 36 17426e04df93 xxxxxxxxxxx 0.10% 9.895MiB / 62.73GiB 0.02% 129MB / 97.7MB 0B / 0B 6 5849332c035d xxxxxxxxxx 0.00% 40.43MiB / 62.73GiB 0.06% 2.59MB / 76.4kB 0B / 0B 12 3168dd810eb9 xxxxxxxxxxxxxxxxxxxx 0.02% 21.53MiB / 62.73GiB 0.03% 539MB / 1.78MB 0B / 0B 9 0b15f326c902 xxxxxxxxxxxxx 45.01% 230.1MiB / 62.73GiB 0.36% 260MB / 103MB 0B / 0B 22 4c04ff55ae43 xxxxxxxxxxxxxxx 0.12% 8.609MiB / 62.73GiB 0.01% 99.3MB / 90.2MB 0B / 0B 6 8e8b98f025c9 xxxxxxxxxxxxxxx 0.08% 9.469MiB / 62.73GiB 0.01% 135MB / 66.4MB 0B / 0B 6 ed9d251c5f8a xxxxxxxxxxxxxxx 0.35% 10.41MiB / 62.73GiB 0.02% 219MB / 138MB 0B / 0B 7 124b0e9b6659 xxxxxxxxxxxxxxxxx 2.03% 178.9MiB / 62.73GiB 0.28% 472MB / 139GB 0B / 0B 42 7af51b6011e3 xxxxxxxxxxxxxx 0.00% 16.75MiB / 62.73GiB 0.03% 2.59MB / 21.8MB 0B / 0B 25 f503e0a68d75 xxxxxxxxxxx 0.89% 66.32MiB / 62.73GiB 0.10% 3.09MB / 338kB 0B / 0B 7 0162686fcb7f xxxxxxxxxxxxxxx 0.09% 597.2MiB / 62.73GiB 0.93% 2.56MB / 39.8kB 0B / 0B 69 ffb7f7b840b4 xxxxxxx 0.04% 313.8MiB / 62.73GiB 0.49% 2.57MB / 861kB 0B / 0B 56 274f9b5071e0 xxxxxxxxxxxxxxxxx 3.87% 127.4MiB / 62.73GiB 0.20% 24.6MB / 8.31MB 0B / 0B 41
As you can see the docker Dashy is hammering on my IO and MEM usage quite intensively with 50gb ram and 44gb net IO. Even though this is a simple dashboard application.
I suspect this to be related to a toggle option in dashy that lets the server handle all the uptime pings for a given service instead of the client (which i toggled to be on)
I could either remove the docker or simply toggle this to let the client handle all the uptime pings again. This is a hypothesis though.

The problem ultimately is Dashy and i will try to narrow down the problem even further.

Some other related Reddit / forum posts i found to be somewhat related:
Unraid 6.8.3 OOM error on Large file transferr: https://www.reddit.com/r/unRAID/comments/ig5xyy/unraid_683_oom_error/
Heavy Disk IO:
Some application generating/checking data all the time (interval extremely short)
https://www.reddit.com/r/unRAID/comments/gip2tv/high_cpu_usage_all_of_a_sudden/
 
Documentation references
Docker stats: https://docs.docker.com/engine/reference/commandline/stats/