System always hanging when Docker is enabled

[ipreferpie]

I've been using unRaid since 6.1 with no many issues and am running 6.8.1 now. But for the last 6 weeks, I've been encountering constant hanging. I noticed that this happens only when Docker is enabled. Without it enabled, there's no problem, but after having Docker running and some containers running (i.e. sabNZBD, Plex, Sonarr, influx, etc.) the system would start having a super high CPU utilization and then would start hanging. This usually takes anywhere from 2-36 hours of running the Docker containers.

 

I would then try to stop processes that are on docker using "kill -9". After that, I'd try to "umount -l /mnt/cache" which would cause the shell to hang. If I skip these steps and try "poweroff" or "powerdown" the script would run, but get stuck at "Forcing shutdown..." after waiting 90 sec for the graceful shutdown option. I have to physically power off and start again.

 

I run a parity check which usually shows 0 errors. But after startup up some containers, the problems persist.

 

I tried checking the files on my /mnt/cache and try moving or syncing files on Krusader. It shows me a small number of /appdata files are corrupt (influx/wal etc which I wasn't even running). So I copied those that could be read and reformatted the cache drives and moved the files back and also deleting/rebuilding docker.img. But the problem still persists. I'm really frustrated at the problem and can't come close to any solution. I think likely it's something happening in the docker containers but don't know what. Can anyone help me on this please? I've attached my diagnostics FYI. Many thanks.

hk-homelab-diagnostics-20200125-2100.zip

Edited January 25, 2020 by ipreferpie

[EgyptianSnakeLegs]

Wow, there are a bunch of us describing very similar problems lately.  I wonder if it's all related to some bug in 6.7.x & 6.8.x?  I thought mine was Ryzen related, and have been troubleshooting it for a couple months with no success.  I just read another post where a guy had the same problem, but thinks he solved it by deleting and rebuilding his docker image.  I think that's going to be my next attempt.  I currently only have Plex and Tautulli running, but have installed and removed a bunch of different docker containers that I was experimenting with.  Maybe one of them broke something.  I'll let you know if I have any luck.

[ipreferpie]

7 hours ago, EgyptianSnakeLegs said:

Wow, there are a bunch of us describing very similar problems lately.  I wonder if it's all related to some bug in 6.7.x & 6.8.x?  I thought mine was Ryzen related, and have been troubleshooting it for a couple months with no success.  I just read another post where a guy had the same problem, but thinks he solved it by deleting and rebuilding his docker image.  I think that's going to be my next attempt.  I currently only have Plex and Tautulli running, but have installed and removed a bunch of different docker containers that I was experimenting with.  Maybe one of them broke something.  I'll let you know if I have any luck.

Yeah, it really started after upgrading to 6.8.0. The thing is that I rebuilt docker.img multiple times to no avail. I think it's too late to roll back to 6.7.x no that I upgraded twice. The likelihood and immediacy of unRAID hanging is dependent on which docker containers I'm running, I feel. Not proven yet, but I have to turn on one container at a time to run for 2-4 days to ensure it's really working and then add another one. I'm starting to do that now, but it's slow going and not really targetted. I hope someone here w/ more experience can look into the diagnostics and see what's happening.

[EgyptianSnakeLegs]

I remade my docker image last night, and also deleted many (most) of the corresponding app data folders.  All except my Plex library config/metadata.  It really seems to have solved 99% of my issues.