High CPU Usage By Docker Service

[Irithor]

I'm getting huge CPU usage from the docker service causing my system/webgui to lock up.

 

It started right in the middle of normal operation (was renaming some files in Sonarr I think). I cannot attribute it to any specific container as the problem persists even with all containers off. I've tried turning them on/off one at a time but no obvious culprit. If I disable the Docker service entirely though, it seems to go away. The only other change I have made recently is increasing the size of the docker image from 64gb to 96gb.

 

Top is showing Dockerd using around 600%.

 

A couple of orphaned docker images were created around this time but not exactly sure when. I am also noticing irregularly high reads in my cache pool (the one with my appdata and docker img). Is this corruption?

 

Thanks in advance.

unraidserver-diagnostics-20230511-0654.zip

[itimpi]

13 hours ago, Irithor said:

The only other change I have made recently is increasing the size of the docker image from 64gb to 96gb.

I think it is unlikely that you need a docker image file this large unless you have a container mis-configured so that is writing data internally to the image.   Have you had problems with it filling up in the past?   If properly configured then any location within a container where anything other than a trivial amount of data is being written should be mapped to a location on the host external to the image.

 

Whether this is related to your current issue I have no idea but thought it was worth mentioning just in case.

[Irithor]

Thanks for taking a look!

I thought 96gb for the Docker Image was very large also. I have since decided to just go ahead and recreate the image and re-download my containers. This has allowed me to at least have the service up and running with no containers and a clean image, so that's progress. I am now in the process of turning them back on one by one. It having one or some of them auto-start with the array causes this issue and doesn't resolve until a full reboot.

 

I know the Docker Image can fill up when paths are incorrectly configured but I feel I have pretty meticulously checked and re-checked they are all correct.

 

I've attached a report of the Container Sizes. I'm not sure if they will mean anything to you as you might not be familiar with them but appreciate letting me know if anything looks off to you.

Container Sizes.txt

[Daniel Sippel]

I have the same or at least a similar problem. I am on 6.12.0-rc5. I am using a directory instead of a docker image, which is placed on the cache pool (2 nvme drives, raid 1, btrfs). The cache pool is not full and not corrupt. Any suggestions? Tried restarting and stopping individual containers, no change. Please find screenshots and diagnostics attached.

 

Best

 

Daniel

 

server-diagnostics-20230516-1420.zip

[Irithor]

A scrub won't return anything specifically in regards to Docker Image corruption. Just overall corruption I believe.

 

I just ended up re-creating the Docker Image and downloading all the images from scratch. I also believe the culprit may be that the Plex Meta Manager container was creating a memory leak due to trying to restart itself in extremely quick succession without just stopping because of a bad update that was pushed that needed to disable part of it's start up process but didn't account for this looping issue.

 

Between those two things the issue seems to have resolved itself for now.