I was doing some server admin earlier when my CPU spiked to 100% and sat there. It looks like dockerd is the cause, but when I run docker stats nothing stands out - I did kill any containers using any level of substantial CPU, but no change. 

 

I've rebooted, but as soon as docker fires up it again nails at 100%. It has dropped a few times, but the GUI isn't responsive enough for me to shut docker down, get to "Fix Common Problems", logs etc.

 

I was able to generate a diag file though, eventually.

server-diagnostics-20231018-1654.zip

Edited October 18, 20232 yr by Nuuki

22 minutes ago, Nuuki said:

I've rebooted, but as soon as docker fires up it again nails at 100%.

Just the docker service or the containers?

Good question, I could shut each container down in turn I guess. Let me try to do that and see what it does.

I previously tried stopping the array (which is a USB drive) and though CPU dropped, it looks like its been stuck trying to actuallyt stop the array for some time. When I check the logs I see a bunch of these log blocks:

 

Oct 18 17:46:19 Server emhttpd: Unmounting disks...
Oct 18 17:46:19 Server emhttpd: shcmd (2345): /usr/sbin/zpool export cache
Oct 18 17:46:19 Server root: cannot unmount '/var/lib/docker/zfs/graph/f9d3a0840bae10b5cb2ff414bab9a4d047e4acd9101f69ba1566b21406728928-init': unmount failed
Oct 18 17:46:19 Server emhttpd: shcmd (2345): exit status: 1
Oct 18 17:46:19 Server emhttpd: Retry unmounting disk share(s)...

 

@JorgeB I rebooted in the end as I couldn't stop the array. After it came back up I was able to stop all running containers, but dockerd is still pinned at 100% CPU.

Edited October 18, 20232 yr by Nuuki

Disable all containers auto start and reboot, see if the CPU gets pinned with just the docker service, if yes recreate the docker image.

OK I've done that and the CPU is looking normal. Shall I enable containers one by one and see if/when it goes crazy?

 

I did change some container volume mounts earlier, but all seemed fine at the time. It seems feasible that that's caused a problem somewhere I guess.

3 minutes ago, Nuuki said:

Shall I enable containers one by one and see if/when it goes crazy?

Try that.

Well, I've been able to bring all the containers online. 

 

However, whenever I started one the CPU would spike much more than I would expect - over 60%, for around 30 seconds, before dropping back down. If I started 2 or 3 at once it would go much higher. Maybe it always does that, but given that there'd usually be 50 or 60 starting in quick succession, it stood out a bit.

 

I also noted that the ZFS resource meter was usually at 85%+

 

I'd note that I did migrate my cache pool from btrfs to zfs at the weekend, though it went find and not had any issues until today - thought it was worth nothing anyway.

 

So right now its all back up and none of the containers seemed individually problematic, but if I turned container auto-start back on and bounced docker, I suspect I might well see the same symptoms again.

11 hours ago, Nuuki said:

I also noted that the ZFS resource meter was usually at 85%+

This is not a problem, normally should always be at 100%.

 

11 hours ago, Nuuki said:

So right now its all back up and none of the containers seemed individually problematic, but if I turned container auto-start back on and bounced docker, I suspect I might well see the same symptoms again.

Not sure what it could be, are you using a docker folder with zfs?

 

Understood.

 

I'm using a ZFS pool. and for Docker I'm using a folder with ZFS.

 

Its running OK now - the real test will be to re-enable auto-start and bounce docker. I may try that later but I'll see if it runs stably for a day or so before I try that. Thanks for your help.

If the same with autostart enabled try using a docker image instead, zfs folder creates a lot of datasets for each container, and that can affect performance.

Ah right - I switched to a (btrfs) folder about a year ago so as not to worry about sizing, but if performance takes a hit its not a major issue to switch back to an image and just increase the size if needed.