Recently (around 1 week ago) I noticed my docker image started to grow. After some reading I found typical causes to be writing files to the image instead of to a mapped path, so I started to look into that route. I however wasn't able to find anything.

Later I decided to change the docker image size since it was approaching 100% and I didn't want to fill it up ( I hadn't found the root cause of the growing image size, however I also did not want everything to go offline as I needed some services available)

after changing the docker image size (increasing) the docker service failed to start throwing these PHP errors

Warning: stream_socket_client(): unable to connect to unix:///var/run/docker.sock (No such file or directory) in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 712
Couldn't create socket: [2] No such file or directory
Warning: Invalid argument supplied for foreach() in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 898

 

ok, I have seen mention of this on the forums before, seems like a corrupted docker image. no big deal, I deleted the image, rebuilt it, and this time I didn't install about 1/2 of the previous dockers that were not as important in an attempt to reduce the image size and find the offending container that was causing the image to grow.

after getting everything going again, I logged in this morning only to find websites failing, and containers not responding. I saw the container was showing ~85% usage of allocated space (30GB) so I went to the docker page and calculated the image size, unfortunately I didn't take a screenshot, so I don't remember the size for the logs/writable portions, however I know the total container size was showing as 30.5GB. Something isn't adding up, the dashboard is reporting ~85% but the calculated size is well over the allocated size.

I shut down the docker service, went to change the image size again to increase to around 60GB (the previous size I had used for years with no problems) and now the docker service is failing to start again with the same PHP errors.

I saw that issues like this can arise from full drives, but everything is reporting with 100s of GB of free space on every drive, with the cache drive have ~500GB free.

In summary:

Docker image growing

Resized image
Docker service failed to start

rebuilt image with less containers

docker image filled up again

resized image

docker service failed to start again.

 

Thanks for any help!

gary-diagnostics-20230421-1032.zip

Reboot and post new diags after array start.

Sorry for the delay, got caught up with work.

After restarting so far it seems like everything is running normally.
 

Attached are the updated diagnostics.

Do you know what may be happening that would require a reboot. normally I am able to leave the server on indefinitely, usually only shutting down when I need to do maintenance, or the power will be out.

gary-diagnostics-20230421-1740.zip

I suspected it would help since before the docker image was failing to mount because it was in use, that means it failed to unmount at array stop, things look normal for now, post new diags it it happens again.