Recently (around 1 week ago) I noticed my docker image started to grow. After some reading I found typical causes to be writing files to the image instead of to a mapped path, so I started to look into that route. I however wasn't able to find anything.
Later I decided to change the docker image size since it was approaching 100% and I didn't want to fill it up ( I hadn't found the root cause of the growing image size, however I also did not want everything to go offline as I needed some services available)
after changing the docker image size (increasing) the docker service failed to start throwing these PHP errors
Warning: stream_socket_client(): unable to connect to unix:///var/run/docker.sock (No such file or directory) in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 712
Couldn't create socket: [2] No such file or directory
Warning: Invalid argument supplied for foreach() in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 898
ok, I have seen mention of this on the forums before, seems like a corrupted docker image. no big deal, I deleted the image, rebuilt it, and this time I didn't install about 1/2 of the previous dockers that were not as important in an attempt to reduce the image size and find the offending container that was causing the image to grow.
after getting everything going again, I logged in this morning only to find websites failing, and containers not responding. I saw the container was showing ~85% usage of allocated space (30GB) so I went to the docker page and calculated the image size, unfortunately I didn't take a screenshot, so I don't remember the size for the logs/writable portions, however I know the total container size was showing as 30.5GB. Something isn't adding up, the dashboard is reporting ~85% but the calculated size is well over the allocated size.
I shut down the docker service, went to change the image size again to increase to around 60GB (the previous size I had used for years with no problems) and now the docker service is failing to start again with the same PHP errors.
I saw that issues like this can arise from full drives, but everything is reporting with 100s of GB of free space on every drive, with the cache drive have ~500GB free.
In summary:
Docker image growing
Resized image
Docker service failed to start
rebuilt image with less containers
docker image filled up again
resized image
docker service failed to start again.
Thanks for any help!
gary-diagnostics-20230421-1032.zip