Updated to 6.3.5 and docker tab is empty

[ciarlill]

I just updated to 6.3.5. After a reboot my Docker tab is completely empty. I ssh'd into the server an it looks like dockerd was not running. I tried running it from the command line and reloaded the docker tab. At that point the tab loaded but none of my dockers were listed.

 

Any ideas on what could have happened? What logs can I provide to help sort this out?

[BRiT]

diagnostics

[ciarlill]

I ran 'diagnostics' and got the following in logs/docker.txt (full zip attached):

 

$ tail docker.txt 
time="2017-05-30T16:41:42.206974508-04:00" level=info msg="libcontainerd: new containerd process, pid: 8833" 
time="2017-05-30T16:41:43.612576754-04:00" level=fatal msg="Error starting daemon: write /var/lib/docker/volumes/metadata.db: no space left on device" 
time="2017-05-30T16:41:44.163676548-04:00" level=info msg="libcontainerd: new containerd process, pid: 8997" 
time="2017-05-30T16:41:45.346545311-04:00" level=fatal msg="Error starting daemon: write /var/lib/docker/volumes/metadata.db: no space left on device" 
time="2017-05-30T17:10:32.184898289-04:00" level=info msg="libcontainerd: new containerd process, pid: 22537" 
time="2017-05-30T17:10:33.338567925-04:00" level=fatal msg="Error starting daemon: write /var/lib/docker/volumes/metadata.db: no space left on device" 
time="2017-05-30T17:10:39.085280285-04:00" level=info msg="libcontainerd: new containerd process, pid: 22749" 
time="2017-05-30T17:10:40.270522174-04:00" level=fatal msg="Error starting daemon: write /var/lib/docker/volumes/metadata.db: no space left on device" 
time="2017-05-30T17:10:49.450429338-04:00" level=info msg="libcontainerd: new containerd process, pid: 23006" 
time="2017-05-30T17:10:50.614545274-04:00" level=fatal msg="Error starting daemon: write /var/lib/docker/volumes/metadata.db: no space left on device" 

 

And here is my disk usage:

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            7.6G     0  7.6G   0% /dev
tmpfs           1.6G  9.3M  1.6G   1% /run
/dev/sda1        48G  7.8G   38G  18% /
tmpfs           7.6G  138M  7.5G   2% /dev/shm
tmpfs           5.0M  4.0K  5.0M   1% /run/lock
tmpfs           7.6G     0  7.6G   0% /sys/fs/cgroup
/dev/sda5       129G   30G   92G  25% /home
cgmfs           100K     0  100K   0% /run/cgmanager/fs
tmpfs           1.6G   24K  1.6G   1% /run/user/1000

 

So it looks like the docker image itself is out of space.... but I'm not sure how to expand it.
 

 

lucien-diagnostics-20170530-1713.zip

Edited May 30, 2017 by ciarlill
Attached full diagnostics

[Squid]

But, with 15 Gig for the docker image, increasing the size may buy you a respite, but the root cause is still there.  Why is it filling up in the first place?  Usually, its because of downloads being saved within the image.  Peruse the Temporary Docker FAQ for some suggestions.

[ciarlill]

Thanks for the reply. Is there a way to do investigate and fix from the shell? My docker tab does not load anything below the navigation bar. No options whatsoever so I cannot turn it off in the UI. I also tried running 

du -ah /var/lib/docker/containers/ | grep -v "/$" | sort -rh | head -60

to investigate container log files as a root cause, but I have no "containers" folder under "/var/lib/docker".

 

I have run into this issue of filling up the docker image before and tracked it down to an individual container that I just ended up rebuilding. But in that case the docker daemon was at least still running so I could interact with the containers. If possible I just want to clean up enough space to get it booted again so I can find the cause.

Edited May 31, 2017 by ciarlill

[ciarlill]

Alright I am back up and running. First, it makes sense that /var/lib/docker/containers does not exist... since the daemon couldn't start and mount it. Second, I found the config file for docker in /boot/config and was able to just edit my image size 20G. Now I can figure out the container causing the problem. Thanks

Edited May 31, 2017 by ciarlill

[Meph]

On 5/31/2017 at 2:26 AM, ciarlill said:

Alright I am back up and running. First, it makes sense that /var/lib/docker/containers does not exist... since the daemon couldn't start and mount it. Second, I found the config file for docker in /boot/config and was able to just edit my image size 20G. Now I can figure out the container causing the problem. Thanks

Thank you! I had the same issue here.. Did not know how this docker stuff does work. But I searched the same error message you had on google and found this article..  My docker image size were already at 20. Set it to 25 now and it started up!

 

What is stored in that image file? and how do i access that data? hmmm 
i can see that my dockers (plex and resilio sync) writes to /mnt/user/appdata/ folder.. But that isnt the image file, is it?

 

[Meph]

Found some forum posts about the same issue i had... turned out i had a logfile growing out of control. over 18GB ... i have now cleaned that file... everything sorted