Docker high image disk utilization: why is my docker image/disk getting full?


Recommended Posts

On 9/7/2017 at 8:06 AM, syrys said:

For those that know about docker, is there really no way for one to browse into a docker containers file system and inspect/delete a file? Say you want to locate and delete a log fine manually, and you are sure its just being saved inside the docker, is there really no way to browse into the docker's file system and try locate and remove said file(s)? This would solve most of our problems (although hackey, its still a viable solution).

 

Sure there is, you can "enter" the docker container with this command:

docker exec -i -t CONTAINER_ID /bin/bash

 

Get the CONTAINER_ID from the "docker ps" command

Link to comment
  • 5 months later...

Ok. Been having this problem for a long time but never actually looked into problem. I just solved it by increasing the image size.

 

But with a docker image being almost 60gb its time to fix the problem.

Im Windows guy, having little knowledge in linux commands so i need expert help.

I have the cadvisor container but i cant find anything useful in it.

Somewhere i read that have the mapping /mnt/user is a bad thing...?

 

Can you see what my problem is?

I attached an image of the configuration.

Please let me know if you need anything else.

 

 

 

unraid docker.JPG

Link to comment
1 hour ago, Rivvern said:

I attached an image of the configuration.

As already mentioned in this thread and the Docker FAQ, in addition to having the container mappings correct, you must check that each application is only writing into folders that are mapped; i.e., on the left side of those mappings you posted.

 

Here is the Docker FAQ, see the section about Docker Image Filling Up

 

https://lime-technology.com/forums/topic/57181-real-docker-faq/

 

 

  • Like 1
  • Upvote 1
Link to comment

I've recently been getting warnings also... and it would appear that, at least in my case, btrfs is the culprit.

 

root@UNRAID:~# du -h -d 1 /var/lib/docker/
16M     /var/lib/docker/containers
85G     /var/lib/docker/btrfs
26M     /var/lib/docker/image
46M     /var/lib/docker/volumes
0       /var/lib/docker/trust
88K     /var/lib/docker/network
0       /var/lib/docker/swarm
2.4M    /var/lib/docker/unraid
0       /var/lib/docker/tmp
0       /var/lib/docker/plugins
16K     /var/lib/docker/builder
85G     /var/lib/docker/

 

85GB for....... what exactly?

 

I've seen a number of sites talking about this issue and it seems that most people end up rebuilding their dockers in order to reduce the size....... I'm not interested in doing that, at all.

 

https://github.com/moby/moby/issues/27653

https://gist.github.com/hopeseekr/cd2058e71d01deca5bae9f4e5a555440

https://github.com/moby/moby/issues/9939

 

Short of simply increasing the size of the docker as @Squid has outlined, is there any way to non-destructively "refresh" btrfs to reduce it's size?

 

 

 

 

Link to comment
1 hour ago, Squid said:

Most causes of docker.img getting that big is either 

  • out of control logging by the apps
  • downloading directly into the image

Both circumstances are covered in the docker FAQ

 

As far as I can tell neither of those situations apply to me.  Any docker app that has a directory to configure is mapped to a share.  I don't think there's anything downloading to the image.

 

Logs...   

du -ah /var/lib/docker/containers/ | grep -v "/$" | sort -rh | head -60

Returns about 20MB of logs.

 

Not sure what else to look at?

 

Edit -- I examined /var/lib/docker/btrfs/subvolumes and there are  167 directories, far more than apps/dockers I've got installed if that matters... some which are quite large.  Example:

1.2G  60ba2059df9b6ee30f3dac1084f7041bc5ff0018c77d34cef3d71aede6ba7076
1.2G  fd00c8da84099fb7c380e06ff82feccdc87862706ef8b371a86c5afaf7e66617
985M  bcd719fbcb62e15f4bb96fe56b441d005a55dede653fd49cfe9b4400f4e482f8
985M c6658bfc2643ad274c31a16229f8bb77edf257ccbd712635cd87c7f325752163

 

So how do I go about figuring out what is what and cleaning some of it up? I can't possibly need all of this to be here....  167? I've got 14 dockers and 4 VMs.. seems excessive.

Edited by Energen
Link to comment
12 hours ago, bonienl said:

 

This is the size of your Docker image file (80G).

 

How much percentage is used as reported on the Dashboard?

 

image.png.95fd3fac948b82f8dc3706965db3c051.png

 

It's currently 37%, after I deleted 2 dockers (Krusader and Dolphin) which also brought my btrfs subvolmes size down to 54GB.

 

The warnings I had been getting were docker image utilization over 70%, as high as 85% at one point.

Link to comment
On 3/18/2018 at 5:41 PM, Energen said:

Any docker app that has a directory to configure is mapped to a share.

That's only partially relevant. What can happen is this. Say you have /data mapped to /mnt/user/downloads. So far, so good. But in the app itself, instead of working totally in /data, the app has a folder pointing to /downloads or something. Anything NOT saved to the mapped folder INSIDE the docker app will download into the docker image itself.

  • Upvote 1
Link to comment
On 3/19/2018 at 11:50 AM, Energen said:

It's currently 37%, after I deleted 2 dockers (Krusader and Dolphin) which also brought my btrfs subvolmes size down to 54GB.

 

Freeing up so much space when deleting the containers means almost certainly that data was stored inside the container and not into a mapped folder.

 

Like @jonathanm says, you need to carefully check the applications settings itself to make sure it is wrtining to the correct (mapped) destination.

  • Upvote 1
Link to comment
  • 2 weeks later...

I went from 79%, found this thread,

 

Did a few commands to see the sizes of my container folders

 

refreshed, 83%,

 

Refreshed 89%.

 

 

Wow. Where do I start?

 

du -h -d 1 /var/lib/docker/
7.4G    /var/lib/docker/containers
69G     /var/lib/docker/btrfs
35M     /var/lib/docker/image
52K     /var/lib/docker/volumes
0       /var/lib/docker/trust
108K    /var/lib/docker/network
0       /var/lib/docker/swarm
2.5M    /var/lib/docker/unraid
0       /var/lib/docker/plugins
16K     /var/lib/docker/builder
0       /var/lib/docker/tmp
76G     /var/lib/docker/

 

Edited by Nyghthawk
Link to comment
40 minutes ago, Nyghthawk said:

refreshed, 83%,

 

Refreshed 89%.

Assuming the refreshes were in fairly quick succession.  In that case, either you're downloading to the docker.img, or an app has out of control logging enabled (usually referred to within the app as "debug".  Couple of entries in the docker FAQ about all this.

Link to comment
  • 2 weeks later...
  • 4 weeks later...
On 4/11/2018 at 10:23 AM, vizi95 said:

I had the same issue.  It has to do with docker log file sizes.


Just run this and your utilization will come down a lot:

 

truncate -s 0 /var/lib/docker/containers/*/*-json.log

 

It's safe to use while the docker is still running.

 

 

This will truncate the docker logs, but just to be certain, data of my individual docker containers aren't affected? I'm not sure if any container logs actual data to the -json.log files.

Edited by Blaze9
  • Like 1
Link to comment
  • 2 weeks later...
  • 1 month later...

I just started getting warnings about high docker.img utilization and found this thread.  The problem is that I have no idea which container might be the culprit.  My docker.img is 24G.  I have 9 containers installed and "docker ps -s" shows usage of no more than 50MB (5GB virtual).  I have bashed into each running docker and run "du -h -d 1 -x /" and I am getting sizes that all agree with the virtual usage from the "docker ps -s" command.  I am getting warnings for about 75% usage, which would be about 16GB, so I have no idea where the other 10+GB of usage is coming from.  I have also run Container Cleanup and have, to the best of my knowledge, no orphaned containers.

 

I would be happy to fix whatever container is causing the problems but I have no ideas at this point how to figure out where that problem is.  Can the docker.img file be mounted somehow to look into it and see what the actual usage is?

 

Help, please!

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.