Jump to content
syrys

Docker high image disk utilization: why is my docker image/disk getting full?

111 posts in this topic Last Reply

Recommended Posts

On 9/7/2017 at 8:06 AM, syrys said:

For those that know about docker, is there really no way for one to browse into a docker containers file system and inspect/delete a file? Say you want to locate and delete a log fine manually, and you are sure its just being saved inside the docker, is there really no way to browse into the docker's file system and try locate and remove said file(s)? This would solve most of our problems (although hackey, its still a viable solution).

 

Sure there is, you can "enter" the docker container with this command:

docker exec -i -t CONTAINER_ID /bin/bash

 

Get the CONTAINER_ID from the "docker ps" command

Share this post


Link to post

Ok. Been having this problem for a long time but never actually looked into problem. I just solved it by increasing the image size.

 

But with a docker image being almost 60gb its time to fix the problem.

Im Windows guy, having little knowledge in linux commands so i need expert help.

I have the cadvisor container but i cant find anything useful in it.

Somewhere i read that have the mapping /mnt/user is a bad thing...?

 

Can you see what my problem is?

I attached an image of the configuration.

Please let me know if you need anything else.

 

 

 

unraid docker.JPG

Share this post


Link to post
1 hour ago, Rivvern said:

I attached an image of the configuration.

As already mentioned in this thread and the Docker FAQ, in addition to having the container mappings correct, you must check that each application is only writing into folders that are mapped; i.e., on the left side of those mappings you posted.

 

Here is the Docker FAQ, see the section about Docker Image Filling Up

 

https://lime-technology.com/forums/topic/57181-real-docker-faq/

 

 

Share this post


Link to post

I had a problem with sync, it filled the disk with a log file (80gb+) i was located in /var/lib/docker/containers/Some"Random"FolderNameofHEX/SameNameAsBefore.json try look for that

Edited by Mikki

Share this post


Link to post
5 hours ago, Rivvern said:

Why do you think i had so many orphaned images?

Orphaned images only tend to happen when a container fails to install and run (ie: conflicting port, etc).

Share this post


Link to post
26 minutes ago, Rivvern said:

Ok so it seems like i keep getting orphaned images.

Uninstall advanced buttons plugin 

Share this post


Link to post
1 hour ago, Squid said:

Uninstall advanced buttons plugin 

Thanks, but ive never had that one installed afaik. :/

Share this post


Link to post

I've recently been getting warnings also... and it would appear that, at least in my case, btrfs is the culprit.

 

root@UNRAID:~# du -h -d 1 /var/lib/docker/
16M     /var/lib/docker/containers
85G     /var/lib/docker/btrfs
26M     /var/lib/docker/image
46M     /var/lib/docker/volumes
0       /var/lib/docker/trust
88K     /var/lib/docker/network
0       /var/lib/docker/swarm
2.4M    /var/lib/docker/unraid
0       /var/lib/docker/tmp
0       /var/lib/docker/plugins
16K     /var/lib/docker/builder
85G     /var/lib/docker/

 

85GB for....... what exactly?

 

I've seen a number of sites talking about this issue and it seems that most people end up rebuilding their dockers in order to reduce the size....... I'm not interested in doing that, at all.

 

https://github.com/moby/moby/issues/27653

https://gist.github.com/hopeseekr/cd2058e71d01deca5bae9f4e5a555440

https://github.com/moby/moby/issues/9939

 

Short of simply increasing the size of the docker as @Squid has outlined, is there any way to non-destructively "refresh" btrfs to reduce it's size?

 

 

 

 

Share this post


Link to post

Most causes of docker.img getting that big is either 

  • out of control logging by the apps
  • downloading directly into the image

Both circumstances are covered in the docker FAQ

Share this post


Link to post
1 hour ago, Squid said:

Most causes of docker.img getting that big is either 

  • out of control logging by the apps
  • downloading directly into the image

Both circumstances are covered in the docker FAQ

 

As far as I can tell neither of those situations apply to me.  Any docker app that has a directory to configure is mapped to a share.  I don't think there's anything downloading to the image.

 

Logs...   

du -ah /var/lib/docker/containers/ | grep -v "/$" | sort -rh | head -60

Returns about 20MB of logs.

 

Not sure what else to look at?

 

Edit -- I examined /var/lib/docker/btrfs/subvolumes and there are  167 directories, far more than apps/dockers I've got installed if that matters... some which are quite large.  Example:

1.2G  60ba2059df9b6ee30f3dac1084f7041bc5ff0018c77d34cef3d71aede6ba7076
1.2G  fd00c8da84099fb7c380e06ff82feccdc87862706ef8b371a86c5afaf7e66617
985M  bcd719fbcb62e15f4bb96fe56b441d005a55dede653fd49cfe9b4400f4e482f8
985M c6658bfc2643ad274c31a16229f8bb77edf257ccbd712635cd87c7f325752163

 

So how do I go about figuring out what is what and cleaning some of it up? I can't possibly need all of this to be here....  167? I've got 14 dockers and 4 VMs.. seems excessive.

Edited by Energen

Share this post


Link to post
2 hours ago, Energen said:

85G     /var/lib/docker/btrfs

 

This is the size of your Docker image file (80G).

 

How much percentage is used as reported on the Dashboard?

 

image.png.95fd3fac948b82f8dc3706965db3c051.png

Share this post


Link to post
12 hours ago, bonienl said:

 

This is the size of your Docker image file (80G).

 

How much percentage is used as reported on the Dashboard?

 

image.png.95fd3fac948b82f8dc3706965db3c051.png

 

It's currently 37%, after I deleted 2 dockers (Krusader and Dolphin) which also brought my btrfs subvolmes size down to 54GB.

 

The warnings I had been getting were docker image utilization over 70%, as high as 85% at one point.

Share this post


Link to post

Not sure what but you definitely have something not working normally. Most people never get that warning with only 20GB docker image.

Share this post


Link to post
On 3/18/2018 at 5:41 PM, Energen said:

Any docker app that has a directory to configure is mapped to a share.

That's only partially relevant. What can happen is this. Say you have /data mapped to /mnt/user/downloads. So far, so good. But in the app itself, instead of working totally in /data, the app has a folder pointing to /downloads or something. Anything NOT saved to the mapped folder INSIDE the docker app will download into the docker image itself.

Share this post


Link to post
On 3/19/2018 at 11:50 AM, Energen said:

It's currently 37%, after I deleted 2 dockers (Krusader and Dolphin) which also brought my btrfs subvolmes size down to 54GB.

 

Freeing up so much space when deleting the containers means almost certainly that data was stored inside the container and not into a mapped folder.

 

Like @jonathanm says, you need to carefully check the applications settings itself to make sure it is wrtining to the correct (mapped) destination.

Share this post


Link to post

I went from 79%, found this thread,

 

Did a few commands to see the sizes of my container folders

 

refreshed, 83%,

 

Refreshed 89%.

 

 

Wow. Where do I start?

 

du -h -d 1 /var/lib/docker/
7.4G    /var/lib/docker/containers
69G     /var/lib/docker/btrfs
35M     /var/lib/docker/image
52K     /var/lib/docker/volumes
0       /var/lib/docker/trust
108K    /var/lib/docker/network
0       /var/lib/docker/swarm
2.5M    /var/lib/docker/unraid
0       /var/lib/docker/plugins
16K     /var/lib/docker/builder
0       /var/lib/docker/tmp
76G     /var/lib/docker/

 

Edited by Nyghthawk

Share this post


Link to post
40 minutes ago, Nyghthawk said:

refreshed, 83%,

 

Refreshed 89%.

Assuming the refreshes were in fairly quick succession.  In that case, either you're downloading to the docker.img, or an app has out of control logging enabled (usually referred to within the app as "debug".  Couple of entries in the docker FAQ about all this.

Share this post


Link to post

I had the same issue.  It has to do with docker log file sizes.


Just run this and your utilization will come down a lot:

 

truncate -s 0 /var/lib/docker/containers/*/*-json.log

 

It's safe to use while the docker is still running.

 

Share this post


Link to post
On 4/11/2018 at 10:23 AM, vizi95 said:

I had the same issue.  It has to do with docker log file sizes.


Just run this and your utilization will come down a lot:

 

truncate -s 0 /var/lib/docker/containers/*/*-json.log

 

It's safe to use while the docker is still running.

 

 

This will truncate the docker logs, but just to be certain, data of my individual docker containers aren't affected? I'm not sure if any container logs actual data to the -json.log files.

Edited by Blaze9

Share this post


Link to post

The latest unRAID 6.5.2 release has an option to open a command line shell within a running docker container.     This might allow you to poke around inside each docker to try and work out what is using up the space.

Share this post


Link to post

I just started getting warnings about high docker.img utilization and found this thread.  The problem is that I have no idea which container might be the culprit.  My docker.img is 24G.  I have 9 containers installed and "docker ps -s" shows usage of no more than 50MB (5GB virtual).  I have bashed into each running docker and run "du -h -d 1 -x /" and I am getting sizes that all agree with the virtual usage from the "docker ps -s" command.  I am getting warnings for about 75% usage, which would be about 16GB, so I have no idea where the other 10+GB of usage is coming from.  I have also run Container Cleanup and have, to the best of my knowledge, no orphaned containers.

 

I would be happy to fix whatever container is causing the problems but I have no ideas at this point how to figure out where that problem is.  Can the docker.img file be mounted somehow to look into it and see what the actual usage is?

 

Help, please!

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.