Docker high image disk utilization: why is my docker image/disk getting full?


Recommended Posts

46 minutes ago, DerfMcDoogal said:

Thank you!  This solved my issue.

I was doing a lot of operations in Plex which I believe is causing all sorts of logging.  I got the warning about 71% usage and by the time I went to investigate it was at 78% and by the time I found the thread and read to page 2 it was 97% and I had to stop all of my containers.  This immediately dropped my usage down to 28%.  All I can think is that Plex is logging and that those logs are normally trimmed every night but since I was doing a lot of operations it spun out of control before the log could be trimmed?

 

No idea.  It was increasing so fast I didn't have time to mess around.

FYI, that command won't "solve" anything, but rather delays it from happening again.  The solution is this

https://forums.unraid.net/topic/57181-real-docker-faq/#comment-564326

 

  • Upvote 1
Link to comment
  • 2 months later...
On 4/11/2018 at 10:23 AM, vizi95 said:

I had the same issue.  It has to do with docker log file sizes.


Just run this and your utilization will come down a lot:

 

truncate -s 0 /var/lib/docker/containers/*/*-json.log

 

It's safe to use while the docker is still running.

 

Aw man, this saved me. I had a docker with a port binding issue, I guess it was filling logs constantly, but after repairing the bind problem, i didn't know what was eating up the docker image file... logs. *shakes fists into air*

Link to comment
Aw man, this saved me. I had a docker with a port binding issue, I guess it was filling logs constantly, but after repairing the bind problem, i didn't know what was eating up the docker image file... logs. *shakes fists into air*
Good to see you back PhAZe

But, you are better off adding the settings into extra params or nuking the image and setting up log rotation

The commands listed is a temporary solution

Sent from my NSA monitored device

Link to comment

Don't know if this will help or not. I had 85% usage a week back. After looking around I found it was Plex sync. My wife had friend's Blu-ray complete series synced to her tablet. Once I deleted the synced files everything returned to normal. I think maybe it caused problems because her tablet storage was full. I honestly don't know. It also filled up my cache drive in the process.

Sent from my Moto Z (2) using Tapatalk

Link to comment
On 6/19/2019 at 1:37 PM, Squid said:

Good to see you back PhAZe

But, you are better off adding the settings into extra params or nuking the image and setting up log rotation

The commands listed is a temporary solution

Sent from my NSA monitored device
 

Yea, this was more to prove that I had the correct docker as the culprit. I just nuked the image and set an extra param path. 

Link to comment
Just now, PhAzE said:

Yea, this was more to prove that I had the correct docker as the culprit. I just nuked the image and set an extra param path. 

docker is weird with log rotations.  The global setting (Settings - docker) only takes effect on new installs.  Any existing installs aren't affected and need the extra params.  It may be academic now, but by nuking docker.img, all the installs became "new", and wouldn't need the extra param if you had set up the global.

 

There was a debate on GitHub about how to handle things because it isn't anywhere near intuitive how the system works on logs.

  • Upvote 1
Link to comment
  • 3 months later...
On 9/6/2017 at 8:44 AM, tillkrueger said:

my thought exactly, Squid...since I have no other way of knowing what Krusader is doing...it's a really cool file manager, I must say! if the UI could be configured to always show the Queue at the bottom and the file manager at the top, that would make it an over the top solution for my file management duties on the server...if I were better at using rsync in Terminal, maybe I'd feel different about that, but for a GUI guy like me, Krusader is pretty great...sure beats Dolphin, in my mind.

 

@tillkrueger I've used Dolphin on Kubuntu and find it very functional. Can you tell me how Krusader is better as I am interested in knowing what features I'm missing out on.

Link to comment
37 minutes ago, frakman1 said:

 

@tillkrueger I've used Dolphin on Kubuntu and find it very functional. Can you tell me how Krusader is better as I am interested in knowing what features I'm missing out on.

I also used Dolphin before I started using Krusader, and felt much more drawn to the UI of Krusader...just seemed more refined and “tight” to me. And when I found binhex Krusader, that impression was further enhanced by binhex’s addition of the windows bar at the bottom, which allows direct access to all active windows of your Krusader instance, like the queue, the directory sizes tool, and everything else Krusader can do in parallel to being the excellent file manager that it is.

  • Like 1
Link to comment
  • 4 weeks later...

Hi,

My situation is a bit different but it fits in this topic I guess.

I have a Nextcloud setup with a Letsencrypt reverse proxy. When I upload a large file to Nextcloud my docker image fills up.

As far as I can tell the Letsencrypt container has the most data transfered but I'm unable to find out where exactly that temporary data is stored. All I could find is the /dev/loop2 gets full and obviously I get an error in Nextcloud.

Any hint is apreciated.

Link to comment

This is the config

server {
       listen         80;
       server_name    .....
       return         301 https://$server_name$request_uri;
}

server {
listen 443 ssl;
server_name .....

root /config/www;
index index.html index.htm index.php;

###SSL Certificates
ssl_certificate /config/keys/letsencrypt/fullchain.pem;
ssl_certificate_key /config/keys/letsencrypt/privkey.pem;

###Diffie–Hellman key exchange ###
ssl_dhparam /config/nginx/dhparams.pem;

###SSL Ciphers
ssl_ciphers .....


###Extra Settings###
ssl_prefer_server_ciphers on;
#ssl_session_cache shared:SSL:10m;

        ### Add HTTP Strict Transport Security ###
add_header Strict-Transport-Security "max-age=63072000; includeSubdomains";
add_header Front-End-Https on;

client_max_body_size 0;

location / {
        proxy_pass .....
        proxy_buffering off;
  }

location = /.well-known/carddav {
      return 301 $scheme://$host/remote.php/dav;
    }
location = /.well-known/caldav {
      return 301 $scheme://$host/remote.php/dav;
    }

add_header Referrer-Policy no-referrer always;

}

 I have proxy buffering off so I could avoid this problem.

Link to comment

Sorry I didn't provide enough details.

A kind of buffering happens. Is not related to volume binding.

If the file is smaller then my docker image free space, after the upload ends, the file goes on the array at the correct destination.

BUT if the file size exceeds my docker image free space, the dicker image gets 100% full, the upload fails and the docker image gets back to normal.

So I need to find out where this cache exists to bind it outside of docker image.

 

Here are my mappings. /localdata is created by me for external storage in Nextcloud. It works fine.

/data -> /mnt/user/DATA/Cloud/
/localdata -> /mnt/user/
/config -> /mnt/user/appdata/nextcloud

Actually this happens only if I upload by shared link and not if I'm logged in as a user. I'll start a new topic on Nextcloud thread I think.

 

Edited by andreidelait
New discovery
Link to comment
2 hours ago, andreidelait said:

Sorry I didn't provide enough details.

A kind of buffering happens. Is not related to volume binding.

If the file is smaller then my docker image free space, after the upload ends, the file goes on the array at the correct destination.

BUT if the file size exceeds my docker image free space, the dicker image gets 100% full, the upload fails and the docker image gets back to normal.

So I need to find out where this cache exists to bind it outside of docker image.

 

Here are my mappings. /localdata is created by me for external storage in Nextcloud. It works fine.


/data -> /mnt/user/DATA/Cloud/
/localdata -> /mnt/user/
/config -> /mnt/user/appdata/nextcloud

Actually this happens only if I upload by shared link and not if I'm logged in as a user. I'll start a new topic on Nextcloud thread I think.

 

Mapping to /mnt/user might well be a problem -you should also have a directory at the end of that path (perhaps something like ‘/mnt/user/appdata/nextcloud/localdata’).

Edited by itimpi
Link to comment
  • 3 months later...

My docker image has also been increasing for a while.  This is what I am seeing based on the prior thread suggestions:

 

root@baconator:~# du -h -d 1 /var/lib/docker/
1.2M    /var/lib/docker/containerd
93M     /var/lib/docker/containers
0       /var/lib/docker/plugins
99G     /var/lib/docker/btrfs
42M     /var/lib/docker/image
120M    /var/lib/docker/volumes
0       /var/lib/docker/trust
208K    /var/lib/docker/network
0       /var/lib/docker/swarm
16K     /var/lib/docker/builder
56K     /var/lib/docker/buildkit
1.4M    /var/lib/docker/unraid
0       /var/lib/docker/tmp
0       /var/lib/docker/runtimes
99G     /var/lib/docker/

99GB docker file??  lol

 

My docker is supposed to be 30GB (see below)

 

Untitled.thumb.png.d0ed3b3cfe96ad52e43301348b6ab148.png

 

I also performed a scrub as well.

 

Untitled.png.189f1e5ca11292ca2d4fa2139a6d38d8.png

 

I dont think it is log files:

 

Untitled.png.04ab7e797a6bc35bb55c9108155d62fe.png

 

Untitled.png.7db9cb6024ddff9c9210fa97daa33791.png

 

Just like everyone else, nothing in these show anything anywhere near 30GB.

 

I have tried to truncate logs and that didnt do much either by using the following:

truncate -s 0 /var/lib/docker/containers/*/*-json.log

I did see this in Cadvisor though:

 

Untitled.thumb.png.af58f663f104966711e1168de62d32e9.png

 

Any ideas?  This is highly frustrating.  Thanks for any help.

Edited by danktankk
added new data
Link to comment
55 minutes ago, danktankk said:

My docker image has also been increasing for a while.  This is what I am seeing based on the prior thread suggestions:

 


root@baconator:~# du -h -d 1 /var/lib/docker/
1.2M    /var/lib/docker/containerd
93M     /var/lib/docker/containers
0       /var/lib/docker/plugins
99G     /var/lib/docker/btrfs
42M     /var/lib/docker/image
120M    /var/lib/docker/volumes
0       /var/lib/docker/trust
208K    /var/lib/docker/network
0       /var/lib/docker/swarm
16K     /var/lib/docker/builder
56K     /var/lib/docker/buildkit
1.4M    /var/lib/docker/unraid
0       /var/lib/docker/tmp
0       /var/lib/docker/runtimes
99G     /var/lib/docker/

99GB docker file??  lol

 

My docker is supposed to be 30GB (see below)

 

Untitled.thumb.png.d0ed3b3cfe96ad52e43301348b6ab148.png

 

I also performed a scrub as well.

 

Untitled.png.189f1e5ca11292ca2d4fa2139a6d38d8.png

 

I dont think it is log files:

 

Untitled.png.04ab7e797a6bc35bb55c9108155d62fe.png

 

Untitled.png.7db9cb6024ddff9c9210fa97daa33791.png

 

Just like everyone else, nothing in these show anything anywhere near 30GB.

 

I have tried to truncate logs and that didnt do much either by using the following:


truncate -s 0 /var/lib/docker/containers/*/*-json.log

I did see this in Cadvisor though:

 

Untitled.thumb.png.af58f663f104966711e1168de62d32e9.png

 

Any ideas?  This is highly frustrating.  Thanks for any help.

Try this user script https://forums.unraid.net/topic/48707-additional-scripts-for-userscripts-plugin/page/8/?tab=comments#comment-682421  It calculates the sizes slightly different than the Container Size button.  Is there a significant difference?  Note that the script can only calculate sizes for running containers.

 

Link to comment
44 minutes ago, Squid said:

Try this user script https://forums.unraid.net/topic/48707-additional-scripts-for-userscripts-plugin/page/8/?tab=comments#comment-682421  It calculates the sizes slightly different than the Container Size button.  Is there a significant difference?  Note that the script can only calculate sizes for running containers.

 

Hey,

 

Thanks for the reply.  I dont see anything here that would indicate an issue either.  This is what I got from running the script:  

Untitled.thumb.png.389506cc00bc0383304469adca5ea9e1.png

Edited by danktankk
Link to comment
9 hours ago, danktankk said:

My docker image has also been increasing for a while.  This is what I am seeing based on the prior thread suggestions:

 


root@baconator:~# du -h -d 1 /var/lib/docker/
1.2M    /var/lib/docker/containerd
93M     /var/lib/docker/containers
0       /var/lib/docker/plugins
99G     /var/lib/docker/btrfs
42M     /var/lib/docker/image
120M    /var/lib/docker/volumes
0       /var/lib/docker/trust
208K    /var/lib/docker/network
0       /var/lib/docker/swarm
16K     /var/lib/docker/builder
56K     /var/lib/docker/buildkit
1.4M    /var/lib/docker/unraid
0       /var/lib/docker/tmp
0       /var/lib/docker/runtimes
99G     /var/lib/docker/

99GB docker file??  lol

 

My docker is supposed to be 30GB (see below)

 

Untitled.thumb.png.d0ed3b3cfe96ad52e43301348b6ab148.png

 

I also performed a scrub as well.

 

Untitled.png.189f1e5ca11292ca2d4fa2139a6d38d8.png

 

I dont think it is log files:

 

Untitled.png.04ab7e797a6bc35bb55c9108155d62fe.png

 

Untitled.png.7db9cb6024ddff9c9210fa97daa33791.png

 

Just like everyone else, nothing in these show anything anywhere near 30GB.

 

I have tried to truncate logs and that didnt do much either by using the following:


truncate -s 0 /var/lib/docker/containers/*/*-json.log

I did see this in Cadvisor though:

 

Untitled.thumb.png.af58f663f104966711e1168de62d32e9.png

 

Any ideas?  This is highly frustrating.  Thanks for any help.

du command goes across filesystem for sure

I've tried this one too and mine returns a 74G /var/lib/docker/btrfs, my docker image having 40G.

However if I insert -x argument it returns 0 for that volume. I'm sure it calculates some volume mappings or something.

 

So usually I use

 

root@Tower:~# docker system df
TYPE                TOTAL               ACTIVE              SIZE                RECLAIMABLE
Images              28                  28                  11.57GB             784.9MB (6%)
Containers          28                  16                  1.744GB             372.3MB (21%)
Local Volumes       11                  1                   39.68MB             0B (0%)
Build Cache         0                   0                   0B                  0B

or

root@Tower:~# df -h | grep /dev/loop2
/dev/loop2       40G   15G   25G  38% /var/lib/docker

 

Edited by andreidelait
Link to comment

My docker size is still an issue.  This should not be that difficult to figure out.  lol

 

If there was a new feature to be requested for unraid, it would be a new mechanism to show precisely what docker containers have what and where the space usage is increasing.  In windows this is extremely easy to do.

 

Thanks again for the help thus far.  :)

Edited by danktankk
Link to comment

Try this:  go to the Shares tab.  Click on the folder icon next to the system share.  Now click on the folder for Docker.  What size does it report on that page for the docker image size?

 

Also, when you scrubbed your container, did you notice the amount scrubbed was only 22GiB?

 

image.png

 

If you're not familiar with how Docker, volume mappings, thin provisioning, etc. work, the du command can certainly confuse you.

Link to comment

@jonp

 

Thank you for the reply!  This is the amount of space it is currently reporting:

Untitled.thumb.png.c43a8a6705badfd6b90824dd3fd63627.png

 

All containeras added up is 15.6GB or so:

Untitled.thumb.png.e5dea328da4537a7a3fecd9e0b1b4fbc.png

 

however, I am still seeing this usage and getting notifications often:

Untitled.png.f6077960d6a0b343e096df5e0f28df2f.png

 

If the amount scrubbed is 22.16GB, it does make sense that it would report, and does report 75-80% usage.

 

Is there any reason for this discrepancy (15.6GB vs 22.1GB you can think of?

 

Thanks again for the input.

 

 

 

Edited by danktankk
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.