Docker high image disk utilization: why is my docker image/disk getting full?


Recommended Posts

Hey Guys,

 

So, recently, i have been getting notifications about my docker image/disk getting full (hitting about 81% now). I havent installed anything new for like the last 6 months (besides the auto updates), and the disk utilization is slowly increasing (ie. few weeks ago, it was at 75%). 

 

When i first created, i allocated 20gb to my docker disk/image. Currently its using up about 17gb/20gb. If this is normal, i have plenty more hdd space, if someone can tell me how i can expand my allocated 20gb to maybe 30 or 40gb, that would solve the problem for a little while. But in the meantime, i want to know what is going on and why its using up 17gb, and still increasing?

 

Anyway, i just installed the cadvisor docker to see if it gives me any information, and i see this on the docker page of cadvisor, 

 

Images

Repository Tags ID Virtual Size Creation Time
linuxserver/sonarr latest sha256:6aa3c326ea81898a9  328.76 MiB 1/21/2017, 12:33:42 PM
linuxserver/sabnzbd latest sha256:183cd9b9813960b1d  257.16 MiB 1/14/2017, 12:41:12 PM
linuxserver/plexpy latest sha256:d6dbfe6c4a38f4863 109.64 MiB 10/1/2016, 11:23:52 AM
linuxserver/plex latest sha256:699f00c8a8c23307e 404.95 MiB 5/6/2017, 10:44:05 AM
linuxserver/htpcmanager latest sha256:879c6373ef5b14f1d 111.90 MiB 10/1/2016, 11:23:30 AM
linuxserver/deluge latest sha256:0dd36542df4e3fc7d 157.01 MiB 1/14/2017, 11:38:21 AM
linuxserver/couchpotato latest sha256:1416f816197775c9b 111.50 MiB 1/14/2017, 12:33:44 PM
limetech/sync latest sha256:54597b6c03906d4f5 300.71 MiB 7/3/2016, 3:37:30 AM
google/cadvisor latest sha256:f9ba08bafdeaf8158 54.66 MiB 3/10/2017, 12:30:29 PM
dreamcat4/jackett latest sha256:80df299519f91d03e 320.86 MiB 10/11/2016, 4:03:57 PM
coppit/duckdns latest sha256:3a52d808122f4b0a4 226.63 MiB 9/2/2015, 3:52:50 PM
 
and as you can see, those sizes doesnt add upto anywhere near the 17gb.
 
Can someone help me debug this further?
 
ps. im on the latest version of unraid, and everything set to auto update.
Link to comment

Odds on either sab or deluge (or possibly sync) is winding up downloading into the docker image (and things like that don't really show up under the virtual size)  Check your path mappings on the template for those apps AND the settings within the apps for where they are downloading to.

 

And expanding is simple.  Settings - docker - disable docker.  Advanced view.  Increase the size and reenable docker.  Apply

 

Off the top of my head, when properly configured your installed apps would utilize maybe 10Gig.

  • Like 3
  • Thanks 5
Link to comment
38 minutes ago, Squid said:

Odds on either sab or deluge (or possibly sync) is winding up downloading into the docker image (and things like that don't really show up under the virtual size)  Check your path mappings on the template for those apps AND the settings within the apps for where they are downloading to.

 

And expanding is simple.  Settings - docker - disable docker.  Advanced view.  Increase the size and reenable docker.  Apply

 

Off the top of my head, when properly configured your installed apps would utilize maybe 10Gig.

 

Thanks for the instructions on expanding, if i dint get a solution for the size issue, i will do this :)

 

As for sab, deluge, and sync... sync isnt connected to anything at all right now, so it shouldnt actually use up anything. 

 

So, on my setup/filesystem, i have a /mnt/user/Docker/AppData/ which contains all the app data for most of the dockers. this folder is like 250+gb thanks to plex app data dir taking up ~250gb (wtf right? this is an issue i had earlier when i had plex server installed on windows too, so its an unrelated issue). Note, this is not the plex media directory, its just the config/cache directory.

Then i have a unassigned devices mapped as /mnt/disks/mydrive1/ which contains multiple download directories which are mapped to sab and deluge as download directories. 

I really cant find anything else in settings that are not either mapped to /mnt/disks/mydrive1/* or Docker/AppData/*

note: my docker image is located at /mnt/user/Docker/Docker/docker.img

 

Is there any easy way for me to look inside docker.img to see whats using up all the space?

Link to comment

Its not so much your mappings on your templates, but rather where you are telling the download clients where to put the incomplete downloads.  If you don't specify to put them onto a folder you've mapped, then they wind up being stored within the docker.img file.

 

Either that, or you're a victim of excessive logging...

 

http://lime-technology.com/oldforum/index.php?topic=40937.msg405070#msg405070

 

http://lime-technology.com/oldforum/index.php?topic=40937.msg475225#msg475225

 

 

 

 

Link to comment
7 hours ago, Squid said:

Its not so much your mappings on your templates, but rather where you are telling the download clients where to put the incomplete downloads.  If you don't specify to put them onto a folder you've mapped, then they wind up being stored within the docker.img file.

 

Either that, or you're a victim of excessive logging...

 

http://lime-technology.com/oldforum/index.php?topic=40937.msg405070#msg405070

 

http://lime-technology.com/oldforum/index.php?topic=40937.msg475225#msg475225

 

 

 

 

 

first of all, your second link didnt work for me (your first link worked once but not anymore, wtf) :( maybe server/forum issues? ill try again tomorrow.

 

Anyway, as for telling where my download clients download to (temp/incompletes/completes), ive double/triple checked in settings, both on the settings of the docker as well as settings within the app it self (for example, the SAB UI), and on both SAB and deluge. Both SAB and deluge seems to be configured correctly (im eyeballing here, but all seems correct to me, im using "/" correctly, case sensitive etc). Only outlier i saw was that, sab has the log and admin folders linking within the docker image, and i have no way to check if those directories are the culprit?

 

That being said, if there was a way i could actually have access to the file structure on docker.img, then i could run something like windirstat to get a idea of what directories are using up the space and i can investigate. Any idea how i could do this?

 

Secondly, its possible that the plex transcorder is one of the culprits. there was a post couple of replies below on the first link that addresses that. but i highly doubt that, since i dont see the space fluctuating massively in a short amount of time. and even if that was the case, remapping the transcode directory wouldn't help clear out what ever it was that is currently clogging up the image. =/

 

any more thoughts?

 

Edited by syrys
Link to comment
  • 2 weeks later...

Can anyone help me with this? My docker image is now at 85% :( I have not installed or done any changes since my earlier post.

 

The links from Squid does not work for me, so i cannot check the issue he is refferings to.

 

Since the last post, i have done few things to debug but with no luck.

 

Using ssh, I tried getting a list of docker containers using

docker ps -a

Then exported every single docker container one by one into tars using:

docker export container_name > container_name.tar
#for example
docker export plex > plex.tar

Looked at the file sizes for the resulting tars, and they barely add upto 3-4gb as expected. But my docker image is 20gb and its hitting 85% usage which is 17gb. What on earth is going on? Is there really no way to accurately look at the size of each container?

Link to comment
13 minutes ago, syrys said:

Can anyone help me with this? My docker image is now at 85% :( I have not installed or done any changes since my earlier post.

 

The links from Squid does not work for me, so i cannot check the issue he is refferings to.

 

Since the last post, i have done few things to debug but with no luck.

 

Using ssh, I tried getting a list of docker containers using


docker ps -a

Then exported every single docker container one by one into tars using:


docker export container_name > container_name.tar
#for example
docker export plex > plex.tar

Looked at the file sizes for the resulting tars, and they barely add upto 3-4gb as expected. But my docker image is 20gb and its hitting 85% usage which is 17gb. What on earth is going on? Is there really no way to accurately look at the size of each container?

peruse the temporary Docker FAQ (I haven't gotten around to a proper table of contents though)

 

 

Link to comment
  • 3 months later...

I, too, just got a few warnings that my docker.img went from just under 70% to over 75%. In my case it appears pretty obvious that my Krusader app is doing this, as this is the only Docker I've used in a long time and have started to use it to migrate data from smaller/older drives in the array to new/bigger drives...since I'll be doing a *lot* of this in the coming days/weeks (paring down a 15 drive unRAID system to an 8 drive system) I see the pressing need to keep the docker.img from reaching 100%.

Could it be that Krusader does excessive logging, and if so, where/how can this be controlled?

After the current move operation finishes (in about 5hrs), I can pause and deal with figuring out how to increase the docker.img size before continuing this lengthy process of migrating/combining data to bigger drives.

Link to comment

the big horizontal Queue window is covering up the Docker menus right now, so I can't operate it until the current move operation is in progress, but will look for "Empty Trash" as soon as it finishes this move (in about 2hrs)...if that solves it, that would be nice; thanks Squid!

do you know how I can get that big horizontal Queue window back after I close it? not the little floating one but the big horizontal one that actually shows you the progress and what file(s) it is doing.

copy/move operations are not persistent in Krusader, or are they? if I close the browser tab/window that Krusader is running in, it stops the files operation(s), or does it not?

Link to comment
2 hours ago, tillkrueger said:

I, too, just got a few warnings that my docker.img went from just under 70% to over 75%. In my case it appears pretty obvious that my Krusader app is doing this, as this is the only Docker I've used in a long time and have started to use it to migrate data from smaller/older drives in the array to new/bigger drives...since I'll be doing a *lot* of this in the coming days/weeks (paring down a 15 drive unRAID system to an 8 drive system) I see the pressing need to keep the docker.img from reaching 100%.

Could it be that Krusader does excessive logging, and if so, where/how can this be controlled?

After the current move operation finishes (in about 5hrs), I can pause and deal with figuring out how to increase the docker.img size before continuing this lengthy process of migrating/combining data to bigger drives.

 

Unfortunately i still havent found a solution for this original issue. Im glad that you atleast have a good idea what app is causing your issue. In my case, i have no idea still (and i dont have a clone system to test it out) :(

 

For starters, it would be super helpful if there was a way to find out the actual HDD usage of every docker running on the system (so we can nail down the culprit). And once we nail down the culprit, atleast you can dive into try solve the problem or go complain to the docker developer about the issue. Ah well, for the time being, im just increasing the docker space allocation. I hope the usage increase is slower than the consumer SSD size improvements haha :P

Link to comment

my thought exactly, Squid...since I have no other way of knowing what Krusader is doing...it's a really cool file manager, I must say! if the UI could be configured to always show the Queue at the bottom and the file manager at the top, that would make it an over the top solution for my file management duties on the server...if I were better at using rsync in Terminal, maybe I'd feel different about that, but for a GUI guy like me, Krusader is pretty great...sure beats Dolphin, in my mind.

if ever you do find a way to get that big Queue display back, do let us know, Squid! (and I'll do the same)

Link to comment

use "docker ps -s" to see where the usage is, and then figure out how to clean it up.  mine was in pydio:

 

# docker ps -s
CONTAINER ID        IMAGE                      COMMAND             CREATED             STATUS              PORTS                                                                                               NAMES               SIZE
2d6bc0931a8c        limetech/plex              "/sbin/my_init"     3 weeks ago         Up 6 days                                                                                                               PlexMediaServer     377.3 kB (virtual 557.3 MB)
ad63faa6b9f5        linuxserver/letsencrypt    "/init"             4 weeks ago         Up 6 days           0.0.0.0:1180->80/tcp, 0.0.0.0:11443->443/tcp                                                        letsencrypt         624.1 kB (virtual 180.2 MB)
e22be497a65d        linuxserver/rutorrent      "/init"             4 weeks ago         Up 6 days           0.0.0.0:51413->51413/tcp, 0.0.0.0:6881->6881/udp, 0.0.0.0:51444->51444/tcp, 0.0.0.0:10080->80/tcp   rutorrent           4.924 MB (virtual 123.9 MB)
c717743877a1        linuxserver/pydio          "/init"             5 weeks ago         Up 6 days           80/tcp, 0.0.0.0:33443->443/tcp                                                                      pydio               58.47 GB (virtual 58.66 GB)
83792be00581        linuxserver/mysql          "/sbin/my_init"     5 weeks ago         Up 6 days           0.0.0.0:3306->3306/tcp                                                                              mysql               796.8 kB (virtual 533.7 MB)
655e33471cbe        linuxserver/sonarr         "/init"             6 weeks ago         Up 6 days           0.0.0.0:18989->8989/tcp                                                                             sonarr              22.27 MB (virtual 528.1 MB)
7088b4c76ff3        linuxserver/couchpotato    "/init"             6 weeks ago         Up 6 days           0.0.0.0:15050->5050/tcp                                                                             couchpotato         18.56 MB (virtual 164.1 MB)
b1aa37b16af1        linuxserver/jackett        "/init"             6 weeks ago         Up 6 days           0.0.0.0:19117->9117/tcp                                                                             jackett             12.13 MB (virtual 510.9 MB)
4776efea5f09        linuxserver/plexpy         "/init"             6 weeks ago         Up 6 days           0.0.0.0:18181->8181/tcp                                                                             plexpy              89.75 MB (virtual 324.2 MB)
da4cf5fb541d        linuxserver/plexrequests   "/init"             6 weeks ago         Up 6 days           0.0.0.0:3000->3000/tcp                                                                              plexrequests        43.48 MB (virtual 373.4 MB)

 

  • Like 2
Link to comment
8 hours ago, assassinmunky said:

use "docker ps -s" to see where the usage is, and then figure out how to clean it up.  mine was in pydio:

 

 

Tried this, but the numbers of this doesnt add up at all. The size column adds upto maybe 3GB at max (3gb virtual, <500mb normal). But my docker usage is well over 18gb. So yeah... :(

 

 

CONTAINER ID        IMAGE                     COMMAND                  CREATED             STATUS              PORTS                                            NAMES               SIZE
d43779fb64f6        linuxserver/plex          "/init"                  10 days ago         Up 8 days                                                            plex                307.9 kB (virtual 432.1 MB)
d6dca3fe11f5        linuxserver/sonarr        "/init"                  3 weeks ago         Up 8 days           0.0.0.0:8989->8989/tcp                           Sonarr              137.1 MB (virtual 643.4 MB)
eed3a5e07642        linuxserver/radarr        "/init"                  6 weeks ago         Up 8 days           0.0.0.0:7878->7878/tcp                           radarr              27.62 MB (virtual 535.3 MB)
2731688ca5f1        linuxserver/sabnzbd       "/init"                  6 weeks ago         Up 8 days           0.0.0.0:8080->8080/tcp, 0.0.0.0:9090->9090/tcp   sabnzbd             339.6 kB (virtual 271.3 MB)
61c0ac7c92c6        linuxserver/deluge        "/init"                  9 weeks ago         Up 8 days                                                            deluge              589.8 kB (virtual 172.3 MB)
0cbc1ab01f86        google/cadvisor:latest    "/usr/bin/cadvisor -l"   3 months ago        Up 8 days           0.0.0.0:8081->8080/tcp                           cadvisor            0 B (virtual 57.32 MB)
cd9f8971ca71        dreamcat4/jackett         "/init"                  10 months ago       Up 8 days           0.0.0.0:9117-9118->9117-9118/tcp                 jackett-public      18.92 MB (virtual 355.4 MB)
e010e90c85bd        linuxserver/plexpy        "/init"                  11 months ago       Up 8 days           0.0.0.0:8181->8181/tcp                           plexpy              128.1 MB (virtual 243 MB)
2386dbb08d28        coppit/duckdns            "/bin/sh -c /root/duc"   11 months ago       Up 8 days                                                            DuckDNS             144 B (virtual 237.6 MB)
b6471e1de692        linuxserver/htpcmanager   "/init"                  11 months ago       Up 8 days           0.0.0.0:8085->8085/tcp                           htpcmanager         31.39 MB (virtual 148.7 MB)

 

Link to comment
1 hour ago, assassinmunky said:

what does the usage in /var/lib/docker look like?    what is your output of this command?


du -h -d 1 /var/lib/docker/

 

 

 

This is my output:

> du -h -d 1 /var/lib/docker/
0       /var/lib/docker/tmp
9.6G    /var/lib/docker/containers
22G     /var/lib/docker/btrfs
12M     /var/lib/docker/image
68K     /var/lib/docker/volumes
0       /var/lib/docker/trust
84K     /var/lib/docker/network
2.4M    /var/lib/docker/unraid
0       /var/lib/docker/swarm
32G     /var/lib/docker/

Not really sure what this is actually telling me though :(

Edited by syrys
Link to comment

emptying the Trash in Krusader did not solve me docker.img ballooning issue (now at 86%, growing by about 1% every hour or so).

I also found that Krusader did *not* continue the file moving operations after I shout down the computer from which I initiated the file management operations...I built a queue with 2 items in addition to the first item, which was at 98% completed), and found everything this morning exactly where it was last night...the queue was still intact, but had paused, in essence, last night when I shut down my control computer.

Krusader also seems to use total file-size when estimating completion...the last 2% (even though calculated at 30mins) are taking many hours, bc they consist mostly of small JPG images from various timelapses I shot. Krusader is amazing, but lots of room for improvement, still.

if a moderator wants to move this into a new thread, I think that would be best...otherwise please excuse my deviation from the OP's main-topic!

Link to comment
28 minutes ago, tillkrueger said:

if a moderator wants to move this into a new thread, I think that would be best...otherwise please excuse my deviation from the OP's main-topic!

 

Haha i dont mind. Maybe suggestions you get might help me with my issue too (fingers crossed) :)

  • Like 1
Link to comment

well damnit, looks like I'm stuck until I can clear the docker.img and/or increase its size...by a *lot*, then, since I filled up the docker.img within only hours and have so many terabytes worth of data ahead of me that it will take days/weeks.

guess I'm in the same shoes as you, syrys, of needing a solution to not ballooning the docker.img in the first place...maybe slightly simpler than your issue, as I *only* run the Krusader docker.

Link to comment
3 hours ago, tillkrueger said:

did you end up increasing the size of your docker.img successfully? I don't seem to be able to find the info on that and my img is now at 90%...I stopped all dockers and am ready to make the change, but can't locate the instructions pertaining to this.

 

Yeah i did, but to do so, you had to turn off docker. I simply just followed squid's instructions:

 

On 5/21/2017 at 4:16 PM, Squid said:

And expanding is simple.  Settings - docker - disable docker.  Advanced view.  Increase the size and reenable docker.  Apply

 

For those that know about docker, is there really no way for one to browse into a docker containers file system and inspect/delete a file? Say you want to locate and delete a log fine manually, and you are sure its just being saved inside the docker, is there really no way to browse into the docker's file system and try locate and remove said file(s)? This would solve most of our problems (although hackey, its still a viable solution).

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.