syrys Posted May 21, 2017 Share Posted May 21, 2017 Hey Guys, So, recently, i have been getting notifications about my docker image/disk getting full (hitting about 81% now). I havent installed anything new for like the last 6 months (besides the auto updates), and the disk utilization is slowly increasing (ie. few weeks ago, it was at 75%). When i first created, i allocated 20gb to my docker disk/image. Currently its using up about 17gb/20gb. If this is normal, i have plenty more hdd space, if someone can tell me how i can expand my allocated 20gb to maybe 30 or 40gb, that would solve the problem for a little while. But in the meantime, i want to know what is going on and why its using up 17gb, and still increasing? Anyway, i just installed the cadvisor docker to see if it gives me any information, and i see this on the docker page of cadvisor, Images Repository Tags ID Virtual Size Creation Time linuxserver/sonarr latest sha256:6aa3c326ea81898a9 328.76 MiB 1/21/2017, 12:33:42 PM linuxserver/sabnzbd latest sha256:183cd9b9813960b1d 257.16 MiB 1/14/2017, 12:41:12 PM linuxserver/plexpy latest sha256:d6dbfe6c4a38f4863 109.64 MiB 10/1/2016, 11:23:52 AM linuxserver/plex latest sha256:699f00c8a8c23307e 404.95 MiB 5/6/2017, 10:44:05 AM linuxserver/htpcmanager latest sha256:879c6373ef5b14f1d 111.90 MiB 10/1/2016, 11:23:30 AM linuxserver/deluge latest sha256:0dd36542df4e3fc7d 157.01 MiB 1/14/2017, 11:38:21 AM linuxserver/couchpotato latest sha256:1416f816197775c9b 111.50 MiB 1/14/2017, 12:33:44 PM limetech/sync latest sha256:54597b6c03906d4f5 300.71 MiB 7/3/2016, 3:37:30 AM google/cadvisor latest sha256:f9ba08bafdeaf8158 54.66 MiB 3/10/2017, 12:30:29 PM dreamcat4/jackett latest sha256:80df299519f91d03e 320.86 MiB 10/11/2016, 4:03:57 PM coppit/duckdns latest sha256:3a52d808122f4b0a4 226.63 MiB 9/2/2015, 3:52:50 PM and as you can see, those sizes doesnt add upto anywhere near the 17gb. Can someone help me debug this further? ps. im on the latest version of unraid, and everything set to auto update. Quote Link to comment
Squid Posted May 21, 2017 Share Posted May 21, 2017 Odds on either sab or deluge (or possibly sync) is winding up downloading into the docker image (and things like that don't really show up under the virtual size) Check your path mappings on the template for those apps AND the settings within the apps for where they are downloading to. And expanding is simple. Settings - docker - disable docker. Advanced view. Increase the size and reenable docker. Apply Off the top of my head, when properly configured your installed apps would utilize maybe 10Gig. 3 6 Quote Link to comment
syrys Posted May 21, 2017 Author Share Posted May 21, 2017 38 minutes ago, Squid said: Odds on either sab or deluge (or possibly sync) is winding up downloading into the docker image (and things like that don't really show up under the virtual size) Check your path mappings on the template for those apps AND the settings within the apps for where they are downloading to. And expanding is simple. Settings - docker - disable docker. Advanced view. Increase the size and reenable docker. Apply Off the top of my head, when properly configured your installed apps would utilize maybe 10Gig. Thanks for the instructions on expanding, if i dint get a solution for the size issue, i will do this As for sab, deluge, and sync... sync isnt connected to anything at all right now, so it shouldnt actually use up anything. So, on my setup/filesystem, i have a /mnt/user/Docker/AppData/ which contains all the app data for most of the dockers. this folder is like 250+gb thanks to plex app data dir taking up ~250gb (wtf right? this is an issue i had earlier when i had plex server installed on windows too, so its an unrelated issue). Note, this is not the plex media directory, its just the config/cache directory. Then i have a unassigned devices mapped as /mnt/disks/mydrive1/ which contains multiple download directories which are mapped to sab and deluge as download directories. I really cant find anything else in settings that are not either mapped to /mnt/disks/mydrive1/* or Docker/AppData/* note: my docker image is located at /mnt/user/Docker/Docker/docker.img Is there any easy way for me to look inside docker.img to see whats using up all the space? 1 Quote Link to comment
Squid Posted May 21, 2017 Share Posted May 21, 2017 Its not so much your mappings on your templates, but rather where you are telling the download clients where to put the incomplete downloads. If you don't specify to put them onto a folder you've mapped, then they wind up being stored within the docker.img file. Either that, or you're a victim of excessive logging... http://lime-technology.com/oldforum/index.php?topic=40937.msg405070#msg405070 http://lime-technology.com/oldforum/index.php?topic=40937.msg475225#msg475225 1 Quote Link to comment
syrys Posted May 21, 2017 Author Share Posted May 21, 2017 (edited) 7 hours ago, Squid said: Its not so much your mappings on your templates, but rather where you are telling the download clients where to put the incomplete downloads. If you don't specify to put them onto a folder you've mapped, then they wind up being stored within the docker.img file. Either that, or you're a victim of excessive logging... http://lime-technology.com/oldforum/index.php?topic=40937.msg405070#msg405070 http://lime-technology.com/oldforum/index.php?topic=40937.msg475225#msg475225 first of all, your second link didnt work for me (your first link worked once but not anymore, wtf) maybe server/forum issues? ill try again tomorrow. Anyway, as for telling where my download clients download to (temp/incompletes/completes), ive double/triple checked in settings, both on the settings of the docker as well as settings within the app it self (for example, the SAB UI), and on both SAB and deluge. Both SAB and deluge seems to be configured correctly (im eyeballing here, but all seems correct to me, im using "/" correctly, case sensitive etc). Only outlier i saw was that, sab has the log and admin folders linking within the docker image, and i have no way to check if those directories are the culprit? That being said, if there was a way i could actually have access to the file structure on docker.img, then i could run something like windirstat to get a idea of what directories are using up the space and i can investigate. Any idea how i could do this? Secondly, its possible that the plex transcorder is one of the culprits. there was a post couple of replies below on the first link that addresses that. but i highly doubt that, since i dont see the space fluctuating massively in a short amount of time. and even if that was the case, remapping the transcode directory wouldn't help clear out what ever it was that is currently clogging up the image. =/ any more thoughts? Edited May 21, 2017 by syrys Quote Link to comment
syrys Posted May 30, 2017 Author Share Posted May 30, 2017 Can anyone help me with this? My docker image is now at 85% I have not installed or done any changes since my earlier post. The links from Squid does not work for me, so i cannot check the issue he is refferings to. Since the last post, i have done few things to debug but with no luck. Using ssh, I tried getting a list of docker containers using docker ps -a Then exported every single docker container one by one into tars using: docker export container_name > container_name.tar #for example docker export plex > plex.tar Looked at the file sizes for the resulting tars, and they barely add upto 3-4gb as expected. But my docker image is 20gb and its hitting 85% usage which is 17gb. What on earth is going on? Is there really no way to accurately look at the size of each container? Quote Link to comment
Squid Posted May 30, 2017 Share Posted May 30, 2017 13 minutes ago, syrys said: Can anyone help me with this? My docker image is now at 85% I have not installed or done any changes since my earlier post. The links from Squid does not work for me, so i cannot check the issue he is refferings to. Since the last post, i have done few things to debug but with no luck. Using ssh, I tried getting a list of docker containers using docker ps -a Then exported every single docker container one by one into tars using: docker export container_name > container_name.tar #for example docker export plex > plex.tar Looked at the file sizes for the resulting tars, and they barely add upto 3-4gb as expected. But my docker image is 20gb and its hitting 85% usage which is 17gb. What on earth is going on? Is there really no way to accurately look at the size of each container? peruse the temporary Docker FAQ (I haven't gotten around to a proper table of contents though) Quote Link to comment
tillkrueger Posted September 6, 2017 Share Posted September 6, 2017 I, too, just got a few warnings that my docker.img went from just under 70% to over 75%. In my case it appears pretty obvious that my Krusader app is doing this, as this is the only Docker I've used in a long time and have started to use it to migrate data from smaller/older drives in the array to new/bigger drives...since I'll be doing a *lot* of this in the coming days/weeks (paring down a 15 drive unRAID system to an 8 drive system) I see the pressing need to keep the docker.img from reaching 100%. Could it be that Krusader does excessive logging, and if so, where/how can this be controlled? After the current move operation finishes (in about 5hrs), I can pause and deal with figuring out how to increase the docker.img size before continuing this lengthy process of migrating/combining data to bigger drives. Quote Link to comment
tillkrueger Posted September 6, 2017 Share Posted September 6, 2017 sollte ich eine neue Diskussion anfangen, da diese hier schon etwas älter ist? Quote Link to comment
Squid Posted September 6, 2017 Share Posted September 6, 2017 Empty the trash bin in krusader 2 Quote Link to comment
tillkrueger Posted September 6, 2017 Share Posted September 6, 2017 the big horizontal Queue window is covering up the Docker menus right now, so I can't operate it until the current move operation is in progress, but will look for "Empty Trash" as soon as it finishes this move (in about 2hrs)...if that solves it, that would be nice; thanks Squid! do you know how I can get that big horizontal Queue window back after I close it? not the little floating one but the big horizontal one that actually shows you the progress and what file(s) it is doing. copy/move operations are not persistent in Krusader, or are they? if I close the browser tab/window that Krusader is running in, it stops the files operation(s), or does it not? Quote Link to comment
syrys Posted September 6, 2017 Author Share Posted September 6, 2017 2 hours ago, tillkrueger said: I, too, just got a few warnings that my docker.img went from just under 70% to over 75%. In my case it appears pretty obvious that my Krusader app is doing this, as this is the only Docker I've used in a long time and have started to use it to migrate data from smaller/older drives in the array to new/bigger drives...since I'll be doing a *lot* of this in the coming days/weeks (paring down a 15 drive unRAID system to an 8 drive system) I see the pressing need to keep the docker.img from reaching 100%. Could it be that Krusader does excessive logging, and if so, where/how can this be controlled? After the current move operation finishes (in about 5hrs), I can pause and deal with figuring out how to increase the docker.img size before continuing this lengthy process of migrating/combining data to bigger drives. Unfortunately i still havent found a solution for this original issue. Im glad that you atleast have a good idea what app is causing your issue. In my case, i have no idea still (and i dont have a clone system to test it out) For starters, it would be super helpful if there was a way to find out the actual HDD usage of every docker running on the system (so we can nail down the culprit). And once we nail down the culprit, atleast you can dive into try solve the problem or go complain to the docker developer about the issue. Ah well, for the time being, im just increasing the docker space allocation. I hope the usage increase is slower than the consumer SSD size improvements haha Quote Link to comment
Squid Posted September 6, 2017 Share Posted September 6, 2017 (edited) There was one post around here about how to get it back (basically if you keep poking around I the menus it's there somewhere) Myself I've never figured it out and make a point to not close the popup Edited September 6, 2017 by Squid Quote Link to comment
tillkrueger Posted September 6, 2017 Share Posted September 6, 2017 my thought exactly, Squid...since I have no other way of knowing what Krusader is doing...it's a really cool file manager, I must say! if the UI could be configured to always show the Queue at the bottom and the file manager at the top, that would make it an over the top solution for my file management duties on the server...if I were better at using rsync in Terminal, maybe I'd feel different about that, but for a GUI guy like me, Krusader is pretty great...sure beats Dolphin, in my mind. if ever you do find a way to get that big Queue display back, do let us know, Squid! (and I'll do the same) Quote Link to comment
assassinmunky Posted September 6, 2017 Share Posted September 6, 2017 use "docker ps -s" to see where the usage is, and then figure out how to clean it up. mine was in pydio: # docker ps -s CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES SIZE 2d6bc0931a8c limetech/plex "/sbin/my_init" 3 weeks ago Up 6 days PlexMediaServer 377.3 kB (virtual 557.3 MB) ad63faa6b9f5 linuxserver/letsencrypt "/init" 4 weeks ago Up 6 days 0.0.0.0:1180->80/tcp, 0.0.0.0:11443->443/tcp letsencrypt 624.1 kB (virtual 180.2 MB) e22be497a65d linuxserver/rutorrent "/init" 4 weeks ago Up 6 days 0.0.0.0:51413->51413/tcp, 0.0.0.0:6881->6881/udp, 0.0.0.0:51444->51444/tcp, 0.0.0.0:10080->80/tcp rutorrent 4.924 MB (virtual 123.9 MB) c717743877a1 linuxserver/pydio "/init" 5 weeks ago Up 6 days 80/tcp, 0.0.0.0:33443->443/tcp pydio 58.47 GB (virtual 58.66 GB) 83792be00581 linuxserver/mysql "/sbin/my_init" 5 weeks ago Up 6 days 0.0.0.0:3306->3306/tcp mysql 796.8 kB (virtual 533.7 MB) 655e33471cbe linuxserver/sonarr "/init" 6 weeks ago Up 6 days 0.0.0.0:18989->8989/tcp sonarr 22.27 MB (virtual 528.1 MB) 7088b4c76ff3 linuxserver/couchpotato "/init" 6 weeks ago Up 6 days 0.0.0.0:15050->5050/tcp couchpotato 18.56 MB (virtual 164.1 MB) b1aa37b16af1 linuxserver/jackett "/init" 6 weeks ago Up 6 days 0.0.0.0:19117->9117/tcp jackett 12.13 MB (virtual 510.9 MB) 4776efea5f09 linuxserver/plexpy "/init" 6 weeks ago Up 6 days 0.0.0.0:18181->8181/tcp plexpy 89.75 MB (virtual 324.2 MB) da4cf5fb541d linuxserver/plexrequests "/init" 6 weeks ago Up 6 days 0.0.0.0:3000->3000/tcp plexrequests 43.48 MB (virtual 373.4 MB) 2 Quote Link to comment
syrys Posted September 6, 2017 Author Share Posted September 6, 2017 8 hours ago, assassinmunky said: use "docker ps -s" to see where the usage is, and then figure out how to clean it up. mine was in pydio: Tried this, but the numbers of this doesnt add up at all. The size column adds upto maybe 3GB at max (3gb virtual, <500mb normal). But my docker usage is well over 18gb. So yeah... CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES SIZE d43779fb64f6 linuxserver/plex "/init" 10 days ago Up 8 days plex 307.9 kB (virtual 432.1 MB) d6dca3fe11f5 linuxserver/sonarr "/init" 3 weeks ago Up 8 days 0.0.0.0:8989->8989/tcp Sonarr 137.1 MB (virtual 643.4 MB) eed3a5e07642 linuxserver/radarr "/init" 6 weeks ago Up 8 days 0.0.0.0:7878->7878/tcp radarr 27.62 MB (virtual 535.3 MB) 2731688ca5f1 linuxserver/sabnzbd "/init" 6 weeks ago Up 8 days 0.0.0.0:8080->8080/tcp, 0.0.0.0:9090->9090/tcp sabnzbd 339.6 kB (virtual 271.3 MB) 61c0ac7c92c6 linuxserver/deluge "/init" 9 weeks ago Up 8 days deluge 589.8 kB (virtual 172.3 MB) 0cbc1ab01f86 google/cadvisor:latest "/usr/bin/cadvisor -l" 3 months ago Up 8 days 0.0.0.0:8081->8080/tcp cadvisor 0 B (virtual 57.32 MB) cd9f8971ca71 dreamcat4/jackett "/init" 10 months ago Up 8 days 0.0.0.0:9117-9118->9117-9118/tcp jackett-public 18.92 MB (virtual 355.4 MB) e010e90c85bd linuxserver/plexpy "/init" 11 months ago Up 8 days 0.0.0.0:8181->8181/tcp plexpy 128.1 MB (virtual 243 MB) 2386dbb08d28 coppit/duckdns "/bin/sh -c /root/duc" 11 months ago Up 8 days DuckDNS 144 B (virtual 237.6 MB) b6471e1de692 linuxserver/htpcmanager "/init" 11 months ago Up 8 days 0.0.0.0:8085->8085/tcp htpcmanager 31.39 MB (virtual 148.7 MB) Quote Link to comment
assassinmunky Posted September 6, 2017 Share Posted September 6, 2017 what does the usage in /var/lib/docker look like? what is your output of this command? du -h -d 1 /var/lib/docker/ Quote Link to comment
syrys Posted September 7, 2017 Author Share Posted September 7, 2017 (edited) 1 hour ago, assassinmunky said: what does the usage in /var/lib/docker look like? what is your output of this command? du -h -d 1 /var/lib/docker/ This is my output: > du -h -d 1 /var/lib/docker/ 0 /var/lib/docker/tmp 9.6G /var/lib/docker/containers 22G /var/lib/docker/btrfs 12M /var/lib/docker/image 68K /var/lib/docker/volumes 0 /var/lib/docker/trust 84K /var/lib/docker/network 2.4M /var/lib/docker/unraid 0 /var/lib/docker/swarm 32G /var/lib/docker/ Not really sure what this is actually telling me though Edited September 7, 2017 by syrys Quote Link to comment
tillkrueger Posted September 7, 2017 Share Posted September 7, 2017 emptying the Trash in Krusader did not solve me docker.img ballooning issue (now at 86%, growing by about 1% every hour or so). I also found that Krusader did *not* continue the file moving operations after I shout down the computer from which I initiated the file management operations...I built a queue with 2 items in addition to the first item, which was at 98% completed), and found everything this morning exactly where it was last night...the queue was still intact, but had paused, in essence, last night when I shut down my control computer. Krusader also seems to use total file-size when estimating completion...the last 2% (even though calculated at 30mins) are taking many hours, bc they consist mostly of small JPG images from various timelapses I shot. Krusader is amazing, but lots of room for improvement, still. if a moderator wants to move this into a new thread, I think that would be best...otherwise please excuse my deviation from the OP's main-topic! Quote Link to comment
syrys Posted September 7, 2017 Author Share Posted September 7, 2017 28 minutes ago, tillkrueger said: if a moderator wants to move this into a new thread, I think that would be best...otherwise please excuse my deviation from the OP's main-topic! Haha i dont mind. Maybe suggestions you get might help me with my issue too (fingers crossed) 1 Quote Link to comment
tillkrueger Posted September 7, 2017 Share Posted September 7, 2017 thanks, syrys! did you end up increasing the size of your docker.img successfully? I don't seem to be able to find the info on that and my img is now at 90%...I stopped all dockers and am ready to make the change, but can't locate the instructions pertaining to this. Quote Link to comment
tillkrueger Posted September 7, 2017 Share Posted September 7, 2017 well damnit, looks like I'm stuck until I can clear the docker.img and/or increase its size...by a *lot*, then, since I filled up the docker.img within only hours and have so many terabytes worth of data ahead of me that it will take days/weeks. guess I'm in the same shoes as you, syrys, of needing a solution to not ballooning the docker.img in the first place...maybe slightly simpler than your issue, as I *only* run the Krusader docker. Quote Link to comment
bonienl Posted September 7, 2017 Share Posted September 7, 2017 Have you tried unRAID 6.4-rc8? This version has the possibility to limit log size and do log rotation, it would at least help if log files are ballooning. Quote Link to comment
syrys Posted September 7, 2017 Author Share Posted September 7, 2017 3 hours ago, tillkrueger said: did you end up increasing the size of your docker.img successfully? I don't seem to be able to find the info on that and my img is now at 90%...I stopped all dockers and am ready to make the change, but can't locate the instructions pertaining to this. Yeah i did, but to do so, you had to turn off docker. I simply just followed squid's instructions: On 5/21/2017 at 4:16 PM, Squid said: And expanding is simple. Settings - docker - disable docker. Advanced view. Increase the size and reenable docker. Apply For those that know about docker, is there really no way for one to browse into a docker containers file system and inspect/delete a file? Say you want to locate and delete a log fine manually, and you are sure its just being saved inside the docker, is there really no way to browse into the docker's file system and try locate and remove said file(s)? This would solve most of our problems (although hackey, its still a viable solution). Quote Link to comment
tillkrueger Posted September 8, 2017 Share Posted September 8, 2017 thanks for reposting, syrys (and for the original post, Squid)! that's the info I had a hard time finding again...followed them and changed the size to 100GB...probably excessive, but until I find the culprit, I don't wanna be bogged down again for a while. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.