Popple2000 Posted October 3, 2015 Share Posted October 3, 2015 Will do as soon as I get a chance, thanks ! Quote Link to comment
kamhighway Posted October 3, 2015 Share Posted October 3, 2015 @TexasDave, I never get the message that disk utilization has returned to normal. Quote Link to comment
johnsanc Posted October 4, 2015 Share Posted October 4, 2015 Just adding that you all are not alone. I also have a continuously growing docker.img file and cannot seem to figure out the cause. I can reboot to get it to return to normal, but it quickly grows to capacity in a matter of days. Quote Link to comment
kamhighway Posted October 5, 2015 Share Posted October 5, 2015 Update: It's been about 4 days now since I changed my Plex settings to perform transcoding in RAM and the amount of space utilized in my docker.img has changed only slightly. This result is consistent with my theory that allowing Plex to transcode inside its container was the problem. I don't think the problem is limited to Plex. Any container that runs an app that uses lots of tmp space is likely to exhibit the same problem. Maybe it should be standard practice to map every docker's /tmp directory to a share on the cache drive or to ram. Now that I'm fairly sure the Plex docker was my problem, here's how we could collaborate to be sure. If you are using a Plex docker, please post your answers to the following questions: 1) Which Plex Docker? Needo, Linux.io, BinHex, etc. 2) Did you follow JonP's instructions to set Plex to transcode either in RAM, Cache, or a Share? 3) Do you know for sure that someone watches content that Plex has to transcode? 4) Does your utilization of docker.img increase in big jumps (hundreds of MB/day)? Quote Link to comment
PhAzE Posted October 6, 2015 Share Posted October 6, 2015 Im getting this same problem but im not using plex as a docker. I haven't had time to inspect yet but i did notice sonarr had a queue of shows to process but couldn't until the docker was restarted, that or sab. Ill verify but i also suspect sab may be unpacking rar files in the tmp folder during the unrar process and crashing if the docker image fills. Quote Link to comment
kamhighway Posted October 6, 2015 Share Posted October 6, 2015 Im getting this same problem but im not using plex as a docker. I haven't had time to inspect yet but i did notice sonarr had a queue of shows to process but couldn't until the docker was restarted, that or sab. Ill verify but i also suspect sab may be unpacking rar files in the tmp folder during the unrar process and crashing if the docker image fills. The problem is most evident when a dockerized app uses /tmp for big files such as transcoding in Plex. Unraring in sab may be another good example. In the sab docker, try mapping /tmp to /mnt/cache/appdata/sabnzbd/tmp. I am considering doing this for all my dockers so they can all use /tmp without utilizing docker.img. I'd be curious to know if this works for you as I am not 100% sure I've diagnosed the primary issue correctly. Quote Link to comment
PhAzE Posted October 6, 2015 Share Posted October 6, 2015 I had a look ag the tmp folder for sonarr sab and ubooquity and they are all pretty clean. Nothing stands out as bloat yet my docker is filling up still. Needs more investigating. Quote Link to comment
kamhighway Posted October 6, 2015 Share Posted October 6, 2015 I had a look ag the tmp folder for sonarr sab and ubooquity and they are all pretty clean. Nothing stands out as bloat yet my docker is filling up still. Needs more investigating. The /tmp folder in the dockers may be empty now since the apps do a pretty good job of cleaning up after themselves. I suspect that what is happening is that utilization of docker.img goes up when a dockerized app is using /tmp. However, after the app cleans up /tmp, utilization of docker.img does not go down. This would explain why the size of the containers does not change eventhough docker.img utilization increases. If I am right, then mapping every docker's /tmp to the cache drive would be an easy solution. Quote Link to comment
PhAzE Posted October 6, 2015 Share Posted October 6, 2015 Ok like the container size grows but doesn't shrink once expanded. Sounds like we need a way to shrink containers via cron or something. Quote Link to comment
kamhighway Posted October 6, 2015 Share Posted October 6, 2015 Ok like the container size grows but doesn't shrink once expanded. Sounds like we need a way to shrink containers via cron or something. The container does shrink, but it does not seem to free up the space in docker.img. We either need to figure out how to shrink docker.img when a docker is done using space in /tmp. Or, we need to move /tmp outside of docker.img by mapping it to the cache drive. Quote Link to comment
BRiT Posted October 6, 2015 Share Posted October 6, 2015 Or there needs to be an option to not use docker.img but have it stored in an actual subdirectory in the cache drive. Quote Link to comment
ikosa Posted October 12, 2015 Share Posted October 12, 2015 If anybody is interested I found some tools to monitor and track docker containers. Prometheus, container_exporter: http://rancher.com/docker-monitoring-continued-prometheus-and-sysdig/ And Prometheus can also get metrics from cadvisor. https://labs.ctl.io/monitoring-docker-services-with-prometheus/ I can't play enough to achieve my goal. Interestingly can't find a metric about container sizes... Quote Link to comment
shooga Posted October 22, 2015 Share Posted October 22, 2015 I have this issue too and have all of my temp folders for sabnzbd and plex pointing outside docker to my cache. I've already expanded by docker image once and would like to avoid doing that repeatedly. Need to find a solution here. Quote Link to comment
PhAzE Posted October 24, 2015 Share Posted October 24, 2015 So after updating all of my dockers to map /tmp out to /tmp (in ram) the docker is still growing only slightly slower now. I think that may have been only a partial solution. I can see some apps are indeed leaving files in /tmp (like Calibre, and Sonarr). I believe some of these apps are also growing log files in /var/log in each container, but i'm not sure if it's enough to bloat 10GB of docker.img space. Quote Link to comment
PhAzE Posted October 24, 2015 Share Posted October 24, 2015 I just wanted to point out, if you need to clear some space in the image and happen to have dangling images, it's best to remove them. First, run: docker images -f "dangling=true" and if you see anything listed then you have unused images that can be cleared out with the following code: docker rmi $(sudo docker images -f "dangling=true" -q) For me, that got my image back into acceptable utilization levels. Enough to clear the unRaid alert for over utilization. If docker updates are in fact leaving images behind, then I suppose you could 'cron' this for a weekly run to clear out leftover images. That is assuming that your containers themselves are not growing. In my case, I found that after mapping /tmp to /tmp for all the containers, the images still grew, but likely due to application auto updates which were in fact leaving empty image files after the updates. 1 Quote Link to comment
kamhighway Posted October 25, 2015 Share Posted October 25, 2015 @PhAze: Thank you for your post. I can believe that there is more than one thing that can cause docker utilization to ramp up. In my case, setting Plex to transcode from the cache drive seems to have solved most of the problem. The img is still growing, albeit at a much slower rate. I did not know of the possibility of a "dangling image" so I'll look for that the next time I see a big jump in utilization of docker.img. It seems to me that we are part of a very small group that is experiencing this problem. Makes me wonder what I'm doing differently from most other users. Quote Link to comment
BRiT Posted October 25, 2015 Share Posted October 25, 2015 You're likely using dockers that auto-update. Each time the main program updates in your container the image usage will grow. I do not use auto-updating programs and my docker image usage has been constant since the initial setup. Quote Link to comment
shooga Posted October 25, 2015 Share Posted October 25, 2015 I took a look and didn't have any dangling images, so that didn't help. Is there a way to view the image sizes? cAdvisor seems to only report the size of the default image because if I add up the numbers that it shows the total is way short of what shows in my docker settings page. Quote Link to comment
shooga Posted November 1, 2015 Share Posted November 1, 2015 An update for anyone monitoring this thread or searching in the future: I used the command "docker exec -it <container name> bash" to drop into the container and then this, "find / -xdev -type f -size +100M" to find files over 100MB. This helped me find that I had a 1.34GB error log in three of my containers! (eroz's AirVideoServer, needo's PlexMediaServer, and needo's PlexWatch). Deleting that file is an easy temporary fix, but I still need to figure out a long term solution. Any idea what would be causing this error: Nov 1 08:10:18 [server name] sshd[435]: error: Bind to port 22 on 0.0.0.0 failed: Address already in use. Nov 1 08:10:18 [server name] sshd[435]: fatal: Cannot bind any address. I'm guessing it's because ssh is already open on the server. How could I fix this? Quote Link to comment
PhAzE Posted November 3, 2015 Share Posted November 3, 2015 i didn't have any large logs in any of my dockers. It appears my utilization is still growing, even with TMP mapped to native TMP drive, and dangling images gone. Docker containers do not appear to be growing in size, so that leaves me to believe that it is an issue with something outside the containers. Perhaps docker has it's own log file outside the containers but still in the image, or maybe there are history entries growing the image usage. Quote Link to comment
hoosierdaddy Posted November 4, 2015 Share Posted November 4, 2015 I'm also getting this Docker Utilization alert. Mine is actually going up to 98% sometimes before then dropping down to normal. The only Docker I'm using is needo/Plex. During this time, if I'm using PLEX, the movie will stop playing like the file has an error. I can then normally restart it and it will play. Frustrating. I need to pay attending to exactly what files it is happening on. I setup to have the transcoding done on my cache drive. This seems to help but I still have the problem... Not installing any other dockers until I get this figured out. Quote Link to comment
Earache Posted November 8, 2015 Share Posted November 8, 2015 I'm also getting this and I have no idea how to fix it. Is there a solution? linuxserver/couchpotato latest 29d0943b24c0 2 weeks ago 502 MB linuxserver/quassel-core latest 0a9f2f07a76f 3 weeks ago 292.6 MB linuxserver/plexpy latest bbda2e6f2f71 3 weeks ago 314.8 MB linuxserver/plex latest 3cc54bb12b37 3 weeks ago 583.4 MB google/cadvisor latest c0e56eeefd42 5 weeks ago 22.77 MB nativedude/plex-requests latest 94ea7dbfd58d 3 months ago 569.1 MB <none> <none> 14a06126bb6c 3 months ago 282.9 MB needo/nzbdrone latest 132b9d473a16 3 months ago 440.4 MB gfjardim/nzbget latest 2accc56105d5 3 months ago 529.7 MB I also don't know what <none> is. Quote Link to comment
PhAzE Posted November 10, 2015 Share Posted November 10, 2015 I'm also getting this and I have no idea how to fix it. Is there a solution? linuxserver/couchpotato latest 29d0943b24c0 2 weeks ago 502 MB linuxserver/quassel-core latest 0a9f2f07a76f 3 weeks ago 292.6 MB linuxserver/plexpy latest bbda2e6f2f71 3 weeks ago 314.8 MB linuxserver/plex latest 3cc54bb12b37 3 weeks ago 583.4 MB google/cadvisor latest c0e56eeefd42 5 weeks ago 22.77 MB nativedude/plex-requests latest 94ea7dbfd58d 3 months ago 569.1 MB <none> <none> 14a06126bb6c 3 months ago 282.9 MB needo/nzbdrone latest 132b9d473a16 3 months ago 440.4 MB gfjardim/nzbget latest 2accc56105d5 3 months ago 529.7 MB I also don't know what <none> is. NONE is a dangling (left over but unused) container. Run this to remove it: docker rmi $(sudo docker images -f "dangling=true" -q) ------------------------------------------------------------------------ My utilization i still going up, and when i compated my container sizes from 3 weeks ago to now, no change, but I've gone up 30% utilization (which is 3gb out of my 10gb. No temp or log files remain in the containers, and I just can't figure out what is making it grow. Quote Link to comment
ljm42 Posted November 11, 2015 Share Posted November 11, 2015 I'm also getting this Docker Utilization alert. Mine is actually going up to 98% sometimes before then dropping down to normal. The only Docker I'm using is needo/Plex. During this time, if I'm using PLEX, the movie will stop playing like the file has an error. I can then normally restart it and it will play. Frustrating. I need to pay attending to exactly what files it is happening on. I setup to have the transcoding done on my cache drive. This seems to help but I still have the problem... Not installing any other dockers until I get this figured out. It really sounds like the transcoding is still being done in the docker image. There are two steps to moving it outside of the docker image: 1) Add a volume mapping for /transcode to /tmp (or a location on your cache drive) 2) In Plex, go to Settings -> Server -> Transcoder. Hit Show Advanced. In the "Transcoder temporary directory" put "/transcode" That should solve it for you. Edit - I expanded on this idea and added it to the Docker FAQ If Plex is your only Docker, and transcoding really is happening outside of the docker image, and you are still getting to 98%, then your docker image is probably just too small. Mine is 20gb. Quote Link to comment
B_Sinn3d Posted November 11, 2015 Share Posted November 11, 2015 It just got worse...I previously set image from 20GB to 30GB, my image exploded last night and completely filled up 100%. I restarted my docker and now I have 5 orphaned images...grrr. Has LT recognized this as an issue yet? I can't keep expanding my image as a workaround. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.