Cleaning out Docker image

[MagicMountainMan]

Sorry, yes I did map it as /tmp.

[rara1234]

unRAID OS Version: 6.1.3

 

Description: I keep getting warnings about "Event: Docker high image disk utilization". Typically, docker runs fine for a few weeks, then suddenly these start and utilization reported goes up from 70% to 100% in a few days. If it gets to 100% the webgui crashes. There are no entries in the logs, and docker.log contains only this:

 

time="2015-11-18T07:12:18.238256056Z" level=fatal msg="Error starting daemon: Unable to open the database file: unable to open database file"

 

How to reproduce: Just usign docker. with teh following apps:

needo/couchpotato:latest

gfjardim/crashplan:latest

gfjardim/cups:latest

emby/embyserver:latest

pinion/docker-mylar:latest

linuxserver/sabnzbd:latest

linuxserver/sonarr:latest

 

 

Expected results: Docker image utilization does not approach 100% over time

 

Actual results: Docker image utilization approaches 100% and the image file needs to be destroyed and recreated to recover service.

 

Other information: this post has had over 4,400 views, so clearly others are also seeing it: https://lime-technology.com/forum/index.php?topic=42654.45

[rara1234]

i couldn't see that anyone else has raised a defect (at least none show up in search) so i raised on here:

http://lime-technology.com/forum/index.php?topic=44816.0

[trurl]

You haven't really given us enough information to reproduce. Most specifically, you haven't given us any details about how you have any of those dockers configured. I suspect your problem is related to the way you have configured one or more of your dockers, and it likely is the same for most in this thread.

 

Not really a defect report so I have merged it back in to this thread where you can give us more details.

[rara1234]

You haven't really given us enough information to reproduce. Most specifically, you haven't given us any details about how you have any of those dockers configured. I suspect your problem is related to the way you have configured one or more of your dockers, and it likely is the same for most in this thread.

 

Not really a defect report so I have merged it back in to this thread where you can give us more details.

 

that's not really very useful, and although I know you're trying to be helpful by keeping the forums clear of junk, with 4833 views on this thread this is clearly an issue for a large number of people. It would be really great if you could help figure out how to analyse or troubleshoot this issue, rather than closing defect reports with an "it's your fault" response without even letting lime tech see the issue report. What, specifically, do you think we need? a specific log file? the outcome of a specific command? Can you actually assist with this issue or highlight it to someone who can? It's already been dragging on for months.

 

I think the main things that I note are:

1. whatever docker containers everyone here is using, interrogating utilisation within the image file shows that consumed space is well below the docker allocated storage reported by unraid.

2. for many people, things appear stable then suddenly rocket up 20-40% within a few hours

3. there are a few containers which appear to be commonly used, but no containers which everyone is using.

 

What would you propose the next steps are to progress this issue?

 

[kamhighway]

@rara1234,

 

There is more than one reason that can cause the docker utilization rate to skyrocket.  In my case, it was needo's Plex docker that by default performs transcoding inside the docker container. I had been using this docker for months without issue. However, after i bought my daughter an ipad, I found that I had to increase my docker image size every few days. At the time, I did not make the connection between the two events and like you I posted in this forum. The people on this forum helped me narrow down the list of suspect dockers. The big breakthrough for me was when I noticed my daughter was spending a lot of time watching her shows on her ipad via plex. Her ipad is the only device that required plex to transcode. This made sense to me because I had been running fine for months before seeing the utilization of docker.img jump by gigabytes some days and then flatline for days. For me, this pattern ruled out an out of control log file as the problem. I found a post JonP wrote a post explaining how to make plex transcode in either ram or on the cache drive.  This is what solved my problem. 

 

Others have said their problem was due dockers which auto update themselves sometimes leaving dangling containers. And, in at least one case, it seemed that it was an out of control log file.

 

Of the dockers you listed, I would check out emby to see where it performs transcoding. You may have to figure out how to map emby's transcode directory to either ram or your cache drive.

 

As for the other dockers you've installed, we don't overlap on any.  However, I am using Binhex's Couchpotato and Sonarr with great success. I've been through several version updates with Binhex's dockers without seeing my docker.img utilization balloon.  Binhex posted in this thread some of the things he builds into his dockers to avoid this problem.  I wish all docker authors would take as much care as he does.

 

[rara1234]

thank for the helpful reply. I have transcoding disabled in emby. I guess the issue is that I simply don't know how to narrow down the issue e.g. how to work out which container is causing the issue without disabling them one by one and waiting 3 months - but that's clearly not an option! And as far as i can tell, when storage is released within a container it should be released in the image too, which apparently isn't happening, and that's where i think the defect lies - badly configured apps within containers shouldn't be able to permanently consume storage...

[BRiT]

Those issues are not an unRaid defect but a defect in the particular docker container or the user's configuration of docker.

 

As such it does not belong in the general unraid defect issue forum but in each individual docker container thread.

[rara1234]

sorry - i updated my answer after you'd replied to add that:

1. I dont know how to figure out which container is causing it without disabling them one by one for 3 months at a time (the average time it takes for it to be noticeable in my case), and

2.I would have expected that a badly configured app within a container shouldn't be able to permanently consume image storage i.e. the container should be reallocating storage before consuming and then not releasing more image storage.

[kamhighway]

@Brit,

 

You may be correct that the problem is container specific and not a problem with Unraid.  However, telling us to post in the support forum for each container is not helpful if we cannot tell which container is causing the problem.

 

@Rara1234,

 

I would start by disabling the last docker you added and seeing if that solves the problem. However, I should tell you that I went down this road and it just so happened that while I had the last-in docker disabled, my daughter went to a girl scouts camping trip so she wasn't doing any transcoding. Consequently, my docker utilization did not grow and the problem appeared to be caused by the docker I disabled. Send me on wild goose chase for a couple of days.

[ikosa]

We need a script to track usage of containers and (not mapped)directories in containers

[kamhighway]

@ikosa,

 

How about a way for users to give a thumbs up to a docker so we can all see which dockers are working well for the biggest number of users?

[trurl]

... with 4833 views on this thread this is clearly an issue for a large number of people...

To some extent I am judging this based on who isn't reporting the problem rather than how many people are viewing the thread. I'm sure many people view threads just to see if they can help. A quick skim back over the whole thread looks like those posting they have the problem and those trying to help are roughly equal.

 

I'm not committed to the view that this isn't a defect, I just haven't seen any evidence that it is.

 

Simplify your setup by starting with a new docker.img and a single docker, and then add things back a little at a time with a period of observation at each step. Maybe start with the docker you need the most, or the one you suspect the most.

[gshlomi]

Got the same problem, have been following silently this thread waiting for a solution.

For now, I hope someone will post a way to view the size of the different containers within the docker image.

AFAIKT, cAsvisor isn't reporting sizes correctly...

[BRiT]

I've been running on the same docker image for close to 5 months now if not longer and it's at exactly the same size as it was when I finished adding my own docker images into it.

 

I do not use any dockers that auto-update.

I do not use any dockers that transcode inside the container.

I do not use any dockers that log inside the container.

 

I run applications of eggdrop, nzbget, transmission, and pytivo.

 

I can guarantee there is not a bug in unRAID relating to docker images growing. If your docker image is growing it is absolutely because of a misbehaving container or a misconfigured container.

 

[rara1234]

...If your docker image is growing it is absolutely because of a misbehaving container or a misconfigured container.

 

i'd be happy to concede that if i could show it - but not using functionality which is important to my family on a daily basis for months at a time isn't a viable approach. Without this, how can i figure out which one it is? I mean, there must be some better way (i.e. some systematic/analytical approach using e.g. logs, tools to interrogate the system etc.) which can be used to narrow it down to the container(s) which are causing this? 

[BRiT]

There have been other suggestions already earlier in this thread, involving "docker exec -it bash" and doing directory listings with "df -h" or the like.

 

Here is one means of systematic troubleshooting, and pretty much what I used early on the 6.0 beta period to track down what my troubled containers were:

 

0. Remove all dockers.

1. Add a docker to your setup.

2. Observe size.

3. Use as docker container as you regularly would for 12 hours or so.

4. Restart your dockers.

4. Observe new size, if excessive grows you found a troubled container

5. If less than 3 observed times, goto step 3.

6. Goto Step 1 if more docker containers are available.

 

 

 

 

 

 

 

 

[kamhighway]

@brit

 

Did you come across any dockers that you stopped using after this process?

 

 

 

 

[BRiT]

No. However, I have stopped updating the main application inside the docker containers. There is absolutely no need to be on the bleeding edge with the latest developer version of an application.

 

I only have dockers for my immediate needs: pytivo, usenet, torrent, and an irc bot. I have not evaluated any plex or emby containers, yet. I have a working and extremely stable setup so I'm not tempted to change it.

 

 

The one docker I did have major issues with was my own creation for eggdrop container where I had the daemon starting incorrectly which caused it to spam the docker log file and chew up hundreds of megs of space each day.

[kamhighway]

@brit

 

Do you have a list of dockers that have passed your tests?

[trurl]

... but not using functionality which is important to my family on a daily basis for months at a time isn't a viable approach...

How did your family get along before docker?

 

The dockers I have been running 24/7 for many, many months without this problem are in my sig.

 

But I have had these applications working for years before I ever built my unRAID server. They were running on my PC, saving data either locally or to some other NAS.

 

If I needed to I could go back to that approach while I tried to troubleshoot individual dockers on my unRAID.

[rara1234]

... but not using functionality which is important to my family on a daily basis for months at a time isn't a viable approach...

How did your family get along before docker?

 

If I needed to I could go back to that approach while I tried to troubleshoot individual dockers on my unRAID.

 

I know what you mean - but I suspect the world might fall apart without Thomas and friends, pepper pig and regular updates on the comings and going a of the kardashians

 

Previously I had a separate machine running all of these apps. I don't have it any more. I could set these up on a vm but I suppose, but again, I'm sure there is some way to identify where the allocated storage has gone, we just don't know it yet.

 

If I get time over Christmas I will try to rebuild the docker image one at a time. The problem is that it works fine for months then I get 30-50% growth (on a 10gb image) in a day or two, completely at random.

[jevans04]

I had issues with my docker image growing larger periodically. I narrowed it down to CouchPotato and I believe it was because it was auto updating inside the container. I switched to the LinuxServer version and disabled auto updates in the CouchPotato settings. Problem completely solved for me.

 

The way I narrowed it down to a specific app was to remove each app one at at time (container and image) and then reinstalled from my template. Each time looking at the docker image size reported on the unRAID Dashboard. For all my other apps the size before and after didn't change much. For CouchPotato the image size was significantly smaller afterward the reinstall.

[CHBMB]

cadvisor container will show container sizes and help narrow it down.

 

I test a lot of containers and am not seeing this problem myself although I was suspicious at one point I may have been.

[Squid]

cadvisor container will show container sizes and help narrow it down.

 

I test a lot of containers and am not seeing this problem myself although I was suspicious at one point I may have been.

Succumb to marketing you have