kamhighway

@CHBMB TRURL and BRiT have said that we should be posting about this topic in the support forums for the containers causing the problem. I think we would all be happy to do that if we could figure out which containers are causing the problem. BRiT's suggestion that we start with a clean slate and enable one container at a time until we find the problem will not identify the problem containers unless the trigger for the problem occurs during the 12 hour period you are testing the container. Just because a container runs fine for 12 hours does not mean there is no problem. I think the approach you are taking is a better one. If we could isolate which container has ballooned in size we can then take the discussion to the right support thread and focus on figuring out how to properly configure the containers to avoid the problem.

Yes I have read the replies. You made this recommendation in the past and several of us have tried it and reported that it did not help. Since you repeated your recommendation again, I thought maybe there's a way to use cadvisor that I have not figured out so I thought I'd ask you.

@CHBMB, I tried using cadvisor to figure out which container was growing. cadvisor always reported all of my containers remaining the same size even as the utilization of docker.img grew by GBs/day. If there is a way to use cadvisor to figure out which container is the problem, please tell us how to use it.

@brit Do you have a list of dockers that have passed your tests?

@brit Did you come across any dockers that you stopped using after this process?

@ikosa, How about a way for users to give a thumbs up to a docker so we can all see which dockers are working well for the biggest number of users?

@Brit, You may be correct that the problem is container specific and not a problem with Unraid. However, telling us to post in the support forum for each container is not helpful if we cannot tell which container is causing the problem. @Rara1234, I would start by disabling the last docker you added and seeing if that solves the problem. However, I should tell you that I went down this road and it just so happened that while I had the last-in docker disabled, my daughter went to a girl scouts camping trip so she wasn't doing any transcoding. Consequently, my docker utilization did not grow and the problem appeared to be caused by the docker I disabled. Send me on wild goose chase for a couple of days.

@rara1234, There is more than one reason that can cause the docker utilization rate to skyrocket. In my case, it was needo's Plex docker that by default performs transcoding inside the docker container. I had been using this docker for months without issue. However, after i bought my daughter an ipad, I found that I had to increase my docker image size every few days. At the time, I did not make the connection between the two events and like you I posted in this forum. The people on this forum helped me narrow down the list of suspect dockers. The big breakthrough for me was when I noticed my daughter was spending a lot of time watching her shows on her ipad via plex. Her ipad is the only device that required plex to transcode. This made sense to me because I had been running fine for months before seeing the utilization of docker.img jump by gigabytes some days and then flatline for days. For me, this pattern ruled out an out of control log file as the problem. I found a post JonP wrote a post explaining how to make plex transcode in either ram or on the cache drive. This is what solved my problem. Others have said their problem was due dockers which auto update themselves sometimes leaving dangling containers. And, in at least one case, it seemed that it was an out of control log file. Of the dockers you listed, I would check out emby to see where it performs transcoding. You may have to figure out how to map emby's transcode directory to either ram or your cache drive. As for the other dockers you've installed, we don't overlap on any. However, I am using Binhex's Couchpotato and Sonarr with great success. I've been through several version updates with Binhex's dockers without seeing my docker.img utilization balloon. Binhex posted in this thread some of the things he builds into his dockers to avoid this problem. I wish all docker authors would take as much care as he does.

I am in the same boat as death.hilarious, thaddeussmith, and wizard84. I can install it, get it running, and start a bash session. However, if I enter a beet command that requires a response, beet ignores keyboard presses so I can't respond.

@Squid There are two that we overlap on: Needo-MariaDB, and Needo-Plex. In diagnosing my issues, I've had Needo-MariaDB turned off for days when the utilization of docker.img increased by gigabytes a day, so I'm pretty sure Needo-MariaDB was not causing my problem. Needo-Plex however was the docker that seems to account for most of my problem. The issue is that in order to make Plex transcode outside of docker.img, you have to change setting that is not on the docker setup screen. See my previous post for details. There may be other dockers that have similar application settings that cannot be configured in the docker setup screen. And, there may be some apps that generate huge log files. Maybe we should always map the log files for any docker app to the cache drive?

The ones where we overlap are binhex-sonarr, and binhex-couchpotato. I too have not had any problems with these two. The one docker that I found an issue with was needo-plex, but I think the same issue would arise with any plex docker. The issue with Plex is that in order to transcode outside of the docker.img, you have to change a setting in Plex that is NOT visible in the docker setup screen. Mapping /tmp in the plex docker to /tmp does not solve the problem. LMJ42 mentions this in his post above: 1) Add a volume mapping for /transcode to /tmp (or a location on your cache drive) 2) In Plex, go to Settings -> Server -> Transcoder. Hit Show Advanced. In the "Transcoder temporary directory" put "/transcode" After changing this setting in Plex, my docker utilization has stopped rising by gigabytes a day. It still rises a little each day, but I no longer have to keep increasing the size of docker.img every few days. It may be that there are other dockers that have a similar setting in the application that cannot be changed in the docker setup screen. If so, it would be a good idea to make a list of such applications and let others know where the relevant settings are in each app. Brit's list of dockers may be particularly useful. Since he doesn't share our problem, apparently all of his dockers can be configured correctly from the docker settings screens. This is useful information for the rest of us to know.

@Brit I posted in here to see if anyone else has the same problem so we can collaborate to fix it. If you don't have the problem, that's great. But telling those of us that do have the problem that its our problem and we have to fix it ourselves is not helpful. What would be helpful is if you could tell us which dockers you are using that have not given you any problems so we can possibly eliminate them from the list of potentially misbehaving dockers.

@PhAze: Thank you for your post. I can believe that there is more than one thing that can cause docker utilization to ramp up. In my case, setting Plex to transcode from the cache drive seems to have solved most of the problem. The img is still growing, albeit at a much slower rate. I did not know of the possibility of a "dangling image" so I'll look for that the next time I see a big jump in utilization of docker.img. It seems to me that we are part of a very small group that is experiencing this problem. Makes me wonder what I'm doing differently from most other users.

The container does shrink, but it does not seem to free up the space in docker.img. We either need to figure out how to shrink docker.img when a docker is done using space in /tmp. Or, we need to move /tmp outside of docker.img by mapping it to the cache drive.

The /tmp folder in the dockers may be empty now since the apps do a pretty good job of cleaning up after themselves. I suspect that what is happening is that utilization of docker.img goes up when a dockerized app is using /tmp. However, after the app cleans up /tmp, utilization of docker.img does not go down. This would explain why the size of the containers does not change eventhough docker.img utilization increases. If I am right, then mapping every docker's /tmp to the cache drive would be an easy solution.

The problem is most evident when a dockerized app uses /tmp for big files such as transcoding in Plex. Unraring in sab may be another good example. In the sab docker, try mapping /tmp to /mnt/cache/appdata/sabnzbd/tmp. I am considering doing this for all my dockers so they can all use /tmp without utilizing docker.img. I'd be curious to know if this works for you as I am not 100% sure I've diagnosed the primary issue correctly.

Update: It's been about 4 days now since I changed my Plex settings to perform transcoding in RAM and the amount of space utilized in my docker.img has changed only slightly. This result is consistent with my theory that allowing Plex to transcode inside its container was the problem. I don't think the problem is limited to Plex. Any container that runs an app that uses lots of tmp space is likely to exhibit the same problem. Maybe it should be standard practice to map every docker's /tmp directory to a share on the cache drive or to ram. Now that I'm fairly sure the Plex docker was my problem, here's how we could collaborate to be sure. If you are using a Plex docker, please post your answers to the following questions: 1) Which Plex Docker? Needo, Linux.io, BinHex, etc. 2) Did you follow JonP's instructions to set Plex to transcode either in RAM, Cache, or a Share? 3) Do you know for sure that someone watches content that Plex has to transcode? 4) Does your utilization of docker.img increase in big jumps (hundreds of MB/day)?

@TexasDave, I never get the message that disk utilization has returned to normal.

Then that's probably where its mounted. Like I said at work so couldn't check. @Squid Thanks again. I'll monitor the size of the directories at /var/lib/docker. The next time I see a big jump in utilization of docker.img, I should now be able to see what is growing. This is a big help.

@squid Is docker.img mounted to /var/lib/docker/unraid or one level up at /var/lib/docker? It looks to me like the containers are one level up at /var/lib/docker. At that level I see: /containers of size 768, and /graph of size 16520. Does this sound right to you? root@Media:/var/lib/docker# ls -l total 20 drwx------ 1 root root 20 Sep 22 16:17 btrfs/ drwx------ 1 root root 768 Sep 29 10:49 containers/ drwx------ 1 root root 16520 Sep 27 15:06 graph/ drwx------ 1 root root 32 Sep 22 16:16 init/ -rw-r--r-- 1 root root 5120 Sep 29 10:49 linkgraph.db -rw------- 1 root root 604 Sep 29 10:49 repositories-btrfs drwx------ 1 root root 0 Sep 27 15:06 tmp/ drwx------ 1 root root 0 Sep 22 16:16 trust/ drwxrwxrwx 1 root root 112 Oct 1 12:03 unraid/ -rw-rw-rw- 1 root root 71 Oct 1 12:03 unraid-autostart -rw-rw-rw- 1 root root 166 Sep 29 10:49 unraid-update-status.json drwx------ 1 root root 256 Sep 27 14:43 volumes/

@squid, thanks for your help. This has been the most progress I've had in understanding what is going on in a week. At that location, I see a /tmp folder that contains dozens of files with names like tmp-1018085421.url which mc reports to be of size 714424 (I don't know what units this is in). However, I see that /community.applications is of size 60, so I'm guessing the other file is very much bigger. The oldest tmp file seems to correspond to the date I last rebuilt docker.img. Here is a list of what's in /tmp inside docker.img. root@Media:/tmp# ls -l total 50472 drwxrwxrwx 3 root root 60 Sep 23 07:29 community.applications/ drwx------ 2 root root 40 Sep 24 16:59 mc-root/ drwxr-xr-x 4 root root 80 Sep 23 07:27 notifications/ drwxr-xr-x 2 root root 200 Sep 28 23:16 plugins/ -rw-rw-rw- 1 root root 0 Sep 28 10:00 preclear_assigned_disks1 -rw-rw-rw- 1 root root 1680 Sep 28 10:00 preclear_report_sda -rw-rw-rw- 1 root root 168 Sep 28 10:00 preclear_stat_sda -rw-rw-rw- 1 root root 10 Sep 28 10:00 read_speedsda -rw-rw-rw- 1 root root 4571 Sep 28 10:00 smart_finish_sda -rw-rw-rw- 1 root root 4571 Sep 27 09:55 smart_mid_after_zero1_sda -rw-rw-rw- 1 root root 144 Sep 27 09:55 smart_mid_pending_reallocate_sda -rw-rw-rw- 1 root root 4571 Sep 26 23:02 smart_mid_preread1_sda -rw-rw-rw- 1 root root 4574 Sep 26 09:22 smart_start_sda -rw-rw-rw- 1 root root 705595 Sep 23 13:47 tmp-1001038311.url -rw-rw-rw- 1 root root 714424 Sep 27 14:46 tmp-1018085421.url -rw-rw-rw- 1 root root 711853 Sep 25 10:04 tmp-1022176830.url -rw-rw-rw- 1 root root 711853 Sep 25 09:58 tmp-1026715570.url -rw-rw-rw- 1 root root 714424 Sep 28 09:40 tmp-1037033344.url -rw-rw-rw- 1 root root 711853 Sep 27 05:01 tmp-1041957111.url -rw-rw-rw- 1 root root 711853 Sep 25 09:57 tmp-104351457.url -rw-rw-rw- 1 root root 705595 Sep 23 11:43 tmp-1079397565.url -rw-rw-rw- 1 root root 714424 Sep 27 15:08 tmp-1114330130.url -rw-rw-rw- 1 root root 714554 Sep 28 23:16 tmp-1115295570.url -rw-rw-rw- 1 root root 714424 Sep 27 15:09 tmp-1115779883.url -rw-rw-rw- 1 root root 714554 Sep 29 10:48 tmp-1184253690.url -rw-rw-rw- 1 root root 714938 Oct 1 08:22 tmp-1244214065.url -rw-rw-rw- 1 root root 714424 Sep 28 02:27 tmp-124625526.url -rw-rw-rw- 1 root root 705595 Sep 23 11:07 tmp-1257712804.url -rw-rw-rw- 1 root root 711853 Sep 26 22:48 tmp-1262417929.url -rw-rw-rw- 1 root root 705595 Sep 23 11:26 tmp-1266159711.url -rw-rw-rw- 1 root root 705595 Sep 23 11:07 tmp-1303099892.url -rw-rw-rw- 1 root root 705595 Sep 23 11:17 tmp-1304736983.url -rw-rw-rw- 1 root root 714424 Sep 27 15:07 tmp-1320279264.url -rw-rw-rw- 1 root root 714424 Sep 27 14:44 tmp-137839263.url -rw-rw-rw- 1 root root 711853 Sep 26 14:04 tmp-1394325280.url -rw-rw-rw- 1 root root 714424 Sep 27 14:47 tmp-1475139423.url -rw-rw-rw- 1 root root 714554 Sep 29 10:48 tmp-1478797206.url -rw-rw-rw- 1 root root 711853 Sep 26 22:58 tmp-1502409291.url -rw-rw-rw- 1 root root 714554 Sep 28 10:04 tmp-1527394120.url -rw-rw-rw- 1 root root 714554 Sep 28 23:16 tmp-1548075922.url -rw-rw-rw- 1 root root 711853 Sep 26 22:47 tmp-1565010257.url -rw-rw-rw- 1 root root 714554 Sep 28 10:05 tmp-1584110921.url -rw-rw-rw- 1 root root 705595 Sep 23 07:30 tmp-160169679.url -rw-rw-rw- 1 root root 714424 Sep 27 15:21 tmp-1661075416.url -rw-rw-rw- 1 root root 711853 Sep 25 10:04 tmp-1672534634.url -rw-rw-rw- 1 root root 705595 Sep 23 11:43 tmp-1701087208.url -rw-rw-rw- 1 root root 714554 Sep 28 15:39 tmp-170252017.url -rw-rw-rw- 1 root root 708172 Sep 24 22:10 tmp-1731846235.url -rw-rw-rw- 1 root root 714554 Sep 29 10:49 tmp-1776699683.url -rw-rw-rw- 1 root root 711853 Sep 25 10:04 tmp-1794736320.url -rw-rw-rw- 1 root root 705595 Sep 23 11:17 tmp-1798316926.url -rw-rw-rw- 1 root root 705595 Sep 23 11:23 tmp-1867079391.url -rw-rw-rw- 1 root root 714938 Sep 30 18:02 tmp-1892408965.url -rw-rw-rw- 1 root root 711853 Sep 27 09:44 tmp-1992924892.url -rw-rw-rw- 1 root root 714424 Sep 27 15:06 tmp-2004304119.url -rw-rw-rw- 1 root root 705595 Sep 23 07:34 tmp-2058323394.url -rw-rw-rw- 1 root root 705595 Sep 23 07:44 tmp-2087387426.url -rw-rw-rw- 1 root root 714424 Sep 27 14:44 tmp-2088655953.url -rw-rw-rw- 1 root root 714554 Sep 28 15:39 tmp-2136823997.url -rw-rw-rw- 1 root root 714554 Sep 29 10:50 tmp-2143980462.url -rw-rw-rw- 1 root root 705595 Sep 23 10:56 tmp-238162114.url -rw-rw-rw- 1 root root 714424 Sep 27 15:07 tmp-244130472.url -rw-rw-rw- 1 root root 711853 Sep 26 11:22 tmp-297073847.url -rw-rw-rw- 1 root root 714424 Sep 27 14:37 tmp-307181750.url -rw-rw-rw- 1 root root 714554 Sep 29 13:18 tmp-319612520.url -rw-rw-rw- 1 root root 708172 Sep 24 05:56 tmp-364760669.url -rw-rw-rw- 1 root root 711853 Sep 26 14:23 tmp-381722333.url -rw-rw-rw- 1 root root 705595 Sep 23 07:29 tmp-45712661.url -rw-rw-rw- 1 root root 705595 Sep 23 07:37 tmp-46253092.url -rw-rw-rw- 1 root root 714554 Sep 28 23:17 tmp-489865132.url -rw-rw-rw- 1 root root 714424 Sep 27 14:46 tmp-49543768.url -rw-rw-rw- 1 root root 705595 Sep 23 07:30 tmp-532269974.url -rw-rw-rw- 1 root root 711853 Sep 27 09:48 tmp-54267583.url -rw-rw-rw- 1 root root 714554 Sep 29 06:36 tmp-569650762.url -rw-rw-rw- 1 root root 714554 Sep 29 07:31 tmp-595762458.url -rw-rw-rw- 1 root root 705595 Sep 23 07:31 tmp-626615610.url -rw-rw-rw- 1 root root 714424 Sep 27 14:46 tmp-741555393.url -rw-rw-rw- 1 root root 711853 Sep 26 11:20 tmp-771294825.url -rw-rw-rw- 1 root root 705595 Sep 23 07:36 tmp-793902811.url -rw-rw-rw- 1 root root 714554 Sep 28 10:05 tmp-870656813.url -rw-rw-rw- 1 root root 708172 Sep 24 13:29 tmp-87708617.url -rw-rw-rw- 1 root root 714424 Sep 27 14:43 tmp-926217946.url -rw-rw-rw- 1 root root 714424 Sep 27 14:37 tmp-9295676.url -rw-rw-rw- 1 root root 714938 Sep 30 19:15 tmp-933241343.url -rw-rw-rw- 1 root root 705595 Sep 23 11:22 tmp-970283086.url drwx------ 2 root root 60 Sep 26 09:22 tmux-0/ -rw-rw-rw- 1 root root 263768 Sep 27 09:55 zerosda Couple of questions: 1) Do you have any idea what is generating these tmp files? 2) If I did not rebuild docker.img, would these tmp files get purged from docker.img on their own?

Thanks Squid. It that where docker.img is mounted in unraid's filesystem?

The Community Applications plugin stores some files in it. Total space is ~2 Meg. This is interesting. So we can confirm that there are other things in docker.img than the containers. Is there any way to see what is in docker.img?

Plex may not be the the only app that temporarily uses large amounts of diskspace. Any media server that does transcoding is likely to store big temporary files in the container. I see you have Serviio installed. I don't have any experience with Serviio, but if it does transcoding it could have the same problem I am thinking that Plex has. In my case, I suspect Plex is the problem because the amount of diskspace utilized in docker.img sometimes jumps by 1 or 2 gb a day, and sometimes it is relatively stable for days at a time. 1 to 2 gbs / day is way more than a container's log files would account for. And, I noticed that the growth spurts seem to coincide with the days my daughter watched a lot of anime's on her ipad. I don't know why cadvisor does not show the size of the Plex container growing. Maybe cadvisor does not count temp files as part of the container's size? Maybe cadvisor is reporting the size of the image since that would be the container size when the container is first started. My container sizes, as reported by cadvisor, have been constant while my docker utilization continues to climb. Questions for Lime Technology: 1) When a docker app creates temporary files in the container, does the utilization of docker.img increase? 2) When a docker app deletes temporary files in the container, does the utilization of docker.img decrease? 3) What else, besides containers, is stored in docker.img?

@trurl Yes, it had no effect on the utilization of docker.img.