Docker high image disk utilization: why is my docker image/disk getting full?


Recommended Posts

1 hour ago, Goguleanu said:

It would be a lot more helpful if you point to the actual links or the post that has them. 

But, thanks! 

You don't seem to be eager to help yourself. If you were, by merely reading 7 posts before your question you would see all that you need is posted by trurl.

 

I'll try to link to it, but mobile doesn't do to well with linking, so if it doesn't go to exact post, you'll have to do the work yourself...

 

 

 

Link to comment
3 hours ago, BRiT said:

You don't seem to be eager to help yourself. If you were, by merely reading 7 posts before your question you would see all that you need is posted by trurl.

 

I'll try to link to it, but mobile doesn't do to well with linking, so if it doesn't go to exact post, you'll have to do the work yourself...

 

 

 

Thanks!

The reason for me asking for details was that there are many other links in this thread. I just wanted to know which ones he was talking about. 

 

Thanks, again!

Link to comment
On 6/9/2021 at 5:25 AM, Seriously_Clueless said:

While we are patiently waiting for the Unraid team to advise on how they would like to tackle ths going forward (i.e. not using BTRFS in the docker img file) I have done the following to reduce my 96G to now around 40G. I might be able to get it down further but wanted to be cautious:

 

1. sudo btrfs subvolume delete /var/lib/docker/btrfs/subvolumes/<name of subvolume>

This will delete the subvolume and free up the space. I did it on a name basis since I wanted to keep some of the subvolumes, if you want to delete all, use * instead of the name of the subvolume

2. btrfs scrub status -d /var/lib/docker/

Shows you how your btrfs volume looks like

3. btrfs scrub start -d /var/lib/docker/

Think of this like a defrag for the btrfs volume

 

if you run btrfs --help you can see a few more commands that should help with listing the subvolumes instead of looking at the directory itself.

 

You will have to have the docker service started for this otherwise the /var/lib/docker will not contain anything. It is the mounting point for the docker.img file.

 

And now my folder looks like this:

root@NameOfServer:/var/lib/docker# du -h -d 1 .
440K    ./containerd
77M     ./containers
0       ./plugins
41G     ./btrfs
39M     ./image
56K     ./volumes
0       ./trust
88K     ./network
0       ./swarm
16K     ./builder
88K     ./buildkit
1.4M    ./unraid
0       ./tmp
0       ./runtimes
41G     .

 

So 50G were recovered without any issue.

 

 

So, how would one know if which, if any, of the subvolumes to keep?  I tried this and it look like it nuked a couple of containers.. they won't reinstall and complain about not finding the subvolume.   oof.   (I thought something like this might happen, but did it anyhow... lol) 

Link to comment

Actually, it looks like it nuked all of them.. 

root@localhost:# /usr/local/emhttp/plugins/dynamix.docker.manager/scripts/docker run -d --name='plex-linuxserver' --net='host' -e TZ="America/Chicago" -e HOST_OS="Unraid" -e 'VERSION'='docker' -e 'NVIDIA_VISIBLE_DEVICES'='' -e 'TCP_PORT_222'='222' -e 'TCP_PORT_32400'='32400' -e 'TCP_PORT_3005'='3005' -e 'TCP_PORT_8324'='8324' -e 'TCP_PORT_32469'='32469' -e 'UDP_PORT_1900'='1900' -e 'UDP_PORT_32410'='32410' -e 'UDP_PORT_32412'='32412' -e 'UDP_PORT_32413'='32413' -e 'UDP_PORT_32414'='32414' -e 'PUID'='99' -e 'PGID'='100' -v '/mnt/user/Media/':'/media':'rw' -v '':'/tv':'rw' -v '':'/music':'rw' -v '/mnt/disks/ScratchDrive/PMS':'/transcode':'rw,slave' -v '':'/movies':'rw' -v '/mnt/disks/PlexData/plex/':'/config':'rw,slave' 'linuxserver/plex'

docker: Error response from daemon: stat /var/lib/docker/btrfs/subvolumes/00004f57903e9d539379506531d2472859171afdef7ef3502d49e2f5c19200e6: no such file or directory.
See 'docker run --help'.

 

 

Edited by tiny-e
cleanup
Link to comment
8 minutes ago, Goguleanu said:

Is there a way to find which container is misconfigured?

Check each of your applications and make sure none of them specify a path that doesn't correspond to a container path in its mappings. Linux is case-sensitive so if you specify the wrong upper/lower case that would be a path inside docker.img. Also relative paths (not beginning with /) are inside docker.img

 

Do you understand docker volume mappings?

 

You might also get a clue by using the Container Size button on the Docker page.

Link to comment
  • 4 months later...

This post might anger some people,

 

But considering the amount of confusion that this issue is causing, I wonder if there needs to be a better solution to this problem that is not instructing people to "run this command from their shell" or "add this advanced configuration to each of your docker files to truncate the logs".

 

I'm in the same boat, I've identified myself that it's logging causing the problem & also a very large btrfs folder!

 

I feel like the intention of docker support in UNRAID is to make management of docker a little bit more simpler than just running docker in a VM, but the management overhead of keeping the volume disk size low seems contradictory to that mission - can we just not make this simpler?

  • Like 1
Link to comment
1 hour ago, ausca said:

advanced configuration to each of your docker files to truncate the logs

Default settings in Settings - Docker have the logs automatically "rotated" daily at 50Meg.  (ie: It's truncated daily to 50Meg) The need to add in an additional extra parameters only applies to containers that were installed previously to that setting being added (~6.7/6.8 IIRC again) (or if you wanted a specific app to have different log storage settings)

 

Most users have zero issues with the space utilized within the docker.img increasing.  But then again there are some users that for whatever reason do have that issue (usually due to a misconfiguration of how things are handled and/or a poorly behaved app.  There are some containers which you can set to log everything they do along with everyone in the world who shares the same birthday as you with the result that their log explodes - This is caused by a setting within the app of something like logging set to be "debug")

 

But, for the cases where it can't be solved, then the solution is to simply use a folder instead (Settings - Docker)  This has the advantage of never taking up the minimum space you've set (eg: 20G) but having the ability to grow (and shrink) to the limits of the device it's on (cache drive)

 

 

Link to comment
14 minutes ago, Squid said:

Default settings in Settings - Docker have the logs automatically "rotated" daily at 50Meg.  (ie: It's truncated daily to 50Meg) The need to add in an additional extra parameters only applies to containers that were installed previously to that setting being added (~6.7/6.8 IIRC again) (or if you wanted a specific app to have different log storage settings)

 

Most users have zero issues with the space utilized within the docker.img increasing.  But then again there are some users that for whatever reason do have that issue (usually due to a misconfiguration of how things are handled and/or a poorly behaved app.  There are some containers which you can set to log everything they do along with everyone in the world who shares the same birthday as you with the result that their log explodes - This is caused by a setting within the app of something like logging set to be "debug")

 

But, for the cases where it can't be solved, then the solution is to simply use a folder instead (Settings - Docker)  This has the advantage of never taking up the minimum space you've set (eg: 20G) but having the ability to grow (and shrink) to the limits of the device it's on (cache drive)

 

 

Interesting! I've only been an UnRAID user since 6.9 and yet my logs don't appear to be getting rotated as described. I had one log JSON @ 9GB (my personal docker container registry that seems to log everything including GET requests to syslog).

 

The problem is that some of the bad log design you're talking about are not at control of the (consumer) of the docker container, but at the control of the author of the container or containerised application. More on that assumed "mission" I referred to in my original post, I don't know if this "mission" extends to masking poor logging within containers/apps in-order to make this as simple as possible for the person consuming the container image. Especially considering that I would say the average UnRAID user might not know how to fix the logging issue. There are plenty of posts here on people trying to identify which container is even the cause of the problem.

 

My feedback comes from a position of love and not hate. Docker is a complex subject and I appreciate the efforts in "consumerising" it in UnRAID. For me I think I may go back to using docker in a VM for now.

Link to comment

image.png.5e9e9a1691fe532615dd918633710102.png

 

This should be the default settings.  If they're not, then do this.  But, this setting only takes effect on newly installed containers, not already existing ones.  (ie: if this doesn't exist you can either uninstall / reinstall the container (or all containers) or add the applicable line to extra parameters of each container)

 

There's absolutely nothing Unraid can do to control an app that logs the motion of every particle in the universe.  The log rotation is the best that the OS (or any OS) can do.

 

The cause of the issue (in most cases) can easily be determined by hitting Container Size on the docker tab.

  • Thanks 2
Link to comment
10 minutes ago, Squid said:

There's absolutely nothing Unraid can do to control an app that logs the motion of every particle in the universe.  The log rotation is the best that the OS (or any OS) can do.

 

That's fair. You can't fix all the container & app problems. I think that the log retention is a viable workaround for them. 

 

@Squid not sure where your screenshot is taken from though, but my UnRAID instance looks different to yours, even though I am running 6.9.2. I am missing the "Docker LOG number of files" and "Docker LOG maximum file size" despite setting the view to Advanced View.

 

Screen Shot 2022-01-23 at 12.24.34 pm.png

Link to comment
  • 1 month later...
On 9/7/2017 at 12:26 AM, assassinmunky said:

what does the usage in /var/lib/docker look like?    what is your output of this command?

du -h -d 1 /var/lib/docker/

Hi,

 

This is my out put from the two commands:

docker ps -s                                                                 ✔  4.18  81%  14:12 
CONTAINER ID   IMAGE                                  COMMAND                  CREATED         STATUS                   PORTS                                                                            NAMES                  SIZE
ea2f09f4daee   netdata/netdata                        "/usr/sbin/run.sh"       7 hours ago     Up 2 minutes (healthy)                                                                                    netdata                131MB (virtual 502MB)
8c3d263037b6   linuxserver/tautulli                   "/init"                  4 days ago      Up 3 minutes             0.0.0.0:8181->8181/tcp                                                           tautulli               37MB (virtual 161MB)
b6d462e3e1d2   lscr.io/linuxserver/mariadb            "/init"                  4 days ago      Up 3 minutes             0.0.0.0:3306->3306/tcp                                                           mariadb                16.6kB (virtual 292MB)
58b7f9e5230d   lscr.io/linuxserver/swag               "/init"                  4 days ago      Up 3 minutes             0.0.0.0:180->80/tcp, 0.0.0.0:1443->443/tcp                                       swag                   10.8MB (virtual 410MB)
de9fc2887c7a   testdasi/grafana-unraid-stack:latest   "tini -- /static-ubu…"   4 days ago      Up 3 minutes (healthy)                                                                                    Grafana-Unraid-Stack   32.2kB (virtual 993MB)
25be88f53c9c   linuxserver/duckdns                    "/init"                  6 days ago      Up 2 minutes                                                                                              duckdns                11.5kB (virtual 26.6MB)
5fa90ac7906a   lscr.io/linuxserver/nextcloud          "/init"                  8 days ago      Up 3 minutes             80/tcp, 0.0.0.0:444->443/tcp                                                     nextcloud              44kB (virtual 441MB)
7f43259eb69d   linuxserver/calibre                    "/init"                  9 days ago      Up 3 minutes             3000/tcp, 3389/tcp, 0.0.0.0:9080->8080/tcp, 0.0.0.0:9081->8081/tcp               calibre                368MB (virtual 1.65GB)
df41962f063a   linuxserver/sonarr                     "/init"                  9 days ago      Up 3 minutes             0.0.0.0:8989->8989/tcp                                                           sonarr                 89.3MB (virtual 722MB)
e12738380365   linuxserver/radarr                     "/init"                  9 days ago      Up 3 minutes             0.0.0.0:7878->7878/tcp                                                           radarr                 86.4MB (virtual 413MB)
0c04aff5c455   linuxserver/plex                       "/init"                  9 days ago      Up 3 minutes                                                                                              plex                   40.9MB (virtual 672MB)
56c9c041eb4b   binhex/arch-get-iplayer                "/usr/bin/tini -g --…"   5 weeks ago     Up 2 minutes                                                                                              binhex-get-iplayer     1.81MB (virtual 1.16GB)
d91c1b568c52   jlesage/qdirstat                       "/init"                  3 months ago    Up 2 minutes             0.0.0.0:7815->5800/tcp, 0.0.0.0:7915->5900/tcp                                   QDirStat               23.6kB (virtual 253MB)
560437ec2976   henrywhitaker3/speedtest-tracker       "/init"                  4 months ago    Up 3 minutes             443/tcp, 0.0.0.0:3765->80/tcp                                                    speedtest-tracker      4.59MB (virtual 320MB)
aedd5c214872   silverwind/droppy:latest               "docker-entrypoint.s…"   4 months ago    Up 3 minutes             0.0.0.0:1000->8989/tcp                                                           droppy                 3.46kB (virtual 135MB)
b8362151eed8   tzahi12345/youtubedl-material          "/app/entrypoint.sh …"   4 months ago    Up 3 minutes             0.0.0.0:1001->17442/tcp                                                          youtube-dlmaterial     1.84MB (virtual 212MB)
28a67a2b067a   jlesage/handbrake                      "/init"                  5 months ago    Up 2 minutes             0.0.0.0:7803->5800/tcp, 0.0.0.0:7903->5900/tcp                                   HandBrake              39kB (virtual 324MB)
de4ab388b940   linuxserver/openvpn-as                 "/init"                  9 months ago    Up 3 minutes             0.0.0.0:943->943/tcp, 0.0.0.0:9443->9443/tcp, 0.0.0.0:1194->1194/udp             openvpn-as             19.9MB (virtual 248MB)
ba300091e2bf   linuxserver/rutorrent                  "/init"                  11 months ago   Up 3 minutes             443/tcp, 0.0.0.0:6003->80/tcp, 0.0.0.0:8003->6881/udp, 0.0.0.0:7003->51413/tcp   rutorrent-music        3.78MB (virtual 214MB)
827ac52cbb66   mace/darkstat                          "/sbin/my_init"          11 months ago   Up 3 minutes                                                                                              DarkStat               402kB (virtual 307MB)
093054dbf7fa   linuxserver/rutorrent                  "/init"                  11 months ago   Up 3 minutes             443/tcp, 0.0.0.0:6004->80/tcp, 0.0.0.0:8004->6881/udp, 0.0.0.0:7004->51413/tcp   rutorrent-ebooks       3.78MB (virtual 214MB)
eec19e8002b7   linuxserver/rutorrent                  "/init"                  11 months ago   Up 3 minutes             443/tcp, 0.0.0.0:6005->80/tcp, 0.0.0.0:8005->6881/udp, 0.0.0.0:7005->51413/tcp   rutorrent-other        3.78MB (virtual 214MB)
e8b783054594   linuxserver/rutorrent                  "/init"                  11 months ago   Up 3 minutes             443/tcp, 0.0.0.0:6002->80/tcp, 0.0.0.0:8002->6881/udp, 0.0.0.0:7002->51413/tcp   rutorrent-tv           3.79MB (virtual 214MB)
7f23382fcb85   linuxserver/rutorrent                  "/init"                  11 months ago   Up 3 minutes             443/tcp, 0.0.0.0:6001->80/tcp, 0.0.0.0:8001->6881/udp, 0.0.0.0:7001->51413/tcp   rutorrent-films        3.78MB (virtual 214MB)
65249963532e   binhex/arch-krusader                   "/usr/bin/tini -- /b…"   11 months ago   Up 2 minutes             5900/tcp, 0.0.0.0:6080->6080/tcp 

and

du -h -d 1 /var/lib/docker/
1.1M    /var/lib/docker/containerd
44M     /var/lib/docker/containers
0       /var/lib/docker/plugins
89G     /var/lib/docker/btrfs
61M     /var/lib/docker/image
100K    /var/lib/docker/volumes
0       /var/lib/docker/trust
172K    /var/lib/docker/network
0       /var/lib/docker/swarm
16K     /var/lib/docker/builder
88K     /var/lib/docker/buildkit
1.7M    /var/lib/docker/unraid
0       /var/lib/docker/tmp
0       /var/lib/docker/runtimes
89G     /var/lib/docker/

Seems it's all coming from /var/lib/docker/btrfs

How do I resolve this?

Link to comment
1 hour ago, Squid said:

Better way to figure it out is to simply click container size on the docker tab.  You'll see which app is storing within the image

Thanks I hadn't seen that before. Unfortunately it doesn't help:

 

Name                              Container     Writable          Log
---------------------------------------------------------------------
binhex-krusader                     1.92 GB      35.6 MB       847 kB
calibre                             1.65 GB       368 MB       330 kB
binhex-preclear                     1.29 GB          0 B      24.3 MB
binhex-get-iplayer                  1.16 GB      1.81 MB       908 kB
Grafana-Unraid-Stack                 993 MB          0 B      3.44 kB
binhex-rclone                        776 MB          0 B      24.3 MB
sonarr                               668 MB      35.0 MB      8.42 kB
plex                                 632 MB       316 kB      5.79 kB
netdata                              551 MB       180 MB      1.66 MB
nextcloud                            442 MB      44.0 kB      5.56 kB
swag                                 410 MB      10.8 MB      27.4 kB
radarr                               327 MB       306 kB      7.30 kB
HandBrake                            324 MB      39.0 kB      19.3 kB
speedtest-tracker                    320 MB      4.59 MB       384 kB
DarkStat                             307 MB       403 kB      1.71 MB
mariadb                              292 MB      16.6 kB      17.4 kB
QDirStat                             253 MB      23.6 kB       411 kB
openvpn-as                           248 MB      19.9 MB       352 kB
beets                                223 MB          0 B      24.3 MB
rutorrent-ebooks                     214 MB      3.78 MB      1.79 MB
rutorrent-films                      214 MB      3.78 MB      1.79 MB
rutorrent-films-4k                   214 MB      3.78 MB       106 kB
rutorrent-music                      214 MB      3.78 MB      1.84 MB
rutorrent-other                      214 MB      3.78 MB      1.78 MB
rutorrent-temp                       214 MB      3.78 MB      9.80 kB
rutorrent-temp2                      214 MB      3.78 MB      8.54 kB
rutorrent-tv                         214 MB      3.79 MB      1.83 MB
youtube-dlmaterial                   212 MB      1.84 MB      23.9 kB
deluge                               191 MB          0 B      24.3 MB
tautulli                             161 MB      37.0 MB       221 kB
droppy                               135 MB      3.46 kB       216 kB
duckdns                             26.6 MB      11.5 kB       240 kB
---------------------------------------------------------------------
Total size                          15.2 GB       726 MB       114 MB

 

It's showing as only 15.2GB but it grew from 70% of 80GB to 99% within 12 minutes at 7am yesterday.

Link to comment
5 minutes ago, Squid said:

Docker Advanced view - See if there is a ton of orphaned containers, and delete them

No orphaned containers. I'm always in advanced view.

The issue seems twofold:

1. rapid increase in size over the course of just a few minutes

2. massive usage in /var/lib/docker/btrfs

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.