Docker image grow indefinitely

[talard]

Hi there,

 

I need your help, I can't get it worked by myself.

Unraid v6.11.5
CPU : AMD 3700x
RAM : 32Gb
Cache : 1To

All start with Docker service failed. I understaind fast that the image where growing annormaly. I try tro recreate docker.img once, the I map directly my cache hard drive and do a fresh install of my docker containers. I thought all where OK, but I meet this new Out of Memery error and I feel like docker image keep going to grow.

Regarding to out of memory error, it shut down Docker and VM service. I try to clean plugins and VM in order to eliminate this. I try to put memory argument to my 2 docker containers ( with --memory=xG in docker container advanced settings) but I can still see my RAM growing.

I find this command line to find what service take ram and how much but I don't know what to do next with rsync services :

Spoiler

root@Lescabannes:~# ps aux | awk '{print $6/1024 " MB\t\t" $11}' | sort -n
0 MB            COMMAND
0 MB            [amd_iommu_v2]
0 MB            [ata_sff]
0 MB            [blkcg_punt_bio]
0 MB            [btrfs-cache]
0 MB            [btrfs-cache]
0 MB            [btrfs-cleaner]
0 MB            [btrfs-cleaner]
0 MB            [btrfs-delalloc]
0 MB            [btrfs-delalloc]
0 MB            [btrfs-delayed-m]
0 MB            [btrfs-delayed-m]
0 MB            [btrfs-endio-met]
0 MB            [btrfs-endio-met]
0 MB            [btrfs-endio-met]
0 MB            [btrfs-endio-met]
0 MB            [btrfs-endio-rai]
0 MB            [btrfs-endio-rai]
0 MB            [btrfs-endio-wri]
0 MB            [btrfs-endio-wri]
0 MB            [btrfs-endio]
0 MB            [btrfs-endio]
0 MB            [btrfs-fixup]
0 MB            [btrfs-fixup]
0 MB            [btrfs-flush_del]
0 MB            [btrfs-flush_del]
0 MB            [btrfs-freespace]
0 MB            [btrfs-freespace]
0 MB            [btrfs-qgroup-re]
0 MB            [btrfs-qgroup-re]
0 MB            [btrfs-rmw]
0 MB            [btrfs-rmw]
0 MB            [btrfs-transaction]
0 MB            [btrfs-transaction]
0 MB            [btrfs-worker-hi]
0 MB            [btrfs-worker-hi]
0 MB            [btrfs-worker]
0 MB            [btrfs-worker]
0 MB            [cpuhp/0]
0 MB            [cpuhp/10]
0 MB            [cpuhp/11]
0 MB            [cpuhp/12]
0 MB            [cpuhp/13]
0 MB            [cpuhp/14]
0 MB            [cpuhp/15]
0 MB            [cpuhp/1]
0 MB            [cpuhp/2]
0 MB            [cpuhp/3]
0 MB            [cpuhp/4]
0 MB            [cpuhp/5]
0 MB            [cpuhp/6]
0 MB            [cpuhp/7]
0 MB            [cpuhp/8]
0 MB            [cpuhp/9]
0 MB            [cryptd]
0 MB            [devfreq_wq]
0 MB            [dio/nvme0n1p1]
0 MB            [edac-poller]
0 MB            [inet_frag_wq]
0 MB            [ipmi-msghandler]
0 MB            [ipv6_addrconf]
0 MB            [irq/25-AMD-Vi]
0 MB            [irq/26-aerdrv]
0 MB            [irq/27-aerdrv]
0 MB            [irq/29-aerdrv]
0 MB            [irq/30-aerdrv]
0 MB            [kblockd]
0 MB            [kcompactd0]
0 MB            [kdevtmpfs]
0 MB            [khugepaged]
0 MB            [kintegrityd]
0 MB            [kipmi0]
0 MB            [ksmd]
0 MB            [ksoftirqd/0]
0 MB            [ksoftirqd/10]
0 MB            [ksoftirqd/11]
0 MB            [ksoftirqd/12]
0 MB            [ksoftirqd/13]
0 MB            [ksoftirqd/14]
0 MB            [ksoftirqd/15]
0 MB            [ksoftirqd/1]
0 MB            [ksoftirqd/2]
0 MB            [ksoftirqd/3]
0 MB            [ksoftirqd/4]
0 MB            [ksoftirqd/5]
0 MB            [ksoftirqd/6]
0 MB            [ksoftirqd/7]
0 MB            [ksoftirqd/8]
0 MB            [ksoftirqd/9]
0 MB            [kstrp]
0 MB            [kswapd0]
0 MB            [kthreadd]
0 MB            [kthrotld]
0 MB            [kworker/0:0-events]
0 MB            [kworker/0:0H-events_highpri]
0 MB            [kworker/0:1-events]
0 MB            [kworker/0:1H-kblockd]
0 MB            [kworker/10:0-events]
0 MB            [kworker/10:0H-kblockd]
0 MB            [kworker/10:1H-kblockd]
0 MB            [kworker/10:2-rcu_par_gp]
0 MB            [kworker/11:0H-kblockd]
0 MB            [kworker/11:1-rcu_par_gp]
0 MB            [kworker/11:1H-kblockd]
0 MB            [kworker/11:3-events]
0 MB            [kworker/12:0-events]
0 MB            [kworker/12:0H]
0 MB            [kworker/12:1-events]
0 MB            [kworker/13:0-events]
0 MB            [kworker/13:0H]
0 MB            [kworker/13:1-events]
0 MB            [kworker/14:0-events]
0 MB            [kworker/14:0H-kblockd]
0 MB            [kworker/14:1-events]
0 MB            [kworker/14:1H-kblockd]
0 MB            [kworker/15:0-events]
0 MB            [kworker/15:0H]
0 MB            [kworker/15:1-events]
0 MB            [kworker/1:0-events]
0 MB            [kworker/1:0H-kblockd]
0 MB            [kworker/1:1-events]
0 MB            [kworker/1:1H-kblockd]
0 MB            [kworker/2:0H-kblockd]
0 MB            [kworker/2:1H-kblockd]
0 MB            [kworker/2:1]
0 MB            [kworker/2:2-events]
0 MB            [kworker/3:0H-kblockd]
0 MB            [kworker/3:1-events]
0 MB            [kworker/3:1H-kblockd]
0 MB            [kworker/3:2-events]
0 MB            [kworker/4:0-events]
0 MB            [kworker/4:0H-kblockd]
0 MB            [kworker/4:1-events]
0 MB            [kworker/4:1H-kblockd]
0 MB            [kworker/5:0-events]
0 MB            [kworker/5:0H]
0 MB            [kworker/5:1-events]
0 MB            [kworker/6:0-events]
0 MB            [kworker/6:0H]
0 MB            [kworker/6:1-events]
0 MB            [kworker/7:0H]
0 MB            [kworker/7:1-events]
0 MB            [kworker/7:2-rcu_gp]
0 MB            [kworker/8:0-events]
0 MB            [kworker/8:0H-events_highpri]
0 MB            [kworker/8:10-events]
0 MB            [kworker/8:1H-kblockd]
0 MB            [kworker/9:0H-kblockd]
0 MB            [kworker/9:1-events]
0 MB            [kworker/9:1H-kblockd]
0 MB            [kworker/9:3-rcu_gp]
0 MB            [kworker/u64:0-btrfs-endio-meta]
0 MB            [kworker/u64:1-btrfs-endio-meta]
0 MB            [kworker/u64:2-btrfs-endio-meta]
0 MB            [kworker/u64:3-btrfs-endio-meta]
0 MB            [kworker/u64:4-btrfs-endio-meta]
0 MB            [kworker/u64:7-btrfs-endio-meta]
0 MB            [kworker/u65:0-hci0]
0 MB            [kworker/u65:2-hci0]
0 MB            [md]
0 MB            [mdrecoveryd]
0 MB            [migration/0]
0 MB            [migration/10]
0 MB            [migration/11]
0 MB            [migration/12]
0 MB            [migration/13]
0 MB            [migration/14]
0 MB            [migration/15]
0 MB            [migration/1]
0 MB            [migration/2]
0 MB            [migration/3]
0 MB            [migration/4]
0 MB            [migration/5]
0 MB            [migration/6]
0 MB            [migration/7]
0 MB            [migration/8]
0 MB            [migration/9]
0 MB            [mld]
0 MB            [mm_percpu_wq]
0 MB            [netns]
0 MB            [nvme-delete-wq]
0 MB            [nvme-reset-wq]
0 MB            [nvme-wq]
0 MB            [oom_reaper]
0 MB            [rcu_gp]
0 MB            [rcu_par_gp]
0 MB            [rcu_preempt]
0 MB            [rcu_tasks_kthread]
0 MB            [rcu_tasks_rude_kthread]
0 MB            [rcu_tasks_trace_kthread]
0 MB            [scsi_eh_0]
0 MB            [scsi_eh_10]
0 MB            [scsi_eh_11]
0 MB            [scsi_eh_12]
0 MB            [scsi_eh_13]
0 MB            [scsi_eh_1]
0 MB            [scsi_eh_2]
0 MB            [scsi_eh_3]
0 MB            [scsi_eh_4]
0 MB            [scsi_eh_5]
0 MB            [scsi_eh_6]
0 MB            [scsi_eh_7]
0 MB            [scsi_eh_8]
0 MB            [scsi_eh_9]
0 MB            [scsi_tmf_0]
0 MB            [scsi_tmf_10]
0 MB            [scsi_tmf_11]
0 MB            [scsi_tmf_12]
0 MB            [scsi_tmf_13]
0 MB            [scsi_tmf_1]
0 MB            [scsi_tmf_2]
0 MB            [scsi_tmf_3]
0 MB            [scsi_tmf_4]
0 MB            [scsi_tmf_5]
0 MB            [scsi_tmf_6]
0 MB            [scsi_tmf_7]
0 MB            [scsi_tmf_8]
0 MB            [scsi_tmf_9]
0 MB            [slub_flushwq]
0 MB            [target_completi]
0 MB            [target_submissi]
0 MB            [unraidd0]
0 MB            [unraidd1]
0 MB            [usb-storage]
0 MB            [usb-storage]
0 MB            [usb-storage]
0 MB            [usb-storage]
0 MB            [vfio-irqfd-clea]
0 MB            [wg-crypt-wg0]
0 MB            [writeback]
0 MB            [xcopy_wq]
0 MB            [xenbus_probe]
0 MB            [xfs-blockgc/md1]
0 MB            [xfs-buf/md1]
0 MB            [xfs-cil/md1]
0 MB            [xfs-conv/md1]
0 MB            [xfs-inodegc/md1]
0 MB            [xfs-log/md1]
0 MB            [xfs-reclaim/md1]
0 MB            [xfs_mru_cache]
0 MB            [xfsaild/md1]
0 MB            [xfsalloc]
0.0976562 MB            /usr/sbin/acpid
0.113281 MB             /usr/sbin/avahi-dnsconfd
0.113281 MB             /usr/sbin/wsdd2
0.242188 MB             /usr/sbin/inetd
0.261719 MB             avahi-daemon:
0.261719 MB             dhcpcd:
0.289062 MB             /usr/local/sbin/shfs
0.289062 MB             dhcpcd:
0.289062 MB             dhcpcd:
0.816406 MB             /sbin/agetty
0.816406 MB             /sbin/agetty
0.847656 MB             sleep
0.882812 MB             sleep
0.890625 MB             sleep
0.898438 MB             /sbin/agetty
0.898438 MB             init
0.917969 MB             /sbin/agetty
0.921875 MB             /sbin/agetty
0.925781 MB             /sbin/agetty
1.48438 MB              /usr/sbin/atd
1.75 MB         /usr/sbin/crond
1.89062 MB              sort
1.90234 MB              dhcpcd:
1.90234 MB              dhcpcd:
2.00781 MB              /usr/bin/dbus-daemon
2.22266 MB              ps
2.42969 MB              /bin/bash
2.49219 MB              awk
2.53125 MB              awk
2.54297 MB              awk
2.59375 MB              awk
2.59375 MB              awk
2.59375 MB              awk
2.86328 MB              sh
2.89062 MB              sh
2.91406 MB              sh
2.91406 MB              sh
2.91797 MB              sh
2.95703 MB              sh
3.03125 MB              /bin/bash
3.03125 MB              sshd:
3.12109 MB              awk
3.18359 MB              avahi-daemon:
3.39062 MB              elogind-daemon
3.5625 MB               /bin/bash
3.94531 MB              /sbin/udevd
4.02734 MB              /usr/sbin/ntpd
4.05078 MB              /usr/sbin/rsyslogd
4.19531 MB              bash
4.28906 MB              nginx:
4.30078 MB              /usr/sbin/smbd
4.55859 MB              /usr/local/sbin/emhttpd
4.84375 MB              /usr/bin/ttyd
6.01562 MB              /usr/sbin/nmbd
7.73438 MB              sshd:
8.79688 MB              /usr/sbin/smbd
9.66016 MB              php-fpm:
10.0781 MB              /usr/sbin/winbindd
11.3672 MB              /usr/sbin/winbindd
13.2812 MB              /usr/sbin/smbd
14.2383 MB              php-fpm:
14.3594 MB              /usr/sbin/winbindd
14.6641 MB              php-fpm:
16.0898 MB              nginx:
16.207 MB               php-fpm:
16.9023 MB              php-fpm:
17.2734 MB              php-fpm:
17.6211 MB              php-fpm:
17.9219 MB              php-fpm:
22.8711 MB              php-fpm:
23.2539 MB              php-fpm:
28.1836 MB              /usr/bin/php
28.375 MB               /usr/bin/php
28.4414 MB              /usr/bin/php
28.5469 MB              /usr/bin/php
28.5547 MB              /usr/bin/php
28.5898 MB              /usr/bin/php
28.6055 MB              /usr/bin/php
28.6602 MB              /usr/bin/php
28.7539 MB              /usr/bin/php
29.1602 MB              /usr/bin/php
114.305 MB              /usr/local/sbin/shfs
163.031 MB              /usr/local/bin/unraid-api/unraid-api
336.152 MB              rsync
338.105 MB              rsync
374.816 MB              rsync
1066.21 MB              rsync
1068.63 MB              rsync
1186.15 MB              rsync
1552.24 MB              rsync
1555.08 MB              rsync
1726.38 MB              rsync
1753.75 MB              rsync
1754.04 MB              rsync
1755.33 MB              rsync
1755.44 MB              rsync
1769.04 MB              rsync
1769.58 MB              rsync
1951.71 MB              rsync
1951.8 MB               rsync
1967.46 MB              rsync

 

Finally I got this error that I did not understaind because I don't have Dynamix docker manager actually installed. I try to find it on app store in order to install/uninstall again the app but I can't find it.
 

Warning: stream_socket_client(): unable to connect to unix:///var/run/docker.sock (No such file or directory) in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 712
Couldn't create socket: [2] No such file or directory
Warning: Invalid argument supplied for foreach() in /usr/local/emhttp/plugins/dynamix.docker.manager/include/DockerClient.php on line 898
appdata

 

 

Any help would be apreciated !
 

unraid-diagnostics-20230608-2108.zip

Edited June 10, 2023 by talard

[itimpi]

2 hours ago, talard said:

[talard]

For now the situation looks stabilised.

I delete /user/system/docker/ folder manually and change back docker settings to XFS image. I reinstall Plex docker only (for now) and it look to work fine.

Lets see if everything keep going well

[talard]

Hi there,

 

I change the title of post because the out of memory error is gone. But I still have troubles with my docker image growing. I have onely Plex and Transmission docker. I put 100GB for the docker image and it's now at 95% full after 24hours runing.

Here is some screens.

Edited June 10, 2023 by talard
add container size

[Vr2Io]

What actual current docker img file size ?

BTW, container size show usage ~30.1 GB. Not sure img file size reduce if actual usage reduce.

[talard]

The image file size is 100GB.

 

I had to delete then recreate the docker image because it ran to 100%. For now the image is 2% full and look stable but I don't know why, at some point it start to grow and getting full.

 

I put logs limit argument in both container to see if it can help (--log-opt max-size=50m --log-opt max-file=1) but container calculation show that it's not logs that take all this space but something else with plex.

Edited June 10, 2023 by talard

[talard]

Until now since my last delete/creation of new docker file, everything is OK. It still 2% full.

What is really wired is the 100% usage of the 100GB docker image but with container calculation their is only 30GB.

 

Let's see what happen next.

[Hoopster]

7 hours ago, talard said:

The image file size is 100GB

Rarely does a docker.img file need to be larger than 20GB.  Even with a lot of docker containers, this is sufficient. 100GB is definitely excessive and indicates configuration problems somewhere.  For example, I have 15 docker containers with only 12 GB used in the 20GB docker.img file.

 

If you keep needing to increase the size of docker.img and usage within the file keeps growing even when you add no new containers, it is almost certain you have one or more containers improperly configured and they are writing data into docker.img instead of to an array, cache/pool or unassigned devices location.  The  most common offenders are file download apps (Deluge, Transmission, the *arrs, etc.) , backup apps, media management apps (Plex, emby, jellyfn), etc.

 

A good way to find potential offenders is to click on the Container Size button and look for containers that are unusually large.  Some are 2-3GB in size and this is normal for that container.  Other are much smaller.  For example, my Plex container  is only 334 MB in size and the Krusader container is 2.38GB.  Both are normal.  If you see one or more that appear to be unusually large, ask for help in the support thread for that container.

 

Docker.img is only meant to be the storage location for the docker container images and not their data. 

Edited June 10, 2023 by Hoopster

[talard]

8 minutes ago, Hoopster said:

Rarely does a docker.img file need to be larger than 20GB.  Even with a lot of docker containers, this is sufficient. 100GB is definitely excessive and indicates configuration problems somewhere.  For example, I have 15 docker containers with only 13 GB used in the 20GB docker.img file.

 

If you keep needing to increase the size of docker.img and usage within the file keeps growing even when you add no new containers, it is almost certain you have one or more containers improperly configured and they are writing data into docker.img instead of to an array, cache/pool or unassigned devices location.  The  most common offenders are file download apps (Deluge, Transmission, the *arrs) , backup apps, media management apps (Plex, emby, jellyfn), etc.

 

Docker.img is only meant to be the storage location for the docker container images and not their data. 

Hi,

Thanks for your reply.

 

Total agree but I do that because I couldn't debug instantly.

 

As you can see I use both (transmission and plex). I don't change the configuration and all work fine for at least a year. You can see folders mapping in my 1rst post and everything is mapped to my storage (at least pre-configured template docker variables)

[talard]

I finnaly found the issue. I found the origine thanks to unraind tools>process section.

Plex was blocked on transcoding task and I don't know why but even with /transcode folder mapping on array, it use docker image. I delete the video file and recreate docker image and everything is OK since then.

 

Thanks Vr2lo and Hoopster for your comments.

Edited June 15, 2023 by talard