trurl Posted September 19, 2021 Share Posted September 19, 2021 4 minutes ago, Goguleanu said: tutorial See links above Quote Link to comment
Goguleanu Posted September 19, 2021 Share Posted September 19, 2021 3 minutes ago, trurl said: See links above It would be a lot more helpful if you point to the actual links or the post that has them. But, thanks! 1 Quote Link to comment
BRiT Posted September 19, 2021 Share Posted September 19, 2021 1 hour ago, Goguleanu said: It would be a lot more helpful if you point to the actual links or the post that has them. But, thanks! You don't seem to be eager to help yourself. If you were, by merely reading 7 posts before your question you would see all that you need is posted by trurl. I'll try to link to it, but mobile doesn't do to well with linking, so if it doesn't go to exact post, you'll have to do the work yourself... Quote Link to comment
Goguleanu Posted September 19, 2021 Share Posted September 19, 2021 3 hours ago, BRiT said: You don't seem to be eager to help yourself. If you were, by merely reading 7 posts before your question you would see all that you need is posted by trurl. I'll try to link to it, but mobile doesn't do to well with linking, so if it doesn't go to exact post, you'll have to do the work yourself... Thanks! The reason for me asking for details was that there are many other links in this thread. I just wanted to know which ones he was talking about. Thanks, again! Quote Link to comment
trurl Posted September 19, 2021 Share Posted September 19, 2021 Sorry I was on mobile at the time. If you want more detailed advice, attach diagnostics to NEXT post. Quote Link to comment
tiny-e Posted September 21, 2021 Share Posted September 21, 2021 On 6/9/2021 at 5:25 AM, Seriously_Clueless said: While we are patiently waiting for the Unraid team to advise on how they would like to tackle ths going forward (i.e. not using BTRFS in the docker img file) I have done the following to reduce my 96G to now around 40G. I might be able to get it down further but wanted to be cautious: 1. sudo btrfs subvolume delete /var/lib/docker/btrfs/subvolumes/<name of subvolume> This will delete the subvolume and free up the space. I did it on a name basis since I wanted to keep some of the subvolumes, if you want to delete all, use * instead of the name of the subvolume 2. btrfs scrub status -d /var/lib/docker/ Shows you how your btrfs volume looks like 3. btrfs scrub start -d /var/lib/docker/ Think of this like a defrag for the btrfs volume if you run btrfs --help you can see a few more commands that should help with listing the subvolumes instead of looking at the directory itself. You will have to have the docker service started for this otherwise the /var/lib/docker will not contain anything. It is the mounting point for the docker.img file. And now my folder looks like this: root@NameOfServer:/var/lib/docker# du -h -d 1 . 440K ./containerd 77M ./containers 0 ./plugins 41G ./btrfs 39M ./image 56K ./volumes 0 ./trust 88K ./network 0 ./swarm 16K ./builder 88K ./buildkit 1.4M ./unraid 0 ./tmp 0 ./runtimes 41G . So 50G were recovered without any issue. So, how would one know if which, if any, of the subvolumes to keep? I tried this and it look like it nuked a couple of containers.. they won't reinstall and complain about not finding the subvolume. oof. (I thought something like this might happen, but did it anyhow... lol) Quote Link to comment
tiny-e Posted September 21, 2021 Share Posted September 21, 2021 (edited) Actually, it looks like it nuked all of them.. root@localhost:# /usr/local/emhttp/plugins/dynamix.docker.manager/scripts/docker run -d --name='plex-linuxserver' --net='host' -e TZ="America/Chicago" -e HOST_OS="Unraid" -e 'VERSION'='docker' -e 'NVIDIA_VISIBLE_DEVICES'='' -e 'TCP_PORT_222'='222' -e 'TCP_PORT_32400'='32400' -e 'TCP_PORT_3005'='3005' -e 'TCP_PORT_8324'='8324' -e 'TCP_PORT_32469'='32469' -e 'UDP_PORT_1900'='1900' -e 'UDP_PORT_32410'='32410' -e 'UDP_PORT_32412'='32412' -e 'UDP_PORT_32413'='32413' -e 'UDP_PORT_32414'='32414' -e 'PUID'='99' -e 'PGID'='100' -v '/mnt/user/Media/':'/media':'rw' -v '':'/tv':'rw' -v '':'/music':'rw' -v '/mnt/disks/ScratchDrive/PMS':'/transcode':'rw,slave' -v '':'/movies':'rw' -v '/mnt/disks/PlexData/plex/':'/config':'rw,slave' 'linuxserver/plex' docker: Error response from daemon: stat /var/lib/docker/btrfs/subvolumes/00004f57903e9d539379506531d2472859171afdef7ef3502d49e2f5c19200e6: no such file or directory. See 'docker run --help'. Edited September 21, 2021 by tiny-e cleanup Quote Link to comment
JonathanM Posted September 21, 2021 Share Posted September 21, 2021 4 minutes ago, tiny-e said: Actually, it looks like it nuked all of them.. Your best bet is remove and recreate the docker.img file, go to previous apps and select the ones you actually want. That's the preferred cleanup method anyway, it's much quicker and more thorough. Quote Link to comment
tiny-e Posted September 21, 2021 Share Posted September 21, 2021 Just now, JonathanM said: Your best bet is remove and recreate the docker.img file, go to previous apps and select the ones you actually want. That's the preferred cleanup method anyway, it's much quicker and more thorough. fingers crossed... 🤞... Quote Link to comment
tiny-e Posted September 21, 2021 Share Posted September 21, 2021 3 hours ago, tiny-e said: fingers crossed... 🤞... Deleting the old docker image did the trick. Hopefully there will be an official fix for this as mine was complaining about high utilization that I couldn't actually find anywhere... Quote Link to comment
trurl Posted September 21, 2021 Share Posted September 21, 2021 2 minutes ago, tiny-e said: an official fix Not clear there is anything to fix. Users often fill docker.img. The usual cause is configuring an application to write to a path that isn't mapped. Quote Link to comment
Goguleanu Posted September 21, 2021 Share Posted September 21, 2021 50 minutes ago, trurl said: Not clear there is anything to fix. Users often fill docker.img. The usual cause is configuring an application to write to a path that isn't mapped. Is there a way to find which container is misconfigured? Quote Link to comment
trurl Posted September 21, 2021 Share Posted September 21, 2021 8 minutes ago, Goguleanu said: Is there a way to find which container is misconfigured? Check each of your applications and make sure none of them specify a path that doesn't correspond to a container path in its mappings. Linux is case-sensitive so if you specify the wrong upper/lower case that would be a path inside docker.img. Also relative paths (not beginning with /) are inside docker.img Do you understand docker volume mappings? You might also get a clue by using the Container Size button on the Docker page. Quote Link to comment
ausca Posted January 22, 2022 Share Posted January 22, 2022 This post might anger some people, But considering the amount of confusion that this issue is causing, I wonder if there needs to be a better solution to this problem that is not instructing people to "run this command from their shell" or "add this advanced configuration to each of your docker files to truncate the logs". I'm in the same boat, I've identified myself that it's logging causing the problem & also a very large btrfs folder! I feel like the intention of docker support in UNRAID is to make management of docker a little bit more simpler than just running docker in a VM, but the management overhead of keeping the volume disk size low seems contradictory to that mission - can we just not make this simpler? 1 Quote Link to comment
Squid Posted January 23, 2022 Share Posted January 23, 2022 1 hour ago, ausca said: advanced configuration to each of your docker files to truncate the logs Default settings in Settings - Docker have the logs automatically "rotated" daily at 50Meg. (ie: It's truncated daily to 50Meg) The need to add in an additional extra parameters only applies to containers that were installed previously to that setting being added (~6.7/6.8 IIRC again) (or if you wanted a specific app to have different log storage settings) Most users have zero issues with the space utilized within the docker.img increasing. But then again there are some users that for whatever reason do have that issue (usually due to a misconfiguration of how things are handled and/or a poorly behaved app. There are some containers which you can set to log everything they do along with everyone in the world who shares the same birthday as you with the result that their log explodes - This is caused by a setting within the app of something like logging set to be "debug") But, for the cases where it can't be solved, then the solution is to simply use a folder instead (Settings - Docker) This has the advantage of never taking up the minimum space you've set (eg: 20G) but having the ability to grow (and shrink) to the limits of the device it's on (cache drive) Quote Link to comment
ausca Posted January 23, 2022 Share Posted January 23, 2022 14 minutes ago, Squid said: Default settings in Settings - Docker have the logs automatically "rotated" daily at 50Meg. (ie: It's truncated daily to 50Meg) The need to add in an additional extra parameters only applies to containers that were installed previously to that setting being added (~6.7/6.8 IIRC again) (or if you wanted a specific app to have different log storage settings) Most users have zero issues with the space utilized within the docker.img increasing. But then again there are some users that for whatever reason do have that issue (usually due to a misconfiguration of how things are handled and/or a poorly behaved app. There are some containers which you can set to log everything they do along with everyone in the world who shares the same birthday as you with the result that their log explodes - This is caused by a setting within the app of something like logging set to be "debug") But, for the cases where it can't be solved, then the solution is to simply use a folder instead (Settings - Docker) This has the advantage of never taking up the minimum space you've set (eg: 20G) but having the ability to grow (and shrink) to the limits of the device it's on (cache drive) Interesting! I've only been an UnRAID user since 6.9 and yet my logs don't appear to be getting rotated as described. I had one log JSON @ 9GB (my personal docker container registry that seems to log everything including GET requests to syslog). The problem is that some of the bad log design you're talking about are not at control of the (consumer) of the docker container, but at the control of the author of the container or containerised application. More on that assumed "mission" I referred to in my original post, I don't know if this "mission" extends to masking poor logging within containers/apps in-order to make this as simple as possible for the person consuming the container image. Especially considering that I would say the average UnRAID user might not know how to fix the logging issue. There are plenty of posts here on people trying to identify which container is even the cause of the problem. My feedback comes from a position of love and not hate. Docker is a complex subject and I appreciate the efforts in "consumerising" it in UnRAID. For me I think I may go back to using docker in a VM for now. Quote Link to comment
Squid Posted January 23, 2022 Share Posted January 23, 2022 This should be the default settings. If they're not, then do this. But, this setting only takes effect on newly installed containers, not already existing ones. (ie: if this doesn't exist you can either uninstall / reinstall the container (or all containers) or add the applicable line to extra parameters of each container) There's absolutely nothing Unraid can do to control an app that logs the motion of every particle in the universe. The log rotation is the best that the OS (or any OS) can do. The cause of the issue (in most cases) can easily be determined by hitting Container Size on the docker tab. 2 Quote Link to comment
ausca Posted January 23, 2022 Share Posted January 23, 2022 10 minutes ago, Squid said: There's absolutely nothing Unraid can do to control an app that logs the motion of every particle in the universe. The log rotation is the best that the OS (or any OS) can do. That's fair. You can't fix all the container & app problems. I think that the log retention is a viable workaround for them. @Squid not sure where your screenshot is taken from though, but my UnRAID instance looks different to yours, even though I am running 6.9.2. I am missing the "Docker LOG number of files" and "Docker LOG maximum file size" despite setting the view to Advanced View. Quote Link to comment
Squid Posted January 23, 2022 Share Posted January 23, 2022 You have to stop the service (Enable Docker: no) 1 Quote Link to comment
cinereus Posted March 16, 2022 Share Posted March 16, 2022 On 9/7/2017 at 12:26 AM, assassinmunky said: what does the usage in /var/lib/docker look like? what is your output of this command? du -h -d 1 /var/lib/docker/ Hi, This is my out put from the two commands: docker ps -s ✔ 4.18 81% 14:12 CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES SIZE ea2f09f4daee netdata/netdata "/usr/sbin/run.sh" 7 hours ago Up 2 minutes (healthy) netdata 131MB (virtual 502MB) 8c3d263037b6 linuxserver/tautulli "/init" 4 days ago Up 3 minutes 0.0.0.0:8181->8181/tcp tautulli 37MB (virtual 161MB) b6d462e3e1d2 lscr.io/linuxserver/mariadb "/init" 4 days ago Up 3 minutes 0.0.0.0:3306->3306/tcp mariadb 16.6kB (virtual 292MB) 58b7f9e5230d lscr.io/linuxserver/swag "/init" 4 days ago Up 3 minutes 0.0.0.0:180->80/tcp, 0.0.0.0:1443->443/tcp swag 10.8MB (virtual 410MB) de9fc2887c7a testdasi/grafana-unraid-stack:latest "tini -- /static-ubu…" 4 days ago Up 3 minutes (healthy) Grafana-Unraid-Stack 32.2kB (virtual 993MB) 25be88f53c9c linuxserver/duckdns "/init" 6 days ago Up 2 minutes duckdns 11.5kB (virtual 26.6MB) 5fa90ac7906a lscr.io/linuxserver/nextcloud "/init" 8 days ago Up 3 minutes 80/tcp, 0.0.0.0:444->443/tcp nextcloud 44kB (virtual 441MB) 7f43259eb69d linuxserver/calibre "/init" 9 days ago Up 3 minutes 3000/tcp, 3389/tcp, 0.0.0.0:9080->8080/tcp, 0.0.0.0:9081->8081/tcp calibre 368MB (virtual 1.65GB) df41962f063a linuxserver/sonarr "/init" 9 days ago Up 3 minutes 0.0.0.0:8989->8989/tcp sonarr 89.3MB (virtual 722MB) e12738380365 linuxserver/radarr "/init" 9 days ago Up 3 minutes 0.0.0.0:7878->7878/tcp radarr 86.4MB (virtual 413MB) 0c04aff5c455 linuxserver/plex "/init" 9 days ago Up 3 minutes plex 40.9MB (virtual 672MB) 56c9c041eb4b binhex/arch-get-iplayer "/usr/bin/tini -g --…" 5 weeks ago Up 2 minutes binhex-get-iplayer 1.81MB (virtual 1.16GB) d91c1b568c52 jlesage/qdirstat "/init" 3 months ago Up 2 minutes 0.0.0.0:7815->5800/tcp, 0.0.0.0:7915->5900/tcp QDirStat 23.6kB (virtual 253MB) 560437ec2976 henrywhitaker3/speedtest-tracker "/init" 4 months ago Up 3 minutes 443/tcp, 0.0.0.0:3765->80/tcp speedtest-tracker 4.59MB (virtual 320MB) aedd5c214872 silverwind/droppy:latest "docker-entrypoint.s…" 4 months ago Up 3 minutes 0.0.0.0:1000->8989/tcp droppy 3.46kB (virtual 135MB) b8362151eed8 tzahi12345/youtubedl-material "/app/entrypoint.sh …" 4 months ago Up 3 minutes 0.0.0.0:1001->17442/tcp youtube-dlmaterial 1.84MB (virtual 212MB) 28a67a2b067a jlesage/handbrake "/init" 5 months ago Up 2 minutes 0.0.0.0:7803->5800/tcp, 0.0.0.0:7903->5900/tcp HandBrake 39kB (virtual 324MB) de4ab388b940 linuxserver/openvpn-as "/init" 9 months ago Up 3 minutes 0.0.0.0:943->943/tcp, 0.0.0.0:9443->9443/tcp, 0.0.0.0:1194->1194/udp openvpn-as 19.9MB (virtual 248MB) ba300091e2bf linuxserver/rutorrent "/init" 11 months ago Up 3 minutes 443/tcp, 0.0.0.0:6003->80/tcp, 0.0.0.0:8003->6881/udp, 0.0.0.0:7003->51413/tcp rutorrent-music 3.78MB (virtual 214MB) 827ac52cbb66 mace/darkstat "/sbin/my_init" 11 months ago Up 3 minutes DarkStat 402kB (virtual 307MB) 093054dbf7fa linuxserver/rutorrent "/init" 11 months ago Up 3 minutes 443/tcp, 0.0.0.0:6004->80/tcp, 0.0.0.0:8004->6881/udp, 0.0.0.0:7004->51413/tcp rutorrent-ebooks 3.78MB (virtual 214MB) eec19e8002b7 linuxserver/rutorrent "/init" 11 months ago Up 3 minutes 443/tcp, 0.0.0.0:6005->80/tcp, 0.0.0.0:8005->6881/udp, 0.0.0.0:7005->51413/tcp rutorrent-other 3.78MB (virtual 214MB) e8b783054594 linuxserver/rutorrent "/init" 11 months ago Up 3 minutes 443/tcp, 0.0.0.0:6002->80/tcp, 0.0.0.0:8002->6881/udp, 0.0.0.0:7002->51413/tcp rutorrent-tv 3.79MB (virtual 214MB) 7f23382fcb85 linuxserver/rutorrent "/init" 11 months ago Up 3 minutes 443/tcp, 0.0.0.0:6001->80/tcp, 0.0.0.0:8001->6881/udp, 0.0.0.0:7001->51413/tcp rutorrent-films 3.78MB (virtual 214MB) 65249963532e binhex/arch-krusader "/usr/bin/tini -- /b…" 11 months ago Up 2 minutes 5900/tcp, 0.0.0.0:6080->6080/tcp and du -h -d 1 /var/lib/docker/ 1.1M /var/lib/docker/containerd 44M /var/lib/docker/containers 0 /var/lib/docker/plugins 89G /var/lib/docker/btrfs 61M /var/lib/docker/image 100K /var/lib/docker/volumes 0 /var/lib/docker/trust 172K /var/lib/docker/network 0 /var/lib/docker/swarm 16K /var/lib/docker/builder 88K /var/lib/docker/buildkit 1.7M /var/lib/docker/unraid 0 /var/lib/docker/tmp 0 /var/lib/docker/runtimes 89G /var/lib/docker/ Seems it's all coming from /var/lib/docker/btrfs How do I resolve this? Quote Link to comment
Squid Posted March 16, 2022 Share Posted March 16, 2022 Better way to figure it out is to simply click container size on the docker tab. You'll see which app is storing within the image Quote Link to comment
cinereus Posted March 16, 2022 Share Posted March 16, 2022 1 hour ago, Squid said: Better way to figure it out is to simply click container size on the docker tab. You'll see which app is storing within the image Thanks I hadn't seen that before. Unfortunately it doesn't help: Name Container Writable Log --------------------------------------------------------------------- binhex-krusader 1.92 GB 35.6 MB 847 kB calibre 1.65 GB 368 MB 330 kB binhex-preclear 1.29 GB 0 B 24.3 MB binhex-get-iplayer 1.16 GB 1.81 MB 908 kB Grafana-Unraid-Stack 993 MB 0 B 3.44 kB binhex-rclone 776 MB 0 B 24.3 MB sonarr 668 MB 35.0 MB 8.42 kB plex 632 MB 316 kB 5.79 kB netdata 551 MB 180 MB 1.66 MB nextcloud 442 MB 44.0 kB 5.56 kB swag 410 MB 10.8 MB 27.4 kB radarr 327 MB 306 kB 7.30 kB HandBrake 324 MB 39.0 kB 19.3 kB speedtest-tracker 320 MB 4.59 MB 384 kB DarkStat 307 MB 403 kB 1.71 MB mariadb 292 MB 16.6 kB 17.4 kB QDirStat 253 MB 23.6 kB 411 kB openvpn-as 248 MB 19.9 MB 352 kB beets 223 MB 0 B 24.3 MB rutorrent-ebooks 214 MB 3.78 MB 1.79 MB rutorrent-films 214 MB 3.78 MB 1.79 MB rutorrent-films-4k 214 MB 3.78 MB 106 kB rutorrent-music 214 MB 3.78 MB 1.84 MB rutorrent-other 214 MB 3.78 MB 1.78 MB rutorrent-temp 214 MB 3.78 MB 9.80 kB rutorrent-temp2 214 MB 3.78 MB 8.54 kB rutorrent-tv 214 MB 3.79 MB 1.83 MB youtube-dlmaterial 212 MB 1.84 MB 23.9 kB deluge 191 MB 0 B 24.3 MB tautulli 161 MB 37.0 MB 221 kB droppy 135 MB 3.46 kB 216 kB duckdns 26.6 MB 11.5 kB 240 kB --------------------------------------------------------------------- Total size 15.2 GB 726 MB 114 MB It's showing as only 15.2GB but it grew from 70% of 80GB to 99% within 12 minutes at 7am yesterday. Quote Link to comment
Squid Posted March 16, 2022 Share Posted March 16, 2022 Docker Advanced view - See if there is a ton of orphaned containers, and delete them Quote Link to comment
cinereus Posted March 16, 2022 Share Posted March 16, 2022 5 minutes ago, Squid said: Docker Advanced view - See if there is a ton of orphaned containers, and delete them No orphaned containers. I'm always in advanced view. The issue seems twofold: 1. rapid increase in size over the course of just a few minutes 2. massive usage in /var/lib/docker/btrfs Quote Link to comment
cinereus Posted March 22, 2022 Share Posted March 22, 2022 Any ideas how to fix this? From what I read 80 GB is extreme... Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.