December 5, 20241 yr This apparently has been an issue for some time but I didn't notice as my notification agent was not set up correctly. I only picked this up after running Fix Common Problems. I have narrowed the issue down (I think) to my Frigate container. When the issue happens, I shut down Frigate and docker image returns to normal almost immediately. I have been working to diagnose it for two weeks now and have been posting on the Frigate GitHub to get help. But other than the evidence above I have not been able to find a smoking gun. Frigate is running through SWAG for external access. I have seen some issues with this and other containers, such as Plex. So I am wondering if this could be causing the issue? I switched to NGINX just to see but it did the same thing. Frigate is setup with these arguments: --shm-size=512mb --mount type=tmpfs,target=/tmp/cache,tmpfs-size=1000000000 --restart unless-stopped --runtime=nvidia Checking all of the container maps nothing should be an issue: I don't think I am using this so I probably need to remove it. I continue to monitor the /tmp to see if there are files there that can provide some clues but I guess this is a matter of timing as I don't seem to find anything large there. So what are some of my next options to try? Thanks. odin-diagnostics-20241205-0756.zip
December 5, 20241 yr Author 1 hour ago, JorgeB said: See if this helps: Thanks - have tried that. I also ran the remove orphaned images and unconnected docker volumes and both reported 0 bytes removed. But this doesn't make sense - My docker is set to 25GB and yet is showing this in the docker setup: Label: none uuid: 539c2bc6-d6d9-461b-ae29-22dc0f3538f0 Total devices 1 FS bytes used 95.81GiB devid 1 size 100.00GiB used 100.00GiB path /dev/loop2 At one point I did set the docker image to 100GB to allow myself some space to figure out what is going on. I moved it back to 25GB at some point but this seems to show that it is holding at 100GB? I did run the scrub - not sure whether that is relevant. Right now the docker image is reporting 100% full. I restart Frigate and I immediately get a returned to normal. So stuck. Space invader script output is shown below but shows nothing out of the ordinary: Script location: /tmp/user.scripts/tmpScripts/check_docker_image/script Note that closing this window will abort the execution of this script ################################################################################## Cleanup before starting (if requested in script) ################################################################################## Not removing orphaned images (this can be set in script if you want to) --------------------------------------------------------------------------------- Not removing unconnected docker volumes (this can be set in script if you want to) ################################################################################## List of Image, Container and docker volume size. ################################################################################## There are 21 Images taking up ......17.47GB There are 21 Containers taking up ......190.9MB There are 8 Local Volumes taking up ......50.04kB There are 0 Build Cache taking up ......0B ################################################################################## List of containers showing size and virtual size ################################################################################## First size is the writable layers of the container (Virtual size is writable and read only layers) 189kB (virtual 180MB) Is being taken up by ......... jlesage/nginx-proxy-manager 4.38MB (virtual 1.92GB) Is being taken up by ......... ghcr.io/blakeblackshear/frigate:stable 348kB (virtual 402MB) Is being taken up by ......... lscr.io/linuxserver/swag 22.4kB (virtual 202MB) Is being taken up by ......... lscr.io/linuxserver/radarr:latest 23.7MB (virtual 2.61GB) Is being taken up by ......... binhex/arch-krusader 25.5kB (virtual 614MB) Is being taken up by ......... lscr.io/linuxserver/code-server 116MB (virtual 316MB) Is being taken up by ......... lscr.io/linuxserver/sonarr:latest 24.2kB (virtual 182MB) Is being taken up by ......... lscr.io/linuxserver/sabnzbd 24.9MB (virtual 2.93GB) Is being taken up by ......... lscr.io/linuxserver/calibre 0B (virtual 1.38GB) Is being taken up by ......... fallenbagel/jellyseerr:latest 78kB (virtual 237MB) Is being taken up by ......... ghcr.io/hotio/readarr:latest 0B (virtual 302MB) Is being taken up by ......... portainer/portainer-ce:latest 17.8kB (virtual 177MB) Is being taken up by ......... ich777/openvpn-client 1.26MB (virtual 716MB) Is being taken up by ......... quantumentangledandy/neolink 7.53MB (virtual 680MB) Is being taken up by ......... emby/embyserver:latest 3.53kB (virtual 105MB) Is being taken up by ......... ich777/pushover-on-start 791kB (virtual 1.02GB) Is being taken up by ......... mmcc73/birdcage_frontend:latest 1.71MB (virtual 2.24GB) Is being taken up by ......... mmcc73/birdcage_backend:latest 9.84MB (virtual 2.61GB) Is being taken up by ......... mmcc73/birdnetserver:latest 0B (virtual 116MB) Is being taken up by ......... redis:latest 0B (virtual 25.7MB) Is being taken up by ......... spants/mqtt ################################################################################## List of containers in size order ################################################################################## 2910MB - lscr.io/linuxserver/calibre 2610MB - mmcc73/birdnetserver 2580MB - binhex/arch-krusader 2240MB - mmcc73/birdcage_backend 1920MB - ghcr.io/blakeblackshear/frigate 1380MB - fallenbagel/jellyseerr 1020MB - mmcc73/birdcage_frontend 714MB - quantumentangledandy/neolink 672MB - emby/embyserver 614MB - lscr.io/linuxserver/code-server 401MB - lscr.io/linuxserver/swag 302MB - portainer/portainer-ce 237MB - ghcr.io/hotio/readarr 202MB - lscr.io/linuxserver/radarr 199MB - lscr.io/linuxserver/sonarr 182MB - lscr.io/linuxserver/sabnzbd 180MB - jlesage/nginx-proxy-manager 177MB - ich777/openvpn-client 116MB - redis 105MB - ich777/pushover-on-start 25.7MB - spants/mqtt ################################################################################## List of docker volumes, the container which they are connected to their size ################################################################################## ID 4a80d720986c44fdf9709def6bd0da1c52f273075b9670fdb251b67c33785b7c This volume connected to has a size of 4.0K ID 4aef825c8565a9af136b33b29f6f771feb71ceee3707e9a3e421a573061d947a This volume connected to has a size of 4.0K ID 56ee3c188838eb81aa0d0f2e9319197df6a8c9c7371a25b8195c1d912745928a This volume connected to has a size of 4.0K ID 65eaef7402ad3ab0bdc7bd308c7d55c9f51d10e2021d51c7dbad6d5dc73b7430 This volume connected to birdcage-redis-1 has a size of 48K ID 73f23ee9023bac733edcc8ba5bdd97edcc07ed240651724e6818df6adc322f9b This volume connected to has a size of 0 ID 61088a7232ea99eb2d39121662684d7222c6371b451d6a034215bbcba9a70d02 This volume connected to has a size of 0 ID c9419b5fb422186ae1494253acedcb6d032f3aa26eb6e20e0df638f51d52670b This volume connected to has a size of 4.0K ID cc5c1f57dbc07f691176849bdb37f23ec8fd599f234b0912208abc95ea678915 This volume connected to has a size of 4.0K ################################################################################## Done. Scroll up to view results DONE
December 5, 20241 yr Community Expert Try recreating the image, and check the used space after, and if it keeps growing, there's still a problem: https://docs.unraid.net/unraid-os/manual/docker-management/#re-create-the-docker-image-file Then: https://docs.unraid.net/unraid-os/manual/docker-management/#re-installing-docker-applications Also see below if you have any custom docker networks: https://docs.unraid.net/unraid-os/manual/docker-management/#docker-custom-networks
December 9, 20241 yr Author Hi, I recreated the vdisk and I am still having the same problem. Is there nothing I can look at when the image starts filling up to see what is causing it? Container size says ~8.3GB and I have the docker image set to 30GB now. odin-diagnostics-20241208-2003.zip
December 9, 20241 yr Community Expert If it image keeps growing, there's like an incorrect mapping in one of the containers.
December 9, 20241 yr Community Expert I don't know much about Frigate but to my understanding it does transcode video streams from your surveillance system. These files have to be written somewhere right? Where are you transcoding the video streams to? Could this be your issue is frigate is transcoding the video into the docker image instead of onto a disk? What about the mount for /tmp/cache? Edited December 9, 20241 yr by MowMdown
December 9, 20241 yr Author I recall looking at cache but will do that the next time it happens - which won't be long. It is definitely frigate. The image went into warning mode this morning and I shutdown Frigate and left SWAG running and things returned to normal. I have posted on the GitHub for Frigate about the transcoding and got the response that Frigate only uses small segments so this can't be the issue. This may be getting out of my depth but a LOT of small segments could lead to a problem?
December 9, 20241 yr Community Expert If the segments aren't being cleaned up it could lead to a ballooning docker image.
December 9, 20241 yr Author I am working my way through the options in Frigate, turning them off, and seeing if there is a problem. I have started with turning off 24 hour recording to see if that is the issue or if the motion events are what is causing it. One thing I did want to try is stepping back to an older version. It occurred to me that since I have not changed settings in a while then this could have started with a specific release. I am a little unclear how to do this although I am searching around looking for folks that have done this. The docker points to: ghcr.io/blakeblackshear/frigate:stable The ui says version 0.14.1-f4f3cfa Yet I can't figure out what the last stable version was. And if I had it can I specify this in the repository line?
December 11, 20241 yr Author Just continuing to update on this. I did step back to the previous version to see if it is something that occurred in an update. No dice. I could go even further back - something to consider. It is a little frustrating because the only evidence I have of the docker image filling up is this: Nowhere else can I find evidence of what is going on and where. I have done the following: Check container size button - 8.54GB Run the docker log size script - ~20MB Run the Space Invader script - 10.29GB Checked the /tmp and /dev/shm both inside the console for the docker and in general and see very low counts Is there really no way to figure out where the files are that are causing that meter on the dashboard to report 85%. What does it represent? It just went to 86%.
December 11, 20241 yr Community Expert what does "du -h" and "df -h" report when ran inside the containers?
December 11, 20241 yr Author 3 minutes ago, MowMdown said: what does "du -h" and "df -h" report when ran inside the containers? Not sure what I am looking at here - but I see some large numbers at the bottom? Would appreciate help in interpreting. # du -h 92K ./frigate/api/__pycache__ 232K ./frigate/api 40K ./frigate/comms/__pycache__ 92K ./frigate/comms 52K ./frigate/detectors/plugins/__pycache__ 108K ./frigate/detectors/plugins 24K ./frigate/detectors/__pycache__ 156K ./frigate/detectors 44K ./frigate/events/__pycache__ 92K ./frigate/events 148K ./frigate/images 12K ./frigate/motion/__pycache__ 32K ./frigate/motion 40K ./frigate/output/__pycache__ 96K ./frigate/output 48K ./frigate/ptz/__pycache__ 136K ./frigate/ptz 44K ./frigate/record/__pycache__ 104K ./frigate/record 20K ./frigate/review/__pycache__ 48K ./frigate/review 16K ./frigate/stats/__pycache__ 32K ./frigate/stats 148K ./frigate/test 16K ./frigate/track/__pycache__ 48K ./frigate/track 64K ./frigate/util/__pycache__ 148K ./frigate/util 192K ./frigate/__pycache__ 2.0M ./frigate 100K ./migrations 9.7M ./web/assets 2.0M ./web/fonts 40K ./web/images 12M ./web 14M . # df -h Filesystem Size Used Avail Use% Mounted on /dev/loop2 50G 42G 6.4G 87% / tmpfs 64M 0 64M 0% /dev shm 512M 19M 494M 4% /dev/shm shfs 36T 24T 12T 67% /config overlay 32G 1.6G 30G 5% /usr/share/zoneinfo/Etc/UTC /dev/loop2 50G 42G 6.4G 87% /etc/hosts tmpfs 954M 124M 831M 13% /tmp/cache tmpfs 32G 12K 32G 1% /proc/driver/nvidia overlay 32G 1.6G 30G 5% /usr/bin/nvidia-smi overlay 32G 1.6G 30G 5% /lib/firmware/nvidia/565.57.01/gsp_ga10x.bin devtmpfs 8.0M 0 8.0M 0% /dev/nvidia0
December 11, 20241 yr Author 12 minutes ago, MowMdown said: what does "du -h" and "df -h" report when ran inside the containers? I see this in NGINX Proxy Manager. Frigate is running through that for external camera feeds: /tmp # du -h 0 ./run/user/app 0 ./run/user 0 ./run 0 . /tmp # df -h Filesystem Size Used Available Use% Mounted on /dev/loop2 50.0G 41.5G 6.3G 87% / tmpfs 64.0M 0 64.0M 0% /dev shm 64.0M 0 64.0M 0% /dev/shm shfs 931.5G 310.4G 618.4G 33% /config /dev/loop2 50.0G 41.5G 6.3G 87% /etc/resolv.conf /dev/loop2 50.0G 41.5G 6.3G 87% /etc/hostname /dev/loop2 50.0G 41.5G 6.3G 87% /etc/hosts tmpfs 31.4G 0 31.4G 0% /proc/acpi tmpfs 64.0M 0 64.0M 0% /proc/kcore tmpfs 64.0M 0 64.0M 0% /proc/keys tmpfs 64.0M 0 64.0M 0% /proc/timer_list tmpfs 31.4G 0 31.4G 0% /sys/firmware tmpfs 31.4G 0 31.4G 0% /sys/devices/virtual/powercap /tmp #
December 11, 20241 yr Community Expert 7 minutes ago, The Transplant said: Not sure what I am looking at here - but I see some large numbers at the bottom? Would appreciate help in interpreting. # du -h 92K ./frigate/api/__pycache__ 232K ./frigate/api 40K ./frigate/comms/__pycache__ 92K ./frigate/comms 52K ./frigate/detectors/plugins/__pycache__ 108K ./frigate/detectors/plugins 24K ./frigate/detectors/__pycache__ 156K ./frigate/detectors 44K ./frigate/events/__pycache__ 92K ./frigate/events 148K ./frigate/images 12K ./frigate/motion/__pycache__ 32K ./frigate/motion 40K ./frigate/output/__pycache__ 96K ./frigate/output 48K ./frigate/ptz/__pycache__ 136K ./frigate/ptz 44K ./frigate/record/__pycache__ 104K ./frigate/record 20K ./frigate/review/__pycache__ 48K ./frigate/review 16K ./frigate/stats/__pycache__ 32K ./frigate/stats 148K ./frigate/test 16K ./frigate/track/__pycache__ 48K ./frigate/track 64K ./frigate/util/__pycache__ 148K ./frigate/util 192K ./frigate/__pycache__ 2.0M ./frigate 100K ./migrations 9.7M ./web/assets 2.0M ./web/fonts 40K ./web/images 12M ./web 14M . # df -h Filesystem Size Used Avail Use% Mounted on /dev/loop2 50G 42G 6.4G 87% / tmpfs 64M 0 64M 0% /dev shm 512M 19M 494M 4% /dev/shm shfs 36T 24T 12T 67% /config overlay 32G 1.6G 30G 5% /usr/share/zoneinfo/Etc/UTC /dev/loop2 50G 42G 6.4G 87% /etc/hosts tmpfs 954M 124M 831M 13% /tmp/cache tmpfs 32G 12K 32G 1% /proc/driver/nvidia overlay 32G 1.6G 30G 5% /usr/bin/nvidia-smi overlay 32G 1.6G 30G 5% /lib/firmware/nvidia/565.57.01/gsp_ga10x.bin devtmpfs 8.0M 0 8.0M 0% /dev/nvidia0 Was trying to see if there was a specific directory ballooning out of proportion but I don't see anything like that.
December 11, 20241 yr Community Expert what does the container size button show on the docker containers page?
December 11, 20241 yr Author 6 minutes ago, MowMdown said: Was trying to see if there was a specific directory ballooning out of proportion but I don't see anything like that. I also posted the NGINX Proxy Manager as I have suspicions that the issue is between Frigate and the reverse proxy. Let me know if those numbers look ok. Also, what line items specifically would I be looking at that are blowing up? With docker container size around 9GB and image set at 25GB one assumes I am looking for something in the 10+GB range to achieve 80+%. Thanks
December 11, 20241 yr Author 6 minutes ago, MowMdown said: what does the container size button show on the docker containers page?
December 11, 20241 yr Community Expert 8 minutes ago, The Transplant said: It appears frigate is only using ~2GB of space. Are you 100% sure that your docker image isn't set to 10GB? because 86% of 10GB is 8.6GB which is how much space is currently being used. Something isn't adding up here and nothing appears to be ballooning out of control from what I see here.
December 11, 20241 yr Community Expert I did a test, I wrote a 10GB file inside one of my dockers to show you what it would look like:
December 11, 20241 yr Author Just now, MowMdown said: I did a test, I wrote a 10GB file inside one of my dockers to show you what it would look like: So I should be seeing something in container size for one of my dockers? I am almost 100% positive it is set to 25GB. I don't know a way to figure out the size without shutting down and I want to leave it running a little longer in its blown up state to see what is going on. The growth has slowed down significantly which points down to the external streaming of my cameras. I shutdown all of the devices that were pulling camera feeds. But still concerned that I can't see a smoking gun in terms of a file system that has gotten big.
December 11, 20241 yr Community Expert 1 minute ago, The Transplant said: I am almost 100% positive it is set to 25GB. I don't know a way to figure out the size without shutting down Go to the settings page, click docker, you might have to enabled the advanced view. There should be a default size setting. Alternatively you can see what the docker.img file's reported size is by checking the file from the webgui under your docker share.
December 11, 20241 yr Author 20 minutes ago, MowMdown said: Go to the settings page, click docker, you might have to enabled the advanced view. There should be a default size setting. Alternatively you can see what the docker.img file's reported size is by checking the file from the webgui under your docker share. I don't see it on the docker settings page. I think you have to stop the docker service. I do see this at the bottom but nothing here matches with what I would expect: And this is what shows under the share: Edited December 11, 20241 yr by The Transplant
December 11, 20241 yr Community Expert Yeah, I'm not really sure where to go from here honestly. I've really never had any of my own containers go rogue and fill up like that.
December 11, 20241 yr Author 16 minutes ago, MowMdown said: Yeah, I'm not really sure where to go from here honestly. I've really never had any of my own containers go rogue and fill up like that. It is a mystery. Thanks for looking. Hoping that someone from Unraid can weigh in and offer advice on figuring out what is in the image.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.