Howdy, I'm getting the `Out of memory errors detected on your server` every 30-60 minutes. I have been rebooting and disabling Docker containers to isolate the issue. I'm unsure what is going on, however I did get a diagnostics export (attached). 

 

Any ideas on the "Out of memory errors detected on your server" message?

 

rustymyers-diagnostics-20241004-0904 2.zip

Edited October 4, 20241 yr by rustymyers
Removed VPN crews from diagnostics zip

You've got a lot of stuff running on there, but it looks like mosquitto or openeats is spawning a bunch of threads and python3 instances. Start by disabling all of your containers then re-enabling them one by one until you find the culprit

 

Also, your VPN credentials are exposed in your diagnostics, you should remove it from your post 

 

@JorgeB

Thank you Michael_P! I removed my credentials and changed them. I've started by disabling mosquitto and openeats, hoping to not see the error again. Looks like mosquitto was updated a day ago, so perhaps an issue is happening there.

 

Much appreciated!

rusty

Still getting the Out of Memory Error. Additional diagnostics attached.

rustymyers-diagnostics-20241004-1056 2.zip

Next guess would be frigate and busybox

Thanks! I'm curious as to what you're looking for to diagnose which containers might be taking over memory?

6 minutes ago, rustymyers said:

Thanks! I'm curious as to what you're looking for to diagnose which containers might be taking over memory?

 

The reaper is consistently killing off python3 because it's got the most memory usage, and there's a lot of nginx processes, too - so I'm looking at whatever's spawning those (and frigate is always a suspect anyway)

 

The containers I see spawning those are busybox, whatever this is 6ba7fd76579f9d0638ab122feba7c103e977ac04b07fe58fcfaeab655fe0b13f (using fail2ban?), netalertx, whatever this is 1d1c308d88054708609009b1a4ab5c02180e54ad530e239c4f418fff5e8138fd, and redis (are you using immich?)

6ba7fd76579f9d0638ab122feba7c103e977ac04b07fe58fcfaeab655fe0b13f is a swag container (https://github.com/linuxserver/docker-swag). It has fail2ban and spans many nginx processes. 

 

1d1c308d88054708609009b1a4ab5c02180e54ad530e239c4f418fff5e8138fd is grocy (https://github.com/grocy/grocy). 

 

Redis is used for Manyfold, but I think that's it.

 

I got another OOM error last night and Home Assistant was the only container not running, so I wonder if that's what's causing the errors.

 

HomeAssistant is constantly dying on me, so I'm wondering if something there is being reaped?

 

Searching for the python3 process that's causing this is certainly a difficult task!

 

Thanks

rusty

21 hours ago, rustymyers said:

so I'm wondering if something there is being reaped?

It's killing python3 in your other logs and I didn't see anything but what I listed as running a python3 process, you can try running without HA for a bit and see what happens, then keep moving down the list until you find it - but I'd start with frigate as that seems to be the culprit in a lot of cases I've seen

I ran without Home Assistant for a few days and things seem to be ok. When starting it again, I got an update for the Alexa Media Player integration to fix an infinite loop and memory exhaustion: https://github.com/alandtse/alexa_media_player/pull/2589

 

After updating and running Home Assistant for another day I have not received any out of memory errors. I'm going to continue to bring containers online one by one to see if the issue returns, but for now I'm going to blame the issue above as the main culprit.

 

I'll continue to post as I check things to ensure I have isolated the issue and share my results.

 

Thanks again!

rusty