So I'm running 6.11 now and the server has stayed up, which is great! Thanks for the suggestion @shaunvis!

 

However, my suspicion may be correct as I was able to capture an OOM issue in the logs. By chance can anyone determine what happened?

truffle-diagnostics-20230928-0905.zip

Edited September 28, 20232 yr by flyize

OOM in the log shows what was killed, usually one the the largest memory consumers, not what caused the OOM, you can try limiting more the RAM for VMs and/or docker containers, the problem is usually not just about not enough RAM but more about fragmented RAM, alternatively a small swap file on disk might help, you can use the swapfile plugin:

 

https://forums.unraid.net/topic/109342-plugin-swapfile-for-691/

Interesting. I assume you can see that I have 80GB in there and I'm pretty sure I'm not hitting that. The way you describe it, its a Linux problem. That seems odd. In the real world, that's solved by adding a swap file?

23 minutes ago, flyize said:

that's solved by adding a swap file?

yes, or 

 

29 minutes ago, JorgeB said:

you can try limiting more the RAM for VMs and/or docker containers

 

@shaunvis As expected, @JorgeB seems to be correct in that memory fragmentation was causing my issues. 6.11 seems to handle this *much* better, in that it only seemed to kill a couple of things - and most importantly put it in syslog. 6.12 seemed to kill almost everything, and had ZERO logging of it.

 

I solved it with a swap file, but it could seemingly also be solved by compacting memory according to these threads:

https://stackoverflow.com/questions/62077590/why-can-a-user-process-invoke-the-linux-oom-killer-due-to-memory-fragmentation

https://stackoverflow.com/questions/60079327/how-to-mmap-a-large-file-without-risking-the-oom-killer/62855363#62855363

 

So maybe try 6.12 again but with swap and see if it help you too.

On 9/29/2023 at 7:54 AM, flyize said:

@shaunvis As expected, @JorgeB seems to be correct in that memory fragmentation was causing my issues. 6.11 seems to handle this *much* better, in that it only seemed to kill a couple of things - and most importantly put it in syslog. 6.12 seemed to kill almost everything, and had ZERO logging of it.

 

I solved it with a swap file, but it could seemingly also be solved by compacting memory according to these threads:

https://stackoverflow.com/questions/62077590/why-can-a-user-process-invoke-the-linux-oom-killer-due-to-memory-fragmentation

https://stackoverflow.com/questions/60079327/how-to-mmap-a-large-file-without-risking-the-oom-killer/62855363#62855363

 

So maybe try 6.12 again but with swap and see if it help you too.

I've actually had OOM issues on 6.11 with no apparent fix. Mine is "avahi-daemon" causing it. 

I hadn't found the swapfile solution or any help at all for my OOM. I'll give that a shot. If that stops the OOM in 6.11, maybe I can get 6.12 to run longer than a few hours. 

Thanks! I'll give that a shot

I didn't get the swap file set up, I was reading other people that set up a swap file & the OOM just filled that up too.  But your post sent me down a rabbit hole that I think FINALLY fixed my OOM errors. 

 

It looks like the "Unassigned Devices" plugin was causing avahi-daemon to eat up all my RAM until it was killed. I reinstalled it had it's been OK since. 

 

Now I'm wondering of your experience of .12 locking up from issues like that instead of showing an OOM errors in the log is what I was seeing. I might tempt 6.12 again to see if it works now. 

On 10/3/2023 at 8:18 AM, shaunvis said:

I didn't get the swap file set up, I was reading other people that set up a swap file & the OOM just filled that up too.  But your post sent me down a rabbit hole that I think FINALLY fixed my OOM errors. 

 

It looks like the "Unassigned Devices" plugin was causing avahi-daemon to eat up all my RAM until it was killed. I reinstalled it had it's been OK since. 

 

Now I'm wondering of your experience of .12 locking up from issues like that instead of showing an OOM errors in the log is what I was seeing. I might tempt 6.12 again to see if it works now. 

How did you determine that it was Unassigned Devices? I too have that installed.

 

edit: Also, after upgrading to the new 6.12 the other day, my server crashed again this morning. Same thing OOM killed everything, but server still responds to pings. I just downgraded back to 6.11 and since I'm headed out of town for a few days, I've set the RAM compact script (posted in the SO threads I linked) to run hourly.

Edited October 4, 20232 yr by flyize

On 9/28/2023 at 10:36 AM, JorgeB said:

yes, or 

 

 

Any chance you have any other suggestions? I thought it was fixed, but OOM is still killing things. I can confirm its not running out of RAM (80GB), I've added a 40GB swap file, and I've started running the memory compact command to defrag the RAM.

1 hour ago, flyize said:

Any chance you have any other suggestions?

Not really anything more than what I've written above, you can try and find which plugin/container is causing that.

But if I'm really not running out of RAM, and I think the fragmentation issue should be properly band-aided, there's nothing that *should* be causing it. Any ideas on how to further troubleshoot this?

Create the small script below with the user scripts plugin and schedule it to run hourly, it will output the memory stats to the syslog, then see if there's anything abnormal in the persistent syslog.

 

#!/bin/bash
free -h |& logger &

 

Will do. I had seutp netdata for that purpose, but it seems that it doesn't store data for very long.

On 10/8/2023 at 5:46 AM, JorgeB said:

Create the small script below with the user scripts plugin and schedule it to run hourly, it will output the memory stats to the syslog, then see if there's anything abnormal in the persistent syslog.

 

#!/bin/bash
free -h |& logger &

 

Just happened again. Memory seems fine?

truffle-diagnostics-20231010-0850.zip

Memory usage seemed fine until last stats at 6:47AM, but then the was an OOM event at 7:39AM, swap when from 7.3G used to fully used, seems to me that it should not be that hard to find out the culprit by disabling docker, confirm the problem stops occurring, then re-enable one container at a time to see if you can find the one causing a memory leak.