Help Requested: Out of memory errors detected on your server

[CMJ1015]

Hi All. I received the above "Out of memory errors detected on your server" in Fix Common problems.

 

Can someone help me determine where the issue lies? Diagnostics attached.

 

Thanks in advance for anyone who has the time to take a look.

tower-diagnostics-20220801-0820.zip

[weirdcrap]

It looks like way back in May python invoked the out of memory killer and it killed QEMU.

 

May  7 21:05:54 Tower kernel: python3 invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0

....

May  7 21:05:54 Tower kernel: oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=3ea9e18112ed2dd06031f35c6351a2de56ee7b8fb2ce9631019b9f7ca51e3881,mems_allowed=0,global_oom,task_memcg=/machine/qemu-1-V-DLC.libvirt-qemu,task=qemu-system-x86,pid=15497,uid=0
May  7 21:05:54 Tower kernel: Out of memory: Killed process 15497 (qemu-system-x86) total-vm:17946876kB, anon-rss:17028860kB, file-rss:72kB, shmem-rss:23104kB, UID:0 pgtables:35208kB oom_score_adj:0
May  7 21:05:54 Tower kernel: oom_reaper: reaped process 15497 (qemu-system-x86), now anon-rss:16kB, file-rss:68kB, shmem-rss:4kB

 

I only see the one OOM kill so it hasn't happened again in several months so it is probably safe to simply ignore it unless it happens again.

 

You appear to have some VMs running, assigned about 24GB of RAM if I'm reading your XMLs correctly. That plus the dockers listed in your sig you may be pushing it a little to close on your RAM usage sometimes as the underlying OS needs some RAM for itself.

 

You could try setting artificial limits for the dockers (using --memory=xG in the extra parameters for each docker where X is the limit in GB) and reducing the assigned RAM on the VMs a gigabyte or two at a time until the errors stop for good.

 

But again if this only happened the one time and things seemed to be running OK otherwise I'd probably just leave it be.

Edited August 1, 2022 by weirdcrap

[CMJ1015]

Thank you so much for your help. I'll look into adjusting the RAM allocation.