Fix Common Problems Reporting Out Of Memory server errors

[Geoff Bland]

The Fix Common Problems plugin is reporting that Out Of Memory errors are detected on my UNRAID server.

 

 

This server has 144GB of RAM and as it is mostly used for storage only 25% of the RAM is actually used.

 

 

This server has been working fine for several years. I have only had this issue occur twice now in the last 3 weeks, I have restarted my server between these reported errors. Other than this being reported by Fix Common Problems the server has been stable.

 

In the logs I just notice constant SMB warnings (100s every minute) but these can be ignored according to another post of the forums.

Mar 20 15:39:50 UNRAID02  smbd[10630]:   synthetic_pathref: opening [<user>@<path> (2023_03_20 15_04_28 UTC)] failed

 

What do I need to investigate to determine what is causing this?

 

This is UNRAID 6.11.5

 

I attach diagnostics.

 

unraid02-diagnostics-20230320-1539.zip

[Geoff Bland]

Checking the syslog I think I see that there's an error caused by a docker container:

 

Mar 20 04:27:41 UNRAID02 kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=75989cd4cdfb95d6d4cc0def9b3ed1360cd6f88458130216d3fb6f75b56036b6,mems_allowed=0-1,oom_memcg=/docker/75989cd4cdfb95d6d4cc0def9b3ed1360cd6f88458130216d3fb6f75b56036b6,task_memcg=/docker/75989cd4cdfb95d6d4cc0def9b3ed1360cd6f88458130216d3fb6f75b56036b6,task=s3cmd,pid=17525,uid=0
Mar 20 04:27:41 UNRAID02 kernel: Memory cgroup out of memory: Killed process 17525 (s3cmd) total-vm:567788kB, anon-rss:516056kB, file-rss:0kB, shmem-rss:0kB, UID:0 pgtables:1152kB oom_score_adj:0
...
Mar 20 04:28:34 UNRAID02 kernel: oom-kill:constraint=CONSTRAINT_MEMCG,nodemask=(null),cpuset=75989cd4cdfb95d6d4cc0def9b3ed1360cd6f88458130216d3fb6f75b56036b6,mems_allowed=0-1,oom_memcg=/docker/75989cd4cdfb95d6d4cc0def9b3ed1360cd6f88458130216d3fb6f75b56036b6,task_memcg=/docker/75989cd4cdfb95d6d4cc0def9b3ed1360cd6f88458130216d3fb6f75b56036b6,task=s3cmd,pid=17526,uid=0

 

But this container is set up so it can only use 2GB of ram (using extra parameter --memory=2G) and usually only uses a fraction of that. 

 

So how can this be causing Out of Memory errors on the server itself?

Edited March 20, 2023 by Geoff Bland

[JorgeB]

If it's a one time thing you can ignore, if it keeps happening try limiting more the RAM for VMs and/or docker containers, the problem is usually not just about not enough RAM but more about fragmented RAM, alternatively a small swap file on disk might help, you can use the swapfile plugin:

 

https://forums.unraid.net/topic/109342-plugin-swapfile-for-691/

[Geoff Bland]

4 hours ago, JorgeB said:

If it's a one time thing you can ignore, if it keeps happening try limiting more the RAM for VMs and/or docker containers, the problem is usually not just about not enough RAM but more about fragmented RAM, alternatively a small swap file on disk might help, you can use the swapfile plugin:

 

 

 

Thanks for the quick response.

 

>If it's a one time thing you can ignore, 

It's ocurred twice now. Both in the last 3 weeks. If it occurs again I'll post back here. 

 

>if it keeps happening try limiting more the RAM for VMs and/or docker containers, the problem is usually not just about not enough RAM but more about fragmented RAM, 

Can you explain some more on this point please? As I have less than 25% utilization of RAM on this server (over 100GB RAM free) it would be strange for fragmentation to occur, this normally only occurs when RAM usage is getting over 50%. A quick look at top shows over 104GB is used by cache and this gets reallocated as application RAM as needed.

 

>alternatively a small swap file on disk might help, you can use the swapfile plugin:

Can you also explain a bit more about this, as I understood it virtual memort won't use swap unless the server gets very low on physical RAM (regardless of any fragmentation)?
 

[Geoff Bland]

OK I've done some further research.

 

This may be a false alarm from the Fix Common Problems plugin.

 

The actual error is "Memory cgroup out of memory", this is not the server running out of memory - rather a single container, which was constrained to 2GB total, has run out of memory.

 

The OS has restarted just that container and flagged this warning. 

 

So the warning from Fix Common Problems about "Your server has run out of memory..." is a bit misleading (and overly worrying). Perhaps Fix Common Problems can detect this as a different kind of error and report "Your container X has run out of memory..." 

 

 

[JorgeB]

You should post that on the plugin support thread.

[Geoff Bland]

8 hours ago, JorgeB said:

You should post that on the plugin support thread.

 

Yes, done. Thanks.