[Solved] Out of Memory errors however memory is hovering around 70%?

[HNGamingUK]

Hello,

 

For reference my server has 32GB of RAM.

 

I wonder if anyone else is able to help me, I am unsure on what is causing this and what I can do to stop it.

 

So far for the past couple of nights I have had fix common problems send an alert about out of memory errors and when I login I see that the MySQL docker has stopped. (Only this one stops so I am guessing OOM killer decides to kill that one for whatever reason)

 

However my memory on the dashboard is hovering around 70% max so I am unsure on what is causing it.

 

I have set each docker to have max memory so that they can't randomly use more memory, however it still keeps happening.

 

The only other item that I run contstantly is a Windows VM (16GB assigned to it) and as such should leave 16GB for the dockers and system.

 

I have also attached my diagnostics, if anyone can help me find out what is causing this it would be wonderful. I don't have the capcity to just upgrade my RAM currently and so need to fix this issue. diagnostics.zip

 

Edited May 1, 2020 by Conmyster

[testdasi]

The 70% utilisation you see right now is after the offending process has been killed so it's likely a red herring.

 

OOM error happens if (a) you actually run out of memory or (b) a docker runs over its memory limit or (c) unusual problem

 

Did you set the dockers memory limits before or after mysql docker was killed?

If before then you need to increase the limit for mysql as it was killed because of reason (b) above.

 

And OOM is not an urgent issue to fix as long as it doesn't kill your running docker / VM.

 

 

Side note: Reason (c) is really unusual and not something you need to worry about. For example, my TR4 server with F12 BIOS will have OOM error if I allocate more than 50% RAM all at once (e.g. start a VM) while having Global C State disabled (enable C state and no more error).

[HNGamingUK]

7 minutes ago, testdasi said:

The 70% utilisation you see right now is after the offending process has been killed so it's likely a red herring.

 

OOM error happens if (a) you actually run out of memory or (b) a docker runs over its memory limit or (c) unusual problem

 

Did you set the dockers memory limits before or after mysql docker was killed?

If before then you need to increase the limit for mysql as it was killed because of reason (b) above.

 

And OOM is not an urgent issue to fix as long as it doesn't kill your running docker / VM.

 

 

Side note: Reason (c) is really unusual and not something you need to worry about. For example, my TR4 server with F12 BIOS will have OOM error if I allocate more than 50% RAM all at once (e.g. start a VM) while having Global C State disabled (enable C state and no more error).

 

Yes I set the MySQL and other docker limits before MySQL was getting killed.

 

I assumed that if the container got close to the limit it would just not use more, but reading your messages suggests that if it goes over the limit I set it will just kill the docker?

Edited May 1, 2020 by Conmyster

[testdasi]

37 minutes ago, Conmyster said:

 

Yes I set the MySQL and other docker limits before MySQL was getting killed.

 

I assumed that if the container got close to the limit it would just not use more, but reading your messages suggests that if it goes over the limit I set it will just kill the docker?

Indeed the limit is for the kernel to know when to kill a docker.

I don't think the docker itself is even aware of the limit (perhaps it can be aware but I don't think that piece of code implemented in all the dockers that I can tell).

 

There's also a side effect for setting docker memory limit is that it is less likely to kill your VM if you actually run out of memory. So don't just go undo all your docker limit.

[HNGamingUK]

Just now, testdasi said:

Indeed the limit is for the kernel to know when to kill a docker.

I don't think the docker itself is even aware of the limit (perhaps it can be aware but I don't think that piece of code implemented in all the dockers that I can tell).

 

There's also a side effect for setting docker memory limit is that it is less likely to kill your VM if you actually run out of memory. So don't just go undo all your docker limit.

Yeh I didn't want my VM to die randomly as I use it for gaming.

 

I will have a look at working out some better docker memory limits. Thanks for helping me understand how it works.