Out of memory error after < 2 days of uptime w/ 128GB of RAM

[CorneliousJD]

Upgraded to 10.6.2, had an issue where the web interface was very, very slow, so I rebooted.

That resolved the issues, but now I have less than 2 days of uptime and got this alert.

 

Out Of Memory errors detected on your server

Your server has run out of memory, and processes (potentially required) are being killed off. You should post your diagnostics and ask for assistance on the Unraid forums

 

Also after getting this and looking in the AM, I'm seeing very normal usage.

 

Attached are my diags.

 

server-diagnostics-20220531-0816.zip

[CorneliousJD]

Bump, as I've rebooted after this and now a little over one day (about 30-35 hours) I am getting the same * **Out Of Memory errors detected on your server** 

[Frank1940]

Start the GUI Terminal and type the following command:

ls -al /mnt

Post up a screen shot of the output.

 

Then run this command:

ls -al /mnt/user

Again get a screenshot and post it up.  Now compare the names of the User Shares  (found under the 'Shares' tab on the GUI) with this screenshot.  Any directory in the screenshot that is not in the User Share list is probably the problem.

[CorneliousJD]

9 minutes ago, Frank1940 said:

Start the GUI Terminal and type the following command:

ls -al /mnt

Post up a screen shot of the output.

 

Then run this command:

ls -al /mnt/user

Again get a screenshot and post it up.  Now compare the names of the User Shares  (found under the 'Shares' tab on the GUI) with this screenshot.  Any directory in the screenshot that is not in the User Share list is probably the problem.

 

Here's the screenshots -- the User Share list also matches the output of the ls -al /mnt/user

 

Side note - could this error because by a docker container running out of memory? I did limit a container to 1GB of memory recently, this could correlate to when I started seeing the issue. 

 

 

 

 

[Frank1940]

I had a look at your screen captures and I observed the following.

 

I have noticed that you do not have a docker.img file listed.  I would have expected to see it there.

 

This is where the location of that file is set:

 

 

Make sure the 'Advance view'  (#1) is turned on.   Then check paths #2 and #3.  These are the defaults.

[CorneliousJD]

19 minutes ago, Frank1940 said:

I had a look at your screen captures and I observed the following.

 

I have noticed that you do not have a docker.img file listed.  I would have expected to see it there.

 

This is where the location of that file is set:

 

 

Make sure the 'Advance view'  (#1) is turned on.   Then check paths #2 and #3.  These are the defaults.

 

Here's my setup - I must have moved it originally thinking that the system share was always going to be cache-only or something?

Here's the ls -al /mnt/user/system/docker as well

 

 

[Frank1940]

I have the feeling that you have some experience in dealing with system level issues.  (I an also one of the persons.  I know just enough to make me really, really dangerous at times!)  What I suspect is that something is writing files to the RAM disk that Unraid is installed on.  I would suspect that it is one of your Dockers.  The problem is figuring out which one it is. 

 

What you could try to reboot your server and on restart, stop each one of your Dockers until you find out which one it is.  I would begin by trying to recall which dockers I made changes from the default configuration and start with those. 

 

One thing I want to point out.  Linux is case sensitive.    From example: the path     /mnt/disk1    is on a hard disk.    The path    /Mnt/disk1   is on the RAM disk!  (I do believe that all of the default paths that Unraid use all lower case letters.)    Another example:   /mnt/disk9   would be in RAM if you  don't have a physical disk 9.  (I am assuming that you have eight physical data disks installed on your system!)

 

One thing that bothers me.  Notice the color of the User Share 'nextcloud' in your screenshot of the ls-al /mnt/user command.  It is blue when all of the other shares are green.  I tried to find out what is the significance of that one being blue and I could not find it.  (Also notice that the permissions are different...) So I would be looking whatever is controlling that User Share.

 

[CorneliousJD]

22 minutes ago, Frank1940 said:

I have the feeling that you have some experience in dealing with system level issues.  (I an also one of the persons.  I know just enough to make me really, really dangerous at times!)  What I suspect is that something is writing files to the RAM disk that Unraid is installed on.  I would suspect that it is one of your Dockers.  The problem is figuring out which one it is. 

 

What you could try to reboot your server and on restart, stop each one of your Dockers until you find out which one it is.  I would begin by trying to recall which dockers I made changes from the default configuration and start with those. 

 

One thing I want to point out.  Linux is case sensitive.    From example: the path     /mnt/disk1    is on a hard disk.    The path    /Mnt/disk1   is on the RAM disk!  (I do believe that all of the default paths that Unraid use all lower case letters.)    Another example:   /mnt/disk9   would be in RAM if you  don't have a physical disk 9.  (I am assuming that you have eight physical data disks installed on your system!)

 

One thing that bothers me.  Notice the color of the User Share 'nextcloud' in your screenshot of the ls-al /mnt/user command.  It is blue when all of the other shares are green.  I tried to find out what is the significance of that one being blue and I could not find it.  (Also notice that the permissions are different...) So I would be looking whatever is controlling that User Share.

 

 

 

Thanks for the info -- I actually no longer use that nextcloud share, I will just be getting rid of it entirely. problem solved w/ the odd permissions then

 

Regarding the out of memory error, if it's possible that it's caused by a docker, then I think it is the 1GB docker limit I set on Nginx Proxy Manager 

I am seeing this in one of the lines of the logs.

 memory: usage 1048576kB, limit 1048576kB, failcnt 12250

 

That kb count equals exactly the 1GB limit which I had set on this container.

I have now changed it to 2GB for now, but I may remove the limit entirely I guess, as I have 128GB of RAM, so plenty to spare.

This template came this way from CA w/ the 1GB limit, so I left it, but I may be using it a bit more heavily than most others... 

 

 

[CorneliousJD]

I removed the memory limit on that container and it seems to have resolved this issue. Fingers crossed!