nginx running out of shared memory

[amateur_wizard]

 

- Saw these errors so I ran "/etc/rc.d/rc.nginx restart"

- The connection to system went down (as expected) took about 15 minutes to come back online on My Server but still wasn't reachable. 
- I VPN'd to the router, still wasn't able to get the WebUI with the LAN IP.

- Ping returned a reply, SSH'd into the server, then did a 'sudo reboot'

 

This is especially scary when you're working in another country 👀 

[sunbear]

Just had this happen to me.

 

Rebooted server.

 

Any info on the cause or a fix?

[amateur_wizard]

@sunbear By any chance is you MyServers plugin out of date? Mine was and I think it is related to the problem but I don't want to jump to any conclusion.

[sunbear]

On 3/14/2023 at 3:51 PM, amateur_wizard said:

@sunbear By any chance is you MyServers plugin out of date? Mine was and I think it is related to the problem but I don't want to jump to any conclusion.

I believe I have updated since then but I think you may be correct.

[SirKelsALot]

Same issue here. No updates for the My Servers plug-in show available, that is on 2023.03.09.1140 and I am on OS version 6.11.5.

unraid-diagnostics-20230323-1652.zip

[MyKroFt]

i just had the same thing happen to me 1st time.  shutdown the array did a rc.nginx stop, wait it to shutdown the service and API, did a start and at the moment not getting the error anymore for the time being.  i did have the red triangle ! on the myserver, I disabled it, went to work and come home and it was reconnected to the API, restarted it and all seemed well and it did a flash basckup.

 

and yes everything is updated

Edited March 25 by MyKroFt

[FriarTuck]

The same thing is happening to me. UnRAID used to be so solid... Ive tried most things and cant fix this. I wanted to leave this running unattended for a few months when I travel, but this will no longer be possible.

[amateur_wizard]

1 hour ago, FriarTuck said:

The same thing is happening to me. UnRAID used to be so solid... Ive tried most things and cant fix this. I wanted to leave this running unattended for a few months when I travel, but this will no longer be possible.

I would recommend if you haven't already. Install "CA Auto Update Applications" set it to search for updates for every container and plugin daily. Reinstall the MyServer/Unraid Connect plugin (doing a reboot inbetween).

LMK how that works for you.

[DrSpaldo]

Last month I updated from 6.9.2 (after staying on it for ages) to 6.11.5 then updated to 6.12.0 then 6.12.1.

 

Since this time I have had ongoing issues with massive spikes in CPU usage. Usually 100% and it cannot recover. I have updated plugins, read 100 forum threads, changed settings within Tips & Tweaks (vm.dirty_background_ratio to 3, vm.dirty_ratio to 6, changed the CPU scaling governor to performance), changed my MB & CPU, changed SATA card, removed a HDD, pinned all dockers to various CPU cores, removed unraid access to a few cores, among various other things suggested in these forums... Still no fix.

 

Today I ran the "/etc/rc.d/rc.nginx restart" command. It stopped the CPU from spiking, then maxed out my RAM?! Then the server froze after that and I had to hard reboot.

 

Generally these spikes occur about once per day. Diagnostics attached. 

 

Does anyone have any other ideas?

 

Example prior to CPU isolation:

 

 

Example after CPU isolation:

 

 

What HTOP looks like on one occasion:

 

 

Old CPU:

 

 

spaldounraid-diagnostics-20230701-1754.zip

Edited July 1 by DrSpaldo

[martial]

On 6.12.2 and similar error:

Jul 13 17:34:17 kasumi nginx: 2023/07/13 17:34:17 [crit] 35027#35027: ngx_slab_alloc() failed: no memory
Jul 13 17:34:17 kasumi nginx: 2023/07/13 17:34:17 [error] 35027#35027: shpool alloc failed
Jul 13 17:34:17 kasumi nginx: 2023/07/13 17:34:17 [error] 35027#35027: nchan: Out of shared memory while allocating message of size 233. Increase nchan_max_reserved_memory.
Jul 13 17:34:17 kasumi nginx: 2023/07/13 17:34:17 [error] 35027#35027: *5835100 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/wireguard?buffer_length=1 HTTP/1.1", host: "localhost"
Jul 13 17:34:17 kasumi nginx: 2023/07/13 17:34:17 [error] 35027#35027: MEMSTORE:01: can't create shared message for channel /wireguard

Restarted nginx but no luck, will restart the OS next.

I also tried to see if it was maybe Nginx Proxy Manager, but the /pub location makes me think I have a browser on another computer logged into the dashboard, causing this issue.

 

Could we get a list of remote connections to the dashboard and shut them down manually?

Edited July 13 by martial
typo fix

[FlyingTexan]

My advice is to get rid of NGINX and do a cloudflare zero trust tunnel.  Removes another potential point of failure and was very easy to setup. Made my life much easier. 

 

 

[Jimmy]

14 hours ago, FlyingTexan said:

My advice is to get rid of NGINX and do a cloudflare zero trust tunnel.  Removes another potential point of failure and was very easy to setup. Made my life much easier. 

 

 

 

I think we need to buy a domain name

[FlyingTexan]

2 hours ago, Jimmy said:

 

I think we need to buy a domain name

You’ll need a domain either way. Just pick something easy to remember and cheap then do it for 5+yrs

[martial]

On 7/13/2023 at 5:59 PM, FlyingTexan said:

My advice is to get rid of NGINX and do a cloudflare zero trust tunnel.  Removes another potential point of failure and was very easy to setup. Made my life much easier. 

 

 

@FlyingTexan Which of the "cloudflared" version in "apps" do you recommend?
I ended up using "CloudflaredTunnel" which uses the "cloudflare/cloudflared" container. Thanks for the recommendation.

Edited July 15 by martial
update content

[bumpkin]

Started happening to me a few days ago.  Suck.  Restarting the server sorts things out, but cannot otherwise find the cause, and it keeps coming back.  Filling the log repeating the following:

 

Aug  1 00:10:53 BigChief nginx: 2023/08/01 00:10:53 [crit] 7226#7226: ngx_slab_alloc() failed: no memory
Aug  1 00:10:53 BigChief nginx: 2023/08/01 00:10:53 [error] 7226#7226: shpool alloc failed
Aug  1 00:10:53 BigChief nginx: 2023/08/01 00:10:53 [error] 7226#7226: nchan: Out of shared memory while allocating message of size 15979. Increase nchan_max_reserved_memory.
Aug  1 00:10:53 BigChief nginx: 2023/08/01 00:10:53 [error] 7226#7226: *895289 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/disks?buffer_length=1 HTTP/1.1", host: "localhost"
Aug  1 00:10:53 BigChief nginx: 2023/08/01 00:10:53 [error] 7226#7226: MEMSTORE:00: can't create shared message for channel /disks

 

[meep]

Reporting this encountered for the first time today on 6.12.3

 

Manifests as empty pages on  the Array, Pool and Boot device tabs.

 

Log just filled with this;

 

Aug 8 13:45:08 UNRAID nginx: 2023/08/08 13:45:08 [error] 9990#9990: nchan: Out of shared memory while allocating message of size 19386. Increase nchan_max_reserved_memory.

Aug 8 13:45:08 UNRAID nginx: 2023/08/08 13:45:08 [error] 9990#9990: *2742737 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/disks?buffer_length=1 HTTP/1.1", host: "localhost"

Aug 8 13:45:08 UNRAID nginx: 2023/08/08 13:45:08 [error] 9990#9990: MEMSTORE:01: can't create shared message for channel /disks

Aug 8 13:45:08 UNRAID nginx: 2023/08/08 13:45:08 [crit] 9990#9990: ngx_slab_alloc() failed: no memory

Aug 8 13:45:08 UNRAID nginx: 2023/08/08 13:45:08 [error] 9990#9990: shpool alloc failed

 

[ziggy99]

Same issue here... log is filling up with:

 

Aug  8 21:16:07 MyServer nginx: 2023/08/09 06:16:07 [crit] 16233#16233: ngx_slab_alloc() failed: no memory
Aug  8 21:16:07 MyServer nginx: 2023/08/09 06:16:07 [error] 16233#16233: shpool alloc failed
Aug  8 21:16:07 MyServer nginx: 2023/08/09 06:16:07 [error] 16233#16233: nchan: Out of shared memory while allocating message of size 8894. Increase nchan_max_reserved_memory.
Aug  8 21:16:07 MyServer nginx: 2023/08/09 06:16:07 [error] 16233#16233: *1133944 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost"
Aug  8 21:16:07 MyServer nginx: 2023/08/09 06:16:07 [error] 16233#16233: MEMSTORE:01: can't create shared message for channel /devices

 

[SomeoneOnLine]

So frustrating with this darn error. it seems to be if I leave a browser session open on the main dashboard it causes this error but cant be for sure. I have to reboot like ever few days. super frustrating.

[dboonthego]

I'm seeing these errors for the first time (or noticing for first time) on two different servers running 6.12.3 and 6.12.4.  I haven't noticed any negative impact though, but I do have GUI tabs and browser terminal sessions open to both servers.  Is the thought that leaving the terminal session open is what causes this error?  Or a tab to the Unraid GUI?  Or both?

[FlyingTexan]

4 hours ago, dboonthego said:

I'm seeing these errors for the first time (or noticing for first time) on two different servers running 6.12.3 and 6.12.4.  I haven't noticed any negative impact though, but I do have GUI tabs and browser terminal sessions open to both servers.  Is the thought that leaving the terminal session open is what causes this error?  Or a tab to the Unraid GUI?  Or both?

try changing macvlan to ipvlan and do the above posted cloudflare zerotrust tunnel.

[Lonewolf147]

I just started seeing this error in my logs now too. I just upgraded to 6.12.4 yesterday. I usually keep a Firefox tab open with my Dashboard displayed, but for the past day I've actually had a tab open in Edge. When I went to look at it after it had sat for a few hours, all the animations on the dashboard (CPU/RAM/temps, etc) started going super fast, as if there were all fastforwarding to catch up with the current time. Then they all returned to normal speed. When I looked in the logs (checking for other issues) I found all these entries which coincide with the time that I was looking.

 

Sep 10 01:39:22 Valaskjalf nginx: 2023/09/10 01:39:22 [crit] 12634#12634: ngx_slab_alloc() failed: no memory
Sep 10 01:39:22 Valaskjalf nginx: 2023/09/10 01:39:22 [error] 12634#12634: shpool alloc failed
Sep 10 01:39:22 Valaskjalf nginx: 2023/09/10 01:39:22 [error] 12634#12634: nchan: Out of shared memory while allocating message of size 28129. Increase nchan_max_reserved_memory.
Sep 10 01:39:22 Valaskjalf nginx: 2023/09/10 01:39:22 [error] 12634#12634: *351766 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost"
Sep 10 01:39:22 Valaskjalf nginx: 2023/09/10 01:39:22 [error] 12634#12634: MEMSTORE:01: can't create shared message for channel /devices
Sep 10 01:39:22 Valaskjalf nginx: 2023/09/10 01:39:22 [crit] 12634#12634: ngx_slab_alloc() failed: no memory
Sep 10 01:39:22 Valaskjalf nginx: 2023/09/10 01:39:22 [error] 12634#12634: shpool alloc failed
Sep 10 01:39:22 Valaskjalf nginx: 2023/09/10 01:39:22 [error] 12634#12634: nchan: Out of shared memory while allocating message of size 16811. Increase nchan_max_reserved_memory.
Sep 10 01:39:22 Valaskjalf nginx: 2023/09/10 01:39:22 [error] 12634#12634: *351769 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/disks?buffer_length=1 HTTP/1.1", host: "localhost"
Sep 10 01:39:22 Valaskjalf nginx: 2023/09/10 01:39:22 [error] 12634#12634: MEMSTORE:01: can't create shared message for channel /disks
Sep 10 01:39:23 Valaskjalf nginx: 2023/09/10 01:39:23 [crit] 12634#12634: ngx_slab_alloc() failed: no memory
Sep 10 01:39:23 Valaskjalf nginx: 2023/09/10 01:39:23 [error] 12634#12634: shpool alloc failed
Sep 10 01:39:23 Valaskjalf nginx: 2023/09/10 01:39:23 [error] 12634#12634: nchan: Out of shared memory while allocating message of size 28129. Increase nchan_max_reserved_memory.
Sep 10 01:39:23 Valaskjalf nginx: 2023/09/10 01:39:23 [error] 12634#12634: *351776 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost"

 

valaskjalf-diagnostics-20230910-0222.zip

[martial]

I too have a tab always open in Firefox to my dashboard, but I have been seeing less of my logs filling lately (logs still at 14% after 16 days -- awaiting a few more days to see the list of issues on 6.12.4)
 

The two things I did were to use CloudFlare tunnels to reach my dashboard (Cloudflare Zero Trust with a 6 digits pin sent to a selected email to give me access to the dasahboard and selected services) vs accessing it from Nginx Proxy Manager (NPM), and I have increased the memory allocated to NPM in the extra parameters: 

--memory=4G

from 1G

 

Hopefully others can reproduce this

[maxistviews]

I’m on 6.12.3 and so far I haven’t had any issues with the log overfilling. I used to have a lot of issues on the old versions. Upgrading my LSI SATA drivers and updating unraid fixed these problems for me.

[wayner]

Me too.  I just saw these - my system was recently upgraded to 6.12.4.  I don't remember ever seeing these under 6.11.5, if I did they didn't cause issues as my system could go for months without having issues.  Not sure it is causing any major problems in 6.12.4 - at least not yet, other than filling the logs.

[semtex41]

Seeing this problem now. Similar to the last post, I was also fine running 6.12.3 but updated to 6.12.4 a few weeks ago. 

 

Ran this:

> /var/log/syslog
/etc/rc.d/rc.syslog stop
/etc/rc.d/rc.syslog start

It cleared the log file and the web console log is working. 

 

But I noticed that the Log utilization bar on the Dashboard isnt reflecting the new empty syslog. 

 

uptime:

06:54:26 up 6 days, 18:53,  1 user,  load average: 2.18, 2.11, 2.17