nginx running out of shared memory

[trurl]

There are things in log besides syslog.

 

What do you get from command line with this?

du -h -d 1 /var/log

 

[semtex41]

3 hours ago, trurl said:

There are things in log besides syslog.

 

What do you get from command line with this?

du -h -d 1 /var/log

 

0       /var/log/pwfail
0       /var/log/swtpm
880K    /var/log/samba
0       /var/log/plugins
12K     /var/log/pkgtools
60M     /var/log/nginx
0       /var/log/nfsd
0       /var/log/libvirt
67M     /var/log

 

 

At the same time as the results above:
 

 du -sh /var/log/*
4.0K    /var/log/apcupsd.events
0       /var/log/btmp
0       /var/log/cron
0       /var/log/debug
4.0K    /var/log/dhcplog
72K     /var/log/dmesg
52K     /var/log/docker.log
0       /var/log/faillog
4.0K    /var/log/lastlog
0       /var/log/libvirt
0       /var/log/maillog
0       /var/log/mcelog
0       /var/log/messages
0       /var/log/nfsd
60M     /var/log/nginx
0       /var/log/packages
12K     /var/log/pkgtools
0       /var/log/plugins
0       /var/log/pwfail
0       /var/log/removed_packages
0       /var/log/removed_scripts
0       /var/log/removed_uninstall_scripts
880K    /var/log/samba
0       /var/log/scripts
0       /var/log/secure
0       /var/log/setup
0       /var/log/spooler
0       /var/log/swtpm
5.8M    /var/log/syslog
0       /var/log/vfio-pci
0       /var/log/vfio-pci-errors
16K     /var/log/wtmp

 

And:

 

Thanks for your help!

Edited September 23, 2023 by semtex41
added current syslog size and dashboard screenshot

[semtex41]

I thought it might have been related to nginx proxy manager, but I disabled the host prior to clearing the logs previously.
But there are new error entries related to nginx in unraid itself, and memory.
Here are the recent logs. 

Sep 23 13:33:22 tower nginx: 2023/09/23 13:33:22 [crit] 22809#22809: ngx_slab_alloc() failed: no memory
Sep 23 13:33:22 tower nginx: 2023/09/23 13:33:22 [error] 22809#22809: shpool alloc failed
Sep 23 13:33:22 tower nginx: 2023/09/23 13:33:22 [error] 22809#22809: nchan: Out of shared memory while allocating message of size 18014. Increase nchan_max_reserved_memory.
Sep 23 13:33:22 tower nginx: 2023/09/23 13:33:22 [error] 22809#22809: *5776466 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost"
Sep 23 13:33:22 tower nginx: 2023/09/23 13:33:22 [error] 22809#22809: MEMSTORE:01: can't create shared message for channel /devices
Sep 23 13:41:17 tower emhttpd: spinning down /dev/sdi
Sep 23 14:12:13 tower kernel: veth249d8c8: renamed from eth0
Sep 23 14:12:14 tower kernel: eth0: renamed from veth87b2b92
Sep 23 14:18:06 tower nginx: 2023/09/23 14:18:06 [error] 22809#22809: *5805991 "/usr/local/emhttp/api/index.html" is not found (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /api/ HTTP/1.1", host: "127.0.0.1"
Sep 23 14:18:06 tower nginx: 2023/09/23 14:18:06 [error] 22809#22809: *5805998 open() "/usr/local/emhttp/status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /status?full&json HTTP/1.1", host: "localhost"
Sep 23 14:18:06 tower nginx: 2023/09/23 14:18:06 [error] 22809#22809: *5805999 open() "/usr/local/emhttp/status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /status?full&json HTTP/1.1", host: "127.0.0.1"
Sep 23 14:18:06 tower nginx: 2023/09/23 14:18:06 [error] 22809#22809: *5806000 open() "/usr/local/emhttp/server-status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /server-status?auto HTTP/1.1", host: "localhost"
Sep 23 14:18:06 tower nginx: 2023/09/23 14:18:06 [error] 22809#22809: *5806001 open() "/usr/local/emhttp/server-status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /server-status?auto HTTP/1.1", host: "127.0.0.1"
Sep 23 14:18:08 tower nginx: 2023/09/23 14:18:08 [error] 22809#22809: *5806027 open() "/usr/local/emhttp/server-status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /server-status?auto HTTP/1.1", host: "localhost"
Sep 23 14:18:08 tower nginx: 2023/09/23 14:18:08 [error] 22809#22809: *5806028 open() "/usr/local/emhttp/server-status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /server-status?auto HTTP/1.1", host: "127.0.0.1"
Sep 23 14:18:09 tower nginx: 2023/09/23 14:18:09 [error] 22809#22809: *5806035 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 127.0.0.1, server: , request: "GET /admin/api.php?auth=&version=true HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "127.0.0.1"
Sep 23 14:18:09 tower nginx: 2023/09/23 14:18:09 [error] 22809#22809: *5806037 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 127.0.0.1, server: , request: "GET /admin/api.php?auth=&version=true HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "localhost"
Sep 23 14:18:09 tower nginx: 2023/09/23 14:18:09 [error] 22809#22809: *5806039 open() "/usr/local/emhttp/status/format/json" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /status/format/json HTTP/1.1", host: "127.0.0.1"
Sep 23 14:18:09 tower nginx: 2023/09/23 14:18:09 [error] 22809#22809: *5806045 open() "/usr/local/emhttp/basic_status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /basic_status HTTP/1.1", host: "127.0.0.1"
Sep 23 14:18:09 tower nginx: 2023/09/23 14:18:09 [error] 22809#22809: *5806046 open() "/usr/local/emhttp/stub_status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /stub_status HTTP/1.1", host: "localhost"
Sep 23 14:18:09 tower nginx: 2023/09/23 14:18:09 [error] 22809#22809: *5806047 open() "/usr/local/emhttp/stub_status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /stub_status HTTP/1.1", host: "127.0.0.1"
Sep 23 14:18:09 tower nginx: 2023/09/23 14:18:09 [error] 22809#22809: *5806048 open() "/usr/local/emhttp/nginx_status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /nginx_status HTTP/1.1", host: "127.0.0.1"
Sep 23 14:18:09 tower nginx: 2023/09/23 14:18:09 [error] 22809#22809: *5806049 open() "/usr/local/emhttp/status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /status HTTP/1.1", host: "127.0.0.1"
Sep 23 14:18:09 tower nginx: 2023/09/23 14:18:09 [error] 22809#22809: *5806050 open() "/usr/local/emhttp/us" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /us HTTP/1.1", host: "localhost"
Sep 23 14:18:09 tower nginx: 2023/09/23 14:18:09 [error] 22809#22809: *5806051 open() "/usr/local/emhttp/us" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /us HTTP/1.1", host: "127.0.0.1"

 

[semtex41]

So is this a bug or user error?
Struggle bus here, trying to figure out if its something on my side. 

[wayner]

I removed the unraid Connect Plugin and that fixed this problem for me - none of these errors in 10 days.

[semtex41]

10 hours ago, wayner said:

I removed the unraid Connect Plugin and that fixed this problem for me - none of these errors in 10 days.

I dont see that plugin installed on my box. Good info though. 

[semtex41]

In todays drama, the log hit 100%. What I should have done was cleared them. Instead I restarted. 

 

That was a mistake. After about 5 mins waiting for the reboot, I realize my tower was booting into UEFI and not recognizing the USB. 
After repairing the  USB in windows, it did boot successfully but failed to start the docker service. 

In fact, the docker config seemed to have been factory reset, creating a blank docker.img. 

 

Luckily, I am now keeping the docker image in an appdata folder, and after stopping the service and changing it to the right path, got the containers going again. BTRFS scrub did not find any errors. 

 

I would like to note this flashdrive is only 2 weeks old and I am certain the fs errors on it are related to the full logs. 

 

This really needs to be fixed and I appreciate any help I can get fixing it in the short term. 

[semtex41]

Continued from the last crash/reboot:

 

root@tower:/var/log# ls -lah syslog*
-rw-r--r-- 1 root root 1.9K Oct  5 22:12 syslog
-rw-r--r-- 1 root root  12M Oct  5 00:00 syslog.1
-rw-r--r-- 1 root root  59M Oct  4 03:00 syslog.2

 

0       /var/log/pwfail
0       /var/log/swtpm
484K    /var/log/samba
0       /var/log/plugins
12K     /var/log/pkgtools
11M     /var/log/nginx
0       /var/log/nfsd
0       /var/log/libvirt
82M     /var/log

 

root@tower:/var/log# grep -o 'Increase nchan_max_reserved_memory' /var/log/syslog.2 | wc -l
81995
root@tower:/var/log# awk -v phrase="Increase nchan_max_reserved_memory" '{count += gsub(phrase, "")} END {print count}' /var/log/syslog.2
81995
root@tower:/var/log# grep -o '"/usr/local/emhttp/us" failed (2: No such file or directory)' /var/log/syslog.2 | wc -l
2

 

Performed:

> /var/log/syslog   &\
> /var/log/syslog.1 &\
> /var/log/syslog.2
/etc/rc.d/rc.syslog stop
/etc/rc.d/rc.syslog start

and that zeroed out the logs. 

[martial]

26 minutes ago, semtex41 said:

 

root@tower:/var/log# grep -o 'Increase nchan_max_reserved_memory' /var/log/syslog.2 | wc -l
81995
root@tower:/var/log# awk -v phrase="Increase nchan_max_reserved_memory" '{count += gsub(phrase, "")} END {print count}' /var/log/syslog.2
81995
root@tower:/var/log# grep -o '"/usr/local/emhttp/us" failed (2: No such file or directory)' /var/log/syslog.2 | wc -l
2

Out of curiosity I checked mine and I see 70K of those nchan_max_... errors too

 

I overwrote the file (> /var/log/syslog) and /etc/rc.d/rc.syslog restart

and within a minute I already see another 100 of those errors

 

And sadly that did not reduce the size of /var/log

tmpfs                   128M   47M   82M  37% /var/log

 

also now the items on my dashboard are not updating anymore, so reboot time

 

Edited October 6, 2023 by martial
updated with additional information

[semtex41]

@martial try stopping and then starting the service, instead of restarting. That kept me from needing to reboot previously. 

[martial]

2 hours ago, semtex41 said:

@martial try stopping and then starting the service, instead of restarting. That kept me from needing to reboot previously. 

if it happens again I will try that.

One thing I have found in the past from searching the "nchan_max_reserved_memory" error is that it seems to happens if you leave a tab open onto the dashboard. If you close that tab (and reopen a new one later) people in other threads seem to indicate that helps

[semtex41]

This is still happening about once every 12 hours, but it alternates between the nginx logs being full and just the syslog being full. Just blindly nuking them at this point. 

 

I am using Edge and noticed it is getting faster since the last update. No evidence for that though, just an observation. 

 

Still willing to help get this resolved. 

[viviolay]

Well, at least I have good company. Same issue here. Logs hit 100%. Would any specific diagnostic files help?

Stopped and restarted array just b/c.

Edited October 14, 2023 by viviolay

[martial]

Just a quick update.

I now access the UI from a CloudFlare Tunnel, not a Nginx Proxy Manager proxy.

I also close the tab to the dashboard after I am done using it.

So far, the log has not shown any sign of growth.

[dhenke1690]

I am having this issue as well. I increased the LOG partition size but the error are still showing up. Has anyone increased the Nginx max reserved memory size to see if that helps? I haven't actually done this yet but I'm curious if it will help. I don't want to cause more issues. I do have 128GB of ram so I do mine giving some extra space to these to stop, or at the very least make these errors less common. 

 

Edit the Nginx Configuration:

Open your Nginx configuration file using a text editor. The configuration file is typically located at /etc/nginx/nginx.conf or /etc/nginx/conf.d/your-config-file.conf.

Add the directive nchan_max_reserved_memory 32M; to the appropriate location, often within the http block.

Save the changes and close the editor.

[semtex41]

After another week, I have determined a few things:

 

• Closing all tabs prevents the errors from building up/cascading. 
• The browser type doesnt seem to matter. Crashes/logs growing happens with Edge, Chrome, and Firefox. 
• My appdata backup (which runs on Monday mornings) has been one of the triggers for the nginx errors in the logs. When the tab is open, the log fills up with the errors while the scheduled job is running. I do not blame the plugin, because when the tab (all tabs) are closed, the errors are not generated. Closing the tab today prevented a hard crash like last week, which required a hard shutdown. 
• This is a webserver based interface. If the primary mechanism for accessing the OS causes the OS to consistently crash, then it is a bug. 

[fmeres]

I've been having this issue on and off for a couple of months now. At first I thought that removing the Connect plugin as someone suggested earlier had solved the issue, but it did not. I now added nchan_max_reserved_memory 128M to my nginx conf and I'm hoping it will fix it. Having the tab open is a big part of my workflow so I'm hoping we can eventually get to the bottom of what's causing this.

[Alintya]

Had this happen to me as well for the past year every 2 weeks or so and almost guaranteed whenever I go home with the unraid browser tab still active.

[semtex41]

The bug report for this issue is in this thread:
6.12.4 - Gets unresponsive (Have to Cut power & hard reboot) - Stable Releases - Unraid

 

Luckily it has been reopened. Hopefully it will get some visibility. 

 

 

[martial]

On 10/5/2023 at 11:34 PM, semtex41 said:

Performed:

> /var/log/syslog   &\
> /var/log/syslog.1 &\
> /var/log/syslog.2
/etc/rc.d/rc.syslog stop
/etc/rc.d/rc.syslog start

and that zeroed out the logs. 

 

Got it happening again, used it to clear the logs. The df confirms that worked

 

Unfortunately the dashboard or main page are empty of content, so the next step is to restart nginx

I tried the steps in

 pkill -9 nginx

/etc/rc.d/rc.nginx start

but the dashboard and main were still not updating with live data

... after a couple of minutes the data was showing up again
 

the "/var/log/nginx/error.log" has many of those "nchan: Out of shared memory while allocating channel /cpuload. Increase nchan_max_reserved_memory" so use:
 

pkill -9 nginx

> /var/log/nginx/error.log &

> /var/log/nginx/error.log.1

/etc/rc.d/rc.nginx start

 

To get a more reasonably sized "df -h /var/log"

I put all this in a small shell script (attached to this post), hopefully this can help others

varlog_nginx.sh

Edited December 17, 2023 by martial

[Healzangels]

Wondering if you could re-share script seems to have been removed - Cheers!

[martial]

3 hours ago, Healzangels said:

Wondering if you could re-share script seems to have been removed - Cheers!

Not sure if this request was for the script I use?

If it is, I just put a copy on GH, here is a direct link to it https://raw.githubusercontent.com/mmartial/unraid-study/main/varlog_nginx.sh

Edited January 19 by martial

[Healzangels]

Fantastic, exactly what I was asking. Thanks for sharing!

Curious how often you found the script necessary to run? Every 12 hours via User-Scripts say? Thanks again.

[martial]

1 hour ago, Healzangels said:

Fantastic, exactly what I was asking. Thanks for sharing!

Curious how often you found the script necessary to run? Every 12 hours via User-Scripts say? Thanks again.

 

The error seems to only trigger for me when I leave a browser open on the Dashboard page (I have not seen it yet on the Docker page or VM page, for example), so as long as I close the browser tab, I do not get the empty dashboard/main. I can still ssh into it and start the script manually if/when it happens, so no user scripts are needed. I only use it when needed

[Healzangels]

On 1/22/2024 at 7:27 PM, martial said:

 

The error seems to only trigger for me when I leave a browser open on the Dashboard page (I have not seen it yet on the Docker page or VM page, for example), so as long as I close the browser tab, I do not get the empty dashboard/main. I can still ssh into it and start the script manually if/when it happens, so no user scripts are needed. I only use it when needed

Makes sense, thanks for the additional insight.