September 23, 20232 yr There are things in log besides syslog. What do you get from command line with this? du -h -d 1 /var/log
September 23, 20232 yr 3 hours ago, trurl said: There are things in log besides syslog. What do you get from command line with this? du -h -d 1 /var/log 0 /var/log/pwfail 0 /var/log/swtpm 880K /var/log/samba 0 /var/log/plugins 12K /var/log/pkgtools 60M /var/log/nginx 0 /var/log/nfsd 0 /var/log/libvirt 67M /var/log At the same time as the results above: du -sh /var/log/* 4.0K /var/log/apcupsd.events 0 /var/log/btmp 0 /var/log/cron 0 /var/log/debug 4.0K /var/log/dhcplog 72K /var/log/dmesg 52K /var/log/docker.log 0 /var/log/faillog 4.0K /var/log/lastlog 0 /var/log/libvirt 0 /var/log/maillog 0 /var/log/mcelog 0 /var/log/messages 0 /var/log/nfsd 60M /var/log/nginx 0 /var/log/packages 12K /var/log/pkgtools 0 /var/log/plugins 0 /var/log/pwfail 0 /var/log/removed_packages 0 /var/log/removed_scripts 0 /var/log/removed_uninstall_scripts 880K /var/log/samba 0 /var/log/scripts 0 /var/log/secure 0 /var/log/setup 0 /var/log/spooler 0 /var/log/swtpm 5.8M /var/log/syslog 0 /var/log/vfio-pci 0 /var/log/vfio-pci-errors 16K /var/log/wtmp And: Thanks for your help! Edited September 23, 20232 yr by semtex41 added current syslog size and dashboard screenshot
September 23, 20232 yr I thought it might have been related to nginx proxy manager, but I disabled the host prior to clearing the logs previously. But there are new error entries related to nginx in unraid itself, and memory. Here are the recent logs. Sep 23 13:33:22 tower nginx: 2023/09/23 13:33:22 [crit] 22809#22809: ngx_slab_alloc() failed: no memory Sep 23 13:33:22 tower nginx: 2023/09/23 13:33:22 [error] 22809#22809: shpool alloc failed Sep 23 13:33:22 tower nginx: 2023/09/23 13:33:22 [error] 22809#22809: nchan: Out of shared memory while allocating message of size 18014. Increase nchan_max_reserved_memory. Sep 23 13:33:22 tower nginx: 2023/09/23 13:33:22 [error] 22809#22809: *5776466 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost" Sep 23 13:33:22 tower nginx: 2023/09/23 13:33:22 [error] 22809#22809: MEMSTORE:01: can't create shared message for channel /devices Sep 23 13:41:17 tower emhttpd: spinning down /dev/sdi Sep 23 14:12:13 tower kernel: veth249d8c8: renamed from eth0 Sep 23 14:12:14 tower kernel: eth0: renamed from veth87b2b92 Sep 23 14:18:06 tower nginx: 2023/09/23 14:18:06 [error] 22809#22809: *5805991 "/usr/local/emhttp/api/index.html" is not found (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /api/ HTTP/1.1", host: "127.0.0.1" Sep 23 14:18:06 tower nginx: 2023/09/23 14:18:06 [error] 22809#22809: *5805998 open() "/usr/local/emhttp/status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /status?full&json HTTP/1.1", host: "localhost" Sep 23 14:18:06 tower nginx: 2023/09/23 14:18:06 [error] 22809#22809: *5805999 open() "/usr/local/emhttp/status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /status?full&json HTTP/1.1", host: "127.0.0.1" Sep 23 14:18:06 tower nginx: 2023/09/23 14:18:06 [error] 22809#22809: *5806000 open() "/usr/local/emhttp/server-status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /server-status?auto HTTP/1.1", host: "localhost" Sep 23 14:18:06 tower nginx: 2023/09/23 14:18:06 [error] 22809#22809: *5806001 open() "/usr/local/emhttp/server-status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /server-status?auto HTTP/1.1", host: "127.0.0.1" Sep 23 14:18:08 tower nginx: 2023/09/23 14:18:08 [error] 22809#22809: *5806027 open() "/usr/local/emhttp/server-status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /server-status?auto HTTP/1.1", host: "localhost" Sep 23 14:18:08 tower nginx: 2023/09/23 14:18:08 [error] 22809#22809: *5806028 open() "/usr/local/emhttp/server-status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /server-status?auto HTTP/1.1", host: "127.0.0.1" Sep 23 14:18:09 tower nginx: 2023/09/23 14:18:09 [error] 22809#22809: *5806035 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 127.0.0.1, server: , request: "GET /admin/api.php?auth=&version=true HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "127.0.0.1" Sep 23 14:18:09 tower nginx: 2023/09/23 14:18:09 [error] 22809#22809: *5806037 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 127.0.0.1, server: , request: "GET /admin/api.php?auth=&version=true HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "localhost" Sep 23 14:18:09 tower nginx: 2023/09/23 14:18:09 [error] 22809#22809: *5806039 open() "/usr/local/emhttp/status/format/json" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /status/format/json HTTP/1.1", host: "127.0.0.1" Sep 23 14:18:09 tower nginx: 2023/09/23 14:18:09 [error] 22809#22809: *5806045 open() "/usr/local/emhttp/basic_status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /basic_status HTTP/1.1", host: "127.0.0.1" Sep 23 14:18:09 tower nginx: 2023/09/23 14:18:09 [error] 22809#22809: *5806046 open() "/usr/local/emhttp/stub_status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /stub_status HTTP/1.1", host: "localhost" Sep 23 14:18:09 tower nginx: 2023/09/23 14:18:09 [error] 22809#22809: *5806047 open() "/usr/local/emhttp/stub_status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /stub_status HTTP/1.1", host: "127.0.0.1" Sep 23 14:18:09 tower nginx: 2023/09/23 14:18:09 [error] 22809#22809: *5806048 open() "/usr/local/emhttp/nginx_status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /nginx_status HTTP/1.1", host: "127.0.0.1" Sep 23 14:18:09 tower nginx: 2023/09/23 14:18:09 [error] 22809#22809: *5806049 open() "/usr/local/emhttp/status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /status HTTP/1.1", host: "127.0.0.1" Sep 23 14:18:09 tower nginx: 2023/09/23 14:18:09 [error] 22809#22809: *5806050 open() "/usr/local/emhttp/us" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /us HTTP/1.1", host: "localhost" Sep 23 14:18:09 tower nginx: 2023/09/23 14:18:09 [error] 22809#22809: *5806051 open() "/usr/local/emhttp/us" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /us HTTP/1.1", host: "127.0.0.1"
September 26, 20232 yr So is this a bug or user error? Struggle bus here, trying to figure out if its something on my side.
September 27, 20232 yr I removed the unraid Connect Plugin and that fixed this problem for me - none of these errors in 10 days.
September 27, 20232 yr 10 hours ago, wayner said: I removed the unraid Connect Plugin and that fixed this problem for me - none of these errors in 10 days. I dont see that plugin installed on my box. Good info though.
October 3, 20232 yr In todays drama, the log hit 100%. What I should have done was cleared them. Instead I restarted. That was a mistake. After about 5 mins waiting for the reboot, I realize my tower was booting into UEFI and not recognizing the USB. After repairing the USB in windows, it did boot successfully but failed to start the docker service. In fact, the docker config seemed to have been factory reset, creating a blank docker.img. Luckily, I am now keeping the docker image in an appdata folder, and after stopping the service and changing it to the right path, got the containers going again. BTRFS scrub did not find any errors. I would like to note this flashdrive is only 2 weeks old and I am certain the fs errors on it are related to the full logs. This really needs to be fixed and I appreciate any help I can get fixing it in the short term.
October 6, 20232 yr Continued from the last crash/reboot: root@tower:/var/log# ls -lah syslog* -rw-r--r-- 1 root root 1.9K Oct 5 22:12 syslog -rw-r--r-- 1 root root 12M Oct 5 00:00 syslog.1 -rw-r--r-- 1 root root 59M Oct 4 03:00 syslog.2 0 /var/log/pwfail 0 /var/log/swtpm 484K /var/log/samba 0 /var/log/plugins 12K /var/log/pkgtools 11M /var/log/nginx 0 /var/log/nfsd 0 /var/log/libvirt 82M /var/log root@tower:/var/log# grep -o 'Increase nchan_max_reserved_memory' /var/log/syslog.2 | wc -l 81995 root@tower:/var/log# awk -v phrase="Increase nchan_max_reserved_memory" '{count += gsub(phrase, "")} END {print count}' /var/log/syslog.2 81995 root@tower:/var/log# grep -o '"/usr/local/emhttp/us" failed (2: No such file or directory)' /var/log/syslog.2 | wc -l 2 Performed: > /var/log/syslog &\ > /var/log/syslog.1 &\ > /var/log/syslog.2 /etc/rc.d/rc.syslog stop /etc/rc.d/rc.syslog start and that zeroed out the logs.
October 6, 20232 yr 26 minutes ago, semtex41 said: root@tower:/var/log# grep -o 'Increase nchan_max_reserved_memory' /var/log/syslog.2 | wc -l 81995 root@tower:/var/log# awk -v phrase="Increase nchan_max_reserved_memory" '{count += gsub(phrase, "")} END {print count}' /var/log/syslog.2 81995 root@tower:/var/log# grep -o '"/usr/local/emhttp/us" failed (2: No such file or directory)' /var/log/syslog.2 | wc -l 2 Out of curiosity I checked mine and I see 70K of those nchan_max_... errors too I overwrote the file (> /var/log/syslog) and /etc/rc.d/rc.syslog restart and within a minute I already see another 100 of those errors And sadly that did not reduce the size of /var/log tmpfs 128M 47M 82M 37% /var/log also now the items on my dashboard are not updating anymore, so reboot time Edited October 6, 20232 yr by martial updated with additional information
October 6, 20232 yr @martial try stopping and then starting the service, instead of restarting. That kept me from needing to reboot previously.
October 6, 20232 yr 2 hours ago, semtex41 said: @martial try stopping and then starting the service, instead of restarting. That kept me from needing to reboot previously. if it happens again I will try that. One thing I have found in the past from searching the "nchan_max_reserved_memory" error is that it seems to happens if you leave a tab open onto the dashboard. If you close that tab (and reopen a new one later) people in other threads seem to indicate that helps
October 14, 20232 yr This is still happening about once every 12 hours, but it alternates between the nginx logs being full and just the syslog being full. Just blindly nuking them at this point. I am using Edge and noticed it is getting faster since the last update. No evidence for that though, just an observation. Still willing to help get this resolved.
October 14, 20232 yr Well, at least I have good company. Same issue here. Logs hit 100%. Would any specific diagnostic files help? Stopped and restarted array just b/c. Edited October 14, 20232 yr by viviolay
October 19, 20232 yr Just a quick update. I now access the UI from a CloudFlare Tunnel, not a Nginx Proxy Manager proxy. I also close the tab to the dashboard after I am done using it. So far, the log has not shown any sign of growth.
October 23, 20232 yr I am having this issue as well. I increased the LOG partition size but the error are still showing up. Has anyone increased the Nginx max reserved memory size to see if that helps? I haven't actually done this yet but I'm curious if it will help. I don't want to cause more issues. I do have 128GB of ram so I do mine giving some extra space to these to stop, or at the very least make these errors less common. Edit the Nginx Configuration: Open your Nginx configuration file using a text editor. The configuration file is typically located at /etc/nginx/nginx.conf or /etc/nginx/conf.d/your-config-file.conf. Add the directive nchan_max_reserved_memory 32M; to the appropriate location, often within the http block. Save the changes and close the editor.
October 23, 20232 yr After another week, I have determined a few things: Closing all tabs prevents the errors from building up/cascading. The browser type doesnt seem to matter. Crashes/logs growing happens with Edge, Chrome, and Firefox. My appdata backup (which runs on Monday mornings) has been one of the triggers for the nginx errors in the logs. When the tab is open, the log fills up with the errors while the scheduled job is running. I do not blame the plugin, because when the tab (all tabs) are closed, the errors are not generated. Closing the tab today prevented a hard crash like last week, which required a hard shutdown. This is a webserver based interface. If the primary mechanism for accessing the OS causes the OS to consistently crash, then it is a bug.
October 23, 20232 yr I've been having this issue on and off for a couple of months now. At first I thought that removing the Connect plugin as someone suggested earlier had solved the issue, but it did not. I now added nchan_max_reserved_memory 128M to my nginx conf and I'm hoping it will fix it. Having the tab open is a big part of my workflow so I'm hoping we can eventually get to the bottom of what's causing this.
November 3, 20232 yr Had this happen to me as well for the past year every 2 weeks or so and almost guaranteed whenever I go home with the unraid browser tab still active.
November 3, 20232 yr The bug report for this issue is in this thread: 6.12.4 - Gets unresponsive (Have to Cut power & hard reboot) - Stable Releases - Unraid Luckily it has been reopened. Hopefully it will get some visibility.
December 17, 20232 yr On 10/5/2023 at 11:34 PM, semtex41 said: Performed: > /var/log/syslog &\ > /var/log/syslog.1 &\ > /var/log/syslog.2 /etc/rc.d/rc.syslog stop /etc/rc.d/rc.syslog start and that zeroed out the logs. Got it happening again, used it to clear the logs. The df confirms that worked Unfortunately the dashboard or main page are empty of content, so the next step is to restart nginx I tried the steps in pkill -9 nginx /etc/rc.d/rc.nginx start but the dashboard and main were still not updating with live data ... after a couple of minutes the data was showing up again the "/var/log/nginx/error.log" has many of those "nchan: Out of shared memory while allocating channel /cpuload. Increase nchan_max_reserved_memory" so use: pkill -9 nginx > /var/log/nginx/error.log & > /var/log/nginx/error.log.1 /etc/rc.d/rc.nginx start To get a more reasonably sized "df -h /var/log" I put all this in a small shell script (attached to this post), hopefully this can help others varlog_nginx.sh Edited December 17, 20232 yr by martial
January 19, 20242 yr 3 hours ago, Healzangels said: Wondering if you could re-share script seems to have been removed - Cheers! Not sure if this request was for the script I use? If it is, I just put a copy on GH, here is a direct link to it https://raw.githubusercontent.com/mmartial/unraid-study/main/varlog_nginx.sh Edited January 19, 20242 yr by martial
January 22, 20242 yr Fantastic, exactly what I was asking. Thanks for sharing! Curious how often you found the script necessary to run? Every 12 hours via User-Scripts say? Thanks again.
January 23, 20242 yr 1 hour ago, Healzangels said: Fantastic, exactly what I was asking. Thanks for sharing! Curious how often you found the script necessary to run? Every 12 hours via User-Scripts say? Thanks again. The error seems to only trigger for me when I leave a browser open on the Dashboard page (I have not seen it yet on the Docker page or VM page, for example), so as long as I close the browser tab, I do not get the empty dashboard/main. I can still ssh into it and start the script manually if/when it happens, so no user scripts are needed. I only use it when needed
January 24, 20242 yr On 1/22/2024 at 7:27 PM, martial said: The error seems to only trigger for me when I leave a browser open on the Dashboard page (I have not seen it yet on the Docker page or VM page, for example), so as long as I close the browser tab, I do not get the empty dashboard/main. I can still ssh into it and start the script manually if/when it happens, so no user scripts are needed. I only use it when needed Makes sense, thanks for the additional insight.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.