trurl Posted September 23, 2023 Share Posted September 23, 2023 There are things in log besides syslog. What do you get from command line with this? du -h -d 1 /var/log Quote Link to comment
semtex41 Posted September 23, 2023 Share Posted September 23, 2023 (edited) 3 hours ago, trurl said: There are things in log besides syslog. What do you get from command line with this? du -h -d 1 /var/log 0 /var/log/pwfail 0 /var/log/swtpm 880K /var/log/samba 0 /var/log/plugins 12K /var/log/pkgtools 60M /var/log/nginx 0 /var/log/nfsd 0 /var/log/libvirt 67M /var/log At the same time as the results above: du -sh /var/log/* 4.0K /var/log/apcupsd.events 0 /var/log/btmp 0 /var/log/cron 0 /var/log/debug 4.0K /var/log/dhcplog 72K /var/log/dmesg 52K /var/log/docker.log 0 /var/log/faillog 4.0K /var/log/lastlog 0 /var/log/libvirt 0 /var/log/maillog 0 /var/log/mcelog 0 /var/log/messages 0 /var/log/nfsd 60M /var/log/nginx 0 /var/log/packages 12K /var/log/pkgtools 0 /var/log/plugins 0 /var/log/pwfail 0 /var/log/removed_packages 0 /var/log/removed_scripts 0 /var/log/removed_uninstall_scripts 880K /var/log/samba 0 /var/log/scripts 0 /var/log/secure 0 /var/log/setup 0 /var/log/spooler 0 /var/log/swtpm 5.8M /var/log/syslog 0 /var/log/vfio-pci 0 /var/log/vfio-pci-errors 16K /var/log/wtmp And: Thanks for your help! Edited September 23, 2023 by semtex41 added current syslog size and dashboard screenshot Quote Link to comment
semtex41 Posted September 23, 2023 Share Posted September 23, 2023 I thought it might have been related to nginx proxy manager, but I disabled the host prior to clearing the logs previously. But there are new error entries related to nginx in unraid itself, and memory. Here are the recent logs. Sep 23 13:33:22 tower nginx: 2023/09/23 13:33:22 [crit] 22809#22809: ngx_slab_alloc() failed: no memory Sep 23 13:33:22 tower nginx: 2023/09/23 13:33:22 [error] 22809#22809: shpool alloc failed Sep 23 13:33:22 tower nginx: 2023/09/23 13:33:22 [error] 22809#22809: nchan: Out of shared memory while allocating message of size 18014. Increase nchan_max_reserved_memory. Sep 23 13:33:22 tower nginx: 2023/09/23 13:33:22 [error] 22809#22809: *5776466 nchan: error publishing message (HTTP status code 500), client: unix:, server: , request: "POST /pub/devices?buffer_length=1 HTTP/1.1", host: "localhost" Sep 23 13:33:22 tower nginx: 2023/09/23 13:33:22 [error] 22809#22809: MEMSTORE:01: can't create shared message for channel /devices Sep 23 13:41:17 tower emhttpd: spinning down /dev/sdi Sep 23 14:12:13 tower kernel: veth249d8c8: renamed from eth0 Sep 23 14:12:14 tower kernel: eth0: renamed from veth87b2b92 Sep 23 14:18:06 tower nginx: 2023/09/23 14:18:06 [error] 22809#22809: *5805991 "/usr/local/emhttp/api/index.html" is not found (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /api/ HTTP/1.1", host: "127.0.0.1" Sep 23 14:18:06 tower nginx: 2023/09/23 14:18:06 [error] 22809#22809: *5805998 open() "/usr/local/emhttp/status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /status?full&json HTTP/1.1", host: "localhost" Sep 23 14:18:06 tower nginx: 2023/09/23 14:18:06 [error] 22809#22809: *5805999 open() "/usr/local/emhttp/status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /status?full&json HTTP/1.1", host: "127.0.0.1" Sep 23 14:18:06 tower nginx: 2023/09/23 14:18:06 [error] 22809#22809: *5806000 open() "/usr/local/emhttp/server-status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /server-status?auto HTTP/1.1", host: "localhost" Sep 23 14:18:06 tower nginx: 2023/09/23 14:18:06 [error] 22809#22809: *5806001 open() "/usr/local/emhttp/server-status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /server-status?auto HTTP/1.1", host: "127.0.0.1" Sep 23 14:18:08 tower nginx: 2023/09/23 14:18:08 [error] 22809#22809: *5806027 open() "/usr/local/emhttp/server-status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /server-status?auto HTTP/1.1", host: "localhost" Sep 23 14:18:08 tower nginx: 2023/09/23 14:18:08 [error] 22809#22809: *5806028 open() "/usr/local/emhttp/server-status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /server-status?auto HTTP/1.1", host: "127.0.0.1" Sep 23 14:18:09 tower nginx: 2023/09/23 14:18:09 [error] 22809#22809: *5806035 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 127.0.0.1, server: , request: "GET /admin/api.php?auth=&version=true HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "127.0.0.1" Sep 23 14:18:09 tower nginx: 2023/09/23 14:18:09 [error] 22809#22809: *5806037 FastCGI sent in stderr: "Primary script unknown" while reading response header from upstream, client: 127.0.0.1, server: , request: "GET /admin/api.php?auth=&version=true HTTP/1.1", upstream: "fastcgi://unix:/var/run/php5-fpm.sock:", host: "localhost" Sep 23 14:18:09 tower nginx: 2023/09/23 14:18:09 [error] 22809#22809: *5806039 open() "/usr/local/emhttp/status/format/json" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /status/format/json HTTP/1.1", host: "127.0.0.1" Sep 23 14:18:09 tower nginx: 2023/09/23 14:18:09 [error] 22809#22809: *5806045 open() "/usr/local/emhttp/basic_status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /basic_status HTTP/1.1", host: "127.0.0.1" Sep 23 14:18:09 tower nginx: 2023/09/23 14:18:09 [error] 22809#22809: *5806046 open() "/usr/local/emhttp/stub_status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /stub_status HTTP/1.1", host: "localhost" Sep 23 14:18:09 tower nginx: 2023/09/23 14:18:09 [error] 22809#22809: *5806047 open() "/usr/local/emhttp/stub_status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /stub_status HTTP/1.1", host: "127.0.0.1" Sep 23 14:18:09 tower nginx: 2023/09/23 14:18:09 [error] 22809#22809: *5806048 open() "/usr/local/emhttp/nginx_status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /nginx_status HTTP/1.1", host: "127.0.0.1" Sep 23 14:18:09 tower nginx: 2023/09/23 14:18:09 [error] 22809#22809: *5806049 open() "/usr/local/emhttp/status" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /status HTTP/1.1", host: "127.0.0.1" Sep 23 14:18:09 tower nginx: 2023/09/23 14:18:09 [error] 22809#22809: *5806050 open() "/usr/local/emhttp/us" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /us HTTP/1.1", host: "localhost" Sep 23 14:18:09 tower nginx: 2023/09/23 14:18:09 [error] 22809#22809: *5806051 open() "/usr/local/emhttp/us" failed (2: No such file or directory), client: 127.0.0.1, server: , request: "GET /us HTTP/1.1", host: "127.0.0.1" Quote Link to comment
semtex41 Posted September 26, 2023 Share Posted September 26, 2023 So is this a bug or user error? Struggle bus here, trying to figure out if its something on my side. Quote Link to comment
wayner Posted September 27, 2023 Share Posted September 27, 2023 I removed the unraid Connect Plugin and that fixed this problem for me - none of these errors in 10 days. Quote Link to comment
semtex41 Posted September 27, 2023 Share Posted September 27, 2023 10 hours ago, wayner said: I removed the unraid Connect Plugin and that fixed this problem for me - none of these errors in 10 days. I dont see that plugin installed on my box. Good info though. Quote Link to comment
semtex41 Posted October 3, 2023 Share Posted October 3, 2023 In todays drama, the log hit 100%. What I should have done was cleared them. Instead I restarted. That was a mistake. After about 5 mins waiting for the reboot, I realize my tower was booting into UEFI and not recognizing the USB. After repairing the USB in windows, it did boot successfully but failed to start the docker service. In fact, the docker config seemed to have been factory reset, creating a blank docker.img. Luckily, I am now keeping the docker image in an appdata folder, and after stopping the service and changing it to the right path, got the containers going again. BTRFS scrub did not find any errors. I would like to note this flashdrive is only 2 weeks old and I am certain the fs errors on it are related to the full logs. This really needs to be fixed and I appreciate any help I can get fixing it in the short term. Quote Link to comment
semtex41 Posted October 6, 2023 Share Posted October 6, 2023 Continued from the last crash/reboot: root@tower:/var/log# ls -lah syslog* -rw-r--r-- 1 root root 1.9K Oct 5 22:12 syslog -rw-r--r-- 1 root root 12M Oct 5 00:00 syslog.1 -rw-r--r-- 1 root root 59M Oct 4 03:00 syslog.2 0 /var/log/pwfail 0 /var/log/swtpm 484K /var/log/samba 0 /var/log/plugins 12K /var/log/pkgtools 11M /var/log/nginx 0 /var/log/nfsd 0 /var/log/libvirt 82M /var/log root@tower:/var/log# grep -o 'Increase nchan_max_reserved_memory' /var/log/syslog.2 | wc -l 81995 root@tower:/var/log# awk -v phrase="Increase nchan_max_reserved_memory" '{count += gsub(phrase, "")} END {print count}' /var/log/syslog.2 81995 root@tower:/var/log# grep -o '"/usr/local/emhttp/us" failed (2: No such file or directory)' /var/log/syslog.2 | wc -l 2 Performed: > /var/log/syslog &\ > /var/log/syslog.1 &\ > /var/log/syslog.2 /etc/rc.d/rc.syslog stop /etc/rc.d/rc.syslog start and that zeroed out the logs. Quote Link to comment
martial Posted October 6, 2023 Share Posted October 6, 2023 (edited) 26 minutes ago, semtex41 said: root@tower:/var/log# grep -o 'Increase nchan_max_reserved_memory' /var/log/syslog.2 | wc -l 81995 root@tower:/var/log# awk -v phrase="Increase nchan_max_reserved_memory" '{count += gsub(phrase, "")} END {print count}' /var/log/syslog.2 81995 root@tower:/var/log# grep -o '"/usr/local/emhttp/us" failed (2: No such file or directory)' /var/log/syslog.2 | wc -l 2 Out of curiosity I checked mine and I see 70K of those nchan_max_... errors too I overwrote the file (> /var/log/syslog) and /etc/rc.d/rc.syslog restart and within a minute I already see another 100 of those errors And sadly that did not reduce the size of /var/log tmpfs 128M 47M 82M 37% /var/log also now the items on my dashboard are not updating anymore, so reboot time Edited October 6, 2023 by martial updated with additional information Quote Link to comment
semtex41 Posted October 6, 2023 Share Posted October 6, 2023 @martial try stopping and then starting the service, instead of restarting. That kept me from needing to reboot previously. Quote Link to comment
martial Posted October 6, 2023 Share Posted October 6, 2023 2 hours ago, semtex41 said: @martial try stopping and then starting the service, instead of restarting. That kept me from needing to reboot previously. if it happens again I will try that. One thing I have found in the past from searching the "nchan_max_reserved_memory" error is that it seems to happens if you leave a tab open onto the dashboard. If you close that tab (and reopen a new one later) people in other threads seem to indicate that helps Quote Link to comment
semtex41 Posted October 14, 2023 Share Posted October 14, 2023 This is still happening about once every 12 hours, but it alternates between the nginx logs being full and just the syslog being full. Just blindly nuking them at this point. I am using Edge and noticed it is getting faster since the last update. No evidence for that though, just an observation. Still willing to help get this resolved. Quote Link to comment
viviolay Posted October 14, 2023 Share Posted October 14, 2023 (edited) Well, at least I have good company. Same issue here. Logs hit 100%. Would any specific diagnostic files help? Stopped and restarted array just b/c. Edited October 14, 2023 by viviolay Quote Link to comment
martial Posted October 19, 2023 Share Posted October 19, 2023 Just a quick update. I now access the UI from a CloudFlare Tunnel, not a Nginx Proxy Manager proxy. I also close the tab to the dashboard after I am done using it. So far, the log has not shown any sign of growth. Quote Link to comment
dhenke1690 Posted October 23, 2023 Share Posted October 23, 2023 I am having this issue as well. I increased the LOG partition size but the error are still showing up. Has anyone increased the Nginx max reserved memory size to see if that helps? I haven't actually done this yet but I'm curious if it will help. I don't want to cause more issues. I do have 128GB of ram so I do mine giving some extra space to these to stop, or at the very least make these errors less common. Edit the Nginx Configuration: Open your Nginx configuration file using a text editor. The configuration file is typically located at /etc/nginx/nginx.conf or /etc/nginx/conf.d/your-config-file.conf. Add the directive nchan_max_reserved_memory 32M; to the appropriate location, often within the http block. Save the changes and close the editor. Quote Link to comment
semtex41 Posted October 23, 2023 Share Posted October 23, 2023 After another week, I have determined a few things: Closing all tabs prevents the errors from building up/cascading. The browser type doesnt seem to matter. Crashes/logs growing happens with Edge, Chrome, and Firefox. My appdata backup (which runs on Monday mornings) has been one of the triggers for the nginx errors in the logs. When the tab is open, the log fills up with the errors while the scheduled job is running. I do not blame the plugin, because when the tab (all tabs) are closed, the errors are not generated. Closing the tab today prevented a hard crash like last week, which required a hard shutdown. This is a webserver based interface. If the primary mechanism for accessing the OS causes the OS to consistently crash, then it is a bug. 2 3 Quote Link to comment
fmeres Posted October 23, 2023 Share Posted October 23, 2023 I've been having this issue on and off for a couple of months now. At first I thought that removing the Connect plugin as someone suggested earlier had solved the issue, but it did not. I now added nchan_max_reserved_memory 128M to my nginx conf and I'm hoping it will fix it. Having the tab open is a big part of my workflow so I'm hoping we can eventually get to the bottom of what's causing this. Quote Link to comment
Alintya Posted November 3, 2023 Share Posted November 3, 2023 Had this happen to me as well for the past year every 2 weeks or so and almost guaranteed whenever I go home with the unraid browser tab still active. Quote Link to comment
semtex41 Posted November 3, 2023 Share Posted November 3, 2023 The bug report for this issue is in this thread: 6.12.4 - Gets unresponsive (Have to Cut power & hard reboot) - Stable Releases - Unraid Luckily it has been reopened. Hopefully it will get some visibility. Quote Link to comment
martial Posted December 17, 2023 Share Posted December 17, 2023 (edited) On 10/5/2023 at 11:34 PM, semtex41 said: Performed: > /var/log/syslog &\ > /var/log/syslog.1 &\ > /var/log/syslog.2 /etc/rc.d/rc.syslog stop /etc/rc.d/rc.syslog start and that zeroed out the logs. Got it happening again, used it to clear the logs. The df confirms that worked Unfortunately the dashboard or main page are empty of content, so the next step is to restart nginx I tried the steps in pkill -9 nginx /etc/rc.d/rc.nginx start but the dashboard and main were still not updating with live data ... after a couple of minutes the data was showing up again the "/var/log/nginx/error.log" has many of those "nchan: Out of shared memory while allocating channel /cpuload. Increase nchan_max_reserved_memory" so use: pkill -9 nginx > /var/log/nginx/error.log & > /var/log/nginx/error.log.1 /etc/rc.d/rc.nginx start To get a more reasonably sized "df -h /var/log" I put all this in a small shell script (attached to this post), hopefully this can help others varlog_nginx.sh Edited December 17, 2023 by martial Quote Link to comment
Healzangels Posted January 19 Share Posted January 19 Wondering if you could re-share script seems to have been removed - Cheers! Quote Link to comment
martial Posted January 19 Share Posted January 19 (edited) 3 hours ago, Healzangels said: Wondering if you could re-share script seems to have been removed - Cheers! Not sure if this request was for the script I use? If it is, I just put a copy on GH, here is a direct link to it https://raw.githubusercontent.com/mmartial/unraid-study/main/varlog_nginx.sh Edited January 19 by martial Quote Link to comment
Healzangels Posted January 22 Share Posted January 22 Fantastic, exactly what I was asking. Thanks for sharing! Curious how often you found the script necessary to run? Every 12 hours via User-Scripts say? Thanks again. Quote Link to comment
martial Posted January 23 Share Posted January 23 1 hour ago, Healzangels said: Fantastic, exactly what I was asking. Thanks for sharing! Curious how often you found the script necessary to run? Every 12 hours via User-Scripts say? Thanks again. The error seems to only trigger for me when I leave a browser open on the Dashboard page (I have not seen it yet on the Docker page or VM page, for example), so as long as I close the browser tab, I do not get the empty dashboard/main. I can still ssh into it and start the script manually if/when it happens, so no user scripts are needed. I only use it when needed Quote Link to comment
Healzangels Posted January 24 Share Posted January 24 On 1/22/2024 at 7:27 PM, martial said: The error seems to only trigger for me when I leave a browser open on the Dashboard page (I have not seen it yet on the Docker page or VM page, for example), so as long as I close the browser tab, I do not get the empty dashboard/main. I can still ssh into it and start the script manually if/when it happens, so no user scripts are needed. I only use it when needed Makes sense, thanks for the additional insight. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.