Cpt. Chaz Posted March 9, 2021 Share Posted March 9, 2021 i'm seeing the same thing on 6.9.0-rc2. running /etc/rc.d/rc.nginx restart only seemed to work after then closing all open browser windows to server gui. also guilty of using chrome, knowing that sometimes chrome and unraid don't play nice. attached logs for anyone digging in on the issue. kal-el-diagnostics-20210309-1053.zip Quote Link to comment
John_M Posted March 9, 2021 Share Posted March 9, 2021 20 minutes ago, Cpt. Chaz said: i'm seeing the same thing on 6.9.0-rc2. Still using -rc2? 6.9.1 is the current version. Quote Link to comment
Cpt. Chaz Posted March 9, 2021 Share Posted March 9, 2021 Just now, John_M said: Still using -rc2? 6.9.1 is the current version. yeah. until now, updating hasn't been a high priority because i wasn't having any issues. it takes my r720 forever and a day to reboot, so i'm usually slow to update. will be updating today. Quote Link to comment
Rubene Posted March 11, 2021 Share Posted March 11, 2021 Same issue here suddenly since yesterday. I'm on 6.8.3 since it got released, never seen this issue before. Curious what triggered this. /etc/rc.d/rc.nginx restart seems like to fix it for now. Quote Link to comment
jbuszkie Posted March 11, 2021 Share Posted March 11, 2021 The only correlation I have figured out might have something to do with using the web browser terminals. I've tried to avoid using them and I haven't had an instance in a while now. I've also made sure they were all closed down right after I used them. Before I was using them and possibly leaving them open on different machines. Not sure if that's it, but that's the only thing I can think of that might slightly correlate. Quote Link to comment
Chunks Posted April 5, 2021 Share Posted April 5, 2021 This has happened a couple times to me as well. In the last few weeks I've left browsers open and viewing the dashboard page for many days at a time (I'm the guy with 10 windows, 50 tabs in each, and since I've been working on a few things I end up with a bunch of Unraid dashboard / Main windows open). I'm using Chrome. I can confirm that running "/etc/rc.d/rc.nginx restart" via SSH resolved the problem for me (temporarily?) before my system completely locks up. Last time this happened the Web UI was completely hosed and I had to restart by force (I forgot to try SSH). Quote Link to comment
TBoneStaek Posted May 18, 2021 Share Posted May 18, 2021 I just had this problem myself. I usually keep Dashboard open in Firefox on my home PC always and when I'm away I also have it open while using my laptop. I'm running 6.9.2 and this is the first time I've run unto this issue. I rebooted my server before finding this temporary fix listed above. I know the webgui becomes unresponsive once the log fills up and I didn't want to chance it. The other thing I've been doing different is tinkering with VMs and I had left one on for a few hours but that shouldn't have caused it. Just throwing in my info in case it helps lead to a solution. I couldn't grab any logs, they were taking too long to compile. Just snagged a screenshot of the syslog so I could "google" it which led me here. Quote Link to comment
aurevo Posted May 21, 2021 Share Posted May 21, 2021 Today I ran into the same problem for the first time. Wondering there is still no solution other than restarting NGINX. This temporary fixes the not working dashboard etc. but the log is still on 100%. How to fix it or clear the logs? Logs attached. tower-diagnostics-20210521-0947.zip Quote Link to comment
joshbgosh10592 Posted May 31, 2021 Share Posted May 31, 2021 (edited) I'm still having this issue on 6.9.1. I have to purge /var/log/syslog (.1 and .2 as well) and then /etc/rc.d/rc.nginx restart in order for the pages to start responding again.... Edited June 1, 2021 by joshbgosh10592 autocorrect.... Quote Link to comment
joshbgosh10592 Posted June 19, 2021 Share Posted June 19, 2021 I've also noticed the /var/log/nginx throwing the out of shared memory error... Spamming this: 2021/06/19 01:10:53 [crit] 6642#6642: ngx_slab_alloc() failed: no memory 2021/06/19 01:10:53 [error] 6642#6642: shpool alloc failed 2021/06/19 01:10:53 [error] 6642#6642: nchan: Out of shared memory while allocating channel /cpuload. Increase nchan_max_reserved_memory. 2021/06/19 01:10:53 [error] 6642#6642: *5824862 nchan: error publishing message (HTTP status code 507), client: unix:, server: , request: "POST /pub/cpuload?buffer_length=1 HTTP/1.1", host: "localhost" 2021/06/19 01:10:53 [crit] 6642#6642: ngx_slab_alloc() failed: no memory 2021/06/19 01:10:53 [error] 6642#6642: shpool alloc failed 2021/06/19 01:10:53 [error] 6642#6642: nchan: Out of shared memory while allocating channel /var. Increase nchan_max_reserved_memory. 2021/06/19 01:10:53 [alert] 6642#6642: *5824863 header already sent while keepalive, client: 10.9.0.240, server: 0.0.0.0:80 2021/06/19 01:10:53 [alert] 27152#27152: worker process 6642 exited on signal 11 2021/06/19 01:10:53 [crit] 6798#6798: ngx_slab_alloc() failed: no memory Quote Link to comment
Gabriel_B Posted June 20, 2021 Share Posted June 20, 2021 Had my first runin with this issue today. Quote Link to comment
crazygambit Posted June 25, 2021 Share Posted June 25, 2021 I've also been having this issue. Has anyone found a solution? Or at least a way to increase the log size permanently, since 128M seems pretty small and for some reason when the logs roll they're saved in the same directory, which kinda negates the point of rolling in the first place (so the log doesn't fill up in the first place!). Quote Link to comment
Dovy6 Posted June 25, 2021 Share Posted June 25, 2021 The actual problem is that nginx keeps crashing, not that the logs get full. Each time nginx crashes, it logs it, and restarts. It then crashes again, logs it, and restarts. this keeps happening, which is why the logs quickly get filled. But the full log itself isn't the problem, and isn't why Unraid slogs down until you reboot. It's the bug causing nginx to crash that's the problem Quote Link to comment
Inenting Posted July 3, 2021 Share Posted July 3, 2021 I'm on Unraid Version 6.9.2 now and this still keeps happening to me. I increased my log size to 1gb but it just keeps on filling up and I don't want to keep on restarting so i just delete the old syslogs and nginx logs (both filled with same errors). Jul 3 04:45:04 unraidserver nginx: 2021/07/03 04:45:04 [error] 3377#3377: nchan: Out of shared memory while allocating channel /disks. Increase nchan_max_reserved_memory. Jul 3 04:45:04 unraidserver nginx: 2021/07/03 04:45:04 [error] 3377#3377: *1125090 nchan: error publishing message (HTTP status code 507), client: unix:, server: , request: "POST /pub/disks?buffer_length=1 HTTP/1.1", host: "localhost" Jul 3 04:45:04 unraidserver nginx: 2021/07/03 04:45:04 [crit] 3377#3377: ngx_slab_alloc() failed: no memory Jul 3 04:45:04 unraidserver nginx: 2021/07/03 04:45:04 [error] 3377#3377: shpool alloc failed I do keep multiple chrome tabs with unraid open for a very long time but this was never a problem before, it just started in december, fixed itself and now it came back? unraidserver-diagnostics-20210703-1357.zip Quote Link to comment
eribob Posted July 7, 2021 Share Posted July 7, 2021 (edited) Same error for me today. Could not even download diagnostics until I restarted nginx. monsterservern-diagnostics-20210707-1012.zip Edited July 7, 2021 by eribob Quote Link to comment
Dovy6 Posted July 7, 2021 Share Posted July 7, 2021 On 7/3/2021 at 7:59 AM, Inenting said: I'm on Unraid Version 6.9.2 now and this still keeps happening to me. I increased my log size to 1gb but it just keeps on filling up and I don't want to keep on restarting so i just delete the old syslogs and nginx logs (both filled with same errors). Jul 3 04:45:04 unraidserver nginx: 2021/07/03 04:45:04 [error] 3377#3377: nchan: Out of shared memory while allocating channel /disks. Increase nchan_max_reserved_memory. Jul 3 04:45:04 unraidserver nginx: 2021/07/03 04:45:04 [error] 3377#3377: *1125090 nchan: error publishing message (HTTP status code 507), client: unix:, server: , request: "POST /pub/disks?buffer_length=1 HTTP/1.1", host: "localhost" Jul 3 04:45:04 unraidserver nginx: 2021/07/03 04:45:04 [crit] 3377#3377: ngx_slab_alloc() failed: no memory Jul 3 04:45:04 unraidserver nginx: 2021/07/03 04:45:04 [error] 3377#3377: shpool alloc failed I do keep multiple chrome tabs with unraid open for a very long time but this was never a problem before, it just started in december, fixed itself and now it came back? unraidserver-diagnostics-20210703-1357.zip 448.46 kB · 0 downloads again, the logs filling up are a symptom, not the disease. You can increase the log size to the size of your array, it's nginx crashing that's the problem, not the log generated by that Quote Link to comment
nlz Posted August 16, 2021 Share Posted August 16, 2021 Had this problem today also. Quote Link to comment
artwodeetwo Posted August 17, 2021 Share Posted August 17, 2021 Undraid 6.9.2 I had this start being an issue for me too, and I'd just recently switched over to using Safari on a Mac. I always keep a tab open to my Unraid machine. After I found this and the query re: Safari I switched to Brave (was just giving Safari a chance really), rebooted the server to clear the log storage and haven't seen the message or an increase in logs (still on 1%) in the last ~6d. Quote Link to comment
jbuszkie Posted August 23, 2021 Share Posted August 23, 2021 This happened again to me last night. I've been really good about not leaving web terminal windows open. Well.. I didn't have any last night open or for a while. What I did have different was I had set my grafana window to auto refresh every 10s. I wonder if that had anything to do with this problem? Also.. The restart command didn't completely fix it this time. I was still getting a bunch of these... Aug 23 09:09:49 Tower nginx: 2021/08/23 09:09:49 [alert] 25382#25382: worker process 1014 exited on signal 6 Aug 23 09:09:50 Tower nginx: 2021/08/23 09:09:50 [alert] 25382#25382: worker process 1044 exited on signal 6 Aug 23 09:09:51 Tower nginx: 2021/08/23 09:09:51 [alert] 25382#25382: worker process 1202 exited on signal 6 Aug 23 09:09:53 Tower nginx: 2021/08/23 09:09:53 [alert] 25382#25382: worker process 1243 exited on signal 6 Aug 23 09:09:54 Tower nginx: 2021/08/23 09:09:54 [alert] 25382#25382: worker process 1275 exited on signal 6 Aug 23 09:09:55 Tower nginx: 2021/08/23 09:09:55 [alert] 25382#25382: worker process 1311 exited on signal 6 Aug 23 09:09:56 Tower nginx: 2021/08/23 09:09:56 [alert] 25382#25382: worker process 1342 exited on signal 6 Aug 23 09:09:57 Tower nginx: 2021/08/23 09:09:57 [alert] 25382#25382: worker process 1390 exited on signal 6 Aug 23 09:09:58 Tower nginx: 2021/08/23 09:09:58 [alert] 25382#25382: worker process 1424 exited on signal 6 Aug 23 09:09:59 Tower nginx: 2021/08/23 09:09:59 [alert] 25382#25382: worker process 1455 exited on signal 6 I started to kill my dockers and after I stopped the HASSIO group, That message stopped. I restarted the docker group and it hasn't comeback. I really wish we could get to the bottom of this!! FYI.. I'm now on 6.9.2 Quote Link to comment
dp100 Posted August 31, 2021 Share Posted August 31, 2021 (edited) Hi I found this page after experiencing the same issue myself. Running 6.9.2. I am guilty of leaving Chrome windows open with my Dashboard and/or Docker container list, which i've now closed. My log file hit 100% and i eventually managed to increase it (my terminal window keeps closing and resetting), by giving more space. I've also ran the '/etc/rc.d/rc.nginx restart' command. When i saw 'nginx' in the log, i wondered if it was a problem with my SWAG container, which i've closed. None of these measures have stopped the terminal window connection from constantly closing. No errors were found on an extended 'Fix Common Problems' scan. I have recently switched Plex to use a ram disk for transcodes, but i've never seen it anywhere near max out the system (i'm running 24gb) ram). I'm going to try a reboot. Any thoughts or suggestions appreciated! Cheers! darren syslog.zip Edited September 1, 2021 by dp100 typos! Quote Link to comment
Flemming Posted September 1, 2021 Share Posted September 1, 2021 I have the same problem and /etc/rc.d/rc.nginx don't fix anything, not even temporarily after: Logs attatched unraid-diagnostics-20210901-1251.zip Quote Link to comment
jbuszkie Posted September 1, 2021 Share Posted September 1, 2021 2 hours ago, Flemming said: I have the same problem and /etc/rc.d/rc.nginx don't fix anything, not even temporarily strange that /etc/rc.d/rc.nginx restart didn't fix it. I assume you made room in /var/log for more messages to come through? After the restart did you still have stuff spewing in the log file? I do recall a time where I had to do a restart to fix it completely. Jim Quote Link to comment
dp100 Posted September 1, 2021 Share Posted September 1, 2021 6 hours ago, jbuszkie said: strange that /etc/rc.d/rc.nginx restart didn't fix it. I assume you made room in /var/log for more messages to come through? After the restart did you still have stuff spewing in the log file? I do recall a time where I had to do a restart to fix it completely. Jim Hi Jim, Thanks for the comments. I'm fairly sure that '/etc/rc.d/rc.nginx restart' didn't fix it, but as the terminal window remained unusable (constantly closing/opening a connection) i was struggling to properly assess things, plus it was getting late. In the end i lost my patience and rebooted the system as i went to bed. Since then, the server has been fine and my log has stayed at 1%. Although it seems 'fixed' for now, i'd love to know what the cause is. Considering the number of users complaining of this issue affecting them too, there must be some underlying common bug or cause... cheers darren Quote Link to comment
mgutt Posted September 4, 2021 Share Posted September 4, 2021 I wonder nobody checked their log path: du -sh /var/log/* Maybe the ngnx error is only a subsequent error. Quote Link to comment
xthursdayx Posted September 17, 2021 Share Posted September 17, 2021 I've just started running into this issue as well, on a machine running unRAID 6.10.0-rc1 with 64Gb of memory, which has always been plenty, though I too am guilty of leaving unRAID WebUI tabs open (as well as mosh/ssh sessions running). Interestingly, I'm able to access the pages of the WebUI and see the header and nav buttons, as well as the major sections of pages like the Dashboard and Main, however they're empty, not showing my disks, docker containers, etc. The only way I've been able to (temporarily) “fix” the issue is by restarting my server. I tried restarting /etc/rc.d/rc.nginx/, but it didn't make any difference, as you can see in these logs: Sep 17 02:25:29 vulfTower nginx: 2021/09/17 02:25:29 [alert] 25209#25209: worker process 32623 exited on signal 6 Sep 17 02:25:29 vulfTower nginx: 2021/09/17 02:25:29 [alert] 25209#25209: worker process 32645 exited on signal 6 Sep 17 02:25:31 vulfTower nginx: 2021/09/17 02:25:31 [alert] 25209#25209: worker process 32647 exited on signal 6 Sep 17 02:25:31 vulfTower nginx: 2021/09/17 02:25:31 [alert] 25209#25209: worker process 334 exited on signal 6 Sep 17 02:25:31 vulfTower nginx: 2021/09/17 02:25:31 [alert] 25209#25209: worker process 339 exited on signal 6 Sep 17 02:25:32 vulfTower nginx: 2021/09/17 02:25:32 [alert] 25209#25209: worker process 350 exited on signal 6 Sep 17 02:25:33 vulfTower nginx: 2021/09/17 02:25:33 [alert] 25209#25209: worker process 405 exited on signal 6 Sep 17 02:25:33 vulfTower nginx: 2021/09/17 02:25:33 [alert] 25209#25209: worker process 565 exited on signal 6 Sep 17 02:25:33 vulfTower nginx: 2021/09/17 02:25:33 [alert] 25209#25209: worker process 591 exited on signal 6 Sep 17 02:25:34 vulfTower nginx: 2021/09/17 02:25:34 [alert] 25209#25209: worker process 594 exited on signal 6 Sep 17 02:25:34 vulfTower rsyslogd: file '/var/log/syslog'[2] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.2102.0 try https://www.rsys log.com/e/2027 ] Sep 17 02:25:34 vulfTower rsyslogd: action 'action-0-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.2102.0 try https:/ /www.rsyslog.com/e/2027 ] Sep 17 02:25:34 vulfTower rsyslogd: rsyslogd[internal_messages]: 561 messages lost due to rate-limiting (500 allowed within 5 seconds) Sep 17 02:25:34 vulfTower rsyslogd: file '/var/log/syslog'[2] write error - see https://www.rsyslog.com/solving-rsyslog-write-errors/ for help OS error: No space left on device [v8.2102.0 try https://www.rsys log.com/e/2027 ] Sep 17 02:25:34 vulfTower rsyslogd: action 'action-0-builtin:omfile' (module 'builtin:omfile') message lost, could not be processed. Check for additional error messages before this one. [v8.2102.0 try https:/ /www.rsyslog.com/e/2027 ] ad infinitum I'm also getting these errors in my logs: Sep 17 02:32:54 vulfTower nginx: 2021/09/17 02:32:54 [alert] 25209#25209: worker process 4936 exited on signal 6 Anyway, I tried running du to check my log sizes as well as df to check how full my cache drives and boot flash drive are, to see if I could identify the issue. You can see the results below. While my NGINX and Syslog are large, neither seem to be large enough to disable access to the WebUI, and neither of my cache drives (where I store my syslog backup). ❯ du -sh /var/log/* 4.0K /var/log/apcupsd.events 0 /var/log/btmp 0 /var/log/cron 0 /var/log/debug 88K /var/log/dmesg 1012K /var/log/docker.log 0 /var/log/faillog 2.8M /var/log/file.activity.log 16K /var/log/gitflash 4.0K /var/log/lastlog 4.0K /var/log/libvirt 4.0K /var/log/maillog 0 /var/log/messages 0 /var/log/nfsd 52M /var/log/nginx 0 /var/log/packages 0 /var/log/pkgtools 0 /var/log/plugins 0 /var/log/preclear.disk.log 0 /var/log/pwfail 0 /var/log/removed_packages 0 /var/log/removed_scripts 0 /var/log/removed_uninstall_scripts 20K /var/log/samba 0 /var/log/scripts 0 /var/log/secure 0 /var/log/setup 0 /var/log/spooler 0 /var/log/swtpm 69M /var/log/syslog 3.6M /var/log/syslog.1 0 /var/log/vfio-pci 8.0K /var/log/wtmp ❯ df -h /mnt/cache Filesystem Size Used Avail Use% Mounted on /dev/sdf1 466G 306G 157G 67% /mnt/cache ❯ df -h /mnt/cache_io Filesystem Size Used Avail Use% Mounted on /dev/sdk1 932G 244G 688G 27% /mnt/cache_io ❯ df -h /boot Filesystem Size Used Avail Use% Mounted on /dev/sda1 7.5G 893M 6.6G 12% /boot Any ideas? For some reason, I'm currently unable to download my diagnostics, but any advice or ideas would be much appreciated. Cheers! Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.