[6.12] Unraid webui stop responding then Nginx crash

2 hours ago, mikeofoslo said:

- set Interface eth0 to IPv4 only

Interesting, I also have IPv6 enabled. Since I still have crashes with 6.12.2 I'll test disabling IPv6 temporarily in unraid and see if that is the cause of the issue.

Btw I'm only using firefox and not chrome, so I doubt it is a browser issue. In general a browser should never be able to crash a webserver regardless of what it does.

Changing to IPV4 only seems to have resolved it. 4 days with no crash, and I leave the tab open on multiple machines constantly.

 

Hilariously, it also fixed my minecraft docker containers failing to start (2/3), because the addons were failing to resolve.

20 hours ago, SpaceInvader said:

In general a browser should never be able to crash a webserver regardless of what it does.

 

 Oh I agree : ) I'm just trying to narrow down potential causes. 

 

 

22 hours ago, mikeofoslo said:

- set Interface eth0 to IPv4 only
...
24h has passed since the router rebuild (reinstall) and the unraid server WebGUI is still running. 

 

18 hours ago, LilDrunkenSmurf said:

Changing to IPV4 only seems to have resolved it. 4 days with no crash, and I leave the tab open on multiple machines constantly.

 

This is great news, and gives us something to focus on. I'd encourage anyone else with nginx issues to disable IPv6 (as a potential short term fix)

I was having the same problem as well with the webUI dashboard updates stopping after less than a minute and nginx hanging after a day or two, then having to manually kill nginx processes.  I temporarily disabled IPv6 via sysctl settings and removed the ::1 references in rc.nginx, restarted nginx, and everything has been working perfectly ever since.

 

When I get a chance, I'll properly disable IPv6 in the Unraid settings and reboot (My ISP's IPv6 support comes and goes and currently I have it disabled on my router so I might as well disable it correctly/permanently in Unraid as well).  Glad to finally know a work-around.

6 hours ago, DieselDrax said:

I temporarily disabled IPv6 via sysctl settings and removed the ::1 references in rc.nginx, restarted nginx, and everything has been working perfectly ever since.

Is /etc/nginx/conf.d/servers.conf the correct place to remove the ::1 referance?

 

# Always accept http requests from localhost
# ex: http://localhost
# ex: http://127.0.0.1
# ex: http://[::1]
#
server {
    listen 127.0.0.1:80; # lo
    listen 127.0.0.1:443; # lo
    listen [::1]:80; # lo
    listen [::1]:443; # lo
 

4 hours ago, mikeofoslo said:

Is /etc/nginx/conf.d/servers.conf the correct place to remove the ::1 referance?

 

# Always accept http requests from localhost
# ex: http://localhost
# ex: http://127.0.0.1
# ex: http://[::1]
#
server {
    listen 127.0.0.1:80; # lo
    listen 127.0.0.1:443; # lo
    listen [::1]:80; # lo
    listen [::1]:443; # lo
 

 

No, that is generated by /etc/rc.d/rc.nginx as part of starting nginx, if you remove the lines from servers.conf then the rc script will add them back and nginx will error that it can't assign the requested address and fail to start.

3 hours ago, DieselDrax said:

 

No, that is generated by /etc/rc.d/rc.nginx as part of starting nginx, if you remove the lines from servers.conf then the rc script will add them back and nginx will error that it can't assign the requested address and fail to start.

Then I figure the two lines below need to be hashed out in /etc/rc.d/rc.nginx
#    echo "${t}listen [::1]:$PORT; # lo"
#    echo "${t}listen [::1]:$PORTSSL; # lo"

 

(I'm a linux n00b and want to be sure)

I have seen some users who disable ipv6 in their 'go' file, using a command like this:

--sysctl net.ipv6.conf.all.disable_ipv6=1

or

echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6

 

You need to remove these lines because it conflicts, Unraid takes care of enabling / disabling IPv6 per interface as configured

 

57 minutes ago, mikeofoslo said:

Then I figure the two lines below need to be hashed out in /etc/rc.d/rc.nginx
#    echo "${t}listen [::1]:$PORT; # lo"
#    echo "${t}listen [::1]:$PORTSSL; # lo"

 

(I'm a linux n00b and want to be sure)

 

Correct, if you want to temporarily test.  This is not the official method.

 

18 minutes ago, bonienl said:

I have seen some users who disable ipv6 in their 'go' file, using a command like this:

--sysctl net.ipv6.conf.all.disable_ipv6=1

or

echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6

 

You need to remove these lines because it conflicts, Unraid takes care of enabling / disabling IPv6 per interface as configured

 

 

Yup, disabling IPv6 needs to be done via the UI and not by manual overrides.  I simply did this to confirm disabling IPv6 resolved the issue (and it did) before stopping all services and disabling IPv6 via UI (which I also did after testing).

I'm having system instability as well (have for quite awhile actually, over various updates).  But it for this particular one it does look like either closing the tab with the GUI open, OR disabling IPV6 fixes the issue.  It is too early for me to tell yet whether this fixes my system instability issues though.  One thing at a time  

im still having stability issues dont have IPV6 enabled and have been closing the tabs but still woke up this morning and the webgui and most of the containers unresponsive. rebooted and then took this diagnostic

 

 

 

synas-diagnostics-20230707-0655.zip

I found my problem, the Microsoft Edge Browser was set to put background tabs to sleep, and that eventually caused the Webgui to lock up.  I don't know if that fixes the problem, but by turning that off it seems to have fixed mine.

8 hours ago, laffer98 said:

I found my problem, the Microsoft Edge Browser was set to put background tabs to sleep, and that eventually caused the Webgui to lock up.  I don't know if that fixes the problem, but by turning that off it seems to have fixed mine.

 

I use Edge and it puts my Unraid dashboard tab to sleep and I haven't had any UI/dashboard issues as a result of this, they have all been related to IPv6 being enabled.  When I click the dashboard tab in Edge it does a bit of a fast-forward before showing the real-time updates which is odd but it has never caused any problems.  This is with Unraid 6.12.2.

Edited July 7, 2023 by DieselDrax

For me this bug seems to be related for unraid webgui tab being open too long (Chrome). Normally I have my computer open 24/7 and also unraid webgui tab open at browser. This haven't been issue until upgrading to 6.12.2. After this upgrade I experienced webgui crash twice. On my server it seems not to be related to IPv6 because I've always had IPv4 only setting.

 

I couldn't get webgui back although I killed nginx and tried to restart and reload it and also tried to restart and reload php but it didn't make any difference. Otherwise everything worked normally, VMs and dockers kept running and smb shares worked and I could connect to server by ssh. I could get to unraid login screen if I cleared cookies but after login I just got http error 500. Only way to get webgui back was to restart whole server.

 

Only common log entry before both crashes was this, it was repeated at 1 hour interval until I noticed webgui crash and rebooted:

Quote

crond[1448]: exit status 255 from user root php /usr/local/emhttp/plugins/community.applications/scripts/notices.php > /dev/null 2>&1

 

I don't know if that is cause or just effect after webgui has been crashed or is it any way related to issue but that was on log before both crashes and I haven't seen that after those crashes.

 

Now I've kept unraid webgui tab closed at browser when not operating and webgui has been up +24 hours. Let's see if it's just good luck.

Edited July 10, 2023 by windiz

On 7/5/2023 at 3:07 AM, SpaceInvader said:

I'll test disabling IPv6 temporarily in unraid and see if that is the cause of the issue.

Since disabling IPv6 in the network settings (6 days ago) I didn't have another crash . Hopefully that helps narrowing the issue down and fixing it.

On 7/7/2023 at 6:59 AM, Syrade said:

im still having stability issues dont have IPV6 enabled and have been closing the tabs but still woke up this morning and the webgui and most of the containers unresponsive. rebooted and then took this diagnostic

 

 

 

synas-diagnostics-20230707-0655.zip

i rolled back to 6.12.1 from 6.12.2 and have not had an issue in 4 days since i did this

On 7/10/2023 at 1:39 PM, windiz said:

Now I've kept unraid webgui tab closed at browser when not operating and webgui has been up +24 hours. Let's see if it's just good luck.

 

Okay it was just a good luck. Woke up this morning again to see webgui has crashed (http error 500). It worked about 4 days. I didn't have webgui open at browser this time and still crashed so that doesn't seem to trigger crash. And as I wrote earlier I don't have IPv6 enabled so that shouldn't also be issue.

 

Again there wasn't anything other special at syslog than that "exit status 255 from user root php" repeated at 1 hour interval which I mentioned at earlier post.

 

Maybe I just need to rollback version for now.

3 hours ago, windiz said:

Maybe I just need to rollback version for now.

You can also try 6.12.3-rc3 and see if that helps.

After updating to 6.12.3 I thought I could enable IPv6 again, but after only a few minutes the webui crashed again.

Nginx log contains this error a bunch of times, when the error occured:

2023/07/16 01:54:54 [alert] 5530#5530: shared memory zone "memstore" was locked by 4007
ter process /usr/sbin/nginx -c /etc/nginx/nginx.conf: ./nchan-1.3.6/src/store/memory/memstore.c:705: nchan_store_init_worker: Assertion `procslot_found == 1' failed.
2023/07/16 01:54:54 [alert] 5530#5530: worker process 4087 exited on signal 6

 

 

Let me know, if I can do anything else, to help find the cause.

nas-diagnostics-20230716-0206.zip

Edited July 16, 2023 by SpaceInvader

1 hour ago, SpaceInvader said:

After updating to 6.12.3 I thought I could enable IPv6 again, but after only a few minutes the webui crashed again.

Nginx log contains this error a bunch of times, when the error occured:

2023/07/16 01:54:54 [alert] 5530#5530: shared memory zone "memstore" was locked by 4007
ter process /usr/sbin/nginx -c /etc/nginx/nginx.conf: ./nchan-1.3.6/src/store/memory/memstore.c:705: nchan_store_init_worker: Assertion `procslot_found == 1' failed.
2023/07/16 01:54:54 [alert] 5530#5530: worker process 4087 exited on signal 6

 

 

Let me know, if I can do anything else, to help find the cause.

nas-diagnostics-20230716-0206.zip

 

And disabling IPv6 makes those messages go away? Would you mind sharing diagnostics that show that?

No problem found in 6.12.3-rc3, just updated 6.12.3, waiting for testing

4 hours ago, ljm42 said:

 

And disabling IPv6 makes those messages go away? Would you mind sharing diagnostics that show that?

The attached diagnostic is with "ipv4 only" in network settings and no issues as well as no logged nginx error. I rebooted after changing to ipv4 and the system was running for about two hours.

 

 

After I wrote the last message I also noticed that the ipv6 network went down (ping -6 gave an error). So it could be that the ipv6 connection breaks for some reason and that causes the nginx crash? Maybe something related to my network (cheap isp router that might not use standards). I'll enable ipv6 again and try to find the cause of the connection issue, when it happens again. ->  After trying again I could not replicate this, so it was probably something else. The error from the screenshot above occured again.

nas-diagnostics-20230716-0607.zip

Edited July 16, 2023 by SpaceInvader

6 hours ago, SpaceInvader said:

After updating to 6.12.3 I thought I could enable IPv6 again, but after only a few minutes the webui crashed again.

 

Can you disable the nginx proxy and test with a plain connection.

 

Jul 16 01:28:17 NAS rc.docker: container Nginx-Proxy-Manager has an additional network that will be restored: none;

 

There is also something wrong with your router and IPv6 assignment. You need to check router configuration.

 

Jul 16 02:06:21 [1123]: br0: part of a Router Advertisement expired
Jul 16 02:06:21 [1123]: route socket overflowed (rcvbuflen 106496) - learning interface state
Jul 16 02:06:21 [1123]: drained 279 messages

 

12 hours ago, bonienl said:

Can you disable the nginx proxy and test with a plain connection.

The nginx proxy manager docker was just for other docker services and I never used it to access the webui. 
To be sure it is not docker related I disabled docker entirely in the ui and still had the same issue (see attached diagnostics).

 

12 hours ago, bonienl said:

There is also something wrong with your router and IPv6 assignment. You need to check router configuration

That could be true, unfortunately it is and ISP provided router with literally no IPv6 configuration options except a basic firewall. But this router has worked with the the same NAS that is now running unraid, when I was running debian on it before. IPv6 also works fine with android and windows devices as well as debian on my notebook. So I don't think it is totally broken.

 

Another thing I just noticed is that with IPv6 enabled there always is this /usr/sbin/atd running a nginx reload every three seconds. This stays there all the time like in my screenshot (the rc.rpc update is not always there)

Also the Task number roughly doubles when enabling IPv6:

 

nas-diagnostics-20230716-2015.zip

Edited July 16, 2023 by SpaceInvader

16 minutes ago, SpaceInvader said:

But this router has worked with the the same NAS that is now running unraid

 

In version 6.12 services are bound to valid IP addresses for security reasons, this includes IPv6 addresses when IPv6 is enabled, this requires both IPv4 and IPv6 to have correct assignments.

 

Earlier versions of Unraid did not have this requirement and an improper IPv6 assignment would probably go unnoticed when the server is accessed on its IPv4 address.

 

Thanks for the additional information, it gives me more insight in what is happening.