[6.12] Unraid webui stop responding then Nginx crash

Just found this thread after pulling my hair out for 3 days of constant crashes.   Turned off IPv6 and no crashes for 18 hours.  Hopefully good to go.  My server had been up for months before this.   All I had changed was new drives for my cache pool and converted to ZFS.  My network settings had been on IPv4 and IPv6 combo for at least a year before this.

Edited August 21, 2023 by Vatash
extra info

I can now confirm that I have had the server running 6.12.3 with disabled IPv6 for: 

Uptime 1 month 4 days 15 hours 43 minutes

We have a 6.12.4 rc release available. This release doesn't specifically target these nginx worker process issues but it does have significant updates to networking so I would expect to see improvements, particularly for people who are having issues related to IPv6.

 

We'd appreciate your feedback on whether nginx issues are improved:

 

 

17 hours ago, ljm42 said:

We have a 6.12.4 rc release available. This release doesn't specifically target these nginx worker process issues but it does have significant updates to networking so I would expect to see improvements, particularly for people who are having issues related to IPv6.

 

We'd appreciate your feedback on whether nginx issues are improved:

 

 

Installing now, running both ipv6 and having the macvlan issue, will update if the issue continues 
Edit: after an hour I'm noticing my webgui graphs are still functioning, before the graphs would fail after about 5 mins, then the gui wuld crash after around 20 mins. Things look good so far  

Edited August 23, 2023 by otakunorth
updating results

24 hours later, still going strong  

my machine happed also, add a user script to restart nginx automaticly, hope it decrease problem.

 

```

#!/bin/bash

pid=$(cat /var/run/nginx.pid)
if [[ $(ps -ef | awk "\$3==$pid" | wc -l) -eq 0 ]]; then
    /etc/rc.d/rc.nginx start
    sleep 5
    exit 0;
fi
ps -ef | awk "\$3==$pid" | grep '<defunct>' > /dev/null
if [[ $? -eq 0 ]]; then
    kill -9 $pid
    sleep 1
    /etc/rc.d/rc.nginx start
    sleep 5
fi

```

 

Edited August 26, 2023 by xiehuc

I have 6.12.4 installed and I crashed hard this morning.  I'll need to setup a syslog server on my workstation to get any idea of why.  I also hooked my box's HDMI output to a TV so I can look at that as well.

on 6.12.3 it maxed out my CPUs, not only ssh or nginx issue. Downgraded to 6.11.5 and running strong 29 days, except Docker has some problems network port are in use, some even force update cannot start, like deluge.

Been running the latest beta for a week now, fully stable  

6.12.4 just can hold 4hr.... still crash

7 hours ago, ianlee168 said:

6.12.4 just can hold 4hr.... still crash

 

这个帖子我全部都看了一遍，如果你有开启 IPV6 的话题，有尝试过这个帖子里面说的关闭 IPV6 或者指定一个固定的 IPV6 DNS 么？

I am also still seeing crashes, seems more frequently than before. 

IPV4 only, ipvlan since 6.12.3, bridging and bonding set to No

oxygen-diagnostics-20230903-2316.zip oxygen-diagnostics-20230904-2039.zip

这个问题，三个版本都还修复不了，真是服了

So my unraid server was acting similar to what was described here, so i wanted to post a solution here in case those watching this post (as i did for weeks), dont see their problems solved with the recent docker networking changes in 6.12.4.

 

A symptom i missed from this whole ordeal was that the unraid GUI was crashing (docker containers all appeared fine, but deluge was ALWAYS not functioning).  If this sounds similar to you, check out the post below.
 

TLDR for below: binhex torrent container has a special release tag (like :latest) to help prevent this issue from happening.

":libtorrentv1"

 

Since this is the longest thread with this issue, and all over the place even on reddit, there's at least 20 of us if not more, that have this issue dating back to July. It's definitely not an IPv6 issue. I have rolled back from 6.12.4 because it persists there, and also in 6.12.3. I'd like to solve this, but it's pretty out of hand. It is now almost November, so this is a really big problem.  Does anyone have an update on this? I lose SSH and access to the ui., as well as unresponsive or slow CLI on local. Which then makes restarting nginx impossible. That doesn't explain SSH going out too. Maybe we should see what plugins we all have in common?

16 minutes ago, Bifrost Systems said:

Since this is the longest thread with this issue, and all over the place even on reddit, there's at least 20 of us if not more, that have this issue dating back to July. It's definitely not an IPv6 issue. I have rolled back from 6.12.4 because it persists there, and also in 6.12.3. I'd like to solve this, but it's pretty out of hand. It is now almost November, so this is a really big problem.  Does anyone have an update on this? I lose SSH and access to the ui., as well as unresponsive or slow CLI on local. Which then makes restarting nginx impossible. That doesn't explain SSH going out too. Maybe we should see what plugins we all have in common?

 

I think there are serveral different issues with similar symptoms being discussed in this thread. The one I had (with the same log messages as the op) was indeed "solved" by disabling automatic ipv6 dns. I did not test yet if the last update fixed the issue, since it has been stable this way for me. But in my case  and the others with that exact issue I saw only the web interface was unresponsive and ssh and other services still worked normally. My issue also occurred with a completely fresh usb stick without any settings and storage setup.

 

Since these issues seem to be similar and are quite severe maybe the unraid team should consider sorting the issues by type and providing commands to collect additional information (maybe start some services in verbose mode or something similar), since the logs in the diagnostics pack didn't contain much useful info, when I looked into it.

I rolled back to 6.12.01 to stop having this issue (losing access to the ui and all the apps pretty much locking up) for my rig and have not even thought about upgrading untill i see reports of the issue being resolved on this thread mine was not ipv6 related 

6.12.4 same problem really frustrating when im away from home!! i keep rebooting my server to get into the UI

 

 

I'm glad this issue appeared for me while I was on a trial, no way I'm paying $400 AUD for Unraid with issues like this that run for a year and no fix.

 

Truenas is free, probably going to be forced to go back.

 

Incredibly frustrating, spent a day setting up an array, caches, setting everything up. No docker containers, barely any plugins.

 

Started a copy and went to bed. Woke up and cannot access via GUI or ssh.

 

/etc/rc.d/rc.php-fpm restart did nothing

/etc/rc.d/rc.nginx reload did nothing

 

 

 

This thread went a bit silent over the last weeks. Which is probably a good sign ?!? Is this issue still there in latest 6.12.6 ? I remember I upgraded last year to Unraid 6.12.x from 6.11.5 and after several restarts, not responding GUI and a lot of frustration. I went to back to 6.11.5 and everything runs rock stable since then.

However I like to keep my system up to date .. so anyone having luck with latest release in regards to these nginx crashes ? Is it worth a try ?

I have been on 6.12.6 for 1 month 25 days without a single crash.  I've been keeping the Dashboard open in multiple browsers.  Prior to the .6 release, the system would normally have issues with 3-5 days of uptime if I would leave the Dashboard open in a browser.

 

At least for me, 6.12.6 seems to have resolved the issues.

For what it’s worth I’ve just found this thread because I believe I’m having this issue running 6.12.6.

 

syslogs shows nginx crash and the whole server becomes unresponsive (even to pings) and I have to power off with the power button. 
 

it seems to happen when the server is idle for a couple of hours. 

Edited February 3 by xreyuk

I just ran into this error for the first time tonight. I am on the latest stable version (cant see what it is right now) I have a docker running Minio and moving 1Gbps sustained to the zfs array. otherwise, it is idle. I SSH'd into the server to see if I can run diagnostics, 30 minutes later, I am still waiting on the command to return. Really just an FYI.

6.12.8: Error still exists. Nothing more to say because users here have said it all. I hope it gets fixed.

I can confirm, 6.12.10, also crashing nginx, server unresponsive and the only way to get it back to life is a hard reboot, which then triggers a parity scan