The web user interface becomes inaccessible due to a crash of the NGINX server.

[Shine]

Explanation of the situation：

 

The web user interface being inaccessible and the abnormal termination of the NGINX service are likely indicative of a system bug. This assumption is reinforced by the fact that two other friends who upgraded with me have also encountered the same issue. As a temporary solution, I have observed that forcefully terminating the NGINX process and manually restarting it restores the web user interface temporarily. However, after a period of time, the error resurfaces.

 

 

 

 

This is the error information from my nginx.

 

2023/06/28 16:22:37 [info] 2466#2466: Using 131072KiB of shared memory for nchan in /etc/nginx/nginx.conf:161
2023/06/28 16:22:52 [info] 3615#3615: Using 116KiB of shared memory for nchan in /etc/nginx/nginx.conf:161
2023/06/28 16:22:52 [info] 3615#3615: Using 131072KiB of shared memory for nchan in /etc/nginx/nginx.conf:161
ter process /usr/sbin/nginx -c /etc/nginx/nginx.conf: ./nchan-1.3.6/src/store/memory/memstore.c:705: nchan_store_init_worker: Assertion `procslot_found == 1' failed.
2023/06/28 16:22:52 [alert] 8504#8504: worker process 3618 exited on signal 6
2023/06/28 16:22:52 [alert] 8504#8504: shared memory zone "memstore" was locked by 3618
ter process /usr/sbin/nginx -c /etc/nginx/nginx.conf: ./nchan-1.3.6/src/store/memory/memstore.c:705: nchan_store_init_worker: Assertion `procslot_found == 1' failed.
2023/06/28 16:22:52 [alert] 8504#8504: worker process 3628 exited on signal 6
2023/06/28 16:22:52 [alert] 8504#8504: shared memory zone "memstore" was locked by 3628
ter process /usr/sbin/nginx -c /etc/nginx/nginx.conf: ./nchan-1.3.6/src/store/memory/memstore.c:705: nchan_store_init_worker: Assertion `procslot_found == 1' failed.
2023/06/28 16:22:52 [alert] 8504#8504: worker process 3629 exited on signal 6
2023/06/28 16:22:52 [alert] 8504#8504: shared memory zone "memstore" was locked by 3629
ter process /usr/sbin/nginx -c /etc/nginx/nginx.conf: ./nchan-1.3.6/src/store/memory/memstore.c:705: nchan_store_init_worker: Assertion `procslot_found == 1' failed.
2023/06/28 16:22:52 [alert] 8504#8504: worker process 3631 exited on signal 6
2023/06/28 16:22:52 [alert] 8504#8504: shared memory zone "memstore" was locked by 3631
ter process /usr/sbin/nginx -c /etc/nginx/nginx.conf: ./nchan-1.3.6/src/store/memory/memstore.c:705: nchan_store_init_worker: Assertion `procslot_found == 1' failed.
2023/06/28 16:22:52 [alert] 8504#8504: worker process 3632 exited on signal 6
2023/06/28 16:22:52 [alert] 8504#8504: shared memory zone "memstore" was locked by 3632
ter process /usr/sbin/nginx -c /etc/nginx/nginx.conf: ./nchan-1.3.6/src/store/memory/memstore.c:705: nchan_store_init_worker: Assertion `procslot_found == 1' failed.

 

Proposed solution：

 

Execute via SSH

pkill -9 nginx

/etc/rc.d/rc.nginx start

 

 

 

[ljm42]

Please upload your full diagnostics.zip file (from Tools -> Diagnostics). It may have clues as to what is causing the issue.

[ki2k]

I've observed same issue after upgrading to Unraid 6.12.2 today. I'm also following https://forums.unraid.net/topic/141105-unable-to-connect-to-webui-ssh-works-diagnostics-hangs/  and  https://forums.unraid.net/bug-reports/stable-releases/612-unraid-webui-stop-responding-then-nginx-crash-r2451/.

Edited July 3, 2023 by ki2k

[seanwalter]

I have also been having this problem since updating to 6.12. Was hoping 6.12.2 would finally resolve it. Diagnostics attached. For now I have this script in my /mnt/cache directory so that I can ssh into unraid and manually restart nginx (many thanks to someone who posted the main two command lines in another unraid forum, and some people at stackoverflow for the error-checking code...):

 

#!/usr/bin/env bash

pkill -9 nginx
pkillexitstatus=$?

if [ $pkillexitstatus -eq 0 ]; then
    echo "pkill exited normally with nginx killed"
elif [ $pkillexitstatus -eq 1]; then
    echo "could not find nginx process to kill"
elif [ $pkillexitstatus -eq 2]; then
    echo "syntax error in trying to kill nginx using \'pkill -9 nginx\'"
elif [ $pkillexitstatus -eq 3]; then
    echo "fatal error using \'pkill -9 nginx\'"
else
    echo UNEXPECTED
fi

/etc/rc.d/rc.nginx start

nginxstatus=$?

if [ $nginxstatus -eq 0 ]; then
    echo "nginx restarted normally with exit code " $nginxstatus
else
    echo "nginx restart returned an abnormal exit code: " $nginxstatus
fi

 

home-raid-diagnostics-20230703-1316.zip

[kekayin]

On 7/4/2023 at 4:24 AM, seanwalter said:

I have also been having this problem since updating to 6.12. Was hoping 6.12.2 would finally resolve it. Diagnostics attached. For now I have this script in my /mnt/cache directory so that I can ssh into unraid and manually restart nginx (many thanks to someone who posted the main two command lines in another unraid forum, and some people at stackoverflow for the error-checking code...):

 

#!/usr/bin/env bash

pkill -9 nginx
pkillexitstatus=$?

if [ $pkillexitstatus -eq 0 ]; then
    echo "pkill exited normally with nginx killed"
elif [ $pkillexitstatus -eq 1]; then
    echo "could not find nginx process to kill"
elif [ $pkillexitstatus -eq 2]; then
    echo "syntax error in trying to kill nginx using \'pkill -9 nginx\'"
elif [ $pkillexitstatus -eq 3]; then
    echo "fatal error using \'pkill -9 nginx\'"
else
    echo UNEXPECTED
fi

/etc/rc.d/rc.nginx start

nginxstatus=$?

if [ $nginxstatus -eq 0 ]; then
    echo "nginx restarted normally with exit code " $nginxstatus
else
    echo "nginx restart returned an abnormal exit code: " $nginxstatus
fi

 

home-raid-diagnostics-20230703-1316.zip 255.62 kB · 2 downloads

Thanks for your workaround. I can access web gui in this way now. 

[kharntiitar]

On 7/4/2023 at 4:24 AM, seanwalter said:

I have also been having this problem since updating to 6.12. Was hoping 6.12.2 would finally resolve it. Diagnostics attached. For now I have this script in my /mnt/cache directory so that I can ssh into unraid and manually restart nginx (many thanks to someone who posted the main two command lines in another unraid forum, and some people at stackoverflow for the error-checking code...):

 

#!/usr/bin/env bash

pkill -9 nginx
pkillexitstatus=$?

if [ $pkillexitstatus -eq 0 ]; then
    echo "pkill exited normally with nginx killed"
elif [ $pkillexitstatus -eq 1]; then
    echo "could not find nginx process to kill"
elif [ $pkillexitstatus -eq 2]; then
    echo "syntax error in trying to kill nginx using \'pkill -9 nginx\'"
elif [ $pkillexitstatus -eq 3]; then
    echo "fatal error using \'pkill -9 nginx\'"
else
    echo UNEXPECTED
fi

/etc/rc.d/rc.nginx start

nginxstatus=$?

if [ $nginxstatus -eq 0 ]; then
    echo "nginx restarted normally with exit code " $nginxstatus
else
    echo "nginx restart returned an abnormal exit code: " $nginxstatus
fi

 

home-raid-diagnostics-20230703-1316.zip 255.62 kB · 2 downloads

Another thank you for this workaround. I'd love to find out a reason for it happening, it just suddenly started for me about 2 weeks ago.

[kharntiitar]

Here is a short script that reboots nginx every hour. I wanted it to run in a screen/tmux session so that *hopefully* when the next update comes out the issue is resolved, and the subsequent reboot will stop the script. if not, I just manually run it again, until the next version...

 

I don't think I'll notice nginx rebooting every hour, but if it becomes an issue, I can always stop it and change the sleep time.

 

#!/bin/bash

# Endless loop
while true; do
    # Kill nginx processes forcefully
    pkill -9 nginx

    # Start nginx
    /etc/rc.d/rc.nginx start

    # Sleep for one hour before next iteration.
    sleep 1h
done

 

[wolfling]

This issue has also troubled me for a long time, until yesterday when I checked the logs. The problem with nginx seems to have occurred after the automatic backup process of Flash started. I have now uninstalled the Unraid Connect plugin, and the system has been running for over 24 hours without any crashes (previously it would crash after approximately 2-3 hours). Everything is normal, and there are no records similar to "memstore" or any records related to nginx in the logs.

[kaleb112494]

On 7/3/2023 at 3:24 PM, seanwalter said:

For now I have this script in my /mnt/cache directory so that I can ssh into unraid and manually restart nginx

 

Thank you for this. I have been experiencing this issue since 6.10.X, as I am notorious for leaving browser sessions open. Back before unraid-api, I was able to just restart rc.nginx without issue, but that solution stopped working after 6.11.3. Similar resolution, different methodology.

EDIT: I suppose we can't really call it a solution until the problem ceases to occur in the first place.

Edited March 6 by kaleb112494