Unable to connect to WebUI, SSH works, `diagnostics` hangs

[Dyon]

I am on Unraid 6.12.1. I also am unable to access the WebUI after some time. It does work initially, but after having the GUI open for some time, I notice it stops updating and becames inaccessible. Everything else just keeps on working fine, but it's just the WebUI being dead. Rebooting the servers via SSH works. After reboot the WebUI works again.

If I SSH to my server and run `diagnostics` it just hangs...

I also don't see anything getting added to /boot/logs. What can I do? Should I upload diagnostics AFTER a reboot when the UI works maybe?

 

I have 2 NICs, via both ports I am still able to access shares or SSH. So there is a link.

[ljm42]

> If I SSH to my server and run `diagnostics` it just hangs...

 

How long do you wait? Most of the commands diagnostics runs have 30 second timeouts, so if 5 commands have issues that would add 2.5 minutes. Let it run for say 10-15 minutes before killing it to see if it is able to gather diags

[Dyon]

19 hours ago, ljm42 said:

> If I SSH to my server and run `diagnostics` it just hangs...

 

How long do you wait? Most of the commands diagnostics runs have 30 second timeouts, so if 5 commands have issues that would add 2.5 minutes. Let it run for say 10-15 minutes before killing it to see if it is able to gather diags

Waited about 30+ minutes.  

 

I think php-fpm probably broke, so running PHP was not possible.

 

Using kill -9 on nginx I was able to kill NGINX and also by running I was able to connect to the WebUI:
/etc/rc.d/rc.nginx restart

/etc/rc.d/rc.nginx reload

/etc/rc.d/rc.php-fpm restart

/etc/rc.d/rc.php-fpm reload

 

/etc/rc.d/rc.nginx stop just didnt kill the proces...

 

After that I was also able to run diagnostics, which took literally 30 seconds indeed.

I also had a `cat /var/log/nginx/error.log` open before shutting down the server. I dont think it's included in the diagnostics. It's not the full log file, but only as much as my terminal could show. Forgot to pull the actual file of the server.

 

dyon-unraid-diagnostics-20230627-1520.zip nginx.log

[ljm42]

Nice troubleshooting

 

The syslog is full of this, anyone have ideas?

Jun 27 04:30:58 Dyon-unRAID kernel: i915 0000:00:02.0: [drm] *ERROR* Unexpected DP dual mode adaptor ID 7f
Jun 27 04:31:01 Dyon-unRAID crond[1519]: exit status 255 from user root /usr/local/emhttp/plugins/dynamix/scripts/monitor &> /dev/null

 

[Shine]

13 hours ago, ljm42 said:

Nice troubleshooting

 

The syslog is full of this, anyone have ideas?

Jun 27 04:30:58 Dyon-unRAID kernel: i915 0000:00:02.0: [drm] *ERROR* Unexpected DP dual mode adaptor ID 7f
Jun 27 04:31:01 Dyon-unRAID crond[1519]: exit status 255 from user root /usr/local/emhttp/plugins/dynamix/scripts/monitor &> /dev/null

 

Yes, I have also encountered this issue where only the web UI cannot be accessed and the logs indicate abnormal termination of the nginx service. I believe this is a system bug, as two other friends who upgraded with me have also encountered this problem. When I forcefully terminate the nginx process and manually restart it, the web UI resumes its services. However, after some time, the error reappears.

[Shine]

This is the error information from my nginx.

 

2023/06/28 16:22:37 [info] 2466#2466: Using 131072KiB of shared memory for nchan in /etc/nginx/nginx.conf:161
2023/06/28 16:22:52 [info] 3615#3615: Using 116KiB of shared memory for nchan in /etc/nginx/nginx.conf:161
2023/06/28 16:22:52 [info] 3615#3615: Using 131072KiB of shared memory for nchan in /etc/nginx/nginx.conf:161
ter process /usr/sbin/nginx -c /etc/nginx/nginx.conf: ./nchan-1.3.6/src/store/memory/memstore.c:705: nchan_store_init_worker: Assertion `procslot_found == 1' failed.
2023/06/28 16:22:52 [alert] 8504#8504: worker process 3618 exited on signal 6
2023/06/28 16:22:52 [alert] 8504#8504: shared memory zone "memstore" was locked by 3618
ter process /usr/sbin/nginx -c /etc/nginx/nginx.conf: ./nchan-1.3.6/src/store/memory/memstore.c:705: nchan_store_init_worker: Assertion `procslot_found == 1' failed.
2023/06/28 16:22:52 [alert] 8504#8504: worker process 3628 exited on signal 6
2023/06/28 16:22:52 [alert] 8504#8504: shared memory zone "memstore" was locked by 3628
ter process /usr/sbin/nginx -c /etc/nginx/nginx.conf: ./nchan-1.3.6/src/store/memory/memstore.c:705: nchan_store_init_worker: Assertion `procslot_found == 1' failed.
2023/06/28 16:22:52 [alert] 8504#8504: worker process 3629 exited on signal 6
2023/06/28 16:22:52 [alert] 8504#8504: shared memory zone "memstore" was locked by 3629
ter process /usr/sbin/nginx -c /etc/nginx/nginx.conf: ./nchan-1.3.6/src/store/memory/memstore.c:705: nchan_store_init_worker: Assertion `procslot_found == 1' failed.
2023/06/28 16:22:52 [alert] 8504#8504: worker process 3631 exited on signal 6
2023/06/28 16:22:52 [alert] 8504#8504: shared memory zone "memstore" was locked by 3631
ter process /usr/sbin/nginx -c /etc/nginx/nginx.conf: ./nchan-1.3.6/src/store/memory/memstore.c:705: nchan_store_init_worker: Assertion `procslot_found == 1' failed.
2023/06/28 16:22:52 [alert] 8504#8504: worker process 3632 exited on signal 6
2023/06/28 16:22:52 [alert] 8504#8504: shared memory zone "memstore" was locked by 3632
ter process /usr/sbin/nginx -c /etc/nginx/nginx.conf: ./nchan-1.3.6/src/store/memory/memstore.c:705: nchan_store_init_worker: Assertion `procslot_found == 1' failed.

Edited June 28, 2023 by Shine

[ljm42]

6 hours ago, Shine said:

This is the error information from my nginx.

 

Sorry you are having trouble, please start your own thread and upload your full diagnostics. It gets very confusing trying to respond to multiple issues in the same thread.

[Squid]

On 6/26/2023 at 6:12 AM, Dyon said:

I am on Unraid 6.12.1. I also am unable to access the WebUI after some time.

Can you retry this with a stock "go" file (/config/go on the flash drive).

 

You're installing a ton of extra packages, and also doing a hack to relocate the docker logs

[Dyon]

1 hour ago, Squid said:

Can you retry this with a stock "go" file (/config/go on the flash drive).

 

You're installing a ton of extra packages, and also doing a hack to relocate the docker logs

 

Will run a clean go file later. I honestly don't expect that the extra packages impact the WebUI personally. I haven't changed anything in that file for about 2 years honestly.

 

The docker logs hack is an older version of this below, didn't even know that I still had that active or why I needed this. I am currently of a big upload via FileZilla, so I can't sadly interrupt that to tes if a clean go file fixes it.

 

 

 

[ljm42]

On 6/28/2023 at 12:21 PM, Dyon said:

Will run a clean go file later. I honestly don't expect that the extra packages impact the WebUI personally. I haven't changed anything in that file for about 2 years honestly.

 

If extra packages are needed, you should create an "extra" directory on the flash drive and put the packages there, then Unraid will install them as it boots and you don't have to tweak the go script.

 

Having said that, adding packages like this on your own is not really a good idea. You need to check for updates on a regular basis and keep the files current to prevent problems.

 

Whatever you are using these packages for, it would be better to keep the base OS as stock as possible and find a Docker container for this functionality

[Dyon]

On 6/30/2023 at 6:28 PM, ljm42 said:

 

If extra packages are needed, you should create an "extra" directory on the flash drive and put the packages there, then Unraid will install them as it boots and you don't have to tweak the go script.

 

Having said that, adding packages like this on your own is not really a good idea. You need to check for updates on a regular basis and keep the files current to prevent problems.

 

Whatever you are using these packages for, it would be better to keep the base OS as stock as possible and find a Docker container for this functionality

Thanks for the tip about the `extra` directory. I know using Docker etc is better. But I ain't gonna install a Docker only to run `nmap` for something or when I need to use `ffmpeg` to quickly convert a file.

Nevertheless, the problem still occurs with a clean `go` file. (cc @Squid )

Still need to kill -9 nginx and start it again for it to work. Also running 6.12.2 now. Same problem.

nginx-error.log dyon-unraid-diagnostics-20190307-1931.zip

[ljm42]

These diags are from 2019

 

Also, in case I wasn't clear, while tracking down this problem you should not have anything in the extra folder either. We need the system to boot without old packages potentially causing issues.

[Dyon]

On 7/3/2023 at 6:47 PM, ljm42 said:

These diags are from 2019

 

Also, in case I wasn't clear, while tracking down this problem you should not have anything in the extra folder either. We need the system to boot without old packages potentially causing issues.

 

My bad, the sorting on my filesystem was sorted the wrong way around. Attached are the proper diagnostics. I naturally did a clean boot without the packages installed.

dyon-unraid-diagnostics-20230702-2131.zip

[ljm42]

We are working on a fix for nginx with IPv6 but for now you should go to Settings -> Network Settings and change eth0 to "IPv4 only" if that works in your environment.

 

Also, please see my comment here about browsers potentially causing issues with backgrounded tabs:

https://forums.unraid.net/bug-reports/stable-releases/612-unraid-webui-stop-responding-then-nginx-crash-r2451/page/3/?tab=comments#comment-25245

 

[ProZac]

Thank you for this post, saved my night hehe. Had same issue. Ipv6 might be the bad egg here too, kill and restart nginx from ssh solved the dead gui and then I disabled Ipv6. Hopefully the problem don't come back (2nd time now since 6.12.0 update)

[xsk]

i have same issue, i just disabled ipv6 and let's wait to see.

[nukecat]

I too facing this similar issue,webui no response,syslog full of nginx errors.

But everything else runs ok,containers work fine,qbittorrent still seeding ,ssh service sitll works etc.

The 'diagnostics' commnad did not work though just hang there forever.

I manage to restart nginx service by first 'kill -9 ${nginx_pid}' and start it again.

Here is my diagnostics.

 

endurance-diagnostics-20230709-1417.zip

[JorgeB]

3 hours ago, nukecat said:

I too facing this similar issue,webui no response,syslog full of nginx errors.

Try disabling IPv6.