WebGUI after reboot no longer loads/connects

[CoZ]

Not sure if this is the correct section for this - so I apologize in advance if it is not. 

I had a power outage the other day that was long enough to have my UPS initiate a graceful shutdown of my tower and router and modem.  The system has been rock solid for years - until this recent power outage.  I can no longer access the WebGUI. 

 

At first, I thought it was a docker Hdhrdvr causing this issue as it and only two others were in Host mode (Plex and Home Assistant).  The reason why I thought it was that docker was due to the fact that every time I tried to access my UnRaid dashboard, the HDHomeRun DVR configuration page would appear instead of the WebGUI.

 

All of my other dockers I can connect to fine <IP:Port> and even the dockers that use VNC show up without issue.  I'm able to SSH/PuTTy into the terminal without issue as well.

 

I've run

nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

then

nginx -s reload:

nginx: [error] invalid PID number "" in "/var/run/nginx.pid"

 

I also tried:

nginx: [emerg] bind() to unix:/var/run/nginx.socket failed (98: Address already in use)

Message from syslogd@CozsNAS at Jul  5 13:24:15 ...
 nginx: 2022/07/05 13:24:15 [emerg] 22438#22438: bind() to unix:/var/run/nginx.socket failed (98: Address already in use)
nginx: [emerg] bind() to unix:/var/run/nginx.socket failed (98: Address already in use)

Message from syslogd@CozsNAS at Jul  5 13:24:16 ...
 nginx: 2022/07/05 13:24:15 [emerg] 22438#22438: bind() to unix:/var/run/nginx.socket failed (98: Address already in use)
nginx: [emerg] bind() to unix:/var/run/nginx.socket failed (98: Address already in use)

Message from syslogd@CozsNAS at Jul  5 13:24:16 ...
 nginx: 2022/07/05 13:24:15 [emerg] 22438#22438: bind() to unix:/var/run/nginx.socket failed (98: Address already in use)
nginx: [emerg] bind() to unix:/var/run/nginx.socket failed (98: Address already in use)

Message from syslogd@CozsNAS at Jul  5 13:24:17 ...
 nginx: 2022/07/05 13:24:15 [emerg] 22438#22438: bind() to unix:/var/run/nginx.socket failed (98: Address already in use)
nginx: [emerg] bind() to unix:/var/run/nginx.socket failed (98: Address already in use)

Message from syslogd@CozsNAS at Jul  5 13:24:17 ...
 nginx: 2022/07/05 13:24:15 [emerg] 22438#22438: bind() to unix:/var/run/nginx.socket failed (98: Address already in use)
nginx: [emerg] still could not bind()

Message from syslogd@CozsNAS at Jul  5 13:24:18 ...
 nginx: 2022/07/05 13:24:15 [emerg] 22438#22438: still could not bind()

 

running netstat -nalp | grep -i 'nginx: master' returned nothing what-so-ever

 

Then trying:

netstat -nalp | grep -E 'tcp\s.*(:80|:443).*LISTEN\s'
tcp        0      0 0.0.0.0:8091            0.0.0.0:*               LISTEN      22673/docker-proxy
tcp        0      0 0.0.0.0:8010            0.0.0.0:*               LISTEN      14385/docker-proxy
tcp        0      0 0.0.0.0:8080            0.0.0.0:*               LISTEN      13325/docker-proxy
tcp        0      0 0.0.0.0:8086            0.0.0.0:*               LISTEN      13733/docker-proxy
tcp        0      0 0.0.0.0:8089            0.0.0.0:*               LISTEN      17567/docker-proxy
tcp        0      0 0.0.0.0:8090            0.0.0.0:*               LISTEN      13311/docker-proxy

 

 

I've pinged TOWERNAME from my WindowsPC and it resolves with the static assigned IP address

 

I managed to figure out how to disable the hdrhomerun docker from auto starting and rebooting the tower. Since disabling the hdhomerun docker and rebooting the server and typing in nginx  nothing happens.  Entering nginx -s reload no errors show up either.   Entering in

docker network inspect host:

"Name": "host",
        "Id": "937fbdeca1b4fcaf5d1930f1fbbc740461b25c5699070a79cfc49b6755722d24",
        "Created": "2020-12-09T11:07:08.920604571-05:00",
        "Scope": "local",
        "Driver": "host",
        "EnableIPv6": false,
        "IPAM": {
            "Driver": "default",
            "Options": null,
            "Config": []
        },
        "Internal": false,
        "Attachable": false,
        "Ingress": false,
        "ConfigFrom": {
            "Network": ""
        },
        "ConfigOnly": false,
        "Containers": {
            "79c3c4cbb2fa30b3a9bec0a621e56e2f51032c120096c115bdead6aa7a881c25": {
                "Name": "home-assistant",
                "EndpointID": "01d6247f536f7cfa0a70cb2314197ec344038e9e975f19c3f862946717ebf5b                   f",
                "MacAddress": "",
                "IPv4Address": "",
                "IPv6Address": ""
            },
            "d236e30221155a37809b821e4dfb64aa37ced3ee1834a6f950ffd9940ed747c3": {
                "Name": "PlexMediaServer",
                "EndpointID": "1ceb127b22c6a92f8975b57f9695382d222b383e8cd472290f38179a764d11c                   4",
                "MacAddress": "",
                "IPv4Address": "",
                "IPv6Address": ""
            }
        },
        "Options": {},
        "Labels": {}

 

shows that the HDHomeRun docker is no longer starting on reboot.

 

I am still unable to access the GUI.  

 

Looking at the "MyServers" plugin on the forums, it reports that my Tower is online.  Trying to access it from the hyperlink on that webpage also just simply timesout.

 

I've exhausted everyone's knowledge and my own trying to figure out what could possibly be wrong now.  Some folks on /r/UnRaid were helping but they have since gone silent - I guess they have run out of ideas as to what to try?   So now I turn to you fine folk.

 

If it makes any difference I am running an Ubiquiti UDM router (UniFi OS1.12.22 / Network7.1.66)  in case the power outage and subsequent reboot of everything might have an effect on something, somewhere.  I've also power cycled the cable modem and rebooted the router a few times as a just in case type of thing.

 

I really could use some help here.  Thank you, in advance.

[CoZ]

Replying to this for posterity’s sake.  The issue has now been resolved.  It was, indeed the HDHomeRun Docker causing the issue.  It took over port80 when the author of said docker updated it due to Silicon Dust releasing a GUI for the DVR hardware.  Unforseen consequence he has stated on GitHub after I opened an issue.

 

In order to access the webgui since the docker took over port 80 the following commands were used:

 

nano -w /boot/config/ident.cfg

then looking for the PORT line entry and changing it to a random number and saving the file.

/etc/rc.d/rc.nginx restart

 

Then typing in the IP:Newly Assigned Port of the UnRaid server into Edge / Firefox.  Going into the Docker tab, stopping the offending Docker and then unchecking "autostart"

 

Reverse the commands above, changing the PORT line entry back to the default of 80 and rebooting the server allowed access once again.

[dfwjensen]

Thanks for the posterity post!
I am having the same problem, but that didn't work. I tried two different random ports, but still can't connect.

Also tried disabling the docker start in the config.

Can access server via ssh, can even access some shared folders over the network, but no GUI access. 

Any other suggestions?

[trurl]

Attach Diagnostics to your NEXT post in this thread 

[dfwjensen]

3 hours ago, trurl said:

Attach Diagnostics to your NEXT post in this thread 

Sorry for not including this earlier.

unraidserver-diagnostics-20230919-0941.zip

[JorgeB]

19 minutes ago, dfwjensen said:

Sorry for not including this earlier.

Sep 19 05:23:16 UnraidServer nginx: 2023/09/19 05:23:16 [emerg] 9345#9345: bind() to unix:/var/run/nginx.socket failed (98: Address already in use)

See here:

https://forums.unraid.net/topic/140510-complete-webui-breakage-after-6120-update/?do=findComment&comment=1271735

 

[ljm42]

5 hours ago, dfwjensen said:

I tried two different random ports, but still can't connect.

 

According to your diags the ports nginx is trying to use are:

PORT="80"
PORTSSL="443"

If you changed them to something else in config/ident.cfg, be sure to include the port when trying to access the webgui.
 

 

 

5 hours ago, dfwjensen said:

Also tried disabling the docker start in the config.

 

Can you give more info on what you did here? Until you reboot, changing this setting in config/docker.cfg won't have any effect.

 

To stop docker without rebooting, you can type this via SSH:

/etc/rc.d/rc.docker stop

but before doing that...

 

 

I think nginx is not stopping properly, which means it can't restart properly.  Please connect via SSH and type:

/etc/rc.d/rc.nginx term

followed by

/etc/rc.d/rc.nginx start

and paste the output of both commands back here.

 

If this doesn't solve the problem, please provide updated diagnostics as well (type "diagnostics" and then grab the latest zip from the logs directory on the flash drive)

 

[dfwjensen]

 

1 hour ago, ljm42 said:

According to your diags the ports nginx is trying to use are:

PORT="80"
PORTSSL="443"

If you changed them to something else in config/ident.cfg, be sure to include the port when trying to access the webgui.

Thank you so much for your support! Really appreciated!

 

I tried both port 88 and 247. Then rebooted the server and tried to access via 192.168.4.41:88 or :247 with no luck. 

So I changed it back to 80.

 

1 hour ago, ljm42 said:

Can you give more info on what you did here? Until you reboot, changing this setting in config/docker.cfg won't have any effect.

For this I changed the /boot/config/docker.cfg to docker_enabled ="no". Then rebooted. Still no effect.

 

I had tried the restart a number of times with no effect.

I though I had uploaded by latest diagnostics last time. I will try again.

 

unraidserver-diagnostics-20230919-1309.zip

Edited September 19, 2023 by dfwjensen
change diagnostics to .zip file

[itimpi]

6 minutes ago, dfwjensen said:

I though I had uploaded by latest diagnostics last time. I will try again.

These should be a single zip file (which makes it easier to navigate when troubleshooting) rather than all the separate files you posted.   Maybe you have your browser set to automatically open zip files to show its contents?

[ljm42]

Ah, it was not clear that you were rebooting between attempts. OK that is good. 

 

Please edit your message to delete all those individual files and upload the zip as a single file instead. I understand Macs can make this type of thing difficult.

 

41 minutes ago, dfwjensen said:

I tried both port 88 and 247. Then rebooted the server and tried to access via 192.168.4.41:88 or :247 with no luck. 

 

If you are choosing random ports, please use something higher than 1000 and lower than 64,000, and make sure it isn't already in use by a Docker container.

 

41 minutes ago, dfwjensen said:

Then rebooted the server and tried to access via 192.168.4.41:88 or :247 with no luck. 

 

Based on that screenshot it looks like nginx is running fine. What error message do you get when you try to access the webgui?

 

Sometimes browsers will assume https, but you are trying to use http. So be sure you type "http" at the beginning of the address:

  http://192.168.4.41

or 

  http://192.168.4.41:port

in the browser's url bar

[dfwjensen]

Appreciate all the help with this.

Been using the server for a couple years with no issues, so this is throwing me a bit.

 

I tried port 1012, but still no change. Still getting an HTTP 500 error.

 

I replaced the long list of files on my previous post with a newly run diagnostics file (sorry about that - noob mistake)

 

49 minutes ago, ljm42 said:

Sometimes browsers will assume https, but you are trying to use http. So be sure you type "http" at the beginning of the address:

  http://192.168.4.41

or 

  http://192.168.4.41:port

in the browser's url bar

Yes. I was ensuring http://

Although I tried https:// just to be sure.

 

I have tried clear browser cache. I have a PC and have tried to access via both Chrome and Firefox.

 

The only change recently is I updated to the latest update of Unraid (can't remember the version.... 6.12.3?).

 

Appreciate all the feedback (and patience with me - a newcomer to forums).

[dfwjensen]

58 minutes ago, itimpi said:

These should be a single zip file (which makes it easier to navigate when troubleshooting) rather than all the separate files you posted.   Maybe you have your browser set to automatically open zip files to show its contents?

Nope. Can't blame anyone but me. Thought my previous upload attempt had been rejected as a .zip and assumed I could be helpful by supplying an unzipped version. I guess the classic case of assuming still holds true!  

[ljm42]

9 minutes ago, dfwjensen said:

Still getting an HTTP 500 error.

 

aha! a 500 error means that nginx is responding to requests.

 

via SSH, type

ls -al /var/log/nginx/

This will show how big the nginx error log is.

 

Then type:

cat /var/log/nginx/error.log

to see the error messages and paste the results back here. 

[dfwjensen]

Hope this helps?

[ljm42]

I am kind of stumped. So you are getting a 500 error but nothing is logged in syslog or errors.log.

 

Is Docker currently disabled? If not, please disable it and reboot. I just want to make sure there is not a Docker container responding on port 80.

[dfwjensen]

2 hours ago, ljm42 said:

Is Docker currently disabled? If not, please disable it and reboot. I just want to make sure there is not a Docker container responding on port 80.

I believe so. It has been for that last 5 or so reboots as per: /boot/config/docker.cfg

[dfwjensen]

Here is what I get when I just run nginx. Not sure if this is helpful or not?

Thanks for all your help.

Is there any path forward to recover my unraid server?

[ljm42]

15 hours ago, dfwjensen said:

Here is what I get when I just run nginx.

To clarify, what is the exact command you ran here?  Please run `/etc/rc.d/rc.nginx term` and then that command, to see if it is any different

[dfwjensen]

3 hours ago, ljm42 said:

To clarify, what is the exact command you ran here?

Just typed "nginx" to get the output.

 

3 hours ago, ljm42 said:

Please run `/etc/rc.d/rc.nginx term` and then that command, to see if it is any different

When I stop nginx and then type it I get nothing.

 

[dfwjensen]

Is there a straight forward way I can fix this by redoing the install?

I don't really want to lose my plex library, or have to redo anymore than I have to.

I figure since I can SSH into the server, I should be able to save most of the configuration?

Thanks for trying!

Edited September 23, 2023 by dfwjensen
added SSH line

[dfwjensen]

Last bump. Is there any way I can fix this? 
My server has been down for over two weeks, and I just don't know how to get it running again.

Should I try installing unraid on a new USB drive and copying over some config file(s) so I don't lose my data?

Appreciate any help out there....

[dfwjensen]

I tried a new install on a new USB stick. It worked fine. Copied by /boot/config/ directory over and the same problem. 

Just gave up and used the info in the config files to set up a my new USB stick and have the system up and running.

Thanks for all your attempts to help, but I am back online.

Cheers,

Dave

[ljm42]

Glad you are back up and running. I wonder what configuration file was the issue.

[dfwjensen]

21 hours ago, ljm42 said:

Glad you are back up and running. I wonder what configuration file was the issue.

Thanks! Me too! Just hope it doesn't happen again! 
Been using the unraid server reliably for a couple years, and just love it!

[yolo]

same problem and don't know what to do

in my case, the nginx service was not started after reboot, webui response 'ERR_CONNECTION_REFUSED'
then i start nginx by '/etc/rc.d/rc.nginx start', webui response '500 Internal Server Error'