WebGUI not responding

[FreeMan]

This is the second time I've had this happen in the last week. I recently upgraded both my servers to 6.8.0 and first my primary server and now my backup server have given me this error when attempting to reach the WebGUI:

This site can’t be reached
backup took too long to respond.
Try:

Checking the connection
Checking the proxy and the firewall
Running Windows Network Diagnostics
ERR_CONNECTION_TIMED_OUT

I did have the GUI open in my browser, I've had a tab open to each browsers GUI for years, just refreshing it when I want to take a quick look. For my main server, it wouldn't powerdown from an SSH connection, nor from a short press on the power button. For my backup server, it did respond to a powerdown command from SSH. After it rebooted (because I completely forgot to do it before I rebooted), it also responded very quickly to diagnostics from SSH, but the WebUI is still unreachable.

 

I get that I shouldn't have a browser window open to the server at all times. I guess this is something new with 6.8, because I've had one open almost continuously since my first unRAID build on 4.5 (or so).

 

Any practical solutions on how to reach the GUI on my freshly rebooted backup server since it's not responding now?

backup-diagnostics-20191228-0947.zip

[FreeMan]

I had to shut the server down for a bit. When I brought it back up, I still cannot reach the WebGUI.

 

I can connect via PuTTY and it responds to a ping. My rsync backups will write to it with no issue. I can browse the server with Win Explorer (which is how I go the diagnostics file attached), I can even connect to it from my phone using ControlR via the ZeroTier docker. I just cannot connect to the web server.

 

Any suggestions?

backup-diagnostics-20191228-1728.zip

Edited December 28, 2019 by FreeMan

[Squid]

Try editing USE_SSL in /config/ident.cfg from "auto" to instead be "no" and see if that makes a difference (reboot will be required).  And if it does, then within the gui change it back to auto in management settings

[FreeMan]

24 minutes ago, Squid said:

Try editing USE_SSL in /config/ident.cfg from "auto" to instead be "no" and see if that makes a difference (reboot will be required).  And if it does, then within the gui change it back to auto in management settings

Edited the file (used Notepad++ on my Win10 machine - it is set to preserve line endings, so I did not overwrite LF with CRLF) and rebooted. I can still access it via SMB from Win10 and can SSH in, but still no GUI, but now I can't ping it and it's not responding to ControlR.

 

Fresh diagnostics (just to be 100% that I didn't botch the file edit - nothing looked weird on a 'cat ident.cfg' from a command line).

backup-diagnostics-20191228-1811.zip

[Squid]

1 minute ago, FreeMan said:

it's not responding to ControlR.

Yeah, that's expected since you were using SSL.  You can set it back to auto. 

 

Have you tried Safe Mode yet?

 

Since you're using IPv6 exclusively, I'm not really sure where to go from here.  The only other thing I can suggest is to delete the contents of ssh / ssl from /config on the flash (they'll get regenerated automatically), but from looking at your go file, you're adding in your own keys.

[FreeMan]

I'm not sure why it's on IPv6 exclusively, TBH. I didn't intentionally set it that way, and was going to ask how to enable IPv4. 

 

6 minutes ago, Squid said:

Yeah, that's expected since you were using SSL

The web browser has always come up telling me the site is 'not secure'. Enabling HTTPS was going to be a follow up question. 

 

The keys are so my two servers (this is to be my offsite backup) can talk to each other. I had it set up and working via ZeroTier, but with the release of the WireGuard plugin, I'm switching to that. I know which ones I generated, should I delete all the rest, set it back to "auto" and try again?

[Squid]

Set in the network.cfg again, USE_DHCP6[0] to be "no"  Then, ethtool should show an IP in the diagnostics (previously it was simply showing the IPv6), and with luck you'll be able to just type in the appropriate IP and go from there

[FreeMan]

29 minutes ago, Squid said:

Set in the network.cfg again, USE_DHCP6[0] to be "no"

Updated network.cfg:  USE_DHCP6[0] = "no"

Left ident.cfg: USE_SSL="no"

 

Still no GUI access. I believe it would respond to a ping prior to the network.cfg update - I had tried pinging by name and by IP. It has responded every time I've tried via name, but it won't respond via IP. I had it set with a static IP address, but changed that back to DHCP before shipping the server to the off-site location. (I brought it back to do the initial backup instead of chewing up my bandwidth allocation and the (non-commercial) host site's.) Since then it's responded to a ping by name, but it only reports the IPv6 address, not the IPv4. I tried pinging with the last known IPv4 addresss - it seems it's not home there, but it is responding now by name.

 

I have not tried safe mode, will do that next. 

 

Any other suggestions?

[FreeMan]

I just booted into GUI safe mode. I logged in (managed to get the password right the first time!  ) and I do get a full GUI and it's reporting the same IP address that it had before I sent it off-site, so I'm not sure why it wasn't responding to a ping via IP... Maybe it didn't like just being a number?  

 

I don't have a mouse plugged into the server, so I'm a bit limited to what I can do, though I'm pretty sure I can scrounge up a mouse from somewhere.

 

Once again, the server will respond to a ping from my desktop when I call by name, but not by number. I can also access shares from my Win10 machine. (Never had to use safe mode before, so I'm not 100% certain what will/won't work. I guess everything but Dockers & plugins, like it says on the tin, eh?)

 

Recommendation for next steps?

 

I found a mouse, plugged it in, ran diagnostics and the WebUI froze up on the console. I SSH'd in and ran diagnostics from the command line.

backup-diagnostics-20191228-1928.zip

[FreeMan]

Anyone have any additional thoughts or suggestions? The WebGUI has failed to start after several reboots.

 

It did come up with the GUI on the console when booting safe mode w/GUI, but it hung.

It comes up with the GUI on the console when booting in regular mode w/GUI. I'm seeing no hang yet - I kicked off the Mover (it needed it).

Just for kicks, I refreshed the browser tab on my desktop, and I was presented with the login screen for this server! I'm able to log in and move around the GUI, at least for now.

 

Fresh diagnostics (including a drive error that's just popped up - I'll post about that separately) attached.

 

I'm still open to input or suggestions on how to resolve this so that I don't have to fiddle to get it to work. I'd like to resolve in the next day or two before my son takes it back to college where it'll serve as my off-site backup.

backup-diagnostics-20191229-1224.zip

[JorgeB]

Dos the GUI work with a new trial flashdrive? Or try redoing current one, save config folder than restore.

[FreeMan]

9 hours ago, johnnie.black said:

Or try redoing current one, save config folder than restore.

I'm heading down this route for now. Making a backup of my entire flash drive to my SSD on my desktop Win10 machine and this:

That speed! Granted, it's USB2, but still!!! 

 

It did pick up to MB/s shortly after the screen grab, but I'll also run a full chkdsk on the drive while I'm at it. Maybe the flash is dying.

[JorgeB]

15 minutes ago, FreeMan said:

Maybe the flash is dying

Could be

[FreeMan]

Win10's "Tools | Error Checking" reported no issues. Just for giggles, I ran HD Tune Pros' Error scan on it too - no reported issues.

 

It reports it as a "Patriot Memory 7gB" drive (marketing size: 8GB) but doesn't give any model number. It's a tiny little thing, with no writing on the drive itself, that's almost impossible to get out of the USB plug. Perfect for leaving in a laptop  except I'm not using it in a laptop . I figured this would be good for off-site use since it's out of the way and nearly unnoticable so less likely to be messed with by my son's college roommates. (Note, it is plugged directly into a MoBo port, not a front port.)

 

I'm copying my entire config directory back and will be rebooting shortly.

 

While waiting for the HD Tune error scan to run, I was thinking... a dying flash drive should not impact Unraid operations except at boot time, should it? Everything's loaded into memory once the OS has booted and it almost never writes anything back to the drive except a "clean powerdown" flag (and config changes, which I haven't have been making as I've set up WireGuard...) when I shut it down, right?

Edited December 30, 2019 by FreeMan

[FreeMan]

Nope, still no dice. I recreated the flash drive as @johnnie.black suggested, booted and got no GUI. I undid the USE_SSL change that @Squid recommended, just to get back toward my "standard" setup. Rebooted, selected the "with GUI" option on the console when I booted and I get GUI on the console and via the network.

 

That's a workable solution, but it's certainly not ideal nor where it was just a couple of days ago. Fresh diagnostics, just for fun.

 

Any other ideas/suggestions?

backup-diagnostics-20191230-1422.zip

[JorgeB]

And if you use a completely new flash drive without current config? Just to confirm is not related to that.

[FreeMan]

1 hour ago, johnnie.black said:

And if you use a completely new flash drive without current config?

Started with a fresh install on a totally different USB stick. It came up with no problem with the GUI - must be something config related.

 

The only changes I made recently was the installation of the WireGuard plugin, and it continued to work just fine for several days before the GUI disappeared on the 28th when I started this thread.

[JorgeB]

You can start over with your flash drive, initially just copy super.dat (disk assignments) and the key from the backup, then reconfigure the server or start copying other config files a few a at time until you find the culprit.

[FreeMan]

Adding WireGuard was the most recent setup change I made to the server.

I updated WireGuard to the 2019.12.30 release, it still wouldn't give me a GUI without booting the server in GUI mode.

 

Interestingly, booting the server into console GUI mode is not sufficient - I actually have to log in at the terminal in order to be served the login page at another machine on the network. I thought this was the case, but that machine boots so slowly and takes so long to time-out that I was getting bored and doing something else and forgot where I was. I definitely confirmed it just now, though.

 

Next trouble shooting steps taken:

1. Uninstalled WireGuard, rebooted, no  GUI
2. Attempted to uninstall ZeroTier docker, the green arrows spun and spun and spun. Finally the browser timed out and gave me the "backup took too long to respond." error message.

I was able to login via SSH and grab attached diagnostics.

backup-diagnostics-20191231-1332.zip

Edited December 31, 2019 by FreeMan

[FreeMan]

Shortly after getting the diagnostics, I issued "powerdown" from the command prompt. Booted in GUI mode & logged in at the console. After ~30 minutes, I still couldn't get access to the GUI from my desktop machine.

 

I've made notes on all the plugins and dockers I've got installed. I'm going to start with a fresh OS install on my USB stick with the only change being the server name. I'll add in one thing at a time until it breaks or decides to work.

 

*Note: this is as much for my note taking purposes as anything else. If, however, anyone cares to chime in with something that they see, please feel free!

 

Note #2: I'm not taking a new backup of the flash drive before I start this - I've already got a backup from before I started messing around with things, and I'm considering that my "gold standard" of where I'd like to be. Though, there are a few things on that server that it really doesn't need that I will probably only install for testing purposes.

Edited December 31, 2019 by FreeMan

[FreeMan]

I reinstalled each plugin one at a time, followed by each docker one at a time, with a reboot between each install (only 6 minutes to reboot this server...). I got a GUI at each reboot without having to boot into GUI mode on the server. Not really sure what happened but it seems to have been resolved by rebuilding everything.

 

The one thing I have not done was reinstall all the individual CLI tools I had turned on in Nerd Pack. I'm not really sure I need any of them, so I'm going to leave everything (Nerd Pack) uninstalled for now.

[FreeMan]

Sadly, this is back. 

 

Now, sometimes, I'll attempt to log in, the spinner icon will stop, and it will tell me that the server is taking too long to respond. Then after a few moments, the spinner (running Brave) will start again and the Dashboard will load.

 

I just replaced 2 2TB drives with one 4TB drive - I had to do that from the GUI on the console. After booting, I was able to connect for a bit and edited my shares to remove the ban on the two drives that I'd pulled (I'd previously emptied them). I got through all of them, clicked the "Main" link and the GUI quit responding.

 

Here are the latest diagnostics. Any thoughts at all on this? Any suggestions other than rebuilding the USB stick again? That can't be my fallback solution - this is going to be an off-site backup machine and I have to be able to access the GUI (using WireGuard) to do the remote management. My son could do things locally, but he'd still need to be able to access the GUI from the local LAN.

backup-diagnostics-20200115-1857.zip

[FreeMan]

Just for fun, the GUI decided to respond! I grabbed new diagnostics just to see if there's anything different.

backup-diagnostics-20200115-1930.zip

[JorgeB]

Don't see anything out of the ordinary on the logs.

[FreeMan]

@limetech @jonp - hate to ping you guys directly, but this one seems to have stumped our otherwise excellent tech support team. Do you guys have any thoughts on what might be going on?

 

Sent from my moto g(7) using Tapatalk