Server Inaccessible via Network After Replacing Flash Drive, then Works When Swapping Back

[draconastar]

I recently started getting Alerts in Unraid that my flash drive was no longer in read-write mode.  After following documentation and trying to repair it (to no avail, all utilities claimed it was in read-only mode and wouldn't fix it), I eventually gave up and decided to replace it with a better flash drive.

I went through all of the documented steps to swap drives: make a zip backup, download and run the USB creation tool, select my zip as the backup and write to the new disk, then power down the server, plug my new flash in and turn it on.  That seems to work fine, but my server is no longer accessible over the network after I do that. 

I know it's alive and kicking, because I can plug a monitor and keyboard in and have access to the Unraid command line as soon as I do.  No boot issues that I can see, I got into the operating system just fine.  It has an IP address assigned to it, and I can ping it from a command line on a separate machine in my network.  But any time I try to browse to the web GUI, I'm presented with "Unable to connect", and developer tools show me "NS_ERROR_CONNECTION_REFUSED" for the request's transfer state.  This is regardless of whether I include https or not, or if I navigate directly to the IP address.  All variants result in a refused connection.

Oddly enough, when I swap back to the old flash, everything works fine.  GUI boots up with no issue, via direct IP or my local DNS address.  This makes it seem like it's some issue with the configuration that was copied over, or perhaps not copied over?

Basic steps that I can follow to reproduce:
- Create new flash drive using documented steps
- Shut down server, plug in new flash drive, restart server

- Plug in monitor and keyboard directly, see that Unraid command line login is waiting for me.  Able to log in and run commands.
- Navigate to http://atlas.local (my server's name).  Connection refused.
- Navigate to http://{myServerIp}.  Connection refused.

- Navigate to https variants of both of the above, https://atlas.local and https://{myServerIp}.  Connection refused.
- Navigate to registered external address for my server (i.e. https://myserver.com).  CloudFlare page showing that host is down.
- Shut down the server, swap back to the old flash drive, restart server

- Plugged in monitor and keyboard still show Unraid command line login, as before.
- Navigate to http://atlas.local.  Web GUI works.
- Navigate to http://{myServerIp}.  See my NPM entry page.
- Navigate to https variants of both.  Both the Web GUI and NPM page still load.
- Navigate to registered external address.  Authelia login loads correctly (I have this guarding external network traffic)

What am I missing here?  The fact that the old flash gets everything working again makes me feel like it's a configuration issue, I just have no idea what that could be, or why the USB tool would complete its write operation without having copied everything.

I've attached the diagnostic zip that I can pull from my server with the old flash loaded, so not sure how useful that is in this scenario, but I'm including it just in case.

 

atlas-diagnostics-20240620-1732.zip

Edited June 20 by draconastar
Changing GUI to command line to avoid confusion

[JorgeB]

Try disabling SSL:

 

use_ssl no

 

[draconastar]

9 hours ago, JorgeB said:

Try disabling SSL:

 

use_ssl no

 

This didn't work.  I set USE_SSL to "no", rebooted the server and tried accessing the GUI again.  Same result - http doesn't work, https doesn't work, doesn't matter if it's DNS name or IP address.

But that kind of makes sense, right?  When everything is functional, I'd expect to have http work when use SSL is no, and to have https work when it's yes (assuming I've got the cert set up, which again, it was working before).

[JorgeB]

Try booting with a different flash drive using a stock install, that will confirm if the problem is the /config or an external issue, if it's the /config, you can redo the flash drive, backup the current one first and then redo it and just restore the bare minimum, like the key, super.dat and the pools folder for the assignments, also copy the docker user templates folder, if all works you can then reconfigure the server or try restoring a few config files at a time from the backup to see if you can find the culprit.

[draconastar]

Well, it seems I am even further up the creek without a paddle now.  The old flash drive that I am trying to replace seems to have finally failed enough that it's no longer usable.  BIOS doesn't see it as a bootable device, even though it still has all of the files on it, and the bootable script won't work because the drive is seemingly permanently write-protected and so it fails to create the files that it wants to.  That has rendered my server unusable - old flash won't boot, new flash will boot but I can't access anything other than plugging into the machine directly.

Scratch that, seems like that was a temporary issue.  Got the drive booting again, but this makes me nervous it's going to fail very soon.

 

I did manage to copy all of the files directly off of the drive just in case that would be useful, before I used the USB creation tool.  Are you suggesting that I manually copy files from the old to the new to see if that fixes things?

Edited June 21 by draconastar

[JorgeB]

First try with a stock install, to make sure that works.

[draconastar]

Apologies for the delay on this... I had some unrelated home network issues that I had to solve before I could jump back into this one.

 

I acquired a fresh flash drive, installed stock Unraid on it and tested it on my server.  It works exactly as I expect - I could navigate to tower.local just fine, I was presented with the account setup screen, etc. 

 

That points to /config, I suppose, or something about how the USB Creation Tool sets up a new flash drive from a backup.  Clearly something about it is different, as I'm still getting the same "Unable to load the web page" errors when trying to navigate to any of its interfaces.  Again, I can see that it is being assigned the correct IP address, and I can ping the machine.  It just won't load any web pages.

 

So my current state is:

• Original flash drive:  works, but is dying
• New flash drive with stock install:  works.  can navigate to tower.local and the GUI loads
• New flash drive with backup:  does not work.  navigating to tower.local or any other interface gives me NS_ERROR_CONNECTION_REFUSED

 

I suppose this is where I would start taking your suggestion to restore the bare minimum to the new drive (the *old* new drive...).  Can I trouble you for some pointers on how I should go about doing this?  What files should I start with, and which ones should I copy over next as I'm testing?  Am I stopping the server, plugging drive into a different machine, copying files, then plugging it back in and restarting for each file I try?

[JonathanM]

Have you manually copied the entire config folder from the original to the new one that is working? All I see is references to USB creator.

[draconastar]

I have not.  My understanding is that the USB creation tool is the recommended way to replace a flash drive using the zip backup of the original, but that's just what I've gotten from documentation.

 

I'll hock the entire /config folder onto the new one and see if that changes anything.

[draconastar]

22 hours ago, JonathanM said:

Have you manually copied the entire config folder from the original to the new one that is working? All I see is references to USB creator.

This did not work either.  I copied the entire /config folder from the original still-working drive to the new one, making sure to delete the entire folder on the new one first to ensure there are no file replacement issues.  I have the exact same result - the server boots up and I can see it is handed the correct IP address on its startup report, but I'm unable to access it.  As soon as I plug the original drive back in, it works again.

I'm really struggling to understand what's going on here, especially given that it seemingly has the right IP address.  It's still giving me these NS_ERROR_CONNECTION_REFUSED messages in browser console.

[JorgeB]

This means there's a problem with the current config, backup the current flash drive first and then redo it with a stock install and just restore the bare minimum, like the key, super.dat and the pools folder for the assignments, also copy the docker user templates folder, if all works you can then reconfigure the server or try restoring a few config files at a time from the backup to see if you can find the culprit.