Hello everyone,

 

Definitions

• PC = my main computer (1st machine)
• Server = the 2nd machine that has Unraid on it
• Everything is purely within the LAN network, both PC and Server are in the same house, i.e. both are connected to the walls in their respective rooms via an ethernet cable.

 

How the problem started:

1. Decided to give Ngninx a try since I wanted to see if I could get HTTPS working.
2. Installed Nginx docker, messed around with it a bit, seemed like I could not get it to work (could get it to make an SSL certificate via DuckDNS and set up some Proxy Hosts with duckdns subdomains, but everything staid HTTP when I clicked on the proxy host).
3. Basically had no idea what I was doing. Was experimenting with having Nginx Proxy Hosts direct to different ports etc. and figured I'd just trial and error things a bit while following some Youtube tutorials etc. to see how far I'd get. Basically that's how I got this far using Unraid over the past several months in general. I was not aware about this being potentially dangerous outside of the docker containers, in hindsight I guess I should have.
4. At some point I was also changing some ports in the different docker settings (as far as I know I did not create any overlapping ports, but who knows) and tried putting some dockers from "Bridge" to "br0" etc. (not having any deep understanding about the differences) but that did not seem to do much, in the end I just set everything back to "Bridge".
5. Installed Traefik docker in Unraid, but then decided against starting to mess around with that as well, so jumped back to Nginx.
6. All of a sudden I did not have LAN access to the server anymore (purely from messing around with Nginx as far as I could tell).
7. Strangely, I could still go to different tabs within the Nginx GUI itself (e.g. "Proxy Host" and "SSL" tabs) but all the other dockers (including Unraid GUI itself) were unavailable so I was basically stuck - I tried to walk my steps back in Nginx but I don't know at what stage everything broke and I don't remember when I changed what, as I had been messing with it on and off over the course of several hours.
8. Whenever I try to connect to the Server (or any of the dockers like Nextcloud or Jellyfin) from my PC via my PC browser (tested multiple browsers, using bookmarks to e.g. Nextcloud so nothing changed there either), both via "tower.local" or the "IP address", I get this message: 

   "This site can’t be reached

   192.168.0.100 [note: this is my local Unraid server IP] refused to connect.

   Try:

   Checking the connection

   Checking the proxy and the firewall

   ERR_CONNECTION_REFUSED"

 

Once the problem occured:

1. I still see the Server on its fixed LAN IP when logging into my modem, so that seems normal.
2. I can get into the Server via physically connecting a screen/keyboard/mouse and booting into GUI (it seems to actually start a Linux instace and automatically opened a Firefox browser showing the Unraid GUI in there - hopefully makes sense?).
3. The GUI in the Firefox browser on the server connects to the tower.local address in the adress bar.
4. When I open a second tab in that Firefox browser on the server I seemingly can use the IP address to get to the login screen as well.
5. However, I cannot get to the login screen via my PC anymore, neither to any of the dockers and SMB is also down (cannot get to my folders via windows explorer on PC).
6. I tried connecting via my PC browser both without the dockers running and after the dockers started. Booting the Server in Unraid safe mode also does not change anything it seems.

 

I think I managed to get the diagnostics file (see attachment).

 

When I open "servers.conf.txt" in the diagnostics file I see some reference to Nginx, but I don't know what to make of it. I just mention it here because it may be relevant for somebody who does know how to read it.

 

[edit: removed fluff]

 

Edited December 17, 20241 yr by UnraidC

Reporting back on some more troubleshooting I did in the meantime:

1. Uninstalled the Nginx docker. Did not change anything.
2. Reinstalled the Nginx docker, which gave errors because it as a standard uses port 80.
3. I changed the Nginx docker settings back to what it was before I removed it.
4. Strangely, when I then launched the Nginx GUI on the server, it just got me straight into Nginx, where it apparently had remembered the login credentials and still had the proxy host and SSL certificate?
5. In other words, removing the Nginx docker apparently did not remove everything related to it and as a result also did not solve the issue

 

[edit: apparently there was some misunderstanding here on my end because the settings of the dockers are saved separately in appdata, so when you remove a docker but don't remove the appdata, the next time you install the docker again it just loads its settings from the appdata again and it looks like nothing happened - clever design but I had no idea how it worked so to me it looked completely illogical when I wrote this post because I assumed that removing the docker is to remove everything including any settings, which in Unraid turned out to not be the case here even though I assume that Nginx related appdata on its own was not what was causing issues, but idk.]

Edited December 17, 20241 yr by UnraidC

May be easier to just redo the flash drive, backup the current one first and then recreate it using the USB tool and just restore the bare minimum, like the key, super.dat and the pools folder for the assignments, also copy the docker user templates folder, if all works you can then reconfigure the server or try restoring a few config files at a time from the backup to see if you can find the culprit.   

1 hour ago, JorgeB said:

May be easier to just redo the flash drive, backup the current one first and then recreate it using the USB tool and just restore the bare minimum, like the key, super.dat and the pools folder for the assignments, also copy the docker user templates folder, if all works you can then reconfigure the server or try restoring a few config files at a time from the backup to see if you can find the culprit.   

 

Thank you for your reply, I am a bit intimidated by what you are saying though as it seems to be a lot of things that I can individually mess up and I don't know what any of it means or what steps to take... I am really at absolute beginner level. If at all possible I'd like to start with the simplest possible solutions where I can cause the least amount of collateral damage.

 

1. Backup the flash drive -> you mean run the Appdata Backup plugin?
   1. I have 2 backup files (.zip) via Appdata Backup:
      1. "Backup A" from today (16 december) which I just manually did via the "Manual Backup" button in the plugin.
      2. "Backup B" from the 2nd of december (when everything was still working normally).
   2. Alternatively, do you mean a "Backup C" that I would need to create now using Unraid Connect.
      1. I found these instructions via https://docs.unraid.net/connect/flash-backup/: "Navigate to Settings → Management Access → Unraid Connect and under Flash backup click the Activate button. Wait for the activation and initial backup job to complete." However, I read there "Important: Backups from the flash device do not include the config/shadow or config/smbpasswd files. User accounts are preserved, but their passwords are not. This means that when you restore from backup, you will need to set passwords for your users including root. We also don't store any of your WireGuard keys". This makes me very nervous as I am afraid of getting locked out entirely / having to mess around with setting up SMB shares again
      2. Also, I had a look but I do not see any "Unraid Connect" options under "Settings → Management Access"?
   3. I see that within the Appadata Backup plugin there is also a "restore" tab, is there any point in trying that before going through the chain you mentioned above?
2. Recreate using the USB tool
   1. Are you referring to https://unraid.net/blog/new-usb-creator ?
   2. I found this description online: "You download the unraid USB creator. Download a zip copy of your flash drive backup from unraid connect. Plug new flash drive into the computer, open the USB creator, on the drop down select local file, point to the flash drive zip file. Select your USB drive then create." -> would this be the Appdata Backup file or the Unraid Connect file in your advice or are they essentially the same?
3. just restore the bare minimum, like the key, super.dat and the pools folder for the assignments -> I don't know what that means or how to do that, could you please elaborate or point me to some information?
4. also copy the docker user templates folder -> copy it to where?
5. if all works you can then reconfigure the server -> what should work before I reconfigure what/how on the server?
6. or try restoring a few config files at a time from the backup to see if you can find the culprit -> I don't know what the actual process is here to restore a few config files at a time? Which config files and what order and so on and finding the culprit would imply breaking it again?

 

 

Thank you for your help, your time and effort is much appreciated!

Edited December 16, 20241 yr by UnraidC

To backup the flash drive you just need to copy the complete /config folder.

 

2 hours ago, UnraidC said:

Recreate using the USB tool

Yes

 

2 hours ago, UnraidC said:

 I don't know what that means or how to do that, could you please elaborate or point me to some information?

Just need to restore those files from the earlier backup.

 

2 hours ago, UnraidC said:

also copy the docker user templates folder -> copy it to where?

To the same folder \config\plugins\dockerMan\templates-user

 

2 hours ago, UnraidC said:

what should work before I reconfigure what/how on the server?

What is not working now, in this case the GUI.

 

 

44 minutes ago, JorgeB said:

What is not working now, in this case the GUI.

 

I probably did not explain well: the GUI is working but only when I physically am connected to the server. The problem is that I cannot connect to the server from the PC anymore.

 

 

Do I understand it correctly that I should basically get the oldest appdata backup that I have (Backup B mentioned above), plug the Unraid USB in my PC and override the config folder with the config folder in the Backup B and that should restore things to normal?

Edited December 16, 20241 yr by UnraidC

1 hour ago, UnraidC said:

Do I understand it correctly that I should basically get the oldest appdata backup that I have

That's not what I recommend, and it won't help if the issue is the flash drive config, but it may if it's a misconfigured container.

Would the assumption be that the issue is likely with network.cfg on the USB drive? Is that something that is affected by Nginx?

 

I am pretty sure the issue originated from / is caused by whatever it is that Nginx is doing or changed, I would like to try to reverse whatever Nginx did first (and just removing the docker apparently is not enough) and just get rid of everything related to Nginx before moving on to more complex potential solutions?

 

Is there perhaps something to be found in the diagnostics files pointing to anything strange? When I open "servers.conf.txt" in the diagnostics file I see some reference to Nginx, but I don't know what to make of it.

Edited December 16, 20241 yr by UnraidC

Just some update, a few more hours down the line:

1) I noticed that when I change anything in the Nginx docker (e.g. setting the network from "host" to "br0") it gives an error message saying something to the effect of "WARNING: Your kernel does not support swap limit capabilities or the cgroup is not mounted. Memory limited without swap" - I don't know if that is what is causing the server's network to fail?

 

I honestly don't know to begin with what you suggested (that is not on you to be clear).

 

After hours and hours and hours of messing around I'm completely lost.

 

2) Would deleting network.cfg start to get me out of this situation? (based on this thread): 

Or maybe Nginx would still be messing with something in the background?

 

[edit: removed fluff]

Edited December 17, 20241 yr by UnraidC

Deleting network.cfg will get you back to default network settings, unclear to me that will have anything to do with nginx issues, but it won't hurt to try.

I don't know why exactly, but this (or one of the things below) is what solved it in the end:

1. I noticed that docker.img had grown >50gb (which is not normal from what I found on reddit somewhere as I only run a few dockers).
2. I removed all dockers via the "Docker" tab in Unraid.
3. I stopped the docker service (under "settings -> docker").
4. I tried to delete the docker.img via the option in the same menu after turning docker off, but nothing happened (was following these instructions: https://docs.unraid.net/unraid-os/manual/docker-management/#re-create-the-docker-image-file), i.e. the file remained in place, even after several tries.
5. I then tried to delete the file directly via system/docker/docker.img (going via the "Share" tab in Unraid) but whenever I hit delete nothing happened either (i.e. the file just remained in place).
6. I rebooted the server, leaving the docker service off.
7. I then tried again to delete the file (same as in step 5). This time it did delete and the system/docker/ folder ended up empty.
8. I then used the "Cleanup Appdata" plugin to remove all of the appdata paths shown there (except for 3 which I was quasi sure were not the ones causing issues as they never did before)
9. I reinstalled most dockers, except nginx and traefik.
10. Remote access to server was working again - problem solved.

 

So in the end it seems one of the dockers or some combination was causing issues (I still assume it was something related to Nginx or maybe Traefik even though I did almost nothing with the latter). Before doing the above I had already just tried deleting different dockers etc. but that did not resolve it.

 

Cannot say why it worked but it did, so maybe it can help somebody else as well at some point without wasting hours of your life on it.

 

In closing, thank you @JorgeB for taking the time and effort to think along with me, I appreciate it!

Edited December 17, 20241 yr by UnraidC