lost My servers connection

[philliphartmanjr]

A few weeks ago my system was running with no issues, then for some reason My servers and community applications lost internet access.  I do have a PFsense + firewall as I have read that could be causing an issue but I have not found any information why this would be the case. The only documentation I could find was on virtualization of PFsense and that is not my case. My Unraid server has internet access (I can ping internally and externally without issue) and my docker containers has internet access as well. I do not currently have anything on PFsense as far as I can tell that would be causing an issue. In PFsense I have an allow all rules on all my local networks and I do not have PFblocker so there is no reason (and no way from what I can tell) to whitelist websites. PFsense is running as DNS reslover and as a troubleshooting step I enabled DNS rebinding to see if that fixed the issue even though is was fine before. I have checked My PFsense firewall logs and nothing has been blocked from my Unraid's server ip. The only other weirdness I noticed is the nerdpack plugin is empty when I open it.

 

I am not sure what step to take from here but i will add I would say I have at least an intermediate understanding of Unraid and I work in IT with a focus on networking and comptia network + certified. I mention this not to prevent any ideas but only to show you probably do not need to explain simple solutions to help speed up the process on finding the issue. I have included my server logs any help would be most appreciated.

talonhawk-diagnostics-20230412-1116.zip

[ljm42]

It looks like the system does not have a DNS server. Go to Settings -> Network Settings and apply a static DNS server of 8.8.8.8, that will likely solve your connectivity issues.

 

Once the network is working, I'd highly recommend that you install the Fix Common Problems plugin, it will alert you to other potential issues.

 

Also, the server has a routable IP address. Unraid is not meant to be put directly on the Internet, it should be behind a firewall. 

 

1 hour ago, philliphartmanjr said:

I enabled DNS rebinding

 

DNS Rebinding Protection must be disabled in order to use ip.hash.myunraid.net urls.

[philliphartmanjr]

Ok I manually set the DNS to googles and it did not fix the issue and after it ran through the ip lease the DNS setting went back to automatic. I did this sever times and finally gained access to community applications and installed the fix common problems plugin. There were a couple of old plugins that it found and i fixed those so the only thing it has now is my logs being saved to flash. I updated everything while I have access just in case. 

 

I am not sure which address I used when teh DNS started to work again so after I had everything updated and the old plugins removed I went back to the network settings to put the DNS on PFsense 127.0.0.1 since it is set up as a DNS resolver if it does not have the address it finds it the same way another service would ie: Google. After I set  the DNS it went back to automatic again is this normal?

 

My server is behind my PFsense firewall the reason why it is routable is because I have the server name set up in PFsense saved in a local DNS setting and that has worked for months with DNS rebinding protection on. That way I can control what hits my servers a bit more securely than allowing DNS rebinding.

 

Now that I have connection to Myservers and the Community Applications the only problem I see in the logs is unraid is trying to send packets to network 172.17.255.255 port 137 and 138. That is not a network in this location and the docker network in 172.18.0.0/16 in the routing table. Is this something that I should be concerned with? I have had Unraid do some crazy stuff in the past with improper network configs.

 

Lastly after I put the DNS on PFsense 127.0.0.1 and it went back to automatic it still does not show a DNS in the network options is that normal? Now that I know the issue is with DNS I am sure I can fix it again but how can I know what Unraid is using for DNS? even though it is working fine now when I change the DNS server in Unraid's network setting even the log does not show what it is using only that it is in the mDNS for my servers ip.

[ljm42]

16 minutes ago, philliphartmanjr said:

put the DNS on PFsense 127.0.0.1 since it is set up as a DNS resolver

 

127.0.0.1 refers to Unraid itself, there is no DNS resolver there. You should put the IP address of the PFsense VM. Or really, just keep it pointed at a public DNS server like 8.8.8.8. Many people cause problems for themselves by pointing at a local DNS server like pihole that does ad blocking and it ends up blocking something Unraid needs.

 

22 minutes ago, philliphartmanjr said:

My server is behind my PFsense firewall the reason why it is routable is because I have the server name set up in PFsense saved in a local DNS setting and that has worked for months with DNS rebinding protection on. That way I can control what hits my servers a bit more securely than allowing DNS rebinding.

 

If your network is using routable IPs then DNS Rebinding Protection doesn't really protect you from anything anyway.

 

It isn't a hard requirement, but I would recommend using a private IP space and then disabling DNS Rebinding Protection for the myunraid.net domain. Using pfsense you can leave it enabled everywhere else.

 

 

18 minutes ago, philliphartmanjr said:

the only problem I see in the logs is unraid is trying to send packets to network 172.17.255.255 port 137 and 138

 

That is a private IP so it is something on your network. If you do a Google search for "unraid 172.17.255.255" it looks like it is related to Docker. If you want more details I'll move this thread to General Support where more folks will see it.

 

[philliphartmanjr]

54 minutes ago, ljm42 said:

 

127.0.0.1 refers to Unraid itself, there is no DNS resolver there. You should put the IP address of the PFsense VM. Or really, just keep it pointed at a public DNS server like 8.8.8.8. Many people cause problems for themselves by pointing at a local DNS server like pihole that does ad blocking and it ends up blocking something Unraid needs.

 

I am not sure what I was thinking 127.0.0.1 is a loop back address I put the PFsense ip back in. I do not have the PFblocker app installed yet so no additional blocking should be happening. Also I am not using a vm for PFsense I have an Netgate appliance with PFsense+ installed so the resolver that I am referring to is PFsense.

 

1 hour ago, ljm42 said:

If your network is using routable IPs then DNS Rebinding Protection doesn't really protect you from anything anyway.

 

I am not using an external routable IP for Unraid. DNS for internal host's only resolves internally and then I have Unraid urls pointed to the host name. Honestly the reason I did it that way was because at the time I was still trying to figure out PFsense as its UI is different than what I was used to and I thought DNS rebind was a blanket on or off.

 

The only question, is there a way for me to see what Unraid is trying to use for DNS? it does not matter what I put in Unraid network settings 8.8.8.8 1.1.1.1 1.0.0.1 it goes back to automatic. The only way I have been able to make it stay is to set a static IP in Unraid's network settings and then set the static DNS. Then it saves and stays static. Currently Unraid is set to DHCP with a static IP set in PFsense and anytime I tried to set a static DNS it just changes back to automatic and I am unsure how unraid handles this. I made no other changes except trying to change the static DNS about 5 times and then I had a connection and the system log did not show any DNS setting only that network being reset. If a network reset is what really fixed it then why would it not restore on one of the several server reboots that i did before posting here? If Unraid sets to auto DNS by default when on DHCP I don't see why I regained internet connection.

[MAM59]

this is pretty much normal.

 

You asked for automatics, you get automatics. (maybe  the gui is misleading here, the manual settings are ignored thats why it reverts to "automatic" afterwards)

 

If you use DHCP, unraid reads ALL settings from DHCP, including the DNS Servers.

 

So either set unraid to static network and specify the DNS server of your desire, or tell the DHCP Server to send this DNS Server to unraid.

 

[philliphartmanjr]

On 4/13/2023 at 2:09 AM, MAM59 said:

this is pretty much normal.

 

You asked for automatics, you get automatics. (maybe  the gui is misleading here, the manual settings are ignored thats why it reverts to "automatic" afterwards)

 

If you use DHCP, unraid reads ALL settings from DHCP, including the DNS Servers.

 

So either set unraid to static network and specify the DNS server of your desire, or tell the DHCP Server to send this DNS Server to unraid.

 

That is what I assumed, My question is with this being the case

 

1. What actually caused the issue. Rebooting Unraid will reestablish network config and rebooting did not fix the issue. about 5 attempts at manually trying to set DNS did.

 

2. The solution given to me was that my Unraid did not have DNS set up how can someone tell? I previously had connection and I have it again but I can't find anything in the log that says DNS was the problem.

 

I feel like this is going to happen again even though my system is working now it seems the underlining issue is unresolved.

[ljm42]

 

On 4/12/2023 at 3:59 PM, philliphartmanjr said:

it does not matter what I put in Unraid network settings 8.8.8.8 1.1.1.1 1.0.0.1 it goes back to automatic.

 

I am not aware of any issues overriding the DNS servers provided by DHCP. If you are able to reproduce this, first try again in another browser (ideally one without plugins or ad blockers)

 

If the problem happens in multiple browsers, please provide me with step by step instructions to reproduce it myself.  A Loom recording of the problem ( https://www.loom.com/ ) could also be helpful.

 

14 hours ago, philliphartmanjr said:

I can't find anything in the log that says DNS was the problem.

 

Unzip your diagnostics and look at the system/urls.txt file

 

It shows that at the time you collected diagnostics, DNS 1 was undefined

 

It also hides your IP address because it determined that the IP is routable. This was logic I added for privacy reasons, but if you feel I got the logic wrong please let me know what the server's IP address so I can review the logic.

 

[philliphartmanjr]

On 4/15/2023 at 2:41 AM, ljm42 said:

 

 

I am not aware of any issues overriding the DNS servers provided by DHCP. If you are able to reproduce this, first try again in another browser (ideally one without plugins or ad blockers)

 

If the problem happens in multiple browsers, please provide me with step by step instructions to reproduce it myself.  A Loom recording of the problem ( https://www.loom.com/ ) could also be helpful.

 

 

Unzip your diagnostics and look at the system/urls.txt file

 

It shows that at the time you collected diagnostics, DNS 1 was undefined

 

It also hides your IP address because it determined that the IP is routable. This was logic I added for privacy reasons, but if you feel I got the logic wrong please let me know what the server's IP address so I can review the logic.

 

Thanks that helped a lot I was looking in the wrong place in the logs for that info I thought is was in the /config /network folder. I am pretty sure its not a browser issue I run firefox and the only 2 extensions that I have is dashlane and honey. I have set my server and myunraid.net so rebinding will work (DNS rebinding protection off for those 2). I still have internet connection, my thinking is... I have some local hosts (including unraid server) set to resolve locally by DNS and this is why it shows as routable in the url.txt file.

 

I have looked at PFsense documentation and all it really mentions is if the client supports it it works. I am not sure why the DNS 1 is blank, Maybe Unraid doesn't support this fully but still sends DNS requests to PFsense since it shows it is routable from that IP? I put up a current log to show that is still says the same thing and internet access is fine. I am curious to see if it happens again and if so I will remove unraid from the local routing table. I assume why Unraid has internet is it send the request then it resolves locally then hits the external DNS (if needed) and this somehow makes everything work without rebinding? Just to be clear any external request to resolve DNS query to my unraid server would be rejected, it is not resolved via WAN.

 

It seems like it is a it kinda works but is maybe not really meant to kinda thing.

talonhawk-diagnostics-20230427-1409.zip

[ljm42]

How many network cards does this system have? The network.cfg thinks it has 12 ?!?!

 

# Generated settings:
IFNAME[0]="GATEWAY6:1"
DHCP_KEEPRESOLV="yes"
DNS_SERVER1="127.0.0.1"
DNS_SERVER2="172.52.52.1"
DHCP6_KEEPRESOLV="no"
IFNAME[1]="GATEWAY:1"
IFNAME[2]="IPADDR6:1"
IFNAME[3]="IPADDR:1"
IFNAME[4]="METRIC6:1"
IFNAME[5]="METRIC:1"
IFNAME[6]="NETMASK6:1"
IFNAME[7]="NETMASK:1"
IFNAME[8]="PRIVACY6:1"
IFNAME[9]="USE_DHCP6:1"
IFNAME[10]="br0"
BRNAME[10]="br0"
BRSTP[10]="no"
BRFD[10]="0"
DESCRIPTION[10]="work"
BRNICS[10]="eth0"
PROTOCOL[10]="ipv4"
USE_DHCP[10]="yes"
DHCP_KEEPRESOLV="yes"
DNS_SERVER1="172.52.52.1"
DNS_SERVER2="10.128.0.1"
USE_DHCP6[10]="yes"
DHCP6_KEEPRESOLV="no"
MTU[10]="1500"
IFNAME[11]="br1"
BRNAME[11]="br1"
BRNICS[11]="eth1"
BRSTP[11]="no"
BRFD[11]="0"
DESCRIPTION[11]="cat_and_dog"
PROTOCOL[11]="ipv4"
USE_DHCP[11]="yes"
MTU[11]="1500"
SYSNICS="12"

 

It looks like this file may have been edited manually because I see some duplicate entries. I'd suggest that you delete config/network.cfg from the flash drive and reboot. The system should pick up a new IP on eth0 from DHCP. Then go to Settings -> Network Settings, make a dummy change and hit save to generate a new network.cfg and start testing things from there.

 

[philliphartmanjr]

16 hours ago, ljm42 said:

How many network cards does this system have? The network.cfg thinks it has 12 ?!?!

 

# Generated settings:
IFNAME[0]="GATEWAY6:1"
DHCP_KEEPRESOLV="yes"
DNS_SERVER1="127.0.0.1"
DNS_SERVER2="172.52.52.1"
DHCP6_KEEPRESOLV="no"
IFNAME[1]="GATEWAY:1"
IFNAME[2]="IPADDR6:1"
IFNAME[3]="IPADDR:1"
IFNAME[4]="METRIC6:1"
IFNAME[5]="METRIC:1"
IFNAME[6]="NETMASK6:1"
IFNAME[7]="NETMASK:1"
IFNAME[8]="PRIVACY6:1"
IFNAME[9]="USE_DHCP6:1"
IFNAME[10]="br0"
BRNAME[10]="br0"
BRSTP[10]="no"
BRFD[10]="0"
DESCRIPTION[10]="work"
BRNICS[10]="eth0"
PROTOCOL[10]="ipv4"
USE_DHCP[10]="yes"
DHCP_KEEPRESOLV="yes"
DNS_SERVER1="172.52.52.1"
DNS_SERVER2="10.128.0.1"
USE_DHCP6[10]="yes"
DHCP6_KEEPRESOLV="no"
MTU[10]="1500"
IFNAME[11]="br1"
BRNAME[11]="br1"
BRNICS[11]="eth1"
BRSTP[11]="no"
BRFD[11]="0"
DESCRIPTION[11]="cat_and_dog"
PROTOCOL[11]="ipv4"
USE_DHCP[11]="yes"
MTU[11]="1500"
SYSNICS="12"

 

It looks like this file may have been edited manually because I see some duplicate entries. I'd suggest that you delete config/network.cfg from the flash drive and reboot. The system should pick up a new IP on eth0 from DHCP. Then go to Settings -> Network Settings, make a dummy change and hit save to generate a new network.cfg and start testing things from there.

 

I have 4 Ethernet ports 2 on the motherboard and 2 on a separate nic. It looks like Unraid counts every port that has its own Mac as a NIC? I did upgrade my system and go through a couple of multiport cards in the last year and I am guessing it doesn't adjust the count on reboot.

 

I did lose connection to Myserver's again, this time the app page still worked along with my VM's, Docker containers, and Unraid it self. I believe I have discovered the problem. there are a lot of video's and guides on these individual topis but nothing that I have found that ties everything together. I believe the Myserver's issue happened as I was trying to setup Rustdesk and Nextcloud.

 

To summerise for anyone that has Pfsense, Unraid, and it trying to reverse proxy a domain or self host a domain you need to make sure to change the web port for Pfsense and uncheck the redirect, otherwise Pfsense can get confused and will assume at least some of the time that everything entering via wan is for the Pfsense firewall its self regardless of any NAT or firewall rules in place. You will also need proper host overrides in place in Pfsense for local access.

 

I would also recommend changing Unraid's web port just in case it can cause confusion port wise for Unraid its self and docker containers.

 

My network tab still wasn't showing a DNS ip even though everything seems to be working properly deleting the network.config file seems to have fixed that. I am pretty sure will this fixes everything is correct now. I will give it a few days to make sure Myservers stays connected and if so Mark this as a solution. Thank you for your time and help.

[MAM59]

41 minutes ago, philliphartmanjr said:

It looks like Unraid counts every port that has its own Mac as a NIC

EVERY Port needs to have its own, unique MAC, thats fundamental basics of Ethernet.

Some fake NICs (tunnels, VLANS) add even more virtual ports, each with its own NIC.

 

Using Network modifying dockers like pihole, pfsense and so on, is nothing for the simple heart, you need to know what you are doing, if in doubt, LEAVE THEM ALONE!

You will do more harm to your LAN that you think you might do something good.

 

[philliphartmanjr]

On 5/4/2023 at 9:41 AM, MAM59 said:

EVERY Port needs to have its own, unique MAC, thats fundamental basics of Ethernet.

Some fake NICs (tunnels, VLANS) add even more virtual ports, each with its own NIC.

 

Every port does have its own unique Mac. My point was the reason Unraid was showing 12 SYSNICS is because it counted every port on each NIC that I went trough that had its own MAC and retained that info even after reboots. The allocation in Unraid network settings is correct. Also every port doesen't always have its own unique MAC. Some multiport cards have to be split by VLAN.

 

On 5/4/2023 at 9:41 AM, MAM59 said:

Using Network modifying dockers like pihole, pfsense and so on, is nothing for the simple heart, you need to know what you are doing, if in doubt, LEAVE THEM ALONE!

You will do more harm to your LAN that you think you might do something good.

 

PFsense is not a docker container, as stated above I have a Netgate official appliance running Pfsense plus. I would say that your comment is very rude and if you would have actually read the thread doesn't even pertain to my situation nor is it relevant to what that fix was.

[MAM59]

1 hour ago, philliphartmanjr said:

PFsense is not a docker container, as stated above I have a Netgate official appliance running Pfsense plus. I would say that your comment is very rude

I did not intent to be rude, sorry. My usual excuse would be "english is not my natural language." Obviously I misunderstood your writing or did not understand some aspects.