Hi Everyone,

 

I am completely out of ideas very frustrated and tired as well...

 

On Friday 19th of august I installed a pfsense server in my network to replace my UDM pro, everything went fine as far as I could tell at that time. I configured 2 site2site ipsec connections to 2 UDM pro's in other locations, the same configuration as before with the 3x UDM pro setup but now with 2x UDM and 1x pfSense. 

 

To make my story a bit more understandable: 

 

1 Site = DC

1 Site = Home

1 Site = Friends Place

 

Site DC:

1 unraid server ( Dell R730XD ) running both docker and vm's with Intel Corporation Ethernet Controller X710 for 10GbE SFP+ but only copper rj45 in use at the moment

1 pfsense router

2 switches 

1 windows server

some miscellaneous stuff

 

all 3 of them connected together with site2site vpn.

 

Everything was working fine immediately with the exception of my Unraid machine, for some reason when ever I started the array I would lose access to the webinterface from home ( 1 site of the site2site connection ) I had still access to IPMI ( iDrac ) and from within a windows machine also in DC site I was able to get to the web interface. I was however able to reach all other devices with web interfaces from home  from DC site so vpn was not the issue. 

 

While troubleshooting this weird problem I figured out and tried a few things

 

- Discovered that pfsense logs told me that 192.168.35.5 ( IP of unraid box internally ) was changing Mac address every minute or so

- Disabled Host access in docker because my problem appears to be very similar to this: 

- Disabled Both Bridging and bonding on the Unraids box Interface

- Discovered that me being unable to reach the webui was not the only problem since Tautulli also spitting out notifications that the Plex server is unstable ( saying UP and Down every few minutes ) and playing movies is also unstable 

- I tried both static and DHCP Ip allocation of Unraid machine same issues remain 

- Downgraded to unraid 6.9.2 from 6.10.3 

- removed all Br0 IP's previously used by a container all docker containers now use either host of a custom docker network

- renamed network.cfg and network-rules.cfg

 

 

there is definitely some troubleshooting steps I did that I have forgot to mention since I have been on this issue for over a week now with no luck.

 

all dockers with a web interface routed through a Cloudflare tunnel and proxy are reachable

 

in a nutshell i experience the following issues:

 

- Plex unstable 

- Some containers seem not to be able to see each other for example sync-thing is constantly trying to see other machines but is only rarely able to set a connection although all endpoints are either inside of the Lan of DC or reachable over Site2Site VPN
 

From Home:

- Unraid Webui unreachable as soon as I start the array

 

- other docker containers through Ip unreachable such as sync-thing

- icmp ping works if array is stopped but switches to unreachable after array start 

 

I hope someone has some ideas left. 

 

 

For anyone in the future in a similar situation...

 

I forgot that a long time ago I added a static route directly into unraid that points Site A's subnet to gateway 192.168.35.1

But with the migration from the UDM pro to pfSense I set pfSense on 192.168.35.3 so that I could run both gateways simultaneously. 

So after the migration when I removed the UDM pro from the setup the traffic meant for Site A went to 192.168.35.1 which at that point didn't exist anymore. 

 

I think above issue together with the host access Mac issues described in the post above and here: 

 I think it was a combination of both these issues.