Virtualized pfSensen unable to access self hosted services

[ySkNight]

Hello,

 

I've successfully setup a vm running pfSense and everything work fine so far beside that I'm unable to access my reverse proxy self hosted nextcloud, bitwarden, prior to that I have everything is working.

 

I've FiOS and the way I've set it up is 

Verizon FiOS ONT(wan) > Intel NIC port1(wan) > Intel NIC port2(lan) > A switch(lan) > Quantum Gateway(wan) for TV guides, etc

 

Most of the physical device is showing up on my "Status > DHCP leases" but all docker ip is not showing up. Regardless if they're on br0, bridge. 

 

I'm using NGINX PROXY MANAGER, at beginning when I access, let say bitwarder using internal IP it show up as "pfsense Potential DNS Rebind attack detected" Already properly port forwarded NGINX PROXY MANAGER and then I play around enable some stuff that goes away, but I'm unable to use external ip / domain to connect it return back timeout.

 

Thing I've tried

*Change pfSense web port to other than 443/80

*Added alternate hostnames under "SystemAdvancedAdmin Access"

*NAT Reflection mode for port forwards - Pure NAT

*Enable NAT Reflection for 1:1 NAT

*Enable automatic outbound NAT for Reflection

 

TL:dR

Problem

1: Unable to access self hosted domain, after using pfSense

2: Duckdns unable to update IP every 5 min even tho it said IP has been updated.

3: DHCP leases in pfSense unable to see docker IP regardless which interfaces was on.

 

Thanks in advances and sorry for a messy writing.

Edited October 28, 2020 by ySkNight

[m0t0k0]

1. pfSnese has rebind protection on by default (its a good idea) what you need to do is tell pfSnese its ok when your DNS resolves your servers IP as an internal rather than an external IP.

 

pfSense: If you are using pfSense internal DNS resolver service, you can add these Custom Option lines:

server:
private-domain: "your domain here"

 

2. I have a physical fw as its seemed like a lot of hassle to virtualise it so cant really help here but I'm sure is has something to do with the virtualisation. Did you passtrough the NIC hardware to the pfSense VM?

How does the Unraid box connect to your LAN?

 

3. Can you not create the containers with static IP addresses and then entre these into the static leases on pfSense

[ySkNight]

1 hour ago, m0t0k0 said:

1. pfSnese has rebind protection on by default (its a good idea) what you need to do is tell pfSnese its ok when your DNS resolves your servers IP as an internal rather than an external IP.

 

pfSense: If you are using pfSense internal DNS resolver service, you can add these Custom Option lines:

server:
private-domain: "your domain here"

 

2. I have a physical fw as its seemed like a lot of hassle to virtualise it so cant really help here but I'm sure is has something to do with the virtualisation. Did you passtrough the NIC hardware to the pfSense VM?

How does the Unraid box connect to your LAN?

 

3. Can you not create the containers with static IP addresses and then entre these into the static leases on pfSense

1. Yea, I fixed by adding that script at end of DNS resolver, but I'm still have problem to let pfSense to see my docker ip in DHCP leases.

 

2. Beginner would like to try something new and learn something new. Since I've unRAID running 24/7, its good ideas to run on unRAID and i would like to consolidate everything in one place/unit. For NIC hardware, yes I've passthru to pfSense VM, that's the first problem I solved, since it was not detect by pfSense and I've edited XML file and it successfully detected by pfSense after. unRAID is connected to my unifi switch, pic for reference (https://imgur.com/a/LryX1Sm)

 

3. I deleted all my docker trying to reinstall them, but as far as I know, normally you wont allow to have same IP in 2 network, I was able create a docker with same IP as one of my PC. It didnt give me error when I create it.

 

[m0t0k0]

1. I assume your network setup is similar to the one I have pictured below? As far as pf is concerned your Unraid box only has one MAC address and as such the other IPs that exist in docker are reached through you unraid box over the docker bridge br0.

 

I did look into this before and it seems you would need to use a docker macvlan where each container is assigned a virtual MAC address and can then reach out to the pf box for an IP. I never went any further with this myself but there is info here https://docs.docker.com/network/

let me know if you get anywhere with this as I'm interested in my containers showing as physical devices on my network.

 

I assign an IP to the container when its created (its in the range of reserved IPs from pf DHCP settings for LAN. I can then get to that IP address via LAN (look at attachment custom br0). However this is just for one or two containers which need it.

 

Most of my containers run on a separate virtual network within docker called proxynet. This was setup after following the reverse proxy guide by spaceinvaderone you should check him out loads of good vids (he also runs virtual pf on unraid.

 

 

2. Yes I understand it fun learn however I have to get a physical device as my old unraid box had no expansion slots free.

This looks like a good place to start for help with this

 

3. Without seeing how you set it up I assume your using the virtual bridge from witin docker and essentially is forwards network traffic for a specific port to the internal container so all traffic going to a container goes though

"unraid IP":"assigned port ie 8080 or whatever"

to

"IP of container on br0":"whatever port you picked"

 

 

[ySkNight]

6 hours ago, m0t0k0 said:

1. I assume your network setup is similar to the one I have pictured below? As far as pf is concerned your Unraid box only has one MAC address and as such the other IPs that exist in docker are reached through you unraid box over the docker bridge br0.

 

I did look into this before and it seems you would need to use a docker macvlan where each container is assigned a virtual MAC address and can then reach out to the pf box for an IP. I never went any further with this myself but there is info here https://docs.docker.com/network/

let me know if you get anywhere with this as I'm interested in my containers showing as physical devices on my network.

 

I assign an IP to the container when its created (its in the range of reserved IPs from pf DHCP settings for LAN. I can then get to that IP address via LAN (look at attachment custom br0). However this is just for one or two containers which need it.

 

Most of my containers run on a separate virtual network within docker called proxynet. This was setup after following the reverse proxy guide by spaceinvaderone you should check him out loads of good vids (he also runs virtual pf on unraid.

 

 

2. Yes I understand it fun learn however I have to get a physical device as my old unraid box had no expansion slots free.

This looks like a good place to start for help with this

 

3. Without seeing how you set it up I assume your using the virtual bridge from witin docker and essentially is forwards network traffic for a specific port to the internal container so all traffic going to a container goes though

"unraid IP":"assigned port ie 8080 or whatever"

to

"IP of container on br0":"whatever port you picked"

 

 

 

 

1. That is correct, that's exactly my network look like, but here is weird thing, I've revert back the setup, I didn't see dockers IP in VZ router's DHCP lease as well, but I was able to use same NGINX proxy manager to reverse proxy some of my self-hosted services. Take look at your possible solutions basically command line:

$ docker network create -d macvlan \

--subnet=192.168.1.0/24 \

--gateway=192.168.1.1 \

-o parent=eth0 pub_net

very simlliar to create a custom proxynet right? Btw, yes I've follow SpaceInvaderone's youtube video from beginning of using unRAID, That's why I decided virtualized pfSense in the first place.

 

2. I have to break IMMOU Group using ACS override in order to break all of them in to separate groups. Follow SpaceInvaderone's video as well.

 

3. That's right, but for some reason I'm able to get it thru locally, but not external, I just cant identify if its docker(nextcloud, bitwarden) it self blocked by pfSense firewall can't get thru, or if its NGINX wasn't setup correctly to properly forward to right address, I'm able to issue a SSL certificate and already forward the right port for NGINX proxy manager. If I didn't forward the right port, I will not able to request a SSL from Let's Encrypt anyway, it will return back Internal error occur some like that. 

 

Additional questions, Am I able to use LAN connect to WAN to work with pfSense? Instead of the really WAN cable from ONT? I would like to keep current network setup, and work with pfSense until I fully solve this issue. 

 

Thank you very much for the help btw!

[m0t0k0]

1. Oh yea that should do the tick. 

I must have done this but assigned it to the bridge br0 instead of eth0.

I'm not sure if this will work though as these containers will still be created with an IP an as such will not be leased one.

I think it would still be best to just assign them a static address on the router so they appear in the GUI

 

Unfortunately I'm locked out of my router at the moment (lastpass did not save the password when I changed it) so I can't have a look. I will need to wait untill after the holidays.

 

2. I have tried many router OS now and currently on OpenWRT but it has been a big hassle having a physical box. I think I'm going to try proxmox and still have a physical box for the router but virtualise the OS so I can change it when I want.

 

3.

a) So when you access from LAN are you going to nextcloud."your domain".whatever or are you doing "server IP":"nextcloud port"?

b) Have you tried accessing the service via the domain name from outside your LAN ie using a mobile phone not connected to Wi-Fi?

c) Does you domain resolve to your external IP?

type you domain name in here https://www.ultratools.com/tools/ipWhoisLookupResult

the result should match the one given here https://whatismyipaddress.com/

^^^^if your using cloudflare for you DNS make sure you have the cloudflare proxy turned off when you do this^^^