Docker and VM networking

[dnLL]

Hi yall,

 

Currently, when I create a VM, it gets an IP address from my pfSense router. However, if I create a docker on br0, it gets the first IP after Unraid's IP even if that IP is already used on pfSense. In fact, I don't see the dockers at all in pfSense, as if Unraid was doing the DHCP itself. I would like that to change, to have my pfSense acts as the DHCP for both my VMs and my dockers.

 

Another thing I noticed about VMs: when VMs communicate between each other (or with the host), they don't go through my pfSense router at all, all the communications are handled within Unraid. So, even if I create rule within pfSense to prevent VM 1 to talk to VM 2, that doesn't work since the traffic never reaches pfSense. That's also something I would like to change, I would like all the traffic to go through my firewall.

 

I pretty much use the default network settings, with a bonding between my 2 network interfaces on my server.

 

Here is the configuration of my 2 network interfaces:

 

 

And here is pretty much what I think is the default routing table for Unraid, at least I didn't make any chance that I am aware of (and need help to understand what's really going on 172.17.0.0/16 and 192.168.122.0/24 since I don't use these networks and don't really want Unraid to use them):

 

 

My goal at the end is to be able to actually use my pfSense firewall to prevent one specific VM from reaching anything else on the local network besides port 53 on pfSense for DNS purposes. There are most likely multiple ways to do that, but I kinda like the idea of having the traffic to go through the pfSense, this way I can properly monitor what's going on on my local network.

[dnLL]

From another thread, I now understand I can't really have the communication between dockers and the Unraid host going through my pfSense router because of the way the docker engine is built, sharing ressources with its host. Can't get DHCP to work with dockers either.

 

Now, my question remains about how to have VMs to host and host to VMs communications go through pfSense rather than be handled within the Unraid host itself. I probably need to edit the routes but last time I played with the routes, I locked myself out my Unraid host.

Edited March 26, 2020 by dnLL

[ken-ji]

To limit host to VM and VM to host communications, you want them to go through a firewall - this can be done on the Unraid level via iptables, but that's a non scalable ugly hack.

 

What you want is easy to do if you have VLAN support on your switches (or at least they happily pass VLAN tagged packets)

Enable a VLAN in Unraid network settings. Make sure not to add an IP address to the new VLAN. (this will create a new network interface eth0.2/br0.2 for VLAN ID 2. Configure pfSense to support this VLAN (DHCP, DNS, gateway). Connect a VM to this network interface. the VM should then get a DHCP IP from pfSense. You can then firewall the IP/Subnet as needed.

 

[dnLL]

7 hours ago, ken-ji said:

To limit host to VM and VM to host communications, you want them to go through a firewall - this can be done on the Unraid level via iptables, but that's a non scalable ugly hack.

 

What you want is easy to do if you have VLAN support on your switches (or at least they happily pass VLAN tagged packets)

Enable a VLAN in Unraid network settings. Make sure not to add an IP address to the new VLAN. (this will create a new network interface eth0.2/br0.2 for VLAN ID 2. Configure pfSense to support this VLAN (DHCP, DNS, gateway). Connect a VM to this network interface. the VM should then get a DHCP IP from pfSense. You can then firewall the IP/Subnet as needed.

 

But how does it work by default? Like, if I create a VM, it does get a DHCP address from pfSense. But traffic doesn't route through pfSense. How does that make sense? Put it another way, why will a VLAN force traffic to go through pfSense basically?

 

That will work for my need (since I basically want to isolate 1 VM) so I'll go and test it now, but I'm trying to understand the inner workings, why default routing doesn't go through the router and why it will with a VLAN.

[dnLL]

Just tested it and it works with a VLAN, I'm fully able to isolate the VM however I want through regular pfSense rules. I guess my questions now are just to help me understand what makes the traffic go through pfSense when on a VLAN and not through pfSense if no VLAN.

[ken-ji]

A VLAN is a different subnet just sharing the same physical connections as the main subnet.

So when a new VLAN is configured, you'll need to configure the router (in your case pfSense) to know about the VLAN and support it as well.

So for devices on different VLANs / subnets to reach each other, it absolutely needs to go through a router. If the devices are on the same VLAN/subnet they will ignore the router and communicate directly.

[dnLL]

1 hour ago, ken-ji said:

If the devices are on the same VLAN/subnet they will ignore the router and communicate directly.

I feel like an idiot. That's the part I was missing. Thank you for the info, basically if you really want all the communications to go through pfSense, you need a dedicated VLAN for every VM. Which makes sense now that I think about it since I've been reading about some people doing exactly that...