How to setup pfSense VM VLAN (on br0) for Unraid Dockers and other VMs

[samba_69]

I have been trying for some time to utilize VLANs to put dockers on a separate VLAN on br0 (instead of different ports on localhost)

 

I have pfSense VM connected to Unraid's br0 via virtio (VNET0) on the same interface I have VLAN setup with id 150, which I intend to use with docker and VMs on Unraid.

 

Setup:

Setings within pfSense:

 

pfSense VM connect VNET0 to br0 (no virtual nic br0.150):

 

Unraid network setup:

 

 

But whenever I connect VMs / Docker container to bridge br0.150, they become isolated from all other networks.

 

How to pass VLAN from pfSense VM to Unraid bridge br0/br0.150 without adding extra nic to pfSense for each VLAN?

 

 

[Ford Prefect]

First things first: I have no clue on how to configure a pfsense, especially not a BSD firewall ... their concept never stuck to my brain...I am a Mikrotik Person 

 

On 6/22/2021 at 10:58 PM, samba_69 said:

But whenever I connect VMs / Docker container to bridge br0.150, they become isolated from all other networks.

...but isn't that the whole point of using VLANs?

What exactly does not work?

 

On 6/22/2021 at 10:58 PM, samba_69 said:

How to pass VLAN from pfSense VM to Unraid bridge br0/br0.150 without adding extra nic to pfSense for each VLAN?

I noticed, that you did not set an IP to the br0.150

Also what IP(s) are you using for the interfaces in your pfsense? 

 

What you are (trying/need to) build is a Router-on-a-Stick Scenario.

VLANs are L2. The port in your pfsense attached to br0 in unraid will work like a trunk port of a switch.

But if you want other devices on different VLANs to communicate with each other, you need routing, which is L3.

So define IP networks and assign IPs and routes.

The default gateway for each network should be the (respective VLAN-)IP of your pfsense interface.

Then the pfsense firewall needs to be set up according to your needs, accepting/dropping packets in the forward-chain.

 

Is the pfsense also the main router for you WAN/Internet connection (is there another NIC passed through to the VM that is, i.e. attached to a modem)?

[samba_69]

Yes, pfSense VM is my primary router/firewall and sole source to connect to internet!

 

I have 4 port NIC passed (vfio) to pfSense, of which 3 ports are used as load balanced WAN port!
2 ports (1 from 4port NIC and 1 form Motherboard) connect to a network switch (to load balance some local and internet traffic)

1 virtio passed to pfSense which connects to br0.

I want this virtio to handle vlan traffic for docker container, as I want to allow only selected local user to connect to docker, but all can connect to unraid

 

What I want to do is pass a single  virtio to pfsense, on which I want to create multiple vlans from pfSense settings, which will eventually be used unraid's bridge to distribute the traffics.

 

I understand that adding virtual interface for pfsense on for each vlan will solve the problem, but adding new interface and managing them from pfsence will become problem as In future I have plans to assign individual vlan for a group of VMs for security reasons as they may be used by different guest. Having separate interfaces for each vlan will be similar to creating a virtual bridge for each network.

[Ford Prefect]

1 hour ago, samba_69 said:

Yes, pfSense VM is my primary router/firewall and sole source to connect to internet!

 

I have 4 port NIC passed (vfio) to pfSense, of which 3 ports are used as load balanced WAN port!
2 ports (1 from 4port NIC and 1 form Motherboard) connect to a network switch (to load balance some local and internet traffic)

1 virtio passed to pfSense which connects to br0.

...OK, so you have 5 NICs in your unraid box, where the first 4 (possibly a quad-NIC) are passed-through to your pfsense.

This will leave you with a single NIC visible to unRaid host, where you also defined br0 (and VALNs, br0.xxx) on it.

This unraid NIC (br0) and one NIC from your pfsense are connected to another external, physical switch.

Is this switch VLAN capable?

 

1 hour ago, samba_69 said:

I want this virtio to handle vlan traffic for docker container, as I want to allow only selected local user to connect to docker, but all can connect to unraid

Did you enable Bridging on your pfsense?

As unraid br0 and pfsense are already connected via a path through the external switch, you create another, second path via that virtio-Interface and br0....there might be a L2 loop.

 

1 hour ago, samba_69 said:

What I want to do is pass a single  virtio to pfsense, on which I want to create multiple vlans from pfSense settings, which will eventually be used unraid's bridge to distribute the traffics.

This is not the way it works.

IMHO you have several L2 components in your setup.

- the external switch, where unraid br0 and pfsense NIC.4 us physically connected

- your pfsense, where the main switch-.chip/bridge is the CPU

- unraid, with br0

 

...in order to make VLANs work, the VLANs need to be configured in each component. Connection between components are either through trunk- or access-ports.

You cannot simply define VLANs in your pfsense only and expect these to be magically distributed to the other parts.

 

1 hour ago, samba_69 said:

I understand that adding virtual interface for pfsense on for each vlan will solve the problem, but adding new interface and managing them from pfsence will become problem as In future I have plans to assign individual vlan for a group of VMs for security reasons as they may be used by different guest. Having separate interfaces for each vlan will be similar to creating a virtual bridge for each network.

You can use br0 / virtio and use it as a trunk port.

But you need to define VLAN tagged and untagged interfaces on each component.

 

In unraid br0 is a trunk interface in a software switch

Each VLAN interface is an access-port to that unraid switch.

When attaching a docker to an unraid VLAN interface, i.e. br.150, tagged traffic with VLAN-ID 150 will travel via br0 (traffic orginating in br0 will be untagged).

 

The virtual connection between br0 and pfsense-virtio is a switch-to-switch interconnection...configure the virtio interface as a trunk port in pfsense.

Then traffic vom br.150 will arrive tagged in pfsense via virtio interface...make your pfsense central bridge (cpu) decide how to handle it.

 

[samba_69]

Am quite new here, hense pardon me if I ask silly questions!

 

 

Let me consider a standalone scenario where I have just my Unraid server, pfSense VM and Unraid Dockers. Based on this I would like to handle the vlans as you have said .

 

 

 

11 minutes ago, Ford Prefect said:

The virtual connection between br0 and pfsense-virtio is a switch-to-switch interconnection...configure the virtio interface as a trunk port in pfsense.

Then traffic vom br.150 will arrive tagged in pfsense via virtio interface...make your pfsense central bridge (cpu) decide how to handle it.

I probably did exactly the same as you instructed earlier:
1. virtio is set as trunk/parent interface for VLAN 150

On 6/23/2021 at 2:28 AM, samba_69 said:

 

2. virtio connects to br0

On 6/23/2021 at 2:28 AM, samba_69 said:

 

 

 

Even on doing this I cannot reach (ping) the dockers on br0.150 lan
neither can dockers can reach internet

 

This is not firewall vlan 150 have pass all rule defined

 

 

What am I doing wrong?

[Ford Prefect]

As said, I have no clue on how to do that in pfsense. 

Your screenshot just shows, that you defined the VLAN tag as part of the base interface virtio/opt3.

in order to use the traffic separately, IMHO you need to add the VLAN to the list of interfaces on the pfsense bridge as well, so you will have an "opt4" in addition to opt3/virtio.

 

Are you able to ping the IP on br.0 ?

Please define an IP on br0.150 on the unraid box as well...then try to ping that from pfsense.

 

I am using VLANs in unraid this way, but with a hardware router and it works like a charm.

Also used it with a Router VM, but with all real, passed-through NICs and not with virtio.

You can also directly connect the unraid NIC and NIC.4 of pfsense (instead over the switch) and do not use virtio-NIC (just in case this is the root-cause) and then try.

[samba_69]

22 hours ago, Ford Prefect said:

Are you able to ping the IP on br.0 ?

Please define an IP on br0.150 on the unraid box as well...then try to ping that from pfsense.

 

I can ping unraid from pfSense on br0,

But cannot ping on Vlan/br0.150

 

What I believed was br0 behaving as a trunk, and any tagged packets in trunk would be untagged and moved to own switch, but this is definitely not the case here

[Ford Prefect]

31 minutes ago, samba_69 said:

 

I can ping unraid from pfSense on br0,

But cannot ping on Vlan/br0.150

 

What I believed was br0 behaving as a trunk, and any tagged packets in trunk would be untagged and moved to own switch, but this is definitely not the case here

 

try to do one step at a time. My guess is, that your pfsense is missing the interface for vlan-150 and unraid is missing this as gateway.

You only showed one screen where you only defined the tag, but no interface and network using it.

 

consider the network at br0 as the untagged/default PVID net....this is working

create the same setup for VLAN-150...mimik the setup from br0

Example:

unraid-br0:  IP: 192.168.0.50, netmask: 255.255.255.0 (192.168.0.0/24),  gateway: 192.168.0.1 /virtio-pfsense-opt3

unraid-br0.150 192.168.150.50 netmask 255.255.255.0 (192.168.150.0/24) gateway 192.168.150.1 /virtio-pfsense-opt???

 

-> with VLAN-ID: 150 create an interface on pfsense and add IP/net 192.168.150.1/24 - also in unraid add IP 192.168.150.50 with gateway 192.168.150.1

...then test if are you able to ping unraid-br0.150 (192.168.150.50, from IF 192.168.150.1 and vice versa.

 

only after that, try to ping accross networks with routing&firewall involved.

[samba_69]

Things are working when am passing a dedicated nic (virtio <-->br0.xxx) to pfSense on br0.xxx. With this config I am able to ping Unraid on br0.xxx VLAN.

 

But, things are falling apart when am not passing a dedicated nic to pfsense, and I define VLAN on opt3 (virtio <--> br0) 

 

But is it not possible to do this without passing a dedicated nic to pfSense?

[Ford Prefect]

Yes, I repeat.

As you only have created the VLAN-tag, but you have not created the IP connectivity and associated VLAN interfaces, when using one virtio as trunk..

 

VLAN is Layer-2, so when using VLANs, each ethernet frame (and *not* IP-packet) gets tagged. The tag is what separates the virtual ethernet connections inside a physical ethernet connection/wire.

In order to extract/split each VLAN from inside a physical connection, you need to create a (virtual - other than using another  virtio, which will look like a physical NIC in your VM). 

Then add an IP to that virtual (VLAN) NIC,

When using individual virtio adapters attached to the individual unraid bridges each, you simply make each NIC used as an Acces-port, not a trunk port.

 

When using a single virtio as trunk, attached to unraid br0, you will receive packets that are tagged (from br0.xxx) and that are untagged (br0 - VLAN default, PVID=1)

Only the IP packets ones from VLAN-ID=1 will be seen on that NIC in pfsense on layer 3.

In order to use IP packets from VLAN 150 (unraid br.150 via br0 trunk) you need to *add* another VLAN *interface* 

See: https://techexpert.tips/pfsense/pfsense-vlan-configuration/

...what I gather from your screenshots, you only did the first step, not the second (and third, whjich is adding an IP to the VLAN 150 Interface in PFSense and defining the IP network used over VLAN 150).

 

Now, did you try all these steps or did you just perform the first?

 

Edit: do not get confused between physical and virtual NICs between pfsense on the inside and on the outside (where you use virtualised NICs - virtio, as your pfsense is a VM).

Example: In the screenshot above the OPT1 interface is a virtual interface from inside the pfsense as it is created *on* a physical NIC (em0 - which is your opt3 in your pfsense-VM, from the virtio-br0).

For VLANs, just ignore the fact, that this pfsense is a VM.

Edited June 26, 2021 by Ford Prefect

[samba_69]

9 hours ago, Ford Prefect said:

Now, did you try all these steps or did you just perform the first?

I have followed all the 3 steps. I have independent networks created for each VLAN, with their own Static IPv4 and DHCP
 

 

Here are the screenshots for VLAN 1

 

 

Unraid settings:

 

pfSense pings:

 

[Ford Prefect]

....you added VLAN-ID = 1 but did not add ID=150, which you originally indented. 

instead you added vtnet3 (virtio/br.150 I guess)

This scenario works, you said or does it not.

However, what you wanted and "we" are talking about here, is to use a trunk (on vtnet0/br0)

 

VLAN 1 is - as far as i know - the default accross all  network components, when untagged traffic arrives/ingress.

I'd try to not use it explicitly unless you know what you are doing.

On 6/25/2021 at 9:12 PM, samba_69 said:

I can ping unraid from pfSense on br0,

...what is your unraid IP on br0 then?

 

On 6/25/2021 at 9:12 PM, samba_69 said:

But cannot ping on Vlan/br0.150

by the looks you have completed the steps, but did this for VLAN1 but not 150.

But I can only go be the pics compared to that other howto I linked...as said, I do not use pfsense and by the looks of it, the concept is completely different from that of a linux bridge.

Maybe you should get rid of vtnet3 and recreate it as VLAN150 on vtnet0 then.

 

On 6/25/2021 at 9:12 PM, samba_69 said:

What I believed was br0 behaving as a trunk, and any tagged packets in trunk would be untagged and moved to own switch, but this is definitely not the case here

...maybe I overlooked that...what do you mean by "...would be untagged and moved to own switch".

Are you talking to that external switch of yours?

Is this switch VLAN enabled?

 

I'd concentrate on pfsense and unraid link via virtio/br0 first....leave everything else out of the equation, as this might cause side effect...one step at a time. 

Enable pfsense to be able to ping the individual unraid IPs on br0 *and* br.150 ... then you can expand from there.

[samba_69]

was working on VLAN id  1 right now for test, now I switched to VLAN id 15 

 

Everything is configured for vlan 15, as well as unraid
Am also testing vms on br0.15 now

 

unraid on br0 is 10.10.10.10

unraid on br0.15 is 10.10.15.1

 

Vlan 15 is have opt3 as parent interface, which connects to br0

Tried pinging 10.10.15.10 (Unraid on br0.15), but failed

 

1 hour ago, Ford Prefect said:

.what do you mean by "...would be untagged and moved to own switch"

Am epecting br0 (virtual bridge within Unraid) to be a VLAN enabled bridge which can move tagged messages in to their respective VLANs

[Ford Prefect]

4 hours ago, samba_69 said:

was working on VLAN id  1 right now for test, now I switched to VLAN id 15 

 

Everything is configured for vlan 15, as well as unraid
Am also testing vms on br0.15 now

 

unraid on br0 is 10.10.10.10

unraid on br0.15 is 10.10.15.1

...unraid is on 10.10.10.10 (br0) and on br.15 it is 10.10.15.1 or is it 10.10.15.10 ??

4 hours ago, samba_69 said:

 

Vlan 15 is have opt3 as parent interface, which connects to br0

Tried pinging 10.10.15.10 (Unraid on br0.15), but failed

...OK, and you did set pfsense 10.10.10.1 (opt3) and 10.10.15.1 (vlan15) IPs and  as gateway for each network (10.10.10.0/24 and 10.10.15.0/24) in unraid on the respective interfaces?

 

4 hours ago, samba_69 said:

Am epecting br0 (virtual bridge within Unraid) to be a VLAN enabled bridge which can move tagged messages in to their respective VLANs

Yes, this is what unraid br0 with enabled VLANs does. I am using it just fine this way.

 

If you did set both sides up, I can only think of virtio.net as the cause.

As said earlier, for tests, you could physically connect unraid on-board NIC (br0) and physical NIC#4, that is passed through to your pfsense, with a patchcable...this will take virtio-nic out of the equation.

This kind of setup I used to run with a virtualised Mikrotik RouterOS (CHR) for years and it worked (no virtio) flawlessly.

I strongly do believe, that the problem still lies in pfsense and not virtio through, but I cannot help with pfsense better, as I already said. I am sorry.

Maybe you should ask in a pfsense forum, then.

[samba_69]

I did all as instructed, but all fails

You may be right! May be virtio is the culprit here which is not passing vlans  

 

let me deep-dive into this in pfSense's forum