I was doing some research and couldn't find definite answer if my idea is doable or not (maybe my Google-Fu took holidays to seriously ), so I am asking here.
Idea is to configure VM with OPNsense firewall, which will serve whole home network - unRAID with Docker containers and VM (or two); and physical wired and wireless devices, which will access internet via OPNsense and services hosted on unRAID.
I plan segmenting home network to several subnets, so all traffic going to from one subnet to other will traverse OPNsense FW. To optimize network traffic, I would like connect OPNsense and Docker containers via internal/virtual network. Is this possible and if yes, how (any guide, how-to... is appreciated)?
I assume the same can't be achieved for unRAID shares as well? unRAID will be in separate subnet than most of the end devices.

 

Thanks!

Doable yes, not easily done can be a convoluted mess. It is better to pass a pcie nic and use real hardware. Unriad is not a networking device.... unraid 7 rc2 has some nicer netwrok stuff.

However, I have used unraid to build and make "forbiden routers".... Most depending on how you setup other unraid networking stuff ...

 

For starters you want to go to docker settigns and switch to macvlan. enable host access and preserver defined networks...

 

then you want to go to VM manager and make sure you using vhost

*This is becasue you want lan access for the VM and Docker network to access the same subnet as unraid before the forbiden router. this is for recovery and to still have default unraid functions....

In my case when testing as i ahve done this 3-4 times with unraid.... I only had 1 nic on my unraid machine, so I turned off bridging and bonding under unraids network settings... (easier when i had a pcie nic to pcie pass and separate due to unirad networks....)

You may now notice that vhost0@interface name taps and copies your eth0 ip... You should treat it like a service for isolation latter. You may want it at a different IP on that interface latter

 

The reason for that is that Docker and VM will then use the vhost tap for their network traffic.

ip a
look for

vhost0@eth0

This is an esential setup I ahve found due to how other network setings have worked and continue to work on unraid.... So this vhot@eth0 will esentail become the "wan" interface into unraid qemu virtual switch for the hyper V... (the nic will grab a dhcp IP from the subnet unriad is on, that vhost0 is on

So Lets make a New VM and review some things:

 

I chose linux. as the template.

Set your settigns q35 vs ifx atleast 1 GB ram.... 10 GB HDD.... install OS..... etc...

Scroll to networking So in the VM we should see:

click the plus to add a second.

Once you set your settings. don't start the VM after creation and under webUI for the vm click edit and go to xml form a switch at the top right, then scroll down to interfaces.

Example

    <interface type='direct' trustGuestRxFilters='yes'>
      <mac address='52:54:00:4e:e2:14'/>
      <source dev='vhost0' mode='bridge'/>
      <model type='virtio-net'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <interface type='direct' trustGuestRxFilters='yes'>
      <mac address='52:54:00:4c:88:c4'/>
      <source dev='vhost0' mode='bridge'/>
      <model type='virtio-net'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

note the source dev that it says vhost0 this will be important latter. the web UI wo't allow you to set different interfaces.... however manual edits will allow and support this...

Now to the fun stuff, we are going to make a dummy interface. This interface will be the lan connection to the vm.
Latter, this interface will be made into a docker network for use. 

So let's make a bridge interface. let's call it vmbrlan0
 

#!/bin/bash
# Create and configure the vmbrlan0 bridge

# Ensure the bridge module is loaded
modprobe bridge

# Create the bridge interface
ip link add name vmbrlan0 type bridge

# Assign an IP address to the bridge if needed (e.g., for LAN routing)
#ip addr add 192.168.x.2/24 dev vmbrlan0
#depends on your router vm settings...

# Set the bridge interface up
#ip link set vmbrlan0 up
#optional for now we want it down..

# (Optional) Attach vhost0 to the bridge
#ip link set vhost0 master vmbrlan0
#THis will be done in the VM...

*Note your router lan interface ip will be 192.168.x.1 you should set this to that subnets lan ip as 192.168.x.2

Now lets add that nic to your VM's xml as a second nic:
 

<interface type='bridge'>
  <mac address='52:54:00:ab:cd:ef'/>
  <source bridge='vmbrlan0'/>
  <model type='virtio'/>
</interface>

    <interface type='direct' trustGuestRxFilters='yes'>
      <mac address='52:54:00:4c:88:c4'/>
      <source dev='vmbrlan0' mode='bridge'/>
      <model type='virtio-net'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>

*Note the model type changed as well you can use either or... I prefer the 2nd option as it allows unraid qemu firewall help and do other things...

It at this step you can start the VM and install the OS.
https://www.ipfire.org/downloads/ipfire-2.29-core190
I prefer ipfire. I can confirm this with it. Openwrt / opensene work ok as well here...

With that we now have a virtual interface getting the VM router Lan IP

With a VM make sure your interface and driver are that and connected and assigned correctly for the lan and wan.

So Now we want to make a docker network:

Macvlan:

docker network create \
  -d macvlan \
  --subnet=192.168.2.0/24 \
  --gateway=192.168.2.1 \
  -o parent=vmbrlan0 \
  macvlan_network

Breakdown:

-d macvlan: Specifies the MACVLAN driver.

--subnet=192.168.2.0/24: The subnet used by the MACVLAN network.

--gateway=192.168.2.1: The gateway for the MACVLAN network.

-o parent=vmbrlan0: Indicates the parent interface (vmbrlan0).

macvlan_network: The name of the Docker network.

IPVLAN:
 

docker network create \
  -d ipvlan \
  --subnet=192.168.2.0/24 \
  --gateway=192.168.2.1 \
  -o parent=vmbrlan0 \
  ipvlan_network

Explanation:

-d ipvlan: Specifies the IPVLAN driver.

--subnet=192.168.2.0/24: The subnet for the IPVLAN network.

--gateway=192.168.2.1: The gateway for the IPVLAN network.

-o parent=vmbrlan0: Specifies the parent interface for the IPVLAN network.

ipvlan_network: The name of the Docker network

As Host side networking is now done. you should test by making a generic VM and adding the interface and make sure you see a client and getting a ip to the VM...

Problems usually begin around here.... This is why you may here it reference as a forbidden router... makes the forbidden router a convulture mess..
You may need to up the interface in unraid... you may need to add a iproute to the vm under unraid network settings...

Once done and configures...

this now means that in the VMs xml, the vhost0 interface is your wan and vmbrlan0 is your lan and the dockers via the template can be used as it is set to the router vm for a ip and a virtual router for more gradual control...

So in the docker template you would choose the macvlan_netwrok or ipvlan_network custom ip in the docker:

 

There are some docker that run routers or VPNs, in that case you would add the network type container to point to the VPN. I prefer ipfire myself...

as you can see issues can raise with how the dummy interface is setup and other settings from iptables to iproutes to bridge utilities can easily break and add broken interfaces to the Host system...

for this Reason I recommend VFIO PCIE pass a network nic so a vm gets 2 physical Ethernet ports 1 for wan 1 for lan.
And unraid host to have 2 aditional onboard nics. 1 for your routers subnet 1 to the lan either via a switch or looped between the 2 nics...

So a host with 4 nics all together... this way you have a easy known netowrk of eth0 is this subnet and eth1 is my vm subnet...
Not much chages in the create docker steps.

 

Edited December 30, 20241 yr by bmartino1
Data - typo

High-Level Overview

Virtualize OPNsense on Unraid:

OPNsense will act as the central firewall/router for your home network.

All traffic between subnets will be routed through OPNsense.

 

Segment the Network:

Create VLANs or subnets for different purposes (e.g., Docker containers, Unraid shares, guest Wi-Fi, IoT devices, etc.).

Assign specific VLANs/subnets to Docker containers, VMs, and Unraid services.

 

Use Internal Virtual Networks:

Leverage Unraid’s macvlan or ipvlan networks for Docker containers.

Use bridge interfaces or virtio NICs to create high-speed internal networks for OPNsense and Unraid VMs.

 

External Devices:

Physical wired and wireless devices will connect via OPNsense using their respective VLANs or subnets.

 

Challenges and potential Solutions

1. Performance

Using virtual NICs and bridges minimizes latency.

Ensure Unraid and OPNsense VMs are allocated enough CPU and RAM for optimal performance.

 

2. Access Control

Use OPNsense firewall rules to control traffic between VLANs/subnets.

Isolate sensitive VLANs (e.g., Unraid shares) from guest or IoT VLANs.

 

3. Device Compatibility

Configure your physical network (switches, access points) to support VLANs.

 

4. Testing and Debugging

Test each VLAN/subnet for connectivity and access.

Use tools like ping and traceroute to diagnose routing issues.

 

Example Network Layout

WAN (Internet): vmbr0 (Physical NIC connected to ISP)

LAN (Internal Network):

VLAN 10: Docker Containers (192.168.10.0/24)

VLAN 20: Unraid Shares (192.168.20.0/24)

VLAN 30: IoT Devices (192.168.30.0/24)

VLAN 40: Guest Wi-Fi (192.168.40.0/24)

@bmartino1, thank you very much for detailed explanation!

I forgot to mention that I have dedicated 4 port PCIe NIC, passed-through to OPNsense VM. In fact, I already have this part working, now I would like to optimize network traffic, so it doesn't need to pass over switch multiple times - for example: PC with Kodi is in subnet A and Emby Docker is in subnet B. Currently traffic is going to OPNsense, where it is routed from subnet A to subnet B and then forwarded to unRAID's NIC (5th physical network interface on unRAID box (MB built-in NIC)).

OPNsense's LAN port and unRAID's port are both connected to same switch.

Nice. since your using physical nic passed this make things a bit easier. you could stillmake a dumbby interface and atache it but that can become problematic.

more of this may be on opensense and how you are subneting. i would do vlans.

On Unraid under netwrok settings enable vlans:

the -o parent=vmbrlan0 \

would become -o parent=br0.10

opensence would need to support "stp" spaningtree for vlans. over the interface.

you may need to make a docekr then

the interface would then be unriad interface br0, eth0, bond0
with . vlan number

High level overview:

Hardware with swtich Opensne software:

VLAN Configuration on the Switch and OPNsense:

Configure VLANs on your switch for subnet A and subnet B.

Assign the relevant VLANs to the ports connected to OPNsense and unRAID.

Ensure OPNsense routes traffic between VLANs efficiently without physically passing through the switch multiple times.

https://www.zenarmor.com/docs/network-security-tutorials/how-to-configure-vlan-on-opnsense

https://forum.opnsense.org/index.php?topic=37478.0

 

or 

Software level with unraid and opensense nic assigments:

Bridge Networking on unRAID:

Enable bridging on unRAID to allow traffic destined for different subnets on the same NIC to stay internal to the host.

Ensure Docker containers like Emby are correctly bound to the respective VLANs.

 

#####

 

Highlevel overview:

 

Direct Docker Networking:

Assign a dedicated IP for Emby Docker on the VLAN corresponding to subnet A, so that traffic does not require routing.

Alternatively, use macvlan or ipvlan Docker networking drivers to isolate and manage container traffic.

 

Optimized Routing Rules on OPNsense:

Implement policy-based routing or static routes to ensure traffic between subnet A (PC with Kodi) and subnet B (Emby Docker) does not traverse unnecessary paths.

 

Leverage L3 Capabilities of the Switch (if supported):

If your switch supports Layer 3 routing, you can offload inter-VLAN routing from OPNsense to the switch, significantly reducing traffic overhead.

 

I'd recommend using the switch: