So I've been working on trying to virtualize my pfSense router for coming up on a week now and I'm starting to get there, but there's one really weird issue I can't seem to nail down. I have a Dell R710 and an Intel E1G44ETBLK Pro/1000 ET PCI-E Quad Port NIC that after much headache, I managed to pass through at a hardware level. If someone finds this thread later and is having similar problems passing through their NIC, I ended up needing to add vfio-pci.ids=[id of NIC], [id of bridge in the same IOMMU group as nics] vfio_iommu_type1.allow_unsafe_interrupts=1 to the append line of the syslinux.cfg file. I don't know if that's the best/only way to get it to work, but it is the only way that it would work for me.

 

Now onto the reason I opened this thread in the first place. After passing through the 4 hardware ports to a pfSense virtual machine, I set up igb0 as my WAN, I set up igb1 to my LAN, and I set up a handful of bridges using the following commands to try and have different networks for different types of vms/docker containers.

 

brctl addbr br9_Nickname

ip link set dev br9_Nickname up

 

All the bridges are named things like br9_TestNet, br9_Tools, br9_Media, etc. This allows me to quickly see what the bridge's purpose was, and the br9 at the beginning makes it show unRAID will show the interfaces in the VM screens. If I left off the br9 part, the wouldn't show up I feel like I could have still manually added them by editing the XML, but I found this route easier. I have several computers plugged into the switch that my LAN connection is plugged into, and those computers have absolutely no problem connecting to the internet. However, the VMs I have on the br9_ bridges are all doing the same thing. If I send the command "ping 8.8.8.8" it pings just fine. If I send the command "ping google.com" it resolves and pings just fine. Same results with ipinfo.io and its ip. However, if I try something like "curl ipinfo.io/ip" or "wget ipinfo.io/ip" it tries for a while, but eventually times out. If I try an apt-get update, it tries to connect to the ubuntu.com apt repos, but again, it times out. I'm able to ping us.archive.ubuntu.com just fine, I just can't read from it.

 

Does anyone have any idea what step I'm missing? I'm not sure what other information you might need, so if you need anything else, please ask. Thanks in advance.

 

EDIT: As it usually works out, the problem is that I didn't ask someone how to do it. Right after I posted this, I found the missing piece. I needed to also run the following command on each of the bridges:

ip addr add XXX.XXX.XXX.XXX dev br9_Name

 

Once I gave the bridges an IP address, curl, wget, apt-get, and everything else started working properly.

 

EDIT 2: Actually, there may be one step I didn't mention that makes a difference and may not be ENTIRELY obvious. You need to set the address to something other than the ip address for pfSense and it needs to be the gateway on your devices. So for example, if your bridge in pfSense uses 192.168.15.1/24, you would type ip addr add 192.168.15.2 dev br9_Name, and then in your dhcp config or in your os, you would set the default gateway to 192.168.15.2

 

EDIT 3: I feel like with as much effort as it has taken me to figure all of this out, I will be writing a guide for pfSense on unRAID in the following days. For now, just adding a note on how to get Docker containers to work properly. Given the example from EDIT 2, the corresponding docker network create command would look like this:

 

docker network create -d bridge --subnet 192.168.15.0/24 --gateway 192.168.15.2 -o parent=br9_Name Name

You would then go into the docker container you want to add, and click Advanced settings, then add something like this to the Extra parameters box: 

--network Name --ip 192.168.15.3

Edited October 30, 20178 yr by bamhm182

unRAID 6.4 supports macvlan (custom) networks from the GUI. Any manually created custom networks will interfere with the automatic creation done in 6.4.

 

Thanks for the info! I've had a few reasons to want to upgrade to 6.4. looks like I've got a real good reason now. I'll check it out later tonight.

So I installed 6.4.0_rc10b, and I'm not seeing what exactly you're talking about. Under the Network Settings, I see where you can set up a routing table now, but it doesn't actually let me set up new routes unless the gateway already exists. The gateways seem to be the bridges I was creating with the method I mentioned earlier, and I'm not seeing a way to create new macvlan gateways in the webgui. Is there something I am missing?

When the docker service is started, it will automatically create custom (macvlan) networks for any interface it finds with an IP address associated. The custom networks get the same name as the original network. E.g. br0, br0.10, eth1.

 

When setting up a container you can choose which custom network to attach, plus the ability to set either a fixed or automatic IP address assignment.

 

Okay, awesome, thanks for clearing that up for me. I thought it was custom networks all around. Not just for Docker containers. In that case, it seems to me like you can go into the command line and run the following 3 commands:

brctl addbr br9_name

ip addr add 192.168.0.2/24 dev br9_name

ip link set dev br9_name up

Docker will then create its network corresponding to that virtual bridge automatically when it starts, and then you're good to go. Testing it out on my box seems to work as I would expect it to, however, there's an issue where if the bridge name has capitol letters in it, it will show up as a network you can select, however, when the docker container goes to get recreated, the command will fail because it converts br9_Name to br9_name and it is case sensitive, therefore, br9_name doesn't exist. I don't know if this is something that would be fixed in your Dynamix code, or elsewhere, but it seems to me like the one of the following things could fix it:

• Filtering networks so those with capitol letters don't get automatically created
• Making the docker run command uses the network's name with the possibility of capitol letter

Either way, thank you for all of your hard work, bonienl! Keep it up!

 

EDIT: After playing around with this more, it looks like in 6.4, personally created networks like I did with the above commands are no longer persistent across reboots. Is it a terrible idea to add the commands to my go file so they're recreated upon reboot?

Edited October 31, 20178 yr by bamhm182

I believe everything you want to do can be achieved through the network functionality present in the GUI. Not sure why you want to go the (difficult) manual way.

 

I want to create a bridge that is not associated with a physical NIC. Is that possible to do in the GUI?

 

I guess I could just set up a bunch of vlans on one of the physical ports and then just not have anything plugged into it. I should still be able to attach pfSense to each of the vlans and have it do my routing. I was just thinking the better way to do it would be to create virtual NICs, but for simplicity sake, maybe just creating a bunch of vlans on an unused port would be better.

Edited October 31, 20178 yr by bamhm182