Immense difficulty getting network card working with OPNsense or PFsense (running in VM)

[suchamoneypit]

Tighten your seat belts fellas, this is a long post. Its long given I have spent probably 8+ hours and worked with several people and I cannot get my OPNsense or PFsense working. I wanted to provide as much info as I could in this post as I have spent a lot of time trying many things already. This is kinda driving me crazy at this point. In a nutshell, if I pass through the entire PCI devices, no interfaces will detect and I can't test anything. If I passthrough the NIC ports via a bridge, they detect and work, but only the WAN port will function, and the LAN port will not. The WAN port will instantly grab an IP on all 4 ports of my NIC, but no matter what is done, LAN will not work on any port, either with DHCP or static addresses. I cannot access the webUI or get an IP.

 

The NIC is a quad port 2.5GB RTL8125 card. Everything Im seeing seems to indicate OPNsense and PFsense support this card, although years prior it did not. I am using the latest versions of OPNsense and PFsense. I have tried OVMF and Q35, and I440fx and Seabios on both. Multiple versions tested. 

 

This is the hardware setup. 

 

Note, I have also tried everything with a single cable from the gateway/router to the WAN/Net3 port, as well as the desktop computer directly to the LAN port. I tried the LAN port on all 4 ports. Same behavior on all setups, all cabling, all VM bios/settings. for LAN, I tried the 10.0.1.1 scheme and 192.168.1.1. Also note I am aware I am probably double NATing, but I don't care right now. I just wanted a proof of concept and confirm its functioning before putting the gateway into bridge mode and switching my entire network to my server. 

 

Trying PCI passthrough, it simply does not work. The VMs never see any interfaces. To get my IOMMU groups broken up, I have PCI ACS override enabled.

 

 

I believe PCI passthrough is correctly done:

               Note: the motherboard itself has a 2.5GB ethernet port using RTL8125. This port is for unraid. It is group 33, used by unraid. 

 

When creating the VMs, the ethernet controllers properly show up and allow me to passthrough:

 

OPNsense and PFsense say the same, no interfaces. I cannot proceed to setup. 

 

 

 

Trying the non-PCI passthrough route, the other thing I tried was disabling all the PCI stuff, and assigning them through bridges. This is where I got the interfaces detected, but only WAN would function and LAN would not. 

 

I did this, once for every port. Only 2 are shown in this screenshot:

 

They were then assigned to the VMs (again, tried with both BIOS types, OPNsense and PFsense):

        note: for network model I tried every single one, same behavior. 

 

I am not sure why PCI passthrough doesn't work whatsoever. I am more confused why when using the bridges, WAN works and works great on all 4 ports of the NIC, but the LAN port refuses to work in any configuration on any port. I thought maybe I got a bad NIC (this is what I purchased), but WAN works perfect on every single port. Like I said, I have spent a significant amount of my time trying to get this working and have been unsuccessful. I am not an unraid or networking pro, so its possible I'm missing things and I won't understand complex responses fully. I am more than willing to learn and very grateful for any help provided. I had 2 people more experienced than me go through everything with me closely and they both were left confused as to why my setup was not working.