[Solved] pfSense VM do not recognize virtio network interface (e1000/vmxnet3 is marked down)

[samba_69]

I have been trying for a couple of days to pass a virtual nic to pfSense (2.5.1) vm on my Unraid 6.92

I tried various settings including changing network interface model to 

             > virtio/virtio-net: cannot recognize the virtual nic. Based on pfSense doc here, virtio should be supported out of the box installation. Based on the instruction I also checked the "Disable hardware checksum offload" within pfSense settings.

             > model type='e1000-82545em': The interface visible but is always down and I cannot get the interface up and running

             > model type='vmxnet3': The interface visible but is always down and I cannot get the interface up and running

 

Requirement:

I actually want to access Unraid from pfSense's local network. I also want other vms/dockers on unraid to be on pfSense's network (different from home lan network), thus I would need a virtual switch within unraid to which other vms and dockers would connect to.

 

You can suggest alternative ideas too (but not connecting to router from one interface and connecting back to unraid on main interface using two physical cables) 

 

Any help on this would be highly appreciated!

Edited June 18, 2021 by samba_69
changed image to reduce confusion

[samba_69]

Based on a reddit post I updated the VM settings and switched Machine to 'Q35-2.4' (from 'Q35-5.1') and eventually I was able to have 'virtio' interface visible to pfSense.

 

 

But still the port is down and I am not able to connect to Unraid from pfSense's lan network!

[SimonF]

11 minutes ago, samba_69 said:

Based on a reddit post I updated the VM settings and switched Machine to 'Q35-2.4' (from 'Q35-5.1') and eventually I was able to have 'virtio' interface visible to pfSense.

 

 

But still the port is down and I am not able to connect to Unraid from pfSense's lan network!

You will need to change the XML for ports to be e1000 from virtio I think. There is a fix coming in the next release to allow e1000 to be selectable from VM template.

 

Change the view to xml and then find the network interfaces and change type.

[samba_69]

20 minutes ago, SimonF said:

You will need to change the XML for ports to be e1000 from virtio I think. There is a fix coming in the next release to allow e1000 to be selectable from VM template.

 

Change the view to xml and then find the network interfaces and change type.

 

I have already tried that. But pfSense says that the port is down as you can see here.

56 minutes ago, samba_69 said:

 > model type='e1000-82545em': The interface visible but is always down and I cannot get the interface up and running

 

Any suggestions how to get a port up and running for pfsense?

[SimonF]

7 minutes ago, samba_69 said:

 

I have already tried that. But pfSense says that the port is down as you can see here.

 

Any suggestions how to get a port up and running for pfsense?

Options I added to the VM Template in the gui are

 

echo mk_option($arrNic['model'], 'e1000', 'e1000');

echo mk_option($arrNic['model'], 'vmxnet3', 'vmxnet3');

 

Have you tried just e1000 without -8254em?

Edited June 18, 2021 by SimonF

[samba_69]

I just tried "E1000" without -8254em,

<interface type='bridge'>
  <mac address='52:54:00:1b:13:df'/>
  <source bridge='br0'/>
  <target dev='vnet0'/>
  <model type='e1000'/>
  <alias name='net0'/>
  <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</interface>

 

but it didn't help. The port is still down!

 

 

I am also attaching my Unraid network setting, incase there is something wrong there

 

[SiNtEnEl]

I wont use the Q35 untill bug 238922 in BSD (https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=236922) is properly fixed, there have been commits done to fix it, but this is on BSD 13.0, and PFsense / Opnsense are still on the 12.2-STABLE train.

There are indeed some workarounds to run a virtio device under q35 on a PCi bus. But i gave up on that, i had similar issues like you have and weird behavior with packet ordering under in my case opnsense. Opnsense still has the advice not to use the q35 either, so if u get it working.. dont get your hopes up it will be error free.
 

So basicly im fallback running virtio-net (10Gbase-T) under i440fx-5.1 and have no issues getting 1GBe on my internet pipe and routing traffic over my 10GBe network with decent speeds. My personal advice will be switch to i440fx-5.1.

 

[samba_69]

Thanks for your suggestions. I just tried a fresh pfSense install with i440fx-5.1, but still virtio port is marked as down (ref: image)! 

 

Can you please suggest way I can make this virtio port up?

[SiNtEnEl]

Can you do a ifconfig br0 in the shell of unraid, see what it reports there?
You could just try to do: ifconfig br0 up
See if that fixes it

Edit: if it's up, and issue is still there do this in the shell: virsh iface-list
See if the br0 is active or not. If its active we need to check in the vm.

 

Edited June 18, 2021 by SiNtEnEl

[samba_69]

20 minutes ago, SiNtEnEl said:

ifconfig br0 in the shell of unraid

I just did, and br0 is UP. Here is what I get form ipconfig br0

 

br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.14.134  netmask 255.255.255.0  broadcast 192.168.14.255
        ether 1c:fd:08:71:2b:01  txqueuelen 1000  (Ethernet)
        RX packets 383670  bytes 334981815 (319.4 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 299769  bytes 355025216 (338.5 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

 

This is the local ip of the Unraid server

 

I am so very confused trying to understand what is going wrong!

[SiNtEnEl]

br0 is the bridge interface that is used by Qemu, what is the hypervisor that runs your pfsense VM.
Where checking if the interface is running correctly on the host, in your case unraid.

If u do in the shell of unraid: virsh iface-list
You will get the list of interfaces that are used by qemu, that u can link to a vm and if its active or not.
My guess its up based on the ifconfig. So its active, then its a issue is likely inside the vm.

But seeing your trying to configure the interface in pfsense from the shell menu and listed down is not normal behavior. (not for me atleast, when i run pfsense)
You could try and configure it in pfsense and see if it comes up then.

 

[samba_69]

Thanks @SiNtEnEl

Based on your feedback, I was able to enable virtio from pfSense setting, as you can see in the image below:

I had to manually check the enable interface that eventually marked the virtio port up!

 

I have set the vnet interface as static IPv4 with gateway 192.168.2.1 (my laptop is on LAN interface with 192.168.1.1 gateway)

 

But now I ran into another problem now! I cannot Unraid obtain dhcp lease on this port, and hence cannot reach Unraid.

 

Even I dont see Unraid in DHCP lease table as you can see! Only my laptop is have a DHCP lease

 

Note: Unraid gets DHCP allocation successfully from eth0 on br0

ref:

15 hours ago, samba_69 said:

I am also attaching my Unraid network setting, incase there is something wrong there

 

I am probably doing something wrong! Any guidance?

 

[samba_69]

I was able to connect to Unraid by providing static IPv4 to Unraid in Settings >> NetworkSettings >> IPv4 address assignment <-- static ip

 

Thanks all for your co-operations and help!

@SimonF looking forward for E1000 and vmxnet3 in upcoming release! It will make life much easy!

[SiNtEnEl]

Almost sound like your blocking dhcp on the firewall (port 67).
Static ip is for me the preferred solution regarding Unraid.

Glad you have a working setup now.

[samba_69]

Not very sure how DHCP is blocked, as whenever I connect a device to Unraid's port (on br0), pfSense's DHCP successfully allocates ip to the new device via virtio/br0. Its just the Unraid that cannot get the IP

 

[Squirrel_CA]

I was having a similar issue with my Virtio port showing down. You can fix this by clicking on the Shell icon on the top right, then typing this in:

 

virsh net-start default

 

That brought my interfaces up. Now I just need to get access to Unraid from the virtio network. I am using a PCI ethernet adapter as my WAN interface and then I want to use the Virtio-net interface for my LAN interface. So I will be troubleshooting this tonight. I started working on this last night at 7pm and went to bed at 2am this morning. I am running a HP Proliant DL380p G8 , I tried splitting up the 4 port NIC using ACS but unfortunately for what ever reason, all 4 ports have the same serial, so even if I split them into separate IOMMU groups, restart, then add each port individually to PFSense, when PFSense starts up, it just shows up as one port. 

 

EDIT: Just to be clear, my Virtio-net is using my servers 4 port network card, the one I mentioned above: HP 4-port 684208-B21 Ethernet 1Gb 331FLR Adapter

 

The issue I have is that PFSense is a VM, so when Unraid is rebooted, PFSense starts after Unraid (Obviously), but due to this, Unraid marks the NICS as "Shutdown-inactive" because nothing is currently using the ports, this causes a chain effect, where PFSense looses connectivity to that port now when the VM auto starts because Unraid has turned that port off. I have to manually start the port, then reboot PFSense to see the port then redo the interface assignments again.

 

I might be doing something wrong, I will tackle this again tonight when I get home.

 

EDIT: I will document my steps and post my config if I manage to come right tonight.

Edited September 22, 2021 by Squirrel_CA

[Squirrel_CA]

Managed to get it working last night. My main issue was that I would get PFSense working but then I could not access Unraid from my laptop while on the network, so I would have internet access but 192.168.1.2 (Unraid) would not be reachable. I would have to manage it directly from the server itself with a monitor, keyboard and mouse plugged into the server.

 

Ok so my issue was that my HP 4 port switch shows all 4 ports as the same ID. Therefor I cannot pass all 4 though to PFSense as individual ports (PFSense sees all 4 as just one port, even though I separated all 4 into separate IOMMU groups). Because of this I had to add 2 more pcie 1G ethernet ports (1x for WAN and 1x for LAN). 

 

First to get the 4 port onboard network card to work on PFSense, you need to set bridge to true on the port you going to use for PFSense (in my case eth0 bridge set to yes). Next you need to statically assign an I.P to this port. I.E 192.168.1.2. (Note that this will only show up as 1 port on PFSense. You can just bond all 4 ports in Unraid if you like for better throughput. Will need to bond the ports on the switch side too or else the ports will not be accessible *Supported switch will be required to do this*)

 

So now on PFSense Console it looked something like this:

 

Virtio (this is the bridge port on Unraid "eth0")

re0 (WAN)

re1 (LAN)

 

I then set re0 as WAN and re1 as LAN and Virtio as Optional1 (Opt1)

 

Next I went to "Interfaces" on PFSense and added a new interface and set a static I.P address, set DNS to PFSense I.P and clicked on Done. (Just to be safe I statically assigned Virtio to 192.168.1.2 DHCP leases too. Its fairly easy. Click on the + sign on the right hand side of any lease, it will open a new window and auto fill that leases MAC address. Now delete that MAC address and add Virtio MAC address and click on done. (services - DHCP Server -  Leases).

 

To do this, check the MAC address assigned to Virtio (can be done by clicking on Settings in Unraid - Network settings, and then scrolling down right to the bottom and there will be a list of all interfaces with MAC addresses. Alternatively should be able to see it by clicking on Edit on the PFSense VM in Unraid and going to the network section at the bottom).

 

Now I have Network connection on the network and I am able to manage Unraid as well from the network. I will see if I can EDIT this post when I get home tonight and add screenshots of my config to help anyone in the future. I ran out of time last night, ended up coming right at 1am.

 

EDIT: OH and if at anypoint during this process, you get an error when trying to start your PFSense VM that says: 

Just open the console in Unraid(top right hand side) and enter the following: 

 

Quote

virsh net-start default

 

This will sort it out for you.

 

Edited September 23, 2021 by Squirrel_CA