pfsense NIC passthrough not working anymore


Yanos!

Recommended Posts

Hi guys,

 

Long story short. I use a Dell T30. I was running unRAID 6.7.2. A VM with pfense was working as it should. I bought a HBA card for my disks along with 2 SSDs and installed the new hardware. The server booted normaly and everything was working fine. About 30 hours after that, my server became unresponsive and thought it died. I have a second T30, so I moved everything in this server. Everything worked fine, except my HP Quad ports NIC that I used for pfsense did not work anymore. The lights of the NIC lit, but the card was closing it down few seconds later. So I thought the NIC died (newbie error. As I changed the NIC for one server to the other, I did not unplug the power cord. When I inserted the NIC in the new server, it started. I thought I created a spark or something). I ordered a new one, same problem in pfsense, no NIC. In the mean time, I was able to resurrect my other server (Windows 2016). Both NICs are working fine... I also upgraded to 6.8 RC7, with no luck...

 

So, for troubleshooting my pfsense VM, I created a new one and I see something weird... I connect one NIC, the VM detects the UP link, but still shows no link detected. Or if I reboot the VM, there is no NIC at all... 🤯

 

I think I'm too close to the tree, so I can't see the forest. What did I missed?

 

Any tips or hint welcomed.

vfio-pci flash drive.JPG

NICs passed to VM.JPG

IOMMU.JPG

No NICs in pfsense - unRAID.JPG

pfsense #1.JPG

voute-diagnostics-20191127-0322.zip

Link to comment

So, as expected, I did not received any comments on this. Let me correct that.

 

I have setup a Windows 2016 VM and the 4 passthrough NICs are working.

I followed the tutorial from Spaceinvader One as the first time and I can't get my pfSense VM detect my NICs. Can it be a driver issue?

 

Any input welcomed.

 

Thank you.

Link to comment
  • 1 month later...
  • 3 weeks later...

I had a heck of a time getting pfsense to see my 4 port intel nic.  Linux could see it fine.  Anyway, I ended up modifying the XML like this:

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/>
    </hostdev>

 

 

Notice now the slot 5s do a function 0x0 and 0x1 and also the same for the slot 6s.  Where before it was just slot 5 6 7 8 and 0x0 for the function.  I hope this makes sense.  The multifunction parts got added on automatically for me.

 

I did pci-stub.ids=8086:10bc in the append line in syslinux config (from flash link on main page) to get the card ready for passthrough.

 

Also if you want to bridge br0 or virbr0 make sure it is using <model type='vmxnet3'/>

 

Hope this helps.  Good luck.

Link to comment
  • 4 weeks later...

So after deleting my libvirt.img thinking I could just readd my pfsense.. I am in quite the pickle.

It was working fine before this on the new OS, but now that I'm trying to create a new VM I cannot pass through both my NICs! Only 1 (bxe0) will pass through

I only have 2 interfaces that need to be used.. A WAN and a LAN.

 

Here's what I have in my xml

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x81' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x81' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>

 

My current work around is using the terrible modem/router from my ISP and it makes me want to cry.

If anybody knows how to potentially resolve this I would oh so happy!

Edited by Partial
Link to comment
  • 1 month later...
On 2/22/2020 at 6:29 AM, Partial said:

So after deleting my libvirt.img thinking I could just readd my pfsense.. I am in quite the pickle.

It was working fine before this on the new OS, but now that I'm trying to create a new VM I cannot pass through both my NICs! Only 1 (bxe0) will pass through

I only have 2 interfaces that need to be used.. A WAN and a LAN.

 

Here's what I have in my xml

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x81' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x81' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>

 

My current work around is using the terrible modem/router from my ISP and it makes me want to cry.

If anybody knows how to potentially resolve this I would oh so happy!

I know this is an old thread, but I have just had similar trying to set up PfSense as a virtual machine.

 

In short a windows vm can see my 4 port nic but PfSense cannot see more than one port on the card.

 

I highly recommend watching the excellent video's SpaceInvader One has created. His video titled Advanced GPU passthrough may be the answer to your problem and also the op.

 

In short study the XML as per the post previously by uek2wooF and described excellently by SpaceInvader One.

 

The card or device you are passing through must be passed in a way to mimic the hardware.

 

With a 4 port intel pro NIC card the virtual bus and slot must be the same with the device marked as multifunction and the different ports corresponding to the different function number.

 

As an example this is the XML for a 4 port nic where all 4 ports are now correctly passed through -

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x43' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x43' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x44' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x2'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x44' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x3'/>
    </hostdev>

Although in hardware the card is split across two bus numbers, mine are bus='0x43' and bus='0x44' but yours may be different, in order to see all 4 ports in a pfsense vm the card must be passed through on just one bus. EDIT I did not select bus number 0x01, this was the bus number as generated automatically within the xml template, yours may be a different bus number to mine.

bus='0x01'

Hope this helps anybody who stumbles upon this thread in the future. :)

Edited by dgs2001
added clarification
Link to comment
  • 5 months later...
On 1/27/2020 at 1:45 AM, uek2wooF said:

I had a heck of a time getting pfsense to see my 4 port intel nic.  Linux could see it fine.  Anyway, I ended up modifying the XML like this:

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/>
    </hostdev>

 

 

Notice now the slot 5s do a function 0x0 and 0x1 and also the same for the slot 6s.  Where before it was just slot 5 6 7 8 and 0x0 for the function.  I hope this makes sense.  The multifunction parts got added on automatically for me.

 

I did pci-stub.ids=8086:10bc in the append line in syslinux config (from flash link on main page) to get the card ready for passthrough.

 

Also if you want to bridge br0 or virbr0 make sure it is using <model type='vmxnet3'/>

 

Hope this helps.  Good luck.

Sorry to quote the entire post and stretch the thread but I wanted to be sure this information is saved for the future. Out of all the information in this thread, this post is the one that got me up and running with my Intel PRO/1000 quad-port NIC. The main difference with this post is that the network card's ports are placed on bus 0 and separated using slots. In the IOMMU groups, the card is split into two pairs on different buses, so I did the whole multifunction='on' thing for both pairs, and set the functions to 0 and 1 for each pair.

 

I _also_ tried this by putting all four ports on the same bus, same slot, different functions, with multifunction on for the two leader ports (0,2 are leaders and 1,3 are secondary to them respectively). Or something. I have no idea what I'm doing but it works. :)

 

So anyway here's my own XML:

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x04' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x3'/>
    </hostdev>

Anyone having trouble should start here and work backwards ^^

Edited by sixsicsix
More better.
  • Like 1
  • Thanks 1
Link to comment
  • 3 months later...

Excellent.  I was worried Google had shunted me to another dead-end thread (as it is wont to do), but this looks like the solution.

 

Interestingly I had this exact same problem with the my Radeon 5700XT GPU, which has a different function for graphics and sound on the same bus.  The xml file automatically puts each function on a different bus, which in essence is splitting single device that isn't supposed to be split.  That's what's happening to our nic basically.

 

For a more thorough explanation of this xml config issue see SpaceInvaderOne's video about advanced GPU passthrough techniques (under five minutes).  Although this video is about GPU's, the issue and the solution are essentially the same.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.