pfSense does not see my NIC's, but Windows VM does


pankleks

Recommended Posts

Hello and thanks in advance for help.

 

I'm trying to run pfSense in VM on UNRAID.

I have Intel 4-port PCI-E card, which is seen in 2 separate IOMMU groups as:

 

image.thumb.png.83fb2bcd3a6b9b64c05f19fa578e2275.png

 

Card is also removed from UNRAID usage in config with `vfio-pci.ids=8086:10e8`

 

All 4 NIC's are passed to VM:

image.png.08b89e5dd8f561c14110f57f96d7d177.png

 

But after running pfSense it sees only single NIC:

 

image.thumb.png.bc0af002972440654776a81a5c7daa8d.png

 

Interestingly, when I create Windows 10 VM and passthough all 4 NIC's, all of them are visible and usable under Windows VM.

 

I'm running UNRAID 6.8.0 on Ryzen 5 2600 with ASRock B450m Pro4-f.

pfSense VM runs on QEMU64 due to: 

 

I'm kind of lost, any thoughts? Can I provide some additional data to help investigating?

 

Thanks!

 

 

 

Link to comment

Hi, I came here looking for an answer to this exact problem.

My test H/W is a little different:

Optiplex i5 7010, 16gb RAM with HP Quad port server NIC.

pfSense is not seeing any of the NIC's when passed through, but Windows 7 Pro is.

 

I also removed the quad card from the PC and tried using just the builtin Intel NIC,

but in my case pfSense doesn't see the br0 or the virbr0 NIC's either.

I'm using using pfSense 2.4.4.iso and Unraid 6.8.0 trail, both downloaded today.

 

Link to comment

I changed previous Intel card to newer Intel card (I350 quad port) - which is listed in supported devices for `pfSense`, but this did not helped.

Still VM is seeing only one port :(

 

I lunched `dmesg` in VM and only suspicious things I see:

 

image.thumb.png.eca834debd8878f990cf98d9f4bc53aa.png

 

and:

 

image.thumb.png.166a6f449d36fd2acf770cebb1f561d5.png

 

and:

 

image.thumb.png.e32908964fab1796fdb4a8d6f063cb61.png

 

Any ideas? Thanks!

 

Link to comment

ok, below is the xml of one of your passed through devices.

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>

the address type is inserted automatically by unraid and its generating a seperate bus for every hostdev. Looking at the error log I assume pfsense does not correctly allocate those. 

You can try the following which worked for me:

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x2'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x3'/>
    </hostdev>

 

I just changed the bus for the following entries to "0x01" (value of the first entry, which I assume is allocated correctly in pfsense) and additionally increased the function value for every entry.

Afterwards I reset pfsense from the command line and after rebooting all passed through network cards were available. I hope this helps.

  • Like 4
  • Thanks 3
Link to comment

Pankleks I have also a split iommou group but I can only get two ports to work did you get 4 to work?

Could you share your final XML?

 

I am going crazy haha, Thanks in advance!

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
    </hostdev>

This works and two ports are seen... but when I add my two remaining ports (split into diffrent group) like so:

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x1'/>
    </hostdev>

then still only two interfaces show up.

Edited by thierrybla
Link to comment

@thierrybla Have you tried giving all cards the same bus with different function value like this?

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x2' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x3'/>
    </hostdev>

 

Link to comment
  • 1 month later...
On 1/8/2020 at 11:04 AM, user2352 said:

ok, below is the xml of one of your passed through devices.

 


    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>

the address type is inserted automatically by unraid and its generating a seperate bus for every hostdev. Looking at the error log I assume pfsense does not correctly allocate those. 

You can try the following which worked for me:

 


    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x2'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x3'/>
    </hostdev>

 

I just changed the bus for the following entries to "0x01" (value of the first entry, which I assume is allocated correctly in pfsense) and additionally increased the function value for every entry.

Afterwards I reset pfsense from the command line and after rebooting all passed through network cards were available. I hope this helps.

This helped me too, thanks a bunch!

  • Like 1
Link to comment
  • 2 months later...
On 3/6/2020 at 6:44 PM, kakali said:

What was the no link issue, I am having the same thing on a pfsence I'm trying to set up.

Well I'm running into the same problem, except 2 of the 4 interfaces are working. When I plug a cable into port 0 & port 1 (ports farther from PCIe pins on the card), I see blinking lights and can see the status change from down to up in the pfsense interface, but when I plug into ports 2 & 3, the link lights on the card go solid (green & orange) and pfsense does not change from down.

 

Anyone else figure anything more out on this?

  • Like 1
Link to comment
  • 2 weeks later...
On 5/8/2020 at 1:25 PM, Douglas_D said:

Well I'm running into the same problem, except 2 of the 4 interfaces are working. When I plug a cable into port 0 & port 1 (ports farther from PCIe pins on the card), I see blinking lights and can see the status change from down to up in the pfsense interface, but when I plug into ports 2 & 3, the link lights on the card go solid (green & orange) and pfsense does not change from down.

 

Anyone else figure anything more out on this?

So it turns out that the CLI after setting up pfSense doesn't detect the other 2 NICs, but once I got into the GUI, I was able to enable each port and see that they worked, so if you run into this problem, just keep pushing forward into the next stages and check if things work after configuring things from the GUI.

  • Like 1
Link to comment
  • 2 months later...
On 1/11/2020 at 3:47 PM, thierrybla said:

Pankleks I have also a split iommou group but I can only get two ports to work did you get 4 to work?

Could you share your final XML?

 

I am going crazy haha, Thanks in advance!

 


    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
    </hostdev>

This works and two ports are seen... but when I add my two remaining ports (split into diffrent group) like so:

 


    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x1'/>
    </hostdev>

then still only two interfaces show up.

It worked for me too.

 

Perfect.

 

Only different thing I am doing is that I have only one PCI with 4 Gb Ethernet. I was able to use 2 for UNRAID and pass the 3rd and 4th to pfSense.

 

Thank you guys.

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.