Jump to content
pankleks

pfSense does not see my NIC's, but Windows VM does

24 posts in this topic Last Reply

Recommended Posts

Hello and thanks in advance for help.

 

I'm trying to run pfSense in VM on UNRAID.

I have Intel 4-port PCI-E card, which is seen in 2 separate IOMMU groups as:

 

image.thumb.png.83fb2bcd3a6b9b64c05f19fa578e2275.png

 

Card is also removed from UNRAID usage in config with `vfio-pci.ids=8086:10e8`

 

All 4 NIC's are passed to VM:

image.png.08b89e5dd8f561c14110f57f96d7d177.png

 

But after running pfSense it sees only single NIC:

 

image.thumb.png.bc0af002972440654776a81a5c7daa8d.png

 

Interestingly, when I create Windows 10 VM and passthough all 4 NIC's, all of them are visible and usable under Windows VM.

 

I'm running UNRAID 6.8.0 on Ryzen 5 2600 with ASRock B450m Pro4-f.

pfSense VM runs on QEMU64 due to: 

 

I'm kind of lost, any thoughts? Can I provide some additional data to help investigating?

 

Thanks!

 

 

 

Share this post


Link to post

just tested on Ubuntu 18.04 VM, passthrough NIC's also seems to work, so I think it's more related to pfSense itself rather than passthrough:

 

image.thumb.png.f970b85167955780a0a9bca557b4c15b.png

 

Share this post


Link to post

No, only mother board NIC - which is not used as interface into pfSense.

 

4-Port card is disabled for UNRAID in settings, with: vfio-pci.ids=8086:10e8

Share this post


Link to post

Ahhh, sorry I missed that in your original posting. 

I’m sure you already used this tutorial.  I would watch it again particularly starting around the 9:40 mark and see if you missed something. This is what I used and have a similar hardware setup as you.  I didn’t have any issues.

 

 

 

Share this post


Link to post

Thanks, I used this exact tutorial, since I had issues, I watched it several times ... I think I did not missed anything.

Edited by pankleks

Share this post


Link to post

Hi, I came here looking for an answer to this exact problem.

My test H/W is a little different:

Optiplex i5 7010, 16gb RAM with HP Quad port server NIC.

pfSense is not seeing any of the NIC's when passed through, but Windows 7 Pro is.

 

I also removed the quad card from the PC and tried using just the builtin Intel NIC,

but in my case pfSense doesn't see the br0 or the virbr0 NIC's either.

I'm using using pfSense 2.4.4.iso and Unraid 6.8.0 trail, both downloaded today.

 

Share this post


Link to post

I changed previous Intel card to newer Intel card (I350 quad port) - which is listed in supported devices for `pfSense`, but this did not helped.

Still VM is seeing only one port :(

 

I lunched `dmesg` in VM and only suspicious things I see:

 

image.thumb.png.eca834debd8878f990cf98d9f4bc53aa.png

 

and:

 

image.thumb.png.166a6f449d36fd2acf770cebb1f561d5.png

 

and:

 

image.thumb.png.e32908964fab1796fdb4a8d6f063cb61.png

 

Any ideas? Thanks!

 

Share this post


Link to post

I had this exact same problem and got it to work. Can you post your VM xml? Maybe I can help.

Share this post


Link to post

I’m having that problem to. Can I post my VM xml? Maybe you guys can help me?

I followed the same tutorial from Spaceinvader one.

Share this post


Link to post

ok, below is the xml of one of your passed through devices.

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>

the address type is inserted automatically by unraid and its generating a seperate bus for every hostdev. Looking at the error log I assume pfsense does not correctly allocate those. 

You can try the following which worked for me:

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x2'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x3'/>
    </hostdev>

 

I just changed the bus for the following entries to "0x01" (value of the first entry, which I assume is allocated correctly in pfsense) and additionally increased the function value for every entry.

Afterwards I reset pfsense from the command line and after rebooting all passed through network cards were available. I hope this helps.

  • Like 2
  • Thanks 1

Share this post


Link to post

I got PFsense to see my 4 port Intel card now thanks to your suggestion, But its saying, No link-up detected when I tried to connect my wan interface?

 

 

52A6709D-BE80-436D-B497-31B622B3B1F6.jpeg

Share this post


Link to post

Glad it worked. Can't help much with the no link-up issue though. You can try to manually enter the WAN interface (4 tries at worst ;)) and see if that works.

Share this post


Link to post

Pankleks I have also a split iommou group but I can only get two ports to work did you get 4 to work?

Could you share your final XML?

 

I am going crazy haha, Thanks in advance!

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
    </hostdev>

This works and two ports are seen... but when I add my two remaining ports (split into diffrent group) like so:

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x1'/>
    </hostdev>

then still only two interfaces show up.

Edited by thierrybla

Share this post


Link to post

@thierrybla Have you tried giving all cards the same bus with different function value like this?

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x2' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x3'/>
    </hostdev>

 

Share this post


Link to post
On 1/8/2020 at 11:04 AM, user2352 said:

ok, below is the xml of one of your passed through devices.

 


    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>

the address type is inserted automatically by unraid and its generating a seperate bus for every hostdev. Looking at the error log I assume pfsense does not correctly allocate those. 

You can try the following which worked for me:

 


    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x2'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x3'/>
    </hostdev>

 

I just changed the bus for the following entries to "0x01" (value of the first entry, which I assume is allocated correctly in pfsense) and additionally increased the function value for every entry.

Afterwards I reset pfsense from the command line and after rebooting all passed through network cards were available. I hope this helps.

This helped me too, thanks a bunch!

Share this post


Link to post
On 1/9/2020 at 10:44 AM, Jb711 said:

I figured out the no link-up issue. 
Thanks for the help!!

What was the no link issue, I am having the same thing on a pfsence I'm trying to set up.

Share this post


Link to post
On 3/6/2020 at 6:44 PM, kakali said:

What was the no link issue, I am having the same thing on a pfsence I'm trying to set up.

Well I'm running into the same problem, except 2 of the 4 interfaces are working. When I plug a cable into port 0 & port 1 (ports farther from PCIe pins on the card), I see blinking lights and can see the status change from down to up in the pfsense interface, but when I plug into ports 2 & 3, the link lights on the card go solid (green & orange) and pfsense does not change from down.

 

Anyone else figure anything more out on this?

Share this post


Link to post
On 5/8/2020 at 1:25 PM, Douglas_D said:

Well I'm running into the same problem, except 2 of the 4 interfaces are working. When I plug a cable into port 0 & port 1 (ports farther from PCIe pins on the card), I see blinking lights and can see the status change from down to up in the pfsense interface, but when I plug into ports 2 & 3, the link lights on the card go solid (green & orange) and pfsense does not change from down.

 

Anyone else figure anything more out on this?

So it turns out that the CLI after setting up pfSense doesn't detect the other 2 NICs, but once I got into the GUI, I was able to enable each port and see that they worked, so if you run into this problem, just keep pushing forward into the next stages and check if things work after configuring things from the GUI.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.