December 28, 20196 yr Hello and thanks in advance for help. I'm trying to run pfSense in VM on UNRAID. I have Intel 4-port PCI-E card, which is seen in 2 separate IOMMU groups as: Card is also removed from UNRAID usage in config with `vfio-pci.ids=8086:10e8` All 4 NIC's are passed to VM: But after running pfSense it sees only single NIC: Interestingly, when I create Windows 10 VM and passthough all 4 NIC's, all of them are visible and usable under Windows VM. I'm running UNRAID 6.8.0 on Ryzen 5 2600 with ASRock B450m Pro4-f. pfSense VM runs on QEMU64 due to: I'm kind of lost, any thoughts? Can I provide some additional data to help investigating? Thanks!
December 28, 20196 yr Author just tested on Ubuntu 18.04 VM, passthrough NIC's also seems to work, so I think it's more related to pfSense itself rather than passthrough:
December 28, 20196 yr In Network Settings of Unraid do you see the interfaces listed? Possibly under the Interface Rules group.
December 28, 20196 yr Author No, only mother board NIC - which is not used as interface into pfSense. 4-Port card is disabled for UNRAID in settings, with: vfio-pci.ids=8086:10e8
December 28, 20196 yr Ahhh, sorry I missed that in your original posting. I’m sure you already used this tutorial. I would watch it again particularly starting around the 9:40 mark and see if you missed something. This is what I used and have a similar hardware setup as you. I didn’t have any issues.
December 28, 20196 yr Author Thanks, I used this exact tutorial, since I had issues, I watched it several times ... I think I did not missed anything. Edited December 28, 20196 yr by pankleks
December 28, 20196 yr Author Yes, I actually changed slot in first place to achieve good IOMMU group split without forcing override flag.
December 29, 20196 yr Hi, I came here looking for an answer to this exact problem. My test H/W is a little different: Optiplex i5 7010, 16gb RAM with HP Quad port server NIC. pfSense is not seeing any of the NIC's when passed through, but Windows 7 Pro is. I also removed the quad card from the PC and tried using just the builtin Intel NIC, but in my case pfSense doesn't see the br0 or the virbr0 NIC's either. I'm using using pfSense 2.4.4.iso and Unraid 6.8.0 trail, both downloaded today.
January 2, 20206 yr Author I changed previous Intel card to newer Intel card (I350 quad port) - which is listed in supported devices for `pfSense`, but this did not helped. Still VM is seeing only one port I lunched `dmesg` in VM and only suspicious things I see: and: and: Any ideas? Thanks!
January 5, 20206 yr I had this exact same problem and got it to work. Can you post your VM xml? Maybe I can help.
January 7, 20206 yr I’m having that problem to. Can I post my VM xml? Maybe you guys can help me? I followed the same tutorial from Spaceinvader one.
January 8, 20206 yr ok, below is the xml of one of your passed through devices. <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x07' slot='0x00' function='0x3'/> </source> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </hostdev> the address type is inserted automatically by unraid and its generating a seperate bus for every hostdev. Looking at the error log I assume pfsense does not correctly allocate those. You can try the following which worked for me: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x07' slot='0x00' function='0x2'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x2'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x07' slot='0x00' function='0x3'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x3'/> </hostdev> I just changed the bus for the following entries to "0x01" (value of the first entry, which I assume is allocated correctly in pfsense) and additionally increased the function value for every entry. Afterwards I reset pfsense from the command line and after rebooting all passed through network cards were available. I hope this helps.
January 9, 20206 yr I got PFsense to see my 4 port Intel card now thanks to your suggestion, But its saying, No link-up detected when I tried to connect my wan interface?
January 9, 20206 yr Glad it worked. Can't help much with the no link-up issue though. You can try to manually enter the WAN interface (4 tries at worst ;)) and see if that works.
January 11, 20206 yr Pankleks I have also a split iommou group but I can only get two ports to work did you get 4 to work? Could you share your final XML? I am going crazy haha, Thanks in advance! <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </hostdev> This works and two ports are seen... but when I add my two remaining ports (split into diffrent group) like so: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x1'/> </hostdev> then still only two interfaces show up. Edited January 11, 20206 yr by thierrybla
January 13, 20206 yr @thierrybla Have you tried giving all cards the same bus with different function value like this? <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x2' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x3'/> </hostdev>
March 1, 20206 yr On 1/8/2020 at 11:04 AM, user2352 said: ok, below is the xml of one of your passed through devices. <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x07' slot='0x00' function='0x3'/> </source> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </hostdev> the address type is inserted automatically by unraid and its generating a seperate bus for every hostdev. Looking at the error log I assume pfsense does not correctly allocate those. You can try the following which worked for me: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x07' slot='0x00' function='0x2'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x2'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x07' slot='0x00' function='0x3'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x3'/> </hostdev> I just changed the bus for the following entries to "0x01" (value of the first entry, which I assume is allocated correctly in pfsense) and additionally increased the function value for every entry. Afterwards I reset pfsense from the command line and after rebooting all passed through network cards were available. I hope this helps. This helped me too, thanks a bunch!
March 7, 20206 yr On 1/9/2020 at 10:44 AM, Jb711 said: I figured out the no link-up issue. Thanks for the help!! What was the no link issue, I am having the same thing on a pfsence I'm trying to set up.
May 8, 20206 yr On 3/6/2020 at 6:44 PM, kakali said: What was the no link issue, I am having the same thing on a pfsence I'm trying to set up. Well I'm running into the same problem, except 2 of the 4 interfaces are working. When I plug a cable into port 0 & port 1 (ports farther from PCIe pins on the card), I see blinking lights and can see the status change from down to up in the pfsense interface, but when I plug into ports 2 & 3, the link lights on the card go solid (green & orange) and pfsense does not change from down. Anyone else figure anything more out on this?
May 21, 20206 yr On 5/8/2020 at 1:25 PM, Douglas_D said: Well I'm running into the same problem, except 2 of the 4 interfaces are working. When I plug a cable into port 0 & port 1 (ports farther from PCIe pins on the card), I see blinking lights and can see the status change from down to up in the pfsense interface, but when I plug into ports 2 & 3, the link lights on the card go solid (green & orange) and pfsense does not change from down. Anyone else figure anything more out on this? So it turns out that the CLI after setting up pfSense doesn't detect the other 2 NICs, but once I got into the GUI, I was able to enable each port and see that they worked, so if you run into this problem, just keep pushing forward into the next stages and check if things work after configuring things from the GUI.
August 15, 20205 yr On 1/11/2020 at 3:47 PM, thierrybla said: Pankleks I have also a split iommou group but I can only get two ports to work did you get 4 to work? Could you share your final XML? I am going crazy haha, Thanks in advance! <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </hostdev> This works and two ports are seen... but when I add my two remaining ports (split into diffrent group) like so: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x1'/> </hostdev> then still only two interfaces show up. It worked for me too. Perfect. Only different thing I am doing is that I have only one PCI with 4 Gb Ethernet. I was able to use 2 for UNRAID and pass the 3rd and 4th to pfSense. Thank you guys.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.