pankleks Posted December 28, 2019 Share Posted December 28, 2019 Hello and thanks in advance for help. I'm trying to run pfSense in VM on UNRAID. I have Intel 4-port PCI-E card, which is seen in 2 separate IOMMU groups as: Card is also removed from UNRAID usage in config with `vfio-pci.ids=8086:10e8` All 4 NIC's are passed to VM: But after running pfSense it sees only single NIC: Interestingly, when I create Windows 10 VM and passthough all 4 NIC's, all of them are visible and usable under Windows VM. I'm running UNRAID 6.8.0 on Ryzen 5 2600 with ASRock B450m Pro4-f. pfSense VM runs on QEMU64 due to: I'm kind of lost, any thoughts? Can I provide some additional data to help investigating? Thanks! Quote Link to comment
pankleks Posted December 28, 2019 Author Share Posted December 28, 2019 just tested on Ubuntu 18.04 VM, passthrough NIC's also seems to work, so I think it's more related to pfSense itself rather than passthrough: Quote Link to comment
beaverly72 Posted December 28, 2019 Share Posted December 28, 2019 In Network Settings of Unraid do you see the interfaces listed? Possibly under the Interface Rules group. Quote Link to comment
pankleks Posted December 28, 2019 Author Share Posted December 28, 2019 No, only mother board NIC - which is not used as interface into pfSense. 4-Port card is disabled for UNRAID in settings, with: vfio-pci.ids=8086:10e8 Quote Link to comment
beaverly72 Posted December 28, 2019 Share Posted December 28, 2019 Ahhh, sorry I missed that in your original posting. I’m sure you already used this tutorial. I would watch it again particularly starting around the 9:40 mark and see if you missed something. This is what I used and have a similar hardware setup as you. I didn’t have any issues. Quote Link to comment
pankleks Posted December 28, 2019 Author Share Posted December 28, 2019 (edited) Thanks, I used this exact tutorial, since I had issues, I watched it several times ... I think I did not missed anything. Edited December 28, 2019 by pankleks Quote Link to comment
bastl Posted December 28, 2019 Share Posted December 28, 2019 @pankleks Did you tried the card in another pci slot of your board? Quote Link to comment
pankleks Posted December 28, 2019 Author Share Posted December 28, 2019 Yes, I actually changed slot in first place to achieve good IOMMU group split without forcing override flag. Quote Link to comment
ZaFord Posted December 29, 2019 Share Posted December 29, 2019 Hi, I came here looking for an answer to this exact problem. My test H/W is a little different: Optiplex i5 7010, 16gb RAM with HP Quad port server NIC. pfSense is not seeing any of the NIC's when passed through, but Windows 7 Pro is. I also removed the quad card from the PC and tried using just the builtin Intel NIC, but in my case pfSense doesn't see the br0 or the virbr0 NIC's either. I'm using using pfSense 2.4.4.iso and Unraid 6.8.0 trail, both downloaded today. Quote Link to comment
pankleks Posted January 2, 2020 Author Share Posted January 2, 2020 I changed previous Intel card to newer Intel card (I350 quad port) - which is listed in supported devices for `pfSense`, but this did not helped. Still VM is seeing only one port I lunched `dmesg` in VM and only suspicious things I see: and: and: Any ideas? Thanks! Quote Link to comment
user2352 Posted January 5, 2020 Share Posted January 5, 2020 I had this exact same problem and got it to work. Can you post your VM xml? Maybe I can help. Quote Link to comment
Jb711 Posted January 7, 2020 Share Posted January 7, 2020 I’m having that problem to. Can I post my VM xml? Maybe you guys can help me? I followed the same tutorial from Spaceinvader one. Quote Link to comment
pankleks Posted January 7, 2020 Author Share Posted January 7, 2020 Thanks, here is xml. vm.xml Quote Link to comment
user2352 Posted January 8, 2020 Share Posted January 8, 2020 ok, below is the xml of one of your passed through devices. <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x07' slot='0x00' function='0x3'/> </source> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </hostdev> the address type is inserted automatically by unraid and its generating a seperate bus for every hostdev. Looking at the error log I assume pfsense does not correctly allocate those. You can try the following which worked for me: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x07' slot='0x00' function='0x2'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x2'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x07' slot='0x00' function='0x3'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x3'/> </hostdev> I just changed the bus for the following entries to "0x01" (value of the first entry, which I assume is allocated correctly in pfsense) and additionally increased the function value for every entry. Afterwards I reset pfsense from the command line and after rebooting all passed through network cards were available. I hope this helps. 4 3 Quote Link to comment
Jb711 Posted January 9, 2020 Share Posted January 9, 2020 I got PFsense to see my 4 port Intel card now thanks to your suggestion, But its saying, No link-up detected when I tried to connect my wan interface? 1 Quote Link to comment
user2352 Posted January 9, 2020 Share Posted January 9, 2020 Glad it worked. Can't help much with the no link-up issue though. You can try to manually enter the WAN interface (4 tries at worst ;)) and see if that works. Quote Link to comment
Jb711 Posted January 9, 2020 Share Posted January 9, 2020 I figured out the no link-up issue. Thanks for the help!! Quote Link to comment
pankleks Posted January 10, 2020 Author Share Posted January 10, 2020 Thank you @user2352 it also helped me. Good stuff! Quote Link to comment
thierrybla Posted January 11, 2020 Share Posted January 11, 2020 (edited) Pankleks I have also a split iommou group but I can only get two ports to work did you get 4 to work? Could you share your final XML? I am going crazy haha, Thanks in advance! <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </hostdev> This works and two ports are seen... but when I add my two remaining ports (split into diffrent group) like so: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x1'/> </hostdev> then still only two interfaces show up. Edited January 11, 2020 by thierrybla Quote Link to comment
user2352 Posted January 13, 2020 Share Posted January 13, 2020 @thierrybla Have you tried giving all cards the same bus with different function value like this? <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x2' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x3'/> </hostdev> Quote Link to comment
Alphahelix Posted March 1, 2020 Share Posted March 1, 2020 On 1/8/2020 at 11:04 AM, user2352 said: ok, below is the xml of one of your passed through devices. <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x07' slot='0x00' function='0x3'/> </source> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </hostdev> the address type is inserted automatically by unraid and its generating a seperate bus for every hostdev. Looking at the error log I assume pfsense does not correctly allocate those. You can try the following which worked for me: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x07' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x07' slot='0x00' function='0x2'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x2'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x07' slot='0x00' function='0x3'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x3'/> </hostdev> I just changed the bus for the following entries to "0x01" (value of the first entry, which I assume is allocated correctly in pfsense) and additionally increased the function value for every entry. Afterwards I reset pfsense from the command line and after rebooting all passed through network cards were available. I hope this helps. This helped me too, thanks a bunch! 1 Quote Link to comment
kakali Posted March 7, 2020 Share Posted March 7, 2020 On 1/9/2020 at 10:44 AM, Jb711 said: I figured out the no link-up issue. Thanks for the help!! What was the no link issue, I am having the same thing on a pfsence I'm trying to set up. Quote Link to comment
Douglas_D Posted May 8, 2020 Share Posted May 8, 2020 On 3/6/2020 at 6:44 PM, kakali said: What was the no link issue, I am having the same thing on a pfsence I'm trying to set up. Well I'm running into the same problem, except 2 of the 4 interfaces are working. When I plug a cable into port 0 & port 1 (ports farther from PCIe pins on the card), I see blinking lights and can see the status change from down to up in the pfsense interface, but when I plug into ports 2 & 3, the link lights on the card go solid (green & orange) and pfsense does not change from down. Anyone else figure anything more out on this? 1 Quote Link to comment
Douglas_D Posted May 21, 2020 Share Posted May 21, 2020 On 5/8/2020 at 1:25 PM, Douglas_D said: Well I'm running into the same problem, except 2 of the 4 interfaces are working. When I plug a cable into port 0 & port 1 (ports farther from PCIe pins on the card), I see blinking lights and can see the status change from down to up in the pfsense interface, but when I plug into ports 2 & 3, the link lights on the card go solid (green & orange) and pfsense does not change from down. Anyone else figure anything more out on this? So it turns out that the CLI after setting up pfSense doesn't detect the other 2 NICs, but once I got into the GUI, I was able to enable each port and see that they worked, so if you run into this problem, just keep pushing forward into the next stages and check if things work after configuring things from the GUI. 1 Quote Link to comment
DrLucasMendes Posted August 15, 2020 Share Posted August 15, 2020 On 1/11/2020 at 3:47 PM, thierrybla said: Pankleks I have also a split iommou group but I can only get two ports to work did you get 4 to work? Could you share your final XML? I am going crazy haha, Thanks in advance! <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </hostdev> This works and two ports are seen... but when I add my two remaining ports (split into diffrent group) like so: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x05' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </source> <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x1'/> </hostdev> then still only two interfaces show up. It worked for me too. Perfect. Only different thing I am doing is that I have only one PCI with 4 Gb Ethernet. I was able to use 2 for UNRAID and pass the 3rd and 4th to pfSense. Thank you guys. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.