antohind Posted June 8, 2019 Share Posted June 8, 2019 Hey guys, I have been searching around for the best way to install PFsense onto a unraid VM I have followed all the tutorials on youTube but I am getting a bit stuck on the interfaces settings, My server has 4 x ethernet ports already installed as part of the chassis but when following the youtube vids they use a seprate PCI network card and then remove it from the Psense management Is it possible for me to use PFsense in this setup or will i need a completely separate NIC Hope someone can help Regards Quote Link to comment
1812 Posted June 8, 2019 Share Posted June 8, 2019 (edited) Assuming pfsense has the correct driver for the hp adapter, You have to use pcie acs override multifunction to see if you can split the 4 ports into individual iommu groups. From there, you have 2 options: the first is try to add eth2 and 3 to the vm (which may or may not work depending on if the system releases them) If its a no-go, and assuming you're on 6.7.0, use the bind function as described here to hide them from the system. You'll then have to add them in the xml. or, just buy a 2-4 port card and pass that through. Edited June 8, 2019 by 1812 Quote Link to comment
antohind Posted June 8, 2019 Author Share Posted June 8, 2019 Thank you I will take a look at that now Quote Link to comment
1812 Posted June 8, 2019 Share Posted June 8, 2019 3 minutes ago, antohind said: Thank you I will take a look at that now also, there are also a few links in my signature regarding proliants and common issues you may run into. Quote Link to comment
antohind Posted June 8, 2019 Author Share Posted June 8, 2019 Thanks really appreciate that, I have just enabled the multifunction acs override and rebooted the server should i then see seperate NICs if that has been successful ? Quote Link to comment
antohind Posted June 8, 2019 Author Share Posted June 8, 2019 25 minutes ago, 1812 said: also, there are also a few links in my signature regarding proliants and common issues you may run into. Below is the output after enabling the multifunctional option doesn't look like i can split the NIC's - 14e4:1639 IOMMU group 0:[8086:3406] 00:00.0 Host bridge: Intel Corporation 5520 I/O Hub to ESI Port (rev 13) IOMMU group 1:[8086:3408] 00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 13) IOMMU group 2:[8086:3409] 00:02.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 2 (rev 13) IOMMU group 3:[8086:340a] 00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 13) IOMMU group 4:[8086:340b] 00:04.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 4 (rev 13) IOMMU group 5:[8086:340c] 00:05.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 5 (rev 13) IOMMU group 6:[8086:340d] 00:06.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 6 (rev 13) IOMMU group 7:[8086:340e] 00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13) IOMMU group 8:[8086:340f] 00:08.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 8 (rev 13) IOMMU group 9:[8086:3410] 00:09.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 13) IOMMU group 10:[8086:3411] 00:0a.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 10 (rev 13) IOMMU group 11:[8086:343a] 00:0d.0 Host bridge: Intel Corporation Device 343a (rev 13) IOMMU group 12:[8086:343b] 00:0d.1 Host bridge: Intel Corporation Device 343b (rev 13) IOMMU group 13:[8086:343c] 00:0d.2 Host bridge: Intel Corporation Device 343c (rev 13) IOMMU group 14:[8086:343d] 00:0d.3 Host bridge: Intel Corporation Device 343d (rev 13) IOMMU group 15:[8086:3418] 00:0d.4 Host bridge: Intel Corporation 7500/5520/5500/X58 Physical Layer Port 0 (rev 13) IOMMU group 16:[8086:3419] 00:0d.5 Host bridge: Intel Corporation 7500/5520/5500 Physical Layer Port 1 (rev 13) IOMMU group 17:[8086:341a] 00:0d.6 Host bridge: Intel Corporation Device 341a (rev 13) IOMMU group 18:[8086:341c] 00:0e.0 Host bridge: Intel Corporation Device 341c (rev 13) IOMMU group 19:[8086:341d] 00:0e.1 Host bridge: Intel Corporation Device 341d (rev 13) IOMMU group 20:[8086:341e] 00:0e.2 Host bridge: Intel Corporation Device 341e (rev 13) IOMMU group 21:[8086:341f] 00:0e.3 Host bridge: Intel Corporation Device 341f (rev 13) IOMMU group 22:[8086:3439] 00:0e.4 Host bridge: Intel Corporation Device 3439 (rev 13) IOMMU group 23:[8086:342e] 00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers (rev 13) IOMMU group 24:[8086:3422] 00:14.1 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 13) IOMMU group 25:[8086:3423] 00:14.2 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 13) IOMMU group 26:[8086:3a40] 00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1 [8086:3a44] 00:1c.2 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 3 [14e4:1639] 02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20) [14e4:1639] 02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20) [14e4:1639] 03:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20) [14e4:1639] 03:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20) IOMMU group 27:[8086:3a34] 00:1d.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1 [8086:3a35] 00:1d.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2 [8086:3a36] 00:1d.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3 [8086:3a39] 00:1d.3 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6 [8086:3a3a] 00:1d.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1 IOMMU group 28:[8086:244e] 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90) [1002:515e] 01:03.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] ES1000 (rev 02) [0e11:b203] 01:04.0 System peripheral: Compaq Computer Corporation Integrated Lights Out Controller (rev 03) [0e11:b204] 01:04.2 System peripheral: Compaq Computer Corporation Integrated Lights Out Processor (rev 03) [103c:3300] 01:04.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller [103c:3302] 01:04.6 IPMI Interface: Hewlett-Packard Company Integrated Lights-Out Standard KCS Interface IOMMU group 29:[8086:3a18] 00:1f.0 ISA bridge: Intel Corporation 82801JIB (ICH10) LPC Interface Controller IOMMU group 30:[103c:323a] 04:00.0 RAID bus controller: Hewlett-Packard Company Smart Array G6 controllers (rev 01) IOMMU group 31:[8086:2c70] 3e:00.0 Host bridge: Intel Corporation Xeon 5600 Series QuickPath Architecture Generic Non-core Registers (rev 02) [8086:2d81] 3e:00.1 Host bridge: Intel Corporation Xeon 5600 Series QuickPath Architecture System Address Decoder (rev 02) IOMMU group 32:[8086:2d90] 3e:02.0 Host bridge: Intel Corporation Xeon 5600 Series QPI Link 0 (rev 02) [8086:2d91] 3e:02.1 Host bridge: Intel Corporation Xeon 5600 Series QPI Physical 0 (rev 02) [8086:2d92] 3e:02.2 Host bridge: Intel Corporation Xeon 5600 Series Mirror Port Link 0 (rev 02) [8086:2d93] 3e:02.3 Host bridge: Intel Corporation Xeon 5600 Series Mirror Port Link 1 (rev 02) [8086:2d94] 3e:02.4 Host bridge: Intel Corporation Xeon 5600 Series QPI Link 1 (rev 02) [8086:2d95] 3e:02.5 Host bridge: Intel Corporation Xeon 5600 Series QPI Physical 1 (rev 02) IOMMU group 33:[8086:2d98] 3e:03.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Registers (rev 02) [8086:2d99] 3e:03.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Target Address Decoder (rev 02) [8086:2d9a] 3e:03.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller RAS Registers (rev 02) [8086:2d9c] 3e:03.4 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Test Registers (rev 02) IOMMU group 34:[8086:2da0] 3e:04.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Control (rev 02) [8086:2da1] 3e:04.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Address (rev 02) [8086:2da2] 3e:04.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Rank (rev 02) [8086:2da3] 3e:04.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Thermal Control (rev 02) IOMMU group 35:[8086:2da8] 3e:05.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Control (rev 02) [8086:2da9] 3e:05.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Address (rev 02) [8086:2daa] 3e:05.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Rank (rev 02) [8086:2dab] 3e:05.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Thermal Control (rev 02) IOMMU group 36:[8086:2db0] 3e:06.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Control (rev 02) [8086:2db1] 3e:06.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Address (rev 02) [8086:2db2] 3e:06.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Rank (rev 02) [8086:2db3] 3e:06.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Thermal Control (rev 02) IOMMU group 37:[8086:2c70] 3f:00.0 Host bridge: Intel Corporation Xeon 5600 Series QuickPath Architecture Generic Non-core Registers (rev 02) [8086:2d81] 3f:00.1 Host bridge: Intel Corporation Xeon 5600 Series QuickPath Architecture System Address Decoder (rev 02) IOMMU group 38:[8086:2d90] 3f:02.0 Host bridge: Intel Corporation Xeon 5600 Series QPI Link 0 (rev 02) [8086:2d91] 3f:02.1 Host bridge: Intel Corporation Xeon 5600 Series QPI Physical 0 (rev 02) [8086:2d92] 3f:02.2 Host bridge: Intel Corporation Xeon 5600 Series Mirror Port Link 0 (rev 02) [8086:2d93] 3f:02.3 Host bridge: Intel Corporation Xeon 5600 Series Mirror Port Link 1 (rev 02) [8086:2d94] 3f:02.4 Host bridge: Intel Corporation Xeon 5600 Series QPI Link 1 (rev 02) [8086:2d95] 3f:02.5 Host bridge: Intel Corporation Xeon 5600 Series QPI Physical 1 (rev 02) IOMMU group 39:[8086:2d98] 3f:03.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Registers (rev 02) [8086:2d99] 3f:03.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Target Address Decoder (rev 02) [8086:2d9a] 3f:03.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller RAS Registers (rev 02) [8086:2d9c] 3f:03.4 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Test Registers (rev 02) IOMMU group 40:[8086:2da0] 3f:04.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Control (rev 02) [8086:2da1] 3f:04.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Address (rev 02) [8086:2da2] 3f:04.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Rank (rev 02) [8086:2da3] 3f:04.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Thermal Control (rev 02) IOMMU group 41:[8086:2da8] 3f:05.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Control (rev 02) [8086:2da9] 3f:05.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Address (rev 02) [8086:2daa] 3f:05.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Rank (rev 02) [8086:2dab] 3f:05.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Thermal Control (rev 02) IOMMU group 42:[8086:2db0] 3f:06.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Control (rev 02) [8086:2db1] 3f:06.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Address (rev 02) [8086:2db2] 3f:06.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Rank (rev 02) [8086:2db3] 3f:06.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Thermal Control (rev 02) Total noob here lol if you spot something i am missing would be a great help - not to sure how to bind the interfaces Quote Link to comment
1812 Posted June 8, 2019 Share Posted June 8, 2019 (edited) did you reboot? try pcie_acs_override=downstream,multifunction after rebooting if its still not split, post your diagnostics zip (tools>diagnostics) Edited June 8, 2019 by 1812 Quote Link to comment
antohind Posted June 8, 2019 Author Share Posted June 8, 2019 20 minutes ago, 1812 said: did you reboot? try pcie_acs_override=downstream,multifunction after rebooting if its still not split, post your diagnostics zip (tools>diagnostics) IOMMU group 19:[14e4:1639] 02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20) [14e4:1639] 02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20) IOMMU group 20:[14e4:1639] 03:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20) [14e4:1639] 03:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20) Quote Link to comment
antohind Posted June 8, 2019 Author Share Posted June 8, 2019 (edited) This looks a little more promising Seem to have split them into two as i have two connected for testing but the values are the same still Edited June 8, 2019 by antohind Quote Link to comment
1812 Posted June 8, 2019 Share Posted June 8, 2019 (edited) 3 minutes ago, antohind said: IOMMU group 19:[14e4:1639] 02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20) [14e4:1639] 02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20) IOMMU group 20:[14e4:1639] 03:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20) [14e4:1639] 03:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20) I'd go with that, using 3:00.0 and 3:00.1, it may be the best you'll get. And no real need to split any further. Leave the first 2 for active and failover. Edited June 8, 2019 by 1812 Quote Link to comment
antohind Posted June 8, 2019 Author Share Posted June 8, 2019 4 minutes ago, antohind said: 2 minutes ago, 1812 said: 3:00.0 and 3:00.1 Do i need to bind this in some way now ? Quote Link to comment
antohind Posted June 8, 2019 Author Share Posted June 8, 2019 https://www.google.com/search?rlz=1C1GCEA_enGB806GB806&ei=BBD8XKzqH4Ce1fAPndOP2AM&q=pfsense+setup+unraid+part+3&oq=pfsense+setup+unraid+part+3&gs_l=psy-ab.3...4783.6313..6474...0.0..1.259.975.2j3j2......0....1..gws-wiz.......0i71j0i22i30j33i160j33i21.61C1Nummh7g#kpvalbx=1 This is the method i was following Quote Link to comment
1812 Posted June 8, 2019 Share Posted June 8, 2019 1 minute ago, antohind said: try to add to the pfsense vm via the web gui first. if its not there, then you'll have to bind them by their address as described above and reboot (since they share a device ID you can't use vfio.pcids=) Then you'll have to add them manually to vm. Quote Link to comment
antohind Posted June 8, 2019 Author Share Posted June 8, 2019 Thanks for your help on this i will try and figure out the binding manually now I don't have any other NIC's displaying in the VM GUI under additional PCI devices Quote Link to comment
antohind Posted June 8, 2019 Author Share Posted June 8, 2019 45 minutes ago, 1812 said: add Quote Link to comment
antohind Posted June 8, 2019 Author Share Posted June 8, 2019 Is this what you mean about assigning br2 & br3 to the PFsense VM ?? Quote Link to comment
1812 Posted June 8, 2019 Share Posted June 8, 2019 31 minutes ago, antohind said: Is this what you mean about assigning br2 & br3 to the PFsense VM ?? no not the bridge, the actual eth2&3 Quote Link to comment
antohind Posted June 8, 2019 Author Share Posted June 8, 2019 Ahh OK no they are not there i must be missing something here Quote Link to comment
1812 Posted June 8, 2019 Share Posted June 8, 2019 28 minutes ago, antohind said: Ahh OK no they are not there i must be missing something here You will likely need to input them manually in the xml. go to edit, unselect any network devices/brX you currently have selected. toggle top right corner to xml view. the line under " </video>" add the following <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </source> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/> </source> </hostdev> save, attempt to boot vm. If unRaid releases it, you should be good to go. if not, follow the directions in the link I posted above to bind those 2 addresses, then reboot and retry to boot., If bound correctly using the "new" way in the link, the 2 ethernet ports should not show up to unRaid (if it truly replaces the xenback.hide, this is what that did.) Quote Link to comment
antohind Posted June 8, 2019 Author Share Posted June 8, 2019 16 minutes ago, 1812 said: You will likely need to input them manually in the xml. go to edit, unselect any network devices/brX you currently have selected. toggle top right corner to xml view. the line under " </video>" add the following <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </source> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/> </source> </hostdev> save, attempt to boot vm. If unRaid releases it, you should be good to go. if not, follow the directions in the link I posted above to bind those 2 addresses, then reboot and retry to boot., If bound correctly using the "new" way in the link, the 2 ethernet ports should not show up to unRaid (if it truly replaces the xenback.hide, this is what that did.) So the interfaces do show up in the VM now but i get this error when tryin to bot the vm Quote Link to comment
1812 Posted June 8, 2019 Share Posted June 8, 2019 18 minutes ago, antohind said: So the interfaces do show up in the VM now but i get this error when tryin to bot the vm is this with both interfaces assigned? if so, post diagnostics.zip file Quote Link to comment
antohind Posted June 8, 2019 Author Share Posted June 8, 2019 See diagnostics tower-diagnostics-20190608-2232.zip Quote Link to comment
antohind Posted June 8, 2019 Author Share Posted June 8, 2019 @1812 I have posted diags above as requested Quote Link to comment
1812 Posted June 8, 2019 Share Posted June 8, 2019 7 minutes ago, antohind said: @1812 I have posted diags above as requested next problem on your road to unRaiding: Jun 8 15:04:30 Tower kernel: vfio_iommu_type1_attach_group: No interrupt remapping support. Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform in my signature, there is an explanation how to allow "unsafe interrupts". Follow those directions, reboot, try to run vm again. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.