PF Sense on HP DL380G6 Server

[antohind]

Hey guys, 

 

I have been searching around for the best way to install PFsense onto a unraid VM I have followed all the tutorials on youTube but I am getting a bit stuck on the interfaces settings, 

 

My server has 4 x ethernet ports already installed as part of the chassis but when following the youtube vids they use a seprate PCI network card and then remove it from the Psense management 

 

Is it possible for me to use PFsense in this setup or will i need a completely separate NIC

 

Hope someone can help 

 

Regards 

 

[1812]

Assuming pfsense has the correct driver for the hp adapter, You have to use pcie acs override multifunction to see if you can split the 4 ports into individual iommu groups. From there,  you have 2 options: the first is try to add eth2 and 3 to the vm (which may or may not work depending on if the system releases them) If its a no-go, and assuming you're on 6.7.0, use the bind function as described here 

  to hide them from the system. You'll then have to add them in the xml. 

 

or, just buy a 2-4 port card and pass that through.

Edited June 8, 2019 by 1812

[antohind]

Thank you I will take a look at that now  

 

 

[1812]

3 minutes ago, antohind said:

Thank you I will take a look at that now  

 

 

also, there are also a few links in my signature regarding proliants and common issues you may run into. 

[antohind]

Thanks really appreciate that, I have just enabled the multifunction acs override and rebooted the server should i then see seperate NICs if that has been successful ?

[antohind]

25 minutes ago, 1812 said:

also, there are also a few links in my signature regarding proliants and common issues you may run into. 

Below is the output after enabling the multifunctional option doesn't look like i can split the NIC's - 14e4:1639 

 

IOMMU group 0:[8086:3406] 00:00.0 Host bridge: Intel Corporation 5520 I/O Hub to ESI Port (rev 13)

IOMMU group 1:[8086:3408] 00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 13)

IOMMU group 2:[8086:3409] 00:02.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 2 (rev 13)

IOMMU group 3:[8086:340a] 00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 13)

IOMMU group 4:[8086:340b] 00:04.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 4 (rev 13)

IOMMU group 5:[8086:340c] 00:05.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 5 (rev 13)

IOMMU group 6:[8086:340d] 00:06.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 6 (rev 13)

IOMMU group 7:[8086:340e] 00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 13)

IOMMU group 8:[8086:340f] 00:08.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 8 (rev 13)

IOMMU group 9:[8086:3410] 00:09.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 13)

IOMMU group 10:[8086:3411] 00:0a.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 10 (rev 13)

IOMMU group 11:[8086:343a] 00:0d.0 Host bridge: Intel Corporation Device 343a (rev 13)

IOMMU group 12:[8086:343b] 00:0d.1 Host bridge: Intel Corporation Device 343b (rev 13)

IOMMU group 13:[8086:343c] 00:0d.2 Host bridge: Intel Corporation Device 343c (rev 13)

IOMMU group 14:[8086:343d] 00:0d.3 Host bridge: Intel Corporation Device 343d (rev 13)

IOMMU group 15:[8086:3418] 00:0d.4 Host bridge: Intel Corporation 7500/5520/5500/X58 Physical Layer Port 0 (rev 13)

IOMMU group 16:[8086:3419] 00:0d.5 Host bridge: Intel Corporation 7500/5520/5500 Physical Layer Port 1 (rev 13)

IOMMU group 17:[8086:341a] 00:0d.6 Host bridge: Intel Corporation Device 341a (rev 13)

IOMMU group 18:[8086:341c] 00:0e.0 Host bridge: Intel Corporation Device 341c (rev 13)

IOMMU group 19:[8086:341d] 00:0e.1 Host bridge: Intel Corporation Device 341d (rev 13)

IOMMU group 20:[8086:341e] 00:0e.2 Host bridge: Intel Corporation Device 341e (rev 13)

IOMMU group 21:[8086:341f] 00:0e.3 Host bridge: Intel Corporation Device 341f (rev 13)

IOMMU group 22:[8086:3439] 00:0e.4 Host bridge: Intel Corporation Device 3439 (rev 13)

IOMMU group 23:[8086:342e] 00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers (rev 13)

IOMMU group 24:[8086:3422] 00:14.1 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 13)

IOMMU group 25:[8086:3423] 00:14.2 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 13)

IOMMU group 26:[8086:3a40] 00:1c.0 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 1

[8086:3a44] 00:1c.2 PCI bridge: Intel Corporation 82801JI (ICH10 Family) PCI Express Root Port 3

[14e4:1639] 02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)

[14e4:1639] 02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)

[14e4:1639] 03:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)

[14e4:1639] 03:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)

IOMMU group 27:[8086:3a34] 00:1d.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1

[8086:3a35] 00:1d.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2

[8086:3a36] 00:1d.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3

[8086:3a39] 00:1d.3 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6

[8086:3a3a] 00:1d.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1

IOMMU group 28:[8086:244e] 00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)

[1002:515e] 01:03.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] ES1000 (rev 02)

[0e11:b203] 01:04.0 System peripheral: Compaq Computer Corporation Integrated Lights Out Controller (rev 03)

[0e11:b204] 01:04.2 System peripheral: Compaq Computer Corporation Integrated Lights Out Processor (rev 03)

[103c:3300] 01:04.4 USB controller: Hewlett-Packard Company Integrated Lights-Out Standard Virtual USB Controller

[103c:3302] 01:04.6 IPMI Interface: Hewlett-Packard Company Integrated Lights-Out Standard KCS Interface

IOMMU group 29:[8086:3a18] 00:1f.0 ISA bridge: Intel Corporation 82801JIB (ICH10) LPC Interface Controller

IOMMU group 30:[103c:323a] 04:00.0 RAID bus controller: Hewlett-Packard Company Smart Array G6 controllers (rev 01)

IOMMU group 31:[8086:2c70] 3e:00.0 Host bridge: Intel Corporation Xeon 5600 Series QuickPath Architecture Generic Non-core Registers (rev 02)

[8086:2d81] 3e:00.1 Host bridge: Intel Corporation Xeon 5600 Series QuickPath Architecture System Address Decoder (rev 02)

IOMMU group 32:[8086:2d90] 3e:02.0 Host bridge: Intel Corporation Xeon 5600 Series QPI Link 0 (rev 02)

[8086:2d91] 3e:02.1 Host bridge: Intel Corporation Xeon 5600 Series QPI Physical 0 (rev 02)

[8086:2d92] 3e:02.2 Host bridge: Intel Corporation Xeon 5600 Series Mirror Port Link 0 (rev 02)

[8086:2d93] 3e:02.3 Host bridge: Intel Corporation Xeon 5600 Series Mirror Port Link 1 (rev 02)

[8086:2d94] 3e:02.4 Host bridge: Intel Corporation Xeon 5600 Series QPI Link 1 (rev 02)

[8086:2d95] 3e:02.5 Host bridge: Intel Corporation Xeon 5600 Series QPI Physical 1 (rev 02)

IOMMU group 33:[8086:2d98] 3e:03.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Registers (rev 02)

[8086:2d99] 3e:03.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Target Address Decoder (rev 02)

[8086:2d9a] 3e:03.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller RAS Registers (rev 02)

[8086:2d9c] 3e:03.4 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Test Registers (rev 02)

IOMMU group 34:[8086:2da0] 3e:04.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Control (rev 02)

[8086:2da1] 3e:04.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Address (rev 02)

[8086:2da2] 3e:04.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Rank (rev 02)

[8086:2da3] 3e:04.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Thermal Control (rev 02)

IOMMU group 35:[8086:2da8] 3e:05.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Control (rev 02)

[8086:2da9] 3e:05.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Address (rev 02)

[8086:2daa] 3e:05.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Rank (rev 02)

[8086:2dab] 3e:05.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Thermal Control (rev 02)

IOMMU group 36:[8086:2db0] 3e:06.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Control (rev 02)

[8086:2db1] 3e:06.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Address (rev 02)

[8086:2db2] 3e:06.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Rank (rev 02)

[8086:2db3] 3e:06.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Thermal Control (rev 02)

IOMMU group 37:[8086:2c70] 3f:00.0 Host bridge: Intel Corporation Xeon 5600 Series QuickPath Architecture Generic Non-core Registers (rev 02)

[8086:2d81] 3f:00.1 Host bridge: Intel Corporation Xeon 5600 Series QuickPath Architecture System Address Decoder (rev 02)

IOMMU group 38:[8086:2d90] 3f:02.0 Host bridge: Intel Corporation Xeon 5600 Series QPI Link 0 (rev 02)

[8086:2d91] 3f:02.1 Host bridge: Intel Corporation Xeon 5600 Series QPI Physical 0 (rev 02)

[8086:2d92] 3f:02.2 Host bridge: Intel Corporation Xeon 5600 Series Mirror Port Link 0 (rev 02)

[8086:2d93] 3f:02.3 Host bridge: Intel Corporation Xeon 5600 Series Mirror Port Link 1 (rev 02)

[8086:2d94] 3f:02.4 Host bridge: Intel Corporation Xeon 5600 Series QPI Link 1 (rev 02)

[8086:2d95] 3f:02.5 Host bridge: Intel Corporation Xeon 5600 Series QPI Physical 1 (rev 02)

IOMMU group 39:[8086:2d98] 3f:03.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Registers (rev 02)

[8086:2d99] 3f:03.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Target Address Decoder (rev 02)

[8086:2d9a] 3f:03.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller RAS Registers (rev 02)

[8086:2d9c] 3f:03.4 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Test Registers (rev 02)

IOMMU group 40:[8086:2da0] 3f:04.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Control (rev 02)

[8086:2da1] 3f:04.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Address (rev 02)

[8086:2da2] 3f:04.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Rank (rev 02)

[8086:2da3] 3f:04.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 0 Thermal Control (rev 02)

IOMMU group 41:[8086:2da8] 3f:05.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Control (rev 02)

[8086:2da9] 3f:05.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Address (rev 02)

[8086:2daa] 3f:05.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Rank (rev 02)

[8086:2dab] 3f:05.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 1 Thermal Control (rev 02)

IOMMU group 42:[8086:2db0] 3f:06.0 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Control (rev 02)

[8086:2db1] 3f:06.1 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Address (rev 02)

[8086:2db2] 3f:06.2 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Rank (rev 02)

[8086:2db3] 3f:06.3 Host bridge: Intel Corporation Xeon 5600 Series Integrated Memory Controller Channel 2 Thermal Control (rev 02)

 

Total noob here lol if you spot something i am missing would be a great help - not to sure how to bind the interfaces 

[1812]

did you reboot?

 

try pcie_acs_override=downstream,multifunction

 

after rebooting if its still not split, post your diagnostics zip (tools>diagnostics)

Edited June 8, 2019 by 1812

[antohind]

20 minutes ago, 1812 said:

did you reboot?

 

try pcie_acs_override=downstream,multifunction

 

after rebooting if its still not split, post your diagnostics zip (tools>diagnostics)

IOMMU group 19:[14e4:1639] 02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)

[14e4:1639] 02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)

 

 

IOMMU group 20:[14e4:1639] 03:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)

[14e4:1639] 03:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)

[antohind]

This looks a little more promising 

 

Seem to have split them into two as i have two connected for testing but the values are the same still 

Edited June 8, 2019 by antohind

[1812]

3 minutes ago, antohind said:

IOMMU group 19:[14e4:1639] 02:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)

[14e4:1639] 02:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)

 

 

IOMMU group 20:[14e4:1639] 03:00.0 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)

[14e4:1639] 03:00.1 Ethernet controller: Broadcom Inc. and subsidiaries NetXtreme II BCM5709 Gigabit Ethernet (rev 20)

 

I'd go with that, using 3:00.0 and 3:00.1, it may be the best you'll get. And no real need to split any further. Leave the first 2 for active and failover.

Edited June 8, 2019 by 1812

[antohind]

4 minutes ago, antohind said:

2 minutes ago, 1812 said:

3:00.0 and 3:00.1

Do i need to bind this in some way now ?

 

[antohind]

https://www.google.com/search?rlz=1C1GCEA_enGB806GB806&ei=BBD8XKzqH4Ce1fAPndOP2AM&q=pfsense+setup+unraid+part+3&oq=pfsense+setup+unraid+part+3&gs_l=psy-ab.3...4783.6313..6474...0.0..1.259.975.2j3j2......0....1..gws-wiz.......0i71j0i22i30j33i160j33i21.61C1Nummh7g#kpvalbx=1

 

This is the method i was following 

[1812]

1 minute ago, antohind said:

 

try to add to the pfsense vm via the web gui first. if its not there, then you'll have to bind them by their address as described above and reboot (since they share a device ID you can't use vfio.pcids=) Then you'll have to add them manually to vm.

[antohind]

Thanks for your help on this i will try and figure out the binding manually now

 

I don't have any other NIC's displaying in the VM GUI under additional PCI devices

[antohind]

45 minutes ago, 1812 said:

add

 

[antohind]

Is this what you mean about assigning br2 & br3 to the PFsense VM ??

[antohind]

[1812]

31 minutes ago, antohind said:

Is this what you mean about assigning br2 & br3 to the PFsense VM ??

no not the bridge, the actual eth2&3

[antohind]

Ahh OK no they are not there i must be missing something here 

[1812]

28 minutes ago, antohind said:

Ahh OK no they are not there i must be missing something here 

You will likely need to input them manually in the xml.

 

go to edit, unselect any network devices/brX you currently have selected.  toggle top right corner to xml view.

 

the line under " </video>" add the following

 <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
      </source>
    </hostdev>

 

 

save, attempt to boot vm. If unRaid releases it, you should be good to go. if not, follow the directions in the link I posted above to bind those 2 addresses, then reboot and retry to boot., If bound correctly using the "new" way in the link, the 2 ethernet ports should not show up to unRaid (if it truly replaces the xenback.hide, this is what that did.)

 

 

[antohind]

16 minutes ago, 1812 said:

You will likely need to input them manually in the xml.

 

go to edit, unselect any network devices/brX you currently have selected.  toggle top right corner to xml view.

 

the line under " </video>" add the following

 <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
      </source>
    </hostdev>

 

 

save, attempt to boot vm. If unRaid releases it, you should be good to go. if not, follow the directions in the link I posted above to bind those 2 addresses, then reboot and retry to boot., If bound correctly using the "new" way in the link, the 2 ethernet ports should not show up to unRaid (if it truly replaces the xenback.hide, this is what that did.)

 

 

So the interfaces do show up in the VM now but i get this error when tryin to bot the vm 

[1812]

18 minutes ago, antohind said:

So the interfaces do show up in the VM now but i get this error when tryin to bot the vm 

is this with both interfaces assigned?

 

if so, post diagnostics.zip file

[antohind]

See diagnostics 

tower-diagnostics-20190608-2232.zip

[antohind]

@1812 I have posted diags above as requested 

[1812]

7 minutes ago, antohind said:

@1812 I have posted diags above as requested 

next problem on your road to unRaiding:

Jun  8 15:04:30 Tower kernel: vfio_iommu_type1_attach_group: No interrupt remapping support.  Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform

in my signature, there is an explanation how to allow "unsafe interrupts". Follow those directions, reboot, try to run vm again.