IOMMU: Bind Group devices (vfio-pci.ids)

[cometship]

Hi, My unraid is using one of 10GbE NIC ports (device 19a2:0710 below).

Can I isolate the unused port to be passed to pfsense VM using vfio-pci.ids, or another method?

Do I need to install another NIC card for pfsense LAN ports? I am planning to use the MB NIC for pfsense WAN.

 

Doing a little googling it looks like a NIC with SR-IOV would allow virtualizing the NIC ports: https://www.juniper.net/documentation/en_US/junos/topics/concept/disaggregated-junos-sr-iov.html

 

I have Emulex 49Y7952 https://lenovopress.com/tips0844-emulex-10gbe-vfa-ii-iii which may allow VNICs if it can be switched to vNIC2 mode?

Thanks!

 

IOMMU group 0:

[1022:1632] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe Dummy Host Bridge

[1022:1633] 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge

[19a2:0710] 01:00.0 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (be3) (rev 02) -> Not connected. Can I isolate this port?

[19a2:0710] 01:00.1 Ethernet controller: Emulex Corporation OneConnect 10Gb NIC (be3) (rev 02) -> Connected and used by UNRAID

Edited January 8, 2022 by cometship

[cometship]

Anybody, please? The UNRAID manual has no current entry for "Network Settings". I am using for reference therefore 2018 video from SpaceInvader.

UNRAID grabs all ethernet ports across IOMMU groups.

Objetive is to isolate one NIC port for pfsense VM, and other NIC port for UNRAID with security and stability. Are these the only options?

 

1) Enable bonding, bridging, and VLANs in tab "Network Settings" for both ports in NIC card.

My switch supports VLAN & link aggregation. The bonding will double the 10GbE throughput for UNRAD and pfsense

The bridging option will allow pfsense VM to access the bonded UNRAID physical ports

The VLAN option can provide isolation between pfsense and UNRAID.

 

2) Motherboard supports bifurcation (I've used this successfully elsewhere). Bifurcate x8x8 and install a second NIC card.

Will this result in bifurcated NICs with separate IOMMU group?

 

3) Get a motherboard that supports virtualization better. Which ones in ITX form?

4) Use 'vfio-pci.ids=19a2:0710' will this give each Ethernet port on same NIC their own IOMMU?

 

The options below fail the security requirement since devices are not truly isolated

5) Use 'pcie_acs_override=downstream', if this fails, try 6)

6) Use 'pcie_acs_override=downstream, multifunction' (may not be truly isolated)

??

Thanks for help with this difficult topic.

 

Edited January 9, 2022 by cometship

[ljm42]

Just to level set... assuming you are running 6.9.2 or one of the 6.10.0 rc's, you can navigate to Tools -> System Devices and put a checkmark next to the IOMMU groups that you want to bind to vfio-pci. After rebooting, the devices in those IOMMU groups will be available to pass through to a VM. For more info see https://forums.unraid.net/topic/93781-guide-bind-devices-to-vfio-pci-for-easy-passthrough-to-vms/

 

 

Also, some people have reported success with SR-IOV by making a second call to vfio-pci after drivers have been loaded:
  https://forums.unraid.net/topic/93781-guide-bind-devices-to-vfio-pci-for-easy-passthrough-to-vms/?tab=comments#comment-957990

 

 

Currently it looks like both of your NICs are in the same IOMMU group though. As a first step I'd recommend upgrading your BIOS, it may improve this. Otherwise, you can either work on ways to split them into separate IOMMU groups (although it sounds like you have already ruled those out) or you can buy another physical NIC and hope that your motherboard will put it in its own IOMMU group. There isn't really a way to know what your motherboard will do until you try, although if you aren't happy with the results you can try moving cards around to different slots.

 

<soapbox> Personally I think virtualizing your router is a bad idea. It complicates your life greatly since you have no Internet if you stop the array. Also be aware Unraid is not optimized for this so it takes longer to boot because the Internet is not available so certain things have to timeout before it can continue. In short, Unraid is great for virtualizing multiple boxes into one, but the router is not an ideal use case for this. </soapbox>

[cometship]

Thank you ljm42 for detailed response and downsides of virtualizing the router.  Just because it can be done it doesn't mean we should.  I think my family will appreciate a stable router.  I was swayed by the related SpaceInvader video, but he has a physical backup router that enables when the array is down 

 

A Shinobi docker seems like a better fit for UNRAID with vlans for security.

Edited January 10, 2022 by cometship

[cometship]

(update)

Asrock provided an engineering BIOS that separated the devices from 5 to 21 IOMMU groups. I've had great customer service from them in the past. They delivered again.

 

[ljm42]

That is great news, it will help with whatever you decide to virtualize!