PCIe Cards in shared IOMMU Group

[faicec]

Hi all,

 

Can anything be done to resolve the below?

 

I've purchased a couple of USB3 PCIe cards in the hopes of passing them thru to a couple of VM's, however they landed in an IOMMU group with other devices.

 

I've tried all empty PCIe slots, same result, they land in group 14.  I did stub out 1106:3483, to the extent I get the choice to pass-thru a card to a VM, however the power on throws this exception:

 

"qemu-system-x86_64: -device vfio-pci,host=06:00.0,id=hostdev0,bus=pci.0,addr=0x8: vfio 0000:06:00.0: group 14 is not viable Please ensure all devices within the iommu_group are bound to their vfio bus driver."

 

Group details:

 

IOMMU group 14:

[1022:43b9] 02:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] X370 Series Chipset USB 3.1 xHCI Controller (rev 02)

[1022:43b5] 02:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] X370 Series Chipset SATA Controller (rev 02)

[1022:43b0] 02:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] X370 Series Chipset PCIe Upstream Port (rev 02)

[1022:43b4] 03:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)

[1022:43b4] 03:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)

[1022:43b4] 03:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)

[1022:43b4] 03:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)

[1022:43b4] 03:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)

[1022:43b4] 03:07.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02)

[1b21:1242] 04:00.0 USB controller: ASMedia Technology Inc. ASM1142 USB 3.1 Host Controller

[8086:1539] 05:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)

[1106:3483] 06:00.0 USB controller: VIA Technologies, Inc. VL805 USB 3.0 Host Controller (rev 01)

[1106:3483] 07:00.0 USB controller: VIA Technologies, Inc. VL805 USB 3.0 Host Controller (rev 01)

 

Is this a dead end?

 

[faicec]

OK,

 

Enabling pcie_acs_override=downstream,multifunction has resolved this to some extent; more IOMMU groups, and I can now pass-thru the PCIe cards.

 

As a consequence, the GPU has been separated into 2 groups, 1 containing the VGA controller, the other the Audio. This shouldn't cause an issue, however I dont use the GPU Audio, instead I use the onboard audio.  The issue I now have, the VM will not output to the display unless I select the soundcard to be the GPU audio!

 

I have found that following a successful boot with the GPU being used as the soundcard, I power down the VM and select the onboard audio, and the VM will boot successfully.  So I guess this is only a problem following an UNRAID reboot.

 

I have stubbed out the audio device in group 34, however it doesnt show as an available PCI device to pass-thru.

 

IOMMU group 27:    [10de:1b82] 0a:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070 Ti] (rev a1)
IOMMU group 28:    [10de:10f0] 0a:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)
IOMMU group 34:    [1022:1457] 0c:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 00h-0fh) HD Audio Controller