IOMMU grouping issues for GPU even with ACS override enabled


Nano-uk

Recommended Posts

Hi,

I am running the following hardware:

 

CPU: AMD Ryzen 5 1600

Motherboard: Gigabyte B350M-GAMING3

RAM: 16 GB DDR4 Corsair (3200mhz)

VGA1 - Console: XFX R5 230

VGA2 - main GPU for the VM: Radeon R9 270

LAN: Onboard 1 gbps

HBA card: PCIe 1x Marvell 88SE9215 Chipset single chip 4-port Gen III 6Gb/s host controller

 

Problem description:

Since I first built my unraid system I haven't been able to pass-trough my R9 270 GPU for my windows10 vm. I keep getting the following error:

 

Execution error

 

internal error: qemu unexpectedly closed the monitor: 2017-09-04T12:18:22.818572Z qemu-system-x86_64: -device vfio-pci,host=07:00.0,id=hostdev0,bus=pci.0,addr=0x4: vfio: error, group 10 is not viable, please ensure all devices within the iommu_group are bound to their vfio bus driver.
2017-09-04T12:18:22.818593Z qemu-system-x86_64: -device vfio-pci,host=07:00.0,id=hostdev0,bus=pci.0,addr=0x4: vfio: failed to get group 10
2017-09-04T12:18:22.818601Z qemu-system-x86_64: -device vfio-pci,host=07:00.0,id=hostdev0,bus=pci.0,addr=0x4: Device initialization faileda

 

After reading several posts I realized about the IOMMU grouping issue and that I need to isolate my GPU it's own IOMMU group. To this end I tried different things:

 

1)  PCIe stub following this guide (but replacing the NIC for my GPU of course): 

2) I tried adding vfio_iommu_type1.allow_unsafe_interrupts=1 on my syslinux.cfg

 

3) I tried enabling ACS override.

 

But after all this, I still get the following IOMMU grouping:

 

IOMMU group 0
	[1022:1452] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
IOMMU group 1
	[1022:1453] 00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453
IOMMU group 2
	[1022:1452] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
IOMMU group 3
	[1022:1452] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
IOMMU group 4
	[1022:1453] 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1453
IOMMU group 5
	[1022:1452] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
IOMMU group 6
	[1022:1452] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
IOMMU group 7
	[1022:1454] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1454
IOMMU group 8
	[1022:1452] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1452
IOMMU group 9
	[1022:1454] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 1454
IOMMU group 10
	[1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 59)
	[1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
IOMMU group 11
	[1022:1460] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1460
	[1022:1461] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1461
	[1022:1462] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1462
	[1022:1463] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1463
	[1022:1464] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1464
	[1022:1465] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1465
	[1022:1466] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1466
	[1022:1467] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Device 1467
IOMMU group 12
	[1022:43bb] 03:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 43bb (rev 02)
	[1022:43b7] 03:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] Device 43b7 (rev 02)
	[1022:43b2] 03:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b2 (rev 02)
	[1022:43b4] 04:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
	[1022:43b4] 04:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
	[1022:43b4] 04:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
	[10ec:8168] 05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
	[1b4b:9215] 06:00.0 SATA controller: Marvell Technology Group Ltd. Device 9215 (rev 11)
	[1002:6811] 07:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Curacao PRO [Radeon R7 370 / R9 270/370 OEM]
	[1002:aab0] 07:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]
IOMMU group 13
	[1002:6779] 09:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Caicos [Radeon HD 6450/7450/8450 / R5 230 OEM]
	[1002:aa98] 09:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6450 / 7450/8450/8490 OEM / R5 230/235/235X OEM]
IOMMU group 14
	[1022:145a] 11:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 145a
	[1022:1456] 11:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Device 1456
	[1022:145c] 11:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Device 145c
IOMMU group 15
	[1022:1455] 12:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Device 1455
	[1022:7901] 12:00.2 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
	[1022:1457] 12:00.3 Audio device: Advanced Micro Devices, Inc. [AMD] Device 1457

My console GPU (VGA1) seams to be OK on it's own group (IOMMU group 13), while my main GPU keeps appearing in a group with many other devices regardless of what I try.

 

Any ideas?

 

Thank you in advanced for any help you can provide!

 

Ignacio.

Link to comment

 

4 minutes ago, Squid said:

Try a different slot

 

Yeah, just a few minutes ago I decided to swap my GPU's and now my R9 270 (installed on the top PCIe slot) started to work on my vm

 

IOMMU group 12
	[1022:43bb] 03:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 43bb (rev 02)
	[1022:43b7] 03:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] Device 43b7 (rev 02)
	[1022:43b2] 03:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b2 (rev 02)
	[1022:43b4] 04:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
	[1022:43b4] 04:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
	[1022:43b4] 04:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b4 (rev 02)
	[10ec:8168] 05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
	[1b4b:9215] 06:00.0 SATA controller: Marvell Technology Group Ltd. Device 9215 (rev 11)
	[1002:6779] 07:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Caicos [Radeon HD 6450/7450/8450 / R5 230 OEM]
	[1002:aa98] 07:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6450 / 7450/8450/8490 OEM / R5 230/235/235X OEM]
IOMMU group 13
	[1002:6811] 09:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Curacao PRO [Radeon R7 370 / R9 270/370 OEM]
	[1002:aab0] 09:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Cape Verde/Pitcairn HDMI Audio [Radeon HD 7700/7800 Series]

 

Now they swapped places. My console video card is on group 12 along with many other devices but at least now I can pass-trough my video card.

 

Thanks for the help! :)

Link to comment
On 06/09/2017 at 1:55 AM, RonUSMC said:

I am almost running the same system as you. Basically the second PCIE slot will not get in a decent grouping at all.

 

Yeah, in my case the solution was to move the primary card (R9 270) into the top PCI-e slot and the secondary card to the bottom one. The R9 270 acts both as console card and VM VGA card when I start my windows 10 virtual machine.

 

Link to comment
On 9/8/2017 at 9:17 AM, Nano-uk said:

 

Yeah, in my case the solution was to move the primary card (R9 270) into the top PCI-e slot and the secondary card to the bottom one. The R9 270 acts both as console card and VM VGA card when I start my windows 10 virtual machine.

 

 

Nothing I tried could get the bottom PCIe slot to be isolated. I couldn't even get the vram from it when it was in the second slot. Are you able to use the secondary card for anything at all?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.