GPU Passthrough Issues (solved)

[Urumiko]

Hi. I'm a complete newbie. I'm trying to pass through my GPU.

I get the following error:

internal error: qemu unexpectedly closed the monitor: 2020-07-19T22:47:48.608490Z qemu-system-x86_64: -device vfio-pci,host=0000:08:00.0,id=hostdev0,bus=pci.0,addr=0x4,romfile=/mnt/disk1/Dump/MSI.GTX1660.6144.190305.rom: vfio 0000:08:00.0: group 21 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver.

 

I tried adding the config:

 

append pci-sub.ids=10de:2184  but that has no effect.

 

 

PCI Devices and IOMMU Groups

IOMMU group 0:[1022:1482] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 1:[1022:1483] 00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge

IOMMU group 2:[1022:1482] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 3:[1022:1482] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 4:[1022:1483] 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge

IOMMU group 5:[1022:1482] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 6:[1022:1482] 00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 7:[1022:1482] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 8:[1022:1484] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]

IOMMU group 9:[1022:1482] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 10:[1022:1484] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]

IOMMU group 11:[1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)

[1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)

IOMMU group 12:[1022:1440] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0

[1022:1441] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1

[1022:1442] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2

[1022:1443] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3

[1022:1444] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4

[1022:1445] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5

[1022:1446] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6

[1022:1447] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7

IOMMU group 13:[1022:57ad] 01:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream

IOMMU group 14:[1022:57a3] 02:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge

IOMMU group 15:[1022:57a3] 02:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge

IOMMU group 16:[1022:57a4] 02:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge

[1022:1485] 05:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP

[1022:149c] 05:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller

[1022:149c] 05:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller

IOMMU group 17:[1022:57a4] 02:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge

[1022:7901] 06:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)

IOMMU group 18:[1022:57a4] 02:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge

[1022:7901] 07:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)

IOMMU group 19:[15b7:5001] 03:00.0 Non-Volatile memory controller: Sandisk Corp WD Black NVMe SSD

IOMMU group 20:[8086:1539] 04:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)

IOMMU group 21:[10de:2184] 08:00.0 VGA compatible controller: NVIDIA Corporation TU116 [GeForce GTX 1660] (rev a1)

[10de:1aeb] 08:00.1 Audio device: NVIDIA Corporation TU116 High Definition Audio Controller (rev a1)

[10de:1aec] 08:00.2 USB controller: NVIDIA Corporation Device 1aec (rev a1)

[10de:1aed] 08:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU116 [GeForce GTX 1650 SUPER] (rev a1)

IOMMU group 22:[1022:148a] 09:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function

IOMMU group 23:[1022:1485] 0a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP

IOMMU group 24:[1022:1486] 0a:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP

IOMMU group 25:[1022:149c] 0a:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller

IOMMU group 26:[1022:1487] 0a:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller

 

CPU Thread Pairings

Pair 1:cpu 0 / cpu 4

Pair 2:cpu 1 / cpu 5

Pair 3:cpu 2 / cpu 6

Pair 4:cpu 3 / cpu 7

 

USB Devices

Bus 001 Device 001:ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 002 Device 001:ID 1d6b:0003 Linux Foundation 3.0 root hub

Bus 003 Device 001:ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 004 Device 001:ID 1d6b:0003 Linux Foundation 3.0 root hub

Bus 004 Device 002:ID 0781:5583 SanDisk Corp. Ultra Fit

Bus 005 Device 001:ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 006 Device 001:ID 1d6b:0003 Linux Foundation 3.0 root hub

Bus 007 Device 001:ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 007 Device 002:ID 0499:1509 Yamaha Corp. Steinberg UR22

Bus 007 Device 003:ID 1a40:0101 Terminus Technology Inc. Hub

Bus 007 Device 004:ID 046d:c52b Logitech, Inc. Unifying Receiver

Bus 007 Device 005:ID 046d:c24d Logitech, Inc. G710 Gaming Keyboard

Bus 008 Device 001:ID 1d6b:0003 Linux Foundation 3.0 root hub

 

SCSI Devices

[0:0:0:0]disk SanDisk Ultra Fit 1.00 /dev/sda 62.1GB

[1:0:0:0]disk ATA ST3500641AS J /dev/sdb 500GB

[4:0:0:0]cd/dvd PIONEER BD-RW BDR-207 1.21 /dev/sr0 -

[5:0:0:0]disk ATA OCZ-VERTEX3 .20 /dev/sdc 120GB

[6:0:0:0]disk ATA OCZ-VERTEX3 .20 /dev/sdd 120GB

[N:0:0:1]disk A400 NVMe SanDisk 512GB__1 /dev/nvme0n1 512GB

Edited July 21, 2020 by Urumiko

[rachid596]

It's because you have to passthrough nvidia usb. 

Try with this in your syslinux configuration. 

 

vfio-pci.ids=10de:1aec,10de:1aed

[Urumiko]

11 hours ago, rachid596 said:

It's because you have to passthrough nvidia usb. 

Try with this in your syslinux configuration. 

 

vfio-pci.ids=10de:1aec,10de:1aed

Thank you. I'll give this a try when I get home. Its a bit odd really. There is no USB etc on the card. I can only assume its something to do with VR headsets, gsync, or some other display port shenanigans.

[testdasi]

3 hours ago, Urumiko said:

Thank you. I'll give this a try when I get home. Its a bit odd really. There is no USB etc on the card. I can only assume its something to do with VR headsets, gsync, or some other display port shenanigans.

You missed the point a bit here.

The device (graphics card) has 4 functions (you can think of it as sub-devices) so all have to be passed through together, regardless if the functionality is used or not. Passing through just part of the 4 functions is the software equivalent of physically take a chainsaw to cut your device into two.

[Urumiko]

20 hours ago, testdasi said:

You missed the point a bit here.

The device (graphics card) has 4 functions

Yes, I got a better handle on this thanks to some videos on youtube by "Spaceinvader one"

It seems I wasn't having the usual issue where it splits the card on to different slots, but rather it was only including the parent device itself and not including the sub functions at all. 
I wrote my own entries for these (and uploaded a vbios), and all seems well.
Further to this I also passed through my NVME drive and am booting to this via modified clover image.

 

The only thing I'm not sure on, is at present, in my physical system, different devices have different buses all to themselves.

At the moment they are mapped on the virtual side all to one bus and then instead different slots within that bus.

Is this an issue?

Can anyone see any optimizations I should make?

I have the certified thunderbolt add on card for my mobo on the way and the real acid test will be if I can pass this through and use my ASIO thunderbolt audio interface effectively without issues.

 

I have attached my VM config.

qwer.txt

Edited July 21, 2020 by Urumiko