GPU Passthrough for LED control & questions regarding PCI, IOMMU groups etc.

[AceRimmer]

Forgive me if my knowledge of this subject isn't great. I am only about 7 or 8 months into using Unraid.

 

So this is stemming from my Nvidia 2070 Super GPU in my rig and its passed through to a Windows VM. GPU works perfectly passed through but i have no control over the LED's on the device. Windows will not detect them and i'm left with a glowing rainbow in my room 24/7. 

 

So in relation to the GPU I have its IOMMU group stubbed on the VFIO file (see below), however on booting Unraid this GPU is still outputting video and its not until the VM manager wakes after boot up that Unraid passes the GPU to the VM. 

 

IOMMU group 25:[10de:1e84] 0a:00.0 VGA compatible controller: NVIDIA Corporation TU104 [GeForce RTX 2070 SUPER] (rev a1)

[10de:10f8] 0a:00.1 Audio device: NVIDIA Corporation TU104 HD Audio Controller (rev a1)

[10de:1ad8] 0a:00.2 USB controller: NVIDIA Corporation TU104 USB 3.1 Host Controller (rev a1)

This controller is bound to vfio, connected USB devices are not visible.

[10de:1ad9] 0a:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU104 USB Type-C UCSI Controller (rev a1)

 

Regardless of the stubbing Unraid still sees & recognizes this device as a "passthroughable" GPU for a VM the same way it would see it if i didn't have it stubbed (see below image). This is leading me to think there are 2 possible scenarios happening.

 

 

Since Unraid has control of the GPU during boot (it outputs to the display) and it still lists it as an available graphics card (as below) it hasn't fully stubbed the device and that is why i can't control the LED's on the device. Unraid has held on to the LED controller but The LED controller isn't listing in Unraid or on the IOMMU group because there is no driver for it on Unraid (the same way folk have issues with bluetooth adaptors not appearing in Unraid before there was any driver support).

 

So I have a few questions you may be able to answer:

 

1. Is there any point of stubbing the GPU since Unraid still sees and recognizes the device as a "passthroughable" GPU the same way it would if i didn't stub it?
    
2. Is it possible to stub an entire PCI lane to a VM? 
    
3. Is it safe to stub manually (as in physically typing it into the VFIO config file on the flash drive) since many of the PCI related bridges can't be selected in the VFIO bind list in system devices. Is the system likely doing that for a good reason (power management for starters or messing with the north/south bridge's control over a device?)
    
4. Pending that stubbing a PCI lane can be done how do i tell what IOMMU group PCI lane my GPU is attached to? Below are the IOMMU groups i can't select to VFIO bind but i also have no reference as to what their purpose is on the motherboard - REFER to below FIG-A
    
5. Finally just for my curiosity i'm still left with some remaining devices on my IOMMU list that i can stub but is there a way to tell what these devices actually are? Obviously i can identify the spare NIC and encrypt controller but the remaining devices i don't have any reference as to what they do. Should i be sending them to my main Windows VM. Am i loosing out by not doing so, example some aided hashing functionality on Windows by not passing through the Cryptographic Coprocessor or does Unraid actually use this but it just lists as being "stubbable" anyway? REFER to below FIG-B

 

FIG-A

IOMMU group 0:                [1022:1482] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 1:                [1022:1483] 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
IOMMU group 2:                [1022:1483] 00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
IOMMU group 3:                [1022:1482] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 4:                [1022:1482] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 5:                [1022:1483] 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
IOMMU group 6:                [1022:1482] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 7:                [1022:1482] 00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 8:                [1022:1482] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 9:                [1022:1484] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
IOMMU group 10:                [1022:1482] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 11:                [1022:1484] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
IOMMU group 15:				[1022:57ad] 02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream
IOMMU group 16:				[1022:57a3] 03:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
IOMMU group 17:				[1022:57a3] 03:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
IOMMU group 18:				[1022:57a3] 03:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge

 

FIB-B

IOMMU group 12:[1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)
[1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
IOMMU group 24:[8086:1539] 06:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
IOMMU group 26:[1022:148a] 0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
IOMMU group 27:[1022:1485] 0c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
IOMMU group 28:[1022:1486] 0c:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP

 

 

 

 

 

Edited February 15, 2021 by CatMilk