Hi,

 

I have HW passtrhough working perfectly for my dGPU a 9070.

 

Now I need to also passthrough mi iGPU (Ryten 9950) to another VM. I'm close but I'm encountering a wired problem.

 

If I start the VM with VNC as primary GPU, then my Kubuntu VM boots up normally. Once logged in I can switch primary display to my monitor and turn off the VNC display. Now I have a picture on my monitor using Thunderbolt (USB4-DP-ALT). I can reboot the VM and it will boot normally except that I'm sitting in front of a black screen until I blindly enter my pwd and then my monitor shows a picture again and all is OK.

 

However if I remove the VNC GPU from my VM config (leaving everything else untouched) then my VM does never output anything to my screen. Waiting and blindly entering pwd does not do the trick either.

 

I have a theory that the VM without detecting a monitor does not boot at all.

Quote

Update: This is wrong, the VM  requests an IP form DHCP and is actually pingable. I will setup ssh so I can hopefully login and get logs. My new assumption is, that Kubuntu does not see any display and therefore refuses to start the GUI. Why it's able to see/find the Thunderbolt displays once the GUI is started I do not understand.

 

How can I fix this? I have tried a OVMF and SEABIOS. Same behavior.

It's not the reset bug. I can start/stop the VM with the VNC attached multiple times and get output via iGPU.

 

I have also tried to modify the VM's GRUB with 

GRUB_CMDLINE_LINUX_DEFAULT="nomodeset"

and updated GRUB with

"sudo update-grub"

 

As output I have to use USB4 Thunderbolt since I need to connect 2 monitors. My HDMI port does not output anything anyway. My theory here is that the HDMI port is not accessible form the VM because it's still bound to the CPU (probably a part I cannot passthrough, or at least I have no clue how to figure out what part).

In order to get a picture via Thunderbolt I also have to passthrough the ASMedia USB Controller where those USB4 ports are connected to. I got this information from my motherboard's manual:

(Asrock X870E Taichi Lite)

 

I have extracted the VBIOS and AMDGopDriver, although currently I don't use the GOP ROM

 

Here is my lspci:

 

root@Tower:~# lspci
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Root Complex
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge IOMMU
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge
00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge GPP Bridge
00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge GPP Bridge
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge
00:02.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge GPP Bridge
00:02.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge GPP Bridge
00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge
00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Dummy Host Bridge
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Internal GPP Bridge to Bus [C:A]
00:08.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Internal GPP Bridge to Bus [C:A]
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 71)
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 3
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 5
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 6
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge Data Fabric; Function 7
01:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch (rev 24)
02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch (rev 24)
03:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Device 7550 (rev c0)
03:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Device ab40
04:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller S4LV008[Pascal]
05:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Upstream Port (rev 01)
06:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01)
06:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01)
06:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01)
06:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01)
06:07.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01)
06:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01)
06:0c.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01)
06:0d.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01)
0a:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8126 5GbE Controller (rev 01)
0b:00.0 Network controller: MEDIATEK Corp. Device 0717
0c:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Upstream Port (rev 01)
0d:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01)
0d:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01)
0d:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01)
0d:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01)
0d:07.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01)
0d:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01)
0d:0c.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01)
0d:0d.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset PCIe Switch Downstream Port (rev 01)
13:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller S4LV008[Pascal]
14:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] 800 Series Chipset USB 3.x XHCI Controller (rev 01)
15:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset SATA Controller (rev 01)
16:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] 800 Series Chipset USB 3.x XHCI Controller (rev 01)
17:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] 600 Series Chipset SATA Controller (rev 01)
18:00.0 PCI bridge: ASMedia Technology Inc. Device 2421 (rev 01)
19:00.0 PCI bridge: ASMedia Technology Inc. Device 2423 (rev 01)
19:01.0 PCI bridge: ASMedia Technology Inc. Device 2423 (rev 01)
19:02.0 PCI bridge: ASMedia Technology Inc. Device 2423 (rev 01)
19:03.0 PCI bridge: ASMedia Technology Inc. Device 2423 (rev 01)
1a:00.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Goshen Ridge 2020] (rev 03)
1b:00.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Goshen Ridge 2020] (rev 03)
1b:01.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Goshen Ridge 2020] (rev 03)
1b:02.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Goshen Ridge 2020] (rev 03)
1b:03.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Goshen Ridge 2020] (rev 03)
1b:04.0 PCI bridge: Intel Corporation Thunderbolt 4 Bridge [Goshen Ridge 2020] (rev 03)
7a:00.0 USB controller: ASMedia Technology Inc. Device 2426 (rev 01)
7b:00.0 USB controller: ASMedia Technology Inc. Device 2425 (rev 01)
7c:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Granite Ridge [Radeon Graphics] (rev c9)
7c:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Rembrandt Radeon High Definition Audio Controller
7c:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 19h PSP/CCP
7c:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge USB 3.1 xHCI
7c:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Raphael/Granite Ridge USB 3.1 xHCI
7c:00.6 Audio device: Advanced Micro Devices, Inc. [AMD] Family 17h/19h/1ah HD Audio Controller
7d:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 15b8
root@Tower:~# 

 

This is my iGPU which I pass as multifunction device with my extracted vbios:

7c:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Granite Ridge [Radeon Graphics] (rev c9)
7c:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Rembrandt Radeon High Definition Audio Controller

 

Here the ASMedia USB4 "Thunderbolt" controller I also need in order to get a picture on my monitor:

7b:00.0 USB controller: ASMedia Technology Inc. Device 2425 (rev 01)

 

What I'm wondering is if I also have to passthrough the 6 Thunderbolt bridges above. But I can't bind them to VFIO (at least not via GUI) and then I also have no clue what I would need to modify the VM config XML.

 

Any hints? Getting this to work it the last piece of my puzzle. Any support is highly appreciated. Thank you

 

Edited March 29, 20251 yr by Tom082

Update:

 

I was wrong about the HDMI port. My VM can use it. The trick is to unplug all the monitor cables (HDMI, DP, Thunderbolt) at host boot time so that neither the BIOS/EFI nor Unraid have a chance to detect the monitor and initialize it. Once the iGPU is bound to VFIO it's safe to plug the cables back in. Starting the VM will produce a picture on the HDMI attached monitor.

Maybe switching off all the monitors during host boot instead of unplugging might be enough -  I have not tested this yet though.

In my case configuring the BIOS to use the dGPU instead of the iGPU does somehow not prevent the BIOS/EFI to use the output ports of the iGPU if the monitors are plugged in and detected.

 

In the meantime my VM also runs without the VNC GPU. When I was first setting up my Kubuntu-VM I did so only with the VNC GPU attached (no passthrough, neither iGPU nor the thunderbolt controller). Reinstalling the VM with both the iGPU and thunderbolt controller attached to the VM from the very beginning did solve the issue and now my Kubuntu-VM displays the login screen directly on my monitor, no VNC needed. Since I'm still a Linux Noob with only 2 weeks of experience now, my best guess is that installing the VM with only VNC did not install or load the drivers for the iGPU and Thunderbolt Controller and they were only loaded/activated after the GUI login. As said, this is just a guess.

 

I can now repeatedly start/stop/reboot the VM with the iGPU attached and also move the iGPU between different VMs. I have done at least 20 reboot cycles, 10 start/stop cycles and moved the iGPU back and forth between 2 different VMs.

I do not have the AMD_vendor-reset" plugin installed. I could not install it if I needed to, because I have to run a 6.13.5 or newer kernel in order for my 9070 to work correctly.

 

This leaves me with one last issue to solve. Thunderbolt!

As mentioned passing through the Thunderbolt Controller is required in order to get a picture to the 2nd monitor. I did test Thunderbolt DP-MST already and it works. In order to get an actual picture on the attached monitors I have to unplug the Thunderbolt cable during host boot. Once the Thunderbolt controller is attached to VFIO it's safe to attach the cable.

The VM does recognize Thunderbolt and monitors attached to it. I do get pictures. What is not working at least not reliably and repeatedly are VM reboots and start/stop cycles. Unfortunately at some point my monitors that are using Thunderbolt remain black. The VM reports that the monitors are not attached. HDMI remains working though. So it looks to me that the issue is with the Thunderbolt controller and not the iGPU.

Unraid reports that the Thunderbolt controller has and supports function level resets (FLR). Also in the Unraid logs I can clearly see the the Thunderbolt controller goes though successful resets (at least according to Unraid) even if the Thunderbolt monitors remain black.

 

Mar 29 11:53:47 Tower kernel: vfio-pci 0000:7c:00.0: resetting
Mar 29 11:53:47 Tower kernel: vfio-pci 0000:7c:00.0: reset done
Mar 29 11:53:47 Tower kernel: vfio-pci 0000:7c:00.1: resetting
Mar 29 11:53:47 Tower kernel: vfio-pci 0000:7c:00.1: reset done
Mar 29 11:53:47 Tower kernel: vfio-pci 0000:7b:00.0: resetting
Mar 29 11:53:47 Tower kernel: vfio-pci 0000:7b:00.0: reset done

7c is the GPU, 7b is the Thunderbolt controller.

 

How I can debug and fix this? Any hints and tips? Thank you very much!

 

Except for the Thunderbolt issue the solution for blackscreen and login issues are the following:

 

Blackscreen:

- detach all video cables from the iGPU (this includes USB-C in case of Thunderbolt).

- boot into Unraid

- attach video cables to the iGPU

- start VM

 

Black screen until after login:

- install the VM with VNC plus iGPU otherwise the VM might not start the required drivers

 

For the Thunderbolt instability I opened another thread at Level1

Edited March 30, 20251 yr by Tom082

I reached a point that I can output iGPU of my Ryzen 9 using Ubuntu VM or Zorin Linux VM, but everytimes I uses a Windows VM (even with primary as iGPU), I didn't get any screen output to the monitor.

Wonder if there's any guide that I can follow to make Windows 11 works on the iGPU, and any reason it works well on Linux/Ubuntu but not for Windows when it comes to the iGPU passthrough? Any guidance will be much appreciated