Question about thunderbolt card and thunderbolt dock passthrough to VM

[Javen]

Hello,

 

Recently I bought a GC Titan ridge card and installed it with my Gigabyte B650M MB.

 

The issue comes when I tried to passthrough the whole card to my VM. I can somehow manage to make the card and dock work, but not fully.

 

Here is my finding:

1. The thunderbolt 3 dock and the GC Titan ridge card are both listed as PCIe devices in system devices and both can be bound to vfio at boot. (I think this might be the cause of the issue)

2. I can passthrough both devices(dock and card) to VM and it works including hotplugging USB devices on the dock and DP display. Thunderbolt software can detect the dock as well.

3. However, the big big drawback of 2# is that the dock is passed through as a PCIe device. That means the dock itself must be connected with the server always. Just as a standard PCIe device, unplug it will cause issue and even VM can't start without it. That means the hotplug feature of the dock itself is completely lost.

4. If I try to only passthrough the card itself, the card can be detected in Windows normally. However, devices connected to the dock will not be recognized. Even there is no any entries in device manager.

 

Is my understanding right that the issue comes because the dock itself is a PCIe device rather than a USB device? The logic is totally different compare to USB controller passthrough.

And is there a way to fix it?

Edited November 30, 2022 by Javen

[rootd00d]

Regarding #2, do you get an explicit option to map through the dock?  I only seem to be able to map through the USB4 and Thunderbolt controllers, but dock and USB hub devices are listed if I don't have VFIO enabled.

 

With my current configuration, the dock seems to get passed through normally and is detectable in Windows (Thunderbolt Control Center), but "regular" USB devices are not detected by the dock, such as wireless device receivers, USB audio devices,  (like your #4 scenario) but *will* be available to pass-through via libvirt USB -- however *storage* devices (flash drives, SD cards, external HDDs) do actually pass through completely fine using any of the USB-A, USB-C data, or Thunderbolt ports.

 

Currently looks like this --

 

I have no idea why I can't individually select these IOMMU groups.  A quick albeit limited test of the video passthrough didn't seem to be working either -- might need to be plugged in at boot as per some other comments I've seen.

 

saturn-diagnostics-20230312-1828.zip