Vaggeto Posted October 20, 2020 Share Posted October 20, 2020 Hello, I have an passthrough issue and I'm stuck on next steps. Motherboard is Gigabyte TRX40 Aorus Master and AMD Ryzen Threadrupper 3970x. Essentially my VM won't load when I try to pass through either onboard audio, or a USB controller even though they are on their own IOMMU groupings. I'm using ACS Downstream but have tried all ACD types. I have two USB controllers on separate IOMMU groupings, but they have the same vendor ID even though they are on different IOMMU groupings, even without ACS enabled. I've tried every method i could find but whenever I start the VM, it basically hangs on that device and the log keeps updating that it's waiting for 23:00.3 (the USB controller in question or whichever device I'm trying to pass through) and then eventually gives up, but I'm unable to stop the VM or even reboot through unRAID (GUI or command line) at that time and I have to do a hard reset. The error in the log is: vfio-pci 0000:23:00.3: not ready 32767ms after FLR; waiting which repeats about 5 times until it fails but everything remains locked up. If I just pass through the USB device itself, it seems to work fine but I need to pass the USB controller and also need the on-board audio. Here are my IOMMU groupings: (Note Groups 17 and 32 for USB controllers, and group 33 for the onboard audio.) IOMMU group 0: [1022:1482] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 1: [1022:1483] 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge IOMMU group 2: [1022:1483] 00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge IOMMU group 3: [1022:1482] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 4: [1022:1482] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 5: [1022:1482] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 6: [1022:1482] 00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 7: [1022:1482] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 8: [1022:1484] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] IOMMU group 9: [1022:1482] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 10: [1022:1484] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] IOMMU group 11: [1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61) [1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51) IOMMU group 12: [1022:1490] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 0 [1022:1491] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 1 [1022:1492] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 2 [1022:1493] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 3 [1022:1494] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 4 [1022:1495] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 5 [1022:1496] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 6 [1022:1497] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship Device 24; Function 7 IOMMU group 13: [15b7:5006] 01:00.0 Non-Volatile memory controller: Sandisk Corp Device 5006 IOMMU group 14: [15b7:5006] 02:00.0 Non-Volatile memory controller: Sandisk Corp Device 5006 IOMMU group 15: [1022:148a] 03:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function IOMMU group 16: [1022:1485] 04:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP IOMMU group 17: [1022:148c] 04:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller IOMMU group 18: [1022:1482] 20:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 19: [1022:1482] 20:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 20: [1022:1482] 20:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 21: [1022:1483] 20:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge IOMMU group 22: [1022:1482] 20:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 23: [1022:1482] 20:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 24: [1022:1482] 20:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 25: [1022:1484] 20:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] IOMMU group 26: [1022:1482] 20:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 27: [1022:1484] 20:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] IOMMU group 28: [10de:1c02] 21:00.0 VGA compatible controller: NVIDIA Corporation GP106 [GeForce GTX 1060 3GB] (rev a1) [10de:10f1] 21:00.1 Audio device: NVIDIA Corporation GP106 High Definition Audio Controller (rev a1) IOMMU group 29: [1022:148a] 22:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function IOMMU group 30: [1022:1485] 23:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP IOMMU group 31: [1022:1486] 23:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP IOMMU group 32: [1022:148c] 23:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Starship USB 3.0 Host Controller IOMMU group 33: [1022:1487] 23:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller IOMMU group 34: [1022:1482] 40:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 35: [1022:1483] 40:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge IOMMU group 36: [1022:1482] 40:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 37: [1022:1482] 40:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 38: [1022:1482] 40:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 39: [1022:1482] 40:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 40: [1022:1482] 40:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 41: [1022:1484] 40:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] IOMMU group 42: [1022:1482] 40:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 43: [1022:1484] 40:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] IOMMU group 44: [1022:57ad] 41:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream IOMMU group 45: [1022:57a3] 42:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge IOMMU group 46: [1022:57a3] 42:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge IOMMU group 47: [1022:57a3] 42:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge IOMMU group 48: [1022:57a4] 42:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge IOMMU group 49: [1022:57a4] 42:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge IOMMU group 50: [1022:57a4] 42:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge IOMMU group 51: [8086:1539] 43:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03) IOMMU group 52: [1d6a:11b1] 44:00.0 Ethernet controller: Aquantia Corp. AQC111 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] (rev 02) IOMMU group 53: [8086:2723] 45:00.0 Network controller: Intel Corporation Wi-Fi 6 AX200 (rev 1a) IOMMU group 54: [1022:1485] 46:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:149c] 46:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c] 46:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller IOMMU group 55: [1022:7901] 47:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) IOMMU group 56: [1022:7901] 48:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51) IOMMU group 57: [1022:148a] 49:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function IOMMU group 58: [1022:1485] 4a:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP IOMMU group 59: [1022:1482] 60:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 60: [1022:1482] 60:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 61: [1022:1482] 60:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 62: [1022:1482] 60:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 63: [1022:1482] 60:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 64: [1022:1482] 60:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 65: [1022:1484] 60:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] IOMMU group 66: [1022:1482] 60:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge IOMMU group 67: [1022:1484] 60:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] IOMMU group 68: [1022:148a] 61:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function IOMMU group 69: [1022:1485] 62:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP I tried the following three methods for passing the USB controller(s): So when I use the vfio-pci.ids=1022:148c I get both USB controllers shown in my list of PCIe devices. Now technically I really don't mind this if I can pass them each to separate VMs, but I can't even get one or both to pass to the VM and have it load. I also tried this xen-pciback.hide=(23:00.3) method and added the XML for the single controller <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x23' slot='0x00' function='0x3'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> I also tried the VFIO-PCI Config plugin and choose the controller at 23:00.3 and that properly presents it within my PCI device list but the VM still hangs on waiting for the USB controller. Any other thoughts or ideas on what I could try? Quote Link to comment
Vaggeto Posted October 20, 2020 Author Share Posted October 20, 2020 (edited) Well thanks to this post, I have been able to figure out the on-board audio piece of this. Turns out the MB reports them as "USB Audio Devices" and you just pass them through as USB devices and ignore the actual IOMMU group 33: [1022:1487] 23:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller. (I haven't tried any other methods to pass these USB audio devices since they are both part of a much larger IOMMU group which also have the unRAID USB flash drive port) I still cannot find a way to pass-through these entire USB controllers vs only USB devices though for mouse/keyboard and other USB devices. Edited October 20, 2020 by Vaggeto Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.