Hi @rachid596
I have a Gigabyte X570 aorus master + 5900x, I am on bios F34 with agesa combov2 1.2.0.3B, also tried with with bios F33i agesa 1.2.0.1A with same problem. I run unraid 6.9.2, which have kernel 5.10, which should have the necessary patches, unless the patches were dropped?
I am trying to passthrough the audio controller, my IOMMU:
IOMMU group 0: [1022:1482] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 1: [1022:1483] 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
IOMMU group 2: [1022:1483] 00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
IOMMU group 3: [1022:1482] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 4: [1022:1482] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 5: [1022:1483] 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
IOMMU group 6: [1022:1483] 00:03.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
IOMMU group 7: [1022:1482] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 8: [1022:1482] 00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 9: [1022:1482] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 10: [1022:1484] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
IOMMU group 11: [1022:1482] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
IOMMU group 12: [1022:1484] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
IOMMU group 13: [1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)
[1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
IOMMU group 14: [1022:1440] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0
[1022:1441] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1
[1022:1442] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2
[1022:1443] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3
[1022:1444] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4
[1022:1445] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5
[1022:1446] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6
[1022:1447] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7
IOMMU group 15: [1e0f:0009] 01:00.0 Non-Volatile memory controller: KIOXIA Corporation NVMe SSD (rev 01)
[N:0:1:1] disk KIOXIA-EXCERIA SSD__1 /dev/nvme0n1 1.00TB
IOMMU group 16: [1022:57ad] 02:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse Switch Upstream
IOMMU group 17: [1022:57a3] 03:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
IOMMU group 18: [1022:57a3] 03:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
IOMMU group 19: [1022:57a3] 03:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
IOMMU group 20: [1022:57a3] 03:03.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
IOMMU group 21: [1022:57a3] 03:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
IOMMU group 22: [1022:57a3] 03:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
IOMMU group 23: [1022:57a3] 03:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
IOMMU group 24: [1022:57a4] 03:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
[1022:1485] 0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
[1022:149c] 0b:00.1 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 001 Device 002: ID 048d:8297 Integrated Technology Express, Inc. ITE Device(8595)
Bus 001 Device 003: ID 05e3:0608 Genesys Logic, Inc. Hub
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
[1022:149c] 0b:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
Bus 003 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
Bus 003 Device 002: ID 8087:0029 Intel Corp.
Bus 003 Device 003: ID 05e3:0608 Genesys Logic, Inc. Hub
Bus 003 Device 004: ID 13fe:4300 Kingston Technology Company Inc. USB DISK 2.0
Bus 003 Device 005: ID 046d:c539 Logitech, Inc. USB Receiver
Bus 003 Device 006: ID 046a:0010 Cherry GmbH SmartBoard XX44
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
IOMMU group 25: [1022:57a4] 03:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
[1022:7901] 0c:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
IOMMU group 26: [1022:57a4] 03:0a.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Matisse PCIe GPP Bridge
[1022:7901] 0d:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
[5:0:0:0] disk ATA HGST HUH721212AL T3R0 /dev/sda 12.0TB
[9:0:0:0] disk ATA HGST HUH721212AL T3R0 /dev/sdd 12.0TB
IOMMU group 27: [1e0f:0009] 04:00.0 Non-Volatile memory controller: KIOXIA Corporation NVMe SSD (rev 01)
[N:1:1:1] disk KIOXIA-EXCERIA SSD__1 /dev/nvme1n1 1.00TB
IOMMU group 28: [1e0f:0009] 05:00.0 Non-Volatile memory controller: KIOXIA Corporation NVMe SSD (rev 01)
[N:2:1:1] disk KIOXIA-EXCERIA SSD__1 /dev/nvme2n1 1.00TB
IOMMU group 29: [10de:1b81] 06:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)
[10de:10f0] 06:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)
IOMMU group 30: [8086:2723] 07:00.0 Network controller: Intel Corporation Wi-Fi 6 AX200 (rev 1a)
IOMMU group 31: [8086:1539] 08:00.0 Ethernet controller: Intel Corporation I211 Gigabit Network Connection (rev 03)
IOMMU group 32: [10ec:8125] 09:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
IOMMU group 33: [144d:a808] 0a:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983
[N:3:4:1] disk SAMSUNG MZVLB512HAJQ-000L7__1 /dev/nvme3n1 512GB
IOMMU group 34: [10de:1b82] 0e:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070 Ti] (rev a1)
[10de:10f0] 0e:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)
IOMMU group 35: [10de:1b81] 0f:00.0 VGA compatible controller: NVIDIA Corporation GP104 [GeForce GTX 1070] (rev a1)
[10de:10f0] 0f:00.1 Audio device: NVIDIA Corporation GP104 High Definition Audio Controller (rev a1)
IOMMU group 36: [1022:148a] 10:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
IOMMU group 37: [1022:1485] 11:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
IOMMU group 38: [1022:1486] 11:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP
IOMMU group 39: [1022:149c] 11:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
This controller is bound to vfio, connected USB devices are not visible.
IOMMU group 40: [1022:1487] 11:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller
I have the Audio device bound to VFIO:
Loading config from /boot/config/vfio-pci.cfg
BIND=0000:07:00.0|8086:2723 0000:11:00.0|1022:1485 0000:11:00.1|1022:1486 0000:11:00.3|1022:149c 0000:11:00.4|1022:1487
---
Processing 0000:07:00.0 8086:2723
Vendor:Device 8086:2723 found at 0000:07:00.0
IOMMU group members (sans bridges):
/sys/bus/pci/devices/0000:07:00.0/iommu_group/devices/0000:07:00.0
Binding...
Successfully bound the device 8086:2723 at 0000:07:00.0 to vfio-pci
---
Processing 0000:11:00.0 1022:1485
Vendor:Device 1022:1485 found at 0000:11:00.0
IOMMU group members (sans bridges):
/sys/bus/pci/devices/0000:11:00.0/iommu_group/devices/0000:11:00.0
Binding...
Successfully bound the device 1022:1485 at 0000:11:00.0 to vfio-pci
---
Processing 0000:11:00.1 1022:1486
Vendor:Device 1022:1486 found at 0000:11:00.1
IOMMU group members (sans bridges):
/sys/bus/pci/devices/0000:11:00.1/iommu_group/devices/0000:11:00.1
Binding...
Successfully bound the device 1022:1486 at 0000:11:00.1 to vfio-pci
---
Processing 0000:11:00.3 1022:149c
Vendor:Device 1022:149c found at 0000:11:00.3
IOMMU group members (sans bridges):
/sys/bus/pci/devices/0000:11:00.3/iommu_group/devices/0000:11:00.3
Binding...
Unbound 0000:11:00.3 from xhci_hcd
Successfully bound the device 1022:149c at 0000:11:00.3 to vfio-pci
---
Processing 0000:11:00.4 1022:1487
Vendor:Device 1022:1487 found at 0000:11:00.4
IOMMU group members (sans bridges):
/sys/bus/pci/devices/0000:11:00.4/iommu_group/devices/0000:11:00.4
Binding...
Successfully bound the device 1022:1487 at 0000:11:00.4 to vfio-pci
---
vfio-pci binding complete
Devices listed in /sys/bus/pci/drivers/vfio-pci:
lrwxrwxrwx 1 root root 0 Jul 14 16:22 0000:07:00.0 -> ../../../../devices/pci0000:00/0000:00:01.2/0000:02:00.0/0000:03:03.0/0000:07:00.0
lrwxrwxrwx 1 root root 0 Jul 14 16:22 0000:11:00.0 -> ../../../../devices/pci0000:00/0000:00:08.1/0000:11:00.0
lrwxrwxrwx 1 root root 0 Jul 14 16:22 0000:11:00.1 -> ../../../../devices/pci0000:00/0000:00:08.1/0000:11:00.1
lrwxrwxrwx 1 root root 0 Jul 14 16:22 0000:11:00.3 -> ../../../../devices/pci0000:00/0000:00:08.1/0000:11:00.3
lrwxrwxrwx 1 root root 0 Jul 14 16:22 0000:11:00.4 -> ../../../../devices/pci0000:00/0000:00:08.1/0000:11:00.4
ls -l /dev/vfio/
total 0
crw------- 1 root root 249, 0 Jul 14 16:22 30
crw------- 1 root root 249, 1 Jul 14 16:22 37
crw------- 1 root root 249, 2 Jul 14 16:22 38
crw------- 1 root root 249, 3 Jul 14 16:22 39
crw------- 1 root root 249, 4 Jul 14 16:22 40
crw-rw-rw- 1 root root 10, 196 Jul 14 16:22 vfio
my VM setting:
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x11' slot='0x00' function='0x4'/>
</source>
<alias name='hostdev2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</hostdev>
log from my VM:
-no-hpet \
-no-shutdown \
-boot strict=on \
-device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x7.0x7 \
-device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x7 \
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x7.0x1 \
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x7.0x2 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 \
-blockdev '{"driver":"file","filename":"/mnt/user/domains/Earth LLC/vdisk1.img","node-name":"libvirt-4-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-4-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-4-storage"}' \
-device virtio-blk-pci,bus=pci.0,addr=0x5,drive=libvirt-4-format,id=virtio-disk2,bootindex=1,write-cache=on \
-blockdev '{"driver":"file","filename":"/mnt/user/domains/Earth LLC/vdisk2.img","node-name":"libvirt-3-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-3-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"qcow2","file":"libvirt-3-storage","backing":null}' \
-device virtio-blk-pci,bus=pci.0,addr=0x6,drive=libvirt-3-format,id=virtio-disk3,write-cache=on \
-blockdev '{"driver":"file","filename":"/mnt/user/isos/Windows_1909.iso","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":true,"driver":"raw","file":"libvirt-2-storage"}' \
-device ide-cd,bus=ide.0,unit=0,drive=libvirt-2-format,id=ide0-0-0,bootindex=2 \
-blockdev '{"driver":"file","filename":"/mnt/user/isos/virtio-win-0.1.173-2.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}' \
-device ide-cd,bus=ide.0,unit=1,drive=libvirt-1-format,id=ide0-0-1 \
-netdev tap,fd=35,id=hostnet0 \
-device virtio-net,netdev=hostnet0,id=net0,mac=52:54:00:ac:47:5b,bus=pci.0,addr=0x3 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev socket,id=charchannel0,fd=36,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-vnc 0.0.0.0:1,websocket=5701 \
-k en-us \
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 \
-device vfio-pci,host=0000:0e:00.0,id=hostdev0,bus=pci.0,addr=0x8,romfile=/mnt/user/isos/evga1070ti.rom \
-device vfio-pci,host=0000:0e:00.1,id=hostdev1,bus=pci.0,addr=0x9 \
-device vfio-pci,host=0000:11:00.4,id=hostdev2,bus=pci.0,addr=0xa \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2021-07-16 17:40:09.939+0000: Domain id=3 is tainted: high-privileges
2021-07-16 17:40:09.939+0000: Domain id=3 is tainted: host-cpu
char device redirected to /dev/pts/1 (label charserial0)
2021-07-16T17:40:16.396552Z qemu-system-x86_64: vfio: Cannot reset device 0000:11:00.4, depends on group 37 which is not owned.
2021-07-16T17:40:17.431841Z qemu-system-x86_64: vfio: Cannot reset device 0000:11:00.4, depends on group 37 which is not owned.
So I add the [1022:1485] 11:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP device to my VM like this:
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x11' slot='0x00' function='0x4'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x11' slot='0x00' function='0x0'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
</hostdev>
and get an error after starting the VM:
internal error: Unknown PCI header type '127' for device '0000:11:00.4'
Any idea, what am I missing?