Jump to content

Yet another thread of cant pass through AMD video card to Windows11 VM


Recommended Posts

I have read many threads, watched the video-passthrough videos for settings, but cant figure out why I keep on getting GPU Passthrough Cannot Reset Device error on my Unraid Server. Below are my settings and my IOMMU group bindings. Please help!

Error message while stating a Windows 11 VM which has passthrough enabled

char device redirected to /dev/pts/0 (label charserial0)
2023-10-25T14:21:20.469704Z qemu-system-x86_64: vfio: Cannot reset device 0000:2b:00.1, no available reset mechanism.
2023-10-25T14:21:20.498751Z qemu-system-x86_64: vfio: Cannot reset device 0000:2b:00.1, no available reset mechanism.

 

My Windows 11 VM xml file which includes multifunction = on for sound/video portions

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>SONI-THEATRE</name>
  <uuid>ea43ec9f-130c-4c6d-829c-453c70a8a0d8</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 11" icon="windows11.png" os="windowstpm"/>
  </metadata>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='8'/>
    <vcpupin vcpu='2' cpuset='1'/>
    <vcpupin vcpu='3' cpuset='9'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-7.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/ea43ec9f-130c-4c6d-829c-453c70a8a0d8_VARS-pure-efi-tpm.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv mode='custom'>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='2' threads='2'/>
    <cache mode='passthrough'/>
    <feature policy='require' name='topoext'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/SONI-THEATRE/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.229-1.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='usb' index='0' model='nec-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:6b:95:1f'/>
      <source bridge='br0'/>
      <model type='virtio-net'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <tpm model='tpm-tis'>
      <backend type='emulator' version='2.0' persistent_state='yes'/>
    </tpm>
    <audio id='1' type='none'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x2b' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/isos/vbios/2023.amd.vbios.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x2b' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x2d' slot='0x00' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source startupPolicy='optional'>
        <vendor id='0x1462'/>
        <product id='0x7c56'/>
      </source>
      <address type='usb' bus='0' port='2'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>

 

and here is my IOMMU group binds

Warning: Your system has booted with the PCIe ACS Override setting enabled. The below list doesn't not reflect the way IOMMU would naturally group devices.
To see natural IOMMU groups for your hardware, go to the VM Manager page and set the PCIe ACS override setting to Disabled.

IOMMU group 0:[1022:1482] 00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 1:[1022:1483] 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge

IOMMU group 2:[1022:1483] 00:01.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge

IOMMU group 3:[1022:1482] 00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 4:[1022:1482] 00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 5:[1022:1483] 00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge

IOMMU group 6:[1022:1482] 00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 7:[1022:1482] 00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 8:[1022:1482] 00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 9:[1022:1484] 00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]

IOMMU group 10:[1022:1482] 00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge

IOMMU group 11:[1022:1484] 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]

IOMMU group 12:[1022:790b] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)

[1022:790e] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)

IOMMU group 13:[1022:1440] 00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 0

[1022:1441] 00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 1

[1022:1442] 00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 2

[1022:1443] 00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 3

[1022:1444] 00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 4

[1022:1445] 00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 5

[1022:1446] 00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 6

[1022:1447] 00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 7

IOMMU group 14:[15b7:5009] 01:00.0 Non-Volatile memory controller: Sandisk Corp WD Blue SN550 NVMe SSD (rev 01)

[N:0:1:1] disk WDC WDS100T2B0C__1 /dev/nvme0n1 1.00TB

IOMMU group 15:[1022:43ee] 02:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] 500 Series Chipset USB 3.1 XHCI Controller

Bus 001 Device 001 Port 1-0 ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 001 Device 002 Port 1-2 ID 05e3:0608 Genesys Logic, Inc. Hub

Bus 001 Device 007 Port 1-7 ID 1462:7c56 Micro Star International MYSTIC LIGHT

Bus 002 Device 001 Port 2-0 ID 1d6b:0003 Linux Foundation 3.0 root hub

Bus 002 Device 002 Port 2-3 ID 21c4:0cd1 Lexar USB Flash Drive

IOMMU group 16:[1022:43eb] 02:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] 500 Series Chipset SATA Controller

[2:0:0:0] disk ATA HGST HDS724040AL A580 /dev/sdb 4.00TB

[3:0:0:0] disk ATA WDC WD140EDFZ-11 0A81 /dev/sdc 14.0TB

[4:0:0:0] disk ATA WDC WD40EZRX-00S 0A80 /dev/sdd 4.00TB

[5:0:0:0] disk ATA WDC WD30EURS-63R 0A80 /dev/sde 3.00TB

[6:0:0:0] disk ATA WDC WD141KFGX-68 0A83 /dev/sdf 14.0TB

IOMMU group 17:[1022:43e9] 02:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] 500 Series Chipset Switch Upstream Port

IOMMU group 18:[1022:43ea] 03:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43ea

IOMMU group 19:[1022:43ea] 03:07.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43ea

IOMMU group 20:[1022:43ea] 03:08.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43ea

IOMMU group 21:[1022:43ea] 03:09.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43ea

IOMMU group 22:[15b7:5030] 04:00.0 Non-Volatile memory controller: Sandisk Corp Western Digital WD Black SN850X NVMe SSD (rev 01)

[N:1:8224:1] disk WD_BLACK SN850X 1000GB__1 /dev/nvme1n1 1.00TB

IOMMU group 23:[1b21:1064] 05:00.0 SATA controller: ASMedia Technology Inc. Device 1064 (rev 02)

[10:0:0:0] disk ATA WDC WD80EZAZ-11T 0A83 /dev/sdg 8.00TB

[20:0:0:0] disk ATA HGST HDN728080AL W91X /dev/sdh 8.00TB

[23:0:0:0] disk ATA WDC WD80EZAZ-11T 0A83 /dev/sdi 8.00TB

[25:0:0:0] disk ATA WDC WD80EZZX-11C 0A03 /dev/sdj 8.00TB

[26:0:0:0] disk ATA WDC WD80EMAZ-00W 0A83 /dev/sdk 8.00TB

IOMMU group 24:[8086:15f3] 29:00.0 Ethernet controller: Intel Corporation Ethernet Controller I225-V (rev 03)

IOMMU group 25:[10ec:8168] 2a:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15)

IOMMU group 26:[1002:67df] 2b:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] (rev e7)

IOMMU group 27:[1002:aaf0] 2b:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590]

IOMMU group 28:[1022:148a] 2c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function

IOMMU group 29:[1022:1485] 2d:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP

IOMMU group 30:[1022:1486] 2d:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP

IOMMU group 31:[1022:149c] 2d:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller

This controller is bound to vfio, connected USB devices are not visible.

IOMMU group 32:[1022:1487] 2d:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller

 

 

CPU Thread Pairings

Pair 1:cpu 0 / cpu 8

Pair 2:cpu 1 / cpu 9

Pair 3:cpu 2 / cpu 10

Pair 4:cpu 3 / cpu 11

Pair 5:cpu 4 / cpu 12

Pair 6:cpu 5 / cpu 13

Pair 7:cpu 6 / cpu 14

Pair 8:cpu 7 / cpu 15

 

USB Devices

Bus 001 Device 001 Port 1-0ID 1d6b:0002 Linux Foundation 2.0 root hub

Bus 001 Device 002 Port 1-2ID 05e3:0608 Genesys Logic, Inc. Hub

Bus 001 Device 007 Port 1-7ID 1462:7c56 Micro Star International MYSTIC LIGHT

Bus 002 Device 001 Port 2-0ID 1d6b:0003 Linux Foundation 3.0 root hub

Bus 002 Device 002 Port 2-3ID 21c4:0cd1 Lexar USB Flash Drive

 

SCSI Devices

[0:0:0:0]disk Lexar USB Flash Drive PMAP /dev/sda 31.0GB

[2:0:0:0]disk ATA HGST HDS724040AL A580 /dev/sdb 4.00TB

[3:0:0:0]disk ATA WDC WD140EDFZ-11 0A81 /dev/sdc 14.0TB

[4:0:0:0]disk ATA WDC WD40EZRX-00S 0A80 /dev/sdd 4.00TB

[5:0:0:0]disk ATA WDC WD30EURS-63R 0A80 /dev/sde 3.00TB

[6:0:0:0]disk ATA WDC WD141KFGX-68 0A83 /dev/sdf 14.0TB

[10:0:0:0]disk ATA WDC WD80EZAZ-11T 0A83 /dev/sdg 8.00TB

[20:0:0:0]disk ATA HGST HDN728080AL W91X /dev/sdh 8.00TB

[23:0:0:0]disk ATA WDC WD80EZAZ-11T 0A83 /dev/sdi 8.00TB

[25:0:0:0]disk ATA WDC WD80EZZX-11C 0A03 /dev/sdj 8.00TB

[26:0:0:0]disk ATA WDC WD80EMAZ-00W 0A83 /dev/sdk 8.00TB

[N:0:1:1]disk WDC WDS100T2B0C__1 /dev/nvme0n1 1.00TB

[N:1:8224:1] disk WD_BLACK SN850X 1000GB__1 /dev/nvm

 

with VFIO log

Processing 0000:2b:00.0 1002:67df
Vendor:Device 1002:67df found at 0000:2b:00.0

IOMMU group members (sans bridges):
/sys/bus/pci/devices/0000:2b:00.0/iommu_group/devices/0000:2b:00.0

Binding...

success...

Device 1002:67df at 0000:2b:00.0 bound to vfio-pci
---
Processing 0000:2b:00.1 1002:aaf0
Vendor:Device 1002:aaf0 found at 0000:2b:00.1

IOMMU group members (sans bridges):
/sys/bus/pci/devices/0000:2b:00.1/iommu_group/devices/0000:2b:00.1

Binding...

success...

Device 1002:aaf0 at 0000:2b:00.1 bound to vfio-pci
---
Processing 0000:2d:00.3 1022:149c
Vendor:Device 1022:149c found at 0000:2d:00.3

IOMMU group members (sans bridges):
/sys/bus/pci/devices/0000:2d:00.3/iommu_group/devices/0000:2d:00.3

Binding...
Unbound 0000:2d:00.3 from xhci_hcd

success...

Device 1022:149c at 0000:2d:00.3 bound to vfio-pci
---
Devices listed in /sys/bus/pci/drivers/vfio-pci:
lrwxrwxrwx 1 root root    0 Oct 25 13:53 0000:2b:00.0 -> ../../../../devices/pci0000:00/0000:00:03.1/0000:2b:00.0
lrwxrwxrwx 1 root root    0 Oct 25 13:53 0000:2b:00.1 -> ../../../../devices/pci0000:00/0000:00:03.1/0000:2b:00.1
lrwxrwxrwx 1 root root    0 Oct 25 13:53 0000:2d:00.3 -> ../../../../devices/pci0000:00/0000:00:08.1/0000:2d:00.3
vfio-pci binding complete

 

and lastly my syslinux configuration(flash drive)

Global ConfigurationUnraid OSUnraid OS GUI ModeUnraid OS Safe Mode (no plugins, no GUI)Unraid OS GUI Safe Mode (no plugins)Memtest86+

Server boot mode:

UEFI

Permit UEFI boot mode 

Boot system in UEFI mode. Please check your system settings to support UEFI boot mode.

unraidiommu.jpg

unraidsyslinux.jpg

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
×
  • Create New...