AMD RX 6800 XT reset bug happens every time

[workermaster]

Hi all, 

 

I got a system with a RX 6800 XT GPU. I heard that when a VM is forcefully shutdown, then it can show the AMD reset bug. 

For some reason I always get that bug. Just rebooting the VM causes the GPU to stop working. 

 

I run a Windows 10 VM. It also happens in the Windows 11 test VM. 

 

Other relevant hardware:

R7 3700X

32GB DDR4

Asus X470-f gaming motherboard.

 

I have used the fix from Spaceinvaderone mentioned here: https://www.youtube.com/watch?v=0uZODoPQH9c

But again, he mentions that this bug only happens on some occasions. I get it every single time. This makes a VM with this GPU not usable. 

 

I also found this thread but creating a new kernel did not help me: https://forums.unraid.net/topic/103897-amd-reset-bug/

It is possible though that I simply did something wrong. I followed the steps and only set the reset vendor to "true". 

 

Here is my XML:

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm' id='2'>
  <name>Windows 11 gaming VM</name>
  <uuid>9ddb5e1c-fc7b-5677-070b-8bbf44d6629e</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>20971520</memory>
  <currentMemory unit='KiB'>20971520</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='4'/>
    <vcpupin vcpu='1' cpuset='12'/>
    <vcpupin vcpu='2' cpuset='5'/>
    <vcpupin vcpu='3' cpuset='13'/>
    <vcpupin vcpu='4' cpuset='6'/>
    <vcpupin vcpu='5' cpuset='14'/>
    <vcpupin vcpu='6' cpuset='7'/>
    <vcpupin vcpu='7' cpuset='15'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-5.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/9ddb5e1c-fc7b-5677-070b-8bbf44d6629e_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='4' threads='2'/>
    <cache mode='passthrough'/>
    <feature policy='require' name='topoext'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/Plex/spaces_win_clover.img' index='3'/>
      <backingStore/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <alias name='virtio-disk2'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/Plex/win10.iso' index='2'/>
      <backingStore/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <alias name='sata0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.190-1.iso' index='1'/>
      <backingStore/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <alias name='sata0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x9'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0xa'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0xb'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xc'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xd'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:e9:d6:6d'/>
      <source bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio-net'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-2-Windows 11 gaming VM/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <rom file='/mnt/user/Plex/AMD.RX6800XT.16384.201029.rom'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0e' slot='0x00' function='0x4'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev2'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc07e'/>
        <address bus='7' device='2'/>
      </source>
      <alias name='hostdev3'/>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc336'/>
        <address bus='7' device='3'/>
      </source>
      <alias name='hostdev4'/>
      <address type='usb' bus='0' port='2'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+100</label>
    <imagelabel>+0:+100</imagelabel>
  </seclabel>
</domain>
 

[ghost82]

15 hours ago, workermaster said:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <rom file='/mnt/user/Plex/AMD.RX6800XT.16384.201029.rom'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>

You are passing through only the video part of the gpu, try to passthrough all the components in a multifunction device.

You should have 4 components as reported by another user:

 

Make sure to use your own vbios and do not download from somewhere.

 

Moreover, the reset bug doesn't happen with a force shutdown only, but it happens if you restart the vm too.

However newer gpu like yours, don't have this bug anymore.

Edited August 22, 2021 by ghost82

[workermaster]

16 minutes ago, ghost82 said:

You are passing through only the video part of the gpu, try to passthrough all the components in a multifunction device.

Make sure to use your own vbios and do not download from somewhere.

 

Moreover, the reset bug doesn't happen with a force shutdown only, but it happens if you restart the vm too.

However newer gpu like yours, don't have this bug anymore.

I have passed through the sound part of the card as well. Did that after I created this post. The reset bug is still there. 

I am using a Vbios that I downloaded from Techpowerup. I will try to dump the one from my own card. 

 

So far, I also have another problem. I am now using a Windows 11 VM (that runs waaaaay better than the previous W10) but gaming on it works somewhat. But the bad gaming performance is something for another thread. Basically the card is not being used 100% by the VM. 

[ghost82]

Please don't call reset bug, as it has being fixed for the 6800 xt, as said.

Have a read also at this:

And install also latest amd drivers in the guest.

 

[workermaster]

44 minutes ago, ghost82 said:

Please don't call reset bug, as it has being fixed for the 6800 xt, as said.

Have a read also at this:

And install also latest amd drivers in the guest.

 

I have dumped the vbios of the card with the help of the script from Spaceinvaderone. That did not change anything. 

The VM has the latest drivers already. 

 

I tried binding the GPU with sound and USB parts at boot but that did not work. 

 

I also read through the link you gave me. That seems to be about people who didn't get the passthrough to work. I did but I am having issues rebooting a VM. There was another person in that thread that also had reboot issues but that person mentioned that when he rebooted, Windows did actually reboot but just not show an image. My VM does not reboot. I see one cpu core at about 85% load wich does not change. The VM does not boot. 

 

Here is the new VM template that I am using:

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm' id='2'>
  <name>Windows 11 gaming VM</name>
  <uuid>9ddb5e1c-fc7b-5677-070b-8bbf44d6629e</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>20971520</memory>
  <currentMemory unit='KiB'>20971520</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>12</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='10'/>
    <vcpupin vcpu='2' cpuset='3'/>
    <vcpupin vcpu='3' cpuset='11'/>
    <vcpupin vcpu='4' cpuset='4'/>
    <vcpupin vcpu='5' cpuset='12'/>
    <vcpupin vcpu='6' cpuset='5'/>
    <vcpupin vcpu='7' cpuset='13'/>
    <vcpupin vcpu='8' cpuset='6'/>
    <vcpupin vcpu='9' cpuset='14'/>
    <vcpupin vcpu='10' cpuset='7'/>
    <vcpupin vcpu='11' cpuset='15'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-5.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/9ddb5e1c-fc7b-5677-070b-8bbf44d6629e_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='6' threads='2'/>
    <cache mode='passthrough'/>
    <feature policy='require' name='topoext'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/Plex/spaces_win_clover.img' index='3'/>
      <backingStore/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <alias name='virtio-disk2'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/disks/ST1000LM035-1RK172_W931P0KJ/Windows11_InsiderPreview_Client_x64_en-us_22000.iso' index='2'/>
      <backingStore/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <alias name='sata0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.190-1.iso' index='1'/>
      <backingStore/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <alias name='sata0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x9'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0xa'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0xb'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xc'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xd'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0xe'/>
      <alias name='pci.7'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:e9:d6:6d'/>
      <source bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio-net'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-2-Windows 11 gaming VM/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <rom file='/mnt/user/isos/vbios/6800xt.rom'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0b' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev2'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc07e'/>
        <address bus='7' device='2'/>
      </source>
      <alias name='hostdev3'/>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc336'/>
        <address bus='7' device='3'/>
      </source>
      <alias name='hostdev4'/>
      <address type='usb' bus='0' port='2'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x2516'/>
        <product id='0x014d'/>
        <address bus='5' device='2'/>
      </source>
      <alias name='hostdev5'/>
      <address type='usb' bus='0' port='3'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+100</label>
    <imagelabel>+0:+100</imagelabel>
  </seclabel>
</domain>
 

 

As you can see, I added the sound counterparts and some extra CPU cores. But other than that nothing has changed. 

[workermaster]

To add more info, here is the log from the W11 VM:

ErrorWarningSystemArrayLogin

-boot strict=on \
-device pcie-root-port,port=0x8,chassis=1,id=pci.1,bus=pcie.0,multifunction=on,addr=0x1 \
-device pcie-root-port,port=0x9,chassis=2,id=pci.2,bus=pcie.0,addr=0x1.0x1 \
-device pcie-root-port,port=0xa,chassis=3,id=pci.3,bus=pcie.0,addr=0x1.0x2 \
-device pcie-root-port,port=0xb,chassis=4,id=pci.4,bus=pcie.0,addr=0x1.0x3 \
-device pcie-root-port,port=0xc,chassis=5,id=pci.5,bus=pcie.0,addr=0x1.0x4 \
-device pcie-root-port,port=0xd,chassis=6,id=pci.6,bus=pcie.0,addr=0x1.0x5 \
-device pcie-root-port,port=0xe,chassis=7,id=pci.7,bus=pcie.0,addr=0x1.0x6 \
-device qemu-xhci,p2=15,p3=15,id=usb,bus=pcie.0,addr=0x7 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x0 \
-blockdev '{"driver":"file","filename":"/mnt/user/Plex/spaces_win_clover.img","node-name":"libvirt-3-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-3-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-3-storage"}' \
-device virtio-blk-pci,bus=pci.3,addr=0x0,drive=libvirt-3-format,id=virtio-disk2,bootindex=1,write-cache=on \
-blockdev '{"driver":"file","filename":"/mnt/disks/ST1000LM035-1RK172_W931P0KJ/Windows11_InsiderPreview_Client_x64_en-us_22000.iso","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":true,"driver":"raw","file":"libvirt-2-storage"}' \
-device ide-cd,bus=ide.0,drive=libvirt-2-format,id=sata0-0-0,bootindex=2 \
-blockdev '{"driver":"file","filename":"/mnt/user/isos/virtio-win-0.1.190-1.iso","node-name":"libvirt-1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":true,"driver":"raw","file":"libvirt-1-storage"}' \
-device ide-cd,bus=ide.1,drive=libvirt-1-format,id=sata0-0-1 \
-netdev tap,fd=33,id=hostnet0 \
-device virtio-net,netdev=hostnet0,id=net0,mac=52:54:00:e9:d6:6d,bus=pci.1,addr=0x0 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev socket,id=charchannel0,fd=34,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \
-device vfio-pci,host=0000:0b:00.0,id=hostdev0,bus=pci.4,addr=0x0,romfile=/mnt/user/isos/vbios/6800xt.rom \
-device vfio-pci,host=0000:0b:00.1,id=hostdev1,bus=pci.5,addr=0x0 \
-device vfio-pci,host=0000:01:00.0,id=hostdev2,bus=pci.6,addr=0x0 \
-device usb-host,hostbus=7,hostaddr=2,id=hostdev3,bus=usb.0,port=1 \
-device usb-host,hostbus=7,hostaddr=3,id=hostdev4,bus=usb.0,port=2 \
-device usb-host,hostbus=5,hostaddr=2,id=hostdev5,bus=usb.0,port=3 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2021-08-22 14:38:23.850+0000: Domain id=2 is tainted: high-privileges
2021-08-22 14:38:23.850+0000: Domain id=2 is tainted: host-cpu
char device redirected to /dev/pts/0 (label charserial0)
2021-08-22T14:38:26.806084Z qemu-system-x86_64: vfio: Cannot reset device 0000:0b:00.1, depends on group 21 which is not owned.
2021-08-22T14:38:26.810067Z qemu-system-x86_64: vfio: Cannot reset device 0000:0b:00.0, depends on group 21 which is not owned.
2021-08-22T14:38:26.927508Z qemu-system-x86_64: vfio: Cannot reset device 0000:0b:00.1, depends on group 21 which is not owned.
2021-08-22T14:38:26.931497Z qemu-system-x86_64: vfio: Cannot reset device 0000:0b:00.0, depends on group 21 which is not owned.

 

This log shows a normal VM boot after I rebooted Unraid, followed by a normal reboot of just the VM, then a forcefull shutdown since the VM stopped working, Then I applied the AMD reset fix from spaceinvader and then restarted the VM. 

 

I am seeing that the log shows that it needs group 21. That is the USB part of the GPU. But I have no idea how to give that to the VM. If I select it in the VM template, then I get an error saying that it can't find the controller. And binding at boot also doesn't work since it doesn't even show up in the VM template at boot. 

 

I have no idea what this is. Do I just happen to have the one single 6800 XT in existence that has the AMD reset bug while litterally all others on the world do not?

[ghost82]

Attach diagnostic file, as it's impossible to help without knowing the hardware. You need to passthrough all the components of the gpu (already told) and also device 0b:00.0 cannot be resetted.

[workermaster]

10 minutes ago, ghost82 said:

Attach diagnostic file, as it's impossible to help without knowing the hardware. You need to passthrough all the components of the gpu (already told) and also device 0b:00.0 cannot be resetted.

Here are the diagnostics. 

 

I have tried to pass through the whole card. The VM currently has the video and sound part of it. But I can't get the USB part to work. When I select that in the VM template then it gives me an error telling me that the usb device cannot be found. I can try to post screenshots of that. 

tower-diagnostics-20210822-1840.zip

[ich777]

@workermaster have you already seen that for unRAID 6.10.0-rc1 is a plugin in the CA App for this issue?

AMD Vendor Reset, if you want to try it just install rc1 download the plugin from the CA App and reboot.

You are only able to see the plugin in the CA App if you are on 6.10.0-rc1

[ghost82]

44 minutes ago, workermaster said:

Here are the diagnostics.

 

Attach to vfio iommu groups: 15 (nvme), 19, 20, 21, 22 (gpu) and 28 (audio).

Reboot unraid.

Replace the whole xml of your win 11 vm with the attached and save.

 

AMD.RX6800XT.16384.201029.rom: as said replace this with your vbios.

Windows 11 gaming VM.xml

Edited August 22, 2021 by ghost82

[workermaster]

30 minutes ago, ich777 said:

@workermaster have you already seen that for unRAID 6.10.0-rc1 is a plugin in the CA App for this issue?

AMD Vendor Reset, if you want to try it just install rc1 download the plugin from the CA App and reboot.

You are only able to see the plugin in the CA App if you are on 6.10.0-rc1

Hi, 

 

I have not seen that yet. I have just tried upgrading to version 6.10.0-RC1 but my VM refuses to start there. It shows one core at about 85% load. The exact same problem that I have when I reboot the VM. 

 

I also noticed that when I stop the VM, the screen shows the black and white Unraid terminal text that you see when it boots. It does not do that in 6.9.2 after I shutdown a VM. I am guessing that the GPU has trouble switching from the Unraid text to being used in a VM. 

Since it gave me some trouble and I am expecting food to arrive in a few minutes, I have reverted back to 6.9.2 for the moment and will try your suggestion later today or tomorrow. 

[workermaster]

11 minutes ago, ghost82 said:

 

Attach to vfio iommu groups: 15 (nvme), 19, 20, 21, 22 (gpu) and 28 (audio).

Reboot unraid.

Replace the whole xml of your win 11 vm with the attached and save.

 

AMD.RX6800XT.16384.201029.rom: as said replace this with your vbios.

Windows 11 gaming VM.xml 8.06 kB · 0 downloads

Thanks for your reply. 

 

I will try to do your fix later today or tomorrow.

 

How do I attach vfio iommu groups to a VM? 

Currently I have attached hardware by selecting it in a dropdown menu when creating the VM template (not the xml view). The only hardware that I have selected to be isolated at boot is the NVME controller. I have done that by selecting it in the Tools -> system devices -> click checkbox of iommu group 15 and then click on "Bind selected to VFIO at boot". Is that what I need to do for the other groups as well?

[ghost82]

4 minutes ago, workermaster said:

I have done that by selecting it in the Tools -> system devices -> click checkbox

this is attaching to vfio: put a checkbox for the iommu groups I listed.

[ghost82]

Note that when attaching to vfio the gpu it will not be available for unraid.

[ich777]

29 minutes ago, workermaster said:

I have not seen that yet. I have just tried upgrading to version 6.10.0-RC1 but my VM refuses to start there. It shows one core at about 85% load. The exact same problem that I have when I reboot the VM.

Try to install 6.10.0-rc1 and then install the AMD Vendor Reset plugin, from what I know it is working just fine on @glennv's machine. Both, 6.10.0-rc1 and the AMD-Vendor-Reset plugin/patch.

[workermaster]

27 minutes ago, ich777 said:

Try to install 6.10.0-rc1 and then install the AMD Vendor Reset plugin, from what I know it is working just fine on @glennv's machine. Both, 6.10.0-rc1 and the AMD-Vendor-Reset plugin/patch.

I have 6.10 running and downloaded the plugin. How do I use it? The screen stays black and the VM won't boot. 

[ich777]

3 minutes ago, workermaster said:

I have 6.10 running and downloaded the plugin. How do I use it? The screen stays black and the VM won't boot. 

Download it from the CA App and reboot, this will apply the Vendor Reset patch to your unRAID server, nothing else to do.

 

If the VM won't boot on 6.10.0-rc1 it would be better to create a dedicated thread for that in the VM subforums with the Diagnostics attached (pull the Diagnostics after you tried to start the VM once), eventually @glennv can help here too.

 

What does the log from the VM and also the syslog say? Any errors?

[workermaster]

11 minutes ago, ich777 said:

Download it from the CA App and reboot, this will apply the Vendor Reset patch to your unRAID server, nothing else to do.

 

If the VM won't boot on 6.10.0-rc1 it would be better to create a dedicated thread for that in the VM subforums with the Diagnostics attached (pull the Diagnostics after you tried to start the VM once), eventually @glennv can help here too.

 

What does the log from the VM and also the syslog say? Any errors?

I think it was just the template that was messed up. I created a new VM template and gave it another name and pointed the boot drive to my NVME. I am now back in my VM on Unraid 6.10. That issue seems to be solved now. 

 

I am now in the process of binding the GPU to the VM and see if that helps. @ghost82 I will let you know what happens when I try your template. Hopefully it works with Unraid 6.10. 

[ich777]

2 minutes ago, workermaster said:

I am now in the process of binding the GPU to the VM and see if that helps. @ghost82 I will let you know what happens when I try your template. Hopefully it works with Unraid 6.10. 

With the AMD Vendor Reset patch applied nothing special has to be done in the template to "fix" the AMD Reset Bug.

 

Btw: If you want to mention someone you actually have to click his name after you wrote his name: @ghost82:

[glennv]

Yeah as @ich777 mentioned, i have a pretty stubborn server that untill his patch never wanted to work properly with any previous amd reset bug patch or other tricks. Since using his patch/plugin i have zero problems and can restart , kill , do whatever i want with the VM without any issues with the AMD card. So extremely highly recommended and i am super thankfull for his amazing job.
Nothing special to do other then just install the plugin.

[workermaster]

8 minutes ago, ich777 said:

With the AMD Vendor Reset patch applied nothing special has to be done in the template to "fix" the AMD Reset Bug.

 

Btw: If you want to mention someone you actually have to click his name after you wrote his name: @ghost82:

Thanks, I did not know that I messed that up. I was typing at the bottom of my screen and did not see that a list was shown with names. 

 

Anyway it all seems to work now. I have Unraid 6.10 RC2 running and after recreating the template and binding all parts (and motherboard audio) to my VM with the steps that @ghost82 described, it works. I have not had to use the template given to me, but I will store that one for later testing. I can now reboot the VM when I want to without rebooting the whole system. 

 

@ich777 I have a question that is not related to this topic but it is something that I have been wondering for a while. You are a community developer and I have seen a lot of work you have done. That also goes for a lot of other things other people created. What I don't understand is why Unraid by itself is so incomplete. To get everything working you need to download a whole list of community applications. Things like "Fix common problems" and "CA auto update applications" and "Unassigned devices". Why doesn't Unraid come with most of these functionalities build in? To me it seems like Unraid builds 80% of the product and the community gets to finish the last 20%. Do you happen to know why that is?

[ghost82]

@ich777
@glennv

Are you speaking about a 6800 xt? I don't know of any case it suffers the reset bug.

Edited August 22, 2021 by ghost82

[ghost82]

2 minutes ago, workermaster said:

I have not had to use the template given to me

The modified template only ensures that the components of the gpu are passed through in a multifuncion device, with a correct topology.

If the device is not multifunction in the target it may not work at all or undergo unexpected behaviour.

[glennv]

37 minutes ago, ghost82 said:

@ich777
@glennv

Are you speaking about a 6800 xt? I don't know of any case it suffers the reset bug.

For me no. I am using a 5700XT. To be honest i have also not heared yet of reset bug in the 6000 series, but symptomps where similar for OP so never hurts to install and try it. 

So either the plugin or something else he did solved his issues aparently

Edited August 22, 2021 by glennv

[workermaster]

4 minutes ago, glennv said:

For me no. I am using a 5700XT. To be honest i have also not heared yet of reset bug in the 6000 series, but symptomps where similar for OP so never hurts to install and try it. 

So either the plugin or something else he did solved his issues aparently

I am willing to try and figure out what did it tomorrow. I am curious to know what exactly the problem was. Personally I think that I just messed up the way that I bound the GPU parts to VFIO and I just had to create a new VM template. But I will test that tomorrow. 

 

I am also happy to report that a Windows 11 virtual machine is feeling just as snappy as my old Windows 10 bare metal install. Previously the Windows 10 virtual machines were laggy for me. Unraid finally starts to look like a permament host option for me.