AMD Radeon RX 6800XT passthrough hints


zeus83

Recommended Posts

I've decided to put it here because just spent several hours trying to figure out AMD Radeon 6800XT passthrough to my Win 10 VM.

Previously I've only worked with Nvidia gpus and experienced zero issues with them. Radeon a bit different story.

 

I'm running unraid on Ryzen 5950X Asus Crosshair VIII Hero. The GPU is Gigabyte Radeon 6800XT (ref). So not all tips may be relevant to your hardware/software.

 

So the hints are:

  1. BIOS: disable CSM compatibility mode and leave only UEFI. I couldn't get GPU passed through in Legacy mode.
  2. Only use Q35 virtual machine. i440fx didn't worked for me. Couldn't install a driver.
  3. Don't install amdgpu host drivers. Couldn't pass through the GPU to VM in that case.
  4. Passthrough didn't work on unRAID 6.8 for me (got error 43 always). So I had to switch to unRAID 6.9rc2 on 5.10 kernel.
  5. For 6800XT passthrough four devices:187035181_.thumb.PNG.f73ab2e588f91dc4ddf074364e75f030.PNG
  6. Enabling the AMD Smart Access Memory (SAM) didn't work for me. Got error 43 in windows vm immediately (seems like it's not yet implemented and exposed in vfio https://github.com/qemu/qemu/commit/3412d8ec9810b819f8b79e8e0c6b87217c876e32).
  7. Encoding delay in Parsec is awful > 20 ms (h.265, 1440p). Something wrong with driver/encoder I think. Didn't figure out what's wrong yet, but saw someone complains on reddit about it as well.
Edited by zeus83
Link to comment
  • zeus83 changed the title to AMD Radeon RX 6800XT passthrough hints
On 1/30/2021 at 4:04 PM, zeus83 said:

I've decided to put it here because just spent several hours trying to figure out AMD Radeon 6800XT passthrough to my Win 10 VM.

Previously I've only worked with Nvidia gpus and experienced zero issues with them. Radeon a bit different story.

 

I'm running unraid on Ryzen 5950X Asus Crosshair VIII Hero. The GPU is Gigabyte Radeon 6800XT (ref). So not all tips may be relevant to your hardware/software.

 

So the hints are:

  1. BIOS: disable CSM compatibility mode and leave only UEFI. I couldn't get GPU passed through in Legacy mode.
  2. Only use Q35 virtual machine. i440fx didn't worked for me. Couldn't install a driver.
  3. Don't install amdgpu host drivers. Couldn't pass through the GPU to VM in that case.
  4. Passthrough didn't work on unRAID 6.8 for me (got error 43 always). So I had to switch to unRAID 6.9rc2 on 5.10 kernel.
  5. For 6800XT passthrough four devices:187035181_.thumb.PNG.f73ab2e588f91dc4ddf074364e75f030.PNG
  6. Enabling the AMD Smart Access Memory (SAM) didn't work for me. Got error 43 in windows vm immediately.
  7. Encoding delay in Parsec is awful > 20 ms (h.265, 1440p). Something wrong with driver/encoder I think. Didn't figure out what's wrong yet, but saw someone complains on reddit about it as well.

 

Thanks for this. I happen to be doing the same thing right now, so your post has great timing. I'll try the things you suggested and see if I get anywhere.

 

Should have known it wouldn't be easy... but the card came highly recommended by Level 1 Techs, so I was hopeful.

Link to comment

Well, overall it works but...

 

  1. SAM doesn't work in virtual machine. On the bare metal enabling SAM yields up to 10% improvement depending on a game.
  2. Remote play is hardly feasible. I used to use Moonlight / Parsec. Moonlight is not an option for Radeon cards so Parsec remains, but encoding latency is awful. In a game that maintains 100fps the Parsecs barely encodes 30fps (1440p). Switching to 1080p helps a little. But no stable 60 fps still. If anyone know if it's the case with 5700 XT as well? I tried my GTX 1650 super and it performs way better with Parsec... I don't understand how this is possible.
Link to comment
On 1/30/2021 at 4:04 PM, zeus83 said:

I've decided to put it here because just spent several hours trying to figure out AMD Radeon 6800XT passthrough to my Win 10 VM.

Previously I've only worked with Nvidia gpus and experienced zero issues with them. Radeon a bit different story.

 

I'm running unraid on Ryzen 5950X Asus Crosshair VIII Hero. The GPU is Gigabyte Radeon 6800XT (ref). So not all tips may be relevant to your hardware/software.

 

So the hints are:

  1. BIOS: disable CSM compatibility mode and leave only UEFI. I couldn't get GPU passed through in Legacy mode.
  2. Only use Q35 virtual machine. i440fx didn't worked for me. Couldn't install a driver.
  3. Don't install amdgpu host drivers. Couldn't pass through the GPU to VM in that case.
  4. Passthrough didn't work on unRAID 6.8 for me (got error 43 always). So I had to switch to unRAID 6.9rc2 on 5.10 kernel.
  5. For 6800XT passthrough four devices:187035181_.thumb.PNG.f73ab2e588f91dc4ddf074364e75f030.PNG
  6. Enabling the AMD Smart Access Memory (SAM) didn't work for me. Got error 43 in windows vm immediately (seems like it's not yet implemented and exposed in vfio https://github.com/qemu/qemu/commit/3412d8ec9810b819f8b79e8e0c6b87217c876e32).
  7. Encoding delay in Parsec is awful > 20 ms (h.265, 1440p). Something wrong with driver/encoder I think. Didn't figure out what's wrong yet, but saw someone complains on reddit about it as well.

 

Which version of Q35 let you successfully install the Radeon software? I'm getting incompatibility messages from the installer on Q35 5.1.

Link to comment
5 hours ago, cyberspectre said:

 

Which version of Q35 let you successfully install the Radeon software? I'm getting incompatibility messages from the installer on Q35 5.1.

 

Nevermind, got it working on Q35 5.0. Very pleased so far.

 

For posterity, let me say this. It required all of the things the OP mentioned + an extra kernel argument to make my GTX-970 work again after switching to UEFI mode. That kernel argument is...

 

video=efifb:off

 

Edited by cyberspectre
Link to comment
  • 1 month later...

I apologise for posting here. I have Unraid 6.9 (stable) and trying to get my damn 6900XT passed through to my Win10 VM.

 

I already had CSM disabled, have `video=efifb:off` kernel parameter, using the Q35-5.0 chipset and hiding the KVM ids from the guest (https://mathiashueber.com/fighting-error-43-nvidia-gpu-virtual-machine/ and https://old.reddit.com/r/VFIO/comments/jicm4c/radeon_adrenaline_software_installation_kills_the/).

 

I am pretty sure I am passing all the devices as seen here:

155919811_ScreenShot2021-03-08at9_20_29AM.thumb.png.89d9ca1dd6ab6bc566e1e7e46cff3b18.png

 

Within the VM editor:

1118943852_ScreenShot2021-03-08at9_25_19AM.thumb.png.48548619a39a94e40171a6b21684b970.png

 

 

Here is my VM XML:

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>Windows 10 - Q35</name>
  <uuid>51a4c2cb-65e5-fb05-ada9-ee55c27179ee</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>33554432</memory>
  <currentMemory unit='KiB'>33554432</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>24</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='4'/>
    <vcpupin vcpu='1' cpuset='20'/>
    <vcpupin vcpu='2' cpuset='5'/>
    <vcpupin vcpu='3' cpuset='21'/>
    <vcpupin vcpu='4' cpuset='6'/>
    <vcpupin vcpu='5' cpuset='22'/>
    <vcpupin vcpu='6' cpuset='7'/>
    <vcpupin vcpu='7' cpuset='23'/>
    <vcpupin vcpu='8' cpuset='8'/>
    <vcpupin vcpu='9' cpuset='24'/>
    <vcpupin vcpu='10' cpuset='9'/>
    <vcpupin vcpu='11' cpuset='25'/>
    <vcpupin vcpu='12' cpuset='10'/>
    <vcpupin vcpu='13' cpuset='26'/>
    <vcpupin vcpu='14' cpuset='11'/>
    <vcpupin vcpu='15' cpuset='27'/>
    <vcpupin vcpu='16' cpuset='12'/>
    <vcpupin vcpu='17' cpuset='28'/>
    <vcpupin vcpu='18' cpuset='13'/>
    <vcpupin vcpu='19' cpuset='29'/>
    <vcpupin vcpu='20' cpuset='14'/>
    <vcpupin vcpu='21' cpuset='30'/>
    <vcpupin vcpu='22' cpuset='15'/>
    <vcpupin vcpu='23' cpuset='31'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-5.0'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/51a4c2cb-65e5-fb05-ada9-ee55c27179ee_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='123456789ab'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <vmport state='off'/>
    <ioapic driver='kvm'/>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='12' threads='2'/>
    <cache mode='passthrough'/>
    <feature policy='require' name='topoext'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Windows 10/vdisk1.img'/>
      <target dev='hdc' bus='sata'/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0xa'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0xb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xc'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:db:7e:25'/>
      <source bridge='br0'/>
      <model type='virtio-net'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0d' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0d' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc539'/>
      </source>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>
<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>Windows 10 - Q35</name>
  <uuid>51a4c2cb-65e5-fb05-ada9-ee55c27179ee</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>33554432</memory>
  <currentMemory unit='KiB'>33554432</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>24</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='4'/>
    <vcpupin vcpu='1' cpuset='20'/>
    <vcpupin vcpu='2' cpuset='5'/>
    <vcpupin vcpu='3' cpuset='21'/>
    <vcpupin vcpu='4' cpuset='6'/>
    <vcpupin vcpu='5' cpuset='22'/>
    <vcpupin vcpu='6' cpuset='7'/>
    <vcpupin vcpu='7' cpuset='23'/>
    <vcpupin vcpu='8' cpuset='8'/>
    <vcpupin vcpu='9' cpuset='24'/>
    <vcpupin vcpu='10' cpuset='9'/>
    <vcpupin vcpu='11' cpuset='25'/>
    <vcpupin vcpu='12' cpuset='10'/>
    <vcpupin vcpu='13' cpuset='26'/>
    <vcpupin vcpu='14' cpuset='11'/>
    <vcpupin vcpu='15' cpuset='27'/>
    <vcpupin vcpu='16' cpuset='12'/>
    <vcpupin vcpu='17' cpuset='28'/>
    <vcpupin vcpu='18' cpuset='13'/>
    <vcpupin vcpu='19' cpuset='29'/>
    <vcpupin vcpu='20' cpuset='14'/>
    <vcpupin vcpu='21' cpuset='30'/>
    <vcpupin vcpu='22' cpuset='15'/>
    <vcpupin vcpu='23' cpuset='31'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-5.0'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/51a4c2cb-65e5-fb05-ada9-ee55c27179ee_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='12' threads='2'/>
    <cache mode='passthrough'/>
    <feature policy='require' name='topoext'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Windows 10/vdisk1.img'/>
      <target dev='hdc' bus='sata'/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0xa'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0xb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xc'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:db:7e:25'/>
      <source bridge='br0'/>
      <model type='virtio-net'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0d' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0d' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc539'/>
      </source>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x04d9'/>
        <product id='0x0203'/>
      </source>
      <address type='usb' bus='0' port='2'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>

 

To clarify my issue is that the device is passed through successfully, however when the AMD drivers load my Win10 VM's display turns black. I have no bloody clue what else I can do at the moment. I would really appreciate any help with this as I would ideally like to use Windows for some light gaming and use the 6900XT within Ubuntu (which is working fine with a kernel upgrade) for programing.

Edited by kprinssu
Added more detail to my post
Link to comment
  • 4 weeks later...
On 1/31/2021 at 12:04 AM, zeus83 said:

I've decided to put it here because just spent several hours trying to figure out AMD Radeon 6800XT passthrough to my Win 10 VM.

Previously I've only worked with Nvidia gpus and experienced zero issues with them. Radeon a bit different story.

 

I'm running unraid on Ryzen 5950X Asus Crosshair VIII Hero. The GPU is Gigabyte Radeon 6800XT (ref). So not all tips may be relevant to your hardware/software.

 

So the hints are:

  1. BIOS: disable CSM compatibility mode and leave only UEFI. I couldn't get GPU passed through in Legacy mode.
  2. Only use Q35 virtual machine. i440fx didn't worked for me. Couldn't install a driver.
  3. Don't install amdgpu host drivers. Couldn't pass through the GPU to VM in that case.
  4. Passthrough didn't work on unRAID 6.8 for me (got error 43 always). So I had to switch to unRAID 6.9rc2 on 5.10 kernel.
  5. For 6800XT passthrough four devices:

I got a reference RX 6700 XT today and replaced a R9 390X.. I guess I'm using UEFI (have to check later), Q35, no GPU Host Drivers, passed only 2 devices (GPU and GPU Audio), but have to supply BIOS file then passthrough works..

Link to comment

Running almost the same specs as OP with differences being Dark Hero vs Hero and Asus Tuf OC 6800xt. Oddly, using UEFI I get locked to 640x480 and grayscale. Enabling CSM gets me back to full 2k resolution. VM is Q35 5.0 and boots/runs just fine so long as I don't install drivers. However, without Radeon drivers I'm unable to use more than one monitor. Installing them will give me 3 monitors and works great until I reboot. Then I drop back down to 1 monitor and display looks like the brightness is turned up (really light in color).

I've messed with all the tips from OP as well as passed the dumped vBIOS to the VM. No bueno.

 

On a side note... Who wants to trade a 6800xt for a 3080 :D.

Link to comment
50 minutes ago, fritzgeralds said:

Running almost the same specs as OP with differences being Dark Hero vs Hero and Asus Tuf OC 6800xt. Oddly, using UEFI I get locked to 640x480 and grayscale. Enabling CSM gets me back to full 2k resolution. VM is Q35 5.0 and boots/runs just fine so long as I don't install drivers. However, without Radeon drivers I'm unable to use more than one monitor. Installing them will give me 3 monitors and works great until I reboot. Then I drop back down to 1 monitor and display looks like the brightness is turned up (really light in color).

I've messed with all the tips from OP as well as passed the dumped vBIOS to the VM. No bueno.

 

On a side note... Who wants to trade a 6800xt for a 3080 :D.

I'll trade you a 5500xt for that 6800xt if you're keen.

 

On a serious note, can you please post your vBios here? Did you dump it via a Windows VM with the GPU as secondary, or some other way?

Link to comment
19 minutes ago, lnxd said:

I'll trade you a 5500xt for that 6800xt if you're keen.

 

On a serious note, can you please post your vBios here? Did you dump it via a Windows VM with the GPU as secondary, or some other way?

I have that system shut down for the night as it's my secondary Unraid that I'm building. I'll boot it up after work tomorrow and post the vBIOS. As far as dumping it goes, I used SpaceinvaderOnes script found here via User Scripts. https://github.com/SpaceinvaderOne/Dump_GPU_vBIOS

Link to comment
8 minutes ago, fritzgeralds said:

I have that system shut down for the night as it's my secondary Unraid that I'm building. I'll boot it up after work tomorrow and post the vBIOS. As far as dumping it goes, I used SpaceinvaderOnes script found here via User Scripts. https://github.com/SpaceinvaderOne/Dump_GPU_vBIOS

No worries at all @fritzgeralds

 

His script produced a valid vBios for one of my GPUs but not the other so I'm very interested to see how yours looks. Your symptoms could possibly be explained by an incomplete / invalid vBios. That said, I just noticed Christoph tagged giganode in an earlier post so looks like some of the real pros might be taking a look for you fairly soon as well 😅

Link to comment

@lnxd Here's the vBIOS that I'm using, not sure how to check it for errors or if it's complete. Oddly, I get Error 43's and brightened display when physically at the system however, I just booted up the VM and remoted in and there's no Error 43 under the Radeon driver and display looks fine. Sadly I didn't leave more than one monitor connected before heading to work.

TUF-RX6800XT-O16G-GAMING_vbios.rom

Link to comment

Hey, so ive been having an issue getting the 6800xt to work after a VM reboot

 

I can get my VM to boot up fine with the 6800 xt after initial unraid boot,  However if i reboot or shutdown the VM i cannot get output from the GPU when it starts up again, when i VNC into the VM i see the dreaded code 43 issue.  

If i reboot unraid i can start up the VM no issues at all (until i reboot it).  

 

Ive tried all sorts of configs to no avail. Updated to latest bios, Unraid 6.9.1, latest GPU drivers etc 

Asus strix x570-e

5800x 

Sapphire 6800xt

Nvidia p400  (Docker Passthrough)  

added video=efifb:off 

Stubbed the Card and Audio portion (no usb) 

image.thumb.png.a8641982742e7866cc18ce0a27dd2546.png

VM Q35 5.0 (tried various types)

Rom dumped and added to template (via spaceinvaders plugin)

Updated the Multifunction and addresses in the XML 

 

Attached is the log of the VM

 

Really dont understand why it works perfectly well on first startup but breaks after a reboot or restart.  Ive tried stopping and starting the array but no cigars until unraid is rebooted!  I dont see any other users having this issue so not sure if its something im doing wrong or maybe the hardware/qemu support isnt quite there yet? 

 

logs2.txt

Link to comment
18 hours ago, fritzgeralds said:

@lnxd Here's the vBIOS that I'm using, not sure how to check it for errors or if it's complete. Oddly, I get Error 43's and brightened display when physically at the system however, I just booted up the VM and remoted in and there's no Error 43 under the Radeon driver and display looks fine. Sadly I didn't leave more than one monitor connected before heading to work.

Normally an extracted vBIOS is the size of the GPU's flash ROM, if you use a tool like ATIFlash to extract it. Spaceinvader One's script is really, really awesome in that instead of making an image of the whole ROM, it just reads the data accessible to the VM host directly into a ROM file. It's much more efficient but it has some caveats, one of which is that if there's something that's not accessible to the VM host, ie. not in the expected place in the card's ROM, it won't be extracted. 

 

Your vBIOS file metadata looks valid:

    Product Name is :    73BFHB.20.1.0.42.AS02P
    Device ID is    :    73BF
    Bios Version    :    020.001.000.042.000000
    Bios P/N is     :    115-D412BS0-101
    Bios SSID       :    04F2
    Bios SVID       :    1043
    Bios Date is    :    11/04/20 02:27

 

But I start work in 5 minutes so I'll have to take a proper look later.

 

EDIT: @fritzgeralds This vBIOS file isn't complete. I don't know for sure it's the cause of your problems but it's missing several values that would be in a valid vBIOS file. I'd say it's possibly your problem. If you have a Windows VM the easiest way to get a valid vBIOS is to:
 

  1. Make sure the latest AMD drivers are installed (you can do this step with the GPU as primary).
  2. Set up your GPU in the secondary slot (don't miss the little green + button on the left), make sure the sound card is also passed through.
  3. Use either ATIFlash or GPU-Z to save the ROM to a file. 


Screenshot for step 2:
1864212541_ScreenShot2021-04-08at7_58_32pm.thumb.png.c54dbf2ecc02a812ac50de0a250479d4.png
 

While it's not what you're asking about, please note that I still wouldn't flash this vBIOS to the card. Although it'd most likely be fine, VFIO might cause problems of its own for the generated ROM. Although it's perfectly fine to flash the card over VFIO. If you needed a flash-able copy some time down the line, the safest way to get a safe-to-flash version is probably from running Windows (or linux, but I don't think there's a GUI version) on metal with the GPU as secondary. 

 

14 hours ago, trig229 said:

Really dont understand why it works perfectly well on first startup but breaks after a reboot or restart.  Ive tried stopping and starting the array but no cigars until unraid is rebooted!  I dont see any other users having this issue so not sure if its something im doing wrong or maybe the hardware/qemu support isnt quite there yet?


This is the sort of behaviour you'd see with the vendor-reset issue, but your card shouldn't be impacted. Can you please post the output of:
 

dmesg | grep amdgpu

 

Run on the host. Preferably within a minute of starting the VM, and then again within a minute of stopping the VM. 

Edited by lnxd
Link to comment
11 hours ago, trig229 said:

@lnxd Thanks for your reply, tried this earlier but getting no results from amdgpu expression before or after the vm shutdown/start up.  To confirm, im running this in the unraid terminal?  

Yup that's right. Can you please post your diagnostics.zip? 
 

4 hours ago, giganode said:

 

Big Navi cards should work ootb, as they fully support flr as specified in pcie specifications. Strange... But as I read here, there seems to be an issue with the vbios.

 

@trig229 you can also try to download your vbios from techpowerup and use that instead.

@giganode I probably confused you by quoting them both in one post, did you mean to tag fritzgeralds or do you think trig229 is having a vBIOS issue as well?

Edited by lnxd
Link to comment
23 hours ago, lnxd said:

EDIT: @fritzgeralds This vBIOS file isn't complete. I don't know for sure it's the cause of your problems but it's missing several values that would be in a valid vBIOS file. I'd say it's possibly your problem. If you have a Windows VM the easiest way to get a valid vBIOS is to...

Thanks. I grabbed the rom from techpowerup and figured something was up with mine when it was 100k(mine) vs 1m(theirs). I'll dump mine with GPU-z later tonight so I know it's for sure my specific card and see how that goes.

 

Also, I've been passing the onboard sound and not the GPU sound. Could that be an issue?

Link to comment
12 minutes ago, fritzgeralds said:

Thanks. I grabbed the rom from techpowerup and figured something was up with mine when it was 100k(mine) vs 1m(theirs). I'll dump mine with GPU-z later tonight so I know it's for sure my specific card and see how that goes.

 

Also, I've been passing the onboard sound and not the GPU sound. Could that be an issue?

No worries. Possibly, I wouldn't passthrough one without the other, because the driver might address the audio device expecting it to be there. You can passthrough multiple audio devices if you need to have onboard audio in the VM.

 

While you're in there by the way, I'd dump it with both so that you have a second vBIOS to try if something goes wrong with the first. It's a pain set up, and ATIFlash is more trusted than GPU-Z for generating a mod-able (complete) vBIOS, where GPU-Z is the go-to for producing vBIOS for vfio passthrough.

Edited by lnxd
Link to comment

@lnxd Got it working. Not sure if it was the vBIOS or the GPU audio but I've now survived several reboots and shutdowns with no Error 43 issues.

 

Per your suggestion, I loaded GPU as 2nd graphics card (I also enabled the GPU audio to be safe) and dumped the rom with GPU-z.

Removed VNC from VM used the rom I just dumped.

VM booted right up with no issues.

Attached second monitor for testing, rebooted, shutdown, still no issues.

 

Thank you for your help on this.

 

EDIT: For anyone wondering, I've loaded two VMs, one with Non-WHQL 2020-21.3.2 drivers and one with 2020-21.3.1 drivers. Both work fine and I'm able to shutdown/switch between both without issue.

Edited by fritzgeralds
Link to comment
28 minutes ago, fritzgeralds said:

@lnxd Got it working. Not sure if it was the vBIOS or the GPU audio but I've now survived several reboots and shutdowns with no Error 43 issues.

VM booted right up with no issues.

Attached second monitor for testing, rebooted, shutdown, still no issues.

 

Thank you for your help on this.

Woohoo! Hopefully it keeps going that way. I'm just happy to help, so any time 😁


You'll be better off with both changes anyway, one might have caused problems once you resolved the other so no harm, no foul.

Edited by lnxd
Link to comment
1 hour ago, trig229 said:

Thanks @trig229, when did you last have this problem? I'm assuming you've restarted since then, right? You said you can still restart the array afterwards? The data explaining the problem isn't in this diagnostics.zip. If you're willing to, I might ask you to:
 

  1. Start the VM with both the VGA controller and the audio device passed through:
    0c:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 [Radeon RX 6800/6800 XT / 6900 XT] [1002:73bf] (rev c1)
    	Subsystem: Sapphire Technology Limited Device [1da2:e438]
    	Kernel driver in use: vfio-pci
    	Kernel modules: amdgpu
    0c:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab28]
    	Subsystem: Advanced Micro Devices, Inc. [AMD/ATI] Device [1002:ab28]
    	Kernel driver in use: vfio-pci

     

  2. Stop it, and test that the VM won't start back up
     
  3. Grab a diagnostics.zip to post here (the issue also might be visible to you if you run the next command at this point. But I also want to see your XML files.)
    dmesg | grep vfio

     

  4. Stop the Array, and allow the system to go through a complete reboot so that you can use it again normally without the card stuck hanging

 

There is a chance your issue might be vBIOS related too if @giganode thinks so; although honestly you should be able to get away with not using a vBIOS file, booting to the Windows login screen, then shutting down and starting back up again. Especially if the VM is using the Microsoft Basic Display Driver. The card should still be able to reset, but it sounds like it's failing to.

Also if you ever get that Intel Corporation Wi-Fi 6 AX200 [8086:2723] working in a linux VM, hit me up 😅

Edited by lnxd
Link to comment

Hi @lnxd, sorry my bad!  I hadn't initiated the issue when i took that diagnostic dump so it wouldn't shown anything! 

 

Procedure;

Shutdown VM

Update XML to point to rom from techpower up as mentioned by @giganode and started VM

 - Booted but no output (code 43) as usual

VM shutdown

Updated XML to Removed the ROM entirely and started VM 

 - Booted but no output (code 43) as usual

 

Took dmesg, see attached

Took Diagnostics (see attached)

 

Rebooted unraid

 - VM boots fine with display 

 

Also attached the XML too

 

 

Not even tried passing through WIFI yet,  that's for another time :P 

win10.png

core2-diagnostics-20210409-1532.zip dmesg.txt win10vm.xml

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.