Hello people of the internet,

now im trying to passthrough my R9 380 to Ubuntu or Windows. Now the Card gets detected by Windows and Ubuntu (lspci).
But when I try to install the drivers on Windows, it seems at first everything works, no error 43 (anymore), but after rebooting Windows gets stuck and the circle stops spinning.
On Ubuntu after downgrading the kernel to 5.4(?) so i can install AMD rocm, no device gets detected so I installed the amdgpu-pro drivers, but also doesnt seem to work.
I had to edit some settings in order to get Windows/Ubuntu to recognize the card things like:
allow unsafe interrupts, change PCIe ACS override to multi-function, also creating a IOMMU group (or whatever its called, under systemdevices selected both gpu and audio device).

I realized a phenomen where after installing the drivers the OS get stucks after shutdown, there is no picture nothing.
Under Windows i used the BIOS SeaBios but that didnt really help.

Because im pretty unfamiliar with Unraid and vms in general, im pretty sure that I missinterpreted some setting or there is some thing I configured wrong.
 

My Hardware:
Processor: Intel Xeon X3470
Mainboard: Supermicro X8SIL (Bios V 1.1)
Ram: 16GB DDR3 Single-Bit ECC Memory
GPU: ASUS Strix R9 380 4GB (VBios was copied by using GPU-Z on another PC)
The VM is on a Harddrive instead of a SSD (probably unrelated, but I am absolutly clueless and start suspecting everything)

Unraid Version 6.9.2
Under Dashboard > Info HVM and IOMMU are enabled.

 

The reason why I am using Q35-4.2 instead of Q35-5.1 is because someone in some forum (idk which) had his issue fix by using this version.

Now here is my Ubuntu 20.04.2 VM:

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>Ubuntu</name>
  <uuid>105ea47c-a14f-8faf-c2ef-0952797da2c1</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Ubuntu" icon="ubuntu.png" os="ubuntu"/>
  </metadata>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='1'/>
    <vcpupin vcpu='1' cpuset='5'/>
    <vcpupin vcpu='2' cpuset='3'/>
    <vcpupin vcpu='3' cpuset='7'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/105ea47c-a14f-8faf-c2ef-0952797da2c1_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='2' threads='2'/>
    <cache mode='passthrough'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Ubuntu/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/ubuntu-20.04-desktop-amd64.iso'/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:bc:e2:e2'/>
      <source bridge='br0'/>
      <model type='virtio-net'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/disk2/GameSaves/VBIos/Tonga.rom'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>

Hopefully someone has an idea or solution to my problem.

 

11 hours ago, darkiyy said:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/disk2/GameSaves/VBIos/Tonga.rom'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>

Some comments:

1- audio part of gpu should be in the same bus and slot, but different function, in a multifunction device, not on different bus, change to this:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/disk2/GameSaves/VBIos/Tonga.rom'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x1'/>
    </hostdev>

2- dump your own video bios, it's safer and you will be sure that the vbios is for YOUR gpu: you can dump it from unraid or it's very easy if you have access to windows to save a copy with gpuz (<--- already done, OK)

 

3- Some of the R9 380 have a physical switch, to use the gpu in uefi or legacy mode. Check your instruction manual and set it to use uefi, since you are using ovmf. Below an image of the switch on a R9 380 Nitro.

 

 

To check UEFI vs legacy mode you can use gpuz to see if there's a check next to "UEFI".

Edited June 4, 20215 yr by ghost82

11 hours ago, darkiyy said:

I realized a phenomen where after installing the drivers the OS get stucks after shutdown, there is no picture nothing

You may also suffer the so-called "amd reset bug".

This may be fixed with the "vendor reset patch", ex:

 

2 hours ago, ghost82 said:

Some comments:

1- audio part of gpu should be in the same bus and slot, but different function, in a multifunction device, not on different bus, change to this:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/disk2/GameSaves/VBIos/Tonga.rom'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x1'/>
    </hostdev>

2- dump your own video bios, it's safer and you will be sure that the vbios is for YOUR gpu: you can dump it from unraid or it's very easy if you have access to windows to save a copy with gpuz (<--- already done, OK)

 

3- Some of the R9 380 have a physical switch, to use the gpu in uefi or legacy mode. Check your instruction manual and set it to use uefi, since you are using ovmf. Below an image of the switch on a R9 380 Nitro.

 

 

To check UEFI vs legacy mode you can use gpuz to see if there's a check next to "UEFI".

The first tip worked, i would never found that issue. Thank you!
I dindt tryed the resetbug fix, but i will look into that a bit later

1 hour ago, darkiyy said:

I dindt tryed the resetbug fix

If your pc crash when restarting the vm with a gpu passthrough, maybe you are affected. If it doesn't crash then don't bother.