Jump to content
nicfitzgerald

AMD RX 590 Pass-Through

12 posts in this topic Last Reply

Recommended Posts

Posted (edited)

First off, I'm brand new to Unraid. Second, I apologize if this has been asked previously. I have been looking for AGES on how to make this work.

I'll start with my machine specs:

  • AMD Ryzen R7 1700
  • PowerColor RX 590
  • Corsair Dominator 16gb @3000

I have Unraid running without a problem, and I have a Ubuntu Server VM running without issue. I'm trying to get a Windows VM running and I've followed just about every tutorial I can find. I have excluded the RX from the host, and have verified that with a syslinux change and the ACS override in VM settings.

My VM Template:

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>Windows</name>
  <uuid>4680f734-d50c-0edf-f250-fc41bc3f39c0</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>10485760</memory>
  <currentMemory unit='KiB'>10485760</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>10</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='8'/>
    <vcpupin vcpu='2' cpuset='1'/>
    <vcpupin vcpu='3' cpuset='9'/>
    <vcpupin vcpu='4' cpuset='2'/>
    <vcpupin vcpu='5' cpuset='10'/>
    <vcpupin vcpu='6' cpuset='4'/>
    <vcpupin vcpu='7' cpuset='12'/>
    <vcpupin vcpu='8' cpuset='6'/>
    <vcpupin vcpu='9' cpuset='14'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-3.0'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='10' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Windows 10/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/Win10_1809Oct_v2_English_x64.iso'/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.160-1.iso'/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:5f:72:e5'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes' xvga='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x1f' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/isos/Powercolor.RX590.8192.181106.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x1f' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x1038'/>
        <product id='0x1730'/>
      </source>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x1b1c'/>
        <product id='0x1b49'/>
      </source>
      <address type='usb' bus='0' port='2'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x2433'/>
        <product id='0xb200'/>
      </source>
      <address type='usb' bus='0' port='3'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>

The syslinux override:

kernel /bzimage
append iommu=pt iommu=1 vfio-pci.ids=1002:67df, 1002:aaf0 pcie_acs_override=downstream,multifunction vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot

System devices:

IOMMU group 25:	[1002:67df] 1f:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] (rev e1)
IOMMU group 26:	[1002:aaf0] 1f:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 580]

 

With the GPU attached to Windows, it either won't boot at all, gives me the `127` error or it tells me that it's not able to power up and stuck in the d3 mode.

 

Logs after trying to boot the VM:

2019-04-15T22:53:36.119948Z qemu-system-x86_64: vfio: Unable to power on device, stuck in D3
2019-04-15T22:53:36.120452Z qemu-system-x86_64: vfio: Unable to power on device, stuck in D3
2019-04-15T22:53:41.588646Z qemu-system-x86_64: vfio_err_notifier_handler(0000:1f:00.0) Unrecoverable error detected. Please collect any data possible and then kill the guest
2019-04-15T22:53:41.589405Z qemu-system-x86_64: vfio_err_notifier_handler(0000:1f:00.1) Unrecoverable error detected. Please collect any data possible and then kill the guest

 

Are there any suggestions?

Edited by nicfitzgerald

Share this post


Link to post
21 minutes ago, nicfitzgerald said:

With the GPU attached to Windows, it either won't boot at all, gives me the `127` error or it tells me that it's not able to power up and stuck in the d3 mode.

Shot in the dark, but what happens if you start that VM first thing after a cold power cycle? As in disable any auto starting services, shut down the machine, remove power, power back up, start the problem VM before running anything else?

Share this post


Link to post
30 minutes ago, jonathanm said:

Shot in the dark, but what happens if you start that VM first thing after a cold power cycle? As in disable any auto starting services, shut down the machine, remove power, power back up, start the problem VM before running anything else?

I haven't tried without the Ubuntu server running, but I am currently trying to get a vbios dump of my own card instead of a third party source.

Share this post


Link to post
Posted (edited)

I am also new to Unraid and I'm having the exact same issue.  I have not purchased a license yet, running on trial...  License purchase will depend on this working.

 

Unraid Version: 6.7.0-rc7

AMD Ryzen 1700X

Radeon RX 570

With the GPU attached to Windows 10 VM, it either won't boot at all, gives me the `127` error or it tells me that it's not able to power up and stuck in the D3 mode.

jarvis-diagnostics-20190416-0400.zip

Edited by dk4dk4

Share this post


Link to post
1 hour ago, dk4dk4 said:

I am also new to Unraid and I'm having the exact same issue.  I have not purchased a license yet, running on trial...  License purchase will depend on this working.

 

Unraid Version: 6.7.0-rc7

AMD Ryzen 1700X

Radeon RX 570

With the GPU attached to Windows 10 VM, it either won't boot at all, gives me the `127` error or it tells me that it's not able to power up and stuck in the D3 mode.

jarvis-diagnostics-20190416-0400.zip

I so badly want to enjoy this, but it's borderline impossible and I definitely don't want to just have a sitting paperweight until it's fixed 😔

Share this post


Link to post

I have a RX 590 passed through to windows 10. I had to convert my windows 10 VM to uefi boot mode to get it working properly. Also, I have observed that if I stop the VM I have to reboot the whole server or upon VM start my server freezes. I do not stop the VM very often however so it has not been much of an issue. I am unable to post the Unraid VM config at the moment but I will tomorrow.

Share this post


Link to post
24 minutes ago, sadkisson said:

I have a RX 590 passed through to windows 10. I had to convert my windows 10 VM to uefi boot mode to get it working properly. Also, I have observed that if I stop the VM I have to reboot the whole server or upon VM start my server freezes. I do not stop the VM very often however so it has not been much of an issue. I am unable to post the Unraid VM config at the moment but I will tomorrow.

Thank you so much 😭

I was at a complete

Share this post


Link to post
57 minutes ago, sadkisson said:

I have a RX 590 passed through to windows 10. I had to convert my windows 10 VM to uefi boot mode to get it working properly. Also, I have observed that if I stop the VM I have to reboot the whole server or upon VM start my server freezes. I do not stop the VM very often however so it has not been much of an issue. I am unable to post the Unraid VM config at the moment but I will tomorrow.

My Win10 is already at UEFI (GPT) VM and still not working...

FYI

Share this post


Link to post

Below is my configuration with a working RX 590 pass through to Windows 10.

 

Hardware

Motherboard Supermicro X9SCM-F (modified pci express slot to fit X16 card) latest BIOS

Processor Intel Xeon E3-1200

Graphics Card XFX Radeon RX 590 Fatboy

 

UnRaid Settings

UnRaid booted in UEFI mode

PCIe ACS override: Downstream

VFIO allow unsafe interrupts: No

Graphics pass through using vfio-pci.cfg method stored on flash drive

 

Windows 10 VM config

Machine: Q35-3.1

BIOS: OVMF

Hyper-V: Yes

Graphics Card: AMD Radeon RX 470/480/570/570X/580/590

 

VM Settings

Windows 10 VM set to boot in UEFI mode

Latest VirtIO drivers

 

When I first tried to use the RX590 my Windows 10 VM was using Machine: i440fx and SeaBios. This resulted in my server giving an error when trying to start the VM. I had to convert my Windows 10 VM to boot via UEFI and then the VM would not boot with SeaBios. I had to recreate my UnRaid Windows 10 VM config to boot with Machine: OVMF and BIOS: Q35 and it loaded right up. Installed the AMD driver from their website and I was up and running.  The only issue I have had with this setup is the known AMD reset bug. So if I have to stop the VM I have to reset the entire server to start the VM again or the whole server will freeze when trying to start the VM. From what I understand it has something to do with the AMD graphics card not properly releasing after VM shutdown. Please if you have any question on my setup ask and I can give you any information I might have left out. Hope you all get yours working as well.
 

 

Share this post


Link to post
3 hours ago, sadkisson said:

Below is my configuration with a working RX 590 pass through to Windows 10.

 

Hardware

Motherboard Supermicro X9SCM-F (modified pci express slot to fit X16 card) latest BIOS

Processor Intel Xeon E3-1200

Graphics Card XFX Radeon RX 590 Fatboy

 

UnRaid Settings

UnRaid booted in UEFI mode

PCIe ACS override: Downstream

VFIO allow unsafe interrupts: No

Graphics pass through using vfio-pci.cfg method stored on flash drive

 

Windows 10 VM config

Machine: Q35-3.1

BIOS: OVMF

Hyper-V: Yes

Graphics Card: AMD Radeon RX 470/480/570/570X/580/590

 

VM Settings

Windows 10 VM set to boot in UEFI mode

Latest VirtIO drivers

 

When I first tried to use the RX590 my Windows 10 VM was using Machine: i440fx and SeaBios. This resulted in my server giving an error when trying to start the VM. I had to convert my Windows 10 VM to boot via UEFI and then the VM would not boot with SeaBios. I had to recreate my UnRaid Windows 10 VM config to boot with Machine: OVMF and BIOS: Q35 and it loaded right up. Installed the AMD driver from their website and I was up and running.  The only issue I have had with this setup is the known AMD reset bug. So if I have to stop the VM I have to reset the entire server to start the VM again or the whole server will freeze when trying to start the VM. From what I understand it has something to do with the AMD graphics card not properly releasing after VM shutdown. Please if you have any question on my setup ask and I can give you any information I might have left out. Hope you all get yours working as well.
 

 

I tried out your setup and still get the black screen on VM startup: RX 570 video card

 

IOMMU group 15:[1002:67df] 0b:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] (rev ff)

[1002:aaf0] 0b:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] (rev ff)

 

cat vfio-pci.cfg
BIND="0b:00.0"
 

VM log ending:

2019-04-18T03:24:07.735594Z qemu-system-x86_64: -device vfio-pci,host=0b:00.0,id=hostdev0,bus=pci.7,addr=0x1: Failed to mmap 0000:0b:00.0 BAR 0. Performance may be slow
2019-04-18T03:24:08.916579Z qemu-system-x86_64: vfio: Unable to power on device, stuck in D3
2019-04-18T03:24:08.920566Z qemu-system-x86_64: vfio: Unable to power on device, stuck in D3
2019-04-18T03:24:09.442392Z qemu-system-x86_64: vfio_err_notifier_handler(0000:0b:00.1) Unrecoverable error detected. Please collect any data possible and then kill the guest

 

Syslog:

Apr 18 00:14:00 Jarvis kernel: pci 0000:0b:00.0: [1002:67df] type 00 class 0x030000
Apr 18 00:14:00 Jarvis kernel: pci 0000:0b:00.0: reg 0x10: [mem 0xe0000000-0xefffffff 64bit pref]
Apr 18 00:14:00 Jarvis kernel: pci 0000:0b:00.0: reg 0x18: [mem 0xf0000000-0xf01fffff 64bit pref]
Apr 18 00:14:00 Jarvis kernel: pci 0000:0b:00.0: reg 0x20: [io  0xc000-0xc0ff]
Apr 18 00:14:00 Jarvis kernel: pci 0000:0b:00.0: reg 0x24: [mem 0xfce00000-0xfce3ffff]
Apr 18 00:14:00 Jarvis kernel: pci 0000:0b:00.0: reg 0x30: [mem 0xfce40000-0xfce5ffff pref]
Apr 18 00:14:00 Jarvis kernel: pci 0000:0b:00.0: BAR 0: assigned to efifb
Apr 18 00:14:00 Jarvis kernel: pci 0000:0b:00.0: supports D1 D2
Apr 18 00:14:00 Jarvis kernel: pci 0000:0b:00.0: PME# supported from D1 D2 D3hot D3cold
Apr 18 00:14:00 Jarvis kernel: pci 0000:0b:00.1: [1002:aaf0] type 00 class 0x040300
Apr 18 00:14:00 Jarvis kernel: pci 0000:0b:00.1: reg 0x10: [mem 0xfce60000-0xfce63fff 64bit]
Apr 18 00:14:00 Jarvis kernel: pci 0000:0b:00.1: supports D1 D2
 

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.