First off, I'm brand new to Unraid. Second, I apologize if this has been asked previously. I have been looking for AGES on how to make this work.
I'll start with my machine specs:
AMD Ryzen R7 1700
PowerColor RX 590
Corsair Dominator 16gb @3000
I have Unraid running without a problem, and I have a Ubuntu Server VM running without issue. I'm trying to get a Windows VM running and I've followed just about every tutorial I can find. I have excluded the RX from the host, and have verified that with a syslinux change and the ACS override in VM settings.
My VM Template:
<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
<name>Windows</name>
<uuid>4680f734-d50c-0edf-f250-fc41bc3f39c0</uuid>
<metadata>
<vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
</metadata>
<memory unit='KiB'>10485760</memory>
<currentMemory unit='KiB'>10485760</currentMemory>
<memoryBacking>
<nosharepages/>
</memoryBacking>
<vcpu placement='static'>10</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='0'/>
<vcpupin vcpu='1' cpuset='8'/>
<vcpupin vcpu='2' cpuset='1'/>
<vcpupin vcpu='3' cpuset='9'/>
<vcpupin vcpu='4' cpuset='2'/>
<vcpupin vcpu='5' cpuset='10'/>
<vcpupin vcpu='6' cpuset='4'/>
<vcpupin vcpu='7' cpuset='12'/>
<vcpupin vcpu='8' cpuset='6'/>
<vcpupin vcpu='9' cpuset='14'/>
</cputune>
<os>
<type arch='x86_64' machine='pc-i440fx-3.0'>hvm</type>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='host-passthrough' check='none'>
<topology sockets='1' cores='10' threads='1'/>
</cpu>
<clock offset='localtime'>
<timer name='rtc' tickpolicy='catchup'/>
<timer name='pit' tickpolicy='delay'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<devices>
<emulator>/usr/local/sbin/qemu</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='writeback'/>
<source file='/mnt/user/domains/Windows 10/vdisk1.img'/>
<target dev='hdc' bus='virtio'/>
<boot order='1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</disk>
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<source file='/mnt/user/isos/Win10_1809Oct_v2_English_x64.iso'/>
<target dev='hda' bus='sata'/>
<readonly/>
<boot order='2'/>
<address type='drive' controller='0' bus='0' target='0' unit='0'/>
</disk>
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<source file='/mnt/user/isos/virtio-win-0.1.160-1.iso'/>
<target dev='hdb' bus='sata'/>
<readonly/>
<address type='drive' controller='0' bus='0' target='0' unit='1'/>
</disk>
<controller type='usb' index='0' model='ich9-ehci1'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci1'>
<master startport='0'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci2'>
<master startport='2'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
</controller>
<controller type='usb' index='0' model='ich9-uhci3'>
<master startport='4'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
</controller>
<controller type='pci' index='0' model='pci-root'/>
<controller type='sata' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</controller>
<controller type='virtio-serial' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</controller>
<interface type='bridge'>
<mac address='52:54:00:5f:72:e5'/>
<source bridge='br0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</interface>
<serial type='pty'>
<target type='isa-serial' port='0'>
<model name='isa-serial'/>
</target>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<channel type='unix'>
<target type='virtio' name='org.qemu.guest_agent.0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<hostdev mode='subsystem' type='pci' managed='yes' xvga='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x1f' slot='0x00' function='0x0'/>
</source>
<rom file='/mnt/user/isos/Powercolor.RX590.8192.181106.rom'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x1f' slot='0x00' function='0x1'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
</hostdev>
<hostdev mode='subsystem' type='usb' managed='no'>
<source>
<vendor id='0x1038'/>
<product id='0x1730'/>
</source>
<address type='usb' bus='0' port='1'/>
</hostdev>
<hostdev mode='subsystem' type='usb' managed='no'>
<source>
<vendor id='0x1b1c'/>
<product id='0x1b49'/>
</source>
<address type='usb' bus='0' port='2'/>
</hostdev>
<hostdev mode='subsystem' type='usb' managed='no'>
<source>
<vendor id='0x2433'/>
<product id='0xb200'/>
</source>
<address type='usb' bus='0' port='3'/>
</hostdev>
<memballoon model='none'/>
</devices>
</domain>
The syslinux override:
kernel /bzimage
append iommu=pt iommu=1 vfio-pci.ids=1002:67df, 1002:aaf0 pcie_acs_override=downstream,multifunction vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot
System devices:
IOMMU group 25: [1002:67df] 1f:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] (rev e1)
IOMMU group 26: [1002:aaf0] 1f:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 580]
With the GPU attached to Windows, it either won't boot at all, gives me the `127` error or it tells me that it's not able to power up and stuck in the d3 mode.
Logs after trying to boot the VM:
2019-04-15T22:53:36.119948Z qemu-system-x86_64: vfio: Unable to power on device, stuck in D3
2019-04-15T22:53:36.120452Z qemu-system-x86_64: vfio: Unable to power on device, stuck in D3
2019-04-15T22:53:41.588646Z qemu-system-x86_64: vfio_err_notifier_handler(0000:1f:00.0) Unrecoverable error detected. Please collect any data possible and then kill the guest
2019-04-15T22:53:41.589405Z qemu-system-x86_64: vfio_err_notifier_handler(0000:1f:00.1) Unrecoverable error detected. Please collect any data possible and then kill the guest
Are there any suggestions?