koiril Posted September 25, 2019 Share Posted September 25, 2019 (edited) I've been trying for the last few weeks to get my primary GPU passed through to a Windows 10 VM, and despite following many different guides and instructions, I'm still not able to get it to work. Now I'm desperately hoping someone here can help me to get this working. I'm also hoping that this post will serve as a sort of guide detailing what I've tried for people that come after me. My Setup UnraidOS: 6.7.2 CPU: AMD Ryzen 7 3700X 3.6 GHz 8-Core Processor Motherboard: ASRock Fatal1ty B450 Gaming-ITX/ac Mini ITX AM4 Motherboard (BIOS version 3.50) Memory: Corsair Vengeance LPX 32 GB (2 x 16 GB) DDR4-3200 Memory Video Card: MSI Radeon RX 580 8 GB ARMOR OC Video Card I've followed a lot of different guides from around the internet. Unfortunately most of them don't work because of how Unraid loads the filesystem into memory, so any changes I make to the config (ie. modprobe blacklist configs) don't get persisted. The farthest I've been able to get so far is using the following steps: 1. Install a fresh copy of windows in a VM with the GPU set to VNC 2. Install all VFIO drivers from the VFIO driver ISO 3. Gracefully shutdown VM 4. SSH in to Unraid and run these commands echo 0 > /sys/class/vtconsole/vtcon0/bind echo 0 > /sys/class/vtconsole/vtcon1/bind echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind 5. Edit VM to add GPU and the HDMI Audio that's connected to it 6. Boot VM and connect using RDP (At this stage, I can see the Radeon GPU listed in the device manager) 7. Download and install Radeon drivers (I've tried the same set of steps with two different drivers and had the same results) 7a. Try video drivers from ASRock website (driver version 19.10.16) 7b. Try video drivers from AMD Website (driver version 19.9.2) 8. Drivers install freezes so I force shutdown VM 9. Now the VM freezes when the spinning dots appear during boot Since my plan is to pass an entire SSD to the VM, I've also tried: 1. Install Windows 10 directly on to the SSD 2. Boot directly in to Windows 10 3. Install all necessary drivers 4. Add SSD to VM and pass in GPU 5. Still freezes during boot I have a feeling this might not be working because Unraid doesn't come with the amdgpu kernal module, but I don't know enough about the inner workings of Linux to say for sure. I'm really hoping to get this working through UnraidOS since I love the interface and the feature set is exactly what I want for this build. ************************************************************************** * See below for extra details about my setup and stuff I've tried ************************************************************************** Guides: The Pragmatic Neckbeard 3: VFIO IOMMU and PCIe (Found here) : I followed this guide after finding it in this post, but of course this didn't work because it has me modify what's found in /etc/modprobe.d and no changes in there get persisted. How to passthrough an nvidia gpu as primary gpu in unRAID (Found here) : The result of this was a VM that didn't boot unless the GPU was set to VNC in the VM settings. How to easily passthough a Nvidia GPU as primary without dumping your own vbios! in KVM unRAID (Found here) : The result of this was a VM that didn't boot unless the GPU was set to VNC in the VM settings. IOMMU Groupings IOMMU Group 0 00:01.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] IOMMU Group 0 00:01.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483] IOMMU Group 0 00:01.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483] IOMMU Group 0 01:00.0 Non-Volatile memory controller [0108]: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981 [144d:a808] IOMMU Group 0 02:00.0 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset USB 3.1 XHCI Controller [1022:43d5] (rev 01) IOMMU Group 0 02:00.1 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller [1022:43c8] (rev 01) IOMMU Group 0 02:00.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge [1022:43c6] (rev 01) IOMMU Group 0 03:00.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01) IOMMU Group 0 03:01.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01) IOMMU Group 0 03:04.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01) IOMMU Group 0 03:05.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01) IOMMU Group 0 03:06.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01) IOMMU Group 0 03:07.0 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port [1022:43c7] (rev 01) IOMMU Group 0 08:00.0 Network controller [0280]: Intel Corporation Dual Band Wireless-AC 3168NGW [Stone Peak] [8086:24fb] (rev 10) IOMMU Group 0 09:00.0 Ethernet controller [0200]: Intel Corporation I211 Gigabit Network Connection [8086:1539] (rev 03) IOMMU Group 1 00:02.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] IOMMU Group 2 00:03.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] IOMMU Group 2 00:03.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge [1022:1483] IOMMU Group 2 0a:00.0 VGA compatible controller [0300]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] [1002:67df] (rev e7) IOMMU Group 2 0a:00.1 Audio device [0403]: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] [1002:aaf0] IOMMU Group 3 00:04.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] IOMMU Group 4 00:05.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] IOMMU Group 5 00:07.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] IOMMU Group 5 00:07.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484] IOMMU Group 5 0b:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function [1022:148a] IOMMU Group 6 00:08.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge [1022:1482] IOMMU Group 6 00:08.1 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484] IOMMU Group 6 00:08.2 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484] IOMMU Group 6 00:08.3 PCI bridge [0604]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B] [1022:1484] IOMMU Group 6 0c:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP [1022:1485] IOMMU Group 6 0c:00.1 Encryption controller [1080]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP [1022:1486] IOMMU Group 6 0c:00.3 USB controller [0c03]: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller [1022:149c] IOMMU Group 6 0c:00.4 Audio device [0403]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller [1022:1487] IOMMU Group 6 0d:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51) IOMMU Group 6 0e:00.0 SATA controller [0106]: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] [1022:7901] (rev 51) IOMMU Group 7 00:14.0 SMBus [0c05]: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller [1022:790b] (rev 61) IOMMU Group 7 00:14.3 ISA bridge [0601]: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge [1022:790e] (rev 51) IOMMU Group 8 00:18.0 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 0 [1022:1440] IOMMU Group 8 00:18.1 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 1 [1022:1441] IOMMU Group 8 00:18.2 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 2 [1022:1442] IOMMU Group 8 00:18.3 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 3 [1022:1443] IOMMU Group 8 00:18.4 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 4 [1022:1444] IOMMU Group 8 00:18.5 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 5 [1022:1445] IOMMU Group 8 00:18.6 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 6 [1022:1446] IOMMU Group 8 00:18.7 Host bridge [0600]: Advanced Micro Devices, Inc. [AMD] Matisse Device 24: Function 7 [1022:1447] More detailed PCI info: Only includes relevant section root@my-nas:~# lspci -v 0a:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere [Radeon RX 470/480/570/570X/580/580X/590] (rev e7) (prog-if 00 [VGA controller]) Subsystem: Micro-Star International Co., Ltd. [MSI] Radeon RX 580 Armor 4G OC Flags: fast devsel, IRQ 100 Memory at e0000000 (64-bit, prefetchable) [size=256M] Memory at f0000000 (64-bit, prefetchable) [size=2M] I/O ports at e000 [size=256] Memory at fce00000 (32-bit, non-prefetchable) [size=256K] Expansion ROM at 000c0000 [disabled] [size=128K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150] Advanced Error Reporting Capabilities: [200] Resizable BAR <?> Capabilities: [270] Secondary PCI Express <?> Capabilities: [2b0] Address Translation Service (ATS) Capabilities: [2c0] Page Request Interface (PRI) Capabilities: [2d0] Process Address Space ID (PASID) Capabilities: [320] Latency Tolerance Reporting Capabilities: [328] Alternative Routing-ID Interpretation (ARI) Capabilities: [370] L1 PM Substates Kernel driver in use: vfio-pci 0a:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Ellesmere HDMI Audio [Radeon RX 470/480 / 570/580/590] Subsystem: Micro-Star International Co., Ltd. [MSI] Device aaf0 Flags: fast devsel, IRQ 101 Memory at fce60000 (64-bit, non-prefetchable) [size=16K] Capabilities: [48] Vendor Specific Information: Len=08 <?> Capabilities: [50] Power Management version 3 Capabilities: [58] Express Legacy Endpoint, MSI 00 Capabilities: [a0] MSI: Enable- Count=1/1 Maskable- 64bit+ Capabilities: [100] Vendor Specific Information: ID=0001 Rev=1 Len=010 <?> Capabilities: [150] Advanced Error Reporting Capabilities: [328] Alternative Routing-ID Interpretation (ARI) Kernel driver in use: vfio-pci Kernal start configuration This configuration is a what's left over after trying a lot of different codes and options. kernel /bzimage append amd_iommu=on iommu=pt vfio-pci.ids=1002:67df,1002:aaf0 rd.driver.pre=vfio-pci vfio_iommu_type1.allow_unsafe_interrupts=1 video=vesafb:off,efifb:off modprobe.blacklist=radeon,amdgpu,fglrx initrd=/bzroot VM Config XML: <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm'> <name>gaming_vm</name> <uuid>5465a153-e190-dcee-ca6a-d30bfcad270b</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/> </metadata> <memory unit='KiB'>16777216</memory> <currentMemory unit='KiB'>16777216</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>8</vcpu> <cputune> <vcpupin vcpu='0' cpuset='4'/> <vcpupin vcpu='1' cpuset='12'/> <vcpupin vcpu='2' cpuset='5'/> <vcpupin vcpu='3' cpuset='13'/> <vcpupin vcpu='4' cpuset='6'/> <vcpupin vcpu='5' cpuset='14'/> <vcpupin vcpu='6' cpuset='7'/> <vcpupin vcpu='7' cpuset='15'/> </cputune> <os> <type arch='x86_64' machine='pc-i440fx-3.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/5465a153-e190-dcee-ca6a-d30bfcad270b_VARS-pure-efi.fd</nvram> </os> <features> <acpi/> <apic/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='1' cores='8' threads='1'/> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='no'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <disk type='block' device='disk'> <driver name='qemu' type='raw' cache='writeback'/> <source dev='/dev/disk/by-id/ata-Samsung_SSD_860_EVO_250GB_S3YHNX0M715853F'/> <target dev='hdc' bus='sata'/> <boot order='1'/> <address type='drive' controller='0' bus='0' target='0' unit='2'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/Win10_1903_V1_English_x64.iso'/> <target dev='hda' bus='ide'/> <readonly/> <boot order='2'/> <address type='drive' controller='0' bus='0' target='0' unit='0'/> </disk> <disk type='file' device='cdrom'> <driver name='qemu' type='raw'/> <source file='/mnt/user/isos/virtio-win-0.1.171.iso'/> <target dev='hdb' bus='ide'/> <readonly/> <address type='drive' controller='0' bus='0' target='0' unit='1'/> </disk> <controller type='pci' index='0' model='pci-root'/> <controller type='ide' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='sata' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </controller> <controller type='virtio-serial' index='0'> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </controller> <controller type='usb' index='0' model='ich9-ehci1'> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/> </controller> <controller type='usb' index='0' model='ich9-uhci1'> <master startport='0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/> </controller> <controller type='usb' index='0' model='ich9-uhci2'> <master startport='2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/> </controller> <controller type='usb' index='0' model='ich9-uhci3'> <master startport='4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/> </controller> <interface type='bridge'> <mac address='52:54:00:bc:b7:a3'/> <source bridge='br0'/> <model type='virtio'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </interface> <serial type='pty'> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> </serial> <console type='pty'> <target type='serial' port='0'/> </console> <channel type='unix'> <target type='virtio' name='org.qemu.guest_agent.0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'/> <input type='keyboard' bus='ps2'/> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/> </source> <rom file='/mnt/user/system/vbios/msi_radeon_rx580.rom'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0a' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </hostdev> <memballoon model='none'/> </devices> </domain> Edited September 25, 2019 by koiril Removed accidental emoji Quote Link to comment
YEAHHWAY Posted April 2, 2020 Share Posted April 2, 2020 Why has no one been able to answer this person's question? 1 Quote Link to comment
mechmess Posted March 28, 2021 Share Posted March 28, 2021 I'm having the exact same issue! @koiril - were you ever able to find a solution? Same as you, I did all the normal steps and eventually tried doing a native SSD install and then passing through the SSD and GPU - the VM always freezes when the drivers are initialized. I *am* able to pass through an Nvidia 710 that I have laying around, but the RX 580 fails, and it's not isolated to windows either - the same thing happens to MacOS and Linux VMs as well. The most frustrating thing for me is that I had it working just fine, but swapped to a different motherboard. Quote Link to comment
koiril Posted March 28, 2021 Author Share Posted March 28, 2021 Glad to hear it was working at one point! I gave up on trying since I could only get it somewhat working. In case you're looking for more info, I posted this same question on Reddit and got some useful responses, so maybe those could help you too. https://old.reddit.com/r/unRAID/comments/d9niuv/radeon_rx_580_primary_gpu_passthrough/ Quote Link to comment
RusmanCool Posted February 13, 2023 Share Posted February 13, 2023 @koiril, the best way to install GPU in VM is to use VNC all the way up unto ALL drivers are installed. I have x470 based mobo and Ryzen 5 CPU with XFX RX580 GPU. I had installed it into Win10 Vm without any issues. Well, there is an issue of Error 43 that appears if I do not shutdown VM completely/safely. Other than that, it works. Also, take a look at SpaceInvaderOne video about advanced GPU pathrough. It shows that it is necessary to passthrough both Video&Audo devices together in the same virtual Bus and Slot but use function 0 and 1 accordingly. Quote Link to comment
Chris Hill Posted February 17, 2023 Share Posted February 17, 2023 I had this same problem, with an RX570. What finally worked for me was to use spaceinvader one's user script to dump the vbios. After that, things finally worked and I could install the most recent AMD Adrenalin drivers without getting a black screen. I did have to restart the VM a few times at one point in the process. Some other details that may be relevant to my answer working for you: - I installed windows 10 pro directly on an NVME drive, then created the VM to use that drive - I had to create and delete several VM Templates with all the different things that I tried - switching between the GPU passthrough and VNC did something funny at one point - The last thing I did was to boot in to windows safe mode, use DDU, then I made the template with the vbios and was able to install drivers in the VM - When the dumped vbios was linked in the template, I had already installed the vrtio drivers. I don't know if that order matters - Using the vbios from TechPowerUp didn't work for me (i didn't check if there was a version mismatch) Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.