Hello Unraid Forums
I'm having major issues trying to setup PCIe GPU passthrough on my Unraid server. Everything else is working perfectly except for GPU passthrough.
I started on 6.8.3 RC and I have upgraded to 6.9.0 beta1 to see if the issue was a bug with Unraid.
Just before we start, I have tried mutliple methodolgies for trying to passthrough a GPU to a VM. Additionally, I have read the forum rules and feel that whilst this subject has been covered before, I still cannot get to the bottom of the issue. I'll go in to more detail. I've attached the diagnostics information, the XML for my Win10 VM. Before we start I'll just outline why I decided to use unraid.
Reasons for starting Unraid project
I have been studying for my CompTIA A+ and passed my first exam with a view to go on to do a CCNX qualification. I work as a 1st Line IT Tech Support and I've been been prodominately using Windows throughout my life. I wanted to learn more about Docker containers, VM's and paravirtualisation and cloud applications. I thought of renting a hosted VM platform but the cost is way to high considering I have hardware I can learn with myself (cloud VM's with GPU's are $$$). I started messing about with VirtualBox and wanted to create a paravirtualised environment with a PCIe GPU passthrough for gaming. It would server as a sort of useful retirement home for spare parts etc.
What I want to achieve
I want to use dockers for more persistent applications, such as P2P, ad-blocking, media server, etc. My intention was to have the following:
Intel I350-T4 Quad Port Gigabit - Passthrough to pfSense
MSI GTX 970 - Passthrough to a stable gaming VM (Win10/Mint/SteamOS)
Docker containers to handle app services, such as ad-blocking, P2P server, Plex, etc
Hackintosh environment
Windows 10 environement
Linux environment
Just quicklly for the record, uptime doesn't really matter to much to me, whilst pfSense is my number one priority to learn I need to do that last, after I have got the gaming/media VM's to work. I've got a failover plan if the Unraid server goes down, so lets leave my desire to use pfSense aside
Brief Specs
CPU Intel i5-4440 - GPU MSI GTX 970 - MB Gigabyte B85 HD3 rev 2.1
1x 240GB SSD - 1x 1TB Parity Drives - 3x 500GB Data Drives
Steam Controller - Intel I350-T4 Quad Port Gigabit
Guides/Videos Used So Far:
https://forums.serverbuilds.net/t/guide-remote-gaming-on-unraid/4248/9
Steps so far
I've been trying to get the GPU to passthrough to a VM for the best part of 10 days solid whilst I'm furloghed and I'm starting to lose it a bit. I know what I want to achieve is possible with the hardware I have (with some limitations, upgrades will be considered if I can get the project to a workable state.
The best I've got so far is to be able to pass the GPU through to Windows 10 Pro VM but I'm getting the dreaded error 43 message.
CPU and Motherboard
Before even starting this project I made sure that my CPU (i5-4440) and MB (Gigabyte B85-HD3) both support VT-d and virtualisation, which they aparently do.
GPU and vBIOS
I have placed the GTX 970 into my main PC and it worked perfectly. I did some stresstesting and benchmarking and it's a perfectly usable card. Whilst I was doing this, I used MSI Afterburner to download my BIOS rom for use as a vBIOS so I could follow the guides. Whats really perplexing me is that I see a lot of users getting thier GTX 970 to passthrough with no problem, then another section saying they just couldn't do it and gave up.
I've tried legacy boot with iGPU for console just to try and make sure that Unraid isn't doing weird things by loading it for console, I know it dissapears when you launch the VM but it's something I thought I'd try as the second slot option hasn't been working for me either.
Unraid
I've tried both UEFI and Legacy boot options for Unraid. I followed the recommendation of @SpaceInvaderOne to install the plug-in.
No modification attempts
Attempted to passthough on Linux Mint 19.3 - Fail
UEFI - Fail (System hangs after installing drivers)
Legacy - Fail (System hangs after installing drivers)
Attempted to passthrough to Windows 10 Pro - Fail
UEFI - Fail (Error 43)
Legacy - Fail (Error 43)
Attempted to passthrough to SteamOS - Fail
UEFI - Fail
Legacy - Fail
SpaceInvaderOne style
Editted VM xml - fail
Eddited Syslinux - fail
Windows 10 XML
<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
<name>VM-WIN10PRO</name>
<uuid>NOT SHARING THIS WITH YOU</uuid>
<metadata>
<vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
</metadata>
<memory unit='KiB'>4194304</memory>
<currentMemory unit='KiB'>4194304</currentMemory>
<memoryBacking>
<nosharepages/>
</memoryBacking>
<vcpu placement='static'>2</vcpu>
<cputune>
<vcpupin vcpu='0' cpuset='1'/>
<vcpupin vcpu='1' cpuset='2'/>
<emulatorpin cpuset='3'/>
</cputune>
<os>
<type arch='x86_64' machine='pc-i440fx-4.2'>hvm</type>
<loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
<nvram>/etc/libvirt/qemu/nvram/NOT SHARING THIS WITH YOU_VARS-pure-efi.fd</nvram>
</os>
<features>
<acpi/>
<apic/>
</features>
<cpu mode='host-passthrough' check='none'>
<topology sockets='1' cores='2' threads='1'/>
<cache mode='passthrough'/>
</cpu>
<clock offset='localtime'>
<timer name='hypervclock' present='yes'/>
<timer name='hpet' present='no'/>
</clock>
<on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot>
<on_crash>restart</on_crash>
<devices>
<emulator>/usr/local/sbin/qemu</emulator>
<disk type='file' device='disk'>
<driver name='qemu' type='raw' cache='writeback'/>
<source file='/mnt/user/domains/VM-WIN10PRO/vdisk1.img'/>
<target dev='hdc' bus='virtio'/>
<boot order='1'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
</disk>
<disk type='file' device='cdrom'>
<driver name='qemu' type='raw'/>
<source file='/mnt/user/isos/virtio-win-0.1.185.iso'/>
<target dev='hdb' bus='sata'/>
<readonly/>
<address type='drive' controller='0' bus='0' target='0' unit='1'/>
</disk>
<controller type='usb' index='0' model='qemu-xhci' ports='15'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
</controller>
<controller type='pci' index='0' model='pci-root'/>
<controller type='sata' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
</controller>
<controller type='virtio-serial' index='0'>
<address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
</controller>
<interface type='bridge'>
<mac address='52:54:00:ac:c2:7f'/>
<source bridge='br0'/>
<model type='virtio'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
</interface>
<serial type='pty'>
<target type='isa-serial' port='0'>
<model name='isa-serial'/>
</target>
</serial>
<console type='pty'>
<target type='serial' port='0'/>
</console>
<channel type='unix'>
<target type='virtio' name='org.qemu.guest_agent.0'/>
<address type='virtio-serial' controller='0' bus='0' port='1'/>
</channel>
<input type='tablet' bus='usb'>
<address type='usb' bus='0' port='4'/>
</input>
<input type='mouse' bus='ps2'/>
<input type='keyboard' bus='ps2'/>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</source>
<rom file='/boot/GPU-vBIOS/nVidia/MSI-GTX970-GM204-200-A1.rom'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0' multifunction='on'/>
</hostdev>
<hostdev mode='subsystem' type='pci' managed='yes'>
<driver name='vfio'/>
<source>
<address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
</source>
<address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x1'/>
</hostdev>
<hostdev mode='subsystem' type='usb' managed='no'>
<source>
<vendor id='0x046d'/>
<product id='0xc226'/>
</source>
<address type='usb' bus='0' port='1'/>
</hostdev>
<hostdev mode='subsystem' type='usb' managed='no'>
<source>
<vendor id='0x046d'/>
<product id='0xc227'/>
</source>
<address type='usb' bus='0' port='2'/>
</hostdev>
<hostdev mode='subsystem' type='usb' managed='no'>
<source>
<vendor id='0x28de'/>
<product id='0x1142'/>
</source>
<address type='usb' bus='0' port='3'/>
</hostdev>
<memballoon model='none'/>
</devices>
</domain>
tango-cloud-diagnostics-20200607-1917.zip