[SOLVED] Pass through GPU to Windows VM

[mPoUtiTUR]

I luckily found Unraid just over a year ago and have had lots of fun learning and figuring it out. I have done pretty much all of Spaceinvaderone's and ibracorp's tutorials. All with successfully making them work on my R-720. Now that the R-720 is essentially a production machine for my media/storage for the house and family, I put together a second machine to mess around with. I'll add the specs below. The first thing I wanted to do was pass a GPU through to a windows VM, why is this so hard? Unraid's website makes it seam like a normal function, but I have not been able to get it to work. I am hoping to get some help. 

 

Hardware:

Motherboard

 Supermicro X9SCM-F Intel C204, Version 1.11A
 American Megatrends Inc., Version 2.3a
 BIOS dated: Wed 06 Jan 2021 12:00:00 AM PST

CPU

 Intel® Xeon® CPU E3-1290 V2 @ 3.70GHz

RAM

 32 gigs of Hynix DDR3 ECC

GPU

 PNY GeForce® GT 710 2048MB PCI-E 2.0 Low Profile

 SKU: VCGGT7102XPB-BB

 

What I have done so far is to follow any and all tutorials I can find. 

 

Spaceinvaderone's vBios videos.

These tutorials:

I still get no output to my monitor and error code 43 on the GT 710. I have the vBios from tech power up I did all  Spaceinvaderone's ways to get it. I used GPUz to get it. I tried all the different vBios from the different methods. 

 

 

I tried to install Nvidia's driver while RDPd into the VM. 

I edited the XML to make sure the sound and video are on the same vbus.

Maybe I am just missing something? 

I attached the sys.log, VM log, and vBios of the card.

XML below.

 

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm' id='5' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>Windows 10</name>
  <uuid>a6fda19b-3dbc-8e64-d39d-cbd42bdc9776</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='4'/>
    <vcpupin vcpu='2' cpuset='1'/>
    <vcpupin vcpu='3' cpuset='5'/>
    <vcpupin vcpu='4' cpuset='2'/>
    <vcpupin vcpu='5' cpuset='6'/>
    <vcpupin vcpu='6' cpuset='3'/>
    <vcpupin vcpu='7' cpuset='7'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-5.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/a6fda19b-3dbc-8e64-d39d-cbd42bdc9776_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='1234567890ab'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <ioapic driver='kvm'/>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='4' threads='2'/>
    <cache mode='passthrough'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Windows 10/vdisk1.img' index='3'/>
      <backingStore/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <alias name='virtio-disk2'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/Windows 10 1909.iso' index='2'/>
      <backingStore/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <alias name='sata0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.190-1.iso' index='1'/>
      <backingStore/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <alias name='sata0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:11:36:8c'/>
      <source bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio-net'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-5-Windows 10/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='3'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <rom file='/mnt/user/isos/vbios/GK208.rom'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc249'/>
        <address bus='2' device='6'/>
      </source>
      <alias name='hostdev2'/>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x413c'/>
        <product id='0x2107'/>
        <address bus='2' device='5'/>
      </source>
      <alias name='hostdev3'/>
      <address type='usb' bus='0' port='2'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+100</label>
    <imagelabel>+0:+100</imagelabel>
  </seclabel>
  <qemu:commandline>
    <qemu:arg value='-cpu'/>
    <qemu:arg value='host,kvm=off,hv_vendor_id=null'/>
    <qemu:arg value='-machine'/>
    <qemu:arg value='q35,kernel_irqchip=on'/>
  </qemu:commandline>
</domain>

 

 

 

GK208.rom VMLog.txt syslog

Edited May 2, 2021 by mPoUtiTUR
solved

[John_M]

3 hours ago, mPoUtiTUR said:

why is this so hard?

 

Most likely because you're trying to pass through your only GPU. You'd probably find it easier if you had a second card or a CPU with integrated graphics that you could dedicate to Unraid.

 

[mPoUtiTUR]

The board has an integrated Matrox G200 on it, but unraid wants to use the GT 710. Any ideas on how to force unraid to us the onboard graphics? 

Edited May 1, 2021 by mPoUtiTUR

[twiikker]

I've had one guy whose card just wouldn't work at all as only GPU. He changed it to different card and it worked without problems.

Everything seems to be in order in your VM xml.

I don't know if some cards have just bioses which wont work. 

I guess you have also tried with latest drivers? After 465 they should have beta virtualizion support. Havent checked it myself yet.

 

[SimonF]

14 minutes ago, mPoUtiTUR said:

The board has an integrated Matrox G200 on it, but unraid wants to use the GT 710. Any ideas on how to force unraid to us the onboard graphics? 

In BIOS. Look for

PCIe/PCI/PnP Configuration

Boot Graphics Adapter Priority Use the feature to select the graphics controller to be used as the primary boot device. The options are Offboard VGA and Onboard VGA.

Edited May 1, 2021 by SimonF

[mPoUtiTUR]

Ok I had already changed that, but I update the bios and I bet it changed. Not sure why I didn’t check that I’ll try when I get back to the house thanks. 

[mPoUtiTUR]

Alright I double checked the bios was set to use the onboard as primary(it was). Added a second monitor from onboard for unraid and hooked the other one up the the GT 710. Still didn’t work I’ll keep trying. Thank you for helping.


Sent from my iPhone using Tapatalk

[SimonF]

Has the onboard worked before. I did notice there is a jumper for onboard vga in manual.

[mPoUtiTUR]

Yeah it has always worked. 
 

[SimonF]

7 hours ago, mPoUtiTUR said:

Yeah it has always worked. 
 

Do you have IPMI setup/enabled? I have a X9DR3 with 

 

09:03.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eW WPCM450 (rev 0a)
82:00.0 VGA compatible controller: NVIDIA Corporation GK208B [GeForce GT 720] (rev a1)
83:00.0 VGA compatible controller: NVIDIA Corporation TU106 [GeForce RTX 2060 Rev. A] (rev a1)

 

Next time I reboot will check BIOS settings. With VGA to HDMI I do get output on a screen. Each of the GPUs is connected to a different VMs, but I do have the IPMI enabled.

 

Have you checked BIOS vers/IPMI vers etc.

 

Running out of suggestions as it should work for you. I don't pass  a ROM to the VMs

 

I do use i440fx for my windows machines and not Q35.

Edited May 2, 2021 by SimonF

[mPoUtiTUR]

Thanks for all the help! I just did two things updated the IPMI (i already updated the bios), and took off the vBios passthrough. Seriously I appreciate the help.