Problems with GPU passthrough to VM


Recommended Posts

Disclaimer: I am new to unraid and have just installed the OS, but I am looking to set up a win 10 instance with a passed-through GPU to use day-to-day

 

Problem

I'm having some issues passing through the GPU to a Win10 VM install.

As soon as I power up the VM (which has my single GPU and 5/6 cores assigned), the screen goes blank and the VM refuses to boot (remote desktop doesn't work, but my router detects the virtual ethernet adapter and gives it a valid IP). You can find the XML of the VM at the end of the post.

 

Hardware
CPU:           Ryzen 5 2600 (6 core 12 thread)

MOBO:        B450 Pro4 (2 PCIe x16, 2 M.2, 2 PCIe x1) (BIOSv4.1 and has AMD AGESA version 1.0.0.4 Patch B )

GPU:           2*GTX 1070 (although I only have one plugged in because I want to use a PCIe SSD and that disables the second x16 PCIe slot in the MOBO according to the docs)

RAM:           32GB 2400Hz (4*8GB S-DIMMs)

STORAGE:   3*3TB HDD, 256GB M.2 nvme SSD, 246GB SATA SSD, 16GB 3.0 USB (for booting unraid)

 

Steps I took

I followed Spaceinvarde one's two videos to set up a windows 10 VM with a passed-through GPU:

The best way to install and setup a windows 10 vm as a daily driver or a Gaming VM

The best way to install and setup a windows 10 vm Part 2 Hardware Passthrough

And, as far as I know, I followed all the needed steps (except certain software installs he carried out on the VM itself)

 

So, the steps that I took were:
- Enabled IOMMU and HVM in my BIOS (I can see them enabled in the unraid system info, and the GPU is in its own IOMMU group)

Screenshot-from-2020-04-14-14-15-18.png

Screenshot-from-2020-04-14-14-22-40.png

- Enabled PCIe ACS override in the settings (but unsafe interrupts are off)

- Booted and installed Win10 and VirtIO dirvers (all the ones from the video) through VNC

- Did a ROM dump for my GTX 1070 and put it in the VM XML description for it to use (XML description at the end of the post)

Screenshot-from-2020-04-14-14-27-32.png

- Passed through the GPU to the VM in the VM settings

Screenshot-from-2020-04-14-14-20-46.png

- Passed through the audio device of the GPU to the VM as a multifunction device (so the GPU and its audio device share the same bus), as described in this video: Advanced GPU passthrough techniques on Unraid

- Installed Nvidia drivers in the VM

 

What I didn't do:

- Enable unsafe interrupts

 

What should I do? I can't get it working even after trying to restart the VM and unraid several times and enabling VM auto-start.

 

 

My diagnostics file is:

tower-diagnostics-20200414-1422.zip

 

 

My VM XML setup is:
 

Quote

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>Windows 10</name>
  <uuid>e3bbb35e-ce5b-1e28-be57-c775fc55fd01</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>10</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='1'/>
    <vcpupin vcpu='1' cpuset='7'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='8'/>
    <vcpupin vcpu='4' cpuset='3'/>
    <vcpupin vcpu='5' cpuset='9'/>
    <vcpupin vcpu='6' cpuset='4'/>
    <vcpupin vcpu='7' cpuset='10'/>
    <vcpupin vcpu='8' cpuset='5'/>
    <vcpupin vcpu='9' cpuset='11'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-4.2'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/e3bbb35e-ce5b-1e28-be57-c775fc55fd01_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='5' threads='2'/>
    <cache mode='passthrough'/>
    <feature policy='require' name='topoext'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/VMs/Windows 10/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:ba:7c:d5'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/biosdump/1070.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0a' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc07e'/>
      </source>
      <address type='usb' bus='0' port='2'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc33a'/>
      </source>
      <address type='usb' bus='0' port='3'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'/>
</domain>

 

 

 

Edited by dockunraiwin
Changed hosting of diagnostics to unraid forums
Link to comment
8 hours ago, dockunraiwin said:

- Passed through the audio device of the GPU to the VM as a multifunction device (so the GPU and its audio device share the same bus), as described in this video: Advanced GPU passthrough techniques on Unraid

Your diagnostics xml does not have the soundcard passed through. Was it saved correctly?

 

3 hours ago, ap90033 said:

I suggest getting the VFIO-PCI Config Plugin/Utility. I have a rtx2080 and it has built in USB which was causing issues until I added them.... 

His card is the GTX 1070 which doesn't have USB devices.

Link to comment
1 hour ago, testdasi said:

Your diagnostics xml does not have the soundcard passed through. Was it saved correctly?

 

His card is the GTX 1070 which doesn't have USB devices.

Correct, but he may have some devices that are preventing bootup due to being in an iommu group and not separate. Just a thought 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.