[SOLVED] GPU passthrough issues on 3rd Gen Threadripper 3960X + ASRock TRX40 Creator + MSI GeForce RTX 2070 Super 8GB Ventus OC


Hawkins

Recommended Posts

At time of posting, the display works, and I can use Windows, but the resolution is terrible x aweful and the nVidia drivers will not install.

 

Things I've tried that haven't yet solved the problem, however I am certain that I have not tried every one of the ~14^2 possible combinations.

  • Q35 vs i440fx
  • SeaBIOS vs OVMF
  • Enable vs disable Hyper-V extension
  • GPU passthrough without BIOS (no display output) as well as with either the same-model factory BIOS or modified BIOS, both with and without the nVidia header
  • pcie_acs_override=downstream,multifunction
  • vfio_iommu_type1.allow_unsafe_interrupts=1
  • Various combinations of BIOS settings:
    • SR-IOV Enabled/Disabled
    • IOMMU Auto/Enabled
    • PCIe ARI Support Enabled/Disabled
    • Enable AER Cap Enable/Disabled (Enabled prevented VM from starting)
    • ACS Enable Enabled/Disabled (Requires that Enable AER Cap be Enabled)

 

Things that I already account for:

  • All four devices associated with my GPU are stubbed via vfio-pci.ids=
  • All four devices (one display, one sound, and two "other") are passed through to VM
  • VM's XML is edited to put all 4 GPU devices into one slot with multimode enabled and functions numbered to match source hardware

 

Has anyone struggled with this before? I'm happy to provide any log files, config files, or screenshots. Thanks in advance!

 

Device Manager

01 https://i.imgur.com/riJoNcR.jpg

 

VM Config

02 https://i.imgur.com/CnvxQMJ.jpg

03 https://i.imgur.com/uetUDyn.jpg

 

VM XML

04 https://i.imgur.com/cjDrYtn.jpg

 

Kernel

05 https://i.imgur.com/27jajNP.jpg

 

IOMMU

06 https://i.imgur.com/SVIc3sm.jpg

07 https://i.imgur.com/8anUXQS.jpg

08 https://i.imgur.com/Fm8EjJO.jpg

09 https://i.imgur.com/6VhCiuy.jpg

10 https://i.imgur.com/lQaKnyb.jpg

 

Hardware IDs

11 https://i.imgur.com/W88cK9q.jpg

Edited by Hawkins
URL formatting
Link to comment

First and foremost, please copy-paste the xml of your VM template instead of using screenshot - USE THE CODE FUNCTIONALITY of the forum i.e. the </> button next to the smiley button.

 

When you say the Nvidia driver "will not install", what does it mean? Was there an error? If so what was the error?

 

Is the RTX2070 the only GPU?

Did you dump the vbios yourself or download from Techpowerup?

 

After answering / doing all of the above, start a brand new template with Q35 + OVMF + Hyper-V On. Then add this line above </hyperv>

      <vendor_id state='on' value='0123456789ab'/>

Then save and see if it works.

Edited by testdasi
Link to comment

I used your email signature, hope you don't mind. :)

I took screenshots because I was short on time when I gathered all that data. I'll spend the time now to get you the data you are asking for. In the mean time I'll address the other questions.

Quote

When you say the Nvidia driver "will not install", what does it mean? Was there an error? If so what was the error?

I get this from the 442.19 Windows 10 64-bit Game Ready Driver (GRD) and Studio Driver (SD) installers:

 

vfXyraj.jpg

 

If I stop at this point of the install, which is before the extracted files are deleted from C:\NVIDIA\, I can go to Device Manager and point each device with an issue to the path containing the .inf files. No success from either driver pack, or on any of the 4 devices with issues.

Quote

Is the RTX2070 the only GPU?

Did you dump the vbios yourself or download from Techpowerup?

Yes it is the only GPU in the systems. Installed in the slot closest the CPU. The BIOS came from TechPowerUp and appears to be a perfect model match based on the description from that site. I've debated dumping my own and then pulling the header off.

<vendor_id state='on' value='0123456789ab'/>

While I give this a try, can you tell me what this is doing?

Edited by Hawkins
Added first line thanking testdasi for his signature.
Link to comment

Here is the functional VM XML prior to testdasi's recommend changes. Note that Windows 10 Pro boots, has video, sound works, keyboard & mouse are good, etc. It appears that only video is an issue. The screenshots at start of thread are based on this config:

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm' id='1'>
  <name>Desktop-Jesse</name>
  <uuid>ddfe75d2-246d-6755-757f-21297e0dfafa</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>24</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='12'/>
    <vcpupin vcpu='1' cpuset='36'/>
    <vcpupin vcpu='2' cpuset='13'/>
    <vcpupin vcpu='3' cpuset='37'/>
    <vcpupin vcpu='4' cpuset='14'/>
    <vcpupin vcpu='5' cpuset='38'/>
    <vcpupin vcpu='6' cpuset='15'/>
    <vcpupin vcpu='7' cpuset='39'/>
    <vcpupin vcpu='8' cpuset='16'/>
    <vcpupin vcpu='9' cpuset='40'/>
    <vcpupin vcpu='10' cpuset='17'/>
    <vcpupin vcpu='11' cpuset='41'/>
    <vcpupin vcpu='12' cpuset='18'/>
    <vcpupin vcpu='13' cpuset='42'/>
    <vcpupin vcpu='14' cpuset='19'/>
    <vcpupin vcpu='15' cpuset='43'/>
    <vcpupin vcpu='16' cpuset='20'/>
    <vcpupin vcpu='17' cpuset='44'/>
    <vcpupin vcpu='18' cpuset='21'/>
    <vcpupin vcpu='19' cpuset='45'/>
    <vcpupin vcpu='20' cpuset='22'/>
    <vcpupin vcpu='21' cpuset='46'/>
    <vcpupin vcpu='22' cpuset='23'/>
    <vcpupin vcpu='23' cpuset='47'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='24' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Desktop-Jesse/vdisk1.img' index='3'/>
      <backingStore/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <alias name='virtio-disk2'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/en_windows_10_multiple_editions_x64_dvd_6846432.iso' index='2'/>
      <backingStore/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <alias name='sata0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.173-2.iso' index='1'/>
      <backingStore/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <alias name='sata0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x9'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0xa'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0xb'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xc'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xd'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0xe'/>
      <alias name='pci.7'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0xf'/>
      <alias name='pci.8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/>
    </controller>
    <controller type='pci' index='9' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='9' port='0x10'/>
      <alias name='pci.9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:a5:00:bd'/>
      <source bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-1-Desktop-Jesse/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <hostdev mode='subsystem' type='pci' managed='yes' xvga='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <rom file='/mnt/user/isos/MSI.RTX2070Super.8192.190701-FactoryNoHeader.rom'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x2'/>
      </source>
      <alias name='hostdev2'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x2'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x3'/>
      </source>
      <alias name='hostdev3'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x3'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x21' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev4'/>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+100</label>
    <imagelabel>+0:+100</imagelabel>
  </seclabel>
</domain>

 

Link to comment
2 hours ago, Chess said:

Patch Windows 10 to the latest build version using the Windows Update Assistant, and then try to run the Nvidia driver install again. I get that if I use my older Windows 10 ISOs. 

[SOLVED] It turns out the nVidia error that the driver wasn't compatible with this version of Windows was accurate. I was running built 10240 which was the first release of WIndows 10. I really need to delete my old ISOs. :) Once I upgraded to Windows 10 Pro version 1909, the graphics driver installed and ran with issue. I still have a few items in my device manager that need to be fixed, but my main issue, GPU passthough, is resolved.

FYI, Microsoft makes it hard to get an ISO of 1909. Easy to get around, try this Google search: https://www.google.com/search?q="index+of"+Win10_1909_English_x64.iso

Link to comment
  • Hawkins changed the title to [SOLVED] GPU passthrough issues on 3rd Gen Threadripper 3960X + ASRock TRX40 Creator + MSI GeForce RTX 2070 Super 8GB Ventus OC
6 minutes ago, Hawkins said:

[SOLVED] It turns out the nVidia error that the driver wasn't compatible with this version of Windows was accurate. I was running built 10240 which was the first release of WIndows 10. I really need to delete my old ISOs. :) Once I upgraded to Windows 10 Pro version 1909, the graphics driver installed and ran with issue. I still have a few items in my device manager that need to be fixed, but my main issue, GPU passthough, is resolved.

FYI, Microsoft makes it hard to get an ISO of 1909. Easy to get around, try this Google search: https://www.google.com/search?q="index+of"+Win10_1909_English_x64.iso

 

Glad you go it working! I was surprised by that issue when I first saw it, but a quick search in google sent me in the right direction. Thanks for the link. I'll update my ISO as well with the above one.

  • Like 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.