Help Needed! - Passthrough GPU Unable to Install Drivers - Low Resolution

[Hankanman]

Hi All,

I have unraid running with a GTX 970 and a shiny new RTX 2080 Super, the 970 works with no issue when assigning to VM and drivers were picked up etc.

For the 2080 super i had to dump the vbios and strip the nvidia header, once i did that i was able to get it to output to a display, i have also passed through the onboard audio and usb controllers successfully, followed all sorts of forum posts and spaceinvader one videos (godsend).

The VM picks up it is an RTX 2080 Super and recognises it. But whether through standard installer or geforce experience i cannot get the drivers to stick! Any help would be appreciated I would love to trace rays as soon as possible

Edited August 1, 2019 by Hankanman

[testdasi]

What do you mean by "cannot get the drivers to stick"? Error code 43?

[Hankanman]

get the VM booted and WIndwos 10 installed at 800x600, Run the nvidia driver install, screen flashes (Which is usually when the resolution moves to the screen native) stays at 800x600. Reboot, and if you run the driver install again, the machine doesn't think it is installed.

[testdasi]

Some potential fixes:

• Disable Hyper-V (start a new template, don't edit your current template)
• Is the 2080 in the primary slot (i.e. does Unraid stuff shows up at boot via the 2080 display)? If so, swap it with the 970 (i.e. have the 970 as primary for Unraid to boot with) and try again - make sure you passed through the right card after the swap.
• vfio-pci stub the 2080
• Boot Unraid in legacy mode (and your VM in OVMF mode)

[Hankanman]

okay great i'll try all that, how to i stub the card?

[testdasi]

15 minutes ago, Hankanman said:

how to i stub the card?

Using vfio-pci.ids. I think SIO showed it in one of his videos.

I basically stub all my devices that need to be passed through.

Found a post which he showed an example of what it should look like.

 

[Hankanman]

Awesome! Got it all sorted now, I had to install the drivers via VNC with the 2080 as the 2nd graphics card, but once the driver was in it all came up on the display, also worth noting i no longer need the vbios there, stubbing and no Hyper-V seemed to have done the trick for running in the second PCI slot.

 

Tried with the fixes back in the first PCI slot, looking to get that working as I want to leverage the x16 speed for the new GPU any thoughts?

Edited August 1, 2019 by Hankanman

[Hankanman]

As an update, using the following hardware:

MSI MEG X570 ACE

Ryzen 3800X

MSI Gaming X Trio RTX 2080 SUPER (PCI Slot 1)

ASUS Strix GTX 970 (PCI Slot 2)

 

Steps so far:

1. Using Dumped vBIOS from techpowerup
2. Modified the vBIOS per Space Invader One's tutorials: 
3. Assigned the card, audio device and usb elements of the GPU using both the stub-pci.ids and vfiopci.ids methods
4. Added <ioapic driver='kvm'/> to the VM XML per the bug noted here: https://bugs.launchpad.net/qemu/+bug/1826422 found when using vfio-pci.ids (which results in the host hanging:
5. Tried both the Q35 and i440fx chipsets (the latest of and a couple prior versions for each)
6. PCIe ACS override set to Both
7. VFIO allow unsafe interrupts is Enabled
8. Finally updated to Unraid 6.7.3-rc2

If in the second pci slot there's no problem, with a GTX 970 in the primary. But i want to run the Ssuper at full x16 speed.

With the config and steps listed I get a screen output at 800x600 (only when using the modified vBIOS) and i am unable to install the drivers in Windows 10 (Also booted outside of Unraid and pre-installed the drivers, but they take no effect in the VM)

 

I am also getting the following in the VM log:

2019-08-01T17:40:48.826299Z qemu-system-x86_64: -device vfio-pci,host=2d:00.0,id=hostdev0,bus=pci.4,addr=0x0: Failed to mmap 0000:2d:00.0 BAR 3. Performance may be slow

and:

2019-08-01T17:41:02.439023Z qemu-system-x86_64: vfio_region_write(0000:2d:00.0:region3+0x142a0, 0x47ad010d,8) failed: Device or resource busy

Also FYI I am booting the VM directly from the NVMe drive, with no vdisks.

 

VM XML:

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>Avenger</name>
  <uuid>ecc6bdbc-7fa7-b22f-f907-af4c77e81c8f</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>10</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='3'/>
    <vcpupin vcpu='1' cpuset='11'/>
    <vcpupin vcpu='2' cpuset='4'/>
    <vcpupin vcpu='3' cpuset='12'/>
    <vcpupin vcpu='4' cpuset='5'/>
    <vcpupin vcpu='5' cpuset='13'/>
    <vcpupin vcpu='6' cpuset='6'/>
    <vcpupin vcpu='7' cpuset='14'/>
    <vcpupin vcpu='8' cpuset='7'/>
    <vcpupin vcpu='9' cpuset='15'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-3.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/ecc6bdbc-7fa7-b22f-f907-af4c77e81c8f_VARS-pure-efi.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <ioapic driver='kvm'/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='10' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.160-1.iso'/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0xa'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0xb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xc'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xd'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0xe'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0xf'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x7'/>
    </controller>
    <controller type='pci' index='9' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='9' port='0x10'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='10' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='10' port='0x11'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='11' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='11' port='0x12'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='12' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <interface type='bridge'>
      <mac address='52:54:00:00:93:39'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x2d' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x2d' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x24' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x28' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x2d' slot='0x00' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x2d' slot='0x00' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x0a' slot='0x00' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>

 

[jonp]

Hi there,

 

To be perfectly honest, I don't think you're going to see any meaningful performance differences between an x8 and x16 slot.  Reference:  https://www.gamersnexus.net/guides/2488-pci-e-3-x8-vs-x16-performance-impact-on-gpus

 

That said, if you're dead set on wanting this, the best thing to do is to purchase hardware that has a built-in graphics device (integrated GPU).  This will then act as the primary graphics device for Unraid OS, allowing your primary add-on GPU to be passed through to a VM without any issues.

 

If you want this to work on the hardware you already have, we at Lime Tech can't really provide direct support for that.  All motherboard/cpu/gpu combinations can have some unique aspects that we can't reproduce unless we have the same gear.  The best thing you could do is to post a bug report to Linux:  https://www.kernel.org/doc/html/latest/admin-guide/reporting-bugs.html.

[Hankanman]

I've had another crack at it and sorted! It was because I was booting Unraid in UEFI mode, switched to legacy and it's all good. When installing initially it just made sense to check the UEFI box because I'm running modern hardware. Only when I found a post somewhere about booting with UEFI causing issues that I tried it. There should be a big warning on that checkbox