Nvidia GPU Passthrough to VM Black Screen

mbaut

By mbaut
in VM Engine (KVM)

Recommended Posts

mbaut Noob

mbaut

Posted
Posted (edited)

Hello, I am having difficulty passing through my GTX 1060 to a Windows Virtual Machine. I have tried just about everything many times on 2 different systems. Spaceinvader One has been a huge help to get me this far. I know I am overlooking or missing something, I just can't figure it out. Any suggestions would be appreciated.

 

I have been successful in passing through an AMD GPU, so I know my bios setting are correct for virtualization.

 

I can set VNC as primary GPU and my GTX 1060 as secondary, which I can access through VNC remote. Doing it that way, I can see my 1060 showing up in windows as disabled (code 43). The only way  can get the VM to start is to use the unedited VBios (hence the error code) and access it though VNC. If I use the edited Vbios, and/or don't use VNC, I just get a black screen no matter what I do.

 

I used this guide to get my Vbios. I also used the more recent method of extracting vbios through the script Spaceinvader wrote, but that method always ends in an error for me.

 

My theory is that I am not setting up the NVIDIA GPU correctly regarding the IOMMU devices with multifunction, or using ACS override. I say that because I have yet to get any VM display output directly from my 1060, whether I use the edited or unedited vbios. My native IOMMU groups are good enough (I think). My Video controller and Audio share the same IOMMU group, so I separated them with ACS and enabled multi function.

 

I have attached my VM XML code if that can help.231900519_GPUIOMMUGroups.png.0651575a4beb2d3e9b5e5a079f480126.png 

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>i440fx OVMF</name>
  <uuid>93eac77d-9943-0499-c639-4f0680b81100</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='8'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='10'/>
    <vcpupin vcpu='4' cpuset='4'/>
    <vcpupin vcpu='5' cpuset='12'/>
    <vcpupin vcpu='6' cpuset='6'/>
    <vcpupin vcpu='7' cpuset='14'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-5.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/93eac77d-9943-0499-c639-4f0680b81100_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='4' threads='2'/>
    <cache mode='passthrough'/>
    <feature policy='require' name='topoext'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/i440fx OVMF/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/Windows.iso'/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.190-1.iso'/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:87:a3:5e'/>
      <source bridge='br0'/>
      <model type='virtio-net'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0e' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/isos/vbios/Edited Extracted.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0e' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc07d'/>
      </source>
      <address type='usb' bus='0' port='2'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc336'/>
      </source>
      <address type='usb' bus='0' port='3'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>

 

Edited by mbaut
Change Title
  • Like 1
Link to comment
  • mbaut changed the title to Nvidia GPU Passthrough to VM Black Screen
bhootz Noob

bhootz

Posted
Posted (edited)

I'm able to passthrough windows 10 on my Quadro P2200 with 'multifunction', but I have issues with Asus Rog Strix 3080ti. 

Windows seems to be the problem for this. I am able to passthrough 3080ti on Linux distros with just the 'multifunction' and reassigning the bus and function.

 

For 3080ti passthrough on windows I have failed using

• multifunction

• Vbios extract through script (spaceinvaderone) gave me a 200kb file and no joy

• nano go to add code (I think it was to unbind the gfx card)

• Vbios hex edit after downloading the vbios from techpowerup. Spaceinvaderone says you can use a similar bios and I couldn't find the OC bios, so I downloaded the LC bios of the same card. It's a 2000kb file (seems very big) and I could find the text 'video' twice when I searched on the hex editor.

Removing the text above the first instance gave me a file of 1800kb

Removing text above second instance gave me a file of about 1000kb

Preserving text of the file from first header to second header gave me a file of about 900kb

All of these don't work and look bigger than the vbios extracted from the card via script.

Any further help would be appreciated.

Edited by bhootz
Link to comment
  • 4 months later...
Doublemyst Apprentice

Doublemyst

Posted
Posted (edited)
On 8/29/2021 at 12:39 AM, mbaut said:

Hello, I am having difficulty passing through my GTX 1060 to a Windows Virtual Machine. I have tried just about everything many times on 2 different systems. Spaceinvader One has been a huge help to get me this far. I know I am overlooking or missing something, I just can't figure it out. Any suggestions would be appreciated.

 

I have been successful in passing through an AMD GPU, so I know my bios setting are correct for virtualization.

 

I can set VNC as primary GPU and my GTX 1060 as secondary, which I can access through VNC remote. Doing it that way, I can see my 1060 showing up in windows as disabled (code 43). The only way  can get the VM to start is to use the unedited VBios (hence the error code) and access it though VNC. If I use the edited Vbios, and/or don't use VNC, I just get a black screen no matter what I do.

 

I used this guide to get my Vbios. I also used the more recent method of extracting vbios through the script Spaceinvader wrote, but that method always ends in an error for me.

 

My theory is that I am not setting up the NVIDIA GPU correctly regarding the IOMMU devices with multifunction, or using ACS override. I say that because I have yet to get any VM display output directly from my 1060, whether I use the edited or unedited vbios. My native IOMMU groups are good enough (I think). My Video controller and Audio share the same IOMMU group, so I separated them with ACS and enabled multi function.

 

I have attached my VM XML code if that can help.231900519_GPUIOMMUGroups.png.0651575a4beb2d3e9b5e5a079f480126.png 

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>i440fx OVMF</name>
  <uuid>93eac77d-9943-0499-c639-4f0680b81100</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>2097152</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='8'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='10'/>
    <vcpupin vcpu='4' cpuset='4'/>
    <vcpupin vcpu='5' cpuset='12'/>
    <vcpupin vcpu='6' cpuset='6'/>
    <vcpupin vcpu='7' cpuset='14'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-5.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/93eac77d-9943-0499-c639-4f0680b81100_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='4' threads='2'/>
    <cache mode='passthrough'/>
    <feature policy='require' name='topoext'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/i440fx OVMF/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/Windows.iso'/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.190-1.iso'/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:87:a3:5e'/>
      <source bridge='br0'/>
      <model type='virtio-net'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0e' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/isos/vbios/Edited Extracted.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0e' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc07d'/>
      </source>
      <address type='usb' bus='0' port='2'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc336'/>
      </source>
      <address type='usb' bus='0' port='3'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>

 

 

Did you manage to find a solution? I have the same problems with my Nvidia GTX 1060 6GB. Here is a reddit thread (https://www.reddit.com/r/unRAID/comments/myup6h/gpu_passthrough_vm_no_display/) with a good explanation of all points "normally" needed to be done, but it didn't help in my case. I have tried to boot unraid in legacy and UEFI mode, but with no luck .. I am out of ideas right now. At the boot there is a screen output (right before unraid starts booting, than it hangs on the windows until VM autostart), but as soon as the VM starts, the screens turn black and goes into "sleep" mode.

 

Edited by Doublemyst
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
Go to topic listing