Nvidia GPU Passthrough to VM


Recommended Posts

Hello everyone, I'm a new unraid user as of April.  I have unix experience from the past, so the shell is not new to me.  I am having problems passing through my Nvidia GTX970 to any of the VMs I have created so far.  Using the Nvidia card as the second GPU in my VM settings allows me to see it in the device manager as an Nvidia GTX 970, but with the error code 43.  Drivers will also not install for the Nvidia GPU.  I have come to understand that this is due to Nvidia detecting that the host is virtualized and disabling the card.  I'll post my current Windows 10 XML file at the bottom of this post, but first I will explain what I have tried so far.

 

  • Setting the Nvidia card to be the secondary graphics card, with a Radeon card as the primary.  I can boot to unraid and get video output from VMs with the radeon card, so having only one gpu is not my problem.
  • Dumping the bios and adding it to the XML.  I have also run the bios file(s) through a rom parser that was suggested elsewhere on the internet just to make sure they seemed valid.
  • Using both SeaBios and OVMF bios settings for my VMs.  Note here that I have windows fully installed on my SeaBios vm, but when I tested OVMF I only attempted to boot to the installation screen and, upon seeing no display output, deleted the VM and moved on to the next suggestion.  If fully installing Windows through VNC would make any difference, please let me know and I will test it, but it seems like OVMF is the stricter of the two BIOS types when it comes to GPU passthrough so I feel there is less of a chance of it working with OVMF than with SeaBios.
  • Disabling hypervisor in the advanced VM settings, as well as trying to manually change all the hyperv settings to off in the bios.
  • Adding the line 
    <feature policy='disable' name='hypervisor'/>

    into my XML within the cpu tag, right under the topology line.  This resulted in my windows 10 vm crashing to a grey screen before getting to the login screen.  At first the display was flickering, so I thought maybe it was detecting the multiple graphics cards, but there was still no output from the nvidia card and the cursor was frozen in place on the radeon.  There was nothing new at the bottom of the VM log file compared to when I run the VM without that line.

  • Changing pcie ports for both graphics cards, as well as ensuring that I tried the Nvidia gpu in pcie ports that were controlled by each of the two CPUs.

  • Making sure that all of the virtualization settings are enabled in the host bios.

  • I tried changing the vendor id of the GPU through the VM's XML.  I just changed it to "testvendor" because I didn't read that there was any specific rule you have to follow when coming up with a new id.

  • My host motherboard's bios has an option for addressing pcie memory greater than 4GB.  Because the Nvidia GPU has 4GB of memory alone, and my Radeon GPU is set as the primary, then it seems to me that I would need to enable this for it to be able to address all the memory of both cards.  No change after enabling.

 

 

Here is my other hardware, in case it may be relevant.

  • Dual socket Dell Precision T7610 motherboard.  Relatively unused in custom builds, and required me to make some of my own power supply adapters because of their proprietary power supply, but the motherboard was the cheapest dual socket board I could find and it is a beast.
  • Two Intel Xeon E5-2670 cpus.
  • 32GB of ddr3 ecc memory, split up with 16GB going into the lanes controlled by each cpu.
  • 2 SSDs, 5 other standard hard drives, and two disc drives running over sata.
  • wifi card, usb hub, sata hub, the Nvidia GTX 970, and the Radeon GPU installed to pcie ports.

 

 

Here is my current XML for the Windows 10 vm.  I inserted some comments in the relevant areas that aren't there in the actual xml.  This is a setup with what I gather should be all the correct settings I need, but results in error code 43 when looking at the nvidia gpu in the device manager.

<domain type='kvm'>
  <name>Windows 10</name>
  <uuid>c3bb1f68-dac8-9d06-dbbd-636f46d2272b</uuid>
  <metadata>
    <!--My template is windows 7 because at the time I created my windows 10 vm, there was no windows 10 template and the windows 7 template was recommended online.-->
    <vmtemplate xmlns="unraid" name="Windows 7" icon="windows.png" os="windows7"/>
  </metadata>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='8'/>
    <vcpupin vcpu='1' cpuset='9'/>
    <vcpupin vcpu='2' cpuset='10'/>
    <vcpupin vcpu='3' cpuset='11'/>
    <vcpupin vcpu='4' cpuset='12'/>
    <vcpupin vcpu='5' cpuset='13'/>
    <vcpupin vcpu='6' cpuset='14'/>
    <vcpupin vcpu='7' cpuset='15'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.7'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='4' threads='2'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Windows 10/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.118-2.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:52:94:3a'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <!--The Radeon GPU.-->
    <hostdev mode='subsystem' type='pci' managed='yes' xvga='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x81' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <!--The Nvidia GPU.-->
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <rom file='/path/to/my/bios.file'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x058f'/>
        <product id='0x6387'/>
      </source>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x1532'/>
        <product id='0x0040'/>
      </source>
      <address type='usb' bus='0' port='2'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x1b1c'/>
        <product id='0x1b15'/>
      </source>
      <address type='usb' bus='0' port='3'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </memballoon>
  </devices>
</domain>

 

Link to comment
28 minutes ago, 1812 said:

did you search the forum for error code 43? Seems to pop up a few times.

 

Yes; I've actually had the issue for about a month now and all the bullet points i listed above were things I found on this site, reddit, and other various computer sites through google.

 

One thing i read on another non-unraid virtualization website was that you might get different results using the qemu command line arguments to pass through the graphics card, but i could never get them to boot or even find a definite guide on how to translate the xml arguments to the command line arguments. I dont know if that holds any stock since we're dealing with unraid here, but i figured i would add it.

Link to comment
12 hours ago, 1812 said:

ok. have you tried booting with just the nvida card? and if that works, then put the radeon as the second?

 

please post your diagnostics.

 

 

 

Yes, I initially only owned the Nvidia GPU, and when I couldn't get that working I went and bought a $20 radeon card just to get some video output to the physical monitors so I could stop using VNC, but also to try all of the methods that required two graphics cards, since my CPUs don't have integrated graphics.  When only the Nvidia GPU is installed in the system, it boots to the unraid console fine and displays that over HDMI to the monitor, but then there is the expected result of absolutely nothing happening when I try to assign that card to a VM, since it's the primary.  I have attached my diagnostics.

unraidbeast-diagnostics-20170606-0842.zip

Edited by cyleleghorn
Added more information about what happens when I try to boot with just the Nvidia GPU
Link to comment

Here is my XML from a working Windows 10 Seabios with Nivida GPU passthrough, I also had to take the bios dump etc.

 

First thing I noticed was the line where you are passing the vbios through, look to be in a different position compared to mine by about 1-2 lines. And I had to pass the vbios through for the onboard audio of the GPU before anything would display.

 

I want to add that this is not my desired configuration, I've also had issues getting OVMF to work but think it related to my windows installer. I'm going to try later with a fresh download from the MS site.

 

<domain type='kvm' id='5'>
  <name>Windows 10 Gaming</name>
  <uuid>9bf2954a-016b-0f03-9b62-5053056ec9f9</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='8'/>
    <vcpupin vcpu='1' cpuset='10'/>
    <vcpupin vcpu='2' cpuset='12'/>
    <vcpupin vcpu='3' cpuset='14'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.7'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='4' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/disks/WDC WDS500G1B0A/VM_Cache/Win10.img'/>
      <backingStore/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <alias name='virtio-disk2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/disks/WDC WDS500G1B0A/VM_Cache/WindowsGaming.img'/>
      <backingStore/>
      <target dev='hdd' bus='virtio'/>
      <alias name='virtio-disk3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/VM_Cache/virtio-win.iso'/>
      <backingStore/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <alias name='ide0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:91:a3:2e'/>
      <source bridge='br0'/>
      <target dev='vnet2'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/4'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/4'>
      <source path='/dev/pts/4'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-5-Windows 10 Gaming/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <hostdev mode='subsystem' type='pci' managed='yes' xvga='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0f' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <rom file='/boot/vbios.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0f' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <rom file='/boot/vbios.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x047f'/>
        <product id='0x02f7'/>
        <address bus='1' device='5'/>
      </source>
      <alias name='hostdev2'/>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x1532'/>
        <product id='0x0016'/>
        <address bus='1' device='3'/>
      </source>
      <alias name='hostdev3'/>
      <address type='usb' bus='0' port='2'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x1c4f'/>
        <product id='0x0002'/>
        <address bus='1' device='2'/>
      </source>
      <alias name='hostdev4'/>
      <address type='usb' bus='0' port='3'/>
    </hostdev>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='none' model='none'/>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+100</label>
    <imagelabel>+0:+100</imagelabel>
  </seclabel>
</domain>

 

Edited by Tuftuf
Link to comment

Try the following

 

Also Hyper-V was disabled via the UI prior to making these changes.

    <!--The Nvidia GPU.-->
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>

	<rom file='/path/to/my/bios.file'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
      </source>

	<rom file='/path/to/my/bios.file'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>

 

 

EDIT - It's worth trying my suggestion, but I have noticed that the Slot and Function lines appear differently on your example.

Are you passing through your GPU's audio?

Edited by Tuftuf
Link to comment

I also have been seeing very similar issues.  I have an Asus X99-M WS motherboard that doesn't allow me to specify primary GPU, also with my case configuration I can't put my GTX 960 in any other slot but the first one.  However for testing I have disassembled my case and put a secondary card in and my VM can use my 960 just fine, but when I put it back in slot one by itself unraid uses when booting and when I start the VM the monitor goes black but nothing after that.  I have followed space invader one's video and all the tips on this post, but still nothing.  Any other ideas?

 

Here is my XML: 

 

<domain type='kvm' id='5'>
  <name>Corey-PC</name>
  <uuid>4de2c82c-7133-6f85-e059-b0687f3ef342</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='3'/>
    <vcpupin vcpu='4' cpuset='14'/>
    <vcpupin vcpu='5' cpuset='15'/>
    <vcpupin vcpu='6' cpuset='16'/>
    <vcpupin vcpu='7' cpuset='17'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.7'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/4de2c82c-7133-6f85-e059-b0687f3ef342_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='4' threads='2'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source dev='/dev/disk/by-id/nvme-Samsung_SSD_950_PRO_512GB_S2GMNCAGA06825F'/>
      <backingStore/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <alias name='virtio-disk2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:a6:af:9e'/>
      <source bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/2'/>
      <target port='0'/>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/2'>
      <source path='/dev/pts/2'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-5-Corey-PC/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <rom file='/mnt/user/kvm/960_vbios.dump'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x041e'/>
        <product id='0x3225'/>
        <address bus='3' device='6'/>
      </source>
      <alias name='hostdev2'/>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc32b'/>
        <address bus='3' device='7'/>
      </source>
      <alias name='hostdev3'/>
      <address type='usb' bus='0' port='2'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc52b'/>
        <address bus='3' device='4'/>
      </source>
      <alias name='hostdev4'/>
      <address type='usb' bus='0' port='3'/>
    </hostdev>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='none' model='none'/>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+100</label>
    <imagelabel>+0:+100</imagelabel>
  </seclabel>
</domain>


 

Edited by zyurph
Link to comment

Same suggestion I made to the OP

 

Try adding the VBIOS to the Audio on the GPU that is passed through as well, make sure you are passing it through but based on your XML you are.

 

Here is an example!

 

 <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <rom file='/mnt/user/kvm/960_vbios.dump'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
      </source>
      <alias name='hostdev1'/>

      <rom file='/mnt/user/kvm/960_vbios.dump'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsys

 

  • Upvote 1
Link to comment

here is an update, I followed Spaceinvader one's other video and edited a vbios rom in a hex editor from techpowerup.  I was abale to get my VM booted, but as soon as I see the win10 logon screen the unraid server crashes hard and I have to flip the Psy swtich or hold the power button.  I did run this "echo "0000:02:00.0"  /sys/bus/pci/drivers/vfio-pci/unbind" and it came back, so I assume that unraid has the card bound still wich is why it is crashing when the VM takes it.  So I ran this "echo "0000:02:00.0" > /sys/bus/pci/drivers/vfio-pci/bind" and it echoed back, but still the same result unraid crashes after the VM boots into the logon screen and not before that point.  

 

Thoughts? 

Link to comment

I don't know for sure but I don't think you can boot like this with a different vbios. The only proof I have of this is I recently upgraded my cards vbios to support UEFI and afterwards tried to boot my old Seabios VM with the old GPU's bios and it no longer booted. I didn't test very much so its possible it could of worked.  -- I've not seen this other video, so need to take a look at that to get up to speed.

 

I take it you are still passing the GPU's audio? I've not been able to boot without passing it, I'll do some tests to see if that still holds true.

 

What file size was the bios you dumped? The first time I attempted to dump the vbios I ended up with a empty file.

 

EDIT - It made it to the logon screen?? Are you not trying to install a fresh copy of windows? Sometimes you need to run the install with only 1 CPU core.

Edited by Tuftuf
  • Upvote 1
Link to comment

I would take a step back and get the windows installer booting. You can add things once you have a running OS.

There is no need for the intel audio until the machine is running.

 

If you are trying to boot a pre-made windows image, that may not work due to differences between the VM and the original system.

 

EDIT - Reread your post and see the VM booted and I assume things worked correctly while in the second slot. Which makes much of what I've written above invalid but still good troubleshooting. I had different issues as I switched from Legacy to UEFI as well.

 

I have had a couple of black screen boots where I needed to power the VM off and on again. Also with UEFI booting I've noticed sometimes I don't see the bios boot screen the screen can stay black until the windows login prompt. Are you sure you waited long enough on the original bios?

 

Edited by Tuftuf
  • Upvote 1
Link to comment
1 minute ago, zyurph said:

I'll do that and see if it makes a difference, but I did install windows fresh on this VM with a second GPU, but can't leave it like that for I need to add the rest of my drives.  

 

Windows fresh from the second slot should of been fine, I started to think you had a old windows image or converted one you were trying to migrate.

 

I'm not sure of a fix. I would test things such as 

One core with old bios and current windows. Did you wait long enough for it to reach the login screen, sometimes mines acts strangely on boot.

One core with new bios and current windows. Is it more stable?

I've not looked at the 960 but I assume both were UEFI bioses. - Did you get the same bios version as your card is running or different?

 

I'll get some tests done for my own information on if my VM works with or without the second part passed through. Plus if I can pass my old bios to the card when I've flashed a new one as I think the two would need to match.

  • Upvote 1
Link to comment

So you don't think that fact that unraid has the card bound has anything to do with it then?  Also, it's doing a party check, so I think I'll let that finish before I make it crash again :).  

 

Will update again tonight or tomorrow. Thanks for all your help!

Edited by zyurph
Link to comment

I've had it running where I had a keyboard & mouse connected to Unraid and separate set connected to the VM. (this was using seabios not OVMF, not tested it with OVMF)

 

I had booted up in Unraid GUI mode and then launched the VM. If I moved the mouse Unraid it attempted to redraw the unraid UI over the top of windows while the VM.

It looked as like a squashed Unraid Full screen displayed 3 times over the top of windows. Like a old VGA monitor error or something freaky display issue.

 

If I stopped moving the mouse windows came back. I don't think its the issue you stated but thats only based what happened to me. 

 

  • Upvote 1
Link to comment

So I changed to a different vbios from techpowerup, and it's work now.  Not sure if it didn't like the one I was using for there were three to choose from, or if I just botched it up in the hex editor.  Either way, I'm just happy it's working.  Again thanks so much for your help Tuftuf!

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.