Windows 10 VM GPU passthrough with GTX 1060 returning code 43

[kwilinsi]

Motherboard: Z390-A Pro

CPU: Intel Core i7-9700K

GPU: EVGA GeForce GTX 1060 3GB

 

Background

 

I'm helping a friend set up a fresh Unraid install with a Windows 10 VM as a daily driver. He's using the beta Unraid 6.9 because of the ability to create pools without depending on plugins. I've been following a host of SpaceInvader One tutorials and we set up the virtual machine completely through VNC. However, when we switched to the graphics card we couldn't get an image on the screen. It turns out that the vbios from techpowerup was the problem. We booted with VNC as the main graphics card and the GPU as the secondary one (with no vbios provided), and from there we were able to install GPU-Z and dump a vbios manually. After removing the header with help from SpaceInvader One, we successfully got the computer to boot with the GPU as the main and only graphics card.

 

However, at this point the screen resolution was incorrect and we only had an image on 1 of 4 monitors. There were no errors in device manager, so we installed an Nvidia driver (without GeForce Experience) and suddenly we had a Code 43 error in device manager. As I understand it, this means that Nvidia knows we're running in a VM and doesn't like that.

 

Things we've tried

 

We tried changing random settings in the VM and recreating it a few times to no avail. After deciding to write this post, we carefully created two VMs and logged the configuration here. 

 

We currently have two virtual machines that are almost identical except one is on Q35 and one is on i440fx machine. They both have

     • All 8 cpu threads

     • 24 gb ram on Q35 and 16 gb of ram on 1440fx

     • OVMF bios

     • No Hyper-V (since I've seen that mentioned in a number of forum posts)

     • USB 2.0 controller

     • The same install ISO and VirtIO driver

     • The ISO and VirtIO driver busses are SATA for the Q35 machine and IDE for the 1440fx machine. These were the default options and I didn't change them because I don't know what they mean.

     • The Q35 has a 150 gb vdisk and the other (due to a typo) only has 100 gb

     • And were initially set up with VNC

     • And after initial setup, we turned them off and changed the graphics to the GPU with vbios and nothing else. Before booting we also edited the XML to follow SpaceInvader One's suggestion of putting the video and sound part of the gpu on the same virtual bus slot.

     • The i440fx machine has a keyboard and mouse that were attached after a successful boot to allow us to sign in and install the NVidia driver

 

If we try booting the Q35 machine nothing happens on the screen and we have to force stop.

 

We were able to successfully boot the i440fx machine with no errors in device manager. Then we ran an Nvidia game ready driver for the GPU, and after it finished installing we had a code 43 error in device manager.

 

Other random info

• Unraid is booting in UEFI mode (is this changeable without losing Unraid config and all that? Is it worth trying to change? Does the motherboard even support non UEFI?)
• IOMMU groups are enabled.
• Due to not knowing much about bios configurations, we haven't touched anything aside from VT-d. We did find the Motherboard/BIOS manual though.
• According to the XML the GPU is physically plugged into domain 0x0000, bus 0x01, slot 0x00 on the motherboard and virtually plugged in to bus 0x00, slot 0x05, and function 0x00 for the video and 0x01 for the audio. This is the same on both VMs.

 

Logs

 

In the i440fx machine we have a whole bunch of nearly identical log entries (each with a different region). Not sure exactly when they appeared though in the boot process or if they happened at the same time as installing the Nvidia driver.

2021-01-08T19:35:25.371451Z qemu-system-x86_64: vfio_region_write(0000:01:00.0:region3+0x16f2f, 0x0,1) failed: Device or resource busy

In the Q35 machine we don't get an error log when it boots despite not getting a picture on the screen. However there is one error message (but I'm not sure if the timestamp is from during a VNC boot or a GPU boot)

2021-01-08T19:07:20.362580Z qemu-system-x86_64: -device vfio-pci,host=0000:01:00.0,id=hostdev0,bus=pci.4,multifunction=on,addr=0x0,romfile=/mnt/cache/isos/gtx1060.rom: Failed to mmap 0000:01:00.0 BAR 3. Performance may be slow

 

Edited January 8, 2021 by kwilinsi
Typo

[Simon2]

I ended up having to upgrade my GPU and I chose a 3090. Sorry mate, but your wallet wont be unraided...

[kwilinsi]

Finally managed to get it to work by (mostly) following this guide.

 

First, unrelated to the guide, we reverted from UEFI to Legacy boot. Went to the Unraid GUI Main tab, clicked on "Flash" in the list of drives, scrolled to the bottom, and unchecked the UEFI checkbox thing. We also connected to the flash drive share from another computer and copied all the contents as a backup, and after that renamed the "EFI-" folder to just "EFI" (simply removing the hyphen). Then we shut down Unraid, went to BIOS, and using the manual found every setting related to UEFI and set it to the Legacy option. Then we boot Unraid and checked to make sure it had boot in Legacy mode (by going to the Main page, clicking "Flash", and scrolling to the bottom).

 

Now we made a new virtual machine and went through the configuration, which I'll copy with slight modification from the guide linked earlier.

 

Machine: latest Q35 option

BIOS: OVMF

Same ISO and VirtIO ISO we've clicked maybe forty times today

Used 100G as disk size. VNC graphics with no sound card.

Unlike the tutorial we were following (but as a way of avoiding using RDP since we don't have Windows 10 Pro) we enabled every single USB device we planned on using in the final setup.

Unchecked "Start VM after creation" and created it.

 

Then edited the VM and went to XML. Under the <apic/> tag added

<ioapic driver='kvm'/>

Left hyperv just like it was except we changed the vender_id value to `1234567890ab` as in the tutorial. Also added a <kvm> tag under it such that all the changes together looked like

<ioapic driver='kvm'/>
<hyperv>
  <relaxed state='on'/>
  <vapic state='on'/>
  <spinlocks state='on' retries='8191'/>
  <vendor_id state='on' value='1234567890ab'/>
</hyperv>
<kvm>
  <hidden state='on'/>
</kvm>

Then at the bottom of the XML before the closing </domain> tag but after the </devices> tag we added this XML:

<qemu:commandline> 
  <qemu:arg value='-cpu'/> 
  <qemu:arg value='host,kvm=off,hv_vendor_id=null'/>
  <qemu:arg value='-machine'/> 
  <qemu:arg value='q35,kernel_irqchip=on'/> 
</qemu:commandline>

There was alternative XML here but it seemed to be for ryzen and we have an Intel CPU and GTX 1060 GPU.

 

Clicked update, started the VM, and quickly opened VNC and pressed a key. I assume you're familiar with SpaceInvader One's tutorial on configuring a Windows 10 VM if you're reading this, and if not you should follow his video first.

 

Then we selected a Windows 10 version (Home in our case) and went through the self explanatory setup. The one thing different than usual was after selecting the drive we clicked browse again and went to NetKVM>win10>amd64 to get the network ethernet adapter. Normally this was done within Windows in the Device Manager, but we wanted to follow the source tutorial closely.

 

Went through the rest of setup. On the Windows 10 Desktop, we opened Device Manager and fixed the two devices without drivers. Unlike SpaceInvader One, we did NOT fix the display driver.

 

At this point we skipped all the parts of the tutorial regarding RDP because this is Windows 10 Home. We did install Teamviewer as an alterative, but it proved unnecessary later since we had enabled the usb keyboard and mouse.

 

We shut down the VM, went into edit mode, and added the GPU and associated vbios, along with the audio controller. Clicked apply, and then edited the VM again in XML mode. We made all the same changes as before (interestingly the kvm part was still there), and additionally removed all the <graphics> and <video> tags related to VNC. We also followed SpaceInvader One's advanced GPU passthrough tutorial and adjusted the bus and slot of the audio controller to match the graphics card.

 

We saved the XML and started the VM, and magically everything was working. At this point the tutorial we were following said to install the gpu driver 399.24, but we didn't do that. Or well, we started to, but magically about 2 minutes into the download everything just suddenly worked. The screen resolution fixed and all the other monitors started up. It worked! I still have absolutely no idea why since we didn't change any drivers, but it just worked. I'd suggest waiting like 1 to 10 minutes, who knows.

 

We messed something up in the XML later and repeated this entire process, and sure enough the same thing happened again. After two minutes the display worked. Note that we never installed any drivers from Nvidia; it just worked. Went from code 43 to no error at all.

 

If you repeat this, make sure that you never ever boot your VM without the XML edits made. Every time you change something in the regular form view, you need to click "Update" and then go back into the edit page of the VM in XML mode and confirm that the changes are still there. In my experience everything around the kvm section works fine, but the gpu and soundcard and the qemu part at the end get reset.

[jlficken]

Having the virtio-win-0.1.185.iso assigned to the machine is absolutely vital I discovered as I was just using virtio-win-0.1.171.iso and couldn't get it to work with my Inno3D 1050 Ti.  I changed over to the latest version and it started working immediately.

[Doublemyst]

On 1/9/2021 at 5:05 AM, kwilinsi said:

Finally managed to get it to work by (mostly) following this guide.

 

First, unrelated to the guide, we reverted from UEFI to Legacy boot. Went to the Unraid GUI Main tab, clicked on "Flash" in the list of drives, scrolled to the bottom, and unchecked the UEFI checkbox thing. We also connected to the flash drive share from another computer and copied all the contents as a backup, and after that renamed the "EFI-" folder to just "EFI" (simply removing the hyphen). Then we shut down Unraid, went to BIOS, and using the manual found every setting related to UEFI and set it to the Legacy option. Then we boot Unraid and checked to make sure it had boot in Legacy mode (by going to the Main page, clicking "Flash", and scrolling to the bottom).

 

Now we made a new virtual machine and went through the configuration, which I'll copy with slight modification from the guide linked earlier.

 

Machine: latest Q35 option

BIOS: OVMF

Same ISO and VirtIO ISO we've clicked maybe forty times today

Used 100G as disk size. VNC graphics with no sound card.

Unlike the tutorial we were following (but as a way of avoiding using RDP since we don't have Windows 10 Pro) we enabled every single USB device we planned on using in the final setup.

Unchecked "Start VM after creation" and created it.

 

Then edited the VM and went to XML. Under the <apic/> tag added

<ioapic driver='kvm'/>

Left hyperv just like it was except we changed the vender_id value to `1234567890ab` as in the tutorial. Also added a <kvm> tag under it such that all the changes together looked like

<ioapic driver='kvm'/>
<hyperv>
  <relaxed state='on'/>
  <vapic state='on'/>
  <spinlocks state='on' retries='8191'/>
  <vendor_id state='on' value='1234567890ab'/>
</hyperv>
<kvm>
  <hidden state='on'/>
</kvm>

Then at the bottom of the XML before the closing </domain> tag but after the </devices> tag we added this XML:

<qemu:commandline> 
  <qemu:arg value='-cpu'/> 
  <qemu:arg value='host,kvm=off,hv_vendor_id=null'/>
  <qemu:arg value='-machine'/> 
  <qemu:arg value='q35,kernel_irqchip=on'/> 
</qemu:commandline>

There was alternative XML here but it seemed to be for ryzen and we have an Intel CPU and GTX 1060 GPU.

 

Clicked update, started the VM, and quickly opened VNC and pressed a key. I assume you're familiar with SpaceInvader One's tutorial on configuring a Windows 10 VM if you're reading this, and if not you should follow his video first.

 

Then we selected a Windows 10 version (Home in our case) and went through the self explanatory setup. The one thing different than usual was after selecting the drive we clicked browse again and went to NetKVM>win10>amd64 to get the network ethernet adapter. Normally this was done within Windows in the Device Manager, but we wanted to follow the source tutorial closely.

 

Went through the rest of setup. On the Windows 10 Desktop, we opened Device Manager and fixed the two devices without drivers. Unlike SpaceInvader One, we did NOT fix the display driver.

 

At this point we skipped all the parts of the tutorial regarding RDP because this is Windows 10 Home. We did install Teamviewer as an alterative, but it proved unnecessary later since we had enabled the usb keyboard and mouse.

 

We shut down the VM, went into edit mode, and added the GPU and associated vbios, along with the audio controller. Clicked apply, and then edited the VM again in XML mode. We made all the same changes as before (interestingly the kvm part was still there), and additionally removed all the <graphics> and <video> tags related to VNC. We also followed SpaceInvader One's advanced GPU passthrough tutorial and adjusted the bus and slot of the audio controller to match the graphics card.

 

We saved the XML and started the VM, and magically everything was working. At this point the tutorial we were following said to install the gpu driver 399.24, but we didn't do that. Or well, we started to, but magically about 2 minutes into the download everything just suddenly worked. The screen resolution fixed and all the other monitors started up. It worked! I still have absolutely no idea why since we didn't change any drivers, but it just worked. I'd suggest waiting like 1 to 10 minutes, who knows.

 

We messed something up in the XML later and repeated this entire process, and sure enough the same thing happened again. After two minutes the display worked. Note that we never installed any drivers from Nvidia; it just worked. Went from code 43 to no error at all.

 

If you repeat this, make sure that you never ever boot your VM without the XML edits made. Every time you change something in the regular form view, you need to click "Update" and then go back into the edit page of the VM in XML mode and confirm that the changes are still there. In my experience everything around the kvm section works fine, but the gpu and soundcard and the qemu part at the end get reset.

 

Thank you! It finally works! I was on the edge of giving up!

[ghost82]

Quote

2021-01-08T19:35:25.371451Z qemu-system-x86_64: vfio_region_write(0000:01:00.0:region3+0x16f2f, 0x0,1) failed: Device or resource busy

 

On 1/9/2021 at 5:05 AM, kwilinsi said:

we reverted from UEFI to Legacy boot

 

On 1/9/2021 at 5:05 AM, kwilinsi said:

and magically everything was working

 

I'm late to the party!

Sorry but that tutorial has nothing to do with the op issue.

What fixed the passthrough was booting unraid in legacy mode.

With uefi boot unraid was attaching efifb to the gpu, so the error about device or resource busy.

Simply disabling efifb should had fixed the gpu passthrough, without disabling uefi.

May be of some interest for others reading this, even if this info is spread all over the forum...