Hyper v and nvidea gpus


Recommended Posts

Hi . I am thinking of upgrading my amd r7 370 gpu to an nvidea gtx970. I am happy with my system and game performance. However i want to use an nvidea so i can stream games full screen to another pc/ tablet using splashtop desktop. My amd has to be windowed when streaming games but nvidea will stream full screen........This is all so my daughter can play sims 4 on her laptop!!!

But before I make my mind up i was hoping for some advise on the following.

 

1. I know that nvidea gpus require the disable hyper v setting. I thought the the hyper v helps with the speed of the vm.

So will using an nvidea card with disabled hyper v effect my vm in a negative way at all?

 

2. Also why do nvidea cards require this setting yet amd dont. I know this is because if the driver detects a hypervisor it gives error 43, but why do nvidea want to stop the cards being used in a vm. It cant be a bug else surely they would address this in a driver update

 

3. So Is the disabled hyper v only necessary for windows vms or does it apply to linux vms aswell?

 

thanks

 

 

 

Link to comment
  • 1 month later...

1) from what I can tell the hyperv extensions dont make any real difference. ive seen benchmarks that proved it (ill try to find link later), plus personal 'look and feel' ive noticed nothing. also my findings:

*hyperv relaxed - mostly exists to surpress some noisy logs from windows 'watchdog timer' service.

*vapic - virtual APIC controller. this flag just tells windows to manage its own interrupts instead of using the emulated hardware APIC. might matter for some virtual pci devices, but shouldnt actually affect passthrough devices.

*spinlocks - think this tells windows to favor spinlocks instead of kernel locks when it can. this probably matters most when you have several windows guests and it causes the hypervisor to context switch too often. shouldnt matter with only a few guests

*hypervclock - high performance virtual timer. It's good to have a high performance timer, but HPET works fine. I think this exists as an alternative to HPET which might not be as secure or scale as well or something.

 

A good thing to keep in mind is that the hyperv extensions enable a hpyervisor to: run more guests and run more securely without performance penalty. So i think for the most case it doesnt apply to the handful of gaming guests.

 

2) My theory is its because of the GRID SDK. NVIDIA does actually offer commercial support for proprietary virtualization solutions. This can be seen first hand by the 'floating point' cloud computing units available through services like AWS. They basically virtualize GPU access to virtual machines. From what i can tell there was ways to subdivide cards and do other fun stuff and you were exposed raw CUDA devices to the virtual machine. I'm guessing the hyperv extensions tip off the drivers that there's additional hypervisor functionality coming its way to tell it what to do and it never gets it and shuts down. Plus the virtual devices exposed to the guest might not even be real GPUs and may be some generic CUDA device. Some day I might try to to wander through the grid SDK and see whats up.

 

3) Only windows. There's generally specialized 'virtual' kernels for linux that are designed to be guests only and offer some of the same performance advantages listed above. I think hyper-v is just also the microsoft preferred flavor.

 

This is what i finally settled on for my nvidia guests:

 

  <features>

    <acpi/>

    <apic/>

    <hyperv>

      <relaxed state='off'/>

      <vapic state='off'/>

      <spinlocks state='off'/>

    </hyperv>

    <kvm>

      <hidden state='on'/>

    </kvm>

  </features>

 

  <clock offset='localtime'>

    <timer name='hypervclock' present='no'/>

    <timer name='rtc' present='no'/>

    <timer name='pit'  present='no'/>

    <timer name='hpet' present='yes'/>

  </clock>

 

EDIT: must make sure HPET is enabled in bios, x64 for x64

  • Like 1
Link to comment
  • 2 years later...
6 hours ago, jj_uk said:

The issue is back because i can't run an nvidia card with a VM that has hyper-v enabled, and i also cant disable it because after clicking "update" in the vm template, its just still enabled.

You can just edit the VM's XML directly, change this section:

<clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
</clock>

And just set it to 'no' instead...

 

That said, I have a GTX1070 passed in to a Win10 VM with the ROM file from the card passed in...  Hyper-V is working fine for me with Fully updated Windows and GeForce drivers version 419.17...  So this sounds like you need to check your system...

Edited by Warrentheo
  • Like 1
Link to comment
On 2/23/2019 at 11:24 PM, Warrentheo said:

You can just edit the VM's XML directly, change this section:


<clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
</clock>

And just set it to 'no' instead...

 

That said, I have a GTX1070 passed in to a Win10 VM with the ROM file from the card passed in...  Hyper-V is working fine for me with Fully updated Windows and GeForce drivers version 419.17...  So this sounds like you need to check your system...

This appears to still be an issue.

 

I am trying to pass through an RTX2060.  It worked on SeaBios, but there where CPU limitations with that.  Set it up with OMVF and the 2060 will not operate.  Get a Code 43, with hyper-v on or off.

 

This card works fine booted up in windows as well, so I know the graphics card is good.  Even ran a few benchmarks while i had it working in Seabios with no issue.

Link to comment

Most of the new RTX cards have 4 "Devices" in their IOMMU group, and so far my experience is that people don't know that they need to pass them all in for the card to work correctly...  This also means setting all those devices to use the vfio-pci driver so that UnRaid doesn't try and use them for itself...  You will also need to pass in the BIOS from the card manually on NVidia 10 series and above...

  • Upvote 1
Link to comment
7 hours ago, Warrentheo said:

Most of the new RTX cards have 4 "Devices" in their IOMMU group, and so far my experience is that people don't know that they need to pass them all in for the card to work correctly...  This also means setting all those devices to use the vfio-pci driver so that UnRaid doesn't try and use them for itself...  You will also need to pass in the BIOS from the card manually on NVidia 10 series and above...

I believe I have tried this already, by manually adding those PCI devices into the syslinux config.

 

I will try it again.

Link to comment
42 minutes ago, InfInIty said:

I believe I have tried this already, by manually adding those PCI devices into the syslinux config.

 

I will try it again.

syslinux just keeps them from being used by UnRaid, but that is not enough, if you assign any device from the IOMMU group, it is easiest to just assign them all (there are workarounds for some devices, but it is easiest to just add them all)...  This is why all the existing instructions for us GTX 10 series card users say to also pass the sound card as well...

 

So I assumed you added something like:

vfio-pci.ids=10de:1b81,10de:10f0

to your syslinux config, but with all 4 devices for you...

 

But even after everything has the vfio-pci driver from the UnRaid kernel, you still need to pass all 4 into the devices section of the VM...

 

P.S. In the new UnRaid 6.7.0-rc, the syslinux can be simplified to just the PCI id with something like:

BIND=01:00.0

Which assigns the vfio-pci driver for everything in that PCIe slot (all 4 devices in your case)... This has the big plus of letting you pick from 2 different identical cards, but has the major downside of not moving with the card when you modify what hardware on your motherboard is connected or turned on, so hardware changes need to be made carefully after this...

Link to comment
9 hours ago, Warrentheo said:

Most of the new RTX cards have 4 "Devices" in their IOMMU group, and so far my experience is that people don't know that they need to pass them all in for the card to work correctly...  This also means setting all those devices to use the vfio-pci driver so that UnRaid doesn't try and use them for itself...  You will also need to pass in the BIOS from the card manually on NVidia 10 series and above...

So i did not manually add the the VGA and the Sound device because those where already available to the VM.  I did however add the USB and Serial component of the GPU and still having the same issue.  I am going to go back and try adding all the devices to that line of syslinux to see if that makes a different.

 

Also I am not passing through the Bios ROM, which seems odd but I am not opposed to doing it, just not sure how.  I see space invader has a video on this so I am watching it now.

 

Attached are screen shots of how I have it setup right now.

 

 

Capture.PNG

Capture2.PNG

Capture3.PNG

Link to comment
14 hours ago, InfInIty said:

So I have added all 4 Devices into the syslinux file

Downloaded the bios and removed the nvidia header per the SpaceInvader video

Still same issue.  Code 43

If you have all the devices correctly passed, than this most likely means that the BIOS file is not correct...

Ever since the NVidia 10 series and the RTX series, NVidia started encrypting their BIOS and the connection between their drivers and their BIOS...

I have a GTX1070, and was able to use the SpaceInvader video to just dump my card with no other modification to get it to work...  The RTX cards may be slightly different, but from other reports I have seen, this just looks like the same sort of things that the GTX 10 series people have been having to deal with so I doubt it...

 

For me, when the BIOS was not correct, but I rebooted the host and auto-launched the Windows VM, it would still work during that boot, but then would not be able to reset the PCIe slot, so if I "rebooted" the VM, it would no longer talk to the card and error 43...  I suspect you are at this stage of the process...

 

Link to comment
4 hours ago, Warrentheo said:

If you have all the devices correctly passed, than this most likely means that the BIOS file is not correct...

Ever since the NVidia 10 series and the RTX series, NVidia started encrypting their BIOS and the connection between their drivers and their BIOS...

I have a GTX1070, and was able to use the SpaceInvader video to just dump my card with no other modification to get it to work...  The RTX cards may be slightly different, but from other reports I have seen, this just looks like the same sort of things that the GTX 10 series people have been having to deal with so I doubt it...

 

For me, when the BIOS was not correct, but I rebooted the host and auto-launched the Windows VM, it would still work during that boot, but then would not be able to reset the PCIe slot, so if I "rebooted" the VM, it would no longer talk to the card and error 43...  I suspect you are at this stage of the process...

 

So I watched his second video on this.

I downloaded the bios from tech power up, removed the "header" with a hex editor

I can try it with the full file and see if that improves things.

 

Should I have Hyper-v off or on, i am getting conflicting information on that one.

Link to comment
17 minutes ago, InfInIty said:

So I watched his second video on this.

I downloaded the bios from tech power up, removed the "header" with a hex editor

I can try it with the full file and see if that improves things.

 

Should I have Hyper-v off or on, i am getting conflicting information on that one.

In the past it was one of the things that triggered the NVidia driver to notice it was in a VM, and so had to be turned off...  That said I have had mine on for about a year with no issues on my GTX1070...  The RTX card may be different, but I doubt it... For troubleshooting, I would start with it off for now, and try turning it on when everything else is working...

Edited by Warrentheo
Link to comment

@InfInIty Just a hint from my side. For me the TechPowerUp BIOSes never worked for 3 different Nvidia cards (780ti, 1050ti, 1080ti). I always had to dump the BIOS myself. Can't find the thread but a couple days ago a user had the same issue and he dumped his own BIOS and guess what happened? Issue solved 😄

Link to comment
10 hours ago, bastl said:

@InfInIty Just a hint from my side. For me the TechPowerUp BIOSes never worked for 3 different Nvidia cards (780ti, 1050ti, 1080ti). I always had to dump the BIOS myself. Can't find the thread but a couple days ago a user had the same issue and he dumped his own BIOS and guess what happened? Issue solved 😄

To my knowledge, only GTX 10 series and above need bios passthrough...  It was not until the 10 series that they started encrypting the BIOS... That is why you can get a program like:

https://www.techpowerup.com/download/maxwell-ii-bios-tweaker/

or

https://www.techpowerup.com/download/kepler-bios-tweaker/

For the GTX 9 series and below, but will never see one again for the GTX 10 series and above...

 

This encryption is what is causing the issues with passthrough, and why BIOS's need to be passed through...

Link to comment

I can try dumping the bios myself.  Just for a quick update I want to provide as much info as I can and please anyone who can take a look and see if I am missing something.  The only thing not functioning is the GPU passthrough.  Everything else appears to be working as intended.

 

syslinux

default menu.c32
menu title Lime Technology, Inc.
prompt 0
timeout 50
label Unraid OS
  menu default
  kernel /bzimage
  append initrd=/bzroot vfio-pci.ids=10de:1ada,10de:1adb,10de:1f08,10de:10f9,1912:0014
label Unraid OS GUI Mode
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui
label Unraid OS Safe Mode (no plugins, no GUI)
  kernel /bzimage
  append initrd=/bzroot unraidsafemode
label Unraid OS GUI Safe Mode (no plugins)
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui unraidsafemode
label Memtest86+
  kernel /memtest

Graphics card Iommu group

IOMMU group 15:	[10de:1f08] 03:00.0 VGA compatible controller: NVIDIA Corporation Device 1f08 (rev a1)
	[10de:10f9] 03:00.1 Audio device: NVIDIA Corporation Device 10f9 (rev a1)
	[10de:1ada] 03:00.2 USB controller: NVIDIA Corporation Device 1ada (rev a1)
	[10de:1adb] 03:00.3 Serial bus controller [0c80]: NVIDIA Corporation Device 1adb (rev a1)

VM XML

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>Windows 10</name>
  <uuid>b5a6fb72-22eb-7f6c-70e5-4bf167b30f9a</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>12</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='16'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='18'/>
    <vcpupin vcpu='4' cpuset='4'/>
    <vcpupin vcpu='5' cpuset='20'/>
    <vcpupin vcpu='6' cpuset='6'/>
    <vcpupin vcpu='7' cpuset='22'/>
    <vcpupin vcpu='8' cpuset='8'/>
    <vcpupin vcpu='9' cpuset='24'/>
    <vcpupin vcpu='10' cpuset='10'/>
    <vcpupin vcpu='11' cpuset='26'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-3.0'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/b5a6fb72-22eb-7f6c-70e5-4bf167b30f9a_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='6' threads='2'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source dev='/dev/disk/by-id/ata-SanDisk_SSD_PLUS_1000GB_190532804966'/>
      <target dev='hdc' bus='sata'/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.160-1.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:73:be:5a'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/downloads/MSI.RTX2060.6144.181218.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0b' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>

And the error code I get in device manager.  I have tried both the BIOS with header removed, and without from tech powerup.

 

Capture4.PNG

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.