Lousy Performance on GPU Passthrough

[ms4sman]

Hi all, I'm currently in a trial of unRAID to see if I want to buy a key at the end, and I'm having some trouble that I hope I can get help with. I hope I am just forgetting something obvious here.

 

I've done GPU passthrough VM builds before, manually with great success. I did the whole thing by hand on my last rig and it worked great, with only a minimal performance hit. However, on my new build using unRAID, I'm getting just awful performance in games that I was able to kill with native Windows directly on the hardware. For example, with Windows 10 on bare metal, I was able to push 60+ FPS in Witcher 3, nearly max settings at 1440p. In the VM, I get 10-15 tops, and fluctuates wildly, and doesn't seem to really matter much what settings I use.

 

My new rig is a Ryzen 1700,

GTX 1080 (only GPU in the system, and I'm passing it through if that makes a difference)

32 GB DDR4 3200 MHz

3TB WD Red (currently the only disk in the array, has several shares on it)

512 GB SSD (which I'm passing directly to the Windows VM to use directly, no vdisk)

 

I have a feeling it's going to turn out to be something obvious that I'm forgetting, and I may even discover it myself now that I've made a forum post. Usually I end up solving my own problem, but not until RIGHT after I ask for help. But I'd appreciate any thoughts or suggestions of what could be causing this.  Thanks!

 

And here's my VM XML:

<domain type='kvm'>
  <name>Windows 10</name>
  <uuid>4d8a2a93-2808-1687-73bf-31aa1aeaa10d</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='8'/>
    <vcpupin vcpu='1' cpuset='9'/>
    <vcpupin vcpu='2' cpuset='10'/>
    <vcpupin vcpu='3' cpuset='11'/>
    <vcpupin vcpu='4' cpuset='12'/>
    <vcpupin vcpu='5' cpuset='13'/>
    <vcpupin vcpu='6' cpuset='14'/>
    <vcpupin vcpu='7' cpuset='15'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-2.7'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/4d8a2a93-2808-1687-73bf-31aa1aeaa10d_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='8' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source dev='/dev/disk/by-id/ata-MKNSSDRE512GB_MK170421100356BF5'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x03' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='nec-xhci'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='dmi-to-pci-bridge'>
      <model name='i82801b11-bridge'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1e' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0'/>
    </controller>
    <controller type='pci' index='2' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='2'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:b1:3b:18'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x01' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/isos/1080_BIOS.dump'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x04' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0b' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc531'/>
      </source>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x0c45'/>
        <product id='0x7605'/>
      </source>
      <address type='usb' bus='0' port='2'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x28de'/>
        <product id='0x1142'/>
      </source>
      <address type='usb' bus='0' port='3'/>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
</domain>

 

[ms4sman]

As an update, among the things I've tried to improve this, I switched to i440fx from Q35 with no improvement, and I've also tried Hyper-V disabled with no effect either. I've tried a number of other things but those just came to mind as ones that would probably get suggested so I thought I'd provide my results.

[1812]

I'm assuming your other more successful builds were not Ryzen. Have you read the Ryzen thread(s) yet?

[ms4sman]

I have done quite a bit of research on Ryzen and GPU passthrough before and after buying my hardware, regarding the IOMMU issues and such. Some was on here, but I didn't do as much specifically regarding unRAID and Ryzen. I wasn't aware of a known glaring performance issue with it, did I miss something important? I'd be happy to do some more reading if you can point me in the right direction! Thanks!

[ms4sman]

Hmm, going through the "Anybody planning a Ryzen build" thread I am starting to see some people with issues. I did read that thread several times leading up to the build, but it has been active all the time since so I hadn't seen some of the more recent posts on it. If I can't get this sorted out, perhaps I'll just give up and go back to native Windows and come back in a few months and see if things have improved yet.

[Beancounter]

I am experiencing the same issue.

 

From the reading I've done the issue is with KVM.  Xen passthrough has no issues on Ryzen. (correct me if I'm wrong)

 

Disabling NPT in the gaming VM fixes the GPU issue (near barebones speed), but this obviously hinders CPU performance.

 

How to disable NPT:

 

https://forums.lime-technology.com/topic/36553-how-do-i-disable-nested-page-tables/

 

NOTE: I'm not sure which syntax is correct.  I've noticed some people write "kvm-amd npt=0" while others use "kvm-amd.npt=0"

 

For more info see:

 http://www.spinics.net/lists/kvm/msg152112.html

https://forum.level1techs.com/t/ryzen-gpu-passthrough/116458

 

And updates on the VFIO/KVM issue:

 

https://www.reddit.com/r/VFIO/

 

 

[ms4sman]

Thanks for the reply, this was the conclusion I've been nearing from my research too. I'll try disabling npt over my lunch break and see if it helps, at least as a temporary fix for the problem till a real one comes along. In the meantime, I've been using Wendell's "duel-booting" strategy (as he called it in one video), since my Windows VM was installed directly to my SSD rather than a vdisk, I can just boot straight into Windows if I want to play a demanding game, and then switch back to unRAID after. It's not ideal but its an acceptable temporary solution, and if NPT doesn't help or causes unacceptable CPU gimping, I'll go back to that. I'll try to post back here with results.

 

Thanks again!

Edited July 25, 2017 by ms4sman

[ms4sman]

Hmm...out of curiosity, have you tried disabling NPT yourself? I'm not sure if I somehow did something wrong or what, but it doesn't seem to have had even the slightest effect sadly. I tried both the temporary solution and changing the GO script itself and nothing seems to happen. Any ideas?

[Beancounter]

I 

On 7/25/2017 at 1:52 PM, ms4sman said:

Hmm...out of curiosity, have you tried disabling NPT yourself? I'm not sure if I somehow did something wrong or what, but it doesn't seem to have had even the slightest effect sadly. I tried both the temporary solution and changing the GO script itself and nothing seems to happen. Any ideas?

 

I had the same results as you.  I was at work researching a solution for myself when I stumbled upon this thread.

 

I added the following (which basically adds options kvm-amd npt=0 to kvm-amd.conf.

echo "options kvm-amd npt=0" > /etc/modprobe.d/kvm-amd.conf

I didn't notice any improvement so I putty'd back into the tower to see what other files were in /etc/modprobe.d.  Sure enough there was a kvm.conf there as well.

 

I added a line to the kvm.conf and rebooted the server just in case the server was only reading the kvm.conf and not the kvm-amd.conf.  My understanding of the modprobe.d directory is minimal.  I believe it runs any .conf files in that directory on boot and that file name does not matter (only what is contained within them).

 

I didn't notice a difference either way.

 

Quite frankly I don't know how to check that NPT is actually disabled.

 

When I get the kids in bed I'm going to spend some time on this tonight.  I'd gladly take a 20% hit on my CPU if it meant close to barebones GPU speeds.

[ms4sman]

4 hours ago, Beancounter said:

I'd gladly take a 20% hit on my CPU if it meant close to barebones GPU speeds.

Yeah I feel the same, at least as a temporary solution. I'll do a little tinkering if I can this evening, and let you know if I have any success. Let me know what you find if you get a chance to do some testing.

 

Quote

Quite frankly I don't know how to check that NPT is actually disabled.

Yeah I was wondering this too, besides a difference in performance, I have no idea how to tell.

Edited July 27, 2017 by ms4sman

[ms4sman]

Tried it with both lines in, one for each file. No difference.  I'm glad I'm not the only one with this problem.  Considering how many people I have seen talking about disabling NPT, I'm surprised I can't really find anything about how to actually go about it besides the thread you sent. I guess everyone else doing it must be more advanced than us or something.  Keep me posted on any progress Beancounter.

 

Does anyone else in here by chance know of a different way to disable NPT, or what we might be doing wrong? Or if there's any good way to tell whether or not it is actually disabled?

[Beancounter]

Ok.... forget the /etc/modprobe.d/kvm-amd.conf file...

 

Change the syslinux.cfg file to show:

 

label unRAID OS
  menu default
  kernel /bzimage
  append initrd=/bzroot kvm-amd.npt=0

Trying this now.  Will let you know how I make out.

[Beancounter]

Ok.  THIS is how to successfully disable NPT on a Ryzen system:

 

1)  Change the syslinux.cfg file to show:

 

label unRAID OS
  menu default
  kernel /bzimage
  append initrd=/bzroot kvm-amd.npt=0

 

2)  You will notice your windows VM will fail to load now.  In order to get it to work properly you now have to go to the unraid GUI and click on VM's and shut down your windows gaming VM.  Click edit and change the CPU mode from raw/physical to emulated (QEMU) .

 

Thats it!

 

I immediately noticed the CPU usage is higher but realistically jI can throw as many cores at this VM when I'm playing games as I want.

 

Gaming FPS went from 10-25 to 60-140 in the same scene.  

 

WOOHOO!

[ms4sman]

That does indeed seem to have done the trick! I wasn't aware of the need to change from a passthrough CPU to an emulated one, perhaps that's part of the reason for the hit to CPU performance. In my case, it seems to be quite substantial, but I can play with settings some more and see if I can't minimize that a bit. It's not unacceptable, and it does bring my GPU performance pretty much to full, but it is a pretty major hit to performance. Throwing more cores at it doesn't seem to help either. Hopefully someone will have some suggestions to help with this, or better yet, the issue with KVM itself will be resolved soon and integrated into unRAID. Here's hoping!

 

Thanks for your help Beancounter, glad we could both get it working. Let me know how you fare performance wise and if you come up with any tweaks to help.

[SudoPacman]

Interesting thread, so worthy of raising from the dead perhaps :P

 

Just got my nVidia (980) passthrough working last night with an Intel CPU (970), and played a quick game of WoT.

Performance was around a third of what I get natively, so now looking to see if I've done something wrong.

 

Is NPT a thing for Intel builds too, or just Ryzen?

 

Cheers

 

 

[jonp]

On 11/26/2018 at 7:47 AM, SudoPacman said:

Is NPT a thing for Intel builds too, or just Ryzen?

Just Ryzen, but there are some games that don't perform well in a VM because of CPU exits that can occur.  It seems pretty rare, but I think WoT is one of them.

 

Try a few other more common games like Fortnite, Counterstrike, Dota, or any of the other big ones and see if you get the same performance dip.  If so, there may be more to look at in your configuration.  If not, it's a game specific issue.

[SudoPacman]

Yeah, I think it must be the game.

I tried passing through a SSD to eliminate the slower I/O and adding more cores and memory, etc, removing virtual memory etc.

 

Using the enCore benchmark for it (which was a preview of the 1.0 graphics engine), I have around 17,000 native, and managed to get to around 11,000 in the VM.

Playable, but not good enough.

Shame, since love the idea of having multiple disks for different things that can share around between different VMs etc.

 

Have ended the experiment for now. I may revisit unRAID when I rebuild my server.

I think I'm resigned to not using it on my desktop though. WoT is what I generally play, so it's performance is key for me.

 

Cheers

[lcadmedia]

On 7/26/2017 at 10:41 PM, Beancounter said:

Ok.  THIS is how to successfully disable NPT on a Ryzen system:

 

1)  Change the syslinux.cfg file to show:

 

label unRAID OS
  menu default
  kernel /bzimage
  append initrd=/bzroot kvm-amd.npt=0

 

2)  You will notice your windows VM will fail to load now.  In order to get it to work properly you now have to go to the unraid GUI and click on VM's and shut down your windows gaming VM.  Click edit and change the CPU mode from raw/physical to emulated (QEMU) .

 

Thats it!

 

I immediately noticed the CPU usage is higher but realistically jI can throw as many cores at this VM when I'm playing games as I want.

 

Gaming FPS went from 10-25 to 60-140 in the same scene.  

 

WOOHOO!

Tried this but my windows boot up just takes forever with all CPU cores/threads at 100%;
macOS doesn't boot at all, just stuck at 100% cpu usage, and seem to be stuck at booting midway.