Stuttering GPU passthrough, or maybe CPU?

[reexe]

Hi.

 

I have a problem where i do not get the performance that i should get out of my VMs.

 

My hardware is the following:

Motherboard: ASUS Sabertooth 990FX R2.0

CPU: AMD FX-8350, 125W

GPU: AMD R9 390x (8G)

RAM: DDR3 16G

Monitor: Dell UHD 3440x1440 resolution

 

So i have passed through my GPU to a windows 10 pro VM and installed the correct drivers (have tried different ones) and whenever i start a game everything looks fine, until i turn the camera in the game. Whenever i turn my head/camera in games my fps is drastically lowered, and then when i stop moving my head it goes back up, if i for example walk forwards in the game i maintain high fps. This makes me believe that this is happening whenever the game tries to load stuff to the GPU memory. The lag is not reduced by lowering the in-game resolution, neither by running in window mode.

I also get lag whenever i try to watch a 4k youtube video, and i have also tried reinstalling everything the same way on a windows 7 VM, but i get the exact same problem.

 

I have also tried blocking the VMs cpu cores with isolcpus from unRAID, so only the VM can use the cores, this did not make any noticeable difference.

 

(I have passes through a USB3 controller as well if that matters at all.)

 

Edit: i forgot to add that i have also tried using a AMD HD7870 and i got the same problems when i was using that, and i have tried multiple games such as GW2 and Planetside 2.

 

My XML file is the following:

<domain type='kvm' id='1' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>

  <name>reexe-win10</name>

  <uuid>3b6f7ec0-4bff-8046-570e-b963d087b5d8</uuid>

  <metadata>

    <vmtemplate name="Custom" icon="windows.png" os="windows"/>

  </metadata>

  <memory unit='KiB'>8388608</memory>

  <currentMemory unit='KiB'>8388608</currentMemory>

  <memoryBacking>

    <nosharepages/>

    <locked/>

  </memoryBacking>

  <vcpu placement='static'>4</vcpu>

  <cputune>

    <vcpupin vcpu='0' cpuset='0'/>

    <vcpupin vcpu='1' cpuset='1'/>

    <vcpupin vcpu='2' cpuset='4'/>

    <vcpupin vcpu='3' cpuset='5'/>

  </cputune>

  <resource>

    <partition>/machine</partition>

  </resource>

  <os>

    <type arch='x86_64' machine='pc-i440fx-2.3'>hvm</type>

  </os>

  <features>

    <acpi/>

    <apic/>

    <hyperv>

      <relaxed state='on'/>

      <vapic state='on'/>

      <spinlocks state='on' retries='8191'/>

    </hyperv>

  </features>

  <cpu mode='host-passthrough'>

    <topology sockets='1' cores='4' threads='1'/>

  </cpu>

  <clock offset='localtime'>

    <timer name='hypervclock' present='yes'/>

    <timer name='hpet' present='no'/>

  </clock>

  <on_poweroff>destroy</on_poweroff>

  <on_reboot>restart</on_reboot>

  <on_crash>restart</on_crash>

  <devices>

    <emulator>/usr/bin/qemu-system-x86_64</emulator>

    <disk type='file' device='disk'>

      <driver name='qemu' type='raw' cache='writeback'/>

      <source file='/mnt/cache/Cache-folder/vm/reexe-win10/vdisk1.img'/>

      <backingStore/>

      <target dev='hdc' bus='virtio'/>

      <boot order='1'/>

      <alias name='virtio-disk2'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>

    </disk>

    <controller type='usb' index='0'>

      <alias name='usb'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>

    </controller>

    <controller type='pci' index='0' model='pci-root'>

      <alias name='pci.0'/>

    </controller>

    <controller type='virtio-serial' index='0'>

      <alias name='virtio-serial0'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>

    </controller>

    <interface type='bridge'>

      <mac address='52:54:00:bc:d9:d0'/>

      <source bridge='br0'/>

      <target dev='vnet0'/>

      <model type='virtio'/>

      <alias name='net0'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>

    </interface>

    <serial type='pty'>

      <source path='/dev/pts/0'/>

      <target port='0'/>

      <alias name='serial0'/>

    </serial>

    <console type='pty' tty='/dev/pts/0'>

      <source path='/dev/pts/0'/>

      <target type='serial' port='0'/>

      <alias name='serial0'/>

    </console>

    <channel type='unix'>

      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/reexe-win10.org.qemu.guest_agent.0'/>

      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>

      <alias name='channel0'/>

      <address type='virtio-serial' controller='0' bus='0' port='1'/>

    </channel>

    <hostdev mode='subsystem' type='usb' managed='yes'>

      <source>

        <vendor id='0x1532'/>

        <product id='0x002e'/>

        <address bus='11' device='2'/>

      </source>

      <alias name='hostdev0'/>

    </hostdev>

    <hostdev mode='subsystem' type='usb' managed='yes'>

      <source>

        <vendor id='0x1532'/>

        <product id='0x011a'/>

        <address bus='10' device='2'/>

      </source>

      <alias name='hostdev1'/>

    </hostdev>

    <hostdev mode='subsystem' type='pci' managed='yes'>

      <driver name='vfio'/>

      <source>

        <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>

      </source>

      <alias name='hostdev2'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>

    </hostdev>

    <memballoon model='virtio'>

      <alias name='balloon0'/>

      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>

    </memballoon>

  </devices>

  <qemu:commandline>

    <qemu:arg value='-device'/>

    <qemu:arg value='ioh3420,bus=pci.0,addr=1c.0,multifunction=on,port=2,chassis=1,id=root.1'/>

    <qemu:arg value='-device'/>

    <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>

  </qemu:commandline>

</domain>

 

Some extra info i can add is that i have used Unigine 'Valley' Benchmark 1.0 with Extreme HD settings and got 2500 scores on a standard windows 8.1 and i got 1800 scores when running on a VM in unRAID.

 

[reexe]

I don't know, maybe i came off as rude or something in the first post, but i rly need some help on the matter, any1 have a clue what is going on?

[bungee91]

It sounds more CPU related than GPU, as 4k video shouldn't be hardware decoded (I believe) and rely on the CPU to decode.

I assume that you have 8 cores to assign, correct?

UnRAID likes to do tasks on core 0 (from what I've read), so I recommend not using that in the VM.

Your XML shows 0,1,4,5 as assigned to this, any particular reason for choosing those?

I'd try your top 4 cores (likely 4-7) or 2,3 & 6,7 if this is anything like Intel cores (I don't think it is, but I'm not certain on the recommendation for AMD assignments).

 

[reexe]

Alright, i only have 4 cores assigned atm but i will switch them around and see if anything changes and report back.

[reexe]

Just tried using 4-7 and i get the same problem

 

I also blocked unraid from core 1-7 so it definitely only uses number 0

[reexe]

Bump..

i am still facing the same problems, i have tried everything i can think of except replacing the cpu... (dont have money for that atm).

 

Anyone have any ideas of what could be done to improve the gaming/4Kyoutube experience? Or at least confirm that 4k video is dependent on the cpu?

[jonp]

Disable hyperv and report back if that makes any difference.

[reexe]

How do i do that? i thought amd processors did not use that?

[reexe]

okay i googled a bit and i ran this command when in a elevated command prompt.

dism.exe /Online /Disable-Feature:Microsoft-Hyper-V-All

 

I do not see any difference in game performance, and 4k youtube videos are still lagging.

I dont know if that command is what u where thinking about?

 

Edit: thought i would post what the command reported back when i ran it:

Disabling feature(s)

[==========================100.0%==========================]

The operation completed successfully.

 

It never asked me to restart the computer, but i did that anyway, still no difference.

[saarg]

okay i googled a bit and i ran this command when in a elevated command prompt.

dism.exe /Online /Disable-Feature:Microsoft-Hyper-V-All

 

I do not see any difference in game performance, and 4k youtube videos are still lagging.

I dont know if that command is what u where thinking about?

 

Edit: thought i would post what the command reported back when i ran it:

Disabling feature(s)

[==========================100.0%==========================]

The operation completed successfully.

 

It never asked me to restart the computer, but i did that anyway, still no difference.

You have to disable it in the VM template 

If you don't see it when in edit, you have to enable advanced view.

[reexe]

Ty! i will try that!

[reexe]

You have to disable it in the VM template 

If you don't see it when in edit, you have to enable advanced view.

 

 

So i did this and still got the same result.. 

[reexe]

Just updated to 6.1.9, no improvements.

[bungee91]

I'd like to take a look at your full diagnostics output, hoping I'll see something useful, please post that.

(tools, diagnostics, download or from command line, type diagnostics, upload file on root of flash drive)

[reexe]

Thx for trying to help!

This is what i got.

reexe-unraid-diagnostics-20160307-1630.zip

[bungee91]

1st question, what are you using for sound in this VM?

I don't see that the GPU sound is being passed (01:00.1), or the onboard sound (00.14.2), so I'm not seeing audio here at all.

Some people have reported issues with passing a GPU without the HDMI sound also.

Even if you don't plan to use the sound card part of the GPU, it seems to cause issues at times.

You can add multiple sound devices to a VM with the + symbol next to the sound card option.

 

 

2nd question (just so I know), what 2 USB devices are you passing in the xml (specifically these)

<hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x1532'/>
        <product id='0x002e'/>
        <address bus='11' device='2'/>
      </source>
      <alias name='hostdev0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x1532'/>
        <product id='0x011a'/>
        <address bus='10' device='2'/>
      </source>
      <alias name='hostdev1'/>
    </hostdev>

Since you're passing this USB controller (04:00.0 USB controller [0c03]: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller [1b21:1042]), any reason you're also passing individual devices?

[reexe]

I pass through my razer black widow keyboard and my razer naga mouse using those lines, and then the controller is for a usb3 hdd and also a logitech g933 headset that wont sound correctly unless i pass through the whole controller.

 

Edit: to specify the g933 is what i use as far as sound goes, i have heard ppl have had problems using hdmi sound so thats why i did not even bother trying it.

[reexe]

If i understand your answer correctly tho i guess u want me to try to pass through the hdmi sound as well to see if that makes a difference? I will try this and report back 

[reexe]

Just tried passing through the GPUs sound (01:00.1) as well, did not make any difference. 

[bungee91]

I think the best point to try and get this down to bare bones is to remove any variables that may be causing lag, or latency.

Can you remove the USB card from the XML, and attempt this without the g933/hdd? You can always pass the onboard audio so that you have sound to test to see.

 

[reexe]

I have tried that already, exept i did not use any audio at all.

[reexe]

I have still not found a solution.

[bungee91]

Unfortunately at this point I'm out of solutions to help (sorry, but didn't want to leave you hanging here).

The other issue is that the amount of people doing this with AMD hardware is lower in comparison to Intel, so that gives less input from users who may have a similar situation and can provide solutions.

There are certain things you can attempt to add to your syslinux.cfg file that may be of help to isolate this (not just isolcpus), but I don't have the specific answer to that.

If you're adventurous, or determined, I'd highly recommend asking your question in the VFIO mailing list, as there are some incredibly bright people there who will likely give more input than I can that may help. While this is not specific to UnRAID, it is specific to VFIO/QEMU/KVM and will likely get you a quicker response than here.

Anyhow the information is below:

Sign up https://www.redhat.com/mailman/listinfo/vfio-users

Archived threads (a lot of good information in here) https://www.redhat.com/archives/vfio-users/

Hope you get it resolved soon! 

[reexe]

Thx for the help man, i may try that, just hoping stuff will resolve itself in 6.2 version or that i at least can try OVMF more successfully that i have been able to in the current version and hope that gets me better results.

[reexe]

I have found some info here https://www.redhat.com/archives/vfio-users/2016-February/msg00206.html its the same processor as i have, and according to this guy it helped to allocate multiple cores to the same virtual core, like this

<vcpupin vcpu='0' cpuset='4-5'/>

However, unRAID does only seem to use the first nr that is entered after the = mark, so in this example unraid only used core nr 4 and not the inteded 4 and 5.

And when i click edit on my VM after i have enabled this using the xml edit only core 4 is checked.

 

Is there any other way i can make it so my VM uses 2 real cores per virtual core?