Lag/Stutter while gaming on Windows 10 VM

[Mr.Bekannt]

Hello,

 

im trying out Unraid for 1 Month now. Before Unraid i was using Proxmox for my Viruatisation. But on both systems i had an issue to Play games on my system. My goal with this project is, to have one PC, whitch is my cloud, plex sever and gaming machine.

Right now i am playing League of Legends, and after a few games (around 4-5 games) my Framerate Drops from stable 60FPS displayed (over 140FPS from the gpu) to around 20-30FPS. This issue goes sometimes after a few moments away and sometimes i need to restart the whole host system. I am using the nvidia monitoring tool to display the data and when this issue comes up my rendering latency jumps from 0.9ms to 16ms or sometimes way over 30ms.

 

What system am i using:
Mainboard: ASRock AB350M Pro4 (Newest BIOS flashed today)

CPU: AMD Ryzen 7 3800X 8-Core @ 3900 MHz
RAM: 32 GB of the same speed 4x8GB

GPU: MSI GeForce GTX 1070 GAMING X 8GB (with the bios flashed for VM capabilty | 3 Monitors are used on this GPU)

Drives: -18TB Media and VM/Docker disc
          -500GB Samsung SSD for Gaming vm directly fed in

 

 

I was thinking, that the issue is ether comming from my Ryzen 1600X cpu or the storage system. So i changed the cpu to the 3800X, it helped a bit but the main issue seems to be still there.

What have i tried do solve the issue:
Lots, lots of searching. I think i am gaming with this issue now way over a year now. I tried both Machine types, both BIOS types. I used the vm with and without balooning. I am using the gouvernor type Performance. I have an scrypt for the zenstates. 

 

Also i tried not just to bruteforce the issue with solutions. My first try to analyze the issue on the unraid system was iotop but i couldn

understand what this tool is telling me. I tryed to understand what exactly is the cpu-pinning and the importance of the whole cpu allocation is, but never came to it.

My VM XML:

 <?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm' id='3'>
  <name>Gaming VM</name>
  <uuid>4f84f7f3-c726-d78d-6c1f-269ea0dbb4ef</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>20971520</memory>
  <currentMemory unit='KiB'>20971520</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>10</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='1'/>
    <vcpupin vcpu='1' cpuset='9'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='10'/>
    <vcpupin vcpu='4' cpuset='3'/>
    <vcpupin vcpu='5' cpuset='11'/>
    <vcpupin vcpu='6' cpuset='4'/>
    <vcpupin vcpu='7' cpuset='12'/>
    <vcpupin vcpu='8' cpuset='6'/>
    <vcpupin vcpu='9' cpuset='14'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-6.2'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/4f84f7f3-c726-d78d-6c1f-269ea0dbb4ef_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv mode='custom'>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='5' threads='2'/>
    <cache mode='passthrough'/>
    <feature policy='require' name='topoext'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source dev='/dev/disk/by-id/ata-Samsung_SSD_850_EVO_500GB_S2RBNB0J110662P' index='3'/>
      <backingStore/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <alias name='virtio-disk2'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/Win10_21H2_German_x64.iso' index='2'/>
      <backingStore/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <alias name='sata0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.190-1.iso' index='1'/>
      <backingStore/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <alias name='sata0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0xa'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0xb'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xc'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xd'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x9'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:68:20:f7'/>
      <source bridge='br0'/>
      <target dev='vnet2'/>
      <model type='virtio-net'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/1'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/1'>
      <source path='/dev/pts/1'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-3-Gaming VM/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <audio id='1' type='none'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <rom file='/mnt/disk1/isos/geil.rom'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x0d8c'/>
        <product id='0x0005'/>
        <address bus='3' device='4'/>
      </source>
      <alias name='hostdev2'/>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x1532'/>
        <product id='0x0067'/>
        <address bus='3' device='3'/>
      </source>
      <alias name='hostdev3'/>
      <address type='usb' bus='0' port='2'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x1532'/>
        <product id='0x021e'/>
        <address bus='3' device='2'/>
      </source>
      <alias name='hostdev4'/>
      <address type='usb' bus='0' port='3'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x1532'/>
        <product id='0x051a'/>
        <address bus='1' device='8'/>
      </source>
      <alias name='hostdev5'/>
      <address type='usb' bus='0' port='4'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+100</label>
    <imagelabel>+0:+100</imagelabel>
  </seclabel>
</domain>

Are there any tools, that i could use to find out where this latency problem comes from or is there any start point to diagnose the issue?

This last year of trying to fix this issue made me unable to see any possible solutions. I was hoping, that the issue will magicly disappear with the change to unraid...

I am very grateful for any help
~Bekannt

bitbunker-diagnostics-20220613-2304.zip

Edited June 13, 2022 by Mr.Bekannt

[ghost82]

I'm using latencymon:

https://www.resplendence.com/latencymon

 

https://www.resplendence.com/download/LatencyMon.exe

 

just run it for some time and it will tell you latencies and drivers which may have issues.

If you find issues, high latency, try the msi utility:

https://forums.unraid.net/topic/76035-help-struggling-with-nvidia-audio-passthrough/?do=findComment&comment=1076592

 

Choose v3 and run it as admin.

Check if there are any irq conflict and if any, enable msi for one of the conflicting devices.

Edited June 14, 2022 by ghost82

[Mr.Bekannt]

Thank you for the fast reply.

 

With LatencyMon and the msi utility i was able to fix one of the latency issues which was caused by the dxgkrnl.sys I set the priority to high and checked the msi checkbox for my gpu. After this the ISR Count went to 0 and the Highest execution went to 4,2ms. But the directx drivers were just the second most leaking driver. The USBPORT.SYS is still there with around 20ms at highest execution. I was trying to fix it like the gpu wit the msi utility. I gave all 4 of these usb controllers the msi checkbox and interrupt priority of "Normal" but this didnt help at all. I allso disabled the USB selective suspect setting explained in this post:

But no change ether.

The screenshots were taken after the restart with low loads. Just some browsing.

The next thing what i was thinking to try, is to forward my four usb inputs directly to the vm. Like Spacinvader is telling in this video:

But unfortenetly this guide doesent work with Unraid 6.10.2 any more. When i try to list the input devices by id, my system tells me, that there is no path for "by-id": 
 

root@Bitbunker:~# ls /dev/input/by-id
/bin/ls: cannot access '/dev/input/by-id': No such file or directory

 

A other thing with this usb issue is, that in the msi mode utility all four controllers translated form german word by word have the name: "Intel(R) ICH9-Fimily USB2 universal Hostcontroller -" with the numbers 2934,2935,2936 and 293A at the end. I don´t think, that i have an Intel controller on my Ryzen Setup.. could this be the issue?

I haven´t been playing long enough to tell if this issue is now gone. But still i am very thankful to you for your fast and constructive response!
~Bekannt

 

Edited June 14, 2022 by Mr.Bekannt

[ghost82]

3 hours ago, Mr.Bekannt said:

that i have an Intel controller on my Ryzen Setup.. could this be the issue?

That are the usb emulated controllers you defined in the xml: ehci/uhci.

You still have a shared irq between an emulated usb controller and the high definition audio of your gpu: the audio supports msi so enable it also for the audio, then as far as it concern irq sharing you are good. If it still lags causes are elsewhere...

Emulated usb controllers can be deleted if you are not passing through any single usb device, or you can decrease their number, I don't think you need 2 usb 2.0 (ehci) and 2 usb 1.1 (uhci).

Note that one ehci always needs its companion uhci.

The lags come from the usb, so maybe that setting msi for conflicting audio will solve the lag.

Edited June 14, 2022 by ghost82

[ghost82]

3 hours ago, Mr.Bekannt said:

I gave all 4 of these usb controllers the msi checkbox and interrupt priority of "Normal" but this didnt help at all

this wont work because they don't support msi: irq line based is defined by "linebased", message signaling interrupts (msi) by "msi" and/or "msix".

[Mr.Bekannt]

I´ve set an checkmark to the msi on the High Definition Audio-Controller. After now 25 mins uptime we are at around 10 ms @ Highest execution for USBPORT.sys. For now i belive this is a improvement. Now i need some friends to come onine and play a bit to get the proof. But im in good spirits about that... this would be great.

About the USB Ports... i am using:

-Mouse (Razer Naga trinity)
-Keyboard (Razer Ornata Chroma)
-Headset (Razer Nari Ultimate)
-Microphone (t.bone)

I don´t know if the Audio-Controller is used by default when i use an wireless usb headset. Nor if the VM setting would change something. I have the options between the GPU/CPU audio controller. I am about to try it our right now.
 

On my old Proxmox setup i did a pci passthrough for one of two usb controllers from the mobo, because of the native max 4 usb port passthrough setting on their os. I was thinking to get an usb-c controller switch, which i can simply forward as one pci item and have an hotplug usb hub for the gaming guest but sadly i bought a wrong one which was a simple dummy hub.

 

~Bekannt

[ghost82]

9 hours ago, Mr.Bekannt said:

About the USB Ports... i am using:

-Mouse (Razer Naga trinity)
-Keyboard (Razer Ornata Chroma)
-Headset (Razer Nari Ultimate)
-Microphone (t.bone)

If you are not passing through the usb controller directly, you are passing through single devices that attach to emulated usb controller.

For example, in an xml you could have this:

    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/>
    </controller>

This defines a single multifunction emulated usb controller, 2.0/1.1 (ehci/uhci).

This emulated usb controller has index=0, this means that if you want to attach usb devices to these virtual controller you need to attach them at bus 0, for example:

    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0x0892'/>
      </source>
      <address type='usb' bus='0' port='2'/>
    </hostdev>

You see bus 0 here that corresponds to index 0 for the usb controller; ehci/uhci is automatically chosen depending on the usb device.

About the port in the usb device block: you see that device is attached to port 2, because:

- port 0 on any usb controller is always the usb root hub

- port 1 is hardcoded for the device "USB Tablet" if you have it in your xml, this attaches to usb controller uhci, port 1

 

So if you have multiple usb devices to passthrough to emulated usb controllers, if you want to use ehci/uhci, just define a single emulated multifunction usb controller and attach usb devices starting from port 2.

Uhci controller supports a maxmum of 2 ports while ehci controller supports 6.

 

So you can definetly delete the second multifunction usb controller you have in the xml, unless you have more than 1 usb 1.1. device (I don't think so..).

 

9 hours ago, Mr.Bekannt said:

I don´t know if the Audio-Controller is used by default when i use an wireless usb headset

Is it not the audio controller of the gpu for digital audio output?

 

Passing through a usb controller instead of single devices is always the best option:

1- for usb devices compatibility

2- to have usb hotplug

Edited June 15, 2022 by ghost82

[Mr.Bekannt]

  

5 hours ago, ghost82 said:

Passing through a usb controller instead of single devices is always the best option:

1- for usb devices compatibility

2- to have usb hotplug

100%! Thats also another reason why i did it on the proxmox system. Now i did the same on Unraid as you said. Again i used the Video of Spacinvader. Very nice guy. :

Instead of doing the exclusion of the controller in the Syslinux config. I used the build in function from unraid in Tools>System Devices>Checking the needed IOMMU group>Bind selected to VFIO at Boot. I had to remove the 4 forwarded usb ports from my VM XML config otherwise the VM Manager won´t accept the new USB controller.

 

We are now down to 5ms on the USBPORT.sys after 20 mins writing this post and listening to music on the VM. Some Screenshots again:

 

 

Spoiler

I finally found the Spoiler button! 😅

 

I tryied to game a bit yesterday but sadly my gamefiles went bad after 2-3 games... so i had to reinstall...

 

5 hours ago, ghost82 said:

Is it not the audio controller of the gpu for digital audio output?

I am using an USB wireless headset. I don´t know if it does access the gpu audio. I thought, that the headset with its dongle is using its own audio drivers. I dont use any kind of hdmi audio source - just the usb headset.
 

~Bekannt

 

[ghost82]

23 hours ago, Mr.Bekannt said:

Instead of doing the exclusion of the controller in the Syslinux config. I used the build in function from unraid in Tools>System Devices>Checking the needed IOMMU group>Bind selected to VFIO at Boot. I had to remove the 4 forwarded usb ports from my VM XML config otherwise the VM Manager won´t accept the new USB controller.

Correct

 

23 hours ago, Mr.Bekannt said:

I am using an USB wireless headset. I don´t know if it does access the gpu audio

No, the usb audio has its own drivers.

The "High definition audio controller" you see in msi utility is that of the gpu, which should be passed through to the vm together with the video component of the gpu (so it's correct what you are doing).