Terrible Perfomance on threadripper


Recommended Posts

Hi everyone

 

I'm pretty new to this still, but I'm kinda hitting a roadblock and could use some help. My brother @m0ngr31 is really selling me on Unraid, and I'm about to purchase a license if i can get my gaming VM working correctly.

 

I'm running a Threadripper 1920x, 12c/24t with 32gb of ram on an MSI x399 carbon pro MB. I'm passing through a gtx 1080ti to my windows 10 vm. I've enabled NUMA in my bios by setting memory interleaving to "Channel" as recommended by @SpaceInvaderOne - I've disabled CPU throttling and C-states in my bios. Made sure my SVM and IOMMU settings are enabled. I've changed a few other things over the course of the last month, but can't remember all what i've set in the bios. I've also got an nvme ssd and a standard sata ssd passed through directly to the vm.

 

I've been following various tips from the forum here.

 

Using LSTOPO, I've verified that my GPU and NVME are both running in NUMA node 1. As such, I've pinned, isolated and assigned every cpu for node 1. I've also followed @Nephilgrim post here, and edited the xml for numatune to pin my vm's ram allocation to node 1. I've verified this with numastat, and it is working.

 

photo_2019-10-06_20-17-59.thumb.jpg.d86e3cf7f294a4a4055b53aef4c99db7.jpg

 

The only things not in node 1 as far as i can tell are my sata controller and my ethernet controller. I've disabled my second drive that is running over sata, but it didn't improve my performance.

 

My pinned and isolated CPUs:

image.png.303ac431e7df2bad97205a488369a531.png

 

image.thumb.png.a64b76e6777dc85965b37a3c5494e4c2.png

 

 

LSTOPO:

lstopomvgpu.thumb.png.8829751c733a8ed33ce33ca9e7066fe9.png

 

Full XML for my VM Here (pastebin)

(Sidenote, i decreased the vm's ram size before uploading this to see if it would help, but no dice. My xml reports 15gb of ram instead of the 16 i've been testing with)

Syslinux:

kernel /bzimage
append isolcpus=6-11,18-23 vfio-pci.ids=10de:1b06,1022:145c pcie_aspm=off kvm_amd npt=1 nested=1 avic=1 initrd=/bzroot

My vfio-pci.ids are my gpu and usb controllers that i'm passing through

 

I've also made a hook for qemu to change the FIFO cpu scheduling or something as per this guy's suggestion. The code for that hook is:

 

#!/bin/bash -x

if [[ $2 == "start"* ]]; then
  if pid=$(pidof qemu-system-x86_64); then
    chrt -f -p 1 $pid
    echo $(date) changing scheduling for pid $pid >> /tmp/libvirthook.log
  fi

fi

 

I've made sure to add the root port info for my pcie slots so they report full x16 pcie 3.0 speeds inside the vm.

 

latencymon and gaming deliver terrible results. Ghost Recon Wildlands nets me about 14fps, unigine gave me a score of about 5000.

 

photo_2019-10-06_20-18-06.thumb.jpg.3145b565fb392801caddbdf50c48c1a4.jpg

I'm really at a loss. From what i read, making sure ram, cpu and gpu were all in the same node seemed to fix the problem for most people. I have done all those things, but my performance is just hot garbage. Can someone offer me some suggestions? I'd really appreciate it

 

Thanks in advance!

Link to comment

https://pastebin.com/7CG7ntKw

 

Take a look at mine, running a similar system (Asrock x399fatality, 1950x, 1080ti, passed through NVME controller and a few usb controllers)

 

Specifically, take a look at the <hyperv>  and <cpu> section differences. I've got latency down to reasonable levels. Still have audio pops and cracks every now and then though, but it brought down my latency from over 2000 to 100ish average. You could also experiment with the "<topology sockets='1' cores='8' threads='1'/>" section and try <topology sockets='1' cores='4' threads='2'/>... well however many cores you allocated.

Edited by Tritech
Link to comment

If you passthrough an NVME the best choice is to passthrough the entire controller and not the by-id. Check my config at the bottom you can find an entry "boot order 1". This is the controller for the NVME. Also using SCSI for the other SSD can give you slightly lower latency and less overhead. I think there is also a SpaceInvader video about how to passthrough an NVME controller. Also you can see in my configs a couple tweaks especialy for TR4 where the CPU is reported as Epyc to make the OS aware of the real amount of L2 and L3 cache of the CPU. Another note, you have set the emulatorpin to cores from node0. I've got better results having it set to the node the cores i'am giving to the VM.

 

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm' id='1' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>01_W10</name>
  <uuid>ab84a67e-428e-36f9-5a88-3c02ffe6f93a</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>14</vcpu>
  <iothreads>1</iothreads>
  <cputune>
    <vcpupin vcpu='0' cpuset='9'/>
    <vcpupin vcpu='1' cpuset='25'/>
    <vcpupin vcpu='2' cpuset='10'/>
    <vcpupin vcpu='3' cpuset='26'/>
    <vcpupin vcpu='4' cpuset='11'/>
    <vcpupin vcpu='5' cpuset='27'/>
    <vcpupin vcpu='6' cpuset='12'/>
    <vcpupin vcpu='7' cpuset='28'/>
    <vcpupin vcpu='8' cpuset='13'/>
    <vcpupin vcpu='9' cpuset='29'/>
    <vcpupin vcpu='10' cpuset='14'/>
    <vcpupin vcpu='11' cpuset='30'/>
    <vcpupin vcpu='12' cpuset='15'/>
    <vcpupin vcpu='13' cpuset='31'/>
    <emulatorpin cpuset='8,24'/>
    <iothreadpin iothread='1' cpuset='8,24'/>
  </cputune>
  <numatune>
    <memory mode='strict' nodeset='1'/>
  </numatune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-3.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/ab84a67e-428e-36f9-5a88-3c02ffe6f93a_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vpindex state='on'/>
      <synic state='on'/>
      <stimer state='on'/>
      <reset state='on'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>EPYC</model>
    <topology sockets='1' cores='7' threads='2'/>
    <cache level='3' mode='emulate'/>
    <feature policy='require' name='topoext'/>
    <feature policy='disable' name='monitor'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='disable' name='svm'/>
    <feature policy='disable' name='x2apic'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='none' io='threads' discard='unmap'/>
      <source dev='/dev/disk/by-id/ata-Samsung_SSD_850_EVO_1TB_S2RFNX0J606029L'/>
      <backingStore/>
      <target dev='hdc' bus='scsi'/>
      <boot order='3'/>
      <alias name='scsi0-0-0-2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/Linux/pop-os_19.04_amd64_nvidia_4.iso'/>
      <backingStore/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <alias name='sata0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <alias name='scsi0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x15'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x16'/>
      <alias name='pci.7'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x6'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0x8'/>
      <alias name='pci.8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='nec-xhci' ports='15'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:65:2d:ab'/>
      <source bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-1-01_W10/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='disconnected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='3'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x43' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <rom file='/mnt/user/Backup/vbios/Strix1080ti/AsusStrix1080TI_dump_edit.rom'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x43' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0a' slot='0x00' function='0x3'/>
      </source>
      <alias name='hostdev2'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x41' slot='0x00' function='0x0'/>
      </source>
      <boot order='1'/>
      <alias name='hostdev3'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev4'/>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc246'/>
        <address bus='5' device='2'/>
      </source>
      <alias name='hostdev5'/>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x1b1c'/>
        <product id='0x1b50'/>
        <address bus='5' device='3'/>
      </source>
      <alias name='hostdev6'/>
      <address type='usb' bus='0' port='2'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+100</label>
    <imagelabel>+0:+100</imagelabel>
  </seclabel>
  <qemu:commandline>
    <qemu:arg value='-global'/>
    <qemu:arg value='pcie-root-port.speed=8'/>
    <qemu:arg value='-global'/>
    <qemu:arg value='pcie-root-port.width=16'/>
  </qemu:commandline>
</domain>

 

  • Like 1
Link to comment

Thanks for your replies! I've still got some wild latency, but your recommendations were a huge step in the right direction. I'm down to about 3500 on latency mon after 3 hours vs 15000 after 24 seconds. GRW doubled in FPS to 30 at 1080p, but realistically with this rig I was able to do 4k at 45fps on bare metal.

 

image.thumb.png.0a4767c155b1cab907eff26764a5f30f.png

 

I unpinned 2 threads in my vm and assigned that to my emulatorpin and iothread

I passed through the controller for my NVME and installed the Samsung driver for that

I added the Epyc info to my <cpu> tag. 

I copied the <hyperv> section from the first reply

 

This is my pinned cpu's now:

image.png.78507cb22fa3006deead60f761111499.png

 

My New XML

 

Current Numastat (no change)

 

image.png.efd4fe84a62e9026d0852cbc6c21c3f9.png

 

anyone have any other ideas I could try to get my latency down?

 

I've checked my bios and that is up to date. Maybe I missed something there?

Link to comment

I'm glad the emulatorpin change make some difference, but you are right, there's still room to improve.

Check a couple of things when the VM is up and running.

1- When the VM is up, go to an unraid console and type:

lspci -s 42:00 -vv | grep Lnk 

if i'm not mistaken, that is your GPU+GPUaudio and will return the current link speed. I hope it will match x16 width and x8 speed. This is just to be sure the GPU is applying the correct speed. I had an issue sometime ago where the gpu got "loose" and, even when working propperly, the speeds and width varied with each boot. Had to unplug and reatach it to solve it and cost me 2 weeks to realize.

 

2- Try a test with a software like passmark to check the CPU raw performance just to see if you are close to baremetal. Even if you are using ~half of your cores it can help to see if your being CPU bottlenecked somehow.

 

3- AFTER this 2 tests, try this (im unsure it can make a difference, but it is more "realistic")

change this in your xml:

<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x42' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <rom file='/mnt/user/appdata/Zotac.GTX1080Ti.11264.170316-fixed.rom'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x42' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </hostdev>

to this:

 

<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x42' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <rom file='/mnt/user/appdata/Zotac.GTX1080Ti.11264.170316-fixed.rom'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x42' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
    </hostdev>

Only change is the "bus" & "function" of the 2nd device. Those 2 are your GPU + GPUAudio (hdmi,displayport). They are in the same place: 42:00.0 and 42:00:1. With this change, the VM will mount a 03:00.0 and a 03:00.1 instead of spliting the device in two (03 and 04, like they were different PCIe devices). It (normally) should not make any difference at all, but since in your case there's no way to know where the computer is suffering the loss I think its better try everything.

Backup your XML before changing it ;)

Edited by Nephilgrim
Link to comment

Thanks for that suggestion :) I haven't had as much time as I'd like to be able to play with it this week.

 

I cleaned out the all my pcie slots with rubbing alcohol and cleaned the contacts on the card itself. I did find something interesting in that I had my gpu in an 8x slot. I've moved it to an x16 slot on the same node, and the command you had me run confirmed that.

 

image.png.c449db5165d733512b575ae9d62898d1.png

 

I tried your xml change too, and that didn't seem to make any difference. My latency is about the same at around 1500. It seems curious to me that the latency peaks with the nvidia driver. I tried an older driver, but that didn't help...

 

In the bios, i forced the pcie slot to run at x16, but it didnt help.

 

Any other ideas?

 

I'm kinda thinking 6.8 will help since it's coming with qemu 4.1 @limetech - Sorry, i don't want to annoy you, but i don't know how far away we are from RC1 and my trial expires this weekend. I'm not ready to drop the $$ unless i can make my vm work right.

Link to comment

Are you sure that the MSI_util patch is applied to the audio and video drivers?

I have a latency of 148.

I somtimes still get stutters on CPU heavy games but it is mostly okay and the benefits of unraid make up for it 🙂

 

My XML to compare:

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm' id='1' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>Monster</name>
  <uuid>18da3180-7fb3-6e5c-7013-963dfa89ec0a</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="freebsd.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>14</vcpu>
  <iothreads>1</iothreads>
  <cputune>
    <vcpupin vcpu='0' cpuset='1'/>
    <vcpupin vcpu='1' cpuset='17'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='18'/>
    <vcpupin vcpu='4' cpuset='3'/>
    <vcpupin vcpu='5' cpuset='19'/>
    <vcpupin vcpu='6' cpuset='4'/>
    <vcpupin vcpu='7' cpuset='20'/>
    <vcpupin vcpu='8' cpuset='5'/>
    <vcpupin vcpu='9' cpuset='21'/>
    <vcpupin vcpu='10' cpuset='6'/>
    <vcpupin vcpu='11' cpuset='22'/>
    <vcpupin vcpu='12' cpuset='7'/>
    <vcpupin vcpu='13' cpuset='23'/>
    <emulatorpin cpuset='1,17'/>
    <iothreadpin iothread='1' cpuset='1,17'/>
  </cputune>
  <numatune>
    <memory mode='preferred' nodeset='0'/>
    <memnode cellid='0' mode='preferred' nodeset='0'/>
  </numatune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-q35-3.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/18da3180-7fb3-6e5c-7013-963dfa89ec0a_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vpindex state='on'/>
      <synic state='on'/>
      <stimer state='on'/>
      <reset state='on'/>
      <vendor_id state='on' value='KVM Hv'/>
      <frequencies state='on'/>
    </hyperv>
  </features>
  <cpu mode='custom' match='exact' check='full'>
    <model fallback='forbid'>EPYC</model>
    <topology sockets='1' cores='7' threads='2'/>
    <cache level='3' mode='emulate'/>
    <feature policy='require' name='topoext'/>
    <feature policy='disable' name='monitor'/>
    <feature policy='require' name='hypervisor'/>
    <feature policy='disable' name='svm'/>
    <feature policy='disable' name='x2apic'/>
    <numa>
      <cell id='0' cpus='0-13' memory='16777216' unit='KiB'/>
    </numa>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='yes'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.160-1.iso'/>
      <backingStore/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <alias name='sata0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Clover_BootFromNVME/spaces_win_clover.img'/>
      <backingStore/>
      <target dev='hdc' bus='sata'/>
      <boot order='1'/>
      <alias name='sata0-0-2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <disk type='block' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source dev='/dev/disk/by-id/ata-Samsung_SSD_850_PRO_512GB_S39FNX0JB11716E'/>
      <backingStore/>
      <target dev='hdd' bus='scsi'/>
      <alias name='scsi0-0-0-3'/>
      <address type='drive' controller='0' bus='0' target='0' unit='3'/>
    </disk>
    <controller type='scsi' index='0' model='virtio-scsi'>
      <alias name='scsi0'/>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'>
      <alias name='pcie.0'/>
    </controller>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x10'/>
      <alias name='pci.1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x11'/>
      <alias name='pci.2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0x12'/>
      <alias name='pci.3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <alias name='pci.4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <alias name='pci.5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0x8'/>
      <alias name='pci.6'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0x9'/>
      <alias name='pci.7'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='8' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='8' port='0xa'/>
      <alias name='pci.8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='9' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='9' port='0xb'/>
      <alias name='pci.9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='qemu-xhci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:d6:6e:98'/>
      <source bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-1-Monster/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0b' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0b' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x45' slot='0x00' function='0x3'/>
      </source>
      <alias name='hostdev2'/>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev3'/>
      <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x1e71'/>
        <product id='0x170e'/>
        <address bus='1' device='3'/>
      </source>
      <alias name='hostdev4'/>
      <address type='usb' bus='0' port='2'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+100</label>
    <imagelabel>+0:+100</imagelabel>
  </seclabel>
  <qemu:commandline>
    <qemu:arg value='-global'/>
    <qemu:arg value='pcie-root-port.speed=8'/>
    <qemu:arg value='-global'/>
    <qemu:arg value='pcie-root-port.width=16'/>
  </qemu:commandline>
</domain>

 

Link to comment
On 10/11/2019 at 8:21 PM, darthcircuit said:

I tried your xml change too, and that didn't seem to make any difference. My latency is about the same at around 1500. It seems curious to me that the latency peaks with the nvidia driver. I tried an older driver, but that didn't help...

I just noticed this part. Did you tried to use msi_util? 

https://www.dropbox.com/s/gymaipg6vprd508/MSI_util.zip?dl=0

apply to your nvidia sound output like this and reboot

 

image.png.42fa5a79d6f55a052c1f24b693250c48.png

 

 

Edited by Nephilgrim
Link to comment

Thanks everyone for your suggestions. I did try the msifix a while ago and that did make a difference too. I’m still getting latency quite a bit higher than I’d like (averages between 6-900), and between that and poor benchmark performance I didn’t do any further testing in most games. 

 

I bit the bullet and purchased unraid, hooked my rig back up to my tv and to my big surprise, every game I tried played almost flawlessly at 4k. I think I’ve been getting too hung up on the benchmarks. Even though ghost recon benchmarks horribly (between 4 and 30 FPS on 1080p), actual gameplay is fine with a pretty solid 45fps at 4k. Wolfenstein 2 played beautifully at 4k 60fps on ultra. 

 

Its not perfect solution  and I’m sure I’ll be tweaking it for a while, but I’m really happy right now. 

Link to comment
On 10/18/2019 at 10:38 AM, darthcircuit said:

Thanks everyone for your suggestions. I did try the msifix a while ago and that did make a difference too. I’m still getting latency quite a bit higher than I’d like (averages between 6-900), and between that and poor benchmark performance I didn’t do any further testing in most games. 

 

I bit the bullet and purchased unraid, hooked my rig back up to my tv and to my big surprise, every game I tried played almost flawlessly at 4k. I think I’ve been getting too hung up on the benchmarks. Even though ghost recon benchmarks horribly (between 4 and 30 FPS on 1080p), actual gameplay is fine with a pretty solid 45fps at 4k. Wolfenstein 2 played beautifully at 4k 60fps on ultra. 

 

Its not perfect solution  and I’m sure I’ll be tweaking it for a while, but I’m really happy right now. 

 

Don't worry use your desktop experience as your benchmark, if your mouse is smooth w/o any jitter while background apps are loading and your sound is fine - your system is responsive enough.   Play games for a few hours.  My Latency Mon numbers would say otherwise, but  I can play many games w/o any stutter, I haven't played one that did.  What I learned a few months late is that you don't need a lot of extra stuff with libvirtd for a smooth overall desktop experience.  

So start simple with your configuration.  

The more I played around with the CPU configuration, the more degraded my performance became.  I learned that by just installing win10 using a basic native qemu setup and using a simple XML and adding what I see from forums, but one line at a time - to troubleshoot. 

So here I am sharing mine.   I borrowed some CPU config lines from Symon's  XML.  Not all of them worked for me because as I'm using qemu 4.0.0
 

<domain type="kvm">
  <name>win10</name>
  <uuid>fa7c22d6-89fa-4c7c-9d4f-86471b96e36f</uuid>
  <metadata>
    <libosinfo:libosinfo xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0">
      <libosinfo:os id="http://microsoft.com/win/10"/>
    </libosinfo:libosinfo>
  </metadata>
  <memory unit="KiB">33554432</memory>
  <currentMemory unit="KiB">33554432</currentMemory>
  <vcpu placement="static">24</vcpu>
  <os>
    <type arch="x86_64" machine="pc-q35-4.0">hvm</type>
    <loader readonly="yes" type="pflash">/mnt/data/firmware/OVMF_CODE.fd</loader>
    <nvram>/mnt/data/firmware/OVMF_VARS_win02.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state="on"/>
      <vapic state="on"/>
      <spinlocks state="on" retries="8191"/>
      <vendor_id state="on" value="132271374ch"/>
    </hyperv>
    <kvm>
      <hidden state="on"/>
    </kvm>
    <ioapic driver="kvm"/>
  </features>
  <cpu mode="host-passthrough" check="full">
    <topology sockets="1" cores="12" threads="2"/>
    <cache mode="passthrough"/>
    <feature policy="require" name="topoext"/>
    <feature policy="disable" name="ibpb"/>
    <feature policy="disable" name="virt-ssbd"/>
  </cpu>
  <clock offset="localtime">
    <timer name="rtc" tickpolicy="catchup"/>
    <timer name="pit" tickpolicy="delay"/>
    <timer name="hpet" present="no"/>
    <timer name="hypervclock" present="yes"/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>destroy</on_crash>
  <pm>
    <suspend-to-mem enabled="no"/>
    <suspend-to-disk enabled="no"/>
  </pm>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type="file" device="disk">
      <driver name="qemu" type="raw"/>
      <source file="/mnt/data/Windows/Win10x-boot.img"/>
      <target dev="vda" bus="virtio"/>
      <boot order="1"/>
      <address type="pci" domain="0x0000" bus="0x0c" slot="0x00" function="0x0"/>
    </disk>
    <disk type="file" device="disk">
      <driver name="qemu" type="raw"/>
      <source file="/mnt/data/Windows/WinDisk.img"/>
      <target dev="vdb" bus="virtio"/>
      <boot order="2"/>
      <address type="pci" domain="0x0000" bus="0x0d" slot="0x00" function="0x0"/>
    </disk>
    <disk type="file" device="disk">
      <driver name="qemu" type="raw"/>
      <source file="/home/me/Downloads/virtio-win-0.1.171.iso"/>
      <target dev="vdc" bus="virtio"/>
      <address type="pci" domain="0x0000" bus="0x0e" slot="0x00" function="0x0"/>
    </disk>
    <disk type="file" device="disk">
      <driver name="qemu" type="raw"/>
      <source file="/mnt/data/Windows/Win10_1903_V1_English_x64.iso"/>
      <target dev="vdd" bus="virtio"/>
      <address type="pci" domain="0x0000" bus="0x0f" slot="0x00" function="0x0"/>
    </disk>
    <controller type="usb" index="0" model="qemu-xhci" ports="15">
      <address type="pci" domain="0x0000" bus="0x02" slot="0x00" function="0x0"/>
    </controller>
    <controller type="sata" index="0">
      <address type="pci" domain="0x0000" bus="0x00" slot="0x1f" function="0x2"/>
    </controller>
    <controller type="pci" index="0" model="pcie-root"/>
    <controller type="pci" index="1" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="1" port="0x10"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="2" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="2" port="0x11"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x1"/>
    </controller>
    <controller type="pci" index="3" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="3" port="0x12"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x2"/>
    </controller>
    <controller type="pci" index="4" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="4" port="0x13"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x3"/>
    </controller>
    <controller type="pci" index="5" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="5" port="0x14"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x4"/>
    </controller>
    <controller type="pci" index="6" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="6" port="0x15"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x5"/>
    </controller>
    <controller type="pci" index="7" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="7" port="0x16"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x6"/>
    </controller>
    <controller type="pci" index="8" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="8" port="0x17"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x02" function="0x7"/>
    </controller>
    <controller type="pci" index="9" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="9" port="0x18"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x0" multifunction="on"/>
    </controller>
    <controller type="pci" index="10" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="10" port="0x19"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x1"/>
    </controller>
    <controller type="pci" index="11" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="11" port="0x1a"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x2"/>
    </controller>
    <controller type="pci" index="12" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="12" port="0x1b"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x3"/>
    </controller>
    <controller type="pci" index="13" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="13" port="0x1c"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x4"/>
    </controller>
    <controller type="pci" index="14" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="14" port="0x1d"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x5"/>
    </controller>
    <controller type="pci" index="15" model="pcie-root-port">
      <model name="pcie-root-port"/>
      <target chassis="15" port="0x1e"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x03" function="0x6"/>
    </controller>
    <controller type="virtio-serial" index="0">
      <address type="pci" domain="0x0000" bus="0x03" slot="0x00" function="0x0"/>
    </controller>
    <controller type="scsi" index="0" model="virtio-scsi">
      <address type="pci" domain="0x0000" bus="0x0b" slot="0x00" function="0x0"/>
    </controller>
    <interface type="direct">
      <mac address="52:54:00:ce:4e:e5"/>
      <source dev="enp5s0" mode="bridge"/>
      <model type="e1000e"/>
      <address type="pci" domain="0x0000" bus="0x05" slot="0x00" function="0x0"/>
    </interface>
    <input type="mouse" bus="ps2"/>
    <input type="keyboard" bus="ps2"/>
    <graphics type="spice" autoport="yes">
      <listen type="address"/>
      <image compression="off"/>
      <gl enable="no"/>
    </graphics>
    <video>
      <model type="cirrus" vram="16384" heads="1" primary="yes"/>
      <address type="pci" domain="0x0000" bus="0x00" slot="0x01" function="0x0"/>
    </video>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x41" slot="0x00" function="0x0"/>
      </source>
      <rom bar="on"/>
      <address type="pci" domain="0x0000" bus="0x01" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x41" slot="0x00" function="0x1"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x06" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x41" slot="0x00" function="0x2"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x07" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="pci" managed="yes">
      <source>
        <address domain="0x0000" bus="0x41" slot="0x00" function="0x3"/>
      </source>
      <address type="pci" domain="0x0000" bus="0x08" slot="0x00" function="0x0"/>
    </hostdev>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x1b1c"/>
        <product id="0x0a14"/>
      </source>
      <address type="usb" bus="0" port="4"/>
    </hostdev>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x046d"/>
        <product id="0xc332"/>
      </source>
      <address type="usb" bus="0" port="8"/>
    </hostdev>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x046d"/>
        <product id="0xc333"/>
      </source>
      <address type="usb" bus="0" port="9"/>
    </hostdev>
    <hostdev mode="subsystem" type="usb" managed="yes">
      <source>
        <vendor id="0x045e"/>
        <product id="0x02ea"/>
      </source>
      <address type="usb" bus="0" port="1"/>
    </hostdev>
    <redirdev bus="usb" type="spicevmc">
      <address type="usb" bus="0" port="2"/>
    </redirdev>
    <redirdev bus="usb" type="spicevmc">
      <address type="usb" bus="0" port="3"/>
    </redirdev>
    <memballoon model="virtio">
      <address type="pci" domain="0x0000" bus="0x04" slot="0x00" function="0x0"/>
    </memballoon>
  </devices>
</domain>

 

 

 

Edited by chromedome
Link to comment
On 10/23/2019 at 8:45 PM, chromedome said:

 

Don't worry use your desktop experience as your benchmark, if your mouse is smooth w/o any jitter while background apps are loading and your sound is fine - your system is responsive enough.   Play games for a few hours.  My Latency Mon numbers would say otherwise, but  I can play many games w/o any stutter, I haven't played one that did.  What I learned a few months late is that you don't need a lot of extra stuff with libvirtd for a smooth overall desktop experience.  

So start simple with your configuration.  

The more I played around with the CPU configuration, the more degraded my performance became.  I learned that by just installing win10 using a basic native qemu setup and using a simple XML and adding what I see from forums, but one line at a time - to troubleshoot. 

So here I am sharing mine.   I borrowed some CPU config lines from Symon's  XML.  Not all of them worked for me because as I'm using qemu 4.0.0
 


<snipped/>

 

 

 

Thanks for that :)


 

The last big issue i had that was noticeable was that my sata and usb controller were on a different numa node, so i'm ordered a combo sata/usb3 controller card. I'm thinking that will improve things a lot. I'm going to pass that through when it gets here, and then leave things as is until 6.8 comes out and i can see if there are any improvements from qemu 4.1...

 

I've been really pleased so far tho, and this is the most helpful community i've been a part of. Thanks everyone!

Edited by darthcircuit
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.