Jerky_san

Weirdly I didn't originally. For some reason my root tweak for the gpu only isn't there anymore. I forgot I had tweaked my GPU. All you need to do is on the GPU PCI you set the bus to 08 if your using my XML.. up higher in the XML is a pci-root that is set to 08 but and its multi function

<?xml version='1.0' encoding='UTF-8'?> <domain type='kvm' id='6' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> <name>Gaming-VM-Test</name> <uuid>2dbff225-ec92-e4e2-d5d8-fdf4cd84353b</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/> </metadata> <memory unit='KiB'>33554432</memory> <currentMemory unit='KiB'>33554432</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>16</vcpu> <cputune> <vcpupin vcpu='0' cpuset='4'/> <vcpupin vcpu='1' cpuset='36'/> <vcpupin vcpu='2' cpuset='5'/> <vcpupin vcpu='3' cpuset='37'/> <vcpupin vcpu='4' cpuset='6'/> <vcpupin vcpu='5' cpuset='38'/> <vcpupin vcpu='6' cpuset='7'/> <vcpupin vcpu='7' cpuset='39'/> <vcpupin vcpu='8' cpuset='8'/> <vcpupin vcpu='9' cpuset='40'/> <vcpupin vcpu='10' cpuset='9'/> <vcpupin vcpu='11' cpuset='41'/> <vcpupin vcpu='12' cpuset='10'/> <vcpupin vcpu='13' cpuset='42'/> <vcpupin vcpu='14' cpuset='11'/> <vcpupin vcpu='15' cpuset='43'/> <emulatorpin cpuset='4,8,36,40'/> </cputune> <numatune> <memory mode='strict' nodeset='0,2'/> </numatune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-q35-3.1'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/2dbff225-ec92-e4e2-d5d8-fdf4cd84353b_VARS-pure-efi.fd</nvram> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <vpindex state='on'/> <synic state='on'/> <stimer state='on'/> <reset state='on'/> <vendor_id state='on' value='KVM Hv'/> <frequencies state='on'/> </hyperv> </features> <cpu mode='custom' match='exact' check='full'> <model fallback='forbid'>EPYC</model> <topology sockets='1' cores='8' threads='2'/> <feature policy='require' name='topoext'/> <feature policy='disable' name='monitor'/> <feature policy='require' name='hypervisor'/> <feature policy='disable' name='svm'/> <feature policy='disable' name='x2apic'/> <numa> <cell id='0' cpus='0-7' memory='16777216' unit='KiB'/> <cell id='1' cpus='8-15' memory='16777216' unit='KiB'/> </numa> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='yes'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <controller type='pci' index='0' model='pcie-root'> <alias name='pcie.0'/> </controller> <controller type='pci' index='1' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='1' port='0x8'/> <alias name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='2' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='2' port='0x9'/> <alias name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='pci' index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='3' port='0xa'/> <alias name='pci.3'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/> </controller> <controller type='pci' index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='4' port='0xb'/> <alias name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/> </controller> <controller type='pci' index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='5' port='0xc'/> <alias name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/> </controller> <controller type='pci' index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='6' port='0xd'/> <alias name='pci.6'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/> </controller> <controller type='pci' index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target chassis='7' port='0xe'/> <alias name='pci.7'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/> </controller> <controller type='pci' index='8' model='pcie-root-port'> <model name='ioh3420'/> <target chassis='8' port='0x1f'/> <alias name='pci.8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1c' function='0x0' multifunction='on'/> </controller> <controller type='pci' index='9' model='pcie-root-port'> <model name='ioh3420'/> <target chassis='9' port='0x1d'/> <alias name='pci.9'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller> <controller type='sata' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller type='usb' index='0' model='qemu-xhci' ports='15'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:d7:c8:2c'/> <source bridge='br0'/> <target dev='vnet2'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/2'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/2'> <source path='/dev/pts/2'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-6-Gaming-VM-Test/org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='mouse' bus='ps2'> <alias name='input0'/> </input> <input type='keyboard' bus='ps2'> <alias name='input1'/> </input> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x41' slot='0x00' function='0x0'/> </source> <alias name='hostdev0'/> <rom file='/mnt/user/domains/1080ti.rom'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x41' slot='0x00' function='0x1'/> </source> <alias name='hostdev1'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0c' slot='0x00' function='0x3'/> </source> <alias name='hostdev2'/> <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x09' slot='0x00' function='0x0'/> </source> <alias name='hostdev3'/> <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x42' slot='0x00' function='0x3'/> </source> <alias name='hostdev4'/> <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x0b05'/> <product id='0x1867'/> <address bus='1' device='8'/> </source> <alias name='hostdev5'/> <address type='usb' bus='0' port='1'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x0b05'/> <product id='0x1868'/> <address bus='1' device='2'/> </source> <alias name='hostdev6'/> <address type='usb' bus='0' port='2'/> </hostdev> <memballoon model='none'/> </devices> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+100</label> <imagelabel>+0:+100</imagelabel> </seclabel> <qemu:commandline> <qemu:arg value='-global'/> <qemu:arg value='pcie-root-port.speed=8'/> <qemu:arg value='-global'/> <qemu:arg value='pcie-root-port.width=16'/> </qemu:commandline> </domain>

Trying now. I can say its not really completely about performance but also latency experienced in windows. DPC latency is all over the place. Can confirm patch is successful =0

Woohoo can't wait for it

When the VM starts the card it does a rate negotiation. The rate negotiation eventually works its way up to the proper but the card when it starts only sees x1 so it sets registers to that effect. Please read the post he did if you'd like further detail. The guys in the posts go into great depth about it

i440fx gives me the 0x while q35 gives me 1x so I assume his is the same

I read the whole thing he posted(it was a hell of a lot) basically when the VM boots it sets a bunch of registers. Those registers impact how the driver and windows interact with the card. Latency and many things are impacted. Not just speed. Basically the patch is to tell the card when it boots "hey your in a x16 slot so set the registers accordingly!" and so it does.

welp hope we get it then or maybe a way to to just run the RC over QEMU 4.0 and have a switch that turns it on & off or something.

Ah shit your right.. still x1 #_# thats depressing

Changed windows 10 machine to q35-3.1 machine Inserted this below the rest of the controllers <controller type='pci' index='8' model='pcie-root-port'> <model name='ioh3420'/> <target chassis='8' port='0x1f'/> <alias name='pci.8'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x1c' function='0x0' multifunction='on'/> </controller> Went where my hostdev's are and inserted this. Change the address in the <source></source> to your GPU's and GPU audio's on the second and it should be more or less plug and play. <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x41' slot='0x00' function='0x0'/> </source> <alias name='hostdev0'/> <rom file='/mnt/user/domains/1080ti.rom'/> <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x41' slot='0x00' function='0x1'/> </source> <alias name='hostdev1'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/> </hostdev>

Inside this thread he talks about how to do it without the patch. It was a big pain the damn ass but I think I got it working..

Second on this

The guest sees the disk as a physical drive so it passes everything to it.

More Tweaking with more cores across more numa Below is what I settled on and the XML.. I actually got much faster L1-L3 read/writes but I think I missed something as L3 cache slowed. I don't know where I messed up but I'll experiment more tmrw probably. I don't really need all the cores though so in a way I'm satisfied with what I got. Remember I am on a 2990wx so adjust accordingly. I am thinking about doing video's like SpaceInvader but I am not great at speaking #_#;; <vcpu placement='static'>16</vcpu> <cputune> <vcpupin vcpu='0' cpuset='4'/> <vcpupin vcpu='1' cpuset='36'/> <vcpupin vcpu='2' cpuset='5'/> <vcpupin vcpu='3' cpuset='37'/> <vcpupin vcpu='4' cpuset='6'/> <vcpupin vcpu='5' cpuset='38'/> <vcpupin vcpu='6' cpuset='7'/> <vcpupin vcpu='7' cpuset='39'/> <vcpupin vcpu='8' cpuset='8'/> <vcpupin vcpu='9' cpuset='40'/> <vcpupin vcpu='10' cpuset='9'/> <vcpupin vcpu='11' cpuset='41'/> <vcpupin vcpu='12' cpuset='10'/> <vcpupin vcpu='13' cpuset='42'/> <vcpupin vcpu='14' cpuset='11'/> <vcpupin vcpu='15' cpuset='43'/> <emulatorpin cpuset='4-11'/> </cputune> <numatune> <memory mode='strict' nodeset='0,2'/> </numatune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-i440fx-3.0'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/3b8790bc-59c0-ff66-e9b9-c3c716abc8b5_VARS-pure-efi.fd</nvram> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <hyperv> <relaxed state='on'/> <vapic state='on'/> <spinlocks state='on' retries='8191'/> <synic state='on'/> <stimer state='on'/> <vendor_id state='on' value='none'/> </hyperv> </features> <cpu mode='custom' match='exact' check='partial'> <model fallback='allow'>EPYC-IBPB</model> <topology sockets='1' cores='8' threads='2'/> <feature policy='require' name='topoext'/> <numa> <cell id='0' cpus='0-7' memory='16777216' unit='KiB'/> <cell id='1' cpus='8-15' memory='16777216' unit='KiB'/> </numa> </cpu> Why I decided to stop is below. It appears you get diminishing returns the more cores you add after a certain point. RAM also gets slower instead of faster. 24 physical cores 50 cores breaks L3 cache - Another thing I thought is that maybe since I am using the EPYC work around it's causing things to break as well. Also ignore the last image don't know why it won't remove.

I'm getting closer to bare metal. Ram is past bare metal now and the L1-L3 latency is less than bare metal. The speed is less I believe because I only have 2 numa involved and would be faster if I involved all the NUMA including the ones who don't have memory controller access. Physical Virtual Old virtual

Never got it to work.. there is a conflict that I posted about but no one more knowledgeable said anything back. I've never tinkered with the base bits of code for qemu but the post I found does a check and that check fails but it shouldn't and it's a bug. The person changed it slightly to pass the check for this nvme. I ended up just getting a Samsung.

Looking at your XML your CPU pairings are very strange Forgot your running at 2950x

I'll just say that if your splitting between NUMA with the EPYC stuff that will basically not work. The QEMU system doesn't provide the windows machine the NUMA information to function properly.

There are ways to do this but if I'm being 100% honest I highly suggest avoiding it. The reason I say that is because Unraid is made to deal with Xeons/Intel more than NUMA and threadripper and because of this if you use the settings that can get this basically perfect everytime and change the CPU count the VM simply disappears as unraid alters the CPU part of the config but doesn't update the settings(honestly I doubt it can as it doesn't have numatune) and the config becomes invalid and is dropped. Lost two damn XML configs because of that. Thats why my current gaming machine is not activated as I rebuilt from scratch and am still tweaking and don't want to have to waste the time to get it activated again.

I hope it turns out great for you.

Yeah sorry on that one

So you may be allocating ram from the other die so if your CPUs are all on a single die then you need to use the node-set parameter and set it to the node its running on. On a 2950x I'm guessing its just 0 or 1. You should also try the EPYC fix though your latency isn't nearly as high as mine was so you probably want to see if your cache is out of wack first. Mine definitely was. <numatune> <memory mode='static' nodeset='0'/> </numatune> Please read the below two posts from me earlier in the thread.

So.. its important to set so you don't go outside a core that has access to the memory. It will introduce stuttering into games with intense graphics. The one I test on is Dying Light. Before I did this little patch I was getting max 80fps. This patch skyrocketed my fps to 120 nearly consistent. Your L3 cache gets the largest boost of from 50ish to 10-11 and your memory latency usually drops from low 100's to very close to bare metal. Its amazing. I'm very glad you got better performance. I struggled with this for weeks/months. I tried literally everything I could think of and it really started to bother me. When I realize the cache was hosed I was baffled I missed that. I went back and read a guide I had read in the beginning https://tripleback.net/post/chasingvfioperformance/ <- this guy and realized he had stuck a patch on it and then I dug to find out what that patch did. I found some stuff related to that which I requested be added in the feature upgrades(if you could go over there and up vote them so we can get more attention). I was super happy that the person on Reddit provided me the CPU information like they did and the other provided me the code snippets(I honestly don't know how to apply them in Unraid though). Anyways I'm really am glad you guys are getting benefits. I'm a higher up server admin but my focus is ESXI/Windows/Netscalers/Netapp. It boggled my mind that I couldn't get the performance I was searching for out of this proc with all my experience with ESXI. Anyways lets keep digging and chasing that performance. We are within striking distance of baremetal but sadly please keep in mind. With this "fix" you CANNOT cross NUMA or you will get a pretty bad latency penalty because the OS no longer understands there is a numa and all my tweaking hasn't been able to get it to understand that it has a NUMA involved so it attempts to access memory on the other controller all the time. Once we can get a legit fix in though and use the name='topoext' only then we can pass NUMA information again I believe but SMT may still be broken. I know QEMU has 3.1.0-RC2 out.. Hopefully they've rolled all the fixes we need into it and when it comes out to 3.1.0 limetech will integrate it. Specifically and but I need to word this one better I believe.

As I just stated it is broken for threadripper. The version of QEMU we have are supposed to have the cache fixes and SMT fixes but it appears they are both missing for some reason. No idea why. There honestly is way to much confusion around it. It took me days of research on the EPYC tweak thing and apparently others claim they don't need it anymore so it just what version of QEMU 3.0 we are running.. With the EPYC cpu tweaks it gives me substantially less latency across the board please see my previous posts in this thread if you'd like to see more. Also its the closet to bare metal I've ever gotten btw.