Jerky_san

This is core pining for the CPU on that particular VM. You need to go to Settings, CPU Pinning and at the bottom "Core isolation" to isolate the cores from the rest of the system so it cannot use it.

Which part? The CPU thing is basically required for better performance/feel of the machine. The stock system doesn't detect cache right and so your VM will be running with all sorts of wonky cache. It also doesn't detect hyper threading right. Keep in mind that these settings can revert everytime you change something in the gui side instead of the XML side. The timer stuff is to lower CPU usage at idle and a slight increase in performance. CPU pinning is required to make sure Unraid doesn't use those cores and dockers don't either so you need to make that you did that. Lastly you have a 3970 so at least you don't have to deal with all the NUMA tuning crap like me with a 2990wx and others on the board. Though we've basically got that down to a science now as well.

Try what I said above and you should get within 90-95% of baremetal.

Ok couple things here right off the bat. You need to be on the latest unraid for this to work btw. <cpu mode='host-passthrough' check='none'> <topology sockets='1' cores='44' threads='1'/> </cpu> Needs to be <cpu mode='host-passthrough' check='none'> <topology sockets='1' cores='22' threads='2'/> <cache mode='passthrough'/> <feature policy='require' name='topoext'/> <feature policy='disable' name='monitor'/> <feature policy='require' name='hypervisor'/> <feature policy='disable' name='svm'/> <feature policy='disable' name='x2apic'/> </cpu> Need to set your clock to this <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='yes'/> </clock> Features to this below <features> <acpi/> <apic/> <hyperv> <vpindex state='on'/> <synic state='on'/> <stimer state='on'/> <reset state='on'/> <vendor_id state='on' value='KVM Hv'/> <frequencies state='on'/> </hyperv> </features> Finally if you have not. You need to reserve all your CPU cores that you will use for your VM. So make sure that is set in Settings/CPU Pinning. Please report back with a CPUz bench of your baremetal and a CPUz of your VM so I can further help you.

Seconded

lol sorry 😃 I've been researching all the changes that QEMU and virtio have been doing.

Coreinfo v3.31 - Dump information on system CPU and memory topology Copyright (C) 2008-2014 Mark Russinovich Sysinternals - www.sysinternals.com Logical to Physical Processor Map: **-------------- Physical Processor 0 (Hyperthreaded) --**------------ Physical Processor 1 (Hyperthreaded) ----**---------- Physical Processor 2 (Hyperthreaded) ------**-------- Physical Processor 3 (Hyperthreaded) --------**------ Physical Processor 4 (Hyperthreaded) ----------**---- Physical Processor 5 (Hyperthreaded) ------------**-- Physical Processor 6 (Hyperthreaded) --------------** Physical Processor 7 (Hyperthreaded) Logical Processor to Socket Map: **************** Socket 0 Logical Processor to NUMA Node Map: ********-------- NUMA Node 0 --------******** NUMA Node 1

I believe it primarily helps with stuttering. Basically what is very important is for you to tell the VM as much as you can about the host topology. By doing that you optimize numerous parts of the system. When it "assumes" things I've found it generally assumes wrong sadly. It is why if you look at my config I have all the NUMA stuff there. So the VM knows where the boundaries are and for what cores.

If you have a power meter. Measure the usage without logging out. Just sitting at the desktop. Then logout and see the usage drop. It is fairly be drop. Before at least on mine even with logging out the machine would be "idle" but in unraid you'd see lots of usage. Also if you look at my CPUZ pic earlier you'll see at the bottom it says "8 cores 16 threads"

https://github.com/qemu/qemu/blob/master/docs/hyperv.txt https://www.reddit.com/r/VFIO/comments/d5z0cb/kvm_high_idle_use_in_win10/

Updated and all went well thus far =0

I get the same thing but audio all works every time I restart. Never doesn't at least. Below is my entire configuration. Maybe you can find something different.. I pass a 1080ti bios, NVME, and sound card. I need to remove the junk again. Just noticed it is all in there again. 2020-01-07T15:31:47.954105Z qemu-system-x86_64: vfio: Cannot reset device 0000:0c:00.3, depends on group 22 which is not owned. 2020-01-07T15:31:49.125110Z qemu-system-x86_64: vfio: Cannot reset device 0000:0c:00.3, depends on group 22 which is not owned. <?xml version='1.0' encoding='UTF-8'?> <domain type='kvm' id='4'> <name>Windows 10</name> <uuid>99642e81-2f13-a916-682c-90191636d75f</uuid> <metadata> <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/> </metadata> <memory unit='KiB'>33554432</memory> <currentMemory unit='KiB'>33554432</currentMemory> <memoryBacking> <nosharepages/> </memoryBacking> <vcpu placement='static'>16</vcpu> <cputune> <vcpupin vcpu='0' cpuset='4'/> <vcpupin vcpu='1' cpuset='36'/> <vcpupin vcpu='2' cpuset='5'/> <vcpupin vcpu='3' cpuset='37'/> <vcpupin vcpu='4' cpuset='6'/> <vcpupin vcpu='5' cpuset='38'/> <vcpupin vcpu='6' cpuset='7'/> <vcpupin vcpu='7' cpuset='39'/> <vcpupin vcpu='8' cpuset='8'/> <vcpupin vcpu='9' cpuset='40'/> <vcpupin vcpu='10' cpuset='9'/> <vcpupin vcpu='11' cpuset='41'/> <vcpupin vcpu='12' cpuset='10'/> <vcpupin vcpu='13' cpuset='42'/> <vcpupin vcpu='14' cpuset='11'/> <vcpupin vcpu='15' cpuset='43'/> <emulatorpin cpuset='4-11,36-43'/> </cputune> <numatune> <memory mode='strict' nodeset='0,2'/> <memnode cellid='0' mode='strict' nodeset='0'/> <memnode cellid='1' mode='strict' nodeset='2'/> </numatune> <resource> <partition>/machine</partition> </resource> <os> <type arch='x86_64' machine='pc-i440fx-4.2'>hvm</type> <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader> <nvram>/etc/libvirt/qemu/nvram/99642e81-2f13-a916-682c-90191636d75f_VARS-pure-efi.fd</nvram> <boot dev='hd'/> </os> <features> <acpi/> <apic/> <hyperv> <vpindex state='on'/> <synic state='on'/> <stimer state='on'/> <reset state='on'/> <vendor_id state='on' value='KVM Hv'/> <frequencies state='on'/> </hyperv> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='1' cores='8' threads='2'/> <cache mode='passthrough'/> <feature policy='require' name='topoext'/> <feature policy='disable' name='monitor'/> <feature policy='require' name='hypervisor'/> <feature policy='disable' name='svm'/> <feature policy='disable' name='x2apic'/> <numa> <cell id='0' cpus='0-7' memory='16777216' unit='KiB'/> <cell id='1' cpus='8-15' memory='16777216' unit='KiB'/> </numa> </cpu> <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='yes'/> </clock> <on_poweroff>destroy</on_poweroff> <on_reboot>restart</on_reboot> <on_crash>restart</on_crash> <devices> <emulator>/usr/local/sbin/qemu</emulator> <controller type='ide' index='0'> <alias name='ide'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/> </controller> <controller type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/> </controller> <controller type='pci' index='0' model='pci-root'> <alias name='pci.0'/> </controller> <controller type='usb' index='0' model='qemu-xhci' ports='15'> <alias name='usb'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/> </controller> <interface type='bridge'> <mac address='52:54:00:23:bf:41'/> <source bridge='br0'/> <target dev='vnet0'/> <model type='virtio'/> <alias name='net0'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </interface> <serial type='pty'> <source path='/dev/pts/0'/> <target type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias name='serial0'/> </serial> <console type='pty' tty='/dev/pts/0'> <source path='/dev/pts/0'/> <target type='serial' port='0'/> <alias name='serial0'/> </console> <channel type='unix'> <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-4-Windows 10/org.qemu.guest_agent.0'/> <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/> <alias name='channel0'/> <address type='virtio-serial' controller='0' bus='0' port='1'/> </channel> <input type='tablet' bus='usb'> <alias name='input0'/> <address type='usb' bus='0' port='1'/> </input> <input type='mouse' bus='ps2'> <alias name='input1'/> </input> <input type='keyboard' bus='ps2'> <alias name='input2'/> </input> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x43' slot='0x00' function='0x0'/> </source> <alias name='hostdev0'/> <rom file='/mnt/user/domains/1080ti.rom'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x0c' slot='0x00' function='0x3'/> </source> <alias name='hostdev1'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x43' slot='0x00' function='0x1'/> </source> <alias name='hostdev2'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x42' slot='0x00' function='0x0'/> </source> <alias name='hostdev3'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x44' slot='0x00' function='0x3'/> </source> <alias name='hostdev4'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/> </hostdev> <memballoon model='none'/> </devices> <seclabel type='dynamic' model='dac' relabel='yes'> <label>+0:+100</label> <imagelabel>+0:+100</imagelabel> </seclabel> </domain>

Mine will pass fine though a time or two it has randomly dropped out completely with the same sort of reset your talking about but it doesn't happen often. I went Zenith because it was the only card people confirmed worked with an LSI9201 card at the time. I actually have an ASRock X399 and it didn't ;-;

Asus x399 Zenith Extreme. The hyper-v stuff is to lower cpu consumption at idle.. There is also a timer thing as well that makes HPET enabled. Windows 10 will idle higher when signed off and will never let the CPU enter lower power states if you don't have the tweaks..

If you say have 8 cores assigned. 4 "physical" cores and 4 SMT cores then you'd say 4 cores 2 threads. PLEASE KEEP IN MIND the below is for a 2990wx configuration. If you don't have a 2990wx you do not need the NUMA tune or NUMA stuff. <vcpu placement='static'>16</vcpu> <cputune> <vcpupin vcpu='0' cpuset='4'/> <vcpupin vcpu='1' cpuset='36'/> <vcpupin vcpu='2' cpuset='5'/> <vcpupin vcpu='3' cpuset='37'/> <vcpupin vcpu='4' cpuset='6'/> <vcpupin vcpu='5' cpuset='38'/> <vcpupin vcpu='6' cpuset='7'/> <vcpupin vcpu='7' cpuset='39'/> <vcpupin vcpu='8' cpuset='8'/> <vcpupin vcpu='9' cpuset='40'/> <vcpupin vcpu='10' cpuset='9'/> <vcpupin vcpu='11' cpuset='41'/> <vcpupin vcpu='12' cpuset='10'/> <vcpupin vcpu='13' cpuset='42'/> <vcpupin vcpu='14' cpuset='11'/> <vcpupin vcpu='15' cpuset='43'/> <emulatorpin cpuset='4-11,36-43'/> </cputune> <numatune> <memory mode='strict' nodeset='0,2'/> <memnode cellid='0' mode='strict' nodeset='0'/> <memnode cellid='1' mode='strict' nodeset='2'/> </numatune> <resource> <partition>/machine</partition> </resource> <features> <acpi/> <apic/> <hyperv> <vpindex state='on'/> <synic state='on'/> <stimer state='on'/> <reset state='on'/> <vendor_id state='on' value='KVM Hv'/> <frequencies state='on'/> </hyperv> </features> <cpu mode='host-passthrough' check='none'> <topology sockets='1' cores='8' threads='2'/> <cache mode='passthrough'/> <feature policy='require' name='topoext'/> <feature policy='disable' name='monitor'/> <feature policy='require' name='hypervisor'/> <feature policy='disable' name='svm'/> <feature policy='disable' name='x2apic'/> <numa> <cell id='0' cpus='0-7' memory='16777216' unit='KiB'/> <cell id='1' cpus='8-15' memory='16777216' unit='KiB'/> </numa> </cpu>

You won't get properly threading sent to the OS. My understand is the OS will not schedule as well if it doesn't know which are "hyper threaded" and are not. If you don't have SMT enabled then this is a moot point..

@testdasi@jbartlett Good news! I recently discovered that we no longer need the EPYC fix =0~~. Cache all comes through right and identifies as threadripper.. This should also translate for Ryzen as well. All you need to do is change it to something like below(of course adjusting cores). All cache shows up properly, CPUZ launches and everything, things seem slightly more responsive and latency tests show same as baremetal. Happy days! <cpu mode='host-passthrough' check='none'> <topology sockets='1' cores='8' threads='2'/> <cache mode='passthrough'/> VM

It's incredibly easy to use. I really enjoyed building my 2990wx system on it and it has great stability. I'd like another storage pool or to be able to add more disks to the 30 disk limit.

@limetech - https://unraid.net/blog/unraid-6-8 <- on the announcement there is a typo. "We started 6.9 development and initial testing" <- should be 6.8. Just wanted to point it out.. Thanks for the hard work btw

That's what mine is but minus the cache level piece. It emulates it properly as long as it passes through as EPYC so I don't use it.

Hmm what work around? I pass it through as an EPYC so it sees the L3 cache right and it does threading correctly that way as well. Also posted video of with and without but feel its not as defined. Both times the machine was just idle/logged in Forgot to mention your watching Core 2 <- Sorry core 2.. I really need to not cook and post on forums.. Bare metal VM

So the bump comes from the 1st core assigned to your VM. If it's a windows 10 machine it seems to consume a lot of kernel in htop. The timer fixes will cause it to drop. You won't see the utilization in the VM itself as it's large overhead happening in the hypervisor level. What you will see though is higher cpuz tests as you notice it does multi thread then tests the first core in windows. I'm curious to know if others see the same cpu throughput in cpuz that I see. Also I kind of wonder if its windows 10 version. I was going to make a video about the differences to show you but today I had updated to 1900 series and it seems the higher kernel usage is gone. So wonder if the scheduler changes brought other changes as well. I was running the 1800 series of windows yesterday when I was doing tweaks on this machine. Captured videos though the difference I feel is not as defined

<vcpu placement='static'>30</vcpu> <cputune> <vcpupin vcpu='0' cpuset='1'/> <vcpupin vcpu='1' cpuset='33'/> <vcpupin vcpu='2' cpuset='2'/> <vcpupin vcpu='3' cpuset='34'/> <vcpupin vcpu='4' cpuset='3'/> <vcpupin vcpu='5' cpuset='35'/> <vcpupin vcpu='6' cpuset='4'/> <vcpupin vcpu='7' cpuset='36'/> <vcpupin vcpu='8' cpuset='5'/> <vcpupin vcpu='9' cpuset='37'/> <vcpupin vcpu='10' cpuset='6'/> <vcpupin vcpu='11' cpuset='38'/> <vcpupin vcpu='12' cpuset='7'/> <vcpupin vcpu='13' cpuset='39'/> <vcpupin vcpu='14' cpuset='8'/> <vcpupin vcpu='15' cpuset='40'/> <vcpupin vcpu='16' cpuset='9'/> <vcpupin vcpu='17' cpuset='41'/> <vcpupin vcpu='18' cpuset='10'/> <vcpupin vcpu='19' cpuset='42'/> <vcpupin vcpu='20' cpuset='11'/> <vcpupin vcpu='21' cpuset='43'/> <vcpupin vcpu='22' cpuset='12'/> <vcpupin vcpu='23' cpuset='44'/> <vcpupin vcpu='24' cpuset='13'/> <vcpupin vcpu='25' cpuset='45'/> <vcpupin vcpu='26' cpuset='14'/> <vcpupin vcpu='27' cpuset='46'/> <vcpupin vcpu='28' cpuset='15'/> <vcpupin vcpu='29' cpuset='47'/> <emulatorpin cpuset='1-15,33-47'/> </cputune> <numatune> <memory mode='strict' nodeset='0,2'/> <memnode cellid='0' mode='strict' nodeset='0'/> <memnode cellid='1' mode='strict' nodeset='2'/> </numatune> The bottom part of the top code snippet is the part that does the back end <numatune> stuff then at the processor you need something like this below. The cell id's correspond to each other. The bottom is what the VM sees but it is also how the VM will be allocated resources <cpu mode='custom' match='exact' check='full'> <model fallback='forbid'>EPYC</model> <topology sockets='1' cores='15' threads='2'/> <feature policy='require' name='topoext'/> <feature policy='disable' name='monitor'/> <feature policy='require' name='hypervisor'/> <feature policy='disable' name='svm'/> <feature policy='disable' name='x2apic'/> <numa> <cell id='0' cpus='0-13' memory='16777216' unit='KiB'/> <cell id='1' cpus='14-29' memory='16777216' unit='KiB'/> </numa> </cpu> Now to get more performance you might want to look at htop.. You'll notice the first core is always using lots of kernel if your using windows 10. To fix that change your <clock> to this below. This will really help your CPUZ score along with a noticeable boost in performance. You'll also notice the kernel in htop has gone away substantially. <clock offset='localtime'> <timer name='hypervclock' present='yes'/> <timer name='hpet' present='yes'/> </clock> Overtime I also did these below for Hyper-v <hyperv> <vpindex state='on'/> <synic state='on'/> <stimer state='on'/> <reset state='on'/> <vendor_id state='on' value='KVM Hv'/> <frequencies state='on'/> </hyperv> I was going to write a little guide on how to make Ryzen work as well as possible. If any of you that have 2990's and 2950's please tell me if these settings replicate into your VM's as well. I can tell you that my non fresh VM didn't take the CPU boosts nearly as well as my fresh test VM. This is within 5-10% of bare metal though which is very good in my book.

I pass NVMEs to it.. I was more or less referring to spanning multiple NUMA without performance problems and some minor hitching in games though last night I think I made a break through. I was wondering what kind of performance you all get with your 2990wx's when spanning two NUMA? Last night I was playing around with my secondary VM's config and I decided to work on the back-end numa presentation more. My problem has always been that I couldn't get memory to span properly between the two numa and I think I finally made a major breakthrough on that. Ironically when I started the machine I accidentally killed my current one because there wasn't enough ram on NUMA 0 to start the new VM with my config I set lol. So here is my documentation I took with it.. If you guys believe I made some progress then I'll write a better guide about it. Also the question was more because if you were going to get one or someone on here I was going to wait and see how good they perform before I consider jumping but the 80 platform interest me to if they support ECC. Opinions welcome - please be gentle ^_^; Configration 2900wx Asus Zenith X399 128 gb of ram G.Skill samsung B-Die 20 Docker's running - Including Plex with 4 transcodes happening during the VM tests. PDO - LVL 3 is enabled for all tests. - These tests for the VM reflect my settings that physically divide NUMA node memory by what I state. One node will always have slightly more due to Hyper visor overhead(I assume) - 117809 machine Baremetal before ram OC Ram is OC'd to 2800 14-14-14-34 CR1 with very tight timings. - Still Baremetal - Note Latency of memory changes to some degree between 69-74ns - same goes with read/writes Last night with XML changes to backend NUMA presentation - L3 reads are low for some reason.. Not figured this out yet but could be that 1 of my cores is always being used for Unraid CPUZ baremetal CPUZ with 2 NUMA - 30 cores - 1 SMT core reserved for unraid Core 0/31 - Test taken while plex was being utilized on other cores so PBO was unable to OC as high I believe. NUMA 0(0/31 missing) & NUMA 2(Fully allocated)

@authorleon yeah #_# i have two x399's thinking that I could use the Taichi for the next jump.. That was a mistake..