Performance Improvements in VMs by adjusting CPU pinning and assignment

December 26, 20196 yr

On 12/14/2019 at 12:18 AM, jbartlett said:

I have not but AMD agrees with you on the latency issues having to utilize the Infinityfabric. They're not creating multiple numa nodes for TR3.

If someone needs to do some testing, I've got a TRX4 up and running. Let me know.

Quote

January 3, 20206 yr

Hello, i'm kinda new on KVM and virtualization. And mostly new on Threadrippers. I made a build a month ago and just found out what was the cause of my unstable system (RAM XMP... ) anyway. Now its rock solid.
And trying to optimize my daily driver VM and dockers with cpu pinning.
I read first pages of this thread but i got more confused in the end
I would really like some of your suggestions on my setup and i could give back any info/tests if you want!

Mostly need help on which cores to assigned based on the CPU topology and making modifications on XML about cpu topology and ram assigment.

My CPU is: Threadripper 2920x (12C24T) - GPU RX5700 8GB- 32GB Ram
My setup / needs are:

Well Unraid - It should be good with 2 Cores i guess, i dont have more than 2 people copying from/to array and of them is me from the VM!

VMS
*Windows VM for Gaming with NVME on unassigned devices(not on cache) and RX5700 (Damn Navi bug.. ) - Will run only while gaming - 6 Cores/16gb ram? Had issues on Borderlands 3 when running under 18GB Ram. Was running out of memory.

*Linux VM for daily use (Not need much power and soon will have a cheap gpu passed) 2 cores/4GB Ram should be ok - Will run 24/7 - 4GB Ram should be ok

*pfSense VM as router/firewall - 2 Cores/4GB Ram should be enough. - Will run 24/7

Lightweight Dockers

(those should all run fine with 1 core) - dont care if the slow down, as those dockers dont do anything important.

*sonarr

*radarr

*bazarr

*tautulli

*jackett

*deluge

*duckdns

*MQTT

*Home-assistant

*unifi-controller

Heaveweight Dockers

*Handbrake - This will never run while gaming so it can be assigned the same cores as gaming vm.

*Plex - All my files are x264 so not much transcoding needed. Max 3 streams concurrently and 2 of the local.

CPU Pairs as taken from VM Editor of Unraid

CPU 0 - HT 12

CPU 1 - HT 13

CPU 2 - HT 14

CPU 3 - HT 15

CPU 4 - HT 16

CPU 5 - HT 17

CPU 6 - HT 18

CPU 7 - HT 19

CPU 8 - HT 20

CPU 9 - HT 21

CPU 10 - HT 22

CPU 11 - HT 23

Thanks! If i should add more info, please tell me what is this you need!

Edited January 3, 20206 yr by skois

Quote

January 10, 20206 yr

Thanks, worked out ... wondered me a little cause i thought i made that before and had errors on my vm´s when trying to start them,

must been a failure by me ... working actually a little better now when the gaming vm is in use so the other can be used a little better ... tubemate get-mobdro.com authorityapk.com

Edited March 27, 20206 yr by johnsmith101

Quote

January 10, 20206 yr

Did some on/off node testing with a TR 2990WX (off node = no direct access to ram/devices). Configured a VM with four numa nodes with two CPU's each (4 with HT) and 4 GB of RAM (1 GB per VM numa node), all set to hit against the same physical numa node. Tests were done on UNRAID 6.8.1 RC1 with the TR being passed through and the guest seeing both the hyperthreaded CPUs & NUMA nodes. Benchmarks were Cinemark R20 for CPU and AIDA64 for memory. Each score is an average of 5 runs with outlayers dropped (tested too far off the variance). Each pass moved a VM node from a numa node with direct access to one without.

On Numa   R20   Read  Write   Copy  Latency
4        3663  31944  32558  32159     93.5
3        3645  30786  30483  30292     93.7
2        3587  27387  29982  27557     93.6
1        3607  19153  19809  20805     93.5
0        3526  14693  15097  17033    162.1

CPU scores saw a diminishing gain as the CPU's got moved off to a numa node without direct memory access as expected by the evenly spread score was surprisingly low. Memory scores shows a clear benefit to having at least one CPU on node in memory access with the times being negatively impacted nearly 100% being completely isolated.

Quote

1

January 22, 20206 yr

On 11/13/2019 at 9:02 AM, bastl said:

It looks like it only works if you pass trough extra infos for the cache like I did.

https://git.qemu.org/?p=qemu.git;a=commit;h=7210a02c58572b2686a3a8d610c6628f87864aed

https://www.reddit.com/r/VFIO/wiki/known_issues#wiki_enabling_smt_on_amd_processors_with_qemu_3.1.2B

I have been using unRAID as my daily driver, and only computer, for about 14 months now. Getting my windows 10 VM to recognize this SMT just gained me like 25% performance, and fixed my microstutter problem in games!

Quote

March 27, 20206 yr

What are the best settings for Ryzen 1700X on a Asus Crosshair X470 and latest bios?

I’d like to move to using only VM as my daily workstation but haven’t been able to get the single core performance good enough for gaming for example. With RX 5700 XT I lose about 30-40% of it’s performance when compared to a baremetal and I think it’s mostly because of this reason. I benchmarked my CPU with Cinebench 15 and while multithreading it’s actually pretty close to a baremetal, the single core performance is really poor.

So... I’ve only isolated and pinned the last cores (4/8)to a fresh Windows 10 VM using Unraid GUI and left the first cores for Unraid. What else do I have to do to improve CPU performance? I hope I could get this sorted out so I can order 3900X and really go all in.

Quote

September 14, 20205 yr

So, I have done two windows vm's on my xeon 10core 3.1ghz.

I have arranged the core assignment, as indicated in the op.

I can't figure out how to change from SeaBIOS. It is grayed out. I am running Nvidia cards, so maybe not good to switch??

Also not sure how to isolate the first two cores for the server. Is that with Docker?

It is mentioned that you shouldn't start with more than 8 gigs. I had already assigned more than that, given that I have 64.

From "pushing the power button" to both vm's ready for log in, we are looking at 6.5min. Seems like a long time

The following are my two machine set up's

Edited September 14, 20205 yr by 22350
MISSING IMAGE

Quote

October 22, 20205 yr

Hi, I have a question, I created a Win 2008 Server VM.

For installing, I used 1 core no HT, but after the installation, I try to add more cores and threads, and then the system detects a "hardware" change and becomes useless.

Any Idea of how to fix it?

Thanks!

Quote

October 22, 20205 yr

14 hours ago, rojarrolla said:

Hi, I have a question, I created a Win 2008 Server VM.

For installing, I used 1 core no HT, but after the installation, I try to add more cores and threads, and then the system detects a "hardware" change and becomes useless.

Any Idea of how to fix it?

Thanks!

That's a Win2008 Server issue and unrelated to KVM. Try reinstalling with the desired number of cores or try to re-register.

Quote

October 23, 20205 yr

8 hours ago, jbartlett said:

That's a Win2008 Server issue and unrelated to KVM. Try reinstalling with the desired number of cores or try to re-register.

Thanks I tried reinstalling with more cores but the screen remains black. How can I re-register it?

cheers!

Quote

October 23, 20205 yr

2 hours ago, rojarrolla said:

Thanks I tried reinstalling with more cores but the screen remains black. How can I re-register it?

cheers!

Try installing it with one core but after the first reboot during the install, update the core count before Windows locks things down.

Quote

October 23, 20205 yr

11 hours ago, jbartlett said:

Try installing it with one core but after the first reboot during the install, update the core count before Windows locks things down.

Hello, Thanks, I've tried as you said. There are 2 reboots when installing, I tried in each one of them and the result is the same, I get the message from windows telling me that the system has changed.

There should be other way, I'll keep searching

Quote

January 11, 20215 yr

On 1/21/2020 at 8:47 PM, tsawind said:

I have been using unRAID as my daily driver, and only computer, for about 14 months now. Getting my windows 10 VM to recognize this SMT just gained me like 25% performance, and fixed my microstutter problem in games!

Hey, I've been having a microstuttering problems. What did you do to fix this? I have a 3900X.

Quote

February 16, 20215 yr

Hi,

i have a AMD Ryzen 7 5800x and i assigned 4 Cores + 4 HT Cores. But i have a really bad perfomance on it, i measure it with 3dmark.

Baremetal: 12.329 Score

Unraid VM: 5.820 Score

I isolated the cores and made the cpu pinning.

Can someone help me?

This is also my config:

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm' id='4'>
  <name>Windows 10</name>
  <uuid>508ccb34-ad84-6c40-60c5-aec62d8f2b67</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>20971520</memory>
  <currentMemory unit='KiB'>20971520</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='4'/>
    <vcpupin vcpu='1' cpuset='12'/>
    <vcpupin vcpu='2' cpuset='5'/>
    <vcpupin vcpu='3' cpuset='13'/>
    <vcpupin vcpu='4' cpuset='6'/>
    <vcpupin vcpu='5' cpuset='14'/>
    <vcpupin vcpu='6' cpuset='7'/>
    <vcpupin vcpu='7' cpuset='15'/>
    <emulatorpin cpuset='0,11'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-5.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/508ccb34-ad84-6c40-60c5-aec62d8f2b67_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='2D76A8B352E2'/>
    </hyperv>
    <kvm>
      <hidden state='on'/>
    </kvm>
    <ioapic driver='kvm'/>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='4' threads='2'/>
    <cache mode='passthrough'/>
    <feature policy='require' name='topoext'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Windows 10/vdisk1.img' index='1'/>
      <backingStore/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <alias name='virtio-disk2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:f9:83:1c'/>
      <source bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio-net'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-4-Windows 10/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='4'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <rom file='/mnt/user/appdata/VBIOS/MSIRTX3070_MOD.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc52b'/>
        <address bus='1' device='5'/>
      </source>
      <alias name='hostdev1'/>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc539'/>
        <address bus='1' device='4'/>
      </source>
      <alias name='hostdev2'/>
      <address type='usb' bus='0' port='2'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x048d'/>
        <product id='0x5702'/>
        <address bus='1' device='2'/>
      </source>
      <alias name='hostdev3'/>
      <address type='usb' bus='0' port='3'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+100</label>
    <imagelabel>+0:+100</imagelabel>
  </seclabel>
</domain>

Quote

February 16, 20215 yr

58 minutes ago, Rexima said:

Can someone help me?

Try reducing the number of cores and memory assigned to the VM to a bare minimum and benchmark again. Something like 2 cores and 8GB RAM.

Quote

February 16, 20215 yr

With 2 Cores and 8GB Ram, the CPU Score is now at 1.510

Quote

February 16, 20215 yr

1 minute ago, Rexima said:

With 2 Cores and 8GB Ram, the CPU Score is now at 1.510

Increase core count in small steps, benchmarking each step.

Quote

1

February 16, 20215 yr

5 hours ago, Rexima said:

Can someone help me?

Set it to a single non-zero CPU or just remove. I've never seen it utilize the HT pair. Free up CPU 0 & it's HT partner from isolation to let UNRAID use it.

Quote

1

February 21, 20215 yr

Thanks, this helped me a lot

Quote

April 6, 20215 yr

Hi I have very low fps on bf v i only get 14-20 fps on low settings

I have a dual xeon e5-2860 2,8ghz 112 gb ram

my wm have 12 cores and 16 gb and 3060 ti

here are my xml settings for right now

I have try more cpu less cpu more ram less ram

but in loading meny i have over 250 Fps

and in begining I have 35-40 fps but it drops down to 14-20 after 5 min playing

i have check temps on cpu it only 55 C

<?xml version='1.0' encoding='UTF-8'?>

<name>Windows 10 test</name>

</metadata>

</memoryBacking>

</cputune>

</numatune>

<partition>/machine</partition>

</resource>

<os>

<loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>

<nvram>/etc/libvirt/qemu/nvram/b04e314d-9a1b-182a-3e0e-ce3d8fd55bfa_VARS-pure-efi.fd</nvram>

</os>

<acpi/>

<apic/>

<vendor_id state='on' value='none'/>

</hyperv>

</features>

</cpu>

</clock>

<on_poweroff>destroy</on_poweroff>

<on_reboot>restart</on_reboot>

<on_crash>restart</on_crash>

<emulator>/usr/local/sbin/qemu</emulator>

</disk>

</disk>

</disk>

</disk>

</controller>

</controller>

</controller>

</controller>

</controller>

</controller>

</controller>

</controller>

</controller>

</controller>

</controller>

</controller>

</controller>

</controller>

</controller>

</controller>

</controller>

</interface>

</target>

</serial>

</console>

</channel>

</input>

</input>

</source>

</hostdev>

</source>

</hostdev>

</source>

</hostdev>

</source>

</hostdev>

</source>

</hostdev>

</devices>

</seclabel>

</domain>

Quote

May 21, 20215 yr

Hi folks,

just wanted to share what helped for me.

Short Answer

Go to Windows sound manager and play around with the sample rate of your audio output (try 44kHz and 48kHz in particular)

Long Answer

After trying the CPU pinning and emulator pinning as suggested (great hint btw.), nothing seemed to work and it even got worse as my audio engine begin to lag for 4-20 seconds every minute PLUS the cracking and stuttering which had been present from the beginning on. I don't know exactly why, but somehow the VM seems to have struggles with down-sampling the audio, so I set all inputs and outputs to the highest sample rate (48000 Hz) and everything works fine.

The setup:

hyper-threading disabled
No cores isolated in the boot config
Cores 0-7 pinned to VM
Emulator tasks pinned to cores 8,9
All audio devices use 48kHz

I hope that someone can benefit from this minor discovery...

best wishes,

TheClockTwister

Quote

May 21, 20215 yr

for audio issues i prefer this on my windows VM's

https://forums.guru3d.com/threads/windows-line-based-vs-message-signaled-based-interrupts-msi-tool.378044/

Quote

September 21, 20214 yr

Anyone seen constant high cpu utilization on 0 cpu thread pair after pinning emulator to it? Is this okay to have?

image.png.d5d18fce0d3123cdce3aac0085f2a2a0.png

Quote

September 21, 20214 yr

Quote

December 31, 20214 yr

I have i7-10510U with 4C/8T with underwhelming performance.

Pairs doesn't go like 0-4 or 1,5... Rather it's like:

Quote

0,1

2,3

4,5

6,7

Problem is when I isolate last 3 cpu's giving emulatorpin+iothread to the first cpu (0,1) I get worse performance than doing (0,4) or EVEN (0,5).

How can this happen?

Edited December 31, 20214 yr by dqmhug

Quote

Performance Improvements in VMs by adjusting CPU pinning and assignment

Featured Replies

Top Posters In This Topic

Popular Days

Most Popular Posts

jbartlett

david279

SpaceInvaderOne

Posted Images

Join the conversation

Top Posters In This Topic

Popular Days

Most Popular Posts

jbartlett

david279

SpaceInvaderOne

Posted Images

Account

Navigation

Search

Configure browser push notifications

Chrome (Android)

Chrome (Desktop)

Safari (iOS 16.4+)

Safari (macOS)

Edge (Android)

Edge (Desktop)

Firefox (Android)

Firefox (Desktop)