Performance problem with a gaming VM

[Mrtj18]

13 minutes ago, ryanm91 said:

Yep all of us gamers seem to be losing frames bad lately. I am on Ryzen and AMD and still same issues. I did notice this morning in halo infinite bare metal I am average of about 20 percent CPU but in the VM I was 70 to 80 percent on same settings. I am using CPU passthrough but it seems like it's bottlenecking

Sent from my SM-G996U using Tapatalk
 

Yeah, I have attempted to pin CPU cores as well from the CPU pinning menu within unraid as well, and that didn't help my situation, either.. In MSI afterburner it shows my memory at 100℅ usage. Yet in games like cyberpunk 2077 and heroes of the storm my GPU usage gets capped at like 70%. So I can't take advantage of the high refresh rate monitor I have.

[ryanm91]

Yeah, I have attempted to pin CPU cores as well from the CPU pinning menu within unraid as well, and that didn't help my situation, either.. In MSI afterburner it shows my memory at 100℅ usage. Yet in games like cyberpunk 2077 and heroes of the storm my GPU usage gets capped at like 70%. So I can't take advantage of the high refresh rate monitor I have.

70 percent I get average of 40 percent GPU usage in my VM

I also can get 100 percent in a benchmark or stress test.

Sent from my SM-G996U using Tapatalk

[ryanm91]

Sooo have any of us tried disabling virtiofs which is new default in unRAID template?

Sent from my SM-G996U using Tapatalk

[DarkyCage]

2 minutes ago, ryanm91 said:

Sooo have any of us tried disabling virtiofs which is new default in unRAID template?

Sent from my SM-G996U using Tapatalk
 

Nop what is that ?

[ryanm91]

Nop what is that ?

Look in VM template you'll see virtiofs towards top turn it off. I was reading in the virtiofs thread that's pinned in this forum people were seeing freezing and performance loss.

Sent from my SM-G996U using Tapatalk

[Mrtj18]

I don't see it, am I missing something?

[ryanm91]

I don't see it, am I missing something?

unRAID share mode it's about mid screen.

Sent from my SM-G996U using Tapatalk

[DarkyCage]

11 minutes ago, Mrtj18 said:

I don't see it, am I missing something?

Same

[hot22shot]

You said that you pinned some CPU cores to the VM but did you also isolated them from the system ?

In my case I pinned half of my core to my gaming VM and isolated them from unraid so that they are fully dedicated to my VM.

 

You should not expect a lot of improvement about static hugepages, according to this link : https://wiki.archlinux.org/title/PCI_passthrough_via_OVMF#Huge_memory_pages

 

Quote

Note: The described procedures have some drawbacks but will not necessarily net you a great performance advantage. According to Red Hat benchmarks, you should not expect more than a 2% performance gain from this over #Transparent huge pages.

 

You can check if you use transparent huge pages by using a simple command, for example on my server :

 

root@XXXX:~# grep AnonHugePages /proc/meminfo
AnonHugePages:  31379456 kB

 

It matches the 32GB I have allocated to my 3 VM.

 

I suggest you read the whole chapter 5 on the link I provided there's a lot of good advice on CPU pinning for example and what core you should bind according to your CPU internal layout.

Edited November 30, 2022 by hot22shot

[ryanm91]

@hot22shot what about emulation cores and iothreads have you found those to be important. I've been seeing those more and more

Sent from my SM-G996U using Tapatalk

[hot22shot]

@ryanm91 I did not try to play with iothreads, I'm still trying to figure out if it useful in my case as I use an NVME disk in passthrough and therefore don't use any virtio-scsi or virtio-blk devices.

 

I do not know what do you mean with emulation core ?

[Mrtj18]

3 minutes ago, hot22shot said:

@ryanm91 I did not try to play with iothreads, I'm still trying to figure out if it useful in my case as I use an NVME disk in passthrough and therefore don't use any virtio-scsi or virtio-blk devices.

 

 

Same, I am passing through my nvme windows install as well. Mainly because so far, I still need it to run games at a high frame rate via dual boot, because of the issues I'm having getting the windows VM stable....

 

I will check out the guide to see if I have my CPU cores pinned correctly, because maybe it is a CPU issue

[ryanm91]

@ryanm91 I did not try to play with iothreads, I'm still trying to figure out if it useful in my case as I use an NVME disk in passthrough and therefore don't use any virtio-scsi or virtio-blk devices.
 
I do not know what do you mean with emulation core ?

I have seen an emulation core parameters on some advanced XML and people tend to point them to the core 0 pair. Was not sure if that helped at all. I am hoping when my new 3070ti comes Friday and SAS card this weekend I can re explore CPU pinning and see if I can get the performance to go from 50% to 80%.

My CPU usage is way higher and GPU usage way lower on my VM. Which I assume is why my frame rates are suffering.

Sent from my SM-G996U using Tapatalk

[hot22shot]

1 hour ago, Mrtj18 said:

I will check out the guide to see if I have my CPU cores pinned correctly, because maybe it is a CPU issue

 

Yes the guide is well written, I couldn't get any simpler explanation how to interpret the CPU layout.

 

1 hour ago, ryanm91 said:

My CPU usage is way higher and GPU usage way lower on my VM. Which I assume is why my frame rates are suffering.

 

Did you try to look at the CPU frequency of your core during your tests ? Not whithin the VM but in unraid ?

Just to be sure the frequency scales up to what it should.

[ryanm91]

 
Did you try to look at the CPU frequency of your core during your tests ? Not whithin the VM but in unraid ?
Just to be sure the frequency scales up to what it should.

I was hitting around 4.5ghz on passed through cores (my unRAID is set to performance scheduling) but I also just assigned the top 18 cores so maybe I need to pick my cores better on my 5900x. Something has to be causing the three of us in this thread to be bottlenecking our graphics cards. They are Intel and Im ryzen so it doesn't appear to be the classic Intel vs Ryzen issues. My c states are also disabled in the bios.

Sent from my SM-G996U using Tapatalk

[hot22shot]

15 hours ago, ryanm91 said:

but I also just assigned the top 18 cores so maybe I need to pick my cores better on my 5900x.

 

According to some random google search of mine, the 5900x core layout is like this :

 

lscpu -e (host):

CPU NODE SOCKET CORE L1d:L1i:L2:L3 ONLINE    MAXMHZ    MINMHZ      MHZ
  0    0      0    0 0:0:0:0          yes 4950.1948 2200.0000 2200.000
  1    0      0    1 1:1:1:0          yes 4950.1948 2200.0000 2200.000
  2    0      0    2 2:2:2:0          yes 4950.1948 2200.0000 2200.000
  3    0      0    3 3:3:3:0          yes 4950.1948 2200.0000 2200.000
  4    0      0    4 4:4:4:0          yes 4950.1948 2200.0000 2200.000
  5    0      0    5 5:5:5:0          yes 4950.1948 2200.0000 2200.000
  6    0      0    6 8:8:8:1          yes 4950.1948 2200.0000 2200.000
  7    0      0    7 9:9:9:1          yes 4950.1948 2200.0000 2200.000
  8    0      0    8 10:10:10:1       yes 4950.1948 2200.0000 2200.000
  9    0      0    9 11:11:11:1       yes 4950.1948 2200.0000 2200.000
 10    0      0   10 12:12:12:1       yes 4950.1948 2200.0000 2200.000
 11    0      0   11 13:13:13:1       yes 4950.1948 2200.0000 2200.000
 12    0      0    0 0:0:0:0          yes 4950.1948 2200.0000 2200.000
 13    0      0    1 1:1:1:0          yes 4950.1948 2200.0000 2200.000
 14    0      0    2 2:2:2:0          yes 4950.1948 2200.0000 3700.000
 15    0      0    3 3:3:3:0          yes 4950.1948 2200.0000 2200.000
 16    0      0    4 4:4:4:0          yes 4950.1948 2200.0000 2200.000
 17    0      0    5 5:5:5:0          yes 4950.1948 2200.0000 3237.737
 18    0      0    6 8:8:8:1          yes 4950.1948 2200.0000 2200.000
 19    0      0    7 9:9:9:1          yes 4950.1948 2200.0000 2200.000
 20    0      0    8 10:10:10:1       yes 4950.1948 2200.0000 2200.000
 21    0      0    9 11:11:11:1       yes 4950.1948 2200.0000 2200.000
 22    0      0   10 12:12:12:1       yes 4950.1948 2200.0000 2200.000
 23    0      0   11 13:13:13:1       yes 4950.1948 2200.0000 2200.000

 

You should try isolating & pinning all the CPUs that share L3 cache to maximize performances, so for example 6-18, 7-19 and up to 11-23.

Otherwise unraid could make use of the L3 cache for itself which would eventually lead to cache evictions and therefore bad performance.

 

Quote

my unRAID is set to performance scheduling

 

As I did for my 5800x, but I found something interesting :

 

Quote

Warning: Depending on your processor, it might actually be detrimental to performance to force the whole CPU to run at full frequency at all times. For instance, modern AMD processors (Zen 2 and Zen 3) depend heavily on being able to scale individual cores in separate core complexes for optimal thermal distribution. If the whole CPU is running at full frequency at all times, there is less headroom for invididual cores to clock high, resulting in worse performance for processes that are not heavily multithreaded such as games. This should be benchmarked in your virtual machine.

 

You could try to set it to "On Demand" : Default for ACPI CPU Freq driver. Scales CPU frequency based on load. Frequency stepping is aggressive.

Maybe you could hit a little bit more than 4.5ghz for a couple of core.

 

@DarkyCage Could you share the output of lscpu -e command on your unraid server ?

Edited December 1, 2022 by hot22shot

[Mrtj18]

@hot22shot So I have a AMD 3600x and I ran the lscpu -e command and I got the attached CPU config.... So for me should my CPU paring be like this : 1+7.... 0+6... 2+8... 3+9...4+10...5+11..? So should I dedicate 1+7 and 0+6 to unraid, and pin the rest to my windows VM?

Edited December 1, 2022 by Mrtj18

[ryanm91]

 
You could try to set it to "On Demand" : Default for ACPI CPU Freq driver. Scales CPU frequency based on load. Frequency stepping is aggressive.
Maybe you could hit a little bit more than 4.5ghz for a couple of core.
 
@DarkyCage Could you share the output of lscpu -e command on your unraid server ?

@hot22shot I really appreciate the insight. When my SAS card shows up I'll boot back into unRAID and see if I can find this bottleneck. I was thinking about it I had enabled MSI interrupts for my 1070ti but since swapping to the 6700xt I had not. My 3070ti comes Friday and so I'll see if I can get closer to bare metal performance and keep chasing the 85 to 90 percent.

Sent from my SM-G996U using Tapatalk

[hot22shot]

4 minutes ago, Mrtj18 said:

So I have a AMD 3600x and I ran the lscpu -e command and I got the attached CPU config.... So for me should my CPU paring be like this : 1+7.... 0+6... 2+8... 3+9...4+10...5+11..? So should I dedicate 1+7 and 0+6 to unraid, and pin the rest to my windows VM?

 

I would interpret your output like this :

 

 

To maximize performance you should associate the CPU that share the same cache level, in particular for L3 cache ("red" column).

So I would test with either the green or yellow (sorry for the messy drawing) group of CPU isolated & pinned to your VM.

 

You could add more CPU but they will share the L3 cache with CPUs not pinned to your VM.

 

 

[Mrtj18]

5 minutes ago, hot22shot said:

 

I would interpret your output like this :

 

 

To maximize performance you should associate the CPU that share the same cache level, in particular for L3 cache ("red" column).

So I would test with either the green or yellow (sorry for the messy drawing) group of CPU isolated & pinned to your VM.

 

You could add more CPU but they will share the L3 cache with CPUs not pinned to your VM.

 

 

Thanks for your assistance by the way....  So I should try grouping cores 0,1,2+6,7,8 to run with unraid or pin to windows VM correct?

[hot22shot]

1 minute ago, Mrtj18 said:

Thanks for your assistance by the way....  So I should try grouping cores 0,1,2+6,7,8 to run with unraid or pin to windows VM correct?

 

Yes that's it. Keep the cpus 0,1,2+6,7,8 for unraid and isolate and pin 3,4,5+9,10,11 for your Windows VM.

[Mrtj18]

18 minutes ago, hot22shot said:

 

Yes that's it. Keep the cpus 0,1,2+6,7,8 for unraid and isolate and pin 3,4,5+9,10,11 for your Windows VM.

Thanks for all your help, I updated my pinned snd core isolation selection, will update once I get home to run another benchmark on cyberpunk 2077.

[DarkyCage]

@hot22shot

[ryanm91]

@hot22shot

12-23, 36-47

Sent from my SM-G996U using Tapatalk

[Mrtj18]

Seems my CPU is not strong enough to isolate and pin according to what was discussed. I was getting highs of 39fps and lows as low as 12fps. I tried pinning and isolating what we discussed and flipping the pinned cores instead of 0,1,2 I used 3,4,5 and it was a no go, same results.

 

Looks like I would need to move my unraid setup to another machine to get my performance back, or leave some performance in the table till I can upgrade to a 8 core 16 thread processor. Because so far it can't keep up. My CPU was stuck at 100% utilization, for the whole benchmark.