Desperate for help / interesting case regarding gaming VM

[nasuellia]

27 minutes ago, testdasi said:

Now that I think about it, that might very well be possible. There are 2 machine types available: Q35 and i440fx. Both are Intel chipsets so it's entirely possible that LBRV is not being emulated. That would sort of explain why you have similar performance in Win 8.1 VM, which supposedly isn't affected by the issue.

When you have a couple minutes maybe check CPUz on your Windows guest, it should provide you with the comprehensive list of CPU-features; if LBRV isn't listed, we found the potential issue.

 

Unless I'm mistaken, i440fx and Q35 should not affect which CPU-flags are (or aren't) passed to the guest, I believe those are two archetypes of chipset emulation. I believe what commands the CPU-features is the part where we can chose host-passthrough or qemu-emulated, and I believe there's way, through direct XML editing, to add flags manually instead. 

 

If that's the case, it's a way. But this is all conjecture, maybe LBRV is already passed and working, and the root cause is simply not that. That's certainly possible... 

 

Let me know if you have the time to check CPUz in the next days.

 

Let me be jolly for a second: how did the testing go, as far as fighting? Eheh, did you die the first time around? Did you enjoy the matches vs AIs to some degree? I hope it wasn't too boring for you! 

 

Thanks again for helping and by the way, wow... what a rig you got going on there... it's nothing short of beastly.

 

PS: by the way, Intel processors do not support LBR virtualization at all, so in any case, my system would still suffer from the issue, regardless of the operating system.

Edited September 20, 2018 by nasuellia

[TType85]

1 minute ago, nasuellia said:

When you have a couple minutes maybe check CPUz on your Windows guest, it should provide you with the comprehensive list of CPU-features; if LVRV isn't listed, we found the potential issue.

Here is mine

Socket 1            ID = 0
    Number of cores        1 (max 1)
    Number of threads    8 (max 
    Name            AMD Ryzen Threadripper 1950X
    Codename        Threadripper
    Specification        AMD Ryzen Threadripper 1950X 16-Core Processor 
    Package         
    CPUID            F.1.1
    Extended CPUID        17.1
    Core Stepping        ZP-B1
    Technology        14 nm
    TDP Limit        180.0 Watts
    Core Speed        3393.0 MHz
    Instructions sets    MMX (+), SSE, SSE2, SSE3, SSSE3, SSE4.1, SSE4.2, SSE4A, x86-64, AMD-V, AES, AVX, AVX2, FMA3, SHA
    Microcode Revision    0x1000065
    L1 Data cache        8 x 64 KBytes, 2-way set associative, 64-byte line size
    L1 Instruction cache    8 x 64 KBytes, 2-way set associative, 64-byte line size
    L2 cache        8 x 512 KBytes, 16-way set associative, 64-byte line size
    L3 cache        8 x 16 MBytes, 16-way set associative, 64-byte line size
    Max CPUID level        0000000Dh
    Max CPUID ext. level    8000001Ah
    Cache descriptor    Level 1, I, 64 KB, 1 thread(s)
    Cache descriptor    Level 1, D, 64 KB, 1 thread(s)
    Cache descriptor    Level 2, U, 512 KB, 1 thread(s)
    Cache descriptor    Level 3, U, 16 MB, 1 thread(s)
    FID/VID Control        yes

[nasuellia]

14 minutes ago, TType85 said:

Here is mine

[large paste removed]

Aaaaaaaand there is no sign of LBRV on the guest...

It's certainly possible KVM/LibVirt still didn't come around to implement LBRV on the software side.

Alternatively (and hopefully) it's just a matter of flags-passing, and we can find a way to add the flag.

 

Do you guys know of someone really (really) experienced we can ask for? Maybe SpaceInvaderOne? I might drop a PM on him tomorrow...

 

Thanks for posting this, it would still be interesting to see your results, if you ever take the time.

[nasuellia]

I found this: https://libvirt.org/formatdomain.html

 

which appears to explain it all about XML editing for the CPU (section: CPU model and topology)

 

I'm literally falling asleep on the keyboard right now, I'll give it a read tomorrow and maybe we can actually try to manually specify the flag. It won't work on my system, but it might make a difference with AMD processors on certain games.

[TType85]

Only had a minute to test before heading to dinner, set it to max, 2560x1440 

 

20-09-2018, 16:50:45 HeroesOfTheStorm_x64.exe benchmark completed, 12534 frames rendered in 93.906 s
                     Average framerate  :  133.4 FPS
                     Minimum framerate  :  114.2 FPS
                     Maximum framerate  :  153.2 FPS
                     1% low framerate   :   70.2 FPS
                     0.1% low framerate :   21.2 FPS
 

Will try 5 min later.  CPU usage is maxing on one core mostly, just like World Of Warcraft.

[nasuellia]

7 hours ago, TType85 said:

Only had a minute to test before heading to dinner, set it to max, 2560x1440 

 

20-09-2018, 16:50:45 HeroesOfTheStorm_x64.exe benchmark completed, 12534 frames rendered in 93.906 s
                     Average framerate  :  133.4 FPS
                     Minimum framerate  :  114.2 FPS
                     Maximum framerate  :  153.2 FPS
                     1% low framerate   :   70.2 FPS
                     0.1% low framerate :   21.2 FPS
 

Will try 5 min later.  CPU usage is maxing on one core mostly, just like World Of Warcraft.

Ok it's pretty evident at this point that the performance-crippling issue is present on AMD processors as well, at least by default. Thanks a lot for the test!

 

I'm about to post another reply with the last resort...

 

[nasuellia]

From what I understand after reading the libvirt documentation (https://libvirt.org/formatdomain.html), there is a way to "force" or at least "require to try" supporting a feature on the guests (obviously assuming the host does).

 

Adding something like following in the XML under the  <cpu></cpu> tag, might do the trick:

<feature policy='require' name='lbrv'/>

 

or maybe

 

<feature policy='force' name='lbrv'/>

 

I also have no idea whether 'lbrv' needs to be in uppercase 'LBRV', it's certainly possible.

 

 

If Threadripper possessors can create a throwaway VM template and try to boot with it, it would be the last chance to figure out this issue...

[Jcloud]

51 minutes ago, nasuellia said:

<feature policy='require' name='lbrv'/>

 

or maybe

 

<feature policy='force' name='lbrv'/>

Tried, wouldn't run. Didn't like either 'lbrv' or 'LBRV'

[nasuellia]

5 minutes ago, Jcloud said:

Tried, wouldn't run. Didn't like either 'lbrv' or 'LBRV'

That's so sad.

Regardless of HotS, AMD users are missing out on a virtualization technology... and Intel is entirely disregarding it altogether. That is really weird.

Maybe the line was improperly built, I hope someone with actual knowledge of this stuff can chip in.

[testdasi]

11 hours ago, nasuellia said:

Unless I'm mistaken, i440fx and Q35 should not affect which CPU-flags are (or aren't) passed to the guest, I believe those are two archetypes of chipset emulation. I believe what commands the CPU-features is the part where we can chose host-passthrough or qemu-emulated, and I believe there's way, through direct XML editing, to add flags manually instead. 

It's not uncommon for a CPU feature to also require motherboard (e.g. chipset) support. Intel VT-x and AMD SVM are 2 well-known examples.

So I am not surprised at all that these chipsets emulation may not support LBRV, causing it to not be exposed to the VM. As mentioned, that fits perfectly with the fact that you have the same issue in Windows 8.1, which supposedly does not have the problem.

 

This is kinda similar to your effort to force the feature on the xml. Some tags are theoretically supported by qemu / libvirt etc. but it doesn't mean it would work.

One recent example (for unRAID) is the numatune tag (at least for Threadripper).

• Any attempt to manually tune RAM using numatune tag will crash libvirt entirely and/or cause nvram corruption.
• Automatic numatune setting doesn't work because numad is not included in unRAID packages.

 

So I guess you might want to consider whether 60 fps average on an actual game sufficient.

You want VM to be 99% BM performance. You don't need VM to be 99% BM performance.

 

 

 

[GHunter]

I know most people use Host Pass Through mode as the CPU Mode when setting up their VM, but have you tried setting up a VM with Emulated (QEMU) Mode? Maybe that will help you in your situation. I'm really not sure but just offering another suggestion.

[nasuellia]

4 minutes ago, testdasi said:

It's not uncommon for a CPU feature to also require motherboard (e.g. chipset) support. Intel VT-x and AMD SVM are 2 well-known examples.

So I am not surprised at all that these chipsets emulation may not support LBRV, causing it to not be exposed to the VM. As mentioned, that fits perfectly with the fact that you have the same issue in Windows 8.1, which supposedly does not have the problem.

 

This is kinda similar to your effort to force the feature on the xml. Some tags are theoretically supported by qemu / libvirt etc. but it doesn't mean it would work.

One recent example (for unRAID) is the numatune tag (at least for Threadripper).

• Any attempt to manually tune RAM using numatune tag will crash libvirt entirely and/or cause nvram corruption.
• Automatic numatune setting doesn't work because numad is not included in unRAID packages.

 

So I guess you might want to consider whether 60 fps average on an actual game sufficient.

You want VM to be 99% BM performance. You don't need VM to be 99% BM performance.

 

 

 

Unfortunately Q35 was already tested before my post, and did not not change anything. 

 

At this point, I'm gonna mess around with VMs for a bit more time, because they are awesome and I'm learning quite a lot of interesting stuff, but in the end, I will go back to bare metal for consistency reasons.

 

I can only thank you for the extended and precious contribution, you're really great man, I can't thank you enough.

 

That said, let me respectfully disagree with your conclusion:

 

it's not a matter of wanting or needing 99% BM performance:

 

- I want my VM to have predictable performance, and not erratic; otherwise I will never know if it's performing correctly or not, which would make spending money on hardware kind of a shaky proposition. As I said on the original post, I expected a general, predictable, measurable drop in performance, and I am willing to accept it, maybe even 15% across the board. On the other hand, I am not up for accepting erratic unpredictable behavior caused by unknown, not fixable issues of this magnitude.

 

- You say that I might have to accept 60 FPS as acceptable, but my system actually often drops to 30 and 40 in real matches online; which is definitely not enjoyable.

 

- The actual reason to do this whole VM thing, was to play locally with friends when they come over. So imagine such an already borderline situation, getting even worse when I split CPU and RAM across two VMs... I already tried it, it's not enjoyable, at all.

 

-- Even assuming it would do 60 FPS stable (which it doesn't as just explained) I am used to 144Hz and therefore 60 FPS is actually quite poor to my perception. It's not a matter of being just picky either, it's that I bought my hardware in accordance and spent money for having 144 FPS, so anything that bars the way to that, is not acceptable.

 

-Imagine I commit to the VM project, accepting the compromise that some apps will run like crap. I spend quite a bit of money with a new processors with more cores, definitely more RAM, and probably more disk space. Then, 6 months from now, another game comes out, a much more modern and performance intensive then HotS in the first place, and it turns out, that it suffers from the same problem... I would find myself with an outrageously expensive machine that can't run that game.

 

- Who knows if this issue applies to other stuff? It's not a matter of accepting playable frame-rate on this particular game and forget about it: what if it turns out that some productivity stuff (that I also need) is also being severely degraded by this problem? What if it turns out that gradle compile times are 50% slower on VM with "whatever" particular specific situation? I'm not up for accepting that, I don't want to fear having halved performance and losing productivity without even knowing, in the future.

[testdasi]

While I agree the issue is annoying to say the least, it's somewhat blown out of proportion.

• First and foremost, you mentioned a lot of "what if" scenario with upgrading. Your original build is not good enough to begin with, at least to the ultra high standard you seem to be expecting, so you have to upgrade regardless. The 6700K, in the most ideal situation, can run 1 2-core VM + 1 1-core VM (leaving the remaining core for emulation and unRAID stuff). I would be astonished if the 1-core VM is not bottle-necked.
• If productivity i.e. work is critical for you then you shouldn't be mixing it with gaming to begin with. You should be building a workstation and then if it can game then it's a bonus. Here you are building a gaming server while wondering the "what if".
• You mentioned you are used to 144Hz, which is fair enough. But that's like having a sports car. You don't drive a sports car pulling a trailer and still expect it to be fast. In other words, if 144Hz is that important to you, you have embarked on the wrong journey to expect the same performance on a VM with ALL games. Even bare metal does not have the same performance with all games, it's not reasonable to expect the same on a VM.
• 30 min frame rate, for a DOTA clone, at ultra settings, is nothing to complain about. So if your friends also need 144Hz then if they come over often enough then you will just have to give each of them a BMW PC, so to speak.

 

Last but not least, I think there's a potential solution here. You can dual boot. On normal days, boot into Windows BM and play to your heart content. When your friends come over, boot into unRAID and accept the compromise. You don't even have to upgrade.

[nasuellia]

1 hour ago, testdasi said:

While I agree the issue is annoying to say the least, it's somewhat blown out of proportion.

• First and foremost, you mentioned a lot of "what if" scenario with upgrading. Your original build is not good enough to begin with, at least to the ultra high standard you seem to be expecting, so you have to upgrade regardless. The 6700K, in the most ideal situation, can run 1 2-core VM + 1 1-core VM (leaving the remaining core for emulation and unRAID stuff). I would be astonished if the 1-core VM is not bottle-necked.
• If productivity i.e. work is critical for you then you shouldn't be mixing it with gaming to begin with. You should be building a workstation and then if it can game then it's a bonus. Here you are building a gaming server while wondering the "what if".
• You mentioned you are used to 144Hz, which is fair enough. But that's like having a sports car. You don't drive a sports car pulling a trailer and still expect it to be fast. In other words, if 144Hz is that important to you, you have embarked on the wrong journey to expect the same performance on a VM with ALL games. Even bare metal does not have the same performance with all games, it's not reasonable to expect the same on a VM.
• 30 min frame rate, for a DOTA clone, at ultra settings, is nothing to complain about. So if your friends also need 144Hz then if they come over often enough then you will just have to give each of them a BMW PC, so to speak.

 

Last but not least, I think there's a potential solution here. You can dual boot. On normal days, boot into Windows BM and play to your heart content. When your friends come over, boot into unRAID and accept the compromise. You don't even have to upgrade.

Look, I won't reply point by point, this particular discussion seems pointless to me, we disagree on multiple points, let's just let it be; besides, you were super cool and I'm thankful for the help, and I don't want to turn into a asshole to you.

 

Disregard the reason why I'm investigating this. There's something clearly wrong/missing/immature (which is unsurprising) with LibVirt/KVM, and I'd love to find out what it is and fix it.

 

If you come up with any other idea, or stumble on any relevant new piece of information,  now or in the future, I would be glad to hear it.

 

 

 

[nasuellia]

4 hours ago, GHunter said:

I know most people use Host Pass Through mode as the CPU Mode when setting up their VM, but have you tried setting up a VM with Emulated (QEMU) Mode? Maybe that will help you in your situation. I'm really not sure but just offering another suggestion.

Hey there, thanks for chipping in.

 

QEMU emulated makes everything run like my Commodore Amiga 500 would have in 1989. =D

[TType85]

2 hours ago, nasuellia said:

Disregard the reason why I'm investigating this. There's something clearly wrong/missing/immature (which is unsurprising) with LibVirt/KVM, and I'd love to find out what it is and fix it.

 

Maybe try VMware ESXi and set up passthrough and see what it does? Other than that, it seems that a virtualized setup isn't quite where you need it yet.  I know a year or so ago it wasn't for me since my main gaming is WoW, which is very single core speed bound. All of the high core count chips were to slow to have the frame rates I wanted.  The 1950x does the job pretty well. 

[nasuellia]

2 hours ago, TType85 said:

Maybe try VMware ESXi and set up passthrough and see what it does? Other than that, it seems that a virtualized setup isn't quite where you need it yet.  I know a year or so ago it wasn't for me since my main gaming is WoW, which is very single core speed bound. All of the high core count chips were to slow to have the frame rates I wanted.  The 1950x does the job pretty well. 

I will try ESXi. Probably in a few days as I just got a really bad fever.

[jonp]

Another test to try would be using a Windows 7 guest.

Since the game is free to play, I will do some experiments with this when I get a few minutes to see if I can help track down the issue.

Sent from my Nexus 6 using Tapatalk

[nasuellia]

3 hours ago, jonp said:

Another test to try would be using a Windows 7 guest.

Since the game is free to play, I will do some experiments with this when I get a few minutes to see if I can help track down the issue.

Sent from my Nexus 6 using Tapatalk
 

Thank you jonp!

 

I suspect the issue is exactly what testdasi originally discovered through some google-fu. =)

There are quite a bit of posts talking about these MSR (Model Specific Register) and LBR (Last Branch Records), that apparently force the hypervisor to "exit", sometimes thousands of times per second in some applications, which is extremely expensive. It is unverified conjecture, but it seems very likely that something is currently missing in KVM/LibVirt and that until implemented, there isn't much to be done.

 

More then anything else, I worry about how many, much more modern games, end up having the same issues (people online talk about Doom, behaving the same way for the same reason, as well as Far Cry Primal).