nasuellia

Hi! Thanks for replying! As I just wrote in the previous reply, that's not always true. I ran tests for weeks, and isolating 6 cores (pinned to the VM) and leaving a couple for exclusive use to UnRaid (0,4) actually brings a measurable decrease of performance. Consider that I run NO dockers and my host is doing NO cpu-intensive task whatsoever, the cpu usage from UnRaid itself is tiny to say the least. I had this issue with both passed-through M.2 disks and with Vdisks, my disk performance are perfect, and my issue has nothing to do with disks anyway. About RAM, it's not even possible to assign all RAM to the VMs, UnRaid prevents you to do so and sets a maximum amount. In any case, all possible reasonable amounts of RAM have been tested before. Anyways, the point here isn't that I have general performance issues, on the contrary: everything is super fine and in-line with what it should be. The weirdness I'd like experienced opinions about is: why on earth, when both the VMs are running, would one of them suffer a decrease in CPU/RAM performance and the other NOT?

Hi thanks for replying! My system is already set on maximum performance regarding power plans (both in the Windows 10 VMs and in UnRaid through the Tips and Tweaks plugin. On top of that, I've also made sure my CPU runs at max clock all the time by disabling Speedstep and manually setting the clock to a fixed frequency. So that's not the issue. You didn't see specs because I'm an absolute cretin and I forgot to paste them, I edited the post now! As far as core-pinning and isolation, I ran a trainload of tests when I was setting up the system initially a couple months ago, and I concluded that, given that I run NO docker and NO cpu intensive operation on my server, it makes no sense for me to renounce a couple logical cores to dedicate for exclusive use of the server, in fact if I do that i LOSE performance, significantly. After a lot of benchmarking, I concluded that in my specific circumstance, letting UnRaid access all cores and just splitting them across the VMs (4 and 4 if both need running, or all 8 to my machine most of the time when I'm alone) works the best, by far. Last month when I was setting up, I think I watched 3/4 of the SpaceInvader's channel on Youtube, hugely informative and well done. Regarding monitoring, I forgot about Netdata, I did watch that video back then! I'll give it a run, thanks!

Hello there everyone! I've been using UnRaid to host my dual gaming machine for about two months now, and I'm very satisfied when it comes to performance: it's 90 to 98% of the bare metal, with the notable exception of Heroes of the Storm that for some unfathomable reason runs a lot worse, you might recall my post about it, about one month back. Today's post isn't about that though; instead, I'd like your opinion about a few minor but annoying issues I've got. 1 - VM becomes stutter-y and requires restarting the host to fix 1 a - Description of the issue Occasionally, my VM ends up in a really weird state: everything's stutter-y, even the mouse cursor over my desktop is barely usable, any operation takes forever (opening a window, opening the start menu, even shutting down); it also seems to get stuck with some actions, for example: I would move the cursor towards the left a bit, and it would slowly keep going that direction (all stutter-y, still) forever. To give you an idea of how slow everything is, any window, say the control panel, is drawn very slowly and fades-in in distinguishable steps (imagine: 10% transparency >> 40% >> 70% >> 100%, in about 3 seconds). Note that in the meantime, the host UnRaid OS is perfectly fine and, If I happen to have the second VM running at that same time, it does not necessarily suffer from the same issue. This does NOT exclusively happens when I'm running both VMs, it happens at random on my VMs (both of them) even when just one is running with full resources. 1 b - Apparent triggers The issue i just described happened in a few cases: - After updating Nvidia drivers (not always). - After a VM restart (not always, but often times). - After spending a night processing GPU intensive tasks (very often). 1 c - The way I fix it - If I restart or shutdown the VM and stat it back up again, the issues does NOT go away. - If I shutdown the VM, then restart the physical machine, it usually goes away (but not always). - If I power cycle the physical machine, it always goes away. 1 d - Questions Obviously: any idea what's going on? Anyone else with the same issue or a comparable one? 2 - My main VM runs significantly worse then the secondary one. 1 a - Description of the issue My main VM gets full resources (my specs at the end of the post) most of the time, and is getting my GTX1070 passed through. Everything works great. When a friend comes over, I shut down my VM, edit the configuration to split resources appropriately, and fire up my VM as well as a second one: they get 2 physical cores each (4 logical cores) and 6.6 GiB of memory; the second VM gets a GTX660 as a video card. I have no IOMMU issues (fixed them all); it works quite well, but I noticed a really really strange thing that I can't figure out: the secondary machine actually runs better then my first when it comes to NON-GPU related performance. Take Rocket League, a game that's super easy on my GTX1070 and is clearly CPU-bound: if I set both with quite low graphical settings (in order to make sure neither GPU is bottlenecking), the secondary VM runs about 20 FPS higher then my main one, which is baffling to say the least. I verified it in a couple more games and the behavior seems consistent: my main VM runs worse then the second one, despite absolute parity in cores and memory. Note that, obviously, if I set higher settings, GPUs become the bottleneck, and my machine runs much better, so it's clearly something with the CPU or memory or some other bottleneck I'm not thinking of that, somehow, only affects one of the two VMs. I tried switching cores around, disabling Hyperthreading, reinstalling the OS multiple times, to no avail. At some point I tried switching the GPUs positions in their respective PCIexpress slots (first and third), but both Windows installations got mad at it and wouldn't start, so I didn't test that. 2 b - Well, any idea? 3 - Is there any way to monitor CPU temperatures on the guests? Probably not, but I'm deeply unsatisfied about Dynamix plugins, I would like something more in-depth, showing me real time temps per core, clock per core, voltage, usage per core. This is probably a moot question but is there any alternative? Or is there a software somehow capable of properly monitoring from the guest? (probably impossible). Specs Motherboard // MSI Z170A Gaming M5 Processor // Intel Core i7 6700K Skylake Cooling // Corsair H110 Memory // 16 GiB Crucial Ballisitx 2400 Storage #1 // Samsung SSD970EVO 250 GiB M.2 Storage #2 // Samsung SSD840EVO 500 GiB SATA Storage #3 // Samsung SSD850EVO 120 GiB SATA Storage #4 // Western Digital Caviar Purple 4 TiB SATA Power Supply // Corsair TX850-EU Cooling Corsair H110 GPU #1 // Nvidia GeForce GTX1070 GPUI #2 // Nvidia GeForce GTX660 Thanks everyone and have a great day!

Thank you jonp! I suspect the issue is exactly what testdasi originally discovered through some google-fu. =) There are quite a bit of posts talking about these MSR (Model Specific Register) and LBR (Last Branch Records), that apparently force the hypervisor to "exit", sometimes thousands of times per second in some applications, which is extremely expensive. It is unverified conjecture, but it seems very likely that something is currently missing in KVM/LibVirt and that until implemented, there isn't much to be done. More then anything else, I worry about how many, much more modern games, end up having the same issues (people online talk about Doom, behaving the same way for the same reason, as well as Far Cry Primal).

I will try ESXi. Probably in a few days as I just got a really bad fever.

Hey there, thanks for chipping in. QEMU emulated makes everything run like my Commodore Amiga 500 would have in 1989. =D

Look, I won't reply point by point, this particular discussion seems pointless to me, we disagree on multiple points, let's just let it be; besides, you were super cool and I'm thankful for the help, and I don't want to turn into a asshole to you. Disregard the reason why I'm investigating this. There's something clearly wrong/missing/immature (which is unsurprising) with LibVirt/KVM, and I'd love to find out what it is and fix it. If you come up with any other idea, or stumble on any relevant new piece of information, now or in the future, I would be glad to hear it.

Unfortunately Q35 was already tested before my post, and did not not change anything. At this point, I'm gonna mess around with VMs for a bit more time, because they are awesome and I'm learning quite a lot of interesting stuff, but in the end, I will go back to bare metal for consistency reasons. I can only thank you for the extended and precious contribution, you're really great man, I can't thank you enough. That said, let me respectfully disagree with your conclusion: it's not a matter of wanting or needing 99% BM performance: - I want my VM to have predictable performance, and not erratic; otherwise I will never know if it's performing correctly or not, which would make spending money on hardware kind of a shaky proposition. As I said on the original post, I expected a general, predictable, measurable drop in performance, and I am willing to accept it, maybe even 15% across the board. On the other hand, I am not up for accepting erratic unpredictable behavior caused by unknown, not fixable issues of this magnitude. - You say that I might have to accept 60 FPS as acceptable, but my system actually often drops to 30 and 40 in real matches online; which is definitely not enjoyable. - The actual reason to do this whole VM thing, was to play locally with friends when they come over. So imagine such an already borderline situation, getting even worse when I split CPU and RAM across two VMs... I already tried it, it's not enjoyable, at all. -- Even assuming it would do 60 FPS stable (which it doesn't as just explained) I am used to 144Hz and therefore 60 FPS is actually quite poor to my perception. It's not a matter of being just picky either, it's that I bought my hardware in accordance and spent money for having 144 FPS, so anything that bars the way to that, is not acceptable. -Imagine I commit to the VM project, accepting the compromise that some apps will run like crap. I spend quite a bit of money with a new processors with more cores, definitely more RAM, and probably more disk space. Then, 6 months from now, another game comes out, a much more modern and performance intensive then HotS in the first place, and it turns out, that it suffers from the same problem... I would find myself with an outrageously expensive machine that can't run that game. - Who knows if this issue applies to other stuff? It's not a matter of accepting playable frame-rate on this particular game and forget about it: what if it turns out that some productivity stuff (that I also need) is also being severely degraded by this problem? What if it turns out that gradle compile times are 50% slower on VM with "whatever" particular specific situation? I'm not up for accepting that, I don't want to fear having halved performance and losing productivity without even knowing, in the future.

That's so sad. Regardless of HotS, AMD users are missing out on a virtualization technology... and Intel is entirely disregarding it altogether. That is really weird. Maybe the line was improperly built, I hope someone with actual knowledge of this stuff can chip in.

From what I understand after reading the libvirt documentation (https://libvirt.org/formatdomain.html), there is a way to "force" or at least "require to try" supporting a feature on the guests (obviously assuming the host does). Adding something like following in the XML under the <cpu></cpu> tag, might do the trick: <feature policy='require' name='lbrv'/> or maybe <feature policy='force' name='lbrv'/> I also have no idea whether 'lbrv' needs to be in uppercase 'LBRV', it's certainly possible. If Threadripper possessors can create a throwaway VM template and try to boot with it, it would be the last chance to figure out this issue...

Ok it's pretty evident at this point that the performance-crippling issue is present on AMD processors as well, at least by default. Thanks a lot for the test! I'm about to post another reply with the last resort...

I found this: https://libvirt.org/formatdomain.html which appears to explain it all about XML editing for the CPU (section: CPU model and topology) I'm literally falling asleep on the keyboard right now, I'll give it a read tomorrow and maybe we can actually try to manually specify the flag. It won't work on my system, but it might make a difference with AMD processors on certain games.

Aaaaaaaand there is no sign of LBRV on the guest... It's certainly possible KVM/LibVirt still didn't come around to implement LBRV on the software side. Alternatively (and hopefully) it's just a matter of flags-passing, and we can find a way to add the flag. Do you guys know of someone really (really) experienced we can ask for? Maybe SpaceInvaderOne? I might drop a PM on him tomorrow... Thanks for posting this, it would still be interesting to see your results, if you ever take the time.

When you have a couple minutes maybe check CPUz on your Windows guest, it should provide you with the comprehensive list of CPU-features; if LBRV isn't listed, we found the potential issue. Unless I'm mistaken, i440fx and Q35 should not affect which CPU-flags are (or aren't) passed to the guest, I believe those are two archetypes of chipset emulation. I believe what commands the CPU-features is the part where we can chose host-passthrough or qemu-emulated, and I believe there's way, through direct XML editing, to add flags manually instead. If that's the case, it's a way. But this is all conjecture, maybe LBRV is already passed and working, and the root cause is simply not that. That's certainly possible... Let me know if you have the time to check CPUz in the next days. Let me be jolly for a second: how did the testing go, as far as fighting? Eheh, did you die the first time around? Did you enjoy the matches vs AIs to some degree? I hope it wasn't too boring for you! Thanks again for helping and by the way, wow... what a rig you got going on there... it's nothing short of beastly. PS: by the way, Intel processors do not support LBR virtualization at all, so in any case, my system would still suffer from the issue, regardless of the operating system.

What if the LBRV flag is not getting passed to the guest? Maybe check CPUz? I know... I sound desperate. That's because I am. I worked so hard to get the VM working on my system and everything seemed so great... Now I'm discovering that, although it's certainly not a majority of the games, quite a few suffer from this issue (whether that's caused by hammered LBR or not).

Yeah, as expected, LBRV is there on AMD processors. Unfortunately it might not be the issue though... Unless NUMA mode is the problem there, but it doesn't seem believable for it to cause such a stupidly large drop... I mean, we're comparing that beast of a CPU with my humble, 3 years old, 6700K... UMA or NUMA, there should be a measurable advantage on your side. Man, I can't thank you enough. You helped a lot. Wait, did I already thank you? =D

Wow, you are great. Seriously. Thanks so much! I have no words... I also just saw your other post, pasting the CPU flags on your processors, and there actually is LBRV there...

And there goes my hope... May I ask you one last thing? It should be a very quick thing to do: can you verify the presence of the LBRV (or LBR-whatever) flag on your host? This command should do the trick: cat /proc/cpuinfo

Hey testdasi, first of all thanks a lot for testing this out for me, you're pure gold! Those results are worrying to say the least. I am assuming you ran the benchmark with the rig in your signature, so a Threadripper 2990WX. May I ask you how many cores did you assign to the VM? From what I know, the 2990WX is actually quite abysmal at gaming because of the similarities with dual-CPU architecture causing lots of latency, especially if the assigned cores are not the ones tightly linked to the PCIexpress controller where the video card resides. That said... we're talking about framerates even lower then mine, so it's... really alarming. Again, consider that my bare metal runs with <low 1%> around 150 FPS in real matches. This makes me very sad.

What a moron I am, I forgot to post the resolution. As I just said to Jcloud, I have a 21:9 monitor so I ran the tests @2560x1080. Running the test at @1920x1080 is perfectly fine, even better as the GPU isn't the limiting factor here, in fact I do not gain a single FPS by going down to 1080p either. Not one. Your assessment of my needs is spot on: I need to verify whether AMD processors actually do not stumble on the issue of the lack of LBR virtualization (they should support it, according to the specs). Threadripper processors even have comparable single core performance to my 6700K so it should actually be a reasonably accurate test no matter the number of cores (as hots uses just two at best): if it runs with a significant advantage over my system, it clearly means the AMD processors do not suffer from the issue.

Hey there, thanks for chipping in! My test performed with 8 logical cores (4 physical) on my 6700K. My resolution is a bit weird because I have a 21:9 monitor so it's 2560x1080, but that doesn't matter at all, as the GTX1070 is waaaaay overshoot for Heroes of the Storm, that's why it sits there at 40% to 60% usage. The limiting factor is my processor, that is being crippled by this lack of LBR virtualization on Intel. Therefore, any test with a comparable GPU would do, as long as it doesn't become the limiting factor (so if you ever decide to run the test, don't run it at 4K, run it at 1080p). Heroes of the Storm doesn't even multithreads very well either, it only uses two cores at most. Hence it doesn't really matter how many cores you commit to the test VM; in order to remain in the domain of reasonable comparison, I'd say keep to 4 or 5 physical cores and their hyperthreaded counterparts; but even a test with all your cores would still serve perfectly well to get an idea of how a Threadripper would perform in the given situation. When it comes to RAM, again, it doesn't really matter as Heroes of the Storm is far from RAM hungry. I was allocating 12 GiB of memory to my VM, but if I recall correctly the game doesn't really uses more then 5 GiB at peak. In any case, thanks a lot for getting interested, whether you end up running the test or not. Sincerely thanks.

- Yes the game is free to play. You just need to download it from the BatteNet client. You also do not need to do anything special to access the try mode needed for the tests, and if I remember correctly, the 3-minutes tutorial is skippable (if it's still in the game at all, which I'm doubtful). - The values I provided in the link to my google spreadsheet (here it is for your convenience: https://goo.gl/wmvGwx) are referring to the testing methodology I described for Heroes of the Storm. If you don't want to visit the link: I'm getting ~227 FPS on BM and ~164 FPS on VM. I see you've got a GTX1070 just like me, so it would actually be a meaningful test to do. Wow. - Don't worry about dying, the Try Mode is truly incredibly easy, it's designed to let the player check out each hero's abilities, you won't die, ever. In fact you can even just stand back and spam abilities at random. Just make sure to keep the camera in the middle, and get both you, the AIs, and all the minions in the frustum of vision all the time (you can move the camera around a bit, just don't go astray and look at empty parts of the map like the spawn area or the north/south of the lane). As for not dying, you don't even need to actually fight the AI at all as the enemy character is purely melee and won't be able to hit you (and you have an ally AI healer so it's 2 v 1 and get healed all the time). Seriously, it's not gonna be an issue. Even if maybe you won't find the time do the test, thank you a lot in advance for proposing it, really appreciated. I'll make sure to remind you about it in the early days of October. Thanks truly. Feel free to ask anything you want!

Wow I did not expect someone to commit to that! If you ever come around to actually do it, well, thanks a lot! Here's how you can reproduce the same test, which would have to be run on bare metal and then on VM in order to compare the results: - Download and install MSI Afterburner (it includes Rivatuner). - Launch Afterburner and enter the settings, benchmark tab, and set hotkeys to start/stop the bench. - Download the battle net client from Blizzard. I won't post a link because I don't want to risk the anti-spam bot killing the post. It's easy to find though! - Install Heroes of the Storm from the battle net client, it's a free to play game! - Launch the game and set all settings to max. Also increase audio channels, audio quality and enable reverb. - Restart the game just to make sure every change was applied correctly. - Set your phone or whatever else with a timer or countdown for 5 minutes. - Enter TRY MODE with any hero (I did it with Mephisto, but it's not relevant). - As soon as your character spawn, start the timer, start the benchmark then head right towards the middle of the lane. - Fight the minions and the AI opponent. Play safe and do not die. It's important for the benchmark. - Keep playing without ever moving the camera away from your character, and stay in the middle of the lane. - When the timer rings, stop the benchmark. - Head to the default location where Rivatuner saves the bench. It should be in documents. - Look at the average framerate. - Do the same for the other machine (VM or bare metal) and post the results to make me cry (either way). PS: I know it's written in a bit of an ELI-5 manner, but I wanted to make it clear in case anyone else wants to try this as well.

I just did a test with a Windows 8.1 VM and unfortunately, it behaves the same exact way. It's certainly possible that Microsoft patched the same functionality in W8.1 in the meantime. I suspect the only real solution is to ditch Intel for a Threadripper instead, but I would need a confirmation before committing to the purchase.

I found the issue and it's actually related to virtualization so it might interest you all. Apparently Heroes of the Storm makes use of some new debugging/tracing feature introduced with Windows10; that feature makes thousands of calls to LBR registers which are not virtualized on Intel processors. Hence, a huge performance drop compared to bare metal. This affects a handful of games, but Heroes of the Storm isn't the only one apparently. Here a few more instances of people stumbling on this: - https://www.spinics.net/lists/kvm/msg132500.html - https://www.reddit.com/r/VFIO/comments/4o0bjk/how_to_fix_the_unsupported_msrs_in_win10/ - https://www.redhat.com/archives/vfio-users/2016-May/msg00134.html If anyone knows a way to disable the feature on Windows 10, please let me know.