Improving Windows VM performance for gaming

[Marshalleq]

Thanks all, a fascinating discussion.  Other than what @Warrentheo mentions above, I would not be sure how to know the passthrough is working - I definitely have the offical Nvidia drivers installed and the 1050ti card shows in my device manager, so should be good -  I first used a hex edited downloaded rom, then dumped my own yesterday to be sure.  From what you're saying it sounds like it's possible to get very good VM performance then.  

 

I might try the q35 setting next though I get stuck with some error about translating the PCI bus, then when creating a new VM I get stuck with a duplicate uuid.  I'll sort that out - this is obviously getting off topic for this thread.

 

I've run the 1050ti on bare metal windows (no Unraid) on this machine before so I know it's performance.  It's definitely not the card causing this and besides for testing I've got all the quality on low (hence my query around what others are seeing).

 

Something interesting is I actually see the mouse lag while the game isn't running just in bare windows - not a lot, but sometimes a bit - at a guess I'd say disk I/0 is lagging the system which makes me think @Warrentheo's suggestion of converting to SCSI is a good idea.  And it's when I'm using the mouse (turning character left and right) that I see all the screen lag / jumpiness - going forward and back doesn't seem to do it for some reason - I guess it redraws more sideways, or it's interrupt based problem.  There are plenty of cores available with Ryzen 1800X , it's on an enterprise SSD, and it has dedicated 14GB of total 32GB memory at present.  Should be good.

 

So keeping on topic - Performance related enhancements, hyper V I assume should be on also?  Seemed to make things a bit better when I enabled it.

[Marshalleq]

Probably another aspect to the performance question is streaming.  I haven't found that to be very good either.  And the graphics quality goes down quite a lot.  I assume there would be some good experience everyone has there too.  

[Marshalleq]

I just ran 3d passmark.  The GPU runs OK at around 130FPS depending on the scenario, but what I notice is during the middle it will drop to less than 1 FPS.  Also noticed, while typing the whole system pauses, then the typing catches up.  There's something wrong with the way the VM is speaking to the host.  So maybe it's not GPU passthrough at all.

[Zer0Nin3r]

19 hours ago, steve1977 said:

I didn't really see any difference in performance since switching to QEMU. Not for the worse, but also not visibly to the better. I'll keep watching it.

Thanks for the feedback. I appreciate it @steve1977. I'll stick with the i440fx for now since it's working for me. I don't feel like making a backup of the vm, rebuilding the drivers upon boot, and tinkering with the vm. But in looking at the documentation for both machine types, logically it would make sense to build Windows 10 machines with Q35; the documentation out there needs to be refreshed, it's been a while.

 

7 hours ago, Warrentheo said:

Linux in general, and therefore UnRaid likes to have its primary core on Core 0... Even if you try and isolate Core 0 Linux will still use it anyway... 

Good to know. I tried searching for something that says it in stone, but can't really find anything substantial — just what the conventional wisdom in the forums dictates. That being said, I asked my question again, since it's been several years since the conventional wisdom has been passed around and the documentation on CPU pinning didn't really give me much to work on; again information atrophy.

 

7 hours ago, Warrentheo said:

Adding NUMA nodes and infinity fabric into this mix makes things even more complicated

I can attest this: Gaming on cores 16–31 with the Threadripper 1950x results in poor performance/FPS. Not sure why this is since the cores pinned should be purely on chip #2. I've tried with 6/8 cores and the FPS is lower than if I were to pin cores from the 2–15 pool.

 

6 hours ago, Warrentheo said:

but typically the way I understand it, you wont get the drivers to load in windows if you don't have a VBIOS for the NVidia card correctly passed through

I've tried like hell to get pass-through to work on the EVGA GTX 1060 6G in Slot 01 and it simply wouldn't work for me. I tried even loading the VBIOS into the Unraid template with no success. It wasn't until I bought a cheapo card and threw it into the server that I was able to successfully pass-through my GPU; I'm passing through the GTX 1060 without having to dump the VBIOS and load it into the template.

 

4 hours ago, Marshalleq said:

, I would not be sure how to know the passthrough is working

You know pass-through is working when you can see the GPU in your Windows device manager and it is not displaying any error codes. Additionally, you know pass-through is working when you can install the video drivers, reboot, and not see any error codes. You should see a "This device is working properly." in your device manager.

 

4 hours ago, Marshalleq said:

Probably another aspect to the performance question is streaming.  I haven't found that to be very good either.  And the graphics quality goes down quite a lot.  I assume there would be some good experience everyone has there too.

Yes, there will be a degradation of graphics quality when game streaming. I am running Unraid in a headless state and using Parsec on a thin client for gaming with First Person Shooters in online multiplayer.

 

I can tell the difference of quality when I look at the playback of my past matches. Moonlight seems to have a better picture quality than Parsec, but you cannot readily access the desktop or your other applications. Also, I seem to get better FPS with Moonlight than Parsec, but Parsec allows me to open and run Discord or any other application that I need to get to.

 

@everyone: Please edit your posts with new thoughts/insight and throwing in a reason for the edit rather than creating multiple subsequent posts — it makes reading the threads easier and allows everyone to use their time more efficiently.

[steve1977]

2 hours ago, Zer0Nin3r said:

I can tell the difference of quality when I look at the playback of my past matches. Moonlight seems to have a better picture quality than Parsec, but you cannot readily access the desktop or your other applications. Also, I seem to get better FPS with Moonlight than Parsec, but Parsec allows me to open and run Discord or any other application that I need to get to.

I've been spending days and weeks on streaming. Moonlight (or gamestream if you have a Shield) are still far superior to Parsec. While the Parsec developers are great and incredibly helpful, the moonlight implementation is still far better. Not sure why you cannot access desktop or other applications. For me, this works very well. At least from my RPi. Haven't tried from IOS devices, but should also work. Only Parsec advantage is that you can play the same game from two devices at the same time.

[Warrentheo]

7 hours ago, Marshalleq said:

I just ran 3d passmark.  The GPU runs OK at around 130FPS depending on the scenario, but what I notice is during the middle it will drop to less than 1 FPS.  Also noticed, while typing the whole system pauses, then the typing catches up.  There's something wrong with the way the VM is speaking to the host.  So maybe it's not GPU passthrough at all.

This sort of thing sounds like a scheduling issue with the cores, most likely the UnRaid kernel and the Windows kernel both have access to the same cores, and some core is getting scheduled for double duty, thus locking up the other while it gets done...  Make sure the VM and the Qemu emulator have their own cores...

[Marshalleq]

22 minutes ago, Warrentheo said:

This sort of thing sounds like a scheduling issue with the cores, most likely the UnRaid kernel and the Windows kernel both have access to the same cores, and some core is getting scheduled for double duty, thus locking up the other while it gets done...  Make sure the VM and the Qemu emulator have their own cores...

I have today (apart from remembering how much I hate the windows long winded everything I have to do installation process", reinstalled the VM from scratch, Q35, dedicated SSD, and assigned it to the last 4 cores / 4 threads of the 1800X.  I have also stopped anything else from accessing those cores, just to be sure.  It's definitely running better.  I have semi-evidence that SABNZbd was causing a bit of grief, but will leave that investigation for now.  It was running on a Samsung PM863a which I am suspicious needs a firmware update - it's supposed to handle a lot of IO, which is why SAB runs from there, but go figure.  So many things to think about with Unraid!

 

4 hours ago, Zer0Nin3r said:

I've tried like hell to get pass-through to work on the EVGA GTX 1060 6G in Slot 01 and it simply wouldn't work for me. I tried even loading the VBIOS into the Unraid template with no success. It wasn't until I bought a cheapo card and threw it into the server that I was able to successfully pass-through my GPU; I'm passing through the GTX 1060 without having to dump the VBIOS and load it into the template.

My card is a 4GB EVGA 1050ti - linked here.  It suited me at the time as it does half height slots and doesn't need extra power.  I might upgrade it but it's remarkable how good it is to be honest.  But I mention it just in case it helps with your EVGA situation.  Actually I dumped an EVGA 710 BIOS while I was at it.  Might be good for a Mac VM.

[Warrentheo]

Yah, it can be an adventure...  AMD cards can have special issues with CPU scheduling that unfortunately I am not familiar with...

4 minutes ago, Marshalleq said:

Actually I dumped an EVGA 710 BIOS while I was at it.

Just so you know, the VBIOS dumping thing is because NVidia encrypted their VBIOS as well as the connection between the VBIOS and their drivers...  However, they only started doing that on the GTX 10 series and the RTX...  You should not need a VBIOS from a GTX710...

[Marshalleq]

Oh excellent so that was a wast of time then - lol.  I don't get much time to be a geek these days, but it sure is fun playing with hardware again - I used to know everything and I now I know nothing!  The principles don't change but the details sure move on.  

[steve1977]

On 2/9/2019 at 12:51 PM, steve1977 said:

I am tempted to give it a try this weekend to run my VM bare-bone and see whether it improves performance. One advantage I see: dockers may sometimes max out non-allocated CPU and non-allocated ram. If I understand it directly, this would leave a bare-bone VM unaffected, while it may have a negative impact on a vdisk VM as some of the non-allocated CPU/ram is still required for visualization and may compete with the dockers? So while my initial worry about dockers and VMS hammerings the same NVME seems a minor issue, there still is an advantage of having the VM 100% isolated from Unraid/dockers (which can only be achieved bare-bone)?

 

If going this route, two questions for the gurus:

 

1) Did you convert the vdisk to bare-bone and (if so) did you do this on a live VM?

 

2) Is there any way back if I eventually decide that I may prefer to go back from bare-bone to vdisk?

@bastl @Warrentheo Any thoughts on my questions above? I am thinking to put my VM bare-metal to try it out, but want to get some sense whether it's better to convert or to create from scratch and whether I can go back to vdisk if no advantage. Also, I'm also curious on what drives you leave the GPU.rom file and the libvirt.img? And lastly, there seems to be another vdisk1.img created together with the libvirt.img, which I am not sure whether I can delete it?

[Tritech]

I've converted to vm and back a few times to test with the same windows install.

 

Quote

Also, I'm also curious on what drives you leave the GPU.rom file and the libvirt.img?

I'm not sure what you mean by this, I'm assuming you mean to config the vm.

This is my current vm, the xml is heavily customized but you should get the idea from here. There is no vdisk location as I've passed through the NVME controller that you see on the bottom. 

 

As far as extra vdisk images, just check when they were last modified, and if there are any existing vms that point to that .img.

 

 

[steve1977]

Great, thanks!
 

How did you convert from vdisk to pass-through? And then even back and forth? Would love to try it out.

 

Is your /mnt/user pointing to the array? If so, you have the GP1202.xxx.rom file on the array?

[Warrentheo]

12 hours ago, steve1977 said:

@bastl @Warrentheo Any thoughts on my questions above? I am thinking to put my VM bare-metal to try it out, but want to get some sense whether it's better to convert or to create from scratch and whether I can go back to vdisk if no advantage. Also, I'm also curious on what drives you leave the GPU.rom file and the libvirt.img? And lastly, there seems to be another vdisk1.img created together with the libvirt.img, which I am not sure whether I can delete it?

Switching Bare-Metal to VM and back again I have seen articles and posts on, but it is outside my personal experience, it would be like yanking a hard-drive back and forth between 2 motherboards...  Windows 10 "ToGo" sticks I am told make this easy, but never tried them out...  Windows also has the ability to boot "Bare-Metal" but from a VHD or VHDX image file which is something I have done before back in Windows 7 days, but flipping the image file between VHD(X) and qemu RAW or QCOW2 just seems like a hassle...

 

I currently run my system exclusively as a VM with GPU passthrough, and you can get near bare-metal performance when needed by just tweaking some settings and turning off background processes on the host, which is the kind of thing you would need to do even if you were running bare-metal, so the point of switching between them seems way more trouble than it is worth IMO...

 

Edit: I have a m.2 960 evo Raid-0 that I use as UnRaid cache drive...  I store them on there, and just set the "system" share to "Cache drive = Yes"

Edited February 16, 2019 by Warrentheo

[steve1977]

Thanks for your help. My thought about moving to bare-bone does not come from potential incremental improvements. My worry is that dockers may sometimes max out non-allocated CPU and non-allocated ram. If I understand it directly, this would leave a bare-bone VM unaffected, while it may have a - potentially even significant - negative impact on a vdisk VM as some of the non-allocated CPU/ram is still required for visualization and may compete with the dockers. Or is this not a major issue?