GPU Throttling while in Game? Low FPS Low GPU usage

[robd]

Hi, all out of thoughts as to why this is happening.  Setup a Win10 VM with a GTX970 passthrough to play GSPro, golf simulator software.  GSPro is graphics intense and GTX970 is on the low end of GPU recommendations but figured I'd give it a try.  GSPro has various graphics level settings (Lite, Low, Medium, High, Very High, Ultra).  My projector is low resolution so I'm just trying all this on Lite settings at 1024x768.  When I play, I get around 30fps (28-32) on the test benchmark course and it stutters at times.  GPU usage NEVER goes above 15%.  Nothing looks to be bottleneck'd on the CPU side, and I'm passing 16gb ram to the GPU.  Whatever I do, I can't get GPU usage to go higher on this app unless I increase graphics settings but GPU usage doesn't go up much, and fps just goes down since it's rendering more.  Using MSI Afterburner and it's built in kombustor benchmarking tool, I'm able to max out the GPU usage. I've even OC'd the GPU some, but it makes no difference in the golf sim sw.  OC'ing the GPU, I get a few extra FPS in the Kombustor benchmark tool.  I'm on a HP Z420 unraid setup with a E5-2680 v2 CPU.  I previously had a 2630v2 but upgraded to see if it was a CPU bottleneck, but it made no difference. To add, this is a headless setup and I’m using Moonlight to connect and stream video. 

 

I've done all the usual to max out performance settings in the Windows VM.  I've followed spaceinvaderone's gaming vm setup.  Anyone have any thoughts?  VM XML is attached.  Thanks in advance! 

win10xml.txt

Edited October 2, 2022 by robd

[ghost82]

10 minutes ago, robd said:

GPU usage NEVER goes above 15%.

 

10 minutes ago, robd said:

To add, this is a headless setup and I’m using Moonlight to connect and stream video

 

This could be the issue, when the os is running headless I think graphics acceleration will be disabled, all will be loaded on the cpu side.

If you have a monitor around, just plug it in the gpu output and turn it on, then stream via moonlight and see if there's any difference.

If it makes a difference you can think about buying a dummy hdmi (or whatever connection has your gpu) to simulate an attached monitor.

Edited October 2, 2022 by ghost82

[robd]

42 minutes ago, ghost82 said:

 

 

This could be the issue, when the os is running headless I think graphics acceleration will be disabled, all will be loaded on the cpu side.

If you have a monitor around, just plug it in the gpu output and turn it on, then stream via moonlight and see if there's any difference.

If it makes a difference you can think about buying a dummy hdmi (or whatever connection has your gpu) to simulate an attached monitor.

thanks! I came across this possible solution lastnight. I connected my tv to the gpu. It’s connected via hdmi but needed a mini hdmi to female hdmi adapter as the gpu had mini hdmi. Didn’t make a difference but will try again today. I did order a dummy plug to see if it does anything. 

[ghost82]

mmm... if the tv was on before starting the vm and it didn't make any difference I'm not confident that the dummy plug will change things

Did you try to force dgpu acceleration for that app/game in windows settings?

For example:

https://pureinfotech.com/set-gpu-app-windows-10/

[robd]

1 hour ago, ghost82 said:

mmm... if the tv was on before starting the vm and it didn't make any difference I'm not confident that the dummy plug will change things

Did you try to force dgpu acceleration for that app/game in windows settings?

For example:

https://pureinfotech.com/set-gpu-app-windows-10/

Yea I was thinking that too regarding the dummy plug probably not changing anything.

 

I did set all the usual windows settings to max out performance and such.  I double checked GPU acceleration as you suggested and did make a small discovery where I added the exe for the 'launcher' of the GSPro sim application.  Diving deeper the launcher just seems to perform update checks and eventually launches a different exe for the actual app so I added that exe for max GPU performance.  fingers crossed I launch the app to test, but still the same results.

 

Someone on the GSPro discord did test on a 1050ti laptop and they get higher FPS than I do as the GPU usage is much higher.  Which is what I would expect, higher GPU usage to get higher FPS, instead of hovering around 15% GPU usage, just to get ~30fps like it's doing for me.  

 

Not sure what else to check.  My limited unraid/VM knowledge, I'm assuming it's a VM config/setting as the hardware should be more than capable.  Just not sure why the GPU is being 'limited' with this app.

[ghost82]

What about other softwares?like benchmark softwares or other games?

It could simply be that that simulator is not programmed well to be run in a vm?

 

As far as the vm settings I can only suggest to:

1. use q35 machine type instead of i440fx for better pcie compatibility

2. put the gpu in a multifunction device, video and audio parts in same bus, same slot, different function.

3. check for irq conflicts inside the guest and use msi fix to switch from irqs to msi(x) if there are irq conflicts

 

(4). Use ovmf instead of seabios (but you need to convert the disk or reinstall windows), so that the gpu will use the uefi vbios instead of the legacy one.

Edited October 2, 2022 by ghost82

[robd]

3 minutes ago, ghost82 said:

What about other softwares?like benchmark softwares or other games?

It could simply be that that simulator is not programmed well to be run in a vm?

 

As far as the vm settings I can only suggest to:

1. use q35 machine type instead of i440fx for better pcie compatibility

2. put the gpu in a multifunction device, video and audio parts in same bus, same slot, different function.

 

Worth noting, HWiNFO app shows GPU Performance Limiters.... Utilization is yes.  So is it the app just not making full use of the GPU?  but if that's the case, on another setup non-VM, laptop setup with a 1050ti, the app makes full use of the GPU.  

 

I'll try q35 machine type.  

 

Can you explain further item #2?

 

Any workaround if there are application limitations if it doesn't like to be run in a VM, or am I just SOL?

 

edit - I used the kombustor benchmark app within MSI afterburner, and it's able to max out the GPU.  So I guess that rules out VM config/setting that is limiting the GPU usage, and it's all pointing at the golf sim app.

Edited October 2, 2022 by robd

[ghost82]

13 minutes ago, robd said:

Can you explain further item #2?

 

This is my gpu, audio and video parts passed:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
      </source>
      <rom file='/opt/gpu-bios/6900xt.rom'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x06' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
    </hostdev>

 

Real hardware addresses (seen by the host) are:
video at 06:00.0

audio at 06:00.1

 

--> add multifunction='on' in the address line, so that I can set the addresses in the guest at:

video: 03:00.0

audio: 03:00.1

Same bus (03), same slot (00), different functions (0 and 1)

 

13 minutes ago, robd said:

HWiNFO app shows GPU Performance Limiters.... Utilization is yes

 

For what I'm reading this is normal if the gpu isn't under load. Are all the cables connected to the power port(s) of the gpu and does your psu have enough power for all?

 

13 minutes ago, robd said:

Any workaround if there are application limitations if it doesn't like to be run in a VM, or am I just SOL?

 

It should just exit or crash, nor limiting performances, I was talking about bad programming, if this is the case..

Edited October 2, 2022 by ghost82

[ghost82]

15 minutes ago, robd said:

edit - I used the kombustor benchmark app within MSI afterburner, and it's able to max out the GPU.  So I guess that rules out VM config/setting that is limiting the GPU usage, and it's all pointing at the golf sim app.

That is making things more difficult, because apparently the gpu works well, try the advices given above, read all again because I edited my posts.

Edited October 2, 2022 by ghost82

[robd]

1 minute ago, ghost82 said:

That is making things more difficult, because apparently the gpu works well, try the advices given above, read all again because I edited my posts.

 

Yea I agree.  just noticed the edits, I'll give them a try and report back.  Really appreciate all the help!

[robd]

7 hours ago, ghost82 said:

What about other softwares?like benchmark softwares or other games?

It could simply be that that simulator is not programmed well to be run in a vm?

 

As far as the vm settings I can only suggest to:

1. use q35 machine type instead of i440fx for better pcie compatibility

2. put the gpu in a multifunction device, video and audio parts in same bus, same slot, different function.

3. check for irq conflicts inside the guest and use msi fix to switch from irqs to msi(x) if there are irq conflicts

(4). Use ovmf instead of seabios (but you need to convert the disk or reinstall windows), so that the gpu will use the uefi vbios instead of the legacy one.

 

So I tried all 4 suggestions.  Reinstalled Windows even to start fresh.  Followed performance tweaks on the Unraid Wiki.  If anything, it's worse.  Windows settings are the same for high performance power management, etc.  VM settings should be the same as well except for the 4 items suggested to try.  I can near max out the GPU in the kombustor benchmark app and MSI Afterburner though.  XML of the new test VM attached.

vmv2.txt

[ghost82]

Is it an hp z420? --> update, yes it is, you wrote it

 

If it is, I'm reading that you should have in your bios some settings about "performance profile" or "power regulator settings": check if you have some dynamic settings there and switch to static high performance, if you have that setting. Check all the other power settings in bios, especially related to pcie (if any...).

I'm reading that some hp servers should have power saving modes in bios that could throttle pcie devices.

Edited October 3, 2022 by ghost82

[robd]

5 hours ago, ghost82 said:

Is it an hp z420? --> update, yes it is, you wrote it

 

If it is, I'm reading that you should have in your bios some settings about "performance profile" or "power regulator settings": check if you have some dynamic settings there and switch to static high performance, if you have that setting. Check all the other power settings in bios, especially related to pcie (if any...).

I'm reading that some hp servers should have power saving modes in bios that could throttle pcie devices.

 

Thanks, will give that a try today.  I almost all but gave up.  Ordered an SSD to dual boot to bare metal windows on this server box to test that way 😔

[robd]

EUREKA!  IT WORKS!

 

PCIe Performance Mode was disabled in Bios.  Enabled it, booted up the original VM.  Golf Sim app 60fps with vsync on Lite graphics mode.  CPU/GPU usage ~26%/~29% respectively!  Disabling vsync, jumps to ~75fps.  This is fantastic!  

 

Thank you so much @ghost82 for all the help!

[ghost82]

Nice that we found the solution, never give up!Still a mistery why the benchmark apps overcome that limit...

 

PS: no problem for the ordered ssd, one ssd more will find its use for sure

 

Edited October 3, 2022 by ghost82

[robd]

1 hour ago, ghost82 said:

Nice that we found the solution, never give up!Still a mistery why the benchmark apps overcome that limit...

 

PS: no problem for the ordered ssd, one ssd more will find its use for sure

 

yea, benchmark apps max'ing the GPU definitely threw a curveball into things.  oh well.  Appreciate all the help, thanks again!