MarKol4 Posted November 1, 2022 Share Posted November 1, 2022 (edited) Hello all, I have HP ZBook 15 G5 (with Xeon E-2186M CPU and nVidia Quadro P2000 dGPU) laptop which I use to explore Unraid features. I'm also running one Win10 guest VM on this Unraid box. For now I'm dealing just with this one VM. My VM has 5 CPUs organized as follows (VM cores do not occupy two HT threads of the same core): <vcpu placement='static'>5</vcpu> <cputune> <vcpupin vcpu='0' cpuset='1,7'/> <vcpupin vcpu='1' cpuset='2,8'/> <vcpupin vcpu='2' cpuset='3,9'/> <vcpupin vcpu='3' cpuset='4,10'/> <vcpupin vcpu='4' cpuset='5,11'/> </cputune> <cpu mode='host-passthrough' check='none' migratable='on'> <topology sockets='1' dies='1' cores='5' threads='1'/> <cache mode='passthrough'/> </cpu> Although my Unraid box is doing virtually nothing except running this VM I have observed that VM performance is worse than expected. First I did a test under native Win10 without Unraid. I have measured frequency and power when one, two and finally three CPU cores are fully utilized. Hera are results which I get under plain Win10: 1 Core load test (affnity CPU 4); temp= 90 C; freq= 4,6 GHz; pwr= ~28,5 W 2 Cores load test (affinity CPU 4, 6); temp= 96 C; freq= 4,57 GHz; pwr= ~45,7 W 3 Cores load test (affinity CPU 4, 6, 8); temp= 98 C; freq= 4,32 GHz; pwr= ~53,1 W, Thermal throttling These are sustainable numbers that I get after the test is running for some time. I think these results are something one could expect. However, when the same test is repeated on a Win10 VM hosted by Unraid I get inferior results. For one core during the first 20 seconds I get: 1 Core load test; temp= 96 C; freq= 4,4 GHz; pwr= ~32 W but when ~20 seconds have past it drops and stabilizes at: 1 Core load test; temp=78 C; freq= 4,0 GHz; pwr= ~22 W For two and three core loads we finally get: First 10 seconds: 2 Cores load test; temp= 97 C; freq= 4,4 GHz; pwr= ~50 W then: 2 Cores load test; temp= 76 C; freq= 3,4 GHz; pwr= ~22 W First ~7 seconds: 3 Cores load test; temp= 97 C; freq= 4,4 GHz; pwr= ~54 W then: 3 Cores load test; temp= 70 C; freq= 3,1 GHz; pwr= ~22 W (results gathered by corefreq-cli form CoreFreq plugin) It looks to me that except for a short period of time, Unraid keeps CPU in some kind of 22W power limit. Results observed in a native Win10 show that machine can do over 50 W in a long run (by the way Throttlestop shows default PL1=50W and PL2=90W), which comes with significant noise but also with a decent performance. For some reason it is not the case when machine is managed by Unraid with mysterious 22 W power limit in use. When VM is doing 3 cores load test, cpufreq-info on Unraid shows the following for all cores: cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009 Report errors and bugs to [email protected], please. analyzing CPU 0: driver: intel_pstate CPUs which run at the same hardware frequency: 0 CPUs which need to have their frequency coordinated by software: 0 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.80 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.80 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 3.10 GHz. analyzing CPU 1: driver: intel_pstate CPUs which run at the same hardware frequency: 1 CPUs which need to have their frequency coordinated by software: 1 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.80 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.80 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 3.10 GHz. which generally seems to be ok. Intel Pstate driver is running and Performance governor is in use. Yet, the actual frequency is low (3.10 GHz) due to some mysterious ~22W power limit. How to make Unraid to use some higher power limit than ~22W? With 22W limit Win10 VM can't even get close to the same Win10 running natively on the same hardware. Any ideas what could be wrong and how to make performance better? Edited November 1, 2022 by MarKol4 Quote Link to comment
Vr2Io Posted November 1, 2022 Share Posted November 1, 2022 3 hours ago, MarKol4 said: Unraid keeps CPU in some kind of 22W power limit. Never got Unraid will touch PL of CPU. Suggest try Disable Pstate driver to check still got PL 22w. FYR, haven't got problem when load test on Win10 VM ( with performance / powersave intel_pstate driver ) cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009 Report errors and bugs to [email protected], please. analyzing CPU 0: driver: intel_pstate CPUs which run at the same hardware frequency: 0 CPUs which need to have their frequency coordinated by software: 0 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.90 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 4.90 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 4.44 GHz. cpufrequtils 008: cpufreq-info (C) Dominik Brodowski 2004-2009 Report errors and bugs to [email protected], please. analyzing CPU 0: driver: intel_pstate CPUs which run at the same hardware frequency: 0 CPUs which need to have their frequency coordinated by software: 0 maximum transition latency: 4294.55 ms. hardware limits: 800 MHz - 4.90 GHz available cpufreq governors: performance, powersave current policy: frequency should be within 800 MHz and 3.60 GHz. The governor "powersave" may decide which speed to use within this range. current CPU frequency is 3.60 GHz. Quote Link to comment
MarKol4 Posted November 1, 2022 Author Share Posted November 1, 2022 (edited) I disabled Intel Pstate driver but it didn't help much. It is all the same that performance settles around 22W after a few seconds. Intel pstate_driver was replaced by acpi-cpufreq driver which can be seen in both - cpufreq-info and Tips&Twiks plugin (governor set to performance). cpufreq-info gives the following output: analyzing CPU 10: driver: acpi-cpufreq CPUs which run at the same hardware frequency: 10 CPUs which need to have their frequency coordinated by software: 10 maximum transition latency: 10.0 us. hardware limits: 800 MHz - 2.80 GHz available frequency steps: 2.80 GHz, 2.80 GHz, 2.70 GHz, 2.60 GHz, 2.40 GHz, 2.30 GHz, 2.10 GHz, 2.00 GHz, 1.80 GHz, 1.70 GHz, 1.50 GHz, 1.40 GHz, 1.20 GHz, 1.10 GHz, 900 MHz, 800 MHz available cpufreq governors: conservative, ondemand, userspace, powersave, performance, schedutil current policy: frequency should be within 800 MHz and 2.80 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 2.80 GHz (asserted by call to hardware). cpufreq stats: 2.80 GHz:95.61%, 2.80 GHz:0.00%, 2.70 GHz:0.00%, 2.60 GHz:0.02%, 2.40 GHz:0.03%, 2.30 GHz:0.00%, 2.10 GHz:0.01%, 2.00 GHz:0.00%, 1.80 GHz:0.02%, 1.70 GHz:0.21%, 1.50 GHz:0.02%, 1.40 GHz:0.10%, 1.20 GHz:0.02%, 1.10 GHz:0.09%, 900 MHz:0.96%, 800 MHz:2.90% (262) analyzing CPU 11: driver: acpi-cpufreq CPUs which run at the same hardware frequency: 11 CPUs which need to have their frequency coordinated by software: 11 maximum transition latency: 10.0 us. hardware limits: 800 MHz - 2.80 GHz available frequency steps: 2.80 GHz, 2.80 GHz, 2.70 GHz, 2.60 GHz, 2.40 GHz, 2.30 GHz, 2.10 GHz, 2.00 GHz, 1.80 GHz, 1.70 GHz, 1.50 GHz, 1.40 GHz, 1.20 GHz, 1.10 GHz, 900 MHz, 800 MHz available cpufreq governors: conservative, ondemand, userspace, powersave, performance, schedutil current policy: frequency should be within 800 MHz and 2.80 GHz. The governor "performance" may decide which speed to use within this range. current CPU frequency is 2.80 GHz (asserted by call to hardware). cpufreq stats: 2.80 GHz:95.62%, 2.80 GHz:0.00%, 2.70 GHz:0.00%, 2.60 GHz:0.01%, 2.40 GHz:0.00%, 2.30 GHz:0.02%, 2.10 GHz:0.01%, 2.00 GHz:0.05%, 1.80 GHz:0.26%, 1.70 GHz:0.11%, 1.50 GHz:0.01%, 1.40 GHz:0.19%, 1.20 GHz:0.02%, 1.10 GHz:0.09%, 900 MHz:0.19%, 800 MHz:3.42% (376) ,which is not fully consistent. According to cpufreq-info frequency should vary between 800 MHz and 2.8 GHz, but it is not exactly true. In the first seconds of the load test when power is not limited to 22W we can see much higher frequency in corefreq-cli and in /proc/cpuinfo: root@Tower:~# grep MHz /proc/cpuinfo cpu MHz : 4299.991 cpu MHz : 4255.252 cpu MHz : 4262.476 cpu MHz : 4290.778 cpu MHz : 4301.467 cpu MHz : 4300.020 cpu MHz : 4293.844 cpu MHz : 4304.088 cpu MHz : 4095.436 cpu MHz : 4296.918 cpu MHz : 4281.509 cpu MHz : 4300.015 Even in stable state when 3 core load is running for some time, the frequency stabilizes at 3.0-3.1 GHz which is higher than 2.8 upper bound shown by cpufreq-info. So in some way cpufreq-info is mistaken here because reported current frequency is lower than it actually is. I have checked loaded modules with "intel" phrase: root@Tower:~# lsmod | grep intel intel_powerclamp 16384 0 intel_gtt 24576 1 i915 kvm_intel 270336 2 intel_wmi_thunderbolt 16384 0 kvm 958464 1 kvm_intel crc32c_intel 24576 2 ghash_clmulni_intel 16384 0 aesni_intel 380928 0 crypto_simd 16384 1 aesni_intel cryptd 24576 2 crypto_simd,ghash_clmulni_intel intel_cstate 20480 0 intel_uncore 200704 0 btintel 36864 1 btusb bluetooth 475136 5 btrtl,btintel,btbcm,btusb agpgart 40960 2 intel_gtt,ttm intel_soc_dts_iosf 16384 1 processor_thermal_device_pci_legacy intel_pch_thermal 16384 0 iosf_mbi 16384 2 intel_soc_dts_iosf,i915 wmi 28672 3 intel_wmi_thunderbolt,wmi_bmof,mxm_wmi It looks like there could be some other modules which can have something to do with power management/limits. All in all intel_pstate driver was disabled and acpi-cpufreq took over, but it did not change anything with power limit. It still does not exceed 20-22 W in long run, which results in low frequency and poor performance. Any suggestions? Edited November 1, 2022 by MarKol4 Quote Link to comment
Vr2Io Posted November 1, 2022 Share Posted November 1, 2022 (edited) 2 hours ago, MarKol4 said: So in some way cpufreq-info is mistaken here because reported current frequency is lower than it actually is. Excellent founding. 👍 2 hours ago, MarKol4 said: Any suggestions? No idea right now. I have apply PL at different Intel platform in BIOS, btw, no observe abnormal at Unraid / VM / Windows baremetal. Will it possible longrun PL 22w was default ( although it doesn't make sense ), just baremetal Windows have correct this ? Could verify this under other Linux OS or try found any BIOS update. Edited November 1, 2022 by Vr2Io Quote Link to comment
MarKol4 Posted November 1, 2022 Author Share Posted November 1, 2022 Well, problem still not solved - but perhaps I have ruled out some possibilities. First I have checked that it does not matter whether load comes from a VM or local process running directly on Unraid box. For load testing I have used Stockfish chess program. Sometimes it is used for benchmarking by some portals. It is small, generates real workload, works from CLI and can use as many threads as you want. You may also easily run multiple single thread instances. All in all it did not change anything - I could see the same 20-22W power limit to kick in after a few seconds and stay at that level during the rest of the test. So even without VM in play it is bad. Then I thought I should follow the suggestion and check the same case on some other linux distro. I went for latest Ubuntu LTS. To my surprise I was able to do everything on Ubuntu Live booted directly from pendrive - even compile CoreFreq. Of course there was no changes to the hardware or any hardware settings - just plug in the pendrive, boot Ubuntu and do the testing. I did the same "Stockfish" test on Ubuntu and I must say - It worked Ok. It was keeping 50W (and running about 4.0-4.1 GHz for 3 core load test) which is the default setting for PL1. I used the same tools and same settings (Intel Pstate, performance governor). I have also checked powersave governor and it worked really well showing almost same results as performance. All in all everything was working just fine here and the power limit was kept at 50W as expected and not at 20-22W as I could see on Unraid. I looks like there is no hardware or hardware config (bios setting or anything else) issue since all is running fine on Ubuntu. There must be something specific in Unraid (6.11.1) which causes the problem with low power limit and poor performance. Maybe someone from Unraid team can have a look and suggest what could be wrong? 1 Quote Link to comment
ghost82 Posted November 2, 2022 Share Posted November 2, 2022 Seems related: https://www.reddit.com/r/FlowZ13/comments/vplgfg/tutorial_compeletely_get_rid_of_the_dumb_cpu/ that was for a z13, it could be the same issue. 1 Quote Link to comment
MarKol4 Posted November 3, 2022 Author Share Posted November 3, 2022 Thanks for the link. Interesting, but also pretty complicated and its hard to determine what really is the issue there and how does it link to my case. We should not forget that all my experiments done so far indicate, that neither Windows nor Ubuntu has a power limit problem here. Only Unraid does. On that basis it seems reasonable to think, that there is no issue with the hardware. And even if there was we have two good examples that it can be dealt with purely by software configuration without doing anything special - as we could see Windows and Ubuntu are working just fine out of the box - literally without doing anything special like manually correcting binary bios images. It must be some other way than extracting and modifying BIOSes to get it right - since it is already right on other OSes. I only hope that with your help we will be able to find that way. I would really wish to switch to Unraid as soon as possible, but these kind of unexpected and bizarre issues hold me back. I need to have them solved before moving on with Unraid. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.