No package C-states reached after starting VM

[alex3305]

Hi,

 

I've recently migrated to a new system. A really similar system to this system by @mgutt. Except I've chosen the Intel Core i5-13500 and a be quiet! Pure Power 12 M 550W PSU. Thanks again, mgutt, for the inspiration and very good guidelines which were easy to follow. I'm trying to make this system (very) energy efficient, which I think is coming along quite nicely.

 

Without anything loaded my idle usage is between 11-13w. Starting the array increases the usage to 14-15w. And starting my 20 or so Docker containers adds another 2-4w, totaling to about 16-19w. This is with 3 SSD's, 2 2.5" mechanical USB drives and 3 internal 3,5" drives.

 

While I'm really pleased with the results, I noticed that when I start a VM (for instance Home Assistant) power consumption increases to around 24-26w. Starting up PowerTOP evidently shows why:

 

 

and when I shut down the VM, C8 state is reached pretty much immediately:

 

 

After some trial and error I'm at a loss why this happens and what I can possibly do about this. Except migrating away from the VM. I was wondering if anyone else has observed this behavior and maybe has a solution? 

 

Thanks!

[mgutt]

1 hour ago, alex3305 said:

and when I shut down the VM, C8 state is reached pretty much immediately:

Which is pretty normal. C-States represent the different idle states of a CPU and its package and if your are using a VM, which produces several percent load on all cores, nothing is able to reach an idle state. Probably it is possible to influence it a little bit by isolating CPU cores and pinning the VM to 1 or 2 specific cores, but finally your target should be not to use VMs at all. Home Assistant for example runs perfectly fine as Docker Container.

[alex3305]

Thanks for your input @mgutt, I really appreciate it.

 

Currently I have pinned the VM to 4 of the 8 E-cores, but I also tried pinning it to a single P-core with CPU isolation, but that didn't make a difference. The host never gets to lower package C-states. 

 

Individual cores seem to get to C6 though, even on the assigned cores. Which makes me wonder if there is something off of my configuration. 

 

Btw I know that I can run HA and associated apps within Docker. And that would be my next test when I cannot get this working. I just kinda like the ease of use of HA OS.

[alex3305]

On 7/22/2023 at 11:24 PM, mgutt said:

[...] C-States represent the different idle states of a CPU and its package and if your are using a VM, which produces several percent load on all cores, nothing is able to reach an idle state. Probably it is possible to influence it a little bit by isolating CPU cores and pinning the VM to 1 or 2 specific cores [...]

 

Like I mentioned in my other reply, I already had isolated and pinned the VM to specific E-cores. But starting a VM - no matter what - prevents my Intel 13500 to enter any package C-state at all.

 

On 7/22/2023 at 11:24 PM, mgutt said:

But finally your target should be not to use VMs at all. Home Assistant for example runs perfectly fine as Docker Container.

 

So today I migrated all of my Home Assistant components to Docker containers. Mostly fairly trivial, except for some specific quirks. 

 

But running either Zigbee2MQTT or Z-Wave JS still prevents package C-state lower than C2. Power consumption is a bit better with about 25W in idle (running 45 containers), which was about around 30W with Home Assistant in a VM. But still nowhere near the 21W I was expecting.

Edited July 30, 2023 by alex3305
Full comment after editor bugged

[okkies]

Maybe forward Zigbee2mqtt towards an raspberry pi solution ? 

now i understand the format of these kind of things...

https://uzg.zig-star.com/
https://zig-star.com/projects/
https://mind.in.ua/ (support for ukraine <3)
(not affiliated)

Edited February 6 by okkies

[Juugis]

I have same CPU and similar issue. Home assistant, kubuntu and ubuntu VM works fine - at idle CPU goes to C3 normaly and total system power draw is 35w. Whenever I start windows vm CPU at idle wont go down to C3 state and total power draw will be 55w. When using linux vms power draw goes to 50-60w. I guess windows just sucks at CPU scheduling.

[alex3305]

On 2/6/2024 at 5:17 PM, okkies said:

Maybe forward Zigbee2mqtt towards an raspberry pi solution ? 

I have tested with a Pi 3B+ and ser2net which worked fine, but was IMHO very underpowered. So currently I have setup my old J4105 NUC-like with Debian, Docker and Docker Compose. There I just run Zigbee2MQTT and Z-WaveJS as a stack. I also added some other applications, such as a secondary backup server and secondary DNS. This setup consumes about 3W. Which is still less than the approximate 7W that the 13500 system consumes.

 

On 2/14/2024 at 12:53 PM, Juugis said:

I have same CPU and similar issue. Home assistant, kubuntu and ubuntu VM works fine - at idle CPU goes to C3 normaly and total system power draw is 35w. Whenever I start windows vm CPU at idle wont go down to C3 state and total power draw will be 55w. When using linux vms power draw goes to 50-60w. I guess windows just sucks at CPU scheduling.

This is probably an unrelated and different issue.

 

----

There also has been some discussion on the Intel Community regarding this issue. But still doesn't acknowledge it, but refers us back to 'Linux', whatever that means.

• Intel Community | High power consumption when using USB1.1 serial devices under Linux
• Intel Community | Poor power efficiency with B660/B760* boards and active** Full-Speed USB devices