Reduce power consumption with powertop

[hot22shot]

Hi,

 

I often see questions about the C1_ACPI to C3_ACPI C status (myself included to be honest) on some systems whereas on others the full list C state are shown.

Basically this seems to be caused by different idle driver used on those systems and this is not necessarly an issue.

More explanations on this thread : https://unix.stackexchange.com/questions/405775/acpi-idle-vs-intel-idle

 

FYI you can check the idle driver used on your system with corefreq.

Mine was using the intel_idle driver and powertop was showing C1_ACPI/.../C3_ACPI status.

When I switched to corefreq as an idle driver powertop was reporting the full C state list and I could see that I could reach C10.

But to be honest it did not make any difference on the power meter.

[bhcompy]

Anyone know if the QXP-800eS-A1164 (Asmedia 1164) PCIe SATA controller will work with autotune? 

 

Secondarily, if I run the autotune command, it doesn't persist on reboot, correct?  So I could try it and if I have a problem, I could reboot and it would resolve?

[DennyK]

Hello, I hope someone here can help me.

I bought a used Lenovo Thinkcentre M720q with i5 8400T, which is now running Unraid. I have deactivated the turbo boost and tried to configure the computer to be as energy efficient as possible. With 3 Docker containers, the power consumption at idle was 5-6W, which I am super happy with.
Now I wanted to move my Home Assistant VM from my older Proxmox server. Moving the VM itself was super easy. My problem now is that the CPU no longer throttles to the lowest 800 MHz when the VM is running (only 2 cores), but almost always at 1700 MHz. The power consumption with this one additional VM then rises to approx. 12 watts. I had already run Powertop with Auto-tune after setting up Unraid. With the VM, the total CPU utilization is also in the low 1-digit range. 

Do you have any idea how I can get the computer to slow down again?

 

[Vitek]

After run powertop --auto-tune &>/dev/null I have everything marked as good

 

but still maximum what I get is C3

 

What I am doing wrong? Do I need to restart the server after run auto-tune command?

 

[Sergii]

Hi,

 

I have moved from TrueNAS to Unraid and in the process of setting up.

On TN my setup was idling at 12W after issueing powertop command, but on Unraid power draw dosn't change for me, staying at 18W. Even all the Tunables are at "Good".

 

What am I doing wrong? How do I make powertop execute at boot?

 

Thanks for your help in advance

 

 

Edited March 28 by Sergii
Added Images with states

[dopeytree]

14 hours ago, DennyK said:

Hello, I hope someone here can help me.

I bought a used Lenovo Thinkcentre M720q with i5 8400T, which is now running Unraid. I have deactivated the turbo boost and tried to configure the computer to be as energy efficient as possible. With 3 Docker containers, the power consumption at idle was 5-6W, which I am super happy with.
Now I wanted to move my Home Assistant VM from my older Proxmox server. Moving the VM itself was super easy. My problem now is that the CPU no longer throttles to the lowest 800 MHz when the VM is running (only 2 cores), but almost always at 1700 MHz. The power consumption with this one additional VM then rises to approx. 12 watts. I had already run Powertop with Auto-tune after setting up Unraid. With the VM, the total CPU utilization is also in the low 1-digit range. 

Do you have any idea how I can get the computer to slow down again?

 

 

Would recommend moving it to a docker container set up & running everything separately bit of work initially but will be lower system overhead compared to a VM.

[_cjd_]

HA VM > docker isn't too bad unless you depend on a lot of internal stuff... In which case you have more than just HA docker needed. It took me a few hours and had to redo a bunch of customized bits as some resources changed name...

I only saved a little bit in power consumption but maybe the improvement on an AMD platform isn't as big. It has also been more stable. I used to have to restart 2-3 times every update. In the end, glad I did the switch.

[measure]

On 7/15/2022 at 7:57 PM, mgutt said:

(...)

But even Intel lost their focus on power efficiency after releasing the 10th gen and it became even worse with the 11th gen. The most efficient builds are still only possible with with 8th and 9th gen. And of course you should never add cards if the same feature is available on a motherboard. So your perfect board would be the Gigabyte W480M (8x SATA, 2x M.2 and 1x 2.5G ethernet port), which allows a 10 to 15W build.

 

On 8/13/2022 at 2:01 PM, mgutt said:

It will. But you need to know that a CPU usually runs inefficient on the highest frequencies. Every CPU has a "sweet spot", and above it, the CPU consumes more energy for a less performance impact. That's why every Intel CPU starting with the 11th generation has a huge Turbo TDP Limit. It's the only way for Intel to beat AMD which have the more efficient node platform at the moment. That's why they use extremely high base clocks, too (which are already outside if the sweet spot).

 

As an example, this German user found out, that the Ryzen 4650G has its sweet spot at 35W, which is the TDP Limit of the 4650GE:

https://www.computerbase.de/forum/threads/renoir-und-b550-die-idle-kuenstler.1967755/

 

 

On 8/27/2022 at 10:02 PM, mgutt said:

I've never seen an efficient MSI Board or 12th Gen Intel.

 

Which one? 2018 or 2021? The 2018 is much less efficient.

 

I think powertop does not display correct c-states for recent CPU models.

 

 

Sorry that I'm replying to such old posts. I have recently spent a lot of time researching this topic, including reading all the posts on this topic.

While reading various websites and forums, I came across at least several configurations where the power consumption was relatively low, despite the use of a 12th generation processor. I'm curious, do you still think that newer generations of Intel processors cannot be energy efficient compared to older generations? Could your assessment be due to observations of C0 stepping instead of H0? I am referring to this issue. Apparently the differences between one stepping and another are large.

Finally, the most important questions.

I know it's difficult to estimate because the processor is only one of the components that influence power consumption, but is it possible to estimate the differences in power consumption between the 10th and 12th generations?

Is it worth looking at older generations of processors in 2024, such as the 10th generation?

Edited March 30 by measure

[FlowDE]

My System whit an 12th Gen Intel CPU uses less than 10 Watts while no Storage is attached and my crapy Intel NIC isn't in use.

I think even the newer generations can use less Watt, maybe not extreme as some models whit 3 to 5 Watts but that is out of my scope.

[Vitek]

On 3/28/2024 at 11:17 AM, maTTi said:

After run powertop --auto-tune &>/dev/null I have everything marked as good

 

but still maximum what I get is C3

 

What I am doing wrong? Do I need to restart the server after run auto-tune command?

 

everybody with Realtek network card like me, try to run this command to force aspm: echo 1 | sudo tee /sys/bus/pci/drivers/r8169/0000\:0*\:00.0/link/l1_aspm replace 0* with your card id

 

after this I am reaching C8 and everything has aspm: enabled

[measure]

19 hours ago, FlowDE said:

My System whit an 12th Gen Intel CPU uses less than 10 Watts while no Storage is attached and my crapy Intel NIC isn't in use.

I think even the newer generations can use less Watt, maybe not extreme as some models whit 3 to 5 Watts but that is out of my scope.

That sounds pretty good. May I know what your CPU stepping is?

I'm guessing this Intel card is Intel I225V 2.5 Gb. 🙂

[FlowDE]

6 hours ago, measure said:

That sounds pretty good. May I know what your CPU stepping is?

I'm guessing this Intel card is Intel I225V 2.5 Gb. 🙂

 Yes its an I225V 

Vendor ID:               GenuineIntel
  BIOS Vendor ID:        Intel(R) Corporation
  Model name:            12th Gen Intel(R) Core(TM) i3-12100
    BIOS Model name:     12th Gen Intel(R) Core(TM) i3-12100 To Be Filled By O.E.M. CPU @ 4.0GHz
    BIOS CPU family:     206
    CPU family:          6
    Model:               151
    Thread(s) per core:  2
    Core(s) per socket:  4
    Socket(s):           1
    Stepping:            5
    CPU(s) scaling MHz:  15%
    CPU max MHz:         5500.0000
    CPU min MHz:         800.0000
    BogoMIPS:            6604.80

[measure]

12 hours ago, FlowDE said:

 Yes its an I225V 

Vendor ID:               GenuineIntel
  BIOS Vendor ID:        Intel(R) Corporation
  Model name:            12th Gen Intel(R) Core(TM) i3-12100
    BIOS Model name:     12th Gen Intel(R) Core(TM) i3-12100 To Be Filled By O.E.M. CPU @ 4.0GHz
    BIOS CPU family:     206
    CPU family:          6
    Model:               151
    Thread(s) per core:  2
    Core(s) per socket:  4
    Socket(s):           1
    Stepping:            5
    CPU(s) scaling MHz:  15%
    CPU max MHz:         5500.0000
    CPU min MHz:         800.0000
    BogoMIPS:            6604.80

Thank you for your response. You have H0 because all 12100 have H0. This is a 6-core die unlike C0 (8 P cores and the same number of E cores). For example, 12400 comes in both variants. Redundant cores are deactivated. The smaller H0 die seems to be more advantageous in terms of energy efficiency.

[KillianMelsen]

Hi, I've set up an Unraid server a couple weeks back and got around to setting up powertop as well. I installed it through the NerdTools plugin.

 

I've got the following system:

- ASUS TUF GAMING B560M-PLUS

- i5-11600K

- 2x Lexar NM710 2TB

- 2x Exos X20 18TB (spins down after 15 mins idle)

- A single fan on the bequiet pure rock 2 (the on included)

 

These are the relevant BIOS settings (I think) I've set up:

 

Intel Adaptive Boost Technology: Disabled

ASUS Performance Enhancement 2.0: Disabled

GPU Boost: Keep Current Settings

PCI Express Native Power Management: Enabled

Native ASPM: Enabled

DMI Link ASPM Control: L1

ASPM: L0sL1

L1 Substrates: L1.1 & L1.2

DMI Gen3 ASPM Control: ASPM L1 (The BIOS says that "L0s is not supported for DMI Aspm for CML/TGP", no idea what that means...)

DMI Gen3 ASPM: ASPM L1 (Same BIOS comment about CML/TGP here...)

PEG - ASPM: L0sL1

PCI Express Clock Gating: Enabled

Intel SpeedStep: Enabled

Intel Speed Shift Technology: Disabled

Turbo Mode: Disabled

CPU C-States: Enabled

Enhanced C-States: Enabled

CPU C6 Report: Enabled

CPU C7 Report: CPU C7

CPU C8 Report: Enabled

CPU C9 Report: Enabled

CPU C10 Report: Enabled

Package C-State Limit: C10

CFG Lock: Disabled

Thermal Monitor: Enabled

Dual Tau Boost: Disabled

SATA Mode Selection: AHCI (I have disabled all SATA ports except for 1 and 3 that have the drives attached)

Aggressive LPM Support: Enabled

Network Stack: Disabled

HD Audio: Disabled

Realtek LAN Controller: Enabled

USB Power delivery in Soft Off State (S5): Disabled

LED lighting when system is in working state: Stealth Mode: All LEDs will be disabled

Max Power Saving: Enabled

ErP Ready: Enable (S4+S5)

Power on by PCI-E: Disabled

Power on by RTC: Disabled

CPU Fan Profile: Silent.

 

I then ran powertop --autotune resulting in the data at the bottom of this post (lspci output at the very bottom in case it's useful). C3 is the highest C-state I can reach. Idle power consumption is now around 23W. Is there any way I can get this lower?

 

Thanks!

 

[kiwijunglist]

 

Ran the command

echo 1 | sudo tee /sys/bus/pci/drivers/r8125/0000:02:00.0/link/l1_aspm

 

It looks ok to my eyes:

 

 

However I still don't seem to get any better than C3 pkg(HW) state.

 

Anything else I should do.

Edited April 9 by kiwijunglist

[DennyK]

On 3/28/2024 at 10:51 PM, _cjd_ said:

HA VM > docker isn't too bad unless you depend on a lot of internal stuff... In which case you have more than just HA docker needed. It took me a few hours and had to redo a bunch of customized bits as some resources changed name...

I only saved a little bit in power consumption but maybe the improvement on an AMD platform isn't as big. It has also been more stable. I used to have to restart 2-3 times every update. In the end, glad I did the switch.

 

I have several addons running, which do not work under Docker. So Docker is not really an option for me.

Furthermore, you can't import the HA backup of the VM under Docker. I've been using HA for quite a while and don't want to reconfigure everything.

 

On the same M720q (i5 8400T, 32GB RAM) with Proxmox, I achieve a power consumption of 8-9W with 1 VM more and otherwise identical, while Unraid requires 12-13W. For a reason I can't explain, Unraid always seems to consume more power than Proxmox in the basic configuration and after optimisation.

 

I actually wanted to switch from my Futro S740 to the Lenovo M720q because I had IO problems with the Futro. After replacing the SATA M2 SSD, the IO problem seems to be solved. As I had never optimised the Futro to save energy, I have now done so. The result is that the Futro S740 with all VMs, LXC (Docker) is now at 5 W by default.

 

I therefore have the M720q available as a test system and will test it again soon with Unraid. In full idle mode under Proxmox (no VM/LXC running), the M720q currently consumes 3.7 W.

Edited April 9 by DennyK

[kiwijunglist]

On 1/18/2024 at 7:42 AM, genesisdoeswhatnintendont said:

Good news everyone!

I managed to get C10 pkg C-State (previously I get no higher than C3) on

Asrock LGA1700 mobo and you can too. Yay!

 

So you have to enable/change hidden BIOS menus by using AMISCE (AMI Setup Control Environment) utility v5.03 or 5.05 for Windows (it sometimes provided with MSI software products and can easily be found on the internet). So you have to install Windows and to enable Administrator password in your BIOS.

1 Run Powershell as admin and cd to folder where your AMISCE extracted when run this command:

 

.\SCEWIN_64.exe /o /s '.\setup_script_file.txt' /a

 

 

 

Thanks for this.

 

I searched for AMISCE utility and found it here - https://www.intel.com/content/www/us/en/download/19551/amisce-ami-setup-control-environment-utility-for-m10jnp2sb.html but the .zip file doesn't have SCEWIN_64.exe

 

Any help for this step?

[genesisdoeswhatnintendont]

@kiwijunglist

https://github.com/amitxv/SCEWIN

[BeersTeddy]

Be very careful playing with SCEWIN.
Some settings once changed doesn't seems to be possible to revert.

Flashing new BIOS, clearing cmos doesn't change this one back

Now all of my discs starting one after the other which takes abosulety forever to boot.

This the settings responsible for it

Setup Question    = PUIS Enable
Help String    = Enable/Disable Power-Up In Standby (PUIS) feature set allows devices to be powered-up into the Standby power management state to minimize inrush current at power-up and to allow the host to sequence the spin-up of devices.   
Token    =10A    // Do NOT change this line
Offset    =86E
Width    =01
BIOS Default    =[00]Disabled 
Options    =[01]Enabled    // Move "*" to the desired Option
         *[00]Disabled

 

Edited April 12 by BeersTeddy

[RedReddington]

Is anybody experiencing a similiar issue?

When I disable the Onboard NIC (Intel I225V) I can only reach C3.
When I enable it, I can reach C10.

 

I also flashed my Intel 710-DA4 to the newest firmware, the older firmware blocked higher C-States (it reported ASPM enabled and L1 before but was still not working).

 

And I can only run the powertop commands for each device on it's own.
With --autotune the server will be unreachable because of the Intel I225V.

That's why I tried to disable it... I can live with it for now, but 39W idle with all discs in idle with C10 still seems to be on a higher side.

[bagican]

yesterday I upgraded to v6.12.10 and did a quick measurement:

 

in average with running containers I have 4,2~4,5 Wh

 

[FlowDE]

21 hours ago, RedReddington said:

Is anybody experiencing a similiar issue?

When I disable the Onboard NIC (Intel I225V) I can only reach C3.
When I enable it, I can reach C10.

 

I also flashed my Intel 710-DA4 to the newest firmware, the older firmware blocked higher C-States (it reported ASPM enabled and L1 before but was still not working).

 

And I can only run the powertop commands for each device on it's own.
With --autotune the server will be unreachable because of the Intel I225V.

That's why I tried to disable it... I can live with it for now, but 39W idle with all discs in idle with C10 still seems to be on a higher side.

 

 

Yeah i got the same problem and probably the system uses more Watt than needed with that card even if its not used.

[1471]

Help!

 

This has been a really useful thread and a bit of a rabbit hole! I've been working my way through applying changes to try and save some power, but have an issue with getting in any C state with my array running.

 

Relevant bits of my setup:

 

ASRock Z790 ITX motherboard
Intel 12500 CPU
6 port MZHOU ASM1166 card
6x 20TB WD 'white lable' disks (1x as parity)
2x Samsung 780 SSD (ZFS mirros, cache)
2x 970 Evo Plus NVME (ZFS mirros, cache)
Lots of containers
2 VM's (Win Server 2022 and Home Assistant)

 

With the above set up but with a LSI 9207-8i I was getting an average power draw of 113W, which seemed high. Running powertop this reduced to 111W but I was unable to get to any C state. This was due to the HBA and a bluetooth dongle I was using.

 

So I did the following:

• I swapped the HBA for the recommended MZHOU 6 port ASM1166 card and updated the firmware to the latest available, HERE for info. I havent had any issues running 'powertop --auto-tune &>/dev/null' with this firmware (that I know of!)
• Following the guide HERE, using SCEWIN_64, I enabled 'Low Power S0 Idle Capability' and deactivated 'Native Aspm'
• I also removed the bluetooth dongle (moved to ESP32 bluetooth proxys)
• Installed the 'tips and tweaks' plugin and set the CPU governor to power save

and made some BIOS changes, relevant ones below:

• SR-IOV support - enabled
• PCI express native control - enabled (tried disabled first, no difference in power usage)
• PCI ASPM support - L0sL1
• PCH PCIE ASPM support - L1
• DMI ASPM support - enabled
• PCH DMI ASPM support - enabled
• Deep sleep - Enabled in S4-S5
• GNA device - disabled
• Onboard audio - disabled
• Onboard WAN device - disabled
• RGB LED - disabled

Left running for a few days and the average power was 95W, so a reduction of 16W, which is about right with the HBA removed, but still quite high.

 

Running powertop it seems like i'm not getting any Pkg(HW) C states which is odd.

 

With the array spun down its mostly running in C2:

 

 

But as soon as I spin up the array, it doesnt get to any C state:

 

 

Checking the ASPM status of all devices shows this. Seems fine, although all the red can't be a good sign..

 

 

Things i've tried and info

• Shutting down VM's - no difference
• All disks are encrypted, but don't think that makes a difference
• powertops showing all tunable options and 'good'
• Feels like there's a BIOS setting i've missed (dont thing I have though) or some hardware stopping things?
• I have a UPS connected via USB

 

From this thread I can see others have reached C8 or C10 with Asrock Z790 motherboards so it seems possible.

 

I've exhausted my limited expertise, so any help would be much appreciated!

Edited April 18 by 1471

[FlowDE]

19 hours ago, 1471 said:

Help!

 

This has been a really useful thread and a bit of a rabbit hole! I've been working my way through applying changes to try and save some power, but have an issue with getting in any C state with my array running.

 

Relevant bits of my setup:

 

ASRock Z790 ITX motherboard
Intel 12500 CPU
6 port MZHOU ASM1166 card
6x 20TB WD 'white lable' disks (1x as parity)
2x Samsung 780 SSD (ZFS mirros, cache)
2x 970 Evo Plus NVME (ZFS mirros, cache)
Lots of containers
2 VM's (Win Server 2022 and Home Assistant)

 

With the above set up but with a LSI 9207-8i I was getting an average power draw of 113W, which seemed high. Running powertop this reduced to 111W but I was unable to get to any C state. This was due to the HBA and a bluetooth dongle I was using.

 

So I did the following:

• I swapped the HBA for the recommended MZHOU 6 port ASM1166 card and updated the firmware to the latest available, HERE for info. I havent had any issues running 'powertop --auto-tune &>/dev/null' with this firmware (that I know of!)
• Following the guide HERE, using SCEWIN_64, I enabled 'Low Power S0 Idle Capability' and deactivated 'Native Aspm'
• I also removed the bluetooth dongle (moved to ESP32 bluetooth proxys)
• Installed the 'tips and tweaks' plugin and set the CPU governor to power save

and made some BIOS changes, relevant ones below:

• SR-IOV support - enabled
• PCI express native control - enabled (tried disabled first, no difference in power usage)
• PCI ASPM support - L0sL1
• PCH PCIE ASPM support - L1
• DMI ASPM support - enabled
• PCH DMI ASPM support - enabled
• Deep sleep - Enabled in S4-S5
• GNA device - disabled
• Onboard audio - disabled
• Onboard WAN device - disabled
• RGB LED - disabled

Left running for a few days and the average power was 95W, so a reduction of 16W, which is about right with the HBA removed, but still quite high.

 

Running powertop it seems like i'm not getting any Pkg(HW) C states which is odd.

 

With the array spun down its mostly running in C2:

 

 

But as soon as I spin up the array, it doesnt get to any C state:

 

 

Checking the ASPM status of all devices shows this. Seems fine, although all the red can't be a good sign..

 

 

Things i've tried and info

• Shutting down VM's - no difference
• All disks are encrypted, but don't think that makes a difference
• powertops showing all tunable options and 'good'
• Feels like there's a BIOS setting i've missed (dont thing I have though) or some hardware stopping things?
• I have a UPS connected via USB

 

From this thread I can see others have reached C8 or C10 with Asrock Z790 motherboards so it seems possible.

 

I've exhausted my limited expertise, so any help would be much appreciated!

Check your Network-Card

[1471]

Thanks, the AsRock Z790 ITX motheboard i'm using has two onboard network cards, an Intel I226V (2.5G) and I219V. I have the I219V disabled, tried swapping round and no change.

 

With a bit more testing I found that with the Win Server 2022 VM disabled I was getting ~20% C2, not brilliant but at least Pkg(HW) is working. Any idea what could cause that? Containers are fine, tried spinning one up at a time with the VM disabled and didnt effect things.

 

I've been through the BIOS quite a few times not and can't see any obvious settings that are wrong, so suspect its something else.