Reduce power consumption with powertop

[FlowDE]

Start from scratch. Zero VMs or Docker, No Storage and disconnect all Network-Cables just a Monitor, Mouse and Keyboard and check the C-State. Then look if you can disable something like bluetooth and wifi or led stuff. 

[1471]

2 hours ago, FlowDE said:

Start from scratch. Zero VMs or Docker, No Storage and disconnect all Network-Cables just a Monitor, Mouse and Keyboard and check the C-State. Then look if you can disable something like bluetooth and wifi or led stuff. 

Thanks, yea probably the best option at this point. A job for next week.

[Napoleon]

I've just got a new motherboard. Changed all the settings in the bios power settings. Aspm is enabled (also says this is OS controlled ASPM), C state settings enabled, and disabled what I'm not using in the bios, powertop has been run and is all on good. It is idling at 20watts but C2-C10 on the pkg(hw) is 0.

Double checked they are enabled in the bios, but it seems C states for the pkg is not working?

C246 board/Xeon E-2174G/2x ECC/EVGA 650GM/USB stick/1 fan.

Edit: 4x 2.5" SSDs and 2x 2.5" HDs and its up to 64Watts with the array not running.

Edit 2: NVME added via PCIE adaptor, no change.

Edit 3: Added: 15 3.5" drives, an SF750 to run them, LSI HBA, SAS expander, 3 fans - 204watts.

Most drives spun down - 106 watts.

Edit 4: 10Gb intel X540 added, onboard ethernet disabled. About 116 watts but one drive won't and 3 SSDs won't spin down.

Now seen it go as low as 78 watts with 1 drive and 3 SSDs still running.

Still no C states though.

Screenshots added below.

Edited April 22 by Napoleon

[Napoleon]

I've gone back to the first config above. Nothing connected but changed the PSU for an HDPLEX 250W passive GaN AIO ATX Power Supply.

C246 board/Xeon E-2174G/2x ECC/USB stick/1 fan.
 

Idles at around 11 watts now, but still 0% on all C states.

 

But it seems to now be giving me 2 lots of 5 beeps at boot. CMOS Reset seemed to fix this one. Unsure which setting gave that beeping error. Edit: New CMOS battery seems to have fixed this one.

C states still aren't working with different pentium CPU, PSU or non ECC ram.

Edited May 4 by Napoleon

[Napoleon]

Still struggling with any C states on the C246 board above, so tried another set I had around.

 

AS Rock H310CM-HDV, Pentium G5400. 2x 8GB sticks. No drives. HDplex PSU mentioned above. Reaches C7. Idling at a whopping 4-6 watts.

Edited April 22 by Napoleon

[methanoid]

I would like my idle state to not be disturbed by random 'Arr activity - if you think likewise feel free to comment here

 

 

[Theroid]

When I try to tune my intel i226 ethernet controller the system becomes totally unresponsive. Is this normal? My solution right now is to tune everything but this ethernet controller.

[NotEnoughStorage]

Hey,
I just wanted to respond to the issue of "sticking at C3 state," which nearly drove me crazy as I spent the past six hours trying to resolve it.

 

I began by implementing all the BIOS configurations discussed in this forum, including ASPM, CPU performance settings, and disabling every onboard device except the Ethernet controller. Later, it turned out that the Ethernet controller was the problem. I also applied the PowerTOP tuning, which allowed me to reach the C3 state, as you can see here:

 

 

However, at this point, I was unable to progress to further states, even though everything appeared to function correctly, with PCI devices showing "ASPM enabled."

 

 

This issue persisted also on Ubuntu 24.04 LTS, which I tested using a live USB and PowerTOP.
Ultimately, I managed to reach C8 by disabling the Ethernet controller:

 

 

My idle power consumption dropped from 30 watts to 18 watts, measured with a basic wall power meter, with two HDDs and one SSD connected and running.
Obviously, this isn't a permanent solution, so I'm planning to buy a NIC with a better Ethernet controller. Does anyone have any recommendations?

My setup:
•    CPU: Intel i3-12100
•    ASUS PRIME H610I-PLUS D4
•    G.Skill Aegis 2 x 16GB DDR4-3200 (part from a gaming PC I already had)
•    Be quiet! Pure Power 11 600W (part from a gaming PC I already had)

 

[hot22shot]

Before changing of NIC did you tried this post : 

 

Not sure it can be applied to your case, as you have a r8111, but it might be worth a try.

[FlowDE]

  

On 5/1/2024 at 9:09 AM, Theroid said:

When I try to tune my intel i226 ethernet controller the system becomes totally unresponsive. Is this normal? My solution right now is to tune everything but this ethernet controller.

Yes its "normal" and we should raise a Support-Ticket at Intel. A Colleague told me that Intel has its own development for linux driver.

[bromerico975]

On 5/1/2024 at 6:54 PM, NotEnoughStorage said:

Hey,
I just wanted to respond to the issue of "sticking at C3 state," which nearly drove me crazy as I spent the past six hours trying to resolve it.

 

I began by implementing all the BIOS configurations discussed in this forum, including ASPM, CPU performance settings, and disabling every onboard device except the Ethernet controller. Later, it turned out that the Ethernet controller was the problem. I also applied the PowerTOP tuning, which allowed me to reach the C3 state, as you can see here:

 

 

However, at this point, I was unable to progress to further states, even though everything appeared to function correctly, with PCI devices showing "ASPM enabled."

 

 

This issue persisted also on Ubuntu 24.04 LTS, which I tested using a live USB and PowerTOP.
Ultimately, I managed to reach C8 by disabling the Ethernet controller:

 

 

My idle power consumption dropped from 30 watts to 18 watts, measured with a basic wall power meter, with two HDDs and one SSD connected and running.
Obviously, this isn't a permanent solution, so I'm planning to buy a NIC with a better Ethernet controller. Does anyone have any recommendations?

My setup:
•    CPU: Intel i3-12100
•    ASUS PRIME H610I-PLUS D4
•    G.Skill Aegis 2 x 16GB DDR4-3200 (part from a gaming PC I already had)
•    Be quiet! Pure Power 11 600W (part from a gaming PC I already had)

 

Hi,

try this:

echo 1 | sudo tee /sys/bus/pci/drivers/r8169/0000:02:00.0/link/l1_2_aspm

This activates ASPM for the Realtec NIC.

[Napoleon]

I'm still struggling with reaching 0% C states on this C246 board. Server seems to work fine aside from that now and I've got 12 hours left to open a return if I have to.

ASPM is enabled in the BIOS, I've also tried auto. Despite that, I get this:

Also tried disabling the onboard ethernet and other PCIE slots. Fitting an X540. Still says disabled.

[mgutt]

1 minute ago, Napoleon said:

this C246 board

Which one?

[Napoleon]

ASMB-586

[Napoleon]

4 minutes ago, mgutt said:

Which one?

https://downloadt.advantech.com/download/downloadsr.aspx?File_Id=1-1V74DQL

Manual there if that helps

[Napoleon]

This is enabled. No MTU setting to be found. Current testing state has nothing but an ethernet card connected, both onboard NICs disabled, even disabled the onboard sata controller.

 

Edited May 4 by Napoleon

[mgutt]

Disconnect all Ethernet ports and test with a display. Test even with Ubuntu and without any USB devices.

 

 

[Napoleon]

Tried the first suggestion. Still 0 on the pkg despite being 97% C7 on the core. Same USB gets C7 on another motherboard, processor and RAM, but I've also tried that processor and RAM in this one with the same results.
 

[Napoleon]

Another data point for everyone:

AS Rock H310CM-HDV, Pentium G5400. 2x 8GB sticks, 1 NVME, 4 2.5" SSDs running from an HDplex PSU mentioned above. LSI HBA, Expander, 17 3.5" drives, all spun down except a WD black SN850X with the drives being ran from an SF750 reaches C3 for 67% and coming in at around 50 watts with the web gui open. Turning off the SF750 sees it come down to around 17 watts.

Same as above but with the C246 and Xeon idles at 60 watts with no pkg c states functioning

Having 2 SSDs on and a downloader see that hover around 20-23 watts and C3 at around 20%.

All drives spun up around 140 watts.

Edited May 4 by Napoleon

[MiguelNdeCarvalho]

On 12/1/2023 at 3:33 PM, unr41dus3r said:

Maybe someone has an idea, i got an Intel Arc 380 and ASPM in Bios is enabled and i also seewit lapci ASPM is enabled.

I try now to get them to lowest state and in Powertop i see it is in RC6 state but not going to RC6p or RC6pp.

Any idea what i could try?

I am also using custom Thor Kernel for the use of ARC, but as i know the RC6 states are normal for intel?

My C states are only from 1-3 but as i read its normal in powertop for AMD CPUs

Hey, I'm exactly on your case, did you find a solution for it?

[1471]

On 4/20/2024 at 7:00 AM, FlowDE said:

Start from scratch. Zero VMs or Docker, No Storage and disconnect all Network-Cables just a Monitor, Mouse and Keyboard and check the C-State. Then look if you can disable something like bluetooth and wifi or led stuff. 

 

Right then, I had a bit of time to do some testing earlier in the week, it seems like:

• Win Server 2022 VM's stops getting to C2
• PCI Sata card stops C6
• NVME / SSD's stops C8

Frst thing to look at is the VM issue, then look at the sata card as supports ASPM and shouldn't be causing problems. I'd be happy if I could get to C6 for now.

 

Any help would be appreciated.

 

Previous post earlier in this thread with server spec and configuration HERE for info and my testing notes below:

_____________________________________________

 

Leaving powertop running for 2 mins before recording C state. Looking at Pkg(HW) only, CPU C states are fine.

 

SERVER TESTING

array on - No C state

array off - C2 77%

array on (VM off, docker on) - C2 21.5%

array on (VM on, docker off) - No C state

array on (VM Win on only, docker off) - No C state

array on (VM HA on only, docker off) - C2 43.3%

array on (VM off, docker off) - C2 54%

 

NOTES: 

- Seems like the Win Server 2022 VM’s stopping the server from getting to C2

- With the Win VM off and getting to C2, power usage drops from an average of 95W to 75W (a 20W saving)

 

THINGS REMOVED (array off)

SATA Card - SSD removed - C2 40.6% + C6 47.9%

SATA Card - SSD - NVME removed - C2 37.4% + C3 11.6 + C6 1.7% + C8 39.4%

SATA Card - NVME removed - C2 27.2.6% + C6 57.5%

SSD - NVME removed - C2 91.9%

 

NOTES:

- Seems like the SATA Card is stopping the server from getting to C6 and the SSD's or NVME's are stopping C8.

- This should be fine as running the latest firmware available which supports ASPM

 

TO TRY:

- Create fresh Win11 VM and see if can get C2 with just that running - DONE - NO DIFFERENCE

- Remove mapped drive in Win VM and turn off indexing? - DONE, NO DIFFERENCE

- Update BIOS version to latest version - DONE, NO DIFFERENCE

- Check firmware of card and maybe lower to recommended?

- Set 'PCI native control' in BIOS to disabled (let the BIOS control ASPM)?

 

Edited May 12 by 1471
added some more testing notes

[1471]

On 5/10/2024 at 10:48 AM, 1471 said:

TO TRY:

- Create fresh Win11 VM and see if can get C2 with just that running - DONE - NO DIFFERENCE

- Remove mapped drive in Win VM and turn off indexing? - DONE, NO DIFFERENCE

- Update BIOS version to latest version - DONE, NO DIFFERENCE

- Check firmware of card and maybe lower to recommended?

- Set 'PCI native control' in BIOS to disabled (let the BIOS control ASPM)?

 

I don't suppose anyone can run a little test for me? Are you able to hit any C States with just a Windows VM running (indeally Win Server and an intel CPU)?

 

Wondering if the reason I can't get to any C state with a Win VM running is local to me or not.

Edited May 12 by 1471

[Napoleon]

More data:

 

Asus H110I Thin ITX, I5 7500T, powered by a 95 watt Dell laptop brick. No drives, 2x 8GB sodimms, usb stick - 6-7 watts. Forgot to check C States.

Dell 3630, C246 board, Xeon E-2186G, 1x 16gb 2666 non ecc, original Dell PSU, only reaches C3 on the PKG but shows up to C10. No drives and idles at 12-14 watts. Only option for C states in the bios is enabled or disabled.

ASRock Rack E3C246D4U2-2T, Xeon E-2186G, 2x 16GB ECC, HDplex ATX 250W PSU, only sees C3 on the PKG with 10gbe running. No drives, idles around 15-16 watts.

Edited May 22 by Napoleon

[nassauer]

Joining in with:

Gigabyte B760M DS3h DDR4

Core i3 1200 (no letter)

32 Gig Corsair Ram

PSU beQuiet BQT9-700W

 

unraid downgraded to 6.12.4

 

I have set everything in Bios to what I have found in almost any thread around here.

I have NOTHING connected, despite the USB stick and network. No Sata, no NVME, no HDMI, no keyboard.

I am stuck at 20 Watts idle. I had C10 yesterday, but still 20 watts. today I only get to C8.

I´m using a 10+ years old PSU beQuiet 700W. But from what I could find it is not too shaby at low loads.

I´f I could get anywhere near 10 Watts idle I would consider sourcing a better PSU.

 

Here is my Go File:

#!/bin/bash
# Start the Management Utility
/usr/local/sbin/emhttp
echo 1 | sudo tee /sys/bus/pci/drivers/r8169/0000\:02\:00.0/link/l1_2_aspm

powertop --auto-tune &>/dev/null

# -------------------------------------------------
# Set power-efficient CPU governor
# -------------------------------------------------
/etc/rc.d/rc.cpufreq powersave

# -------------------------------------------------
# Disable CPU Turbo
# -------------------------------------------------
[[ -f /sys/devices/system/cpu/intel_pstate/no_turbo ]] && echo "1" > /sys/devices/system/cpu/intel_pstate/no_turbo
[[ -f /sys/devices/system/cpu/cpufreq/boost ]] && echo "0" > /sys/devices/system/cpu/cpufreq/boost

# -------------------------------------------------
# Enable power-efficient ethernet
# -------------------------------------------------

# enable IEEE 802.3az (Energy Efficient Ethernet): Could be incompatible to LACP bonds!
for i in /sys/class/net/eth?; do dev=$(basename $i); [[ $(echo $(ethtool --show-eee $dev 2> /dev/null) | grep -c "Supported EEE link modes: 1") -eq 1 ]] && ethtool --set-eee $dev eee on; done

# Disable wake on lan
for i in /sys/class/net/eth?; do ethtool -s  $(basename $i) wol d; done

# -------------------------------------------------
# powertop tweaks
# -------------------------------------------------

# Enable SATA link power management
echo med_power_with_dipm | tee /sys/class/scsi_host/host*/link_power_management_policy

# Runtime PM for I2C Adapter (i915 gmbus dpb)
echo auto | tee /sys/bus/i2c/devices/i2c-*/device/power/control

# Autosuspend for USB device
echo auto | tee /sys/bus/usb/devices/*/power/control

# Runtime PM for disk
echo auto | tee /sys/block/sd*/device/power/control

# Runtime PM for PCI devices
echo auto | tee /sys/bus/pci/devices/????:??:??.?/power/control

# Runtime PM for ATA devices
echo auto | tee /sys/bus/pci/devices/????:??:??.?/ata*/power/control

Edited May 29 by nassauer

[mgutt]

12 minutes ago, nassauer said:

 

I am stuck at 20 Watts idle. I had C10 yesterday, but still 20 watts. today I only get to C8.

I´m using a rather old PSU beQuiet 700W so it is not all too efficient. I´f I could get anywhere near 10 Watts idle I would consider sourcing a better PSU.

Not sure about the 10W, but I think saving 7W should be realistic. Can you find your specific power supply model in a review of tweakpc.de ? They test the efficiency at 10W load. 

 

PS it seems you tested your c state with the unraid web terminal. The webgui produces load, so you should close all browsers and connect through SSH to your server.