Reduce power consumption with powertop

[Vrozaksen]

@Louisgaga
Yes 00:01.0
There is modified script just for this one:
 

#!/bin/bash
ENDPOINT="00:01.0"
ROOT_COMPLEX="00:01.0"

# Hex  Binary  Meaning
# -------------------------
# 0    0b00    L0 only
# 1    0b01    L0s only
# 2    0b10    L1 only
# 3    0b11    L1 and L0s
ASPM_SETTING=3

function aspm_setting_to_string()
{
	case $1 in
	0)
		echo -e "\t${BLUE}L0 only${NORMAL}, ${RED}ASPM disabled${NORMAL}"
		;;
	1)
		;;
	2)
		echo -e "\t${GREEN}L1 only${NORMAL}"
		;;
	3)
		echo -e "\t${GREEN}L1 and L0s${NORMAL}"
		;;
	*)
		echo -e "\t${RED}Invalid${NORMAL}"
		;;
	esac
}

# Pretty colors
GREEN="\033[01;32m"
YELLOW="\033[01;33m"
NORMAL="\033[00m"
BLUE="\033[34m"
RED="\033[31m"
PURPLE="\033[35m"
CYAN="\033[36m"
UNDERLINE="\033[02m"

# we can surely read the spec to get a better value
MAX_SEARCH=20
SEARCH_COUNT=1
ASPM_BYTE_ADDRESS="INVALID"

ROOT_PRESENT=$(lspci | grep -c "$ROOT_COMPLEXT")
ENDPOINT_PRESENT=$(lspci | grep -c "$ENDPOINT")

if [[ $(id -u) != 0 ]]; then
	echo "This needs to be run as root"
	exit 1
fi

if [[ $ROOT_PRESENT -eq 0 ]]; then
	echo "Root complex $ROOT_COMPLEX is not present"
	exit
fi

if [[ $ENDPOINT_PRESENT -eq 0 ]]; then
	echo "Endpoint $ENDPOINT is not present"
	exit
fi

# XXX: lspci -s some_device_not_existing does not return positive
# if the device does not exist, fix this upstream
function device_present()
{

	PRESENT=$(lspci | grep -c "$1")
	COMPLAINT="${RED}not present${NORMAL}"

	if [[ $PRESENT -eq 0 ]]; then
		if [[ $2 != "present" ]]; then
			COMPLAINT="${RED}disappeared${NORMAL}"
		fi

		echo -e "Device ${BLUE}${1}${NORMAL} $COMPLAINT" 
		return 1
	fi
	return 0
}

function find_aspm_byte_address()
{
	device_present $ENDPOINT present
	if [[ $? -ne 0 ]]; then
		exit
	fi

	SEARCH=$(setpci -s $1 34.b)
	# We know on the first search $SEARCH will not be
	# 10 but this simplifies the implementation.
	while [[ $SEARCH != 10 && $SEARCH_COUNT -le $MAX_SEARCH ]]; do
		END_SEARCH=$(setpci -s $1 ${SEARCH}.b)

		# Convert hex digits to uppercase for bc
		SEARCH_UPPER=$(printf "%X" 0x${SEARCH})

		if [[ $END_SEARCH = 10 ]]; then
			ASPM_BYTE_ADDRESS=$(echo "obase=16; ibase=16; $SEARCH_UPPER + 10" | bc)
			break
		fi

		SEARCH=$(echo "obase=16; ibase=16; $SEARCH + 1" | bc)
		SEARCH=$(setpci -s $1 ${SEARCH}.b)

		let SEARCH_COUNT=$SEARCH_COUNT+1
	done

	if [[ $SEARCH_COUNT -ge $MAX_SEARCH ]]; then
		echo -e "Long loop while looking for ASPM word for $1"
		return 1
	fi
	return 0
}

function enable_aspm_byte()
{
	device_present $1 present
	if [[ $? -ne 0 ]]; then
		exit
	fi

	find_aspm_byte_address $1
	if [[ $? -ne 0 ]]; then
		return 1
	fi

	ASPM_BYTE_HEX=$(setpci -s $1 ${ASPM_BYTE_ADDRESS}.b)
	ASPM_BYTE_HEX=$(printf "%X" 0x${ASPM_BYTE_HEX})
	# setpci doesn't support a mask on the query yet, only on the set,
	# so to verify a setting on a mask we have no other optoin but
	# to do do this stuff ourselves.
	DESIRED_ASPM_BYTE_HEX=$(printf "%X" $(( (0x${ASPM_BYTE_HEX} & ~0x7) |0x${ASPM_SETTING})))

	if [[ $ASPM_BYTE_ADDRESS = "INVALID" ]]; then
		echo -e "No ASPM byte could be found for $(lspci -s $1)"
		return
	fi

	echo -e "$(lspci -s $1)"
	echo -en "\t${YELLOW}0x${ASPM_BYTE_ADDRESS}${NORMAL} : ${CYAN}0x${ASPM_BYTE_HEX}${GREEN} --> ${BLUE}0x${DESIRED_ASPM_BYTE_HEX}${NORMAL} ... "

	device_present $1 present
	if [[ $? -ne 0 ]]; then
		exit
	fi

	# Avoid setting if already set
	if [[ $ASPM_BYTE_HEX = $DESIRED_ASPM_BYTE_HEX ]]; then
		echo -e "[${GREEN}SUCESS${NORMAL}] (${GREEN}already set${NORMAL})"
		aspm_setting_to_string $ASPM_SETTING
		return 0
	fi

	# This only writes the last 3 bits
	setpci -s $1 ${ASPM_BYTE_ADDRESS}.b=${ASPM_SETTING}:3

	sleep 3

	ACTUAL_ASPM_BYTE_HEX=$(setpci -s $1 ${ASPM_BYTE_ADDRESS}.b)
	ACTUAL_ASPM_BYTE_HEX=$(printf "%X" 0x${ACTUAL_ASPM_BYTE_HEX})

	# Do not retry this if it failed, if it failed to set.
	# Likey if it failed its a good reason and you should look
	# into that.
	if [[ $ACTUAL_ASPM_BYTE_HEX != $DESIRED_ASPM_BYTE_HEX ]]; then
		echo -e "\t[${RED}FAIL${NORMAL}] (0x${ACTUAL_ASPM_BYTE_HEX})"
		return 1
	fi

	echo -e "\t[${GREEN}SUCCESS]${NORMAL}]"
	aspm_setting_to_string $ASPM_SETTING

	return 0
}

device_present $ENDPOINT not_sure
if [[ $? -ne 0 ]]; then
	exit
fi

echo -e "${CYAN}Root complex${NORMAL}:"
ROOT_COMPLEX="00:01.0"
enable_aspm_byte $ROOT_COMPLEX
echo

echo -e "${CYAN}Endpoint${NORMAL}:"
ENDPOINT="00:01.0"
enable_aspm_byte $ENDPOINT
echo

 

[theothermatt_b]

On 10/28/2023 at 7:21 AM, Lolight said:

Yeah, but his board's manual doesn't mention anything about its existence.

According to the manual all SATA ports are connected to the chipset.

Yeah wait hold up. I thought I was just not understanding something.

It DOES say that everything is connected to the Intel B660 chipset.

 

In fact there aren't any duck duck go results (in english at least) containing "asus" "b660m" and "asm1061"

 

@mgutt is it possible you're thinking of the ASRock B660M? That one mentions ASM1061...

 

EDIT: Looking at the Device Stats screen in Powertop, I do see mention of ASMxxxx stuff, but its ASM1062 and ASM107x. Unclear if that matters.

Edited October 30, 2023 by theothermatt_b

[Louisgaga]

 

Thanks!
I needed to add the BC command with nerd pack.

ASPM is now enabled, unfortunately it didn't change anything on Pkg Cstates : c3 average at 75% but it don't go lower.

Idle at 23W

[nebb00]

Yet another C-State question.

 

Motherboard: Asrock B760M Steel Legend WiFi

CPU: Intel i5-13500

Memory: Corsair Vengeance DDR5 5600MHz 64GB (2x 32GB)

PSU: Seasonic Focus PX 550 Platinum

M.2: 2x Crucial P5 Plus 2TB

HDD: 2x 20TB Toshiba MG10ACA20TE

 

Currently have no containers or VMs running as its a fresh build. Currently can hit C3 and drawing 28w at the wall.

 

Things done

• Powertop --auto-tune
• BIOS updated
• All C-States enabled
• HD Audio / WAN disabled
• All the ASPM functions enabled

Anything else I could possibly do? All the tunables are good.

 

Edited November 6, 2023 by nebb00

[jit-010101]

Well I recon its the SATA-Controller / PCIe device / Mainboard Components difference here at play in terms of hardware.

 

Maybe its related to firmware of the controllers too ... apparently ASRock is known for blocking lower then C3 on many of their boards (I can confirm that for my AM4 mainboard, even with ASPM enabled) so there seems to be a limit as to what you can do to improve idle consumption.

 

But apparently not all Chinaboards are bad either too!

 

I have this here:

 

https://www.amazon.de/gp/product/B0BYVMNMR9/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1

 

Since my board (Green PCB - BKHD-NAS-XXXX Label Onboard), almost the same as the Changwang NAS-Board (Black PCB - CW-NAS-XXXX Label onboard) - biggest difference was that mine has a ASM1166 SATA-Controller for the 6x SATA-Ports On-Board.

 

All PCIe Slots had ASPM disabled and I was stuck at C3 - after modding the BIOS and enabling it afterwards per Port (auto) I'm idling at 11.8W now successfully moving into C8 state with N5105 with 3HDDs and 1 NVMe connected (spindown, obviously) after all the powertop tuning (that's with an PicoPSU 200W, mind you). 

 

 

 

 

 

Edited November 8, 2023 by jit-010101

[Vrozaksen]

Is there someone with Asus TUF Z390M-PRO GAMING and got more than C3?

[fightforlife]

Did someone ever had problems with the onboard Intel SATA controller? As soon as it's used it is adding 4-5W for me, since c states are limited to C2.

 

My setup:

- i5 11400

- Asus prime b560m-a (latest bios)

- 2x 8gb 2666 ddr4 1.2V

- bequiet system power 9 400 W

- only power and LAN connected

 

When I Connect the same SATA boot SSD to different ports, I get very strange results.

1. To chipset b560 SATA ~16W, C2 max

2. To chipset USB3 via Adapter ~11W C8 max

3. To Pcie Asm1166 in B560 pcie slot ~11W C8 max

 

I also tried different SATA devices while booting from USB. As soon as one is connected to the b560 chipset the max cstate will be C2.

 

I already checked the following:

- sata hot plug disabled (/enabled does not matter)

- sata alpm enabled (/disabled does not matter)

- pcie and dmi ASPM native and L1

- powertop --auto-tune and TLP installed.

 

Does anyone have an idea?

I would try bios downgrades next. 

 

P.S.: setting the cpu memory controller to Gear 2 instead of Gear 1 also saves 3-4W.

Edited November 19, 2023 by fightforlife
Added info regarding other sata devices.

[fightforlife]

Solution: 
In the end I plugged every single device individually into all SATA ports to find the cultprits.
My findings are:

• old HDD SAMSUNG HD501LJ is limiting package C-States to PC6.
• new SSD Emtec X150 240GB is limiting package C-States to PC2.
• dvd writer HL-DT-ST DVDRAM GH24NSD5 is limiting package C-States to PC2.

I did a comparison with the drives sata cables attached and not attached, while booting from USB. (all power cables still attached)
The difference between package C-state PC6 and PC2 seems the be around 2,5-3W.
The processor either way seems to be going into C7 or lower. For now I will just accept 3W more instead of buying new drives. My current power consumption with all devices attached but in spindown/sleep is around 20W. 
I hoped to get lower like here https://mattgadient.com/7-watts-idle-on-intel-12th-13th-gen-the-foundation-for-building-a-low-power-server-nas/
But strangely my PC6 vs. PC2 consumption isnt nearly is dramatic as described there.
The lowest I was able to get wihtout any drives and no ASM1166 was 13W.

 

Configurration for reference:
Intel i5-11400 (-0.05V)
2x8GB + 1x 16GB 2666 DDR4
Asus PRIME B560M-A
ASM1166 PCIE NVME to 6x Sata Adapter
2x SATA SSD + 4x SATA HDD + 1x DVD
All C-States enabled + ASPM + ALPM
TLP + Powertop Auto tune

[jit-010101]

On 11/24/2023 at 8:53 PM, fightforlife said:

20w ...

 

Configurration for reference:
Intel i5-11400 (-0.05V)
2x8GB + 1x 16GB 2666 DDR4
Asus PRIME B560M-A
ASM1166 PCIE NVME to 6x Sata Adapter
2x SATA SSD + 4x SATA HDD + 1x DVD
All C-States enabled + ASPM + ALPM
TLP + Powertop Auto tune

 

 

That consumption levels seem to be quite fine - 20W for 4x HDD and 2x SSD is quite good.

 

I do reach 15.8W with a lot of tuning an C8 on a N5105 board with the same amount of drives

 

I'd say the difference is that you use 3x RAM-Sticks and maybe C6 vs C8 - and obviously the drives themselves.

 

if you remove everything but one RAM-Stick you might be able to see lower consumption - have you also enabled GEAR2 for the memory ... that could give you a little bit more power savings too ...

[Pete]

Finally got my hands on a Gigabyte C246M-WU4 mainboard. Came with an Intel Xeon E-2126G SR3WU CPU and 96GB of Kingston ECC RAM. A bit overkill for UNRAID but nice to have for dockers and VMs. Put in 2x Kingston NVME SSD 2 GB in RAID 1 for cache and 6 Ironwolf 12 TB harddrives for the array.

 

Tweaked BIOS and edited the Go files according to the recommendations by @mgutt at the beginnning of this thread.

 

System reaches C9 (nice) but lowest power consumption with all drives idle is around 17 W. Even with all harddrives disconnected it won't go below 13 W.

 

What is even worse, that power consumption varies between 17 and 30 W occasionally spiking to 50 W.

 

 

Something must be running in the background. 

 

The top power consuming process according to powertop is tick_sched_timer which presumably changes the CPU frequency but how to find out what is triggering it?

 

Frankly after having read about idle power consumption of 8 W possible on the C246M-WU4, I've expected a bit more.

 

Different from the reference C246M-WU4 design is the XEON CPU, the 96 GB ECC RAM, the 2 Kingston NVME SSDs and the power supply. Yes it is a Corsair RM550X but the old one from 2018. But as long as I don't reach sub 10 W power consumption, the 2021 modell probably won't make much difference.

 

I'll attach the powertop HTML report to this post. Would be nice if someone could have a look and help me to find the reason, why my system does not reach lower power consumption.

 

 

powertop.html

Edited November 29, 2023 by Pete

[dopeytree]

Anyone with a HBA been able to get anywhere?

I have noticed on occasion my system dips from 110w to 88w.

Reaches c3.

This figure also includes the wifi router and a 10port small switch (powered from ups) so real figure is likely about 10-20w less.

[unr41dus3r]

Maybe someone has an idea, i got an Intel Arc 380 and ASPM in Bios is enabled and i also seewit lapci ASPM is enabled.

I try now to get them to lowest state and in Powertop i see it is in RC6 state but not going to RC6p or RC6pp.

Any idea what i could try?

I am also using custom Thor Kernel for the use of ARC, but as i know the RC6 states are normal for intel?

My C states are only from 1-3 but as i read its normal in powertop for AMD CPUs

[dopeytree]

I find quicker default to low C state (and lower wattage) with ASPM on MSI motherboard set to bios controlled. IE native control is off.

 

See if that does anything?

 

Unraids default as all this thread proves is beast mode is permanently  on.

Edited December 1, 2023 by dopeytree

[vmasip]

Hello,

 

Great opening post. Full of insights. I've been reading it a few times, and many other topics too. And I cannot find the cause my system is not achieve what you get. I am using

• n100dc-itx,
• crucial nvme
• 16 gb crucial ram 3200 cl22
• sata controller asm 1166 without any disk connected to it
• using two different external powerbrick transformers. I've got two by the way19V and 12V (I've readed here 12V is valid, but I guess I won't be able to get normal speed performance)
• ssh connection (no keyboard, gui, or hdmi at this moment)
• My power measurer hasn't arrived yet, so no measures by the moment.

 

After activating every power consumption measure at the bios (c-states,asp,)reproducing your steps:

 

 

 

Ok,looks fine but:

 

 

As yo can see I don't go beyond C3. 

ASPM:

 

 

 

CPU freq:

too much?

 

So:, my doubts:

No need to update asm 1166 firmware I guess. But ethernet controller is disabled. Maybe is not possible?
I really don't know what is this:

00:1c.6 PCI bridge: Intel Corporation Device 54be

After all, maybe I am going beyond C3, but is one of those cases powertop doesn't show rigth info?
Are laptop 75W and 45W watts powerbricks better than corsair you show? I am going to use use ssd sata disks.  So I guess I've got 15W(45w - 30w max watt motherboard) - 2W(controller) margin for them. Isn't it? How many ssd sata will I be able to connect?

 

[fightforlife]

Your power top screen looks really good! 

CPU cores are even reaching C10, which is quite a feat.

pc3 = is the package c state includibg all controller. (Memory, SATA, network. Etc..)

cc3 = is the core c state of the CPU. 

 

The core c state should be more important than the package c state, since the cores can use quite a bit of power.

The package power will not fluctuate a lot and is more difficult to get lower, because of all the different controllers involved.

 

Laptop power should be more efficient than a full ATX PSU. I am really interested in the power values you will measure.

[mgutt]

4 hours ago, vmasip said:

No need to update asm 1166 firmware I guess

As long you don't connect any drives, you won't see the but which is caused by the old firmware.

 

4 hours ago, vmasip said:

After all, maybe I am going beyond C3, but is one of those cases powertop doesn't show rigth info?

As long powertop is showing the other states, it should be able to display their usage. If it does not support anything above C3 it won't show them at all.

 

4 hours ago, vmasip said:

I really don't know what is this:

00:1c.6 PCI bridge: Intel Corporation Device 54be

Usually a CPU has one or more PCI bridge devices to provide the lanes for all the PCI devices. You could output a tree to understand it better:

https://superuser.com/questions/1768544/how-can-i-understand-the-device-topology

 

4 hours ago, vmasip said:

CPU freq:

too much?

Yes and no. If low packages states would be reached, the CPU should throttle to 800 Mhz. But the high frequency is not the reason for the hight pgk c states. Usually the reason are the other devices or the BIOS itself. 

 

Please test with the least amount of connected devices as possible. So remove the ASM1166 and maybe play around with the Ethernet/USB ports. For example I tried on a different board to install Ubuntu and removed USB and Ethernet and finally it I found out that it only reached C8 when I disconnected the Ethernet port.

[vmasip]

 What can you tell me about usb? I run unraid from usb,so I guess I can’t try disconnecting it.

 

you disconnect ethernet and see states from hdmi?

[mgutt]

20 minutes ago, vmasip said:

I run unraid from usb,so I guess I can’t try disconnecting it.

That's why it can be useful to install Ubuntu to check if this could be a reason. Even specific USB ports can influence c states.

 

21 minutes ago, vmasip said:

you disconnect ethernet and see states from hdmi?

Yupp

[vmasip]

Install Ubuntu where? at usb or nvme disk?

By now, without ethernet and with hdmi,I have reached C8 state. I don't know the difference between C8 and C10 and if I should get C8 . ASPM command output remains the same: with realtek ethernet Disabled.

I've found this, same board. Probably same issue:

 

 

Edited December 13, 2023 by vmasip
new info found

[dopeytree]

Is there a simple way to track C states over time?

[mgutt]

On 12/14/2023 at 12:30 AM, vmasip said:

Install Ubuntu where? at usb or nvme disk?

NVMe or you won't be able to exclude USB as the source of the problem 😉

[heyelly]

Hello everyone!

 

Yesterday I completed my first build and I am quite impressed, I was able to reach 8w in idle with HA running as a VM. The only issue I have is about my Sonoff Zigbee Dongle, as soon as I plug in and pass through to the HA VM, I got a 9/10w increase in power consumption. CPU sits almost at C2 states, while without that dongle I am able to reach C6/7.

 

Do you guys have any suggestions? 
 

Gigabyte B760m 

i3 13100

Crucial NVMe 1TB

SP NVMe 512GB

BEQUIET Pure Power 12 M 550w

[mgutt]

29 minutes ago, heyelly said:

Do you guys have any suggestions? 

Please try all USB ports. In addition test one of the USB onboard connectors with an adapter like this one:

https://www.startech.com/en-us/cables/usbmbadapt2

 

 

[heyelly]

9 minutes ago, mgutt said:

Please try all USB ports. In addition test one of the USB onboard connectors with an adapter like this one:

https://www.startech.com/en-us/cables/usbmbadapt2

 

 

I will try with that adapter. The dongle is currently attached to a USB 2.0 port that I have right next to the old PS2 port. I tried all USB ports I have but nothing changes.

[heyelly]

On 12/16/2023 at 8:42 AM, mgutt said:

Please try all USB ports. In addition test one of the USB onboard connectors with an adapter like this one:

https://www.startech.com/en-us/cables/usbmbadapt2

 

 

Same power consumption. Package stuck in C2. As soon as I unplug the dongle, 8-9W less and C8 reached.