Reduce power consumption with powertop

[pOpYRaid]

@dopeytree thanks for your answer.

 

Ok so when i got your right, just enable XMP 
Will try it and measure power consumption again.

 

So my current config is:

• BIOS: EIST/C-States enabled, Turbo boost, Speedstep & XMP disabled
• Added /boot/config/go tweaks from OP (to set powersave cpu governor & other small tweaks)
• Idle consumption is ~11,4 Watt (All docker running, HDD's in standby)

So your suggestions are:

• BIOS: EIST/C-States enabled, Turbo boost, Speedstep & XMP enabled
• Remove tweaks from /boot/config/go
• Use AutoTweak plugin to set powersave cpu governor and other tweaks

Right?

 

 

[dopeytree]

I keep the go config tweaks just remove cpu boost if that is where you control boost on / off. (it can be done in software rather than bios).

• BIOS: EIST/C-States enabled, Turbo boost, Speedstep & XMP enabled
• Only Remove turbo tweaks from /boot/config/go (keep all other settings)
• Use AutoTweak plugin to set powersave cpu governor and other tweaks
• reboot

 

Share how you get on, does it help?

Edited September 16 by dopeytree

[pOpYRaid]

2 hours ago, dopeytree said:

I keep the go config tweaks just remove cpu boost if that is where you control boost on / off. (it can be done in software rather than bios).

• BIOS: EIST/C-States enabled, Turbo boost, Speedstep & XMP enabled
• Only Remove turbo tweaks from /boot/config/go (keep all other settings)
• Use AutoTweak plugin to set powersave cpu governor and other tweaks
• reboot

 

Share how you get on, does it help?

thx @dopeytree i will test this and post my measurement results.

One question, i have seen in the AutoTweaks screenshots that you can set ASPM forthe realtek 8125B driver.
I and others (see above) had bad experiences with this chip on current unraid 7 beta 2 build.
System is stuck C3 or C6 and doesnt reach higher C states.
Also when ASPM is shown enabled by linux.

Do you have such an NIC card and does the AutoTweak's plugin fixes this issue?
Just asking because the 8125 cards are cheaper than other intel cards.

thx

[pOpYRaid]

@dopeytree tested the settings. The idle is up to ~13 watt and peaking much more (yeah the boost is visible here).
Now gone back to disable all turbo boost settings and XMP.
Thinked about it an all worked well without and for our use case, the performance is enought.

Anayway, thx for your time.

[dopeytree]

Wonder if its because of the i3-12100T?

In a previous build I had a 11900T and the base clock was 1.1Ghz which is too slow so always boosting.

My current build is a B760M-ATX ERYING with i5-13500H with base @ 2.5 GHz Boost up to 4.7Ghz

Performance-cores - 4

Efficient-cores - 8

Total Threads - 16

 

anyway enjoy 9w - 13w seems very good. How many drives do you have?

I have 2x 16bay chassis but all spin down when not in use.

Use the server with coral card & frigate for CCTV so on 24/7 now avg around 60w from previous 120w.

 

Edited September 17 by dopeytree

[pOpYRaid]

@dopeytree
Yeah, the base clock of the i3-12100T is 2,2 Ghz @ 4x p-cores & 8x threads, which is enough for us.
I have 1x Samsung 980 nvme & 3x Seagate (2x 20TB & 1x 14 TB) hdd's.

 

I am happy with this build now regarding power consumption.
 

One thing i have to solve is the 2,5 GBE Upgrade.
With the RTL8125B i had a bad experience stuck in C3.

Maybe i'll try the i226-T1.

Do you have 2,5 GBE?
If yes, which adapter?

 

[mgutt]

The Broadcom 9500-16i HBA card supports C7:

 

[pOpYRaid]

@mgutt thx for your info. So when i assume right, that's without the intel i226-T1 card, right?
Would be so nice if you could plug it in and check c states and stability.

 

thx a lot

[mgutt]

36 minutes ago, pOpYRaid said:

Would be so nice if you could plug it in and check c states and stability

I will. I'm waiting for a bracket to re-position the power supply. I think I receive it in 2 weeks.

[SDUGoten]

root@nas:~# lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM )'
00:06.0 PCI bridge: Intel Corporation 12th Gen Core Processor PCI Express x4 Controller #0 (rev 04) (prog-if 00 [Normal decode])
                LnkCap: Port #5, Speed 16GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <16us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
00:07.0 PCI bridge: Intel Corporation Alder Lake-P Thunderbolt 4 PCI Express Root Port #0 (rev 04) (prog-if 00 [Normal decode])
                LnkCap: Port #2, Speed 2.5GT/s, Width x4, ASPM L1, Exit Latency L1 <16us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk-
00:07.2 PCI bridge: Intel Corporation Alder Lake-P Thunderbolt 4 PCI Express Root Port #2 (rev 04) (prog-if 00 [Normal decode])
                LnkCap: Port #4, Speed 2.5GT/s, Width x4, ASPM L1, Exit Latency L1 <16us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk-
00:1c.0 PCI bridge: Intel Corporation Device 51ba (rev 01) (prog-if 00 [Normal decode])
                LnkCap: Port #3, Speed 8GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <16us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
00:1c.6 PCI bridge: Intel Corporation Device 51be (rev 01) (prog-if 00 [Normal decode])
                LnkCap: Port #7, Speed 8GT/s, Width x2, ASPM L0s L1, Exit Latency L0s <1us, L1 <16us
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
00:1d.0 PCI bridge: Intel Corporation Alder Lake PCI Express Root Port #9 (rev 01) (prog-if 00 [Normal decode])
                LnkCap: Port #9, Speed 8GT/s, Width x2, ASPM L0s L1, Exit Latency L0s <1us, L1 <16us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
01:00.0 Non-Volatile memory controller: Seagate Technology PLC FireCuda 530 SSD (rev 01) (prog-if 02 [NVM Express])
                LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
56:00.0 Non-Volatile memory controller: Phison Electronics Corporation PS5013-E13 PCIe3 NVMe Controller (DRAM-less) (rev 01) (prog-if 02 [NVM Express])
                LnkCap: Port #1, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 unlimited
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+
57:00.0 Ethernet controller: Aquantia Corp. AQtion AQC113 NBase-T/IEEE 802.3an Ethernet Controller [Antigua 10G] (rev 03)
                LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM not supported
                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+
58:00.0 SATA controller: ASMedia Technology Inc. ASM1164 Serial ATA AHCI Controller (rev 02) (prog-if 01 [AHCI 1.0])
                LnkCap: Port #0, Speed 8GT/s, Width x2, ASPM L0s L1, Exit Latency L0s <4us, L1 <64us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+

 

 

I wonder if anyone how to enable these two?  This is uGreen DXP6800 Pro.  Thanks.

[spyrosj]

Does the recommendation to disable turbo still stand for newer Intel processors? Since Alder Lake (12th gen) there's been PL1 and PL2 where you can set the upper limits to CPU power draw.

[mgutt]

On 9/17/2024 at 3:32 PM, pOpYRaid said:

Would be so nice if you could plug it in and check c states and stability.

As I had problems to get my new WD Black 8TB running in the C246N-WU2, I disassembled everything and made several new power measurements and c-state tests.

 

7.15 watts and C10 without any additional cards:

 

8.44 watts and C7 with Intel I226-LM card in PCIE X16 slot and 2.5G connection ("LM" means "long support from Intel"):

 

8.01 watts and C7 with Intel I226-LM card in PCIE X16 slot and 1G connection:

 

As mentioned the WD Black NVMe didn't like to run, so I tried to install the 2.5G with an M.2 to PCIE X4 riser in the M.2 slot as follows, but although the server booted, the card was not recognized by Unraid (not visible through lspci):

 

But I had a second 2.5G X1 card from IOCrest with Intel I225 controller and surprisingly this one worked:

 

8.36 watts and C9:

 

And a last offtopic test was the WD Black 8TB inside a C-Payne X8X4X4 Bifurcation Riser:

 

measurement was without the HBA card:

 

Reached 7.86 watts at C7:

 

So the Bifurcation Riser has only a slight overhead of 0.7 watts. Note: Of course such Risers do not work with every board. For example recent Intel boards only support X8X8 and there must be a setting in the BIOS to enable it. The C246N-WU2 allows X8X8 and X8X4X4. An AMD board often allows X4X4X4X4.

 

Now back to the 2.5G cards: I think every 2.5G card with Intel controller should reach good C-states. I recommend a card with I226 because regarding the Intel Specs it is the only controller which supports 802.3az (EEE):

 

Sadly I was not able to enable it. It seems my QNAP QSW-2104-2T supports only EEE with 1G connections, which makes me mad as this restriction is not mentioned anywhere.

 

 

[mgutt]

16 hours ago, spyrosj said:

Does the recommendation to disable turbo still stand for newer Intel processors?

Nothing has changed about that. I would even think about throttling an i5/i7/i9 to 125 watts as they are wasting a huge amount of energy in higher clocks for a tiny amount of (gaming) power. 

 

Things change only if you use low power models like the N100 or already throttled T-CPUs (don't buy them!).

 

 

 

 

 

 

 

[pOpYRaid]

@mgutt thx for your tests regarding the I226. Looks promising. I have now ordered a I226-T1 card and will post my results.
 

@spyrosj i have an i3-12100T and disabled turbo boost/speedstep & XMP. With those enabled it took about ~2 Watt more idle on no load but spiked much more often (because of frequency spikes (turbo boost)). The performance for my use case (media server & NAS) is ok. Because it's idling most of the time of a day, i have decided to get the lowest idle consumption and not performance.

But sure, it's up to your workload and usage of the sever.

 

Ps.: See above here for reference: 

 

Edited September 23 by pOpYRaid

[dopeytree]

9w seems excessive for 2.5GB. 

Do you think it is worth it?

 

How does it compared to running a primary Gigabit Lan with 2nd 10GB DAC or fibre link?

[pOpYRaid]

ok, so here is my update about the genuine Intel I226-T1 (Intel number: I226T1BLK).

 

Tests done with:

• Mainboard: Gigbyte H610i
• RAM: 2X16GB
• PSU: picoPSU-XT-160
• CPU: 12th Gen Intel® Core™ i3-12100T @ 2178 MHz
• Cache: 1x Samsung 980 NVMe M.2 SSD, 1 TB
• Array: Seagate 1x 14TB & 2x 20TB

Bios Settings:

• Enabled all BIOS settings regarding power save, ASPM, ... some of which
• PCIe ASPM L0sL1 and just L0s
• Intel C-State: Enabled
• C1E Support: Enabled
• Package State Limit: C10
• Disabled HD Audio Controller/COM/LPT

Unraid state:

• booted
• all Dockers & VM stopped
• HDD's spun down
• EEE enabled in go boot config file
• No device is accessing unraid, so it's idle

The results are:

• With onboard I219-V enabled with 1G link, no PCIe card:
  • Power consumption: ~12,4 Watt
  • deepest c state: C8
     
• With onboard I219-V disabled and I226-T1 in PCIe slot with 2,5GBE Link:
  • Power consumption: ~12,1 Watt
  • deepest c state: C6

It's basically the same power consumption (but i have not meassured the average!)
It seems the I226-T1 is preventing the cpu to reach C8.
Seems like the same @mgutt above posted, but he reached C7.
Maybe the power consumption is not higher because the I226-T1 is the newest chipset with EEE support.

 

So the result's are ok for me.

 

UPDATE: NIC showed an 2,5GBE link an hour later, but was NOT reachable!
So it's not stable for me :-( 
P.S.: Same Switch & cable where the realtek 2,5GBE was stable.
So i will return it and activate the onboard 1 GBE NIC again.


 

Edited September 25 by pOpYRaid

[pOpYRaid]

@mgutt did you found a way to control the leds of the I225/I226?

They are to bright on my card.

It seems there was an patch in the kernel that they get exposed in /sys/class/leds but they are not here.
Maybe because our kernel is too old because it's the LTS one.

 

 

Here is the patch: https://lore.kernel.org/all/[email protected]/

 

Not needed anymore: See above!

Edited September 25 by pOpYRaid

[Snubbers]

Awesome thread!

I'm on a bit of  journey to get a reasonably balance of power for 24/7 use but also efficiency since most of the time its ticking over with 30+ docker containers doing all manner of stuff.

 

My question really is that for testing, I am just using user scripts to run the various commands in the OP manually.. however, I want a script that will effectively revert everything back to default should I see any issues testing, and then I can selectively turn things back on again..

Does anyone know what commands I'd need to revert each item from?

These are the ones I'm currently trialling:

# -------------------------------------------------
# Enable power-efficient ethernet
# -------------------------------------------------
# enable IEEE 802.3az (Energy Efficient Ethernet): Could be incompatible to LACP bonds!
for i in /sys/class/net/eth?; do dev=$(basename $i); [[ $(echo $(ethtool --show-eee $dev 2> /dev/null) | grep -c "Supported EEE link modes: 1") -eq 1 ]] && ethtool --set-eee $dev eee on; done


# -------------------------------------------------
# powertop tweaks
# -------------------------------------------------
# Enable SATA link power management
echo med_power_with_dipm | tee /sys/class/scsi_host/host*/link_power_management_policy

# Runtime PM for I2C Adapter (i915 gmbus dpb)
echo auto | tee /sys/bus/i2c/devices/i2c-*/device/power/control

# Autosuspend for USB device
echo auto | tee /sys/bus/usb/devices/*/power/control

# Runtime PM for disk
echo auto | tee /sys/block/sd*/device/power/control

# Runtime PM for PCI devices
echo auto | tee /sys/bus/pci/devices/????:??:??.?/power/control

# Runtime PM for ATA devices
echo auto | tee /sys/bus/pci/devices/????:??:??.?/ata*/power/control

 

[pOpYRaid]

10 minutes ago, Snubbers said:

Awesome thread!

I'm on a bit of  journey to get a reasonably balance of power for 24/7 use but also efficiency since most of the time its ticking over with 30+ docker containers doing all manner of stuff.

 

My question really is that for testing, I am just using user scripts to run the various commands in the OP manually.. however, I want a script that will effectively revert everything back to default should I see any issues testing, and then I can selectively turn things back on again..

Does anyone know what commands I'd need to revert each item from?

These are the ones I'm currently trialling:

# -------------------------------------------------
# Enable power-efficient ethernet
# -------------------------------------------------
# enable IEEE 802.3az (Energy Efficient Ethernet): Could be incompatible to LACP bonds!
for i in /sys/class/net/eth?; do dev=$(basename $i); [[ $(echo $(ethtool --show-eee $dev 2> /dev/null) | grep -c "Supported EEE link modes: 1") -eq 1 ]] && ethtool --set-eee $dev eee on; done


# -------------------------------------------------
# powertop tweaks
# -------------------------------------------------
# Enable SATA link power management
echo med_power_with_dipm | tee /sys/class/scsi_host/host*/link_power_management_policy

# Runtime PM for I2C Adapter (i915 gmbus dpb)
echo auto | tee /sys/bus/i2c/devices/i2c-*/device/power/control

# Autosuspend for USB device
echo auto | tee /sys/bus/usb/devices/*/power/control

# Runtime PM for disk
echo auto | tee /sys/block/sd*/device/power/control

# Runtime PM for PCI devices
echo auto | tee /sys/bus/pci/devices/????:??:??.?/power/control

# Runtime PM for ATA devices
echo auto | tee /sys/bus/pci/devices/????:??:??.?/ata*/power/control

 

Just do the oposite.
Remove the script, open an terminal and input every command you want to manually test.

[Snubbers]

51 minutes ago, pOpYRaid said:

Just do the oposite.
Remove the script, open an terminal and input every command you want to manually test.

I've probably not been very clear!

 

I want an 'undo' script that will reset/remove all the above for testing purposes.... i.e. if I find one of the commands is problematic, I'll undo everything via this 'undo' script..

What I'm struggling with (for example) is what is the 'undo' of:

"med_power_with_dipm | tee /sys/class/scsi_host/host*/link_power_management_policy"

etc..

 

I've only done a 'do' script to apply the commands so I don't forget what I applied.. it's easy to try 2 or 3 commands and a day later forget which you applied.. having a 'do' script and an 'undo' script allows me to remember and have a method of resetting without rebooting the server.

Hopefully that makes more sense!

[mgutt]

3 hours ago, Snubbers said:

if I find one of the commands is problematic

Then you can be sure you need to restart the server. This is kind of one-way if you face problems. The alternative is to log all settings with cat /path/to/file. Not impossible, but I don't think the benefit is huge compared to the work.

[TheLinuxGuy]

Hi everyone, I am doing a new build project and I need recommendations for low power HBA (SFF-8643 or SFF-8087) that supports 12 disks or more for this build. Which ones can you recommend? This case will be external.

 

thanks in advance!

[dopeytree]

On 9/17/2024 at 2:05 PM, mgutt said:

The Broadcom 9500-16i HBA card supports C7:

 

 

@TheLinuxGuyThere isn't one really but there are newer HBA cards like above that reach C7. You would need cable adapters and its expensive card to start with so...

 

P.S - I'm running 2x of those boxes. Awesome kit from aliexpress. Average 60-90w with drives spun down.

Edited Thursday at 05:30 PM by dopeytree

[TheLinuxGuy]

4 hours ago, dopeytree said:

 

@TheLinuxGuyThere isn't one really but there are newer HBA cards like above that reach C7. You would need cable adapters and its expensive card to start with so...

 

P.S - I'm running 2x of those boxes. Awesome kit from aliexpress. Average 60-90w with drives spun down.

 Thanks for sharing that you are running this setup successfully yourself. I can't wait for this to arrive will take 4-8 weeks but patience is a virtue...

 

Looking at the Broadcom 9500-8i and 9500-16i - is there a reason for me to pay double for a 16i? Because it looks like this enclosure from aliexpress is actually a SAS expander and I should be able to daisy chain multiple of them... so I am thinking maybe I am OK with a 9500-8i card... ebay has them for $70

[mgutt]

22 minutes ago, TheLinuxGuy said:

SAS expander

SAS Expanders, failover power supplies, SAS Backplanes... I would not wonder if such a storage unit consumes 50 watts alone.