Reduce power consumption with powertop

[TheLinuxGuy]

On 4/18/2023 at 10:03 AM, mgutt said:

If you stick to i3, than you could buy the i3-12100 or i3-13100 instead. Every CPU can be throttled to a T-CPU.

Mind explaining a bit more about how to throttle say a 13600K to 13600T?

I thought that Intel blocked undervolting - which I think is one of the things you would do in addition to lowering PL1 and PL2 settings in BIOS. But I think intel just blocked undervolting... 

 

[mgutt]

6 hours ago, TheLinuxGuy said:

I thought that Intel blocked undervolting

T-CPUs are underclocked, not untervolted. This is done through a different base clock multiplier and reduced PL1 and PL2. The voltage depends only on the frequency. Here is an example of an older CPU at a similar frequency:

i3-8100T http://valid.x86.fr/u1j0ym vs i3-8100 https://valid.x86.fr/tfx7nt

I mean, maybe (I still don't believe that) its even a little be less voltage, but finally this saves only 1W in idle and maybe 5% in load scenarios. So there is no real reason to pay more for a T-CPU.

 

I mean yes, at the moment both cost the same, but in the future the K will be much cheaper (for example the i7-12700T costs more than the i7-12700K) and finally you could sell the K CPU or use it in a gaming system... while the T-CPU is only useful in very special scenarios like an OEM system which does not allow to set PL1/PL2 through the BIOS.

 

6 hours ago, TheLinuxGuy said:

13600K to 13600T?

Note: These are two completely different CPUs. The T-CPU uses Golden Cove cores and the K-CPU Raptor Cove. Only the complete lineup of i7 and i9 uses Raptor Cove cores. So you could maybe decide between an i7-13700K and i7-13700T.

[TheLinuxGuy]

Thanks for sharing the knowledge @mgutt!

 

As I shop around for an intel lga 1700 motherboard, I found these 8 SATA port motherboard by Biostar but apparently nobody in USA is selling them. Maybe its of interest to some here in other markets:

 

B760MXC PRO 2.0 https://www.biostar.com.tw/app/en/mb/introduction.php?S_ID=1112

B660MXC PRO https://www.biostar.com.tw/app/en/mb/introduction.php?S_ID=1072

 

I only found B660 board on newegg but seller wants $50 shipping which is 1/4 cost of the board so not worth the risk.

[mgutt]

4 hours ago, TheLinuxGuy said:

I found these 8 SATA port motherboard by Biostar

B chipsets do not support 8 SATA ports, so these boards will use an additional controller which will raise power consumption. Which means you could finally buy any 4 SATA board and install an ASM1166 card to get 10 SATA ports and getting the same result. Ok not true. It still depends on the board. PS the Gigabyte B660M DS3H is very efficient with 8W in idle.

[Stubbs]

When writing echo med_power_with_dipm, how am I supposed to find out which disks are on which host?

I know sdf and sde are connected to an ASMedia 1062 controller so I have to skip them. I don't see anything in disk information telling me what their host number is.

[Jess3]

I have a asrock deskmini 310 with an i3 9100. i followed all the steps in the first post (including the bios settings) but it won't go in a state below C2. I do run an kingston A2000 which i read in this topic is notorious but the linked article also states that the problem was fixed with a kernel update back in 2021.  Is it lickly its still the cullprint? or are there other factors like docker containers i can check (i see 650 wapeups/second for example).

 

 

[quietwalker]

I join this thread instead of opening a new one (I don't know if this is a good idea )

Anyway I'm trying to get a less watt/h draw from the outlet but it looks like it's impossible in my case.

 

I'm using this setup:

CPU: intel core i3-12100

MoBo: GIGABYTE z690 UD DDR4

PSU: Corsair RM550x

RAM: 16GB (2x8GB) Corsair DDR4 VENGEANCE-LPX C16 3200MHz

Storage:

• 2x500GB Crucial P3 NVME
• 1x2TB SSD Crucial
• 1x2TB SSD Patriot

Without disks and in idle, the system uses 25 watt/h. With the disks attached it uses almost the same: 25 to 27 watt/h.

The C-state I'm able to reach is C-8 if I remove the SSD Patriot (I'm planning to change it with a Samsung Evo tomorrow to see if there is any difference), if not it hang on C-6. It's not too bad, right?

So why is it using so much power? From this link here I see that a user with almost the same setup as mine (he was using an i5-12gen instead of i3-12gen) - using Unraid - is able to achieve 13.5watt. How that is possible?

 

I've put the intel_pstate in passive mode, but nothing changed honestly. I've scaled the governor to PowerSave (in both normal and powersave settings, in order to always use powersave settings) but nothing changed. Now I set the Normal Scaling Governor in  OnDemand and PowerSave governor in PowerSave.

 

Regarding the ASPM status, everything looks fine to me except for the error pcilib: sysfs_read_vpd: read failed: No such device :

root@HomeServer:~# lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM )'
00:1a.0 PCI bridge: Intel Corporation Device 7ac8 (rev 11) (prog-if 00 [Normal decode])
                LnkCap: Port #25, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
00:1b.0 PCI bridge: Intel Corporation Device 7ac0 (rev 11) (prog-if 00 [Normal decode])
                LnkCap: Port #17, Speed 8GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk-
00:1b.4 PCI bridge: Intel Corporation Device 7ac4 (rev 11) (prog-if 00 [Normal decode])
                LnkCap: Port #21, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
00:1c.0 PCI bridge: Intel Corporation Alder Lake-S PCH PCI Express Root Port #1 (rev 11) (prog-if 00 [Normal decode])
                LnkCap: Port #1, Speed 8GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk-
00:1c.2 PCI bridge: Intel Corporation Device 7aba (rev 11) (prog-if 00 [Normal decode])
                LnkCap: Port #3, Speed 8GT/s, Width x1, ASPM L1, Exit Latency L1 <64us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
01:00.0 Non-Volatile memory controller: Micron/Crucial Technology P2 NVMe PCIe SSD (rev 01) (prog-if 02 [NVM Express])
                LnkCap: Port #1, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 unlimited
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
pcilib: sysfs_read_vpd: read failed: No such device
03:00.0 Non-Volatile memory controller: Micron/Crucial Technology P2 NVMe PCIe SSD (rev 01) (prog-if 02 [NVM Express])
                LnkCap: Port #1, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 unlimited
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
                LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+

 

What can I do to lower the power consumption more? Do you have some suggestion?

I'm waiting to add 2x12TB HDD so I'm really scared of how bad the power consumption can increase

 

 

Edited April 26, 2023 by quietwalker

[mgutt]

13 minutes ago, quietwalker said:

PSU: Corsair RM550x

2018 or 2021 revision?

 

 

Quote

2x500GB Crucial P3 NVME

RAID1? Permanent Read / Writes? Check your dashboard and set to transfer rate / s.

[MagickPistacho]

Hi people!

New to Unraid, been building my first NAS and following what I could with PowerTop.

My system is as follows:

• CPU: Intel i3 12100
• Board: ASUS ROG STRIX B660-G GAMING WIFI
• PSU: Corsair RM550x (2021)
• RAM: 1x16 GB Crucial CT16G56C46U5 5600 MHz (clocked at 3200 in BIOS)
• 1x Crucial P3 1TB
• 3x Toshiba N300 8TB
• And the case is a Nox Hummer Vault

As of now, I've managed to have C8 tweking BIOS settings and using --quiet --auto-tune, around 11-13W on idle, 25 while discs are spinning, and about 32-46W on load.

 

Thing is, I can't reach C10 no matter what. I've tried with no HDD and SSD connected, still C8.
Different RAM module (CT16G52C42U5, 5200 MHz, granted it's not that different), same  result.
Dissabling Wifi/Bluetooth on board, same power usage and C states.

 

I don't know what's preventing C10 and lower power consumption, I've seem others with similar systems reach 6-8W on idle or even less.

 

I think I've looked at all BIOS settings but perphaps I've misplaced something, I don't know. Maybe it's the on board LAN? Maybe the board itself?

 

Anyway, thanks for this thread and all the info!

[mgutt]

6 minutes ago, MagickPistacho said:

Thing is, I can't reach C10 no matter what. I've tried with no HDD and SSD connected, still C8.

Please test with Ubuntu + powertop --auto-tune. But don't forget to install an ssh server to connect remotely. As long the display is connected C10 should be impossible.  If you are able to reach C10 with Ubuntu, than we maybe have an ASPM problem which can only be solved with manually enabling ASPM on some PCIe devices:

https://forums.unraid.net/topic/111921-why-is-aspm-disabled-in-unraid-while-its-enabled-in-ubuntu/

 

If you can't reach C10 even in Ubuntu, this is related to the board / BIOS. And probably because your board has an ASMedia onboard controller for USB 3.2:

https://www.hardwareluxx.de/index.php/artikel/hardware/mainboards/58080-dank-bclk-oc-eine-absolute-empfehlung-asus-rog-strix-b660-g-gaming-wifi-im-test.html

 

[MagickPistacho]

Oh, I see. Thanks a lot.
 

I'll have to investigate about installing a ssh server and managing it with Ubuntu, right now I don't know where to start with that, but I'll take a look as soon as I can.

In the meantime, if it serves any purpose, I've tried this and everything checks as enabled:

[TheLinuxGuy]

7 hours ago, mgutt said:

As long the display is connected C10 should be impossible. 

This is a very good pointer... I just received the motherboard + cpu for my new build - without HDDs I am seeing 8.5 watts right now. I partially tried to replicate @riduxd setup (same motherboard)

Here's my hardware:

MB: Asus Prime Z790M-Plus D4 Version: Rev 1.xx

- BIOS version 02/22/2023 v0810

CPU: Intel 13400

CPU cooler: stock intel cooler

RAM: 16GB (2x8GB) G.Skill F4-3000C16-16GISB Aegis DDR4 DDR4-3000 CL16-18-18-38 1.35V

PSU: RGEEK 12V 300W Pico ATX

HDD: only the unraid USB (for now)

Idle: ~8.5 watt.

 

- I am not sure if I am reading powertop correctly but this seems to say that I am not hitting C10 at all, and mostly I am on C8?

 

 

@riduxd any chance you could share what options you have set in BIOS? Just in case I am missing something silly.

 

- Any recommended tutorials on how to force downclock this i5-13400 (65w) to i5-13400T (35w)?

 

Also not sure why at idle my CPU freq is this high... doesn't seem normal to me.

 

root@Tower:~# lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM )'
00:1a.0 PCI bridge: Intel Corporation Device 7a48 (rev 11) (prog-if 00 [Normal decode])
                LnkCap: Port #25, Speed 16GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk-
00:1b.0 PCI bridge: Intel Corporation Device 7a40 (rev 11) (prog-if 00 [Normal decode])
                LnkCap: Port #17, Speed 8GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk-
00:1c.0 PCI bridge: Intel Corporation Device 7a38 (rev 11) (prog-if 00 [Normal decode])
                LnkCap: Port #1, Speed 8GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk-
00:1d.0 PCI bridge: Intel Corporation Device 7a30 (rev 11) (prog-if 00 [Normal decode])
                LnkCap: Port #9, Speed 16GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk-
root@Tower:~#

 

Edited April 27, 2023 by TheLinuxGuy

[riduxd]

Well unfortunately @TheLinuxGuy, I don't have that mobo anymore. I broke it while I was reinstalling my CX4.... LOL. The mobo that I originally got to replace the Asus Prime Z790M-Plus D4, wasn't letting me go into deeper CStates, so I returned that for an Asus Tuf Z790-Plus D4, which also wasn't letting me go into deeper states. So yeah full size ATX mobo's are no go, especially for Z790. Anyways, those seem right.. I have a 13600k so I'm not sure if that's enough variance... What I do remember though is I set Multicore enhancement to disabled, and there was a setting for "Best Case Scenario" (i forget which one it was...). Anything motherboard vendor specific settings was basically turned off, and I used mostly Intel's default settings. I also capped my PL1 to 35W and PL2 at 125W, and unplugged my HDMI cable.

 

Forgot to mention that I was only able to achieve C10 when I let it sit long enough, but in reality, it was mostly at C8, similar to what you have. Oh and for ASPM settings, I used C10 as max, not Auto or CPU Default.

 

More edits: Install corefreq-cli, it should show you frequencies at faster poll rate.

Edited April 27, 2023 by riduxd

[TheLinuxGuy]

2 hours ago, riduxd said:

 

Forgot to mention that I was only able to achieve C10 when I let it sit long enough, but in reality, it was mostly at C8, similar to what you have. Oh and for ASPM settings, I used C10 as max, not Auto or CPU Default.

 

Thank you for confirming that you rarely saw the system go below C8. This is probably as good as it gets with this board, I am happy with it. My other B660M build eats 30 watts idle while this one is 1/3 - same cpu generation too, different chipset and board manufacturer (AsRock vs. Asus)

 

These should be the settings an owner of an Asus Prime Z790M-Plus D4 needs to set to match my power measurements below (unraid usb plugged in + 2TB NVME drive - hdmi + kb/mouse unplugged).

 

I'm still lost on where to set the CPU multiplier in this asus bios (but did set PL1 and PL2 to 35w) - something to look at this weekend I guess when the intel x710 arrives in the mail.

 

*edit note: apparently summary screenshots below are not showing but there are some settings in BIOS with the option "Max battery" and also noticed the scroll window hid my other APM settings where I am forcing L1 on everything plus S4+S5 and CPU deep c-states...

Edited April 27, 2023 by TheLinuxGuy

[quietwalker]

14 hours ago, mgutt said:

2018 or 2021 revision?

I'm not sure how accurate is this statement is: "[ in the Serial Number the ] First 4 are the date code. First 2 = year, next 2 = week."

If this is true, then my PSU was made in 2021, the first week (SN: 2101...)

 

14 hours ago, mgutt said:

RAID1? Permanent Read / Writes? Check your dashboard and set to transfer rate / s.

The whole system is not yet configured at all, so the NVME and SSDs are attached to the board but not configured or used in any way. I'm waiting for the 2x12TB spinning disks which I should receive later today to configure everything.

[quietwalker]

@mgutt I'm actually able to reach C10 with the 2x500GB Crucial P3 NVME and 1x2TB Samsung QVO SSD 

 

So this means also that the 2TB Crucial SSD seems to stop the server to go deep on C10 (with it inserted it hang on C8).

Anyway, in C10 the server is using 20watt which is really strange.

[mgutt]

9 hours ago, quietwalker said:

If this is true, then my PSU was made in 2021, the first week (SN: 2101...)

It is simpel: White letters on black background, then it is 2021:

 

 

9 hours ago, quietwalker said:

The whole system is not yet configured at all, so the NVME and SSDs are attached to the board but not configured or used in any way

Please add them all to the array as disks. Leave the parity slot free, so the array starts without creating a parity. Through Tools > New Config you can delete the array afterwards. Did you close the WebGUI while measuring? Even the Unraid Web Terminal must be closed. This has a huge impact on power consumption. Instead enable SSH and connect through your PCs terminal and "ssh root@tower"

[quietwalker]

29 minutes ago, mgutt said:

Please add them all to the array as disks. Leave the parity slot free, so the array starts without creating a parity.

 Done. The power misurement (without GUI) reports 19watt, which is better than before but not so significant. The strange thing I notice when I connect to the server is that the CPU frequence is not 800Mhz as I was expecting, it's always at 1.2GHz even if the Governors are set in OnDemand (Normal) and PowerSave (Power Safe).

I've tried to enable again pstates and the result doesn't change: with Governors sets as Performances and PowerSafe the power consumtion float from 19w to 22watt

[riduxd]

15 hours ago, TheLinuxGuy said:

 

Thank you for confirming that you rarely saw the system go below C8. This is probably as good as it gets with this board, I am happy with it. My other B660M build eats 30 watts idle while this one is 1/3 - same cpu generation too, different chipset and board manufacturer (AsRock vs. Asus)

 

These should be the settings an owner of an Asus Prime Z790M-Plus D4 needs to set to match my power measurements below (unraid usb plugged in + 2TB NVME drive - hdmi + kb/mouse unplugged).

 

I'm still lost on where to set the CPU multiplier in this asus bios (but did set PL1 and PL2 to 35w) - something to look at this weekend I guess when the intel x710 arrives in the mail.

 

*edit note: apparently summary screenshots below are not showing but there are some settings in BIOS with the option "Max battery" and also noticed the scroll window hid my other APM settings where I am forcing L1 on everything plus S4+S5 and CPU deep c-states...

I honestly gave up on achieving the deepest C-States.. Even if I get the X710, once I run my Home Assistant and zwavejsui container, C-States immediately drop to C2. I guess the Zooz Zwave dongle probably is the culprit, not necessarily the zwavejsui, but I do use quite a few ZWave devices so I can't just drop ZWave.. Oh well.

[MagickPistacho]

On 4/26/2023 at 10:04 PM, mgutt said:

Please test with Ubuntu + powertop --auto-tune. But don't forget to install an ssh server to connect remotely.

 

Ok, I've managed to install the ssh server and powertop in Ubuntu, and yeah, seems that I can't reach C10...

 

No HDMI and USB cables connected, only power, LAN and the USB stick. I don't know if I should've used a non-GUI mode or anything else to test this, didn't see the option while booting Ubuntu.

Even then, while achieving C8 here, it uses about 34W instead of 13 like in UnRAID, is that normal?

[mgutt]

34 minutes ago, MagickPistacho said:

I don't know if I should've used a non-GUI mode or

The desktop version is perfectly fine as long no monitor is connected.

 

34 minutes ago, MagickPistacho said:

Even then, while achieving C8 here, it uses about 34W instead of 13 like in UnRAID, is that normal?

This is the first time I read that Unraid has lower power consumption than Ubuntu. Strange. Especially because of the huge difference. 

[MagickPistacho]

43 minutes ago, mgutt said:

The desktop version is perfectly fine as long no monitor is connected.

 

Yep, then it seems that's it, my system can't go to C10, I'll have to change the motherboard and try with another one. Bummer, but at least I have some spotted thanks to hardwareluxx.

 

And many thanks for the tips and help

 

43 minutes ago, mgutt said:

This is the first time I read that Unraid has lower power consumption than Ubuntu. Strange. Especially because of the huge difference. 

 

My post was misleading, I was comparing Unraid with spun down disks to Ubuntu as is.
 

I thought Unraid used only about 25W while discs were spinning, but it seems that when all 3 are active it can go up to 35W too, so it should be about the same, sorry for the confusion.

Edited April 28, 2023 by MagickPistacho

[mgutt]

1 hour ago, MagickPistacho said:

I thought Unraid used only about 25W while discs were spinning, but it seems that when all 3 are active it can go up to 35W too

You should try it without any disk except a single SSD. As mentioned in the first post, some HDDs have an impact on C-States.

[TheLinuxGuy]

@mguttI have a curious question; is it possible to disable HDMI (or virtually unplug) via command line and achieve the same lower power consumption as if I had unplugged the actual HDMI cables from the board?

I am planning to install PIKVM  https://geekworm.com/products/pikvm-a8 which uses a loopback HDMI connection + USB and allows remote reboot and management of the PC.

 

It would be great if at boot time of unraid i could virtually disable the USB+HDMI connections to drop the consumption but if unraid crashes (and via pikvm I 'reset' the motherboard to get into bios) the video/usb is back for when troubleshooting is necessary. WDYT?

[mgutt]

19 minutes ago, TheLinuxGuy said:

is it possible to disable HDMI (or virtually unplug)

I don't think this does really reduce power consumption as a monitor is switched off after 2 minutes or so.