Reduce power consumption with powertop

[cpu]

@mguttdo you know by any chance if PCIe NVME adapter can cause lack of >c2 state? Those PCIe cards can differ from each other as I have 2 x 970 Evo and I don't want to ditch mirror for my cache pool yet my system had no problems in past to go as low as C8 -> i5-7500 with Fujitsu D3401 motherboard.

[mgutt]

2 hours ago, cpu said:

do you know by any chance if PCIe NVME adapter can cause lack of >c2 state?

It can have a bad impact on C-States as more active PCIe lanes, means a more active CPU. But I had only one motherboard which fell back to C2. All other setups reached C7 or better. So finally it's as often the same answer: It depends 🤪

[yupop]

Hi there,

 

I’m another one of those poor souls who can’t get their pkg c-states any higher than C3. Here’s my setup:

 

Motherboard: Asrock B660M-ITX/ac

CPU: i3-12100

NVME: Samsung 970 EVO 500Gb

RAM: 1 x Kingston Value RAM 16Gb 3200

PSU: Silverstone Strider 550W 80+ Gold ST55F-G

 

I do also have a 6TB Ironwolf HDD as well, but at the moment I have it unplugged. Also unplugged are my front panel USBs, front audio ports and a case fan. I’m currently running Ubuntu 22.04 LTS Server with kernel 6.2.0-31-generic. This is a completely fresh install, so far I have only setup ssh, ufw, powertop, and turbostat.

 

I believe I have setup my BIOS correctly, it is also updated to latest release. C-States and APSM are all enabled, along with SATA Aggressive Link Power. Intel turbo boost has been disabled, along with Audio, RGB LEDS, Intel Ethernet I219-V. WAN device is still enabled as this is what I am using at the moment to connect to my network but I have tried with wifi disabled and LAN enabled and there was no difference. I even tried with both disabled, again no difference.

 

When running powertop I can get no better than C3. When I run powertop with auto-tune there is no difference. My governor is set to powersave.

 

I’ve been measuring my power use at the wall and I am getting 19watts without the HDD and 27 with.

 

Here is a screenshot of my powertop

 

 

Here is a screen shot of my APSM status

 

 

Here are my BIOS photos

 

https://ibb.co/4sfx1Kz

https://ibb.co/hHB80s1

https://ibb.co/vBrzBK5

https://ibb.co/mDsrr2N

https://ibb.co/nfXb52J

https://ibb.co/sJ2c2B4

https://ibb.co/XDQHWTj

https://ibb.co/JBtk785

https://ibb.co/44dwkPd

https://ibb.co/nbq8sLb

https://ibb.co/022Yg3c

 

Questions:

 

• Does there appear to be anything wrong with my setup?
• My PSU is old, could it be preventing the higher C-States?
• Is perhaps powertop not registering the higher C-States?
• If powertop is not registering the C-States correctly and I am actually at say C6 what would explain the high power use. The old PSU? Or the mobo is just crap?
• If the PSU is a problem what PSUs are currently recommended. The Corsair RM550x (2021) seems to be no longer available.
• If I’m just unlucky and have a crap mobo what Itx 12th gen boards are recommended

 

Thanks, and appreciate the help.

[mgutt]

3 hours ago, yupop said:

If I’m just unlucky and have a crap mobo what Itx 12th gen boards are recommended

I say that's the case. Those things happen often with LGA1700 Setups:

- ASRock Boards reach bad C-States

- using the CPU M.2 Slot causes bad C-States

- using the CPU PCIe Slot causes bad C-States

 

3 hours ago, yupop said:

If the PSU is a problem

Your PSU will have a bad efficiency at low loads, but does not influence C-States.

 

 

[yupop]

56 minutes ago, mgutt said:

I say that's the case. Those things happen often with LGA1700 Setups:

- ASRock Boards reach bad C-States

- using the CPU M.2 Slot causes bad C-States

- using the CPU PCIe Slot causes bad C-States

 

Your PSU will have a bad efficiency at low loads, but does not influence C-States.

 

 

Ok Thanks, not what I wanted to hear but was prepared for it. I'm going to try running it with a live usb without the m2 drive connected and see what happens.

 

Do you know of any board, preferably ITX, that would work (regarding cstates) with the i3-12100? and what brand motherboard is usally the best?

Edited September 1, 2023 by yupop

[mgutt]

Check the last 5 pages. Some people posted their results. 

[mihcox]

Trying to run the script on my box results in the below, any suggestions:

powertop --quiet --auto-tune
powertop: /lib64/libncursesw.so.6: no version information available (required by powertop)
powertop: /lib64/libtinfo.so.6: no version information available (required by powertop)
Cannot load from file /var/cache/powertop/saved_results.powertop
Cannot load from file /var/cache/powertop/saved_parameters.powertop
File will be loaded after taking minimum number of measurement(s) with battery only
Cannot load from file /var/cache/powertop/saved_parameters.powertop
File will be loaded after taking minimum number of measurement(s) with battery only

[yupop]

Tried the live usb just now with the nvme drive removed. No change. Asrock B660M-ITX/ac is definitely a no go for a low power system. I've contacted Asrock support and if they come back with anything useful I'll post here.

[mgutt]

6 hours ago, yupop said:

I've contacted Asrock support

I bet they will answer you with the hint to use the aspm off Kernel option (two other users already quoted this). Their support is incompetent. 😒

[denishay]

20 hours ago, mihcox said:

Trying to run the script on my box results in the below, any suggestions:

powertop --quiet --auto-tune
powertop: /lib64/libncursesw.so.6: no version information available (required by powertop)
powertop: /lib64/libtinfo.so.6: no version information available (required by powertop)
Cannot load from file /var/cache/powertop/saved_results.powertop
Cannot load from file /var/cache/powertop/saved_parameters.powertop
File will be loaded after taking minimum number of measurement(s) with battery only
Cannot load from file /var/cache/powertop/saved_parameters.powertop
File will be loaded after taking minimum number of measurement(s) with battery only

exact same issue for me

[yupop]

On 9/2/2023 at 6:34 PM, mgutt said:

I bet they will answer you with the hint to use the aspm off Kernel option (two other users already quoted this). Their support is incompetent. 😒

Not exactly.

 

Firstly they said that I should turn my C-State package on in the BIOS (even though I already told them I did). I then emailed all the screenshots I posted here to them. They emailed again and said that the C-States work on exactly the same setup (except on windows) for them and they posted a screenshot to prove so.

 

When I looked at the screenshot however I could see they were only getting c2 and c3. I point this out to them and then got back the following reply

 

Quote

Regarding your question, to get to C6, C8 and C10 should enable modern standby.

The motherboard does not support modern standby.

 

I checked out modern standby and according to this website it is indeed required

 

Quote

Modern Standby is a platform state. On display time out the OS requests the processor to enter package C10 and platform devices at RTD3 (or disabled) in order to attain low power in idle. Modern Standby requires proper BIOS and OS configuration.

 

So, I think this time they were helpful and I would advise anyone looking for a low power board to make sure that it is modern standby compatible.

[mgutt]

5 hours ago, yupop said:

I checked out modern standby and according to this website it is indeed required

I'm sorry to disappoint you, but ASRock is talking bullshit and you understood the website wrong. C10 is C10 and Modern Standby is Modern Standby. Modern Standby is an additional Standby State, beside S1, S2 and S3. Read for example here:

https://www.dell.com/support/kbdoc/en-us/000177661/what-is-modern-standby-and-how-does-it-differ-from-s3-standby

 

And yes, if the PC is in Modern Standby, the CPU reaches C10, but it's not a requirement of C10, to get your PC into Modern Standby first. If this would be the case, it wouldn't be possible to reach C10 while the server is running. And suprisingly there are many people reaching C10 with unRAID.

 

And of course it has absolutely nothing to do with other C-States of the CPU, too. 

 

Maybe you should ask how to reach C8 and if this state is visible as a setting in the BIOS, ask why it is offered, although it's not working.

[yupop]

12 hours ago, mgutt said:

Maybe you should ask how to reach C8 and if this state is visible as a setting in the BIOS, ask why it is offered, although it's not working.

Well damn, they are being very misleading.

 

I've emailed them back. It will be interesting to see what they come up with next.

 

To be fair to them, the BIOS does not give any visable indication that C6, C8, or C10 can be achieved. The option is simply 'enable' or 'disable' pkg C-States. But it would just be nice to know what the issue is exactly so I can avoid it in the future. Maybe next time check the manual out ahead of time and if the BIOS doesn't give specific C8/C10 options, don't buy it.

[mgutt]

1 hour ago, yupop said:

To be fair to them, the BIOS does not give any visable indication that C6, C8, or C10 can be achieved. The option is simply 'enable' or 'disable' pkg C-States.

That's typically ASRock. On some Asrock boards you can only choose C6. It seems ASRock made an internal decision to limit C-States on all their boards to reduce driver issues / support requests. Because some bad hardware drivers work in for example C3, but crash on C8 and maybe the customers than ask ASRock for support although it's because of a bad driver of a PCIe card of a different manufacturer. Finally it means for us not to buy any ASRock products.

[bally12345]

On 9/1/2023 at 4:12 PM, mihcox said:

Trying to run the script on my box results in the below, any suggestions:

powertop --quiet --auto-tune
powertop: /lib64/libncursesw.so.6: no version information available (required by powertop)
powertop: /lib64/libtinfo.so.6: no version information available (required by powertop)
Cannot load from file /var/cache/powertop/saved_results.powertop
Cannot load from file /var/cache/powertop/saved_parameters.powertop
File will be loaded after taking minimum number of measurement(s) with battery only
Cannot load from file /var/cache/powertop/saved_parameters.powertop
File will be loaded after taking minimum number of measurement(s) with battery only

 

On 9/2/2023 at 12:31 PM, denishay said:

exact same issue for me

 

I am having same issue, tried a few bios settings but still no success. Motherboard is MSI B760

[havocann]

On 9/1/2023 at 5:12 PM, mihcox said:

Trying to run the script on my box results in the below, any suggestions:

powertop --quiet --auto-tune
powertop: /lib64/libncursesw.so.6: no version information available (required by powertop)
powertop: /lib64/libtinfo.so.6: no version information available (required by powertop)
Cannot load from file /var/cache/powertop/saved_results.powertop
Cannot load from file /var/cache/powertop/saved_parameters.powertop
File will be loaded after taking minimum number of measurement(s) with battery only
Cannot load from file /var/cache/powertop/saved_parameters.powertop
File will be loaded after taking minimum number of measurement(s) with battery only

Same here, trying to run on new Gigabyte B760 DS3H DDR4 motherboard with Intel i3-13100.

Any idea's?

[yupop]

4 hours ago, mgutt said:

That's typically ASRock. On some Asrock boards you can only choose C6. It seems ASRock made an internal decision to limit C-States on all their boards to reduce driver issues / support requests. Because some bad hardware drivers work in for example C3, but crash on C8 and maybe the customers than ask ASRock for support although it's because of a bad driver of a PCIe card of a different manufacturer. Finally it means for us not to buy any ASRock products.

 

That's the kind of thing their support should be telling me. I don't mind if that is this case it would just be nice to know so I don't waste my time with it further.

 

I'll give Asrock a miss next time, but will also be checking for that sort of thing no matter the brand.

 

Thanks for the explanation.

[mihcox]

Bumping for multiple people with my issue:
 

On 9/6/2023 at 5:24 AM, havocann said:

Same here, trying to run on new Gigabyte B760 DS3H DDR4 motherboard with Intel i3-13100.

Any idea's?

 

On 9/6/2023 at 3:42 AM, bally12345 said:

 

 

I am having same issue, tried a few bios settings but still no success. Motherboard is MSI B760

 

On 9/2/2023 at 6:31 AM, denishay said:

exact same issue for me

 

On 9/1/2023 at 10:12 AM, mihcox said:

Trying to run the script on my box results in the below, any suggestions:

powertop --quiet --auto-tune
powertop: /lib64/libncursesw.so.6: no version information available (required by powertop)
powertop: /lib64/libtinfo.so.6: no version information available (required by powertop)
Cannot load from file /var/cache/powertop/saved_results.powertop
Cannot load from file /var/cache/powertop/saved_parameters.powertop
File will be loaded after taking minimum number of measurement(s) with battery only
Cannot load from file /var/cache/powertop/saved_parameters.powertop
File will be loaded after taking minimum number of measurement(s) with battery only

 

[mgutt]

54 minutes ago, mihcox said:

with my issue

You don't have any issue. These are only warnings without impact.

[Daniel15]

Another thing you should add to the list is configuring Intel HWP to prefer lower power consumption over performance:

for x in /sys/devices/system/cpu/cpu*/cpufreq/; do echo balance_power > $x/energy_performance_preference; done

Here's the documentation on intel_pstate for Linux 6.1: https://www.kernel.org/doc/html/v6.1/admin-guide/pm/intel_pstate.html

Edited September 7, 2023 by Daniel15

[yupop]

On 9/6/2023 at 10:29 PM, yupop said:

 

That's the kind of thing their support should be telling me. I don't mind if that is this case it would just be nice to know so I don't waste my time with it further.

 

I'll give Asrock a miss next time, but will also be checking for that sort of thing no matter the brand.

 

Thanks for the explanation.

 

Finally an honest reply from Asrock

 

Quote

Regarding your question, we have confirmed with R&D department again that the motherboard does not support C6, C8 and C10.

Sorry for inconvenience.

 

[justGrinding]

hi guys, i'm trying to enable the C states in my bios but idk what i'm missing here. 

 

ASPM is enable, C states enable to C10 did i forgot somehting? or i'm just dumb? 

 

motherboard B760m gaming X DDR4 from gigabyte

cpu I3 13100

1 m2 and 1 HDD for now

35/40 W at idle  i was at 13W with my I3 9100 in dex optiplex

 

 

 

 

 

[mgutt]

1 hour ago, justGrinding said:

ASPM is enable, C states enable to C10 did i forgot somehting? or i'm just dumb? 

 

You have to select the Package C-State Limit. "Auto" and "Default" means in many BIOS "Disabled".

 

1 hour ago, justGrinding said:

1 m2 and 1 HDD for now

Which models exactly? Are you using the CPU M.2 Slot or Chipset? Chipset could be more efficient.

 

Were does the load come from?

 

Did you install the GPU driver (Intel GPU Top plugin)?

 

Did you close the Unraid Web GUI and connect through SSH while checking powertop?

 

 

[frodr]

Sorry guys, this is a long one.

 

Thanks @mgutt for this great tread. I´ve been thru all 19 pages of posts to get a understanding of the topic. I started with a AMD gamer PC which I almost never used. It ran at 160-180W without doing much. So, I terminated the game PC, put the Nvidia GPU in the Server1 and sat up the a gamer VM there.

 

Then the plan was to set up a (somewhat) power efficiant Server 2 to run 24/7 for Plex, RoonServer, plus a few other dockers and possible som Forex trading Windows VM that have to be up 24/7. I bought a LGA1700 mobo and I7 processor with iGPU for transcoding. I sat up a zfs pool of 5 hdd included 4 NVMe ssd for spesial cache. I acquired Highpoint 1508 for 8 NVMe ssd due to LGA1700 not supporting pcie X4/X4/X4/X4 bifurcation. The 1508 has switching on the card. The power usage came down, to around 100W if I remeber correct. But the rig was still to beefy. I then removed the water cooling, a 20W+ pump and 10 x 120 mm fans. The strange thing, this was only 10W difference. My messuments was probably not that prisise. I have had (and still have) major problems with the mobo. Sometimes it reset the bios and all the settings.

 

Then I moved from hhd´s to 8TB sata ssd, pulled the 4 NVMe ssd cache drives and the 1508 hba. Removed the X540-T2 NIC and today the ConnectX3 40gbe NIC. The ConnectX3 draws 10W on idle.

 

So, now power draw is about 30W idle with Plex and and RoonServer running (no streaming). I want to add a highspeed nic, maybe an Intel Intel XXV710. That will pull some.

 

But the big question, can I reduse power draw futher from 30W on current HW?

 

• I am running powertop --auto-tune.
• Powertop tunables all Good.
• Am I correct that Powertop do not support newest CPU´s?
• Idle stats Pkg only C2 and CPU(OS) only C3.
• ASPM in bios enabled, spesific ASPM mostly set to value other than Auto.
• Turbo and Asus Performance disabled in bios.
• Asus tweaker (bios) can not be disabled, only 3 different setting.
• Measure power draw with TPLink Smart Plug.
• No pcie card to remove - only X1 IPMI card that was part of the mobo.
• ASPM status show all L1 enabled except ASPEED AST1150 and a Intel device, see ASPM report.
• For detailed hw listing, see server 2 in the signature.
• Under "Commands" I ran "enable IEEE 802.3az" and "Enable SATA link power management".

 

As usual, happy for any feedback.

 

//

 

 

 

 

 

 

 

 

[mgutt]

7 minutes ago, frodr said:

• Am I correct that Powertop do not support newest CPU´s?

 

Yes. The version you have installed seems to detect the package c-states but has problem with the cpu c-states:

 

 

Is this a problem? No. Powertop is only a monitoring tool or sets some power state functions of the hardware. So if it displays "C3_ACPI" it means something between C3 and C10, but they still work.

 

Did you install the Intel GPU top plugin to install the Intel iGPU drivers? Package C-States won't work on some hardware as long the iGPU driver is not installed and the iGPU is not in RC6 state. Scroll down the "Idle stats" page. At the bottom you find the iGPU status.

 

11 minutes ago, frodr said:

• Idle stats Pkg only C2 and CPU(OS) only C3.
• ASPM in bios enabled, spesific ASPM mostly set to value other than Auto.

I'd say the BIOS settings look good. I think the problem is the board itself. I suggest:

- remove all disks

- boot Ubuntu through USB flash drive

- execute powertop --auto-tune

- execute powertop

- confront Asus with the results