Reduce power consumption with powertop


Recommended Posts

Ok, I almost finished building my new server and made some tests :

  • Intel i3-1300
  • ASRock Z690 Pro RS
  • 2*16GB Corsair 
  • 7HDD + 1 SSD
  • PSU Corsair RM750X (pre-2021 version unfortunately)

I'm not sure I was able to configure in the BIOS my 13100 to clock as a 13100T would as the options are not as clear I would like.

 

No PCIE card or anything else.

 

My Shelly Plug is telling me it draw around 32 to 33W at idle with all the disk spinned down.

I'm not sure how much I would gain by changing the PSU.

And I still need to add some fan to keep all this at cool temp so I suppose it will add up some W.

 

I'm not seeing any cstate in powertop but I suppose the CPU is too new.

 

I tested the default intel_pstate and Intel_pstate passive without any real difference.

 

 

 

Link to comment
3 hours ago, mgutt said:

You should try it without any disk except a single SSD. As mentioned in the first post, some HDDs have an impact on C-States.

 

Yeah, I already tried my system without HDDs before and the results were the same on Unraid.

But I've tried it again on Ubuntu following your suggestion.
Same case, C8 max, but now this one's weird: I get 20W on Ubuntu without HDDs and with Powertop auto-tune while I get those 12-13W on Unraid with discs connected.

 

 

I've already ordered a different mobo (Gigabyte B660M DS3H DDR4, another user managed to get around 5W with it), I'll try to reach C10/lower consumption with that one, we'll see.

But the above was for sure not the result I expected! 🤣 

 

 

Edited by MagickPistacho
Link to comment
55 minutes ago, mgutt said:

It seems no 600/700 chipset mobo reaches C9/C10. In the meantime I got feedback for multiple B660 and B760 and all of them stuck in C8. Some of them even reach only C3 or C6 🤨

I think only ITX and mATX boards only though. I've tried 3 Z790 mobos, and only the Asus Prime Z790M-P D4 was able to achieve C10. Both my Asrock and Tuf ATX motherboards were only able to achieve C6 max.

Link to comment
12 hours ago, riduxd said:

I think only ITX and mATX boards only though. I've tried 3 Z790 mobos, and only the Asus Prime Z790M-P D4 was able to achieve C10. Both my Asrock and Tuf ATX motherboards were only able to achieve C6 max.

 

I have achieved C10 (~5 watt) on the Asus Prime Z790M-Plus D4... but there's a catch. I got there by installing Proxmox + kernel 6.xx + powertop 2.15 (from source) + scaling governor set to powersave.

 

image.thumb.png.32be685dcc7fc8645e28c326aa50d3a9.png

 

root@dom:/# uname -mrs
Linux 6.0.19-edge x86_64
root@dom:/# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

 

I wonder if unraid's kernel was newer if we would be able to get to C10 natively as I am seeing its possible here... 

Also I confirm that Intel X710-DA2 C10 is also achieved under Debian 11/Proxmox kernel 6.0 - the power consumption with 10GB x710 is ~8.7 watt idle.

 

*edit... decided to test rc4 after reading it uses kernel 6.1* results are very close (proxmox is more efficient by 3 watt for some reason - I think C10 is stable in proxmox since unraid is 40% at C10 20% at C8 and random stuff in other states) - 12 watt idle with X710-DA2 connected on unraid 6.12-rc4

 

root@Tower:~# cat /etc/unraid-version
version="6.12.0-rc4.1"
root@Tower:~# uname -mrs
Linux 6.1.26-Unraid x86_64

Edited by TheLinuxGuy
Link to comment
11 hours ago, TheLinuxGuy said:

 

I have achieved C10 (~5 watt) on the Asus Prime Z790M-Plus D4... but there's a catch. I got there by installing Proxmox + kernel 6.xx + powertop 2.15 (from source) + scaling governor set to powersave.

 

image.thumb.png.32be685dcc7fc8645e28c326aa50d3a9.png

 

root@dom:/# uname -mrs
Linux 6.0.19-edge x86_64
root@dom:/# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

 

I wonder if unraid's kernel was newer if we would be able to get to C10 natively as I am seeing its possible here... 

Also I confirm that Intel X710-DA2 C10 is also achieved under Debian 11/Proxmox kernel 6.0 - the power consumption with 10GB x710 is ~8.7 watt idle.

 

*edit... decided to test rc4 after reading it uses kernel 6.1* results are very close (proxmox is more efficient by 3 watt for some reason - I think C10 is stable in proxmox since unraid is 40% at C10 20% at C8 and random stuff in other states) - 12 watt idle with X710-DA2 connected on unraid 6.12-rc4

 

root@Tower:~# cat /etc/unraid-version
version="6.12.0-rc4.1"
root@Tower:~# uname -mrs
Linux 6.1.26-Unraid x86_64

I was actually going to go Proxmox route before unRAID but I couldn't get the installer working. Mine was just getting stuck at the installer.. Got a guide somewhere?

 

Speaking of unRAID version, what were you using before? My first install with the Asus Prime Z790M-P D4 was RC2. I was able to reach 5sh watts at C10 but it's not constant and was more like 8w on average.

 

Man I wish I got the Intel NIC instead since those 8w with it installed is good! My thought with the CX4 is that it's SFP28 and it's more future proof in the event I upgrade my switch in the future.

 

Also more data points. I decided to try the be  quiet! Pure Power M 850W PSU because I'm think of using my unRAID server to host a gaming PC VM. To my surprise, idle power is the same (if not lower since I added an extra fan) as the PICO PSU. Dark Power 13 850W may probably even more efficient but I don't want to spend more $$ lol.

  • Thanks 1
Link to comment
50 minutes ago, riduxd said:

I was actually going to go Proxmox route before unRAID but I couldn't get the installer working. Mine was just getting stuck at the installer.. Got a guide somewhere?

 

Speaking of unRAID version, what were you using before? My first install with the Asus Prime Z790M-P D4 was RC2. I was able to reach 5sh watts at C10 but it's not constant and was more like 8w on average.

 

Man I wish I got the Intel NIC instead since those 8w with it installed is good! My thought with the CX4 is that it's SFP28 and it's more future proof in the event I upgrade my switch in the future.

 

Also more data points. I decided to try the be  quiet! Pure Power M 850W PSU because I'm think of using my unRAID server to host a gaming PC VM. To my surprise, idle power is the same (if not lower since I added an extra fan) as the PICO PSU. Dark Power 13 850W may probably even more efficient but I don't want to spend more $$ lol.

 

Yeah I faced problems with proxmox GUI installer; I had to do CTRL+ALT+F2 to get console then add some forced configuration override to properly detect the iGPU.

Here's the guide for the error (embedded framebuffer): https://forum.proxmox.com/threads/generic-solution-when-install-gets-framebuffer-mode-fails.111577/

 

Speaking of unRAID version, what were you using before? --- Stable was the version I was using previously during my tests.

 

I do confirm, unRAID C10 is unstable with this board I do not know why that is... Proxmox with kernel 6.0 is stable and will 90%+ stick in C10 with no vms running.

I made a VM inside proxmox to boot my unraid RC4 usb and it worked, I only played with it for like an hour last night but from my early observations once unraid VM boots and no longer uses resources - the parent proxmox host will keep C10 stable - until you start fiddling around unraid web interface to do stuff then it cycles thru higher C states (but thats expected I think).

 

I'm leaning towards having proxmox as my base OS and unraid as the VM with passthru for the SATA controller (since proxmox OS is installed on 2TB nvme) but will do more testing this evening.

  • Thanks 3
Link to comment
On 4/29/2023 at 12:33 PM, TheLinuxGuy said:

 

Yeah I faced problems with proxmox GUI installer; I had to do CTRL+ALT+F2 to get console then add some forced configuration override to properly detect the iGPU.

Here's the guide for the error (embedded framebuffer): https://forum.proxmox.com/threads/generic-solution-when-install-gets-framebuffer-mode-fails.111577/

 

Speaking of unRAID version, what were you using before? --- Stable was the version I was using previously during my tests.

 

I do confirm, unRAID C10 is unstable with this board I do not know why that is... Proxmox with kernel 6.0 is stable and will 90%+ stick in C10 with no vms running.

I made a VM inside proxmox to boot my unraid RC4 usb and it worked, I only played with it for like an hour last night but from my early observations once unraid VM boots and no longer uses resources - the parent proxmox host will keep C10 stable - until you start fiddling around unraid web interface to do stuff then it cycles thru higher C states (but thats expected I think).

 

I'm leaning towards having proxmox as my base OS and unraid as the VM with passthru for the SATA controller (since proxmox OS is installed on 2TB nvme) but will do more testing this evening.

Thanks for the info!

Link to comment
On 4/26/2023 at 9:53 PM, MagickPistacho said:
  • CPU: Intel i3 12100
  • Board: ASUS ROG STRIX B660-G GAMING WIFI
  • PSU: Corsair RM550x (2021)
  • RAM: 1x16 GB Crucial CT16G56C46U5 5600 MHz (clocked at 3200 in BIOS)
  • SSD: Crucial P3 1TB
  • HDD: 3x Toshiba N300 8TB
  • Fans: Arctic P8 PWM PST CO & Arctic P12 PWM PST CO
  • Case: Nox Hummer Vault

 

Well, I've changed my board now to a Gigabyte B660M DS3H DDR4 and the RAM to G.Skill Aegis F4 16GB 2400, and it reaches C10, even with the HDMI connected.

 

0008.thumb.png.c4a85396b204a8197468fe645771ae48.png

Right now it sits at 9-10W on spindown, no HDMI, no keyboard or mouse, WebGUI closed. One disc spinning bumps it up to 13-14W, two around 25-35 it seems.

 

I've tried with no discs and it goes down to about 7-8W, and without SSD and fans it makes little to no difference, 6-7 but sometimes 8 as well.

 

It's not bad but this user here managed to reach 4.5W at idle and 9W with 2 HDDs spinning and the same board, CPU, PSU and RAM: https://www.hardwareluxx.de/community/threads/sparsamer-nas-mediaserver-neuling.1329924/#post-29668608

So I don't know what I'm doing wrong, maybe I need to change some settings on BIOS but I haven't found anything that could do the trick.

Edited by MagickPistacho
grammar, typos and picture
  • Like 1
Link to comment
23 hours ago, MagickPistacho said:

It's not bad but this user here managed to reach 4.5W at idle and 9W with 2 HDDs spinning and the same board, CPU, PSU and RAM

Seagate HDDs have several different power states, I don't think the HDD is spinning at full speed while consuming 9W. And in addition it's a slower version with 5400rpm and 4TB should have only 2 platters = low weight = less power needed to rotate them.

 

Simple solution: Disconnect the HDDs an measure again. 

 

Link to comment

 

1 hour ago, mgutt said:

Simple solution: Disconnect the HDDs an measure again. 

 

 

Yeah, these measurements here
 

On 5/1/2023 at 11:29 PM, MagickPistacho said:

I've tried with no discs and it goes down to about 7-8W

 

were without any HDDs on the NAS, all of them disconected.
That's why I find it strange that he can reach 4.5W, even with discs connected, and I can only go to 7-8W without HDDs.

 

That information about those Seagate HDDs is very interesting though, totally explains how they can have such low consumption. A pity I'm stuck with my Toshiba's though.

I'm gonna try tu upgrade Unraid to the latest version, maybe there's been an upgrade in the kernel, and if that doesn't change anything I'll look some more within the BIOS options.

Link to comment

Hi,
thanks for this topic.

I felt stupid when my dell T20 got power thirsty in idle mode after plugging in an M.2 SSD (with a passive PCIE adaptercard).
Seems like the Lexar NM620 2TB is one of those drivers, that keep the cpu package above C3 level.
With this SSD plugged in, the XEON can't get below C2 anymore. Unplugging the Lexar and it can reach C6 - swapping to a Samsung 970 EVO Plus and C6 is back again.

Trying the Lexar NM620 in a thinkcentre M900 Tiny with I5 6500T show the same "defect by design". Nothing else than C2 reachable. With the 970 the 6500T gets down to C8.
Idling on 4.6W vs. 7W....

Link to comment

CPU: Intel G7400
MOBO: Asrock z690 pg riptide
RAM: Team T-FORCE VULCAN Z 32GB 32gb ddr4 3200
PSU: EVGA P2 650W & Antec EDGE 550W
Storage: 5 HDD (14tb, 10tb, 10tb, 8tb, 6tb), 2 SATA SSD (Crucial M4 64gb, 120gb Samsung 850 EVO 120GB), 1 M2 (WD SN770 1tb)

 

With drives spun down and --auto-tune, idles at 24W with EVGA P2 650w and 27W with Antec EDGE 550W.

 

Can't get past C3. BIOS were changed to enable all ASPM and package states. Tried running UBUNTU with only CPU, RAM, and USB flash drive which was in USB3_1 since its Intel Z690 rather than USB3_2 which is ASMedia ASM3042 but still stuck at C3.

 

Tried echo auto | tee /sys/bus/pci/devices/0000:03:00.0/power/control for ASMedia USB controller but ASPM is still disabled.

 

Assuming Asrock mobo is limiting further package states. PSU could be changed as well but tried with what I had available.

 

ss.JPG

Edited by superpap
Link to comment

I cannot seem to get my system to go past C3, its spec; 

 

M/B:ASUSTeK COMPUTER INC. MAXIMUS VI GENE Version Rev 1.xx - s/n: 130713231100205

BIOS:American Megatrends Inc. Version 1603. Dated: 08/15/2014

CPU:Intel® Core™ i7-4790K CPU @ 4.00GHz

HVM:Enabled

IOMMU:Enabled

Cache:256 KiB, 1 MB, 8 MB

Memory:16 GiB DDR3 (max. installable capacity 32 GiB)

Network:bond0: fault-tolerance (active-backup), mtu 1500

Kernel:Linux 5.19.17-Unraid x86_64

OpenSSL:1.1.1s

 

with a quadro p4000 and around 10 drives + nvme cache via pcie adapter

 

Ive tried many suggestions on this thread, the powertop auto-tune runs and all tunables report as "good" I even swapped out my PCIE sata card for an ASM1166 card and no change, system uses 50w as a bare minimum with all drives spun down, and most times its around 130w, was hoping to do better than that, any suggestions? 

Link to comment

Measurement of my new Unraid Server:

 

Gigabyte B760m DS3H DDR4 + I5-13600k

Bequiet Dark Rock Pro 4 cooler with the standard two blowers

Bequiet Pure Power 12M 550W (PSU)

2x 32GB Kingston Fury DDR4-3600 memory

2x 2TB Samsung 970 EVO plus NVME on board in btrfs Raid 1 ( as long as ZFS is not finally releast with 6.12)

1x Toshiba MB09 18TB parity

1x Seagate Exos X16 12TB data

1x WD red 4TB data

(all HDDs spinned down by Unraid)

Fractal Design R5 black with 2 standard blowers installed

Powertop 2.15

Power consumption in idle: 13W (measured without Unraid GUI on any cliean and server headless, no keyboard/monitor)

 

 

 

Link to comment
3 hours ago, Fantomm3 said:

I cannot seem to get my system to go past C3, its spec; 

 

M/B:ASUSTeK COMPUTER INC. MAXIMUS VI GENE Version Rev 1.xx - s/n: 130713231100205

BIOS:American Megatrends Inc. Version 1603. Dated: 08/15/2014

CPU:Intel® Core™ i7-4790K CPU @ 4.00GHz

HVM:Enabled

IOMMU:Enabled

Cache:256 KiB, 1 MB, 8 MB

Memory:16 GiB DDR3 (max. installable capacity 32 GiB)

Network:bond0: fault-tolerance (active-backup), mtu 1500

Kernel:Linux 5.19.17-Unraid x86_64

OpenSSL:1.1.1s

 

with a quadro p4000 and around 10 drives + nvme cache via pcie adapter

 

Ive tried many suggestions on this thread, the powertop auto-tune runs and all tunables report as "good" I even swapped out my PCIE sata card for an ASM1166 card and no change, system uses 50w as a bare minimum with all drives spun down, and most times its around 130w, was hoping to do better than that, any suggestions? 

 

Check BIOS Settings to ensure C levels are properly set up and that clock and vcore do indeed go down at low loads.

 

I have a very similar processor (4770k with OC at 4.00Ghz) and found that in my case, an option on my gigabyte motherboard (CPU VRIN Loadline Calibration set to Extreme) was preventing the board from lowering the vcore at low loads.

 

Changed this setting to standard and updated the PSU; reducing power draw from ~50w to ~35w with only the ssd spinning (power consumption as reported by the UPS).

 

For the quadro check a video by spaceinvader to help reduce power consumption at idle. 

 

image.thumb.png.a674eae97e92d87d4ccac481a283f806.png

Link to comment
1 hour ago, SP67 said:

 

Check BIOS Setting

 

Thanks, I made sure everything C state related is enabled in bios, I see that the cpu goes down to 800mhz when idle. 

 

Im not sure how to check vclock in unraid? 

 

Also my board is asus (maximus gene vi) not sure the equivalent setting? 

 

Ill have a look for the spaceinvader quadro video, but I dont think it is using power, could be wrong. 

 

 

Link to comment
On 5/9/2023 at 11:10 PM, ullibelgie said:

Measurement of my new Unraid Server:

 

Gigabyte B760m DS3H DDR4 + I5-13600k

Bequiet Dark Rock Pro 4 cooler with the standard two blowers

Bequiet Pure Power 12M 550W (PSU)

2x 32GB Kingston Fury DDR4-3600 memory

2x 2TB Samsung 970 EVO plus NVME on board in btrfs Raid 1 ( as long as ZFS is not finally releast with 6.12)

1x Toshiba MB09 18TB parity

1x Seagate Exos X16 12TB data

1x WD red 4TB data

(all HDDs spinned down by Unraid)

Fractal Design R5 black with 2 standard blowers installed

Powertop 2.15

Power consumption in idle: 13W (measured without Unraid GUI on any cliean and server headless, no keyboard/monitor)

 

 

 

Do you have the idle stats for the C states from Powertop?

I am looking to build something very similar with the B760m DS3H and Pure Power 12M 550W but would like to get an idea if the B760m can get into C10 as easily as the B660m model used by "MagickPistacho".

 

I currently run an Intel NUC which is obviously very low power draw but I need to expand storage. My proposed build is as follows:

  • CPU: Intel i3 13100
  • Board: Gigabyte B760m DS3H
  • PSU: Be quiet! Pure Power 12 M 550W
  • RAM: 1x 16GB G.Skill Aegis F4 3200
  • SSD: 2x 1TB Samsung 970 EVO plus NVME (zfs raidz1 if possible)
  • HDD: 2x 16TB Seagate IronWolf Pro 
  • Case: ? As small as possible with room for 6+ drives (can mod)

Anyone got thoughts on this? The parts (not including case or storage) will set me back $630AUD (~420USD). It's kind of expensive for what I need...but realistically I just want as low idles as possible. Happy to hear suggestions on a different route (granted 2x m.2  and 2.5gbit lan)

Link to comment

The Problem is, that Powertop 2.15 doesn't show correctly the C-states of the most recent CPUs (yet).

Although it shows C8 for some percentage, the total of all C-states is far less than 100%... so there is a mismatch what is shown and what is reality.

@mgutt has tested the Gigabyte B760M Gaming and measured with even Sata expansion card below 10Watt power consumption in the German Forum:

https://forums.unraid.net/topic/138498-gigabyte-b760m-gaming-x-ddr4-messung-und-sata-erweiterung/

 

So I guess that both boards to indeed reach C10 states with Raptor Lake CPUs

Link to comment

I hope someone can help me. No matter what i try the c states are not going lower that state 3 in powertop 2.15.

I testes it with ubuntu without gpu in headless, no usb attached and all powersaving options in bios enabled (pcie, cstates, disabled usb, audio, ethernet, wlan, disbled turbo…).

 

Here is my config:

CPU: Intel i3 13400f

Board: ASUS MB Prime H610I-Plus | AsRock B660M-ITX/ac

RAM: 2x Curcial 8GB (stock non overclocked)

SSD: Lexar NM620 | Samsung 980 pro | Crucial P3

 

„|“ means that I tried these different parts by swapping them out.

Link to comment
1 hour ago, Theroid said:

are not going lower that state 3 in powertop 2.15.

Are you seeing other PKG states than C3? 

 

2 hours ago, Theroid said:

Samsung 980 pro

This is the best choice to test the c-statea I think. The Lexar could have a negative impact.

Link to comment
2 hours ago, mgutt said:

Are you seeing other PKG states than C3? 

 

This is the best choice to test the c-statea I think. The Lexar could have a negative impact.


Thank you very much for your help.

I am indeed seeing other states than C3, as low as to C10.

However everything greater than C3 remains at 0% 

If needed I can send a picture.

Link to comment
58 minutes ago, Theroid said:

However everything greater than C3 remains at 0% 

Ok. I suggest to test with the Asus Board and the Samsung NVMe again. Then check the BIOS settings. You said you enabled all powersaving options. What does it mean? Did you only enable C-States or did you enable each C-State separately and set C10 as the maximum? As of my experience, the default "auto" does nothing.

 

Do you have a SATA SSD to exclude the CPU M.2 Slot as the problem? Or maybe boot without any drive except the USB drive. Another idea would be to switch the USB ports. I would try one under the PS2 and after that one under the ethernet port. Sometimes this has an impact as well.

 

PS I checked this photo of the Asus board and I can't see any additional controllers which could have a negative impact on C-States, but regarding the specs it is using a Realtek 1 Gbit Ethernet controller. Maybe this is the reason for the bad C-State. Regarding the Asrock board: Sadly those really often have bad C-States.

Link to comment
1 hour ago, mgutt said:

Ok. I suggest to test with the Asus Board and the Samsung NVMe again. Then check the BIOS settings. You said you enabled all powersaving options. What does it mean? Did you only enable C-States or did you enable each C-State separately and set C10 as the maximum? As of my experience, the default "auto" does nothing.

 

Do you have a SATA SSD to exclude the CPU M.2 Slot as the problem? Or maybe boot without any drive except the USB drive. Another idea would be to switch the USB ports. I would try one under the PS2 and after that one under the ethernet port. Sometimes this has an impact as well.

 

PS I checked this photo of the Asus board and I can't see any additional controllers which could have a negative impact on C-States, but regarding the specs it is using a Realtek 1 Gbit Ethernet controller. Maybe this is the reason for the bad C-State. Regarding the Asrock board: Sadly those really often have bad C-States.

Again. Thank you very much for you help and input. I was near on giving up.
 

I could only activate the max C-State, there are no single options. 
I will upload pictures with the settings highlighted. All other settings are kept stock.

 

I will try to boot from a usb stick only and no other connected drives. Will update when Im done testing it.
 

 

IMG_2754.jpeg

IMG_2753.jpeg

Edited by Theroid
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.