Reduce power consumption with powertop

[Aldamir]

So, maybe this is helpful for someone down the line.

Currently i run this setup:

AsRock H570M-ITX ,

i5-11400,

8GB Ram (1 Stick) - previously 16GB with 2 sticks

1 Cache SSD SN570 1TB (previously 2 SSDs)

3x Seagate IronWolf 8TB,

PSU: Corsair RM650x (previously SF450 Platinum, SFX)

 

I managed to reduce my C-state from C7 to C10 and reduce idle power consumtion (meaning HDD spun down but with various docker containers running) from 21W to 11W.

 

1. With a Reddit guide that utilizes AMISCE, i manage to reach C8, but only leading to 1-2W less
2. Removing the SSD of the CPU PCIe-lane, got me to C10 and another few Watts
3. Removing a Ram slot lead to a neglible difference
4. What really changed a lot was using the ATX-PSU RM650x which performs way better than the SF450 Platinum (which was already tested for the RM550x, for other low wattage PSUs check Wolfgangs PSU db.

Maybe that helps someone down the line

[travelingalex]

I'm pulling out my hair trying to figure this out...I have an issue running powertop --auto-tune where it would freeze my entire system. I tried resetting my bios and install a fresh version of Unraid - after I run the basic steps in the original post, the server still freezes. I even removed all drives/usb devices/drives, except the Unraid usb. I'm going crazy trying to figure this out. I just wanted to be able to run powertop autotune with no issues and then start tweaking the bios to get the power consumption down. 

 

Asus WS W680M-ACE SE

Intel Core i5-12400 

32GB Kingston ECC

 

Sometimes it doesn't freeze and I get this error:

Edited October 7 by travelingalex

[pOpYRaid]

@travelingalex That's weird. Some ideas:

• Is your BIOS up-to-date?
• Reset BIOS to factory default's

Give it a try.

[mgutt]

3 hours ago, travelingalex said:

I just wanted to be able to run powertop autotune with no issues

Simple answer: Not possible. One of your devices does not support sleep states, although it claims to do so (powertop enables only sleep states of devices which offer this setting).

 

3 hours ago, travelingalex said:

I tried resetting my bios and install a fresh version of Unraid

This won't help. You need to find the problematic device. This can be done be executing the commands in the first post line by line in the unraid terminal. After you find out which command causes the crash, we can try to find the specific device.

 

3 hours ago, travelingalex said:

Sometimes it doesn't freeze and I get this error

Those aren't errors, only warnings. Can be ignored.

[travelingalex]

Thanks @pOpYRaid and @mgutt!

 

Turns out the issue was with running the command below. Played around with the BIOS (including updating and starting at factory default) and was able to get it working! Not sure exactly what BIOS settings I changed to get it going though lol. 

echo auto | tee /sys/bus/pci/devices/????:??:??.?/ata*/power/control

 

I'm still stuck on C3. I've searched through every post and can't seem to figure out why I can't go any higher (disconnected all drives and usb drives except for keyboard). I'll keep it at and report back! 

 

Edited October 7 by travelingalex

[mgutt]

7 hours ago, travelingalex said:

I've searched through every post

And why are you then using the web terminal? From the first post:

 

"close your Unraid WebGUI and wait 1 minute! (this closes many background jobs, which cost your energy, use your PC's terminal and "ssh root@tower" to monitor your C-States with powertop)"

 

Some users had even hardware which did not allow deep c states as long an Ethernet cable was connected. They found it out after connecting a monitor and executing powertop locally.

 

7 hours ago, travelingalex said:

I'm still stuck on C3

Do you see more than pkg C3? If not, again first post:

 

"Powertop shows only C1, C2 and C3 Pgk-States - This happens if powertop does not recognize your CPU (happens for most AMD CPUs and recent Intel CPUs). Your CPU could still reach C9 or even C10, but powertop is not able to obtain this and shows only C3."

[pOpYRaid]

8 hours ago, travelingalex said:

Thanks @pOpYRaid and @mgutt!

 

Turns out the issue was with running the command below. Played around with the BIOS (including updating and starting at factory default) and was able to get it working! Not sure exactly what BIOS settings I changed to get it going though lol. 

echo auto | tee /sys/bus/pci/devices/????:??:??.?/ata*/power/control

 

I'm still stuck on C3. I've searched through every post and can't seem to figure out why I can't go any higher (disconnected all drives and usb drives except for keyboard). I'll keep it at and report back! 

 

Have you disconnected HDMI cable to the monitor?
In my case when the iGPU is active, it also reaches lower C states and consumes more power.
Try to measure the power consumption maybe as @mgutt stated, it reaches low power state.

 

This is a good starting point:

• boot ubuntu live from usb
• disconnect all not needed peripheral
• set hdd sleep to 1 min (if any hdd connected)
• install and run powertop (sudo apt get powertop)
• disconnect monitor
• wait 2 min
• meassure with an kill-a-watt (or similar)
• Wait again 2 min
• connect monitor
• check powertop results

Good luck

[spyrosj]

Quote

10G networking

SFP+ DAC consumes less power than SFP+ SR/LR and both less than 10G RJ45. Depending on your internet connection the most efficient scenario is not to use a 10G switch. Instead directly connect your client with your server and use a second 1G cable to connect with your internet router. By that your 10G controller/card goes to sleep while your client is powered off. In comparison an active connection to a 10G switch never allows sleep states.

 

Any suggestions on low power 10g SFP+ nics that are the best for direct connection? I'm contemplating on getting one for each of my servers so that backup jobs can run faster and potentially save power. 

 

The backup job would probably be:

1. Wake-on-lan or wake on RTC my backup server every day at night
2. Run a CRON rsync job on reboot that transfers from the main server's cache to the backup server's cache
3. After the rsync job run mover on the backup server to move cache contents the main array which is JBOD
4. Let the S3 Sleep plugin or the script shutdown the backup server once the job completes and the drives are idle.

 

[Espressomatic]

Not sure if anyone else has seen these type of power-saving settings drastically affect their VM-based network speeds, but I finally tracked down (roughly) what happened to my system.

 

Mini system running Intel i5-1235U with 16GB and 256GB NVME. Unraid 7.00 (beta 2, 3 or 4)

4x 2.5Gbit ports

2x 10Gbit ports

 

VM running pfSense with 1x 2.5Gbit bridged port for WAN and x 1x 10Gbit bridged port for LAN (virtio).

 

Unraid runs on (other) 1x10Gbit port eth0 connection.

 

Prior to making any BIOS changes I could measure a consistent 1700 Mbit download and 1080 - 1100 Mbit upload directly on the pfSense VM or a wired Mac. This matched the speed measured directly on the modem.

 

If I apply the various settings in BIOS required/suggested, such as non-auto setting for max C-state, Turbo OFF, my download speeds drop to 600-800 Mbit and upload to 800-900 Mbit consistently.

 

However, if I boot pfSense bare-metal on it own, I don't see these speed reductions.

 

It took me a few days to figure out what was going on because I hadn't correlated the BIOS changes to the time at which the issue started happening. Eventually it was the only thing left to look at and simply resetting BIOS to defaults immediately resolved the issue. Since then I've gone back and applied only a few settings as mentioned above, and immediately the issues come back.

 

 

Edited October 23 by Espressomatic

[generalxxxstark]

Just wanted to share my journey to optimize my PC build for lower power consumption. Paying almost $0.30 per kWh kicked off this mission to create an efficient setup while maintaining PCIE expansion options. I tested four motherboards on the LGA 1700 platform with the following base specs:

 

My base specs:

 

• i5 12400
• 32gb Corsair DDR5/DDR4
• 1TB WD SN730 nvme
• 3x18 TB Toshiba N300 HDDs
• Seasonic PX750
• 2x80mm fans

 

Initially I bought the ASRock B760M Pro RS/D4 since I had some DDR4 ram lying around but quickly found out no matter what I did it would not go past C3. There are some mods that let you unlock additional C states, but the best I got was C6, saving about 1W. In Unraid, with HDDs spun down, ~12 dockers running, it idled at ~25W.

 

Next, I switched to the Asus Prime B760M-A AX DDR5. It reaches C10 without an SSD, idling at ~15W. Interestingly, with SSD in the CPU-connected NVMe slot, it reached C3. With the SSD in the chipset-connected slot, it reached C6. Overall power with HDDs spun down in Unraid, overall power consumption was ~20W.

 

Then came the Asus Z790-AYW. This board reached C10 without an SSD at ~15W. With the SSD in any NVMe slot, it consistently hit C10, idling at ~15W. Power consumption with HDDs spun down was ~17W. Interestingly, with HDDs spun up, it stays at C10, but with them spun down, it only reaches C8. It was been quite stable and offers ample PCIe expansion with 4X16 and 3 NVMe slots.

 

Finally, I tested the Asus TUF Gaming Z790-PLUS Wifi. Unfortunately, the Intel 226 NIC showed up as ASPM disabled and the system would not go past C2.

 

As a side note, my PSU’s efficiency is about 70% at 20W. Perhaps switching to the Corsair RM750x could save another ~2W. All power measurements were taken using a Sonoff S31 running Tasmota.

 

 

Edited October 27 by generalxxxstark
formatting

[hot22shot]

On 10/27/2024 at 8:24 PM, generalxxxstark said:

As a side note, my PSU’s efficiency is about 70% at 20W. Perhaps switching to the Corsair RM750x could save another ~2W. All power measurements were taken using a Sonoff S31 running Tasmota.

 

Finding an efficient PSU at 20% load is a challenge, i'm not sure what it is worth but I found this site https://www.cybenetics.com/index.php?option=psu-performance-database that has some data on PSUs including graphics that show the PSU`s efficiency throughout its entire operational range.

[FlowDE]

On 10/7/2024 at 5:36 AM, travelingalex said:

I'm pulling out my hair trying to figure this out...I have an issue running powertop --auto-tune where it would freeze my entire system. I tried resetting my bios and install a fresh version of Unraid - after I run the basic steps in the original post, the server still freezes. I even removed all drives/usb devices/drives, except the Unraid usb. I'm going crazy trying to figure this out. I just wanted to be able to run powertop autotune with no issues and then start tweaking the bios to get the power consumption down. 

 

Asus WS W680M-ACE SE

Intel Core i5-12400 

32GB Kingston ECC

 

Sometimes it doesn't freeze and I get this error:

You Intel Card could be the culprit
I reported the issue on https://bugzilla.kernel.org/show_bug.cgi?id=218499

[generalxxxstark]

I just got the Intel X710 DA-2 so that I can run dual 10G to my switch. It does seem to support ASPM L1 but for some reason the system won't go past C3. Is there anything additional I need to do?

 

I have disabled the onboard RTL8125 which used to hit C10.

 

Edit: its now working.

 

I pulled out the SSDs, tested the NIC it reached C8. Now with everything plugged in, it reaches C10. Lol this is some odd behavior. Nevertheless, the system runs about ~3W lower than with the RTL8125 but at 10G.

 

 

 

Edited November 6 by generalxxxstark

[Solverz]

What does the below tunables actually do for the drives?

I read online that they allow for the drives to enter idle Device states like D3 Hot and D3 Cold.

 

But wouldn't this be spinning down the drives? If so, is it not better to prevent this and allow UNRAID to spin down the drives after the idle time you set? As if not, I worry these tunables will spin down the drives too quickly.

 

Or do I misunderstood what these tunables do for drives?

 

`Runtime PM for disk sdX`

 

`Runtime PM for port ataX of PCI Device:... `

[mgutt]

20 hours ago, Solverz said:

Or do I misunderstood what these tunables do for drives?

Yes. It has nothing to do with spin down. It influences the sata connection itself. "PM" stands for "Power Management":

https://en.wikipedia.org/wiki/Aggressive_Link_Power_Management

 

 

[Solverz]

2 hours ago, mgutt said:

Yes. It has nothing to do with spin down. It influences the sata connection itself. "PM" stands for "Power Management":

https://en.wikipedia.org/wiki/Aggressive_Link_Power_Management

 

 

 

`Enable SATA link power management for...`

So this tunable is to configure the LPM level?

 

`Runtime PM for port ataX of PCI Device:... `

This tunable is to enable LPM for specific ports?

 

`Runtime PM for disk sdX`   

Then what is this one for?

 

Appreciate the help, thank you!

 

 

 

Edited November 10 by Solverz
Changed thoughts.

[bpham]

On 11/5/2024 at 9:19 PM, generalxxxstark said:

I just got the Intel X710 DA-2 so that I can run dual 10G to my switch. It does seem to support ASPM L1 but for some reason the system won't go past C3. Is there anything additional I need to do?

 

I have disabled the onboard RTL8125 which used to hit C10.

 

Edit: its now working.

 

I pulled out the SSDs, tested the NIC it reached C8. Now with everything plugged in, it reaches C10. Lol this is some odd behavior. Nevertheless, the system runs about ~3W lower than with the RTL8125 but at 10G.

 

 

 

If you start a VM do you lose your c-states? I managed to get C10 working with a X710-da2 as well but as soon as I start a VM I lose the c-states

[generalxxxstark]

On 11/12/2024 at 8:21 AM, bpham said:

If you start a VM do you lose your c-states? I managed to get C10 working with a X710-da2 as well but as soon as I start a VM I lose the c-states

 

I only have a single VM running Proxmox backup server and 12 docker containers but I do not lose any C-states.

 

The VM is using virtio-net for its network adaptor.

Edited November 13 by generalxxxstark

[1471]

On 11/12/2024 at 4:21 PM, bpham said:

If you start a VM do you lose your c-states? I managed to get C10 working with a X710-da2 as well but as soon as I start a VM I lose the c-states

Running a different set up to you, but I have the same issue with VM's. Can hit C8, but can't get any C State soon as I start a VM (Server 2022 + Home Assistant).

 

Bit of info here:

 

[bpham]

I'm running a win 10 VM and can't hit c-states as soon as I start it. Haven't done any additional testing to see if windows is the cause, but it seems like it since you're running server.

[xyzz1887]

Not sure if there are still people active here, but I'm getting a little frustrated and need some advice.

I can't reach anything higher than pkg C2.

My Server is running on a ASRock H310-DVS Mainboard with an I3 8100. No SATA Extension cards, no NICs, nothing. Just 1x Seagate HDD and 2x (no name)SATA SSDs.

I've enabled all C-States in the BIOS and pretty much every other power setting i could find.

 

I also tried unplugging my network cable and using powertop directly, didn't change anything.

Also tried stopping the array, so i got no containers or VMs active, didn't change anything.

In the "Tunables"-Tab in Powertop everything shows "Good", so no issue there ig.

 

Any1 got ideas?

 

Images attached below:

Edited 7 hours ago by xyzz1887
grammar correction

[xyzz1887]

53 minutes ago, xyzz1887 said:

and 2x (no name)SATA SSDs.

Yea i guess that's the problem here.. Unplugged them both, now i'm reaching C7 Gotta buy new SSDs i guess..