Reduce power consumption with powertop


Recommended Posts

Powertop [Support auf Deutsch]

 

Powertop is an Intel tool (yes, works for AMD, too) to check power consumption states of sata, pcie, usb, etc devices. You can install powertop through the NerdPack Plugin and execute it as follows:

powertop

 

Now press "TAB" until you reach "Idle Stats" and check your CPU Pkg (HW). Depending of the brand and model of your CPU you can reach different C-States. If your system isn't optimized, you will see this:

179241080_2021-07-1312_14_03.png.5a2dc3644c02b04d052699f05a312332.png

 

If you press "TAB" you will see additional information about standby states of different hardware components of your system. The following command will optimize them:

powertop --auto-tune

 

By that all compents will return "Good":

1191854226_2020-10-2112_53_35.png.1465ea3c92903b3a9bc63dfe4b51d0d0.png

 

Note: If you still don't reach the Pkg C-States after executing "--auto-tune", you probably need to activate the C-States in the BIOS.

 

How much energy can be saved?

After reboot my server's power consumption was reduced by 4W (of 24W, which is 20%)! You can save even more energy with Intel Undervolting and enabling powersave for your CPU through the Tips & Tweaks Plugin.

 

Powertop Auto Tune

If powertop is permanently installed on your server (can be installed through "Nerd Pack Plugin"), you can add this to your Go File:

# -------------------------------------------------
# Reduce power consumption
# -------------------------------------------------
powertop --auto-tune

 

Execute this command in addition through your WebTerminal and you don't need to restart your server.

 

Without Powertop

If you don't want to install powertop, you can add the different power-saving commands to your go file manually (read this thread if you are interested in the "haveged" line):

 

 

# -------------------------------------------------
# disable haveged as we trust /dev/random
# https://forums.unraid.net/topic/79616-haveged-daemon/?tab=comments#comment-903452
# -------------------------------------------------
#/etc/rc.d/rc.haveged stop

# -------------------------------------------------
# powertop tweaks
# -------------------------------------------------

# Enable SATA link power management
echo med_power_with_dipm | tee /sys/class/scsi_host/host*/link_power_management_policy

# Runtime PM for I2C Adapter (i915 gmbus dpb)
echo auto | tee /sys/bus/i2c/devices/i2c-*/device/power/control

# Autosuspend for USB device
echo auto | tee /sys/bus/usb/devices/*/power/control

# Runtime PM for disk
echo auto | tee /sys/block/sd*/device/power/control

# Runtime PM for PCI devices
echo auto | tee /sys/bus/pci/devices/????:??:??.?/power/control

# Runtime PM for ATA devices
echo auto | tee /sys/bus/pci/devices/????:??:??.?/ata*/power/control

# Disable wake on lan
for i in /sys/class/net/eth?; do ethtool -s  $(basename $i) wol d; done

 

Config File Editor:

1127764090_2020-10-2112_59_21.png.6c68b2422b81593f8d85f261748b34b1.png

 

image.png

  • Like 1
  • Thanks 2
Link to comment

Seems to work great for me!

 

Had to disable this as the Array did not come online with this enabled:

 

# Runtime PM for port ata od PCI devices
#for i in /sys/bus/pci/devices/????:??:??.?/ata*/power/control; do
#    echo 'auto' > $i
#done


 

Also did add this:

# VM writeback timeout
for i in /proc/sys/vm/dirty_writeback_centisecs; do
    echo '1500' > $i
done

 

Have a Supermicro X9DRi-LN4+ MB

Edited by Allram
Added code brackets
Link to comment
31 minutes ago, Allram said:

Had to disable this as the Array did not come online with this enabled:

Generate a report with powertop. It will contain the commands to set auto for unused pci slots. I don't know why this general rule is a problem for your server. It seems one of your pci cards has a problem with automatic power control.

  • Like 1
Link to comment

Thanks for the tip!
I ran the option:

--auto-tune

All seems fine after 16hrs. Current consumption at an average of 8.5W. Coming from Ubuntu where I had an average of 6.5W but that was without the samsung flash-drive and ConbeeII stick. I have more USB devices attached but those where already there under Ubuntu.

I do suspect a rather high consumption from the Samsung 32 GB Bar Plus flash-drive. After switching to this drive from a Kingston one during the Unraid trial period, consumption rose by 1W.  

 

Link to comment
6 hours ago, CS01-HS said:

There has to be some cost to this, no? I haven't noticed a difference but nothing's free.

Possible reduced stability. It's almost like the inverse of overclocking. Circuits are designed and tested to run at spec, deviate and you risk bit errors. Depending on the quality of your specific silicon you might be fine, but you don't know for sure.

 

How valuable is your data integrity?

 

See @mgutt's reply, I was referencing undervolting, wrong thread.

Edited by jonathanm
wrong thread
  • Thanks 1
Link to comment
23 minutes ago, jonathanm said:

Possible reduced stability. It's almost like the inverse of overclocking. Circuits are designed and tested to run at spec, deviate and you risk bit errors. Depending on the quality of your specific silicon you might be fine, but you don't know for sure.

 

How valuable is your data integrity?

I wouldn't take any meaningful risk to save 4W.


Would you distinguish between any of the power-saving tweaks (SATA links, I2C, USB, PCI and increasing dirty_writeback) in terms of risk, assuming a UPS/no unexpected power loss?

Link to comment
4 hours ago, CS01-HS said:

There has to be some cost to this, no?

No, because by default Linux does not switch off unused pcie, usb, sata, iGPU, etc. This script sets only all devices from (permanently) ON to AUTO (reduce energy consumption / switch off, if not used). This is different compared to Windows or Apple where energy saving is the default.

 

Source:

https://www.kernel.org/doc/html/v4.14/driver-api/pm/devices.html#sys-devices-power-control-files

"The setting can be adjusted by user space by writing either “on” or “auto” to the device’s power/control sysfs file. Writing “auto” calls pm_runtime_allow(), setting the flag and allowing the device to be runtime power-managed by its driver. Writing “on” calls pm_runtime_forbid(), clearing the flag, returning the device to full power if it was in a low-power state, and preventing the device from being runtime power-managed. User space can check the current value of the runtime_auto flag by reading that file."

 

This has nothing to do with underclocking currently used devices. It only reduces power of a device that is not used (which is controlled by its driver). This is the same as CPU C-States. 

Edited by mgutt
  • Like 1
  • Thanks 1
Link to comment
22 minutes ago, CS01-HS said:

dirty_writeback) in terms of risk, assuming a UPS/no unexpected power loss?

dirty_writeback is the only risky setting, but it's not raised by powertop, it's reduced. The default is 30 seconds. Powertop sets it to 15 seconds. This means Linux starts 15 seconds earlier to empty the RAM to disk. This can save energy as all devices can reach the sleep state 15 seconds earlier.

 

Note: A UPS is absolutely recommend or all data of the last 15 seconds will be lost. Unraid default: Even 30 seconds.

 

Example: You move important data to your server (so it's "deleted" from your client). If now a power outage occurs in the 30 seconds since the first file was uploaded, everything is lost.

 

I think this point needs more attention through Limetech. vm.dirty_ratio should be reduced to 1% and vm.dirty_writeback_centisecs should be reduced to 0.5 seconds (disabling has a huge negative impact on performance) if no UPS is connected.

 

I raised my dirty_ratio to 50% and left the 30 seconds because I'm using a UPS and it allows to upload a huge amount of files even to HDDs with maximum 10G speeds (as it's written to RAM and emptying the RAM happens later)

Edited by mgutt
  • Like 1
Link to comment
1 hour ago, mgutt said:

No, because by default Linux does not switch off unused pcie, usb, sata, iGPU, etc. This script sets only all devices from (permanently) ON to AUTO (reduce energy consumption / switch off, if not used). This is different compared to Windows or Apple where energy saving is the default.

Sorry for the misunderstanding, I thought I was in the undervolting power save thread.

  • Haha 1
Link to comment
  • 2 weeks later...

I saw a few of these hard resetting link errors during my mover run. Thankfully (?) no CRC errors reported. ata3 is a spinning disk attached to an integrated ASM1062 controller.

 

I wonder if it might be related to the power-saving tweaks because nothing else changed. For now I've disabled them and will see if they reappear. Maybe coincidence but I'm posting in case others have the same issue.

Nov  3 23:17:30 NAS move: move: file /mnt/cache/Download/movie_1.mp4
Nov  3 23:17:33 NAS kernel: ata3.00: exception Emask 0x10 SAct 0x80 SErr 0x4050002 action 0x6 frozen
Nov  3 23:17:33 NAS kernel: ata3.00: irq_stat 0x08000000, interface fatal error
Nov  3 23:17:33 NAS kernel: ata3: SError: { RecovComm PHYRdyChg CommWake DevExch }
Nov  3 23:17:33 NAS kernel: ata3.00: failed command: WRITE FPDMA QUEUED
Nov  3 23:17:33 NAS kernel: ata3.00: cmd 61/00:38:58:44:51/04:00:2c:02:00/40 tag 7 ncq dma 524288 out
Nov  3 23:17:33 NAS kernel:         res 40/00:30:58:40:51/00:00:2c:02:00/40 Emask 0x10 (ATA bus error)
Nov  3 23:17:33 NAS kernel: ata3.00: status: { DRDY }
Nov  3 23:17:33 NAS kernel: ata3: hard resetting link
Nov  3 23:17:33 NAS move: move: file /mnt/cache/Download/movie_1.mp4
Nov  3 23:17:33 NAS kernel: ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Nov  3 23:17:33 NAS kernel: ata3.00: supports DRM functions and may not be fully accessible
Nov  3 23:17:33 NAS kernel: ata3.00: supports DRM functions and may not be fully accessible
Nov  3 23:17:33 NAS kernel: ata3.00: configured for UDMA/133
Nov  3 23:17:33 NAS kernel: ata3: EH complete
Nov  3 23:17:35 NAS move: move: file /mnt/cache/Download/movie_2.mp4

 

Edited by CS01-HS
Link to comment

Just tested powertop and undervolt. Great tools !

 

My configuration is describe in my signature (and in this post : 

With 

undervolt --gpu -75 --core -100 --cache -100 --uncore -100 --analogio -100

and your startup script to enable autosleep mode on devices, I see some results but not that much.

 

First, 15mn after reboot, my server uses 28-30W. Before modification, it was 29-31W.

But now, my powermeter seems a bit "crazy" because the power usage varies lot more than before. It is even difficult to read a number !! it changes so quickly.

Be careful, my "idle" status is maybe not yours. My server never sleeps. I always have my seedbox running (torrent) + nextcloud + vpn and a dozen of users behind (my family phones + laptop, etc...). They do not consumes that much of server load, but they prevent disks to enter in sleep mode. For example I hardly see more than 2 data disks in sleep mode, often I can see one, and most of the time none are asleep.

 

Under load, (recording a live TV + watching 1 movie with HW transcoding), my server uses 39-42W. Before modifications, it was around 42-44W.

With no HW trancoding, up to 50W.

 

I already did some test with spin down, and even If I stop the array, power usage is around 25-27W. So with your modifications I am maybe around 24-26W now...

Most of power optimizations is not on disks (I have 8x2.5 disks). 

 

Maybe I missed something, but powertop does not show me watt usage per device. I would like to really estimate the power usage per device, for example the my LSI HBA controler and my TBS DVB-T tuner.

 

 

 

 

Edited by tapodufeu
Link to comment

Bingo.... just to be sure I restarted my server and checked the BIOS. Gigabyte Platform Power Management was disable !!

I have also disable the embed audio controler of the motherboard.

 

So now, just a couple of minutes after reboot, I am around 25-28W. 

If I spin down all disk, 24-25W.

 

I have already tested in my previous post results of undervolting my CPU under load.

 

 

Link to comment

powertop can't show you the power usage of devices (as far as I know). The best method to check the energy consumption is to stop the array so dockers / vms are stopped as well and spindown each HDD with this command:

mdcmd spindown 0
mdcmd spindown 1
mdcmd spindown 2
mdcmd spindown 3
mdcmd spindown 4
...

And if you want to know how much the DVB-T card consumes, you need to restart the server without it (and disabled array auto start). Thats how I measured my 10G card. Regarding the LSI card it would be a little bit more complicate. I would connect one HDD to it and then restart the server with the HDD connected to an onboard port, so you get the difference.

 

Do you use all PCIe slots? That could be the reason why powertop has not a huge impact as your SATA and TV cards are permanently in use.

Link to comment

Yes, all are on PCIe slots... and I was investigating just a couple of minutes ago and I found:

 

01:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
        Subsystem: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]
...
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+

 

04:00.0 Multimedia controller: TBS Technologies DVB Tuner PCIe Card
        Subsystem: Device 6281:0002
...
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk+

 

05:00.0 Non-Volatile memory controller: Toshiba Corporation Device 011a (prog-if 02 [NVM Express])
        Subsystem: Toshiba Corporation Device 0001
...
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes Disabled- CommClk+

 

ASPM is only enable for my nvme drive on pcie.

Any idea ?

 

in dmesg I found the following lines:

[    0.151347] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it

[   19.362570] r8169 0000:02:00.0: can't disable ASPM; OS doesn't have ASPM control

[   19.385178] mpt3sas 0000:01:00.0: can't disable ASPM; OS doesn't have ASPM control

 

r8169 is the network card, and I am happy ASPM is disabled for it.

mpt3sas is related to the LSI HBA.

 

I see some posts about linux kernel and ASPM issues with PCIe... still investigating

 

About power usage of HBA controler, I found this post listing many LSI cards

https://www.servethehome.com/lsi-host-bus-adapter-hba-power-consumption-comparison/

 

For the TBS 6281 SE, when not used, it is difficult to say, maybe 1 or 2 watt. My wattmeter is not accurate enough. I don't really see any change with or without.

When recording + transcoding with plex (hw transcode), it consumes approx 10W. Recording without transcoding failed, I was not able to see the file even with VLC. First time I tried so maybe I used wrong settings...

Posts on internet reports up to 25W for the TBS DVB-T card, I was not able to reproduce at all this consumption. I tested on the french DVB-T on the channel France 5.

 

Edited by tapodufeu
add lsi power usage listing and tbs test
Link to comment
  • 4 weeks later...

Hello @mgutt,

 

Nice work with this power consumption tweaking guide !

I use this script from many years on my nas (OMV then Unraid), without any problem. (6 sata HDD with spindown + 2 sata SSD, DVB-t tuner)

It's basically setting pci/sata power management to auto.

 

I'll try your undervolting guide as soon as I have free time. ^^

 

 

Link to comment
  • 2 months later...

Hey,

i just want to start to optimize my new installed unraid server, but i cannot find the powertop command. Is it not installed anymore on 6.8.3?

 

Allready try to find it with "find / -name "powertop"" but, nothing. 

 

Do i have to install it manually?

Link to comment
2 minutes ago, teqqyde said:

Hey,

i just want to start to optimize my new installed unraid server, but i cannot find the powertop command. Is it not installed anymore on 6.8.3?

 

Allready try to find it with "find / -name "powertop"" but, nothing. 

 

Do i have to install it manually?

You have to install "nerdpack" from the Apps first. After install is done you run Nerdpack, and within you find Powertop

Link to comment
  • 2 weeks later...

Very nice !

 

With my setup (viewable in my signature), I went from 25w in idle (no spinning) to 21W !

 

I switched values from Bad to Good. Is persistent after a reboot ?

Edited by Alex.b
Link to comment
  • 1 month later...
On 11/4/2020 at 1:30 PM, CS01-HS said:

I saw a few of these hard resetting link errors during my mover run. Thankfully (?) no CRC errors reported. ata3 is a spinning disk attached to an integrated ASM1062 controller.

 

I wonder if it might be related to the power-saving tweaks because nothing else changed. For now I've disabled them and will see if they reappear. Maybe coincidence but I'm posting in case others have the same issue.

Nov  3 23:17:30 NAS move: move: file /mnt/cache/Download/movie_1.mp4
Nov  3 23:17:33 NAS kernel: ata3.00: exception Emask 0x10 SAct 0x80 SErr 0x4050002 action 0x6 frozen
Nov  3 23:17:33 NAS kernel: ata3.00: irq_stat 0x08000000, interface fatal error
Nov  3 23:17:33 NAS kernel: ata3: SError: { RecovComm PHYRdyChg CommWake DevExch }
Nov  3 23:17:33 NAS kernel: ata3.00: failed command: WRITE FPDMA QUEUED
Nov  3 23:17:33 NAS kernel: ata3.00: cmd 61/00:38:58:44:51/04:00:2c:02:00/40 tag 7 ncq dma 524288 out
Nov  3 23:17:33 NAS kernel:         res 40/00:30:58:40:51/00:00:2c:02:00/40 Emask 0x10 (ATA bus error)
Nov  3 23:17:33 NAS kernel: ata3.00: status: { DRDY }
Nov  3 23:17:33 NAS kernel: ata3: hard resetting link
Nov  3 23:17:33 NAS move: move: file /mnt/cache/Download/movie_1.mp4
Nov  3 23:17:33 NAS kernel: ata3: SATA link up 6.0 Gbps (SStatus 133 SControl 300)
Nov  3 23:17:33 NAS kernel: ata3.00: supports DRM functions and may not be fully accessible
Nov  3 23:17:33 NAS kernel: ata3.00: supports DRM functions and may not be fully accessible
Nov  3 23:17:33 NAS kernel: ata3.00: configured for UDMA/133
Nov  3 23:17:33 NAS kernel: ata3: EH complete
Nov  3 23:17:35 NAS move: move: file /mnt/cache/Download/movie_2.mp4

 

 

Due to the error above I've run my integrated ASM1062 controller with /sys/class/scsi_host/host*/link_power_management_policy set to max_performance (the default) vs --auto-tune's med_power_with_dipm

 

After updating to 6.10.0-rc1 a few days ago I tried powertop's med_power_with_dipm again and so far the error hasn't reappeared. Whatever incompatibility appears to be solved.

 

EDIT: I spoke too soon. Under heavy load I saw hard resets again and the (Seagate) drive's Command timeout entry increased by two.

Edited by CS01-HS
  • Like 1
Link to comment
  • 5 weeks later...
~# powertop --auto-tune
modprobe cpufreq_stats failedLoaded 0 prior measurements
RAPL device for cpu 0
RAPL device for cpu 0
Devfreq not enabled
glob returned GLOB_ABORTED
 the port is sda
 the port is sdb
 the port is sdc
 the port is sdd
 the port is sde
 the port is sdf
 the port is sdg
 the port is sdh
 the port is sdi
 the port is sdj
 the port is sdk
 the port is sdl
 the port is sdm
 the port is sdn
 the port is sdo
Leaving PowerTOP

 

OK, I'm new to this. What do those failed and aborted mentions mean? I suppose I need to take my old monitor to the crawlspace and change BIOS settings for this machine, no?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.