Reduce power consumption with powertop

[Stubbs]

Are the Pure Power 11 FM and RM550x still the only known PSUs that are extremely efficient at low loads? I can't buy them anywhere.

[vmasip]

On 12/13/2023 at 3:13 PM, fightforlife said:

Your power top screen looks really good! 

CPU cores are even reaching C10, which is quite a feat.

pc3 = is the package c state includibg all controller. (Memory, SATA, network. Etc..)

cc3 = is the core c state of the CPU. 

 

The core c state should be more important than the package c state, since the cores can use quite a bit of power.

The package power will not fluctuate a lot and is more difficult to get lower, because of all the different controllers involved.

 

Laptop power should be more efficient than a full ATX PSU. I am really interested in the power values you will measure.

Hello

 

N100DC-ITX with 12V 72W Leicke power brick:

Ethernet connected(reaching c3 state):

• Idle: Just board with unraid usb + nvme + -> 7.5W
• Idle: Just board with unraid usb + nvme + ASM1116-> 9W
• Idle: Just board with unraid usb + nvme + ASM1116 + Samsung Evo SSD (just connected) -> 9W

 

Ethernet disconnected(reaching c8 state)

• Idle: Just board with usb and nvme -> 5.5 W
• Idle: Just board with unraid usb + nvme + ASM1116-> 7W
• Idle: Just board with unraid usb + nvme + ASM1116 + Samsung Evo SSD> 7.1~7.8W
  • (ssd just connected trough jonsbo n2 sata board, and is recognized and available to unraid)

 

How have i get this data? First I have checked as suggested by @mgutt, I connected with HDMI in the 3 scenarios. I've checked C-States. When ethernet is diconnected. realtek 8111 aspm kernel bug is avoided, and reach c8 state. So, to measure without hdmi consumption I just repeat the process , disconnect hdmi, and see how power consumption goes down as normally does when c8 package is reached.

 

At this point, I haven't tested Ubuntu yet,but a few caveats:

ASPM is set enabled for ASM1166, with ssd sata connected. And c8 is achieved. Can we already say that asm1166 firmware is enough?

For last scenario, for example, this is the capture:

By the way,I never see c8 state going beyond >42% everytime I open powertop. As time pass by at this state, shouldn't it increase? Maybe powertop stops refreshing too.

 

Which power effiency approaches can be done for ssd sata? In my case I am going to create an unraid with all ssd.

 

Regards

 

 

 

[bagican]

Hello, I'm very new to Unraid.

Here is my first day with trial Unraid on Odroid H3:

 

 

Idle power consumption: ⚡️ 2.4—3.5W (on DC side)

 

Edited January 4 by bagican

[bagican]

  

Continuation from previous post 👆

 

What filesystem is most lightweight / efficient?
In my previous post I had array stopped.

When my array is started with one SSD with ZFS with compression then I have only C1-C2 State and idle power consumption is 4—5W.

 

 

//update (2023-01-02) when I unplugged cheap SATA SSD (128GB from AliExpress for 10 Eur) and replaced it with NVME Samsung 970 Evo then package CPU is able to reach C10 again and idle power consumption is back to ⚡️ 2.4—3.5W. With started array and with running 2 docker containers! wohooo! 🙂

 

Spoiler

 

here is with logged off and closed WebUI with 1 docker container running:

 

.

 

 

here is my powertop report: https://jsbin.com/hifepomaki/edit?output

 

[bagican]

Continuation from previous post 👆

 

Here is consumption from my Shelly 1PM:

 

So previously measurements 2.5—3 W on the DC side

corresponds to 3.6—3.7Wh according to Shelly 1PM (on the AC side).

Average power consumption for last 11 hours is 3.75 Wh ⚡.

Cool

 Consumption Time, Wh 

  05/01/2024 10:00 , 4.71 
  05/01/2024 11:00 , 3.7 
  05/01/2024 12:00 , 3.67 
  05/01/2024 13:00 , 3.62 
  05/01/2024 14:00 , 3.7 
  05/01/2024 15:00 , 3.6 
  05/01/2024 16:00 , 3.68 
  05/01/2024 17:00 , 3.66 
  05/01/2024 18:00 , 3.6 
  05/01/2024 19:00 , 3.67 
  05/01/2024 20:00 , 3.66 

=============================
          average = 3.75

 

with these running docker containers:

HW: Odroid H3 (Intel N5105, Turbo disabled)

RAM: 1x 16GB DDR4 Crucial 3200

SSD: 1x m.2 Samsung 970 Evo 500GB

Power supply: 12V 150W from AKASA (https://www.akasa.com.tw/update.php?tpl=product/product.detail.tpl&model=AK-PD150-02K)

+ into power consumption measurements is included also OLED DC power meter: https://www.aliexpress.com/item/33012984676.html

 

Edited January 6 by bagican

[_cjd_]

Thought I'd post a quick summary of my journey till now. I realized a while back that on a temperate low energy day (no heating or cooling, no dishwasher, no laundry) my network, security cameras, and unraid setup could push to a bit over 6kWh/day during parity check and an oversized percentage of my total daily consumption. That's now ~5.2kWh, with normal usage around 4.3kWh; not all Unraid tuning to get there (scheduling when certain APs are powered on also helped).

 

Build as it stands:

• ASRock Rack X570D4U
• AMD 5600x
• 32GB ECC 3200 DDR4 (single stick)
• 5 WD/Ultrastar HC550 16TB storage array (2x parity)
• 5x SATA SSD: Raid 0+1 cache array of 2 WD Red 2TB, 2 Crucial MX500 2TB,  and a Samsung 840EVO 500GB media (music) cache
• 1x WD Red SN700 M.2 nvme 1TB (docker containers and VMs)
• Intel x710 dual port SFP+
• 550W be quiet! Dark Power Pro Platinum
• ASM1166 m.2 > 6 SATA adapter

Attached USB peripheral: Rainforest EMU-2, Cyberpower UPS

 

VM usage is on demand and infrequent, most of the time VM support is disabled entirely.

 

When I started I had an HBA - swapping in the 1166 was roughly 12W drop, though at the time I only had NUT for monitoring power usage. Around that time I also had a few other things in flight prompting additional changes - a big one was needing to move my core data files to a network share as I have two separate systems now from which I might be working. This prompted the move from a single SSD pair to the Raid 0+1 array; this now serves both as a write cache for the main array as well as a dedicated file share for 10Gb networked systems... I can't saturate the network, but get close. It only added ~10% throughput for full system image backups.

 

Heat was becoming an issue, so I jumped from an old decently cooled desktop tower to a Rosewill RSV-L4500U, customized to run 3x140mm fans behind the drive cages in addition to the usual 3x120mm at the front of the cages, 2x80mm rear, 2x120mm cpu, 1x40mm chipset. SSD temps dropped ~14C during sustained write (was hitting 50C, now the WD Reds peak at 36C) and still only about 30% on the fan speeds. Chipset fan is an add-on but keeps overall temps down a decent amount - net is lower consumption than running the case fans to keep things cool enough.

 

BIOS settings have been run through multiple times, disabling the serial port etc, enabling power optimizations explicitly (vs auto), and CPU tuning.

 

At this point I added a Shelly Plug Plus (US) in front of just the server to have more fine-grained consumption detail.

 

I also have a Mellanox Connect-x 3 dual port SFP+ and tried both it and the Intel x710; probably a wash in my system, the x710 may possibly be 2W improvement but probably only half a watt; As I was testing this I discovered one of my fans was not spinning and I had quite a few troubles with the motherboard fan settings (they kept misbehaving when I'd switch cards and reboot). My best Mellanox low power reading was about half a watt higher than my best x710 low power reading. I ran out of energy to keep swapping and testing.

 

Fans play an observable role in power consumption - I've opted to run a bit more here for overall system longevity (but also lets me run lower fan speeds which seems to net out better anyway).

 

Lots of stuff defaults to "bad" when running powertop; testing through and getting it all set to "good" results in only about 1W difference. 92% C2 either way.

 

Idle is now 48-49W mean (43.x the lowest, 53.x typical highest I see, but it nets out to 49) with drives spun down (except the m.2), and I can shave another 4W or so stopping HomeAssistant (container not VM) - there has to be a bit more in the noise to shave off with the logging I have going to InfluxDB and the bits to wire up the logging, but stopping all the services didn't result in an observable change. The EMU-2 doesn't show an observable difference either. I haven't tried disabling the onboard i210 network ports as I use them (bonded for redundancy on my IoT network) - I could just run this instead as a vlan on one of the 10g ports so I may try that along the way.

 

I also have a backup server I built recently; an older consumer board with I believe 6th gen Intel i3, 2 Seagate Ironwolf pro 16TB, an m.2 WD Blue SATA 1TB, 2 80mm fans, 16gb ram (2 sticks), dual port Intel x540 10gbe card, 400w Seasonic platinum fanless PSU. It idles around 36W but is powered off most of the time. Complexity running a long DAC cable meant rj45 here (if it were on all the time I may have dealt with the effort), but I also wanted the 10Gb network - makes drive speed the limiting factor on backup operations, and less time powered on is lower power consumption overall (it's about half the time for an equivalent rsync vs 1gbe). 0.1W powered off (WoL trigger from the main server for those backups). It's tuned with everything disabled I can have disabled, powertop all green... 46% c7 is the best I've seen. Not curious enough yet to see how much that x540 is hurting, or the m.2 SATA (I had it sitting on a shelf).

 

For those counting, I estimate my security cameras and related bits (including a hard drive spun up 100% of the time) is about 50W, and the cable modem, gateway, switches, and APs account for another 50W.

 

I admit I'm curious whether/how much I could shave more off jumping to Intel, but can't justify the cost. The boards with the known really low numbers won't work for me as they all seem to be ITX with insufficient ports. I'd prefer ECC, so ASRock Rack W480D4u + Xeon could work. I'm really not sure that'd do much better, if at all. I know the power supply is also hurting me a little, but availability challenges vs cost and all that... given the backup server main board is already down an SATA port (nonfunctional) it may decide I have to put $ into parts sooner than later at which point I may well rotate the current setup over to the backup server and give Intel bits a try just for fun. More $ than needed just to have a backup system around....

 

I am curious if anyone else chasing efficiency has specific experience with hardware that includes ECC, ability to get at least 12 SATA ports (and I'd prefer 14 to give me a bit of wiggle room, but that's 6-8 onboard + a spot for an add on card), support for at least one m.2 NVME, and an x8 slot for the SFP+... though the RAID 0+1 is still a bit of a bottleneck so that may have to change to m.2 or u.2 meaning I'll need at least another x8...

 

Thanks to everyone that has contributed so far - quite an interesting read.

[dopeytree]

Your system sounds ok (not 200w+ etc). Could change the cctv to use an SSD?

 

Perhaps instead chase down your AP & switch energy although its similar to mine.

 

I got rid of my 2.5Gb switch and use a n100 box with 2.5Gb as pfsense direct to the unraid system then use cheap TPlink switches for all 1Gb stuff and they use less than 1w. Old 2.5Gb switches use alot of power.

 

Also changed my wifi AP's to energy efficient ones that are slightly older but work great and still provide faster than our 600Mb fibre link. These use barley any power 2-5w each & I have 3x wifi APs. I use these TP-Link TL-WA1201 https://www.amazon.co.uk/gp/product/B084BGKJZT/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&th=1

Edited January 7 by dopeytree

[_cjd_]

APs are 3.44, 4.23, 4.24, 4.56 watts as measured at the PoE port, within another ~4.03W which comes online evenings only, and one which usually runs ~8W but is only on when I need wifi in the back yard (garage/garage top deck really; nice days I like to work up there). They're tuned to be rather quiet and give me good performance in the city on 25x125 foot lots and most consumers just set to max power because they don't know any better. Even still, there are a couple IoT devices with just bad enough signal I can't do things like synchronize lights on different circuits...

Switches, I'm at 8/8 in a 10Gb SFP+ switch and 22 1Gbe with attached devices - though there are 3 PoE powered 1+4 switches I could add to timed downtime and shave another 4-5W overnight - that's worth doing and easy enough. CCTV HDD runs >50C so I'm hesitant to throw an SSD in there, especially because I'd want a minimum 8GB and that's $$$$. Some day - would be nice just for the seek performance. No 2.5g switches. Gateway router/network controller/firewall I might be able to find more efficient alternatives, maybe not. I'm not sure. I'll get around to chasing power consumption on some of the pieces but it's near impossible to find reviews that include power consumption... and that varies by use anyway.

 

Thanks for the thoughts and data points on network hardware.

[bagican]

Question about Idle stats in powertop: I consider to run one VM (Home Assistant). I suppose that running a VM will affect "Idle stats", right?

 

Currently I have ~ 60—70% in C10 Pkg state with couple of docker containers.

 

PowerTOP 2.15     Overview   Idle stats   

           Pkg(HW)  |            Core(HW) |            CPU(OS) 0
                    |                     | C0 active   2.7%
                    |                     | POLL        0.0%    0.0 ms
                    |                     | C1_ACPI     5.5%    0.4 ms
C2 (pc2)    5.9%    |                     | C2_ACPI     6.4%    1.0 ms
C3 (pc3)    2.3%    | C3 (cc3)    0.0%    | C3_ACPI    84.0%   16.8 ms
C6 (pc6)    0.0%    | C6 (cc6)   90.0%    |
C7 (pc7)    2.6%    | C7 (cc7)   89.6%    |
C8 (pc8)    0.4%    |                     |
C9 (pc9)    0.0%    |                     |
C10 (pc10) 64.3%    |                     |

 

Does running a linux VM will significantly affect Idle stats?❓

Thank you for an answer.

 

[dopeytree]

Docker containers are practically direct on metal so lower level (nearer) the hardware than VM's. This is why you get such good performance & why steam headless gives much greater performance while actually running the cpu cores lower than when compared to a windows VM.

 

 

To understand if your VM is affecting idle state you'd need to provide more detail like which cores are allocated to the VM & are any of those cores isolated to the VM. Bearing in mind home-assistant is pretty small on performance, I wouldn't expect it to affect much UNLESS you are running something like frigate or other addons in the VM.

Edited January 7 by dopeytree

[bagican]

I am familiar with the distinctions between Docker and virtual machines (VMs). However, as I have never actively utilized VMs, I am concerned about the potential resource consumption associated with VM technology.

 

However, based on your statement, it appears that running home-assistant as a lightweight application would have minimal impact on resources.

So virtual machine (VM) infrastructure would also have minimal resource impact, right? This is indeed wonderful news to me.

Thank you @dopeytree!

[mgutt]

5 minutes ago, bagican said:

So virtual machine (VM) infrastructure would also have minimal resource impact, right?

No, because the virtualization itself has an impact on idle load, even if the machine does nothing. I read about custom settings in the xml vm config files to reduce this load, but never tested them by myself. Here is an example:

 

 

But even Docker containers can have an impact on load / c-states. For example a Minecraft server kills all my c-states. Or some people reported loosing c-states after passing through a Zigbee USB Stick to a HA container, but this does not happen for me, so it is maybe depended on the used motherboard?!

[dopeytree]

Further down the rabbit hole.. this was my experience with raytracing gaming with a nvidia low power 75w graphics card rtx a2000.

 

Allocated 6/12 cores to the container & VM. Then tested 1 by 1.

 

- Steam-headless 55fps (container) = all cores 2.2Ghz

- WindowsVM 23fps (vm) = all cores 3.4Ghz

 

The myth that gaming needs the latest 5Ghz cpu is long overdue a visit from the grip reaper.

 

(Have since sold the card and use a steamdeck for gaming 10-20w)

 

Edited January 8 by dopeytree

[xreyuk]

I am hopefully after some help.

 

I've just setup a new unraid server with just the array bit with some data on it. I have no installed dockers/VMs and only a few plugins at this stage.

 

I have followed the guide in the first post however according to powertop I am not even entering c2 after doing all the tweaks. Powertop autotune command shows all the options as 'good', however on the dashboard there always seems to be a bit of load on the CPU.

 

Disks are fully spinning down and tips and tweaks is set as per the guide, as well as the BIOS settings I could find. The only 'weird' BIOS setting I had was enabling C-States C7/C8/C9 etc was 'Package C-State limit' which is set to Auto, as it wanted me to pick a specific C state setting for idle.

The only other setting was ‘native aspm’ which make you choose between BIOS or OS to control ASPM - I left this at BIOS, does that need changing? 

 

Spec is as follows:
 

Core i3-12100

16 GB Corsair Vengeance LPX

Gigabyte H610 DDR4 Mini ITX

Crucial P3 Plus M2 SSD 500GB (brand new - Cache)

2 x 4TB Seagate Ironwolf HDD (brand new - 1 Parity, 1 Array)

1 x 4TB Seagate Desktop HDD (unsure on model, used - Array)

1 x 120GB Crucial MX300 Sata SSD (used - Cache for plex only - but plex isn't installed yet)

 

Any ideas if I can figure out what's going on?

Edited January 8 by xreyuk

[Lolight]

1 hour ago, xreyuk said:

The only 'weird' BIOS setting I had was enabling C-States C7/C8/C9 etc was 'Package C-State limit' which is set to Auto, as it wanted me to pick a specific C state setting for idle.

In "C-States control" choose C10,

also do In "Package C-State limit"

 

Enable "Platform Power Management" and then configure the available ASPM entries.

Also enable "S3 Save mode", "ErP" and "RC6"

[DevanteWeary]

Guys sorry to ask but there seem to be a couple install methods in this thread now.

 

What's the current way to install this for the first time?

 

And should I install and run the auto-tune while a parity is going?

[Ruuddie]

Hi all. I currently have a Xeon 2640 v4 with 128GB RAM, Quadro P4000, 6 SATA disks, 2 NVME SSD's and 1 SATA SSD. It runs about 20 docker containers, a Home Assistant and a Kasm VM. It has a lot of spare headroom for me to fire up extra VM's and even ESXi as a VM, but that's for occasional use, they are not always on. Under normal load, I'm on ~30GB RAM usage. I use the P4000 for Plex transcoding because the Xeon doesn't have an iGPU.

 

I have applied all power saving scripts from this thread. With this, the system's idle usage with all disks spun down is around 70W.

I feel like this isn't too much for a big Xeon system, but still I'd be interested in saving power. Or well, I guess that's all about saving money down the line  So I could invest in newer hardware, but since I prefer to have at least 64GB RAM and preferably have 96/128GB, that would be quite expensive. Also, I don't want something like an N100, I want something that has some performance to run VM's now and then. Doesn't need to be a full 10+ cores setup, but I guess 6+ performance cores (so not the E-cores from the N) would work. Let's say i5 12400-ish? That would also be able to take over transcoding from the P4000, so I wouldn't need a dGPU anymore.

 

If I browse through this thread, I have a feeling that with my requirements will not be giving me a 10W idle system, but it'll go more towards 40W-ish idle. That'd be a savings of 30W, which is about 100 euro per year. So to get a couple hundreds of euros investment back, would take at least a couple of years. That barely feels like it's worth the effort, also because my system now can do everything I want, with plenty headroom performance-wise and it has a built-in remote KVM which is great.

 

What are your thoughts on this?

[xreyuk]

21 hours ago, Lolight said:

In "C-States control" choose C10,

also do In "Package C-State limit"

 

Enable "Platform Power Management" and then configure the available ASPM entries.

Also enable "S3 Save mode", "ErP" and "RC6"

Thanks, I'll give this a go and report back. It's not like the consumption isn't low on my current system, it's at 20w idle with disks spun down, but would like to get even lower if possible.

[Finnicely]

I'm pretty much stumped with my config. Maybe someone here can help me.

Specs:

Intel Core i3-12100

2x 8GB Kingston FURY Single Rank DDR4-2400 DIMM CL15

Gigabyte H610M S2H V2 DDR4 with newest BIOS

2x Crucial BX500 240GB (Cache)

1 x 16TB Seagate Exos X16 (Parity)

1 x 8TB WD Red (Array)

2x Xilence XPF80.W

 

Powered by a 200W picoPSU with a 150W Power Brick.

 

Only 2 devices are plugge in via USB. The Boot Drive and the UPS.

 

I activated RC6. ErP Mode and set the C-States to C10 as well as deactivating Turbo and a bunch of onboard devices i don't need (SerialPorts, GNA for example)

 

 I have no devices reporting that ASPM is disabled.

 

Powertop says everything is "good" yet i can't get lower than C3 on the Pkg(HW).

 

What else can i do? Any ideas?

 

[Lolight]

1 hour ago, Finnicely said:

What else can i do? Any ideas?

Welcome!

 

Try to enable "Race To Halt" and "Energy efficient turbo".

[xreyuk]

On 1/8/2024 at 2:31 PM, Lolight said:

In "C-States control" choose C10,

also do In "Package C-State limit"

 

Enable "Platform Power Management" and then configure the available ASPM entries.

Also enable "S3 Save mode", "ErP" and "RC6"

 

Thanks this has seemed to make it work. Idle has dropped from 20-23w to 11-13w, so a good power saving. Once I left it a bit longer after the image below I was dropping into c10 state.

 

 

My question is do you think this will affect plex transcoding? I'm using hardware transcoding to the Intel iGPU - will turning boost off have any effect on that?

 

I will see how server performance goes otherwise with these settings enabled. I'm running plex/sabnzbd/sonarr/radarr/pi-hole/unbound dockers so I think even in it's non turbo boosting I should be okay? 

 

 

Edited January 9 by xreyuk

[Lolight]

9 minutes ago, xreyuk said:

My question is do you think this will affect plex transcoding? I'm using hardware transcoding to the Intel iGPU - will turning boost off have any effect on that?

 

I will see how server performance goes otherwise with these settings enabled. I'm running plex/sabnzbd/sonarr/radarr/pi-hole/unbound dockers so I think even in it's non turbo boosting I should be okay?

My understanding is that the iGPU shouldn't be affected (I don't transcode).

Yeah, you should not have problems running those.

You might want to enable turbo boosting and see how it affects your C-state (it should not).

[xreyuk]

34 minutes ago, Lolight said:

My understanding is that the iGPU shouldn't be affected (I don't transcode).

Yeah, you should not have problems running those.

You might want to enable turbo boosting and see how it affects your C-state (it should not).

Thanks.

 

On the original guide it says to disable turbo boosting but if you think that won’t affect it much I’ll re-enable it.

 

I’m assuming that if I re-enable boosting in bios, I can just use the tips and tweaks plugin to disable/re-enable it? If I do this I’m assuming my normal governer should still be set to powersave? 
 

Finally can I just use the tune all command in my go file to run that at startup rather than using the individual commands?

 

thank you for all your help. 

Edited January 9 by xreyuk

[Lolight]

2 hours ago, xreyuk said:

On the original guide it says to disable turbo boosting but if you think that won’t affect it much I’ll re-enable it.

I've given that advice strictly from the try-n-see perspective and my understanding of the CPU power management and general BIOS options tuning.

I don't use the power saving settings in the plugin - can't say what it would do.

[xreyuk]

49 minutes ago, Lolight said:

I've given that advice strictly from the try-n-see perspective and my understanding of the CPU power management and general BIOS options tuning.

I don't use the power saving settings in the plugin - can't say what it would do.

 

Ahh okay no worries, thanks. 

 

From my 'extensive' testing today, turning off power boost and going into power saver mode saves about 15w under a load of sabnzbd downloading, doesn't really have an affect under idle.

 

However, disabling the 'boost' feature under my BIOS, for whatever reason saves me basically 50% under idle so I think I might run with this for the forseeable until I notice a performance impact.

 

Thanks.