Reduce power consumption with powertop

[nassauer]

25 minutes ago, mgutt said:

Not sure about the 10W, but I think saving 7W should be realistic. Can you find your specific power supply model in a review of tweakpc.de ? They test the efficiency at 10W load. 

 

PS it seems you tested your c state with the unraid web terminal. The webgui produces load, so you should close all browsers and connect through SSH to your server.

You where correct. I´ve switched to putty and now I get C10. I attribute about 2-3 Watts to my APC I have. I´ve found test data for the PSU and it looks like it is actually pretty good.

But given that literally nothing else is connected to the board itself the idle draw still seems high. I am measuring with a Tapo plug via the app and my APC can display power draw.

[mgutt]

1 hour ago, nassauer said:

I´ve found test data for the PSU and it looks like it is actually pretty good.

This test data is useless. Power supplies are much less efficient on low power consumption. This means a Titanium PSU can consume 20W at 10W load, while a Bronze PSU consumes only 13W at 10W load. Depends fully on the individual model. As an example this specific model is very efficient at low loads:

 

 

 

[nassauer]

53 minutes ago, mgutt said:

This test data is useless. Power supplies are much less efficient on low power consumption. This means a Titanium PSU can consume 20W at 10W load, while a Bronze PSU consumes only 13W at 10W load. Depends fully on the individual model. As an example this specific model is very efficient at low loads:

alright, got it. I´ll be looking out for a new PSU. Could probably source a rm550x.

 

Meanwhile I´m testing with an nvme in the slot, but that leads to c2 state only. Haven´t found something that works and enables ASPM for it.

 

 

 

 

[mgutt]

1 hour ago, nassauer said:

with an nvme

Which one exactly?

[nassauer]

36 minutes ago, mgutt said:

Which one exactly?

It´s a Kingston NV2 500G I had left over.

[DataCollector]

1 hour ago, nassauer said:

Meanwhile I´m testing with an nvme in the slot, but that leads to c2 state only. Haven´t found something that works and enables ASPM for it.

 

Just for Info:

I did also do a lot of testing with several NVMe SSD on my B760M:

 

If german language is a problem here the short of it in english:

naked Board (with CPU and Ram unraid 6.12.4): under 10W (around 9W)
M.2 NVMe/PCIe SSD AData Legend 960 4TB PCIe 4.0 x4: again idle under 10W
M.2 NVMe/PCIe SSD Patriot Viper VP4300 Lite 4TB PCIe 4.0 x4: again idle under 10W
M.2 NVMe/PCIe SSD Samsung SSD 960 PRO 512GB,PCIe 3.0 x4: agein. under 10W
M.2 NVMe/PCIe SSD Lexar NM790 4TB PCIe 4.0 x4: what a surprise: again under 10W idle

 

So it depends also on the specific NVMe if it confuses the Mainboard(Firmware/UEFO/BIOS) and keeps the CPU partally awake.

[mgutt]

46 minutes ago, nassauer said:

Kingston NV

Hmm ok, never read about ASPM problems with this model, but it can happen. Some boards even act differently if you use the CPU M.2 slot or chipset M.2 slot. A pcie m.2 adapter card allows for example to use it in a pcie slot.

[nassauer]

1 hour ago, mgutt said:

Hmm ok, never read about ASPM problems with this model, but it can happen. Some boards even act differently if you use the CPU M.2 slot or chipset M.2 slot. A pcie m.2 adapter card allows for example to use it in a pcie slot.

well it´s ok for me not to use the nvme at the moment and save myself a little headache figuring it out. I could get my hands on a 2018 rm550x PSU. Any differences to the 2021 model?

[mgutt]

2 minutes ago, nassauer said:

Any differences to the 2021 model?

Yes, much worse. Not comparable. This is the reason why you can't get the 2021, but still the 2018 version 😉

[nassauer]

4 minutes ago, mgutt said:

Yes, much worse. Not comparable. This is the reason why you can't get the 2021, but still the 2018 version 😉

Thanks for the warning. Well then I´ll keep searching.

[Arno]

Hi All
Looking to build new system around 13500 CPU

What is the consensus on motherboard for easiest power management?

[nickp85]

I started using powertop and added the auto tune to my go file however all of my percent on the left of the idle stats page are 0.0%. The tweaks page shows everything as good.

 

I have an ASUS PRIME Z-370a motherboard with a 9900K CPU, 8 spinning disks on a SAS to SATA controller card, two NVME for cache and one standalone SATA SSD for passthrough VM. I also have an nvidia 1080 Ti for passthrough to VM.

Turned on all the ASPM options in motherboard BIOS and I don't usually run the VM unless I need to. GPU, a USB controller, and one extra NIC is marked as passed through to VM in Unraid.

 

How can I get my idle stats above 0%?

[mgutt]

4 hours ago, nickp85 said:

How can I get my idle stats above 0%?

Hard task with GPU and SAS card. Read the first post.

[Stubbs]

Does running powertop autotune interrupt with Wake-on-LAN?

 

I've enabled WOL + S3 in my BIOS, have the Unraid S3 Sleep Plugin, and wol is set to g on my eth0's interface(which is my server's network card). Even with all that, I still can't wake it up with a magic packet. 

[crushjz]

I'm new to Unraid but I have quite a bit of experience with Linux in general and like most of you, I'm trying to build a NAS/Homelab that has to be as power efficient as possible 🔋. 

I have the following components:
- Gigabyte B760M DS3H AX DDR4
- Intel i5-12400
- 16GB RAM
- Corsair CX550

With the above configuration, using a cheap Kingston SSD, Ubuntu Server 24.04 with the amazing `powertop`, I'm able to reach Package C10 and a power draw at the wall of just 8-10w. I followed most of the advices in this blog post, specifically for the internal Realtek RTL8125 2.5GbE Controller which requires a different driver than the one shipped with Ubuntu.

Using Unraid tho, the power draw jumps to 15/16w after  `powertop --auto-tune` and the Package can only reach C3. From an initial investigation looks like that the "xHCI Host Controller" (The USB controller) is responsible for that and the Unraid USB Flash Drive  (usb-device-058f-6387 in the screenshot below, which is the one where the Unraid OS is) prevents the controller to be suspended:

As you can see it's the only difference from the Ubuntu Server setup:

Does someone have an advice on how can I force the USB to go to sleep? 🙏

Edited June 23 by crushjz

[obowan]

On 6/22/2024 at 1:10 PM, crushjz said:

I'm new to Unraid but I have quite a bit of experience with Linux in general and like most of you, I'm trying to build a NAS/Homelab that has to be as power efficient as possible 🔋. 

I have the following components:
- Gigabyte B760M DS3H AX DDR4
- Intel i5-12400
- 16GB RAM
- Corsair CX550

With the above configuration, using a cheap Kingston SSD, Ubuntu Server 24.04 with the amazing `powertop`, I'm able to reach Package C10 and a power draw at the wall of just 8-10w. I followed most of the advices in this blog post, specifically for the internal Realtek RTL8125 2.5GbE Controller which requires a different driver than the one shipped with Ubuntu.

Using Unraid tho, the power draw jumps to 15/16w after  `powertop --auto-tune` and the Package can only reach C3. From an initial investigation looks like that the "xHCI Host Controller" (The USB controller) is responsible for that and the Unraid USB Flash Drive  (usb-device-058f-6387 in the screenshot below, which is the one where the Unraid OS is) prevents the controller to be suspended:

As you can see it's the only difference from the Ubuntu Server setup:

Does someone have an advice on how can I force the USB to go to sleep? 🙏

@crushjzWhat do you see when you run spci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM )' ? I have the same NIC as you and in Unraid the stock driver does not support ASPM. If you install the R8125 plugin, you can reach higher C states but for me at least, my speed becomes drastically reduced in my docker containers. One of my PCIE devices states "ASPM disabled" until I load the Realtek plugin, so that might be part of what you're seeing also. Hope this helps!

Edited July 16 by obowan

[Andrea P.]

On 10/21/2020 at 1:05 PM, mgutt said:

wget https://github.com/mgutt/unraid-

As a total noob I opened the terminal, launched the first two commands, then the quoted one. I got a github 404 error, not found.

 

[ChatNoir]

12 hours ago, Andrea P. said:

As a total noob I opened the terminal, launched the first two commands, then the quoted one. I got a github 404 error, not found.

The command is wget https://github.com/mgutt/unraid-packages/raw/main/6.11.0/powertop-2.15-x86_64-1.txz and works fine. Just tested it. Maybe you added a space after the dash ?

[xTazy]

On 1/8/2024 at 10:22 AM, mgutt said:

No, because the virtualization itself has an impact on idle load, even if the machine does nothing. I read about custom settings in the xml vm config files to reduce this load, but never tested them by myself. Here is an example:

 

 

But even Docker containers can have an impact on load / c-states. For example a Minecraft server kills all my c-states. Or some people reported loosing c-states after passing through a Zigbee USB Stick to a HA container, but this does not happen for me, so it is maybe depended on the used motherboard?!

I'm fighting exactly with the same issue (passing through a Zigbee USB Stick to a HA VM). I'm able to rich C10 state without passing it to VM (~9W without VM and ~13W with VM but without stick). I'm find out that switching VM USB controller to 3.0 Qemu XHCI allow me to rich C2 ~30% and reduce power consumption for 2W. Maybe do you hear any other setting for such case ? 

[Andrea P.]

On 7/17/2024 at 7:52 AM, ChatNoir said:

Maybe you added a space after the dash ?

I retried again, this time it worked. Not sure about extra spaces, probably not, 'cause I was simply copy/pasting, however, thanks a lot for the attention. 💜

[methanoid]

@mgutt  thank you for this guide.  Inevitably I have questions

 

1) Is there an AMD Ryzen version of the Disable CPU Turbo?
 

[[ -f /sys/devices/system/cpu/intel_pstate/no_turbo ]] && echo "1" > /sys/devices/system/cpu/intel_pstate/no_turbo
[[ -f /sys/devices/system/cpu/cpufreq/boost ]] && echo "0" > /sys/devices/system/cpu/cpufreq/boost

 

2) Autosuspend for USB device - Is this advisable when the unRAID USB stick is on the bus? I think I have to disable this anyway as I pass through most of my USB controllers for VMs

 

echo auto | tee /sys/bus/usb/devices/*/power/control

 

[Sandbank7]

I have spent the past weeks trying to optimize the power consumption of my server, and I have finally had success.

 

I need to preface this, I am not using UnRaid, but I have been lurking in this thread for a long time, and wanted to share my results with the community. I am using Debian, so I believe it should be transferable.

 

Relevant components:

Intel i5 13500

ASRock B760 Pro

LSI 9400-16i

A bunch of HDDs

NVME drive

Realtek 2.5G NIC

 

What I did

I will do my best to remember everything, as there has been a lot of trial and error, so don't consider this a guide, but rather a collection of things I had to do to finally solve this.

 

• Enable every power saving feature you can in the bios, ensure all the C-states are enabled, intel speedstep, etc.
• Enable ASPM in bios (L1 is max power saving, L0 least).
• For my case, I had to edit my BIOS using SCEWIN. Using this tool, I was able to disable Multi-VC on all root ports (see here) and also enable further power saving functionality relevant only to ASRock (see here)
• Avoid the use of any PCI-E directly connected to the CPU. This means using the NVME slot connected to the chipset, as well as the PCI-E x4 slot for my HBA (not the top x16 slot). This does limit the overall bandwidth of the HBA to 4GB/s instead of 8GB/s, but is still enough for spinning rust.
• Ensure ASPM is supported by your PCI-E devices. The LSI 9400 does support ASPM L1.
• For me, ASPM was disabled out of the gate when I booted my system. I used the following script to forcefully enable ASPM on the ports where it was not enabled. I had to enable it for both my LSI card AND the root ports themselves (as both were disabled when the LSI card was used). What this means is that the root complex and endpoint should be the same when editing the ASPM level of the port, while you change endpoint to target your LSI card as necessary.
• I am not using the R8125 driver, as that is not working with the latest kernel. Instead I use the default R8169 driver, but explicitly enabled ASPM by using the following line:

  echo 1 > /sys/bus/pci/devices/0000:04:00.0/link/l1_aspm

• Run powertop (see below) or any equivalent script to set the basic power saving features.

  powertop --quiet --auto-tune

• Additionally, you can save a little bit of power through some additional commands:

  echo "powersupersave" > /sys/module/pcie_aspm/parameters/policy;
  
  # See https://wiki.archlinux.org/title/CPU_frequency_scaling for details
  cpupower frequency-set -g powersave
  cpupower idle-set -E
  cpupower set -b 15
  
  echo "balance_power" | tee /sys/devices/system/cpu/cpu*/cpufreq/energy_performance_preference

• Disabled onboard sata controller (everything goes through the HBA)

 

All of this has lead to the final culmination of my system finally hitting C10 pkg states.

 

Ensure that whatever you plug into your PCI-E ports support ASPM. Older LSI cards typically do not support ASPM, which means you're out of luck.

 

The following guide helped me a lot in debugging and gaining a broader understanding into this whole mess. If you don't want to read the whole thing, at least use the following commands to check if your LSI card does support ASPM:

> sudo apt-get install acpica-tools
> sudo acpidump -b -n FACP
> iasl -d facp.dat 

Then look for the line that says PCIe ASPM Not Supported (V4). If that says 1, you're out of luck. If it says 0, you're good to go.

 

I hope this helps somebody out there.

Edited August 26 by Sandbank7

[SnugglyDino]

Looking to get an HL15 for my next chassis and need to figure out the best way to add HDD for my novice setup. My motherboard is a MSI MAG B660 Tomahawk with a 12th Gen Intel i5. I'm not sure if I should be getting a HBA card or some sort of PCIe to Sata expander. Right now this is what I'm eyeing up on amazon...

 

1.) PCIe x1 to 12-Port SATA3.0 Expansion Card,6Gbps SATA Adapter for SSD/HDD, Plug and Play with 12 Status LEDs, Compatible with Windows/Linux/Ubuntu/ESXi/Synology NAS

2.) PCIe Sata Expansion Card M.2 to SATA3.0 Adapter Card, 6Gbps High Speed Sata Expansion Card, ASM1166 M.2 PCIE to SATA Expansion Card with Smart Indicator, M.2 Sata Adapter

3.) LSI Logic Controller Card 05-25600-00 9300-16i 16Port 12Gb/s SAS x8 PCI-Express 3 Full Height Brown Box

 

Which option do you recommend I pick or just tell me while all these options are wrong. I'm mainly looking for low power consumption and being able to connect all 15 hard drive slots on the HL15. Speed is 3rd. 

[Sandbank7]

On 8/27/2024 at 2:30 AM, SnugglyDino said:

Looking to get an HL15 for my next chassis and need to figure out the best way to add HDD for my novice setup. My motherboard is a MSI MAG B660 Tomahawk with a 12th Gen Intel i5. I'm not sure if I should be getting a HBA card or some sort of PCIe to Sata expander. Right now this is what I'm eyeing up on amazon...

 

1.) PCIe x1 to 12-Port SATA3.0 Expansion Card,6Gbps SATA Adapter for SSD/HDD, Plug and Play with 12 Status LEDs, Compatible with Windows/Linux/Ubuntu/ESXi/Synology NAS

2.) PCIe Sata Expansion Card M.2 to SATA3.0 Adapter Card, 6Gbps High Speed Sata Expansion Card, ASM1166 M.2 PCIE to SATA Expansion Card with Smart Indicator, M.2 Sata Adapter

3.) LSI Logic Controller Card 05-25600-00 9300-16i 16Port 12Gb/s SAS x8 PCI-Express 3 Full Height Brown Box

 

Which option do you recommend I pick or just tell me while all these options are wrong. I'm mainly looking for low power consumption and being able to connect all 15 hard drive slots on the HL15. Speed is 3rd. 

Read my post above, get any card that satisfies your constraints and supports ASPM. Option 1 seems terrible, option 2 eats up a valuable NVME slot, option 3 uses a card that runs crazy hot. I'd go LSI 9400-16i.

[bagican]

@Sandbank7 did you measure real power consumption? I'm just wondering how many watts it was before and after?