Reduce power consumption with powertop

[Alistar]

I'm back with some updates @mgutt, and thanks for your quick reply.
 

11 hours ago, mgutt said:

Two active disks cost you 10 to 16W. Why are they active?

Parity + Disk1. I'm kinda failing to troubleshoot what causes Parity drive to constantly spin up. Even if stopped manually. Possibly will create separate topic for that.
Disk1 keeps mainly the files that my family is using or content being read by qbit, so I can live with that one active.
 

11 hours ago, mgutt said:

Not the best for super low power consumptions, but at the moment not your main problem.

 

As per 80Plus Gold, should have something around 90% efficiency at loads below 20%. Platinum gives only +2%, rated 92%
 

11 hours ago, mgutt said:

Ask the biostar support. If their answer doesn't make you happy: Throw it in the trash.

Created support ticket, but no idea about their response times. Will see.
 

11 hours ago, mgutt said:

Disable array auto start, remove both components, spindown your HDDs, execute powertop and check if this influences your PKG states. If yes, find out, which of both is your problem.

Removed both components and enabled back motherboard 1gbe.
disabled array auto start, and used guide command to spin down all drives:


Executed auto tune and powertop (via ssh), to see if theres any difference with removed devices:
All Tunables are good, but it still sits at C3, even with cores doing C7 and frequency going down to 800mhz.



Main change is that only 2 devices are now listed as ASPM disabled/not supported.


btw. with removed devices, and all disk spun down, its at 23-24W.

Any hope for me, or just unlucky with mobo selection for going low with power consumption?

Edited January 29 by Alistar
formatting

[mgutt]

4 minutes ago, Alistar said:

As per 80Plus Gold, should have something around 90% efficiency at loads below 20%. Platinum gives only +2%, rated 92%

Read the first post. The rating levels are completely irrelevant for low power consumption efficiency.

 

6 minutes ago, Alistar said:

or just unlucky with mobo selection

At the moment it looks like a very bad board for low power consumption 😔  I think the main problem of this board is the ASM1064 SATA controller. As far as I know this controller is incompatible to ASPM, which means it stops the whole cpu package reaching a lower state than C3. 

 

 

[Alistar]

35 minutes ago, mgutt said:

Read the first post. The rating levels are completely irrelevant for low power consumption efficiency.

Right. I've seen it earlier, but forgot about this specific fragment.
 

37 minutes ago, mgutt said:

ASM1064 SATA controller. As far as I know this controller is incompatible to ASPM

 

Would you recommend any "confirmed" mATX motherboard or nvme? Considering I will use this config for at least few years, might be worth selling biostar + lexar (it has aspm not supported as well) and upgrade to something else (to be honest I do not expect any miracle from biostar support).
 

[FlowDE]

Its so annoying that the Intel i226-v 2.5gbe network card prevent any higher C-States...

[Alistar]

5 minutes ago, FlowDE said:

Its so annoying that the Intel i226-v 2.5gbe network card prevent any higher C-States...

I was not able to directly test it, as ASPM settings are not available on motherboard level.

One of the commands says it supports ASPM L1. Unless L1 is not sufficient and what really matters is only L0sL1 ?

[FlowDE]

Maybe in your BIOS configuration there are some option not enabled.

 

 

If i activate Auto Powercontrol the system gets unstable - If the Networkcable is pluged in i can reach C2/C3 - whitout Networkcable i can reach C8 whit Monitor + Keyboard but it looks like even my USB- Network card does the same and prevent any higher c-states

 

[Trainerbert]

Hey there,

 

I built a new server with this config:

 

Hardware:

Gigabyte B760M Gaming X DDR4

Intel i7-12500T

2x M2 Patriot P300 M.2 PCIe Gen 3x4 256 GB SSD (MirroredCache)

PNY CS900-2TB (VMs)

3x16 TB Toshiba Enterprise MG08ACA (Parity & Storage)

BeQuiet PURE POWER 12 M 750W

ASM1166 6xSATA Card

 

I set the Bios Settings like mgutt but I can't even reach C-State 3.

I stopped the VMs, Dockers,their services and the hdds connecting via ssh to watch with powertop.

 

 

The setup is running in this stage with 30w. I removed every storage except one m2 Patriot.

Still the same C2 but with 28W Idle. Then I removed the m2. Now I can get to C3 with 18W.

But how is this possible that even without anything except the usb flash I still can't go lower than C3?

 

EDIT: I removed all Plugins to be sure that nothing blocks this. But same result as before.

EDIT2: I did a bios update from F15 to F16 without a change. What is different to mgutt settings is that PEG PCH DMI ASPM are enabled but greyed out. So maybe a problem with the Intel 12th series like someone stated here.

Edited January 30 by Trainerbert
value changed

[PSchwarz]

I had the same Problem with the Gigabyte B760M. I downgraded unraid to 6.12.4 and with HDMI detached i reach C10. The power consumption the is around 20W, if the array is stopped.

[Trainerbert]

29 minutes ago, PSchwarz said:

I had the same Problem with the Gigabyte B760M. I downgraded unraid to 6.12.4 and with HDMI detached i reach C10. The power consumption the is around 20W, if the array is stopped.

You were right.  I downgraded to 6.12.4 and I am now reaching C8 with 10 Watt. Thanks!

 

 

But is it really the problem of this motherboard or with every motherboard at the current version?

Did someone ever reached C10 😄?

[PSchwarz]

For C10 you have to detach HDMI.
The problem is the realtek network interface. But it's a kernel driver, so it's not an unraid problem.....

 

[Trainerbert]

10 minutes ago, PSchwarz said:

For C10 you have to detach HDMI.
The problem is the realtek network interface. But it's a kernel driver, so it's not an unraid problem.....

 

I detached HDMI but didn't reach C10. Maybe I would have to wait longer than 3 minutes 😄. But I am pretty happy with C8.

Now I will put every drive little by little and check every time if I keep the c state. 

[Trainerbert]

So I tested now a lot of scenarios:

1. M2 first slot no c state at all.

2. M2 second slot C8.

3. M2 second slot and asm1166 card with ssd 2tb C10 - wtf

4. M2 second slot and asm1166 card with ssd 2tb and 3x16 hdd - C8

5. M2 second slot without asm1166card with ssd 2tb and 3x16 hdd - also C8

 

It looks like the asm1166 card isn't necessary for me. I am running now some dockers like plex, arrs and C8 is being held while using 15 watts. Thats impressive. I think I am going to use my 2tb ssd only for cache and vms and removing the m2s totally.

Edited January 30 by Trainerbert

[bagican]

On 1/5/2024 at 9:45 PM, bagican said:

Continuation from previous post 👆

 

Here is consumption from my Shelly 1PM:

 

So previously measurements 2.5—3 W on the DC side

corresponds to 3.6—3.7Wh according to Shelly 1PM (on the AC side).

Average power consumption for last 11 hours is 3.75 Wh ⚡.

Cool

 Consumption Time, Wh 

  05/01/2024 10:00 , 4.71 
  05/01/2024 11:00 , 3.7 
  05/01/2024 12:00 , 3.67 
  05/01/2024 13:00 , 3.62 
  05/01/2024 14:00 , 3.7 
  05/01/2024 15:00 , 3.6 
  05/01/2024 16:00 , 3.68 
  05/01/2024 17:00 , 3.66 
  05/01/2024 18:00 , 3.6 
  05/01/2024 19:00 , 3.67 
  05/01/2024 20:00 , 3.66 

=============================
          average = 3.75

 

with these running docker containers:

HW: Odroid H3 (Intel N5105, Turbo disabled)

RAM: 1x 16GB DDR4 Crucial 3200

SSD: 1x m.2 Samsung 970 Evo 500GB

Power supply: 12V 150W from AKASA (https://www.akasa.com.tw/update.php?tpl=product/product.detail.tpl&model=AK-PD150-02K)

+ into power consumption measurements is included also OLED DC power meter: https://www.aliexpress.com/item/33012984676.html

 

 

With better SSD a bit better power consumption.

I swapped Samsung 970 Evo to Samsung 980 Pro 2TB and idle power consumption (mostly in C10) is:

 

⚡️on the DC side:

 

⚡️on the AC side: ~2,8 Wh (from Shelly 1PM) is average consumption for last 2 hours.

 

It's known that Pro models from Samsung SSDs have better power management and they are more efficient.

Supported Power States
St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
 0 +     8.49W       -        -    0  0  0  0        0       0
 1 +     4.48W       -        -    1  1  1  1        0     200
 2 +     3.18W       -        -    2  2  2  2        0    1000
 3 -   0.0400W       -        -    3  3  3  3     2000    1200
 4 -   0.0050W       -        -    4  4  4  4      500    9500

 

command: sudo smartctl --all /dev/disk/by-partlabel/root

 

 - where Samsung 970 EVO Plus 500GB has higher numbers:

• Supported Power States
  St Op     Max   Active     Idle   RL RT WL WT  Ent_Lat  Ex_Lat
   0 +     7.80W       -        -    0  0  0  0        0       0
   1 +     6.00W       -        -    1  1  1  1        0       0
   2 +     3.40W       -        -    2  2  2  2        0       0
   3 -   0.0700W       -        -    3  3  3  3      210    1200
   4 -   0.0100W       -        -    4  4  4  4     2000    8000

 

Note that I also changed USB from old Patriot 16GB to new 16GB Transcend JetFlash 180I (industrial-grade USB flash drive with SLC mode https://us.transcend-info.com/embedded/product/embedded-flash-solutions/jetflash-180i  ).

 

 

 

Edited January 31 by bagican

[OddMagnet]

Hey guys, I've followed the guide, but I can't reach better than C3 PKG states. Hoping to get some help here

 

My System is:

CPU: i3-13100

MB: AsRock H610m-itx

RAM: Mushkin 2x32GB

SSD: Teamgroup MP33 Pro 2TB

I haven't installed HDDs yet, since I'm still waiting for my case, lol. (So I decided to dive into this topic while waiting xD)

Not sure if this matters, but I've not created the Cache Pool with my SSD either.

The iGPU driver however is installed.

 

Current Idle Draw: 15W (I assume/hope this can still go lower)

 

I've attached screenshots of my powertop idle stats, device stats and tunables, as well as the ASPM stats.

Device Stats looks suspicious to me, since some devices are permanently at 100%?

When checking the CPU frequency there are always some cores jumping to ~4GHz before going down to 800MHz again.

Unraid WebGUI was not open, I only used the terminal to access Unraid.

 

Additionally, here are my BIOS settings:

CPU Config:

• Hyper Threading: Enabled
• Active P-Cores: All
• CPU C States Support: Enabled
• Enhanced Halt State: Enabled
• CPU C6 & C7 Support: Enabled
• CPU Package C State Support: Enabled
• CFG Lock: Disabled (couldn't find any real info on what this does ¯\_(ツ)_/¯ )
• C6DRAM: Enabled
• AVX/AVX2: Enabled
• Intel Virtualization Technology: Enabled
• Hardware Prefetcher: Enabled
• Adjacent Cache Line Prefetch: Enabled

Chipset Config:

• Primary Graphics Adapter: Onboard
• Above 4G Decoding: Enabled
• Clever Access Memory: Enabled
• VI-d: Enabled
• SR-IOV Support: Enabled
• DMI Link Speed: Gen4
• PCIE1 Link Speed: Auto
• PCI Express Native Control: Enabled
• PCIE ASPM Support: L1
• PCH PCIE ASPM Support: L1
• DMI ASPM Support: Enabled
• PCH DMI ASPM Support: Enabled
• iGPU Multi Monitor: Disabled
• Intel® Ethernet Connection I219-V: Enabled
• Onboard HD Audio: Disabled
• Onboard WAN Device: Disabled
• Deep Sleep: Enabled in S4-5
• Restore on AC/Power Loss: Power Off
• Restore Onboard LED Default: Disabled
• RGB LED: Off
• GNA Device: Disabled

Storage Config:

• SATA Controller(s): Enabled
• SATA Mode Selection: AHCI
• Hybrid Storage Detection and Configuration Mode: Disabled
• SATA Aggressive Link Power Management: Enabled
• Hard Disk S.M.A.R.T: Enabled

ACPI Config:

• Suspend to RAM: Auto (other choice was disabled)
• PS/2 Keyboard Wake Support: Disabled
• PCIE Devices Power On: Disabled
• I219 LAN Power On: Disabled
• RTC Alarm Power On: By OS
• USB Keyboard/Mouse Power On: Disabled

USB Config

• Legacy USB Support: Enabled
• XHCI Hand-off: Disabled

 

Lastly: I found this post on askubuntu.com, which was about the my exact MB, is that something I should look into? Don't want to go around installing Windows on my SSD and doing some potential dangerous stuff before asking here first

 

Edited January 31 by OddMagnet

[dopeytree]

High friends. Just curious who's got the lowest wattage for an active system running frigate NVR & plex and whats your build.

Edited January 31 by dopeytree

[mgutt]

5 hours ago, OddMagnet said:

Lastly: I found this post on askubuntu.com, which was about the my exact MB, is that something I should look into?

Another ASRock owner did the same in this thread and this was his only option:

https://forums.unraid.net/topic/98070-reduce-power-consumption-with-powertop/page/26/#comment-1355095

 

 

5 hours ago, OddMagnet said:

Teamgroup MP33 Pro 2TB

Maybe this SSD prevents lower C-States. Check without it.

 

 

[OddMagnet]

4 hours ago, mgutt said:

Another ASRock owner did the same in this thread and this was his only option:

https://forums.unraid.net/topic/98070-reduce-power-consumption-with-powertop/page/26/#comment-1355095

Gonna try a couple more things in BIOS before giving this a try, will report back after that.

 

4 hours ago, mgutt said:

Maybe this SSD prevents lower C-States. Check without it.

Checked without it, sadly not the cause of my problems. (Or luckily I guess, since I don't need to send it back, lol)

[vmasip]

Please, please, can anyone give some light to this issue?????:

 

[OddMagnet]

Hey @genesisdoeswhatnintendont, thanks so much for your guide, I followed it and am able to reach C10 now (mostly) and dropped down to 10W Idle.

I have a couple of questions, since you have the exact same Mainboard that I do. Not sure if you're still around since your post was a year ago, but here's to hoping  (not sure why I thought that, lol)

 

On 1/17/2024 at 7:42 PM, genesisdoeswhatnintendont said:

I also changed this settings because I wanted to

 

.\SCEWIN_64.exe /i /lang 'en-US' /ms "LED MCU" /qv 0x0 /hb
.\SCEWIN_64.exe /i /lang 'en-US' /ms "Native ASPM" /qv 0x0 /cpwd YOUR-BIOS-ADMIN-PASSWORD /hb
.\SCEWIN_64.exe /i /lang 'en-US' /ms "Discrete Bluetooth Interface" /qv 0x0 /cpwd YOUR-BIOS-ADMIN-PASSWORD /hb
.\SCEWIN_64.exe /i /lang 'en-US' /ms "UnderVolt Protection" /qv 0x0 /hb
.\SCEWIN_64.exe /i /lang 'en-US' /ms "Password protection of Runtime Variables" /qv 0x0 /cpwd YOUR-BIOS-ADMIN-PASSWORD /hb

Could you explain what those do exactly? I looked at the `setup_script_file`, but it's not quite clear to me.

It looks like "LED MCU" disabled USB devices, so that's a no-no for me since I'm using Unraid I guess?

"Native ASPM" disabled means that the BIOS handles ASPM, correct? If so that should be good for me as well? (already enabled it)

"Discrete Bluetooth Interface" not sure if this applies to the AsRock H610m-itx. Does it use Bluetooth over the Wifi Antennas? Would that also disable Wifi? 

"Password protection of Runtime Variables" just means I could use that tool without having to set an admin password?

 

@genesisdoeswhatnintendont, could I compare my BIOS settings with yours?

Currently I'm still getting about 28% C2, which I'd like to eliminate as well if possible

 

Edited February 1 by OddMagnet

[tmodev]

3 hours ago, OddMagnet said:

It looks like "LED MCU" disabled USB devices, so that's a no-no for me since I'm using Unraid I guess?

"Native ASPM" disabled means that the BIOS handles ASPM, correct? If so that should be good for me as well? (already enabled it)

 

I am also trying to apply @genesisdoeswhatnintendont guide to the asrock z790 pro rs d4. I disabled it and my USBs are still working. I think its some sort of RGB LED Controller maybe?

 

In my case i still cant reach deeper C-States than C3 eventhough I enabled ASPM for all my devices. Enabled C-States. Deactivated every unnecessery device and yet i am only able to reach c3. "Native ASPM" is disabled. Any suggestions what else i could try to tune?

 

Also does anybody know why i get an error when running lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM )'

could this be the culprit?

EDIT: I am also using a PiKVM which is plugged in via HDMI.

 

Storage:

3x 1Tb Samsung 970 Evo Plus (can do c10)

8x 16Tb Toshiba MG08 (can do c8 atleast)

 

Throttlestop on Windows:

Edited February 1 by tmodev

[genesisdoeswhatnintendont]

20 hours ago, OddMagnet said:

Lastly: I found this post on askubuntu.com, which was about the my exact MB, is that something I should look into? Don't want to go around installing Windows on my SSD and doing some potential dangerous stuff before asking here first

 

oh boy 😁 I even added link on askubuntu to this thread bc I quoted @mgutt in the comments.

Btw there is a linux utility similar to AMISCE, but sadly it's not maintained anymore. It's called amifldrv

And I believe  there is a linux version of AMISCE but it's pretty hard to find if you are not a motherboard manufacturer.

 

Quote

It looks like "LED MCU" disabled USB devices, so that's a no-no for me since I'm using Unraid I guess?

 

It's for annoying RGB LED, as @tmodev said below, just skip it bc it doesn't work. There is another workaround:

 

lsusb

output is: Bus 001 Device 005: ID 26ce:01a2 ASRock LED Controller

 

create new rule in /etc/udev/rules.d

 

/etc/udev/rules.d/01-asrock-led.rules

 

with these lines:

 

# Disable Asrock LED controller

SUBSYSTEM=="usb", ATTRS{idVendor}=="26ce", ATTRS{idProduct}=="01a2", ATTR{authorized}="0"

 

Reboot

 

Quote

"Native ASPM" disabled means that the BIOS handles ASPM, correct? If so that should be good for me as well? (already enabled it)

 

You are right

 

Quote

"Discrete Bluetooth Interface" not sure if this applies to the AsRock H610m-itx. Does it use Bluetooth over the Wifi Antennas? Would that also disable Wifi? 

 

Just skip it. In my case I don't need neither BT nor Wifi (but there is another settings for Wifi anyways)

 

Quote

"Password protection of Runtime Variables" just means I could use that tool without having to set an admin password?

 

Correct. And I answered this question on askubuntu below in the comments :)

 

Quote

Currently I'm still getting about 28% C2, which I'd like to eliminate as well if possible

 

My BIOS settings are just the same. In my case I have no USB devices and display turned off (consoleblank=130 kernel boot parameter)

Edited February 1 by genesisdoeswhatnintendont

[KarlMeyer]

2 hours ago, tmodev said:

I am also trying to apply @genesisdoeswhatnintendont guide to the asrock z790 pro rs d4. I disabled it and my USBs are still working. I think its some sort of RGB LED Controller maybe?

 

In my case i still cant reach deeper C-States than C3 eventhough I enabled ASPM for all my devices. Enabled C-States. Deactivated every unnecessery device and yet i am only able to reach c3. "Native ASPM" is disabled. Any suggestions what else i could try to tune?

 

Also does anybody know why i get an error when running lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM )'

could this be the culprit?

EDIT: I am also using a PiKVM which is plugged in via HDMI.

 

Storage:

3x 1Tb Samsung 970 Evo Plus (can do c10)

8x 16Tb Toshiba MG08 (can do c8 atleast)

 

Throttlestop on Windows:

Same board, same problem. It's got to be something with the hardware, either the sata controller or the RGB controller would be my guess. It's almost like turning off the RGB controller in bios doesn't actually turn it off, as if it's off but still getting power. I don't know. It would literally be the best Z790 board if you could easily enable C10. Maybe a new bios version will fix it? I've done just about everything I can at this point.

[genesisdoeswhatnintendont]

3 hours ago, tmodev said:

I enabled ASPM for all my devices. Enabled C-States. Deactivated every unnecessery device and yet i am only able to reach c3. "Native ASPM" is disabled. Any suggestions what else i could try to tune?

 

 My guess is Realtek 2.5G LAN is the culprit.  God I hate Realtek so much! 😡

My 2 cents: ATX motherboard (especially with Realtek NIC) and with Z-chipset is not the best choise for low power consumption setup.

 

@KarlMeyer i've had the same issue with RGB LED, just try solution from my previous post.

Edited February 1 by genesisdoeswhatnintendont

[tmodev]

1 hour ago, genesisdoeswhatnintendont said:

 My guess is Realtek 2.5G LAN is the culprit.  God I hate Realtek so much! 😡

My 2 cents: ATX motherboard (especially with Realtek NIC) and with Z-chipset is not the best choise for low power consumption setup.

I guess so as well. However, im idling at 65w with 8 hdds and 3 nvmes and 20+ container running which is still good i suppose. and yet id love to know how deep i could go. 

my proxmox build idles between 20 and 25 watt (5 ssds + 4 hdds)  and reaches c8. Just for science id love to know how big the impact is.

 

Ill try your suggestions regarding the RGB LED controller and report back. Maybe it helps somehow

 

EDIT: just checked with lsusb. It looks like i dont have a RGB Controller (or its deactivated already).

Edited February 1 by tmodev

[FlowDE]

I got a response from Asus for the  Intel i226-v 2.5gbe Networkchip. They could reach deeper C-States like C10 so they assume that the driver inside Linux (Unraid) could be the issue. Does some one know how to proceed?