Update.

Moved nvme-> sata asm1666 adapter from m.2 slot1 to slot2 to check power usage.
Max Package state C2 in slot2 is only C2.

Adapter moved back to m2 slot1 which previously was reaching C10 state.
It no longer reaches C10 in slot1

Max is C2
I'm totally lost why would this could happen

Edited December 14, 20241 yr by btTeddy

Is someone else also affected of not reaching anything higher than C2/C3 with Realtek Network-Cards since the last 2-5 Unraid updates?

On 12/14/2024 at 9:56 PM, btTeddy said:

Update.

Moved nvme-> sata asm1666 adapter from m.2 slot1 to slot2 to check power usage.
Max Package state C2 in slot2 is only C2.

Adapter moved back to m2 slot1 which previously was reaching C10 state.
It no longer reaches C10 in slot1

Max is C2
I'm totally lost why would this could happen

  

On 2/18/2024 at 12:10 PM, FlowDE said:

 

Probably the same issue like my Motherboard that the Intel Network-Card prevent any higher C-States.

I opend a bug Request at https://bugzilla.kernel.org/show_bug.cgi?id=218499

 

After almost three weeks of researching and playing, I was able to achieve pkg C10 with W680D4ID-2T/G5/X550 board (full story there: https://forum.level1techs.com/t/asrock-rack-w680d4id-2t-g5-x550-c-states-aspm-problem/221103), unfortunately only without one NVMe and dGPU installed.

 

As far as I understand C8 and C10 requires ASPM L1 substates (more particularity L1.1 or L1.2), but for some unknown reason CPU PCIe lines don't support it (at least Linux's lspci doesn't show those capabilities), but BIOS has L1 substate option enabled (though I cannot change it).

 

Is it a BIOS bug or modern Intel CPU doesn't support L1 substates on those lines?

 

UPD. I was able to enable L1 substates for CPU PCIe lines, but unfortunately Linux for some reason doesn't use them, moreover it limits the system by pkg C3 instead of C6.

Edited December 20, 20241 yr by lordprotector

Hi, i am new to all this and just installed powertop.

 

followed first post done all that is required, so am i correct in i am only getting C3 states?

 

my motherboard is: Asrock Z690 Extreme with an i5 12600K.

On 12/14/2024 at 8:46 AM, btTeddy said:

It's a victory!
Asrock Z790M-ITX WiFi + asm1166 (nvme -> 6x sata)
even with full stack of arrs containers running, managed to reach C10
 

Now I need to figure out which of many settings I changed helped out in here

 

Update 1
Qbittorrent once seeding a bit prevents reaching any package state.
Despite not reaching any package state above C2 overall power consumption is much lower.
Very much the same use scenario with JMB582 (connected to m.2 A+E wifi port) the whole system used 55W+
Swap to asm1166 lowered this straight away to average 45-47W with 4x exos spinning (parity down) and reading data + SSD for system.
Array started, all 5x exos spun down + SSD, Docker disabled = 21W Idle

Don't think this can be lowered much more due to i5-13500 and 2x 32GB DDR5.

Further update how to reach C10 on the mighty asrock later on

Chief,  I could really use your help here.  I have almost identical hardware (ASRock 760I ITX w/ wifi, 13500k and an NVME>6x Sata adapter.

I have enabled all ASPM items in BIOS, ran Powertop, auto-tune etc and simply can't get my rig into even C3 territory.

 

I have insured all items have ASPM enabled (lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM )').

 

Could you confirm your BIOS settings for me?  Any other suggestions or tips that you can remember?  I saw your setup and you've giving me hope.  Thank you for your help!

Could someone help me with why I have so many devices that claim capable of ASPM but Unraid is not enabling? I have used the pcie_aspm=force option and ensured that ASPM is enabled in the BIOS yet it does not enable.

 

 lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM )'

00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 0d) (prog-if 00 [Normal decode])

                LnkCap: Port #2, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <256ns, L1 <8us

                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+

00:1b.0 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #17 (rev f0) (prog-if 00 [Normal decode])

                LnkCap: Port #17, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <16us

                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+

00:1b.4 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #21 (rev f0) (prog-if 00 [Normal decode])

                LnkCap: Port #21, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <16us

                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-

00:1c.0 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #1 (rev f0) (prog-if 00 [Normal decode])

                LnkCap: Port #1, Speed 8GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <4us

                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, LnkDisable- CommClk-

00:1c.4 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #5 (rev f0) (prog-if 00 [Normal decode])

                LnkCap: Port #5, Speed 8GT/s, Width x2, ASPM L1, Exit Latency L1 <16us

                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+

00:1c.7 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #8 (rev f0) (prog-if 00 [Normal decode])

                LnkCap: Port #8, Speed 8GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <16us

                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+

00:1d.0 PCI bridge: Intel Corporation 200 Series PCH PCI Express Root Port #9 (rev f0) (prog-if 00 [Normal decode])

                LnkCap: Port #9, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <16us

                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+

01:00.0 VGA compatible controller: NVIDIA Corporation GP102 [GeForce GTX 1080 Ti] (rev a1) (prog-if 00 [VGA controller])

                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us

                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+

01:00.1 Audio device: NVIDIA Corporation GP102 HDMI Audio Controller (rev a1)

                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us

                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+

02:00.0 Non-Volatile memory controller: Seagate Technology PLC E18 PCIe SSD (rev 01) (prog-if 02 [NVM Express])

                LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us

                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+

03:00.0 PCI bridge: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA (rev ca) (prog-if 00 [Normal decode])

                LnkCap: Port #1, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <4us

                LnkCtl: ASPM Disabled; LnkDisable- CommClk-

04:00.0 PCI bridge: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA (rev ca) (prog-if 00 [Normal decode])

                LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <4us

                LnkCtl: ASPM Disabled; LnkDisable- CommClk-

04:08.0 PCI bridge: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA (rev ca) (prog-if 00 [Normal decode])

                LnkCap: Port #8, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <4us

                LnkCtl: ASPM Disabled; LnkDisable- CommClk-

04:09.0 PCI bridge: PLX Technology, Inc. PEX 8724 24-Lane, 6-Port PCI Express Gen 3 (8 GT/s) Switch, 19 x 19mm FCBGA (rev ca) (prog-if 00 [Normal decode])

                LnkCap: Port #9, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <4us

                LnkCtl: ASPM Disabled; LnkDisable- CommClk-

05:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)

                LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM not supported

                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-

07:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)

                LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM not supported

                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk-

pcilib: sysfs_read_vpd: read failed: No such device

09:00.0 USB controller: ASMedia Technology Inc. ASM2142/ASM3142 USB 3.1 Host Controller (prog-if 30 [XHCI])

                LnkCap: Port #0, Speed 8GT/s, Width x2, ASPM L0s L1, Exit Latency L0s <2us, L1 unlimited

                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+

0a:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 04)

                LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us

                LnkCtl: ASPM Disabled; RCB 64 bytes, LnkDisable- CommClk+

0b:00.0 Non-Volatile memory controller: Seagate Technology PLC E18 PCIe SSD (rev 01) (prog-if 02 [NVM Express])

                LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us

                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, LnkDisable- CommClk+

3 hours ago, nickp85 said:

Could someone help me with why I have so many devices that claim capable of ASPM but Unraid is not enabling?

Probably the drivers of the devices are not installed or do not support ASPM. I mean you are having a huge amount of devices like dGPU, HBA, PCIe Multiplexer, ASMedia USB... This setup will never be power efficient.

did all the things in op´s first post.

after BIOS check and powertop --auto-tune &>/dev/null my unraid consumes +7W
and yes its still +7W even after 1h and all disk are spin down.

 

plz tell me there is a correct way to undo this? because deleting powertop and undo the bios settings doesnt change anything...

53 minutes ago, joker1319 said:

because deleting powertop and undo the bios settings doesnt change anything...

Unraid does a fresh install on every reboot into the RAM. This means everything is lost on reboot. Only small changes are stored in config files on your USB flash drive like for example the network settings, but those are only known by Unraid or its plugins. Powertop does not change settings which survive a reboot or does even "know" about the config files on the USB flash drive. It wouldn't be even necessary to remove powertop. It's only a executable file which does nothing as long the user isn't using it. So your problem has a different cause. 

 

You should check what causes your load. Check the processes with for example "top".

Changed the motherboard to a C246M from Gigabyte and swapped out the HBA and expander to 2x ASM 6 ports. Spec is Xeon 2126G, 4x 16gb ECC, 15x 3.5" drives, 2x 2.5" drives, 3x 2.5" ssds, and 1 NVME running from an EVGA SFX 650 PSU.  Enabled all power save functions, left turbo on, ran powertop. Now reaches C8 on the PKG and the UPS reports 21 watts at idle which is about a quarter of yesterday. But, none of the SSDs will spin down anymore, so that is with the 4 running.

Looks like it might benefit from a different PSU. Is there an up to date list of PSUs with the low power mode? RM550x isn't available anywhere.

1 hour ago, Napoleon said:

Looks like it might benefit from a different PSU. Is there an up to date list of PSUs with the low power mode? RM550x isn't available anywhere

The Corsair RM750x (2024) consumes 1 watt more than the RM550x (2021) which is still good. And it has the nice feature to adjust a minimum fan speed (or semi passive as usual).

 

As of other users the Bequiet Pure Power 12M 550W consumes 1 watt more than the RM550x (2021), too.

 

Avoid the seasonic power supplies mentioned in this list as they have changed the electronics:

https://docs.google.com/spreadsheets/d/1TnPx1h-nUKgq3MFzwl-OOIsuX_JSIurIq3JkFZVMUas/edit#gid=110239702

 

Maybe the listed Cooler Master MWE400 is an option?! 

Can I just check, is THIS still the recommended way to install powertop?

 

mkdir /boot/extra
cd /boot/extra
wget https://github.com/mgutt/unraid-packages/raw/main/6.11.0/powertop-2.15-x86_64-1.txz

 

I’ve previously installed Powertop using Nerd Tools but see it’s deprecated in Unraid 7.0.0.

Edited January 10, 20251 yr by 1471

2 hours ago, 1471 said:

is THIS still the recommended

You can use the method you like. For 7.0.0 we probably need a new package. If this is available I can provide it through github and add it to the manual. Or maybe there will be an alternative to nerd tools in the future?!

Thanks @mgutt,

 

Looks like the latest Slackware package is:  powertop-2.15-i586-2

 

Which looks like the same version as you're using on your Github repo:  powertop-2.15-x86_64-1.txz

and the current version installed via Nerd Tools:  powertop-2.15-x86_64-1_nerdtools.txz

 

Looking at the Nerd Tools thread, looks like it might be dying for good this time.

Edited January 10, 20251 yr by 1471

1 hour ago, 1471 said:

Which looks like the same version

Sometimes an application needs to be rebuild if the newer OS version contains changes on library files, which are used by the application. So even if the application wasn't changed in the meantime, it could fail to start. So someone needs to test it.

On 1/10/2025 at 11:37 AM, mgutt said:

Sometimes an application needs to be rebuild if the newer OS version contains changes on library files, which are used by the application. So even if the application wasn't changed in the meantime, it could fail to start. So someone needs to test it.

Cool, i'll test when I update to 7.0 over the weekend and update this post with how it goes.

 

Edit:  yup, works fine using the latest slackware package.

 

Edited January 14, 20251 yr by 1471

I can confirm the package from Slackware current, as of today, seems to work fine on Unraid 7.0.0
 

https://mirrors.slackware.com/slackware/slackware64-current/slackware64/ap/powertop-2.15-x86_64-2.txz

 

I can get down to C8 like before.

 

As a side note, I also tested running powertop on a privileged Ubuntu container and it seems to work fine as well, adjusting everything on the host system

Edited January 10, 20251 yr by Jestern
Unraid version clarification

If I don't want to manually enter powertop --auto-tune &>/dev/null after a Boot/Restart of Unraid, do I just nano /boot/config/go and copy paste everything which is on the first page?

28 minutes ago, Ulairi said:

do I just nano /boot/config/go and copy paste everything which is on the first page?

You could use the list of commands or powertop auto tune and add it to the go file. I instead prefer the user scripts plugin and start the script on array start. By that I'm able to start the server without plugins (so without the script) if an OS Update causes any incompatibilities.

On 10/21/2020 at 1:05 PM, mgutt said:

# -------------------------------------------------
# Set power-efficient CPU governor
# -------------------------------------------------
/etc/rc.d/rc.cpufreq powersave

# -------------------------------------------------
# Disable CPU Turbo
# -------------------------------------------------
[[ -f /sys/devices/system/cpu/intel_pstate/no_turbo ]] && echo "1" > /sys/devices/system/cpu/intel_pstate/no_turbo
[[ -f /sys/devices/system/cpu/cpufreq/boost ]] && echo "0" > /sys/devices/system/cpu/cpufreq/boost

# -------------------------------------------------
# Enable power-efficient ethernet
# -------------------------------------------------

# enable IEEE 802.3az (Energy Efficient Ethernet): Could be incompatible to LACP bonds!
for i in /sys/class/net/eth?; do dev=$(basename $i); [[ $(echo $(ethtool --show-eee $dev 2> /dev/null) | grep -c "Supported EEE link modes: 1") -eq 1 ]] && ethtool --set-eee $dev eee on; done

# Disable wake on lan
for i in /sys/class/net/eth?; do ethtool -s  $(basename $i) wol d; done

# -------------------------------------------------
# powertop tweaks
# -------------------------------------------------

# Enable SATA link power management
echo med_power_with_dipm | tee /sys/class/scsi_host/host*/link_power_management_policy

# Runtime PM for I2C Adapter (i915 gmbus dpb)
echo auto | tee /sys/bus/i2c/devices/i2c-*/device/power/control

# Autosuspend for USB device
echo auto | tee /sys/bus/usb/devices/*/power/control

# Runtime PM for disk
echo auto | tee /sys/block/sd*/device/power/control

# Runtime PM for PCI devices
echo auto | tee /sys/bus/pci/devices/????:??:??.?/power/control

# Runtime PM for ATA devices
echo auto | tee /sys/bus/pci/devices/????:??:??.?/ata*/power/control

If I upgrade to 7.0, do I still need these lines in the GO file
or should/must I delete them first?!

28 minutes ago, luckyheiko said:

If I upgrade to 7.0, do I still need these lines in the GO file
or should/must I delete them first?!

you can comment it out/remove it if you want. (the powergovenor)

The setting is in "Settings -> PowerMode" now and does the same.

Edited January 11, 20251 yr by Mainfrezzer

15 hours ago, mgutt said:

You could use the list of commands or powertop auto tune and add it to the go file. I instead prefer the user scripts plugin and start the script on array start. By that I'm able to start the server without plugins (so without the script) if an OS Update causes any incompatibilities.

May I ask how that script look?

On 12/12/2024 at 12:59 PM, PlanetDyna said:

Package C State is used "Auto".

Auto is often "disabled". 

4 hours ago, Ulairi said:

May I ask how that script look?

It contains the commands? 😅