20 hours ago, bigup said:

 

My server runs i5-11500 and a ASRock H570 Steel Legend motherboard and Unraid 6.12.8

 

Its running 12x HDD, 1 WD RED nvme cache for appdata, and 2xSSD in RAID0 for downloads cache.

 

I use 2x ASM1166 SATA cards as well as onboard SATA for the drives

 

PSU is a brand new Corsair 650W (2021)

 

Network card is an Intel i226 PCI 2.5Gbe 1 port.

 

Onboard card is Realtek 2.5gbe but have disabled it and all other rubbish like RGB, audio etc all disabled.

 

BIOS has all Power states options enabled and ASPM enabled (L0L1)

 

When i run lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM )' everything is APSM enabled.

 

Powertop autotune ran, kernel in powersave mode and intel turbo is off (using tips and tweaks plugin)

 

No hdmi cable fitted and no unraid GUI left on in any browser.

 

Unraid flash drive is running off an internal USB header

 

Right now the server is running at 27W/28W at idle with all drives in standby apart from nvme which is always on due to dockers and powertop shows it's only in C3, ive yet to see it go to C6.

 

Am i missing anything else? whats left to check? or can it just be the cpu/motherboard combo? or something else?

 

Probably the same issue like my Motherboard that the Intel Network-Card prevent any higher C-States.

I opend a bug Request at https://bugzilla.kernel.org/show_bug.cgi?id=218499

3 hours ago, FlowDE said:

 

Probably the same issue like my Motherboard that the Intel Network-Card prevent any higher C-States.

I opend a bug Request at https://bugzilla.kernel.org/show_bug.cgi?id=218499

 

An update, so I had 2x ASM1166 Sata cards fitted, one in the 4x slot and one in the 16x slot (where the GPU would normally go), i moved some drives around so i didnt need 1 card and removed it from the 16x slot

 

rebooted and now i have C6 state in powertop, getting there!

 

im now down to 24W/25W idle with dockers enabled.

 

i now need to prove a theory that the nvme isnt causing issues either, will need to work out how to best do this.

not sure i can achieve any better, i have 12 HDD and if they are 0.8W each when spun down, thats 9.6W right there,

 

According to the internet, the WD Red NVMe does 9.24W average power and it being on all the time due to being where my app data is, bloody hell, i should of checked this before buying it!

 

https://m.hexus.net/tech/reviews/storage/148435-wd-red-sn700-ssd-1tb/

 

will update when i do more tests.

Is the consensus that using the cpu's [PCIe 4.0] lanes reduces C-states when compared to using a [PCIe 3.0] slot?

 

What happens if we force pcie3.0 on pcie4.0 slot. Is it the speed or just being directly connected to cpu that dictates power efficiency?

Edited February 18, 20242 yr by dopeytree

root@Unraid:~# powertop --auto-tune 
modprobe cpufreq_stats failed
Loaded 0 prior measurements
Cannot load from file /var/cache/powertop/saved_parameters.powertop
File will be loaded after taking minimum number of measurement(s) with battery only 
RAPL device for cpu 0
RAPL Using PowerCap Sysfs : Domain Mask 5
RAPL device for cpu 0
RAPL Using PowerCap Sysfs : Domain Mask 5
Devfreq not enabled
glob returned GLOB_ABORTED
Cannot load from file /var/cache/powertop/saved_parameters.powertop
File will be loaded after taking minimum number of measurement(s) with battery only
Leaving PowerTOP

i keep getting this if i do anything with powertop:
Q1:
modprobe cpufreq_stats fail <-- is this an issue? 

Q2

100% is bad right?
Q2.2
but this is all good right ?

root@Unraid:~# lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM )'
00:06.0 PCI bridge: Intel Corporation Raptor Lake PCIe 4.0 Graphics Port (rev 01) (prog-if 00 [Normal decode])
                LnkCap: Port #5, Speed 16GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <16us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
00:1a.0 PCI bridge: Intel Corporation Raptor Lake PCI Express Root Port #25 (rev 11) (prog-if 00 [Normal decode])
                LnkCap: Port #25, Speed 16GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk-
00:1b.0 PCI bridge: Intel Corporation Raptor Lake PCI Express Root Port #17 (rev 11) (prog-if 00 [Normal decode])
                LnkCap: Port #17, Speed 8GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk-
00:1c.0 PCI bridge: Intel Corporation Raptor Lake PCI Express Root Port #1 (rev 11) (prog-if 00 [Normal decode])
                LnkCap: Port #1, Speed 8GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk-
00:1c.2 PCI bridge: Intel Corporation Device 7a3a (rev 11) (prog-if 00 [Normal decode])
                LnkCap: Port #3, Speed 8GT/s, Width x1, ASPM L1, Exit Latency L1 <64us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
00:1d.0 PCI bridge: Intel Corporation Raptor Lake PCI Express Root Port #9 (rev 11) (prog-if 00 [Normal decode])
                LnkCap: Port #9, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
00:1d.4 PCI bridge: Intel Corporation Device 7a34 (rev 11) (prog-if 00 [Normal decode])
                LnkCap: Port #13, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller PM9A1/PM9A3/980PRO (prog-if 02 [NVM Express])
pcilib: sysfs_read_vpd: read failed: No such device
                LnkCap: Port #0, Speed 16GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8125 2.5GbE Controller (rev 05)
                LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
06:00.0 SATA controller: ASMedia Technology Inc. ASM1166 Serial ATA Controller (rev 02) (prog-if 01 [AHCI 1.0])
                LnkCap: Port #0, Speed 8GT/s, Width x2, ASPM L0s L1, Exit Latency L0s <4us, L1 <64us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
07:00.0 Non-Volatile memory controller: Phison Electronics Corporation E12 NVMe Controller (rev 01) (prog-if 02 [NVM Express])     
                LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+

 

sometimes i see it go into C2. but mostly its all on 0.0

 

Edited February 18, 20242 yr by okkies

15 hours ago, bigup said:

 

An update, so I had 2x ASM1166 Sata cards fitted, one in the 4x slot and one in the 16x slot (where the GPU would normally go), i moved some drives around so i didnt need 1 card and removed it from the 16x slot

 

rebooted and now i have C6 state in powertop, getting there!

 

im now down to 24W/25W idle with dockers enabled.

 

i now need to prove a theory that the nvme isnt causing issues either, will need to work out how to best do this.

not sure i can achieve any better, i have 12 HDD and if they are 0.8W each when spun down, thats 9.6W right there,

 

According to the internet, the WD Red NVMe does 9.24W average power and it being on all the time due to being where my app data is, bloody hell, i should of checked this before buying it!

 

https://m.hexus.net/tech/reviews/storage/148435-wd-red-sn700-ssd-1tb/

 

will update when i do more tests.

 

Well i have two of them too and if i am not wrong there is nothing which we can compare in terms of endurance.

 

My System used 9 Watts without Harddisk / NVME if it goes to C8/C10 if the Networkcable is unplugged. Now with 2 RED NVMEs, 3 Red Harddisks and the Networkcable pluged in it uses around 23 Watts and wont entering any Higher C-States then C2.

 

So i am assuming that those NVMEs won't use 9 Watts

Edited February 19, 20242 yr by FlowDE

3 hours ago, FlowDE said:

 

Well i have two of them too and if i am not wrong there is nothing which we can compare in terms of endurance.

 

My System used 9 Watts without Harddisk / NVME if it goes to C8/C10 if the Networkcable is unplugged. Now with 2 RED NVMEs, 3 Red Harddisks and the Networkcable pluged in it uses around 23 Watts and wont entering any Higher C-States then C2.

 

So i am assuming that those NVMEs won't use 9 Watts

 

ok, so i removed the WD RED Nvme and still the same, only got to C6, so it must be the ASM1166 card thats in there or something else

without the NVME i saved a total of 1W over having it in, so just going to put it back in, il leave for now, i think its the motherboard, just not good enough?

 

I also tried lowering the Gen in bios on the PCIe 16x to Gen3 and it using it with a ASM1166 card and it wont let me get better than C3.

 

Im now at 24W/25W at idle with Dockers on (but idle) in C6

5 hours ago, bigup said:

 

ok, so i removed the WD RED Nvme and still the same, only got to C6, so it must be the ASM1166 card thats in there or something else

without the NVME i saved a total of 1W over having it in, so just going to put it back in, il leave for now, i think its the motherboard, just not good enough?

 

I also tried lowering the Gen in bios on the PCIe 16x to Gen3 and it using it with a ASM1166 card and it wont let me get better than C3.

 

Im now at 24W/25W at idle with Dockers on (but idle) in C6

 

Tryed without the ASM Cards?

56 minutes ago, FlowDE said:

 

Tryed without the ASM Cards?

not tried it, but will do, so just pull the ASM cards out and load up unraid ? how do i manually spin down drives as they will still be powered?

My powertop auto tune doesn't survive reboot.

Do I just add "powertop --auto-tune &>/dev/null" to go file?

On 2/17/2024 at 3:19 PM, bigup said:

 

My server runs i5-11500 and a ASRock H570 Steel Legend motherboard and Unraid 6.12.8

 

Its running 12x HDD, 1 WD RED nvme cache for appdata, and 2xSSD in RAID0 for downloads cache.

 

I use 2x ASM1166 SATA cards as well as onboard SATA for the drives

 

PSU is a brand new Corsair 650W (2021)

 

Network card is an Intel i226 PCI 2.5Gbe 1 port.

 

Onboard card is Realtek 2.5gbe but have disabled it and all other rubbish like RGB, audio etc all disabled.

 

BIOS has all Power states options enabled and ASPM enabled (L0L1)

 

When i run lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM )' everything is APSM enabled.

 

Powertop autotune ran, kernel in powersave mode and intel turbo is off (using tips and tweaks plugin)

 

No hdmi cable fitted and no unraid GUI left on in any browser.

 

Unraid flash drive is running off an internal USB header

 

Right now the server is running at 27W/28W at idle with all drives in standby apart from nvme which is always on due to dockers and powertop shows it's only in C3, ive yet to see it go to C6.

 

Am i missing anything else? whats left to check? or can it just be the cpu/motherboard combo? or something else?

I'm having a simliar issue with an i5-11500 and H570M-ITX since the last updates..

53 minutes ago, Aldamir said:

I'm having a simliar issue with an i5-11500 and H570M-ITX since the last updates..

what were you getting pre update?

14 hours ago, OrdinaryButt said:

My powertop auto tune doesn't survive reboot.

Do I just add "powertop --auto-tune &>/dev/null" to go file?

same here

Hi,

 

i follow the tuto and my server his stuck in C3 with this :

 

- Asus TUF GAMING B560M-PLUS

- i5-10400

 

Any ideas ?

1 hour ago, JamesAdams said:

Hi,

 

i follow the tuto and my server his stuck in C3 with this :

 

- Asus TUF GAMING B560M-PLUS

- i5-10400

 

Any ideas ?

 

Try to Update your Unraid to 6.12.8

"Problems due to Realtek network cards

Upgraded kernel fixes the deadlock issue with jumbo frames and the in-tree Realtek 8125 driver. However, we still recommend following the advice above regarding jumbo frames.

If you continue to have network stability issues and Tools > System Devices shows that you have a Realtek ethernet controller, grab the part number shown and search Community Apps to see if there is a Realtek vendor-supplied driver plugin for that device. For more information"

And if i am not wrong in the "Appstore" there are some Realtek-Drivers

 

  

On 2/19/2024 at 4:56 PM, bigup said:

not tried it, but will do, so just pull the ASM cards out and load up unraid ? how do i manually spin down drives as they will still be powered?

You don't need to you can check if the CPU enters higher C-States with Powertop after unplugging those cards (care about your Data bevor doing that) 

Edited February 20, 20242 yr by FlowDE

1 hour ago, FlowDE said:

 

Try to Update your Unraid to 6.12.8

"Problems due to Realtek network cards

Upgraded kernel fixes the deadlock issue with jumbo frames and the in-tree Realtek 8125 driver. However, we still recommend following the advice above regarding jumbo frames.

If you continue to have network stability issues and Tools > System Devices shows that you have a Realtek ethernet controller, grab the part number shown and search Community Apps to see if there is a Realtek vendor-supplied driver plugin for that device. For more information"

And if i am not wrong in the "Appstore" there are some Realtek-Drivers

 

  

You don't need to you can check if the CPU enters higher C-States with Powertop after unplugging those cards (care about your Data bevor doing that) 

Thanks, but after the update in 6.12.8, i'm still stuck in c3

Edited February 20, 20242 yr by JamesAdams

Hi JamesAdams,

 

Have you try to disable NIC from the bios and connect locally to launch powertop?

This is how I found my Intel 2.5G was preventing any c6-c8 state.

 

If it's the case with your Realtek 2.5G you can try to force gigabit speed instead of 2.5G (where eth0 is your network card name)

ethtool -s eth0 speed 1000 duplex full autoneg on

 

This is something that everyone has asked here.. but for anything if anybody can find a fault in my system please suggest. I am not able to hit anything past C3. I have gone through this forum multiple times and have done most of the part suggested here. My idle power is 54-60 watts with array shut down. Please find the screenshots of my system attached. Thanks in advance.

I've tried the optimisations and they seem to work. But now I need to test a few things out and I would like to restore the system to what it was before I applied this optimisation(powertop --auto-tune &>/dev/null). How can I go back to default?

Edited February 21, 20242 yr by s1l3nce

11 hours ago, s1l3nce said:

I've tried the optimisations and they seem to work. But now I need to test a few things out and I would like to restore the system to what it was before I applied this optimisation(powertop --auto-tune &>/dev/null). How can I go back to default?

The command is not persistent, if you haven't put it in your Go file, then just reboot the server

Oh! Excellent. Thank you very much 🙏

On 2/20/2024 at 4:56 PM, samarrs said:

This is something that everyone has asked here.. but for anything if anybody can find a fault in my system please suggest. I am not able to hit anything past C3. I have gone through this forum multiple times and have done most of the part suggested here. My idle power is 54-60 watts with array shut down. Please find the screenshots of my system attached. Thanks in advance.

Show us a screenshot of this console command output:
lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM )'

38 minutes ago, OrdinaryButt said:

Show us a screenshot of this console command output:
lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM )'

Thank you for your response. Screenshot's attached.

38 minutes ago, samarrs said:

Thank you for your response. Screenshot's attached.

Dang. ASPM enabled everywhere. Your previous screenshots show high CPU usage to me, or missconfiguration. Your CPU is not going into lower (higher number) C states EDIT - only first core. I am not an expert, but I would start by looking into BIOS settings. Make sure speed step is enabled and stepping are not restricted. I didn't look for your previous posts, but if this is some sort of OEM specialty motherboard, you may be restricted with options in the BIOS.

Read to the very first post again and make sure you didn't miss something.

Edited February 22, 20242 yr by OrdinaryButt

@OrdinaryButt Motherboard is Gigabyte B460 with i3 10th gen. I have gone through the bios settings multiple times with all the suggested settings done. But I will re-check again. What is Speed state and stepping? Any relative setting on Gigabyte Bios I want to look for?