Reduce power consumption with powertop

[Brutos]

Since the last Unraid update I cant seem to get beyond C3 now. Anyone else experience anything similar. Before i could reach c8/c10

[KarlMeyer]

1 hour ago, Brutos said:

Since the last Unraid update I cant seem to get beyond C3 now. Anyone else experience anything similar. Before i could reach c8/c10

Same, I think it has something to do with the RTL125B drivers plugin. Could hit C10 before, but now only hit C3. Bought a x710 & replaced & disabled the onboard realtek Nic. Can now reach C8.

[Brutos]

21 hours ago, KarlMeyer said:

Same, I think it has something to do with the RTL125B drivers plugin. Could hit C10 before, but now only hit C3. Bought a x710 & replaced & disabled the onboard realtek Nic. Can now reach C8.

Thanks I will get a similar network card and disable the onboard one.

[KarlMeyer]

1 hour ago, Brutos said:

Thanks I will get a similar network card and disable the onboard one.

Let me know how it goes, i'm curious. I bought mine from here if that helps. This seller gives you both bracket mounts.

 

https://www.ebay.com/itm/165479772746

 

Good luck.

[KarlMeyer]

There doesn't seem to be a lot of discussion about GPUs & package c states. No matter what I try, it seems that if you have an Nvidia GPU in your server, the package c states can't go lower than C3. Is there anything that can be done to allow it to go lower? I've tried with a 2070 super & a 3090.

[pOpYRaid]

On 9/1/2024 at 4:42 AM, KarlMeyer said:

Same, I think it has something to do with the RTL125B drivers plugin. Could hit C10 before, but now only hit C3. Bought a x710 & replaced & disabled the onboard realtek Nic. Can now reach C8.

Thanks a lot, this saved me ~10 watts of idle consumption!

When the RTL8125B is installed in the system, pkg state just got to C2.
Now it's hitting C6 again.
Will post my reports soon to get help an maybe get to C10 :-)

[pOpYRaid]

@mgutt thanks for all the informations. i have read a lot in the thread followed guide to keep 24/7 power consumption low.
Up until now i have used standby with the s3 script, but i want it to be online 24/7 (because of other issues with standby).

 

Here is my setup :

• Mainboard: ASRock B660M-ITX/ac
• RAM: 2X16GB
• PSU: be quiet BQT L7-350W (85plus) - rather old from before 2010!!!!!!!!!!!!!!!!!!
• CPU: 12th Gen Intel® Core™ i3-12100T @ 2178 MHz
• Cache: 1x Samsung 980 NVMe M.2 SSD, 1 TB
• Array: Seagate 1x 14TB & 1x 20TB
• PCIe NIC RTL8125B

Starting point was ~50 Watt, disks spun up and ~35 Watt spun down (all VM and dockers running, Display disconnected).

As stated in the above post, the RTL8125B PCIe nic card was a culprit!
It prevented the cpu from entering pkg c states and consumed about ~10 w (incl. cpu).
After i read the posts above i immidattly removed it and used the internal intel I219V.

 

These was the idle stats before:

 

device stats:

 

lspci cmd:

 

cpufreq:

 

 

 

This is what i have done to lower the consumption:

• Update BIOS from 3.01 to 11.05 (had a rather 2 year old one)
• Disconnect the monitor (was connected all the time before to the AVR)
• Edit the /boot/config/go as in OP and also add "powertop --auto-tune" at the end, which gives another watt
• Enabled all BIOS settings regarding power save, ASPM, ... some of which
  • PCIe ASPM L0sL1 and just L0s
  • Intel C-State: Enabled
  • C1E Support: Enabled
  • Package C6 State Limit: Enabled
  • Package C7 State Limit: Enabled
  • Disabled HD Audio Controller/COM/LPT
• Enable the hidden Low Power BIOS setting for my asrock  to reach higher c states: https://www.reddit.com/r/ASRock/comments/1998ozl/how_to_get_higher_pkg_cstates_on_asrock/
  !! BE AWARE !! when enabling this you will reach higher c states (C8 instead of C3 for me) BUT s3 sleep is not working!!!!!!!!!!

 

This is the result:

 

 

I am now reaching C6 and been down to ~19 watt with disk spun down (all docker & vms are running).

Which is good, but electricity costs are high in europe, so i want to come down further.

 

My ideas are:

• Enable hidden "Native ASPM" which is default on "Auto" (with AMI tool)
  • -> Result: does'nt make a difference
• Remove cache ssd and retest (Samsung 980 1 TB PCIe 3.0 (bis zu 3.500 MB/s) NVMe M.2 Internes Solid State Drive (SSD) (MZ-V8V1T0BW))
  • Result: tested in Ubuntu, with nvme it reaches also C6 (like in unraid ~19w), without nvme it reaches C8 (~17w)
• Other ITX LGA1700 Mainboard which supports C10 state? (any hints which one)?
• Another ATX PSU (any recommendations)?
   

Thanks a lot

pOpY

Edited September 7 by pOpYRaid

[mgutt]

56 minutes ago, pOpYRaid said:

PSU: be quiet 450W (dont know which efficiency, but usally they are good)

As you only have two disks connected, what about a PicoPSU?

 

57 minutes ago, pOpYRaid said:

Other ITX LGA1700 Mainboard which supports C10 state? (any hints which one)?

I found one here:

https://www.hardwareluxx.de/community/threads/die-sparsamsten-systeme-30w-idle.1007101/page-110#post-30363543

 

Maybe you want to search further in this thread.

[pOpYRaid]

27 minutes ago, mgutt said:

As you only have two disks connected, what about a PicoPSU?

Will look into it, i think the PicoPSU-90 will be sufficent?

 

28 minutes ago, mgutt said:

I found one here:

https://www.hardwareluxx.de/community/threads/die-sparsamsten-systeme-30w-idle.1007101/page-110#post-30363543

 

Maybe you want to search further in this thread.

I have an Lian Li PC-Q25 case which just holds mini-ITX.
Will look in the thread.
Is there any recommended chipset for LGA1700 cpu`s (regarding c states).

 

thx

 

 

[mgutt]

2 hours ago, pOpYRaid said:

Is there any recommended chipset for LGA1700 cpu`s (regarding c states).

I don't think that this really matters. I think the main issues nowadays are additional chips/controllers on the board like ASM media Chipsets for USB 3.1, Realtek 2.5G etc

 

2 hours ago, pOpYRaid said:

i think the PicoPSU-90 will be sufficent?

I'd say buy a little bit bigger version to be more flexible for some additional HDDs in the future?!

 

2 hours ago, pOpYRaid said:

just holds mini-ITX.

ITX....meh. that's my recent investment as there is such a limited amount of good ITX boards:

 

Compared to the measurements of the last months I would try a Gigabyte ITX board:

https://geizhals.de/?cat=mbp4_1700&xf=4400_Mini-ITX~544_GIGABYTE&sort=r&hloc=at&hloc=de&v=e&pg=1

 

Maybe the GIGABYTE B760I AORUS Pro DDR4 as it has 2.5G from Intel and 4 SATA ports?!

 

Note: The DDR5 version has only 2 SATA ports.

 

Here are multiple tests of LGA1700 itx boards:

https://forums.unraid.net/topic/147723-itx-messungen-gigabyte-b760i-aorus-pro-und-asrock-z790-phantom-gaming-itxtb4/

 

 

[pOpYRaid]

11 hours ago, mgutt said:

I don't think that this really matters. I think the main issues nowadays are additional chips/controllers on the board like ASM media Chipsets for USB 3.1, Realtek 2.5G etc

 

I'd say buy a little bit bigger version to be more flexible for some additional HDDs in the future?!

 

ITX....meh. that's my recent investment as there is such a limited amount of good ITX boards:

 

Compared to the measurements of the last months I would try a Gigabyte ITX board:

https://geizhals.de/?cat=mbp4_1700&xf=4400_Mini-ITX~544_GIGABYTE&sort=r&hloc=at&hloc=de&v=e&pg=1

 

Maybe the GIGABYTE B760I AORUS Pro DDR4 as it has 2.5G from Intel and 4 SATA ports?!

 

Note: The DDR5 version has only 2 SATA ports.

 

Here are multiple tests of LGA1700 itx boards:

https://forums.unraid.net/topic/147723-itx-messungen-gigabyte-b760i-aorus-pro-und-asrock-z790-phantom-gaming-itxtb4/

 

 

Thx, for your suggestions.

Maybe i will try a PicoPSU (120W).
 

Have now booted ubuntu and it was also stuck at c6 (~19w).
After removing my nvme SSD it got c8 with ~17 watt.

Not much and i dont want to remove the nvme :-(

 

Have found this thread:

 

What i am reading the "Gigabyte H610I DDR 4" would fit my needs and runs in higher c-states.
Maybe i will also try it.

Otherwise i am happy with the current results of ~20 watt when all docker and VM's are running (on nvme cache) and disks spun down. Before the tweaks it was 30-35w.

I must decide for myself if i want it 24/7 or in s3 sleep (which i had up until now).

 

 

[mgutt]

7 minutes ago, pOpYRaid said:

Not much and i dont want to remove the nvme 😞

Maybe trying the other slot?

[pOpYRaid]

7 hours ago, mgutt said:

Maybe trying the other slot?

Sadly my Mobo just has 1 slot which seems to be connected to the CPU, hence preventing it from sleep.
This i can solve later and now i know this. I also have laying arround another WD nvme 1TB and an 1TB sata SSD.
Thos i will try later.

BUT becaue you have mentioned the PSU. I have looked at it an it's an "be quiet BQT L7-350W (85plus)", rather old from before 2010!!!!!!!!!!!!!!!!!! (Have corrected my first post from above).

I dont find any reviews regarding eficiency on low power draw, maybe that's one culprit.

 

I dont want an external brick + PicoPSU-160-XT + Internal adapters, so is there an replacement for the rather good "corsair RM550x 2021"?
Ok, after more reading, i will go with an PicoPSU-160XT + LEICKE 150W power supply 12V 12.5A + SATA Adapters.

My plan now is:

• picoPSU
• Mainboard (i think the Gigabyte H610I)
• check cstates and power draw again and... maybe replace the "Samsung 850" NVME

 

Edited September 7 by pOpYRaid

[Duc]

Hello,

 

i bought that motherboard cwwk H670, with a i5 14500, 96gb ram, 8hdd, 3nvme, with the tweaks in the bios, i consume around 25W idle (hdd spin down), however i'm stuck to c3 in powertop, did someone reach c10 with that board ? (i could not reach c10 even with just the cpu/ram and unraid booted)

 

do we have a list of bios tweaks to reach c10 ?

 

 

 

Edited September 8 by Duc

[Sandbank7]

On 8/30/2024 at 9:23 AM, bagican said:

@Sandbank7 did you measure real power consumption? I'm just wondering how many watts it was before and after?

I don't have super solid numbers on this, as I was monitoring it using an old meter (which requires you to look at a physical display to see), but it would never dip below 40W, rarely below 45W. With my new meter I store the data on the server, and I'm regularly below 40W (36-39W) when all drives are spun down, and around 45-50W with a few drives spinning.

 

It's not a massive difference, but it's "free". My server rarely actually does nothing, so the minimum draw of around 36W from the wall seems reasonable.

[pOpYRaid]

2 hours ago, Duc said:

Hello,

 

i bought that motherboard cwwk H670, with a i5 14500, 96gb ram, 8hdd, 3nvme, with the tweaks in the bios, i consume around 25W idle (hdd spin down), however i'm stuck to c3 in powertop, did someone reach c10 with that board ? (i could not reach c10 even with just the cpu/ram and unraid booted)

 

do we have a list of bios tweaks to reach c10 ?

What manufacturer of the board is it?

If ASRock and you have just up to "Enable C7" in the BIOS you can try the following to reach higher c states:

 

Enable the hidden Low Power BIOS setting to reach higher c states:

 

 

 

!! BE AWARE !! when enabling this you will reach higher c states (C8 instead of C3 for me) BUT s3 sleep is not working in my case!!!!!!!!!!

When you don't use S3 sleep (standby) it's ok.

 

Also try to boot Ubuntu with the nvmes removed.

In my case my Samsung 980 nvme prevents C8 and it's stuck in C6, when it's installed.

[pOpYRaid]

I have now replaced my rather old PSU with picoPSU.
Now when all docker and VM's stopped i am down to ~14w from ~19w - 20w.

It reaches c6 with the asrock board and Samsung nvme not removed (which i know prevents it from c8).
So yesss, picoPSU saved 25% power draw, yeah.

 

My next step is to replace the known bad Asrock B660M-ITX/ac board with the gigabyte H610i,

which should perform better, see here: 

 

 

Keep you updated.

Edited Tuesday at 05:36 PM by pOpYRaid

[woodman]

Hey Everyone,

 

Has anyone had success getting a Dell LSI 9211-8i, the "Serial Attached SCSI controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)" to ASPM Enabled?

 

Are there any recommended alternatives for a controller with ASPM support if it is not possible?

 

I think the SCSI controller is my roadblock to getting higher than C3.

 

 

M/B:ASRock H97M Pro4 s/n M80-47012401538

BIOS:American Megatrends Inc. Version P2.30 Dated 03/07/2018

CPU:Intel® Xeon® CPU E3-1241 v3 @ 3.50GHz

 

All tunables are good.

 

Thank you!

[pOpYRaid]

So keep you updated, i will post my results here.

!! FIRST A HUGE THANKS TO @mgutt and @Chr1sch !!

From this thread: https://forums.unraid.net/topic/147723-itx-messungen-gigabyte-b760i-aorus-pro-und-asrock-z790-phantom-gaming-itxtb4/

 

 

I have now replaced my ~15 year old PSU with an pixoPSU-XT-160 and today the shitty Asrock "B660M-ITX/ac" with an Gigabyte "H610I DDR 4" all other hardware is unchanged.
See my post above for the other hardware https://forums.unraid.net/topic/98070-reduce-power-consumption-with-powertop/?do=findComment&comment=1462149

 

After changing the PSU to the pixoPSU-XT-160 i got an improvment (with Asrock BIOS hack to reach C6-C8), see here

https://forums.unraid.net/topic/98070-reduce-power-consumption-with-powertop/?do=findComment&comment=1463177

 

Now after swapping the mainboard but all other hardware connected (idle, no HDMI/DP, 1x NVME, 3x HDD (in standby)):

• Ubuntu: ~9,6 Watt  / No Dockers/VM's / C8 (see picture 1)
• Unraid:  ~10,3 Watt / All Dockers/VM's stopped / C8
• Unraid:  ~11,3 Watt /  All Dockers started & VM (Windows 11) removed/stopped / C8

I am happy with the results, so finally i will leave my unraid 24/7 on.

 

Thanks a lot for the help on c-states journey 

 

Picture 1 - 9,6 Watt:

 

Picture 2 - 11,3 Watt:

 

 

[pOpYRaid]

On 9/1/2024 at 4:42 AM, KarlMeyer said:

Same, I think it has something to do with the RTL125B drivers plugin. Could hit C10 before, but now only hit C3. Bought a x710 & replaced & disabled the onboard realtek Nic. Can now reach C8.

Any cheap 2,5 GBE REplacement card which has no issues with higher c-states?

 

P.S.: Dont need 10 GBE

 

Thank you

[mgutt]

18 minutes ago, pOpYRaid said:

Any cheap 2,5 GBE REplacement card which has no issues with higher c-states?

Not tested by now, but I bought this one:

https://www.idealo.de/preisvergleich/OffersOfProduct/203754581_-i226-t1-intel.html

 

This is a genuine Intel card. Of course there are cheaper versions with i225/i226 chip of other manufacturers.

[pOpYRaid]

18 hours ago, mgutt said:

Not tested by now, but I bought this one:

https://www.idealo.de/preisvergleich/OffersOfProduct/203754581_-i226-t1-intel.html

 

This is a genuine Intel card. Of course there are cheaper versions with i225/i226 chip of other manufacturers.

Thanks a lot for the hint.

Is it possible for you test c-states of unraid with this card?
If it get's worse with it or not.

 

If you search the net there are many many reported issues with I225 and (yes intel did it again :-) the I226.
Would be really nice if you can test your card and give us feedback.

 

Thank you

pOpY

Edited 8 hours ago by pOpYRaid

[pOpYRaid]

@Brutos @KarlMeyer @mgutt
I have now tested the 8125B card again.
Sadly if it's installed, the machine is stuck in C6 which consumes 4-5 Watts more idle.
 

I have also tested it with different drivers and tweaks (verifed with lsmod that it's loaded):

• Default r8169 driver -> ASPM shown enabled -> Same issue as stated above.
• Driver R8125 plugin () -> ASPM shown enabled -> Same issue as stated above.
• Tried the aspm=1 mod from here: https://forums.unraid.net/topic/141349-plugin-realtek-r8125-r8126-r8168-and-r815267-drivers/?do=findComment&comment=1379870
   -> Same issue as stated above.

So no chance to get low idle power consumption when this card is installed.

Would be nice if anyone of you can recommend an "cheap" 2,5GBE PCIe card which is capable of high cstates (low power consumption). When i got it right there not so much chip manufactures, there is Intel, Marvel and Realtek.
Maybe a marvel one will do the job?

thx