Reduce power consumption with powertop

[Aldamir]

Ah makes sense, thanks!
I have 2 M.2s, one in the CPU slot, one in the Chipset slot...

Thanks for the help!

[Aldamir]

On 3/24/2023 at 5:08 PM, mgutt said:

It enables the power saving setting on all pcie slots, USB and SATA ports. This is a one-time setting (which gets lost on every reboot). It is the same as the custom commands in the first post. So it's nothing like a background process watching and reducing power consumption if you thought about that.

 

The bios settings are looking good to me. Did you place the NVMe into the CPU M.2 slot? Try a Chipset M.2 slot then.

I remove the drive from the CPU-M.2 slot but still only C3..

[Aldamir]

Is it possibly caused by the SN570 Drives itself? I read that it might be different depending on the ssd products..

[meganie]

I get this output when I execute:

powertop --quiet --auto-tune

 

[mgutt]

Maybe someone likes to test the following command:

 

echo powersave > /sys/module/pcie_aspm/parameters/policy

 

This is something which is not set by powertop --auto-tune and was not possible to set in older Unraid versions. After fresh boot Unraid uses "default":

 

root@tower:~# cat /sys/module/pcie_aspm/parameters/policy
[default] performance powersave powersupersave

 

Regarding the Linux Kernel source code, this enables ASPM for all PCIe links "on the fly":

https://github.com/torvalds/linux/blob/master/drivers/pci/pcie/aspm.c

 

More Info:

https://wiki.archlinux.org/title/Power_management#:~:text=Bus power management

 

 

[hot22shot]

14 hours ago, mgutt said:

Maybe someone likes to test the following command:

 

echo powersave > /sys/module/pcie_aspm/parameters/policy

 

 

I did, no difference whatsoever.

 

EDIT : Also tried powersupersave (catchy name 😁), same result as default/powersave.

Edited March 29, 2023 by hot22shot

[mgutt]

Accurate power measurement

- do some research how accurate your power meter is (UPS integrated power meters are usually bad)

- close your Unraid WebGUI and wait 1 minute! (this closes many background jobs, which cost your energy, use your PC's terminal and "ssh root@tower" to monitor your C-States with powertop)

- don't use Unraid in GUI mode (you should avoid this mode, if you want low power consumption)

 

Powertop shows only C1, C2 and C3 Pgk-States

This happens if powertop does not recognize your CPU (happens for most AMD CPUs and recent Intel CPUs). Your CPU could still reach C9 or even C10, but powertop is not able to obtain this and shows only C3.

 

Those two points were added to the main post.

[Takeshi]

Hi everyone,
Thank you very much for previous information especially @mgutt-san.
I have started to use unraid last month and I tried to reduce power consumption in according to this topic.

My report might be helped for new users, I report my hardware informaton, idle power consumption and briefly note.

I am Japanese, so please forgive me if my English is a little strange.

[Conclusion]

- I could reduce idle power from 23W to 19.5W with 3.5x4 HDDs and 2.5x4 SSDs.

- The information provided here is mostly correct, so new user should decide based on the cost-effectiveness of the additional devices.

 

[Hardware]

Fujitsu TX1320 M2 with genuine PSU 250W (used)

CPU: core i5 6400 (changed from G4400)

MEM: 8GB ECC single

HBA: additional cheap asm1166/6ports with silverstone FW

NIC: additional cheap 2.5GbE RLT8125B

HDD: 3.5 4TB x4 (Ultrastar x1, WD blue x2, Seagate SMR x1)

SDD: 1TB x2, 1.6TB, 500GB (Micron DC 1TB, Intel DC 1.6TB, normal QLC 1TB, normal MLC 500GB)

 

[Note]

- asm1166 with original FW does not work using "powertop --auto-tune" well.
  This site is helpful for me how to update and confirm.
  https://docs.phil-barker.com/posts/upgrading-ASM1166-firmware-for-unraid/
  After updated, I can reduce 2-3W to just call "powertop -auto-tune". Excellent.

- Fujitsu PSU is the best efficiency in my PSUs. It seems to work efficiency almost the same as PicoPSU.
  This used PC is easy to get very cheap price around $40-50 in Japan and it includes all for creating home NAS.

-  I tried some HBA. They sometimes cost only around $5-7 like IBM ServerRAID H1110. H1110 costs 5W for 4 ports. 

   They are a cost effective option in countries with low electricity costs.
   However, I really wanted to try ASM1166 myself, so I eventually adopted ASM1166.

-  HPE Microserver Gen10 lite (X3216) does not have enought CPU power to calc parity as full speed.

   Idle power is almost the same as TX1320 M2. I don't recommend to use it.

Edited April 6, 2023 by Takeshi

[mgutt]

3 hours ago, Takeshi said:

SDD: 1TB x2, 1.6TB, 500GB (Micron DC 1TB, Intel DC 1.6TB, normal QLC 1TB, normal MLC 500GB

Regarding my experience Datacenter / Enterprise SSDs prevent deep C-States and they never reach sleep states by itself. You should measure without all SSDs and reconnect one after each other. I'm pretty sure that your power consumption could be much lower.

[Takeshi]

Hi @mgutt

Thank you for your comment.

I think same thing... At the first time, I bought Intel DC for cache drive because my wife sometimes transfer over 100GB RAW photos one time from working PC.  It works so good especially continuously writing I expected, but I noticed 1.6TB cache is too much for our use case. After that I found cheap 1TB DC micron ($50). I switched it for cache.

 

I mainly stored small files such as photoes and documents to SSD, you know,,, those files are very important for family I choosed to store them to Intel DC due to it's reliability. If I bought an affordable 2TB SSD with enough reliability, I would probably replace it from  Intel DC, but right now I feel there is little cost-effective benefit. For example, 1W reduction in power consumption can reduce electricity bills by approximately 25 JP yen/month (1kW=30-35yen) in Japan, it means reduction of 300 yen/year, 1500 yen/5years.

I always struggle with this balance because if the investment for this purpose exceeds this amount, the original objective will be lost.

As you comment, my system can be tuned more for power consumption, but I'm still exploring what is the best for me.

Your detailed advice has been invaluable by all Unraid users and I would have continued to use TrueNAS without your information. Thank you so much.

[mgutt]

6 hours ago, Takeshi said:

due to it's reliability

I don't think Unraid needs such reliable SSDs. I never saw people complaining about dead SSDs and there are several users with Consumer QLC SSDs.

 

6 hours ago, Takeshi said:

 

I always struggle with this balance because if the investment for this purpose exceeds this amount, the original objective will be lost.

That's why I asked you to remove all SSDs and measure without them. Only if you know how much energy can be saved, you can make the decision to buy other SSDs or not. Only as an example: I measured Samsung 870 Evo 4TB and Samsung DC PM883 7.68TB and the latter raised power consumption by 6W, because it does not support DIPM (low sleep states). Or another build of mine: Samsung 970 Evo Plus 2TB vs WD Black SN750 1TB. The latter added 4W, because the C-State changed from C7 to C2.

[Takeshi]

Using a trial license, I built a small NAS with a small PC and an M.2 type ASM1166 as prototype.

 

 

The PC itself is a gigabyte gb-bsi3h-6100 with ac adopter, four consumer 2.5 HDDs are connected as external box powered by PICO PSU. With this configuration, total idle power consumption was 13-14W, and auto-tune worked without any problems. As gb-bsi3h-6100 has sata x1 and m.2 x1, we can use  7 sata ports as maximum using asm1166.

 

Perhaps there are users who have already achieved something similar, but if you choose to build a NAS using an old small computer, please take a look at it. 

 

The challenge is what to do with the enclosure....

[riduxd]

Long time lurker, and I have been reading this thread for some time now. I was able to get a deep idle power of around 7W - 8W with the following build. The 8W was achieved with the array running, but with no active Docker containers. With 2 running containers (Plex and HomeBridge) the idle power is about 20W. If I remove the other drives I'm not using, I'm sure I can lower the deep idle wattage even more.

 

MB: Asus Prime Z790M-Plus

CPU: Intel 13600K

CPU cooler: Noctua NH-D15 (1 fan)

RAM: CORSAIR Vengeance LPX 32GB (2PK x 16GB) 3.2GHz

PSU: RGEEK 12V 300W Pico ATX

NVME 1: Crucial P3 4TB

NVME 2: Samsung 980 Pro 500GB

NVME 3: Inland Performance 1TB

SSD 1: SanDisk Ultra II 960GB

Boot drive: Kingston DataTraveler 2.0

Ethernet: Built-in 1Gbit

Case fans: 2

Idle: ~8W (C10)

Idle with running Docker container: ~20w (C6)

Idle with Connectx-4 LX and Docker containers running: 38w (C2)

 

All is well thanks to following this guide, but how the heck do I let my CX4 go into deeper C-State? It's plugged to my Mac using a DAC and should be inactive when my Mac is off..

 

@mgutt

 

Quote

PCIe cards

Some PCIe cards prevent states other than C0 or C3 (Intel). As an example my Qnap Aquantia 10G card allowed only C3 while all my Intel 10G cards allowed C7. Without any installed card I'm able to reach C10.

 

Which Intel card? X520??

Edited April 15, 2023 by riduxd
tagged mgutt

[mgutt]

4 hours ago, riduxd said:

Idle with Connectx-4 LX and Docker containers running: 38w (C2)

This is a huge difference. Much more than only resulting from bad c-states. 

 

4 hours ago, riduxd said:

Which Intel card?

I've tested X550 and X710:

https://forums.unraid.net/topic/105909-mein-10-zoll-server/?do=findComment&comment=1082756

 

If possible you should use an Intel card with SFP+ and DAC cables. Those are the most power efficient. If your board has an open X1 Slot, I would use this. One lane is almost enough for 10G and less lanes save even more energy. Or you try for tape the additional pins of the card to throttle it to X1. You could try the same for your current card.

 

 

[TheLinuxGuy]

5 hours ago, riduxd said:

Long time lurker, and I have been reading this thread for some time now. I was able to get a deep idle power of around 7W - 8W with the following build. The 8W was achieved with the array running, but with no active Docker containers. With 2 running containers (Plex and HomeBridge) the idle power is about 20W. If I remove the other drives I'm not using, I'm sure I can lower the deep idle wattage even more.

 

MB: Asus Prime Z790M-Plus

CPU: Intel 13600K

CPU cooler: Noctua NH-D15 (1 fan)

RAM: CORSAIR Vengeance LPX 32GB (2PK x 16GB) 3.2GHz

PSU: RGEEK 12V 300W Pico ATX

NVME 1: Crucial P3 4TB

NVME 2: Samsung 980 Pro 500GB

NVME 3: Inland Performance 1TB

SSD 1: SanDisk Ultra II 960GB

Boot drive: Kingston DataTraveler 2.0

Ethernet: Built-in 1Gbit

Case fans: 2

Idle: ~8W (C10)

Idle with running Docker container: ~20w (C6)

Idle with Connectx-4 LX and Docker containers running: 38w (C2)

 

All is well thanks to following this guide, but how the heck do I let my CX4 go into deeper C-State? It's plugged to my Mac using a DAC and should be inactive when my Mac is off..

 

@mgutt

 

 

Which Intel card? X520??

 

wow impressive. I have the 13600k on a different board B660M chipset and the most I was able to squeeze was 40 watt idle. I think I am going to buy this motherboard to see if the newer chipset makes a difference.

I was going to build my second NAS anyway and need a board like this  

[riduxd]

7 hours ago, mgutt said:

This is a huge difference. Much more than only resulting from bad c-states. 

 

I've tested X550 and X710:

https://forums.unraid.net/topic/105909-mein-10-zoll-server/?do=findComment&comment=1082756

 

If possible you should use an Intel card with SFP+ and DAC cables. Those are the most power efficient. If your board has an open X1 Slot, I would use this. One lane is almost enough for 10G and less lanes save even more energy. Or you try for tape the additional pins of the card to throttle it to X1. You could try the same for your current card.

 

 

Oh I haven't thought about that.. I'll try the bottom slot.. If it still doesn't idle efficiently, I'll just get one of those Intel NIC you mentioned.

 

6 hours ago, TheLinuxGuy said:

 

wow impressive. I have the 13600k on a different board B660M chipset and the most I was able to squeeze was 40 watt idle. I think I am going to buy this motherboard to see if the newer chipset makes a difference.

I was going to build my second NAS anyway and need a board like this  

I was pretty surprised myself when I achieved that. The only other computers in my house that can achieve that on idle are my Apple Silicon Macs

Edited April 16, 2023 by riduxd
Typo

[mgutt]

9 hours ago, TheLinuxGuy said:

the most I was able to squeeze was 40 watt idle

Remove everything except CPU+single fan, one ram module, one ssd and measure again until you find the cause. And of course: Check if your power supply is efficient at low loads.

[riduxd]

Quick update... I ended up bricking my motherboard LOL! 🤣. Are mATX boards really that more power efficient vs ATX? I'm about to go to Micro Center to get a new mobo.

 

EDIT: I will just go with an ASRock Z790 PG Lightning. The PCIEx1 slot I think is notched so therefore I can just install my CX4 in there if it is.. Modding my Asus mobo to make the CX4 fit was a BAD idea lolol.

EDIT 2: My idle with the new motherboard is the same despite it being larger in size and have more features (ie 2.5G port, which is plugged in). The CX4 is plugged in a PCIEx1 slot, and bandwidth is limited. I'll play more with this motherboard connections. I may end up just plugging it back to a x16 slot (or get those Intel NIC).

Edited April 17, 2023 by riduxd
Added more info

[TheLinuxGuy]

I'm building a second unraid server; anyone has any SATA pcie controller recommendation that allows deep C states / power efficiency?

 

My current build has ASM1064 controller and I am stuck at C3 - I am hoping my backup unraid (new) server to be more efficient. The motherboards I am looking at have at most 4 SATA ports; so I will definetely need to add at least 4 or 6 ports more. 

[hot22shot]

18 hours ago, TheLinuxGuy said:

I'm building a second unraid server; anyone has any SATA pcie controller recommendation that allows deep C states / power efficiency?

 

My current build has ASM1064 controller and I am stuck at C3 - I am hoping my backup unraid (new) server to be more efficient. The motherboards I am looking at have at most 4 SATA ports; so I will definetely need to add at least 4 or 6 ports more. 

 

I'm interested as well for an answer, I'm considering moving to an 13100T cpu, but as I need 8 sata ports I'm wondering what would be best from a power efficiency pov : using a cheap mobo and an external sata controller or targeting a pricy mobo that includes 8 sata ports.

 

Edit : same question for a 2.5GB network card.

Edited April 18, 2023 by hot22shot

[mgutt]

19 hours ago, TheLinuxGuy said:

My current build has ASM1064 controller and I am stuck at C3

Here is someone talking about C6 with ASM1166:

https://forums.unraid.net/topic/102010-recommended-controllers-for-unraid/?do=findComment&comment=1198808

 

 

[mgutt]

1 hour ago, hot22shot said:

I'm interested as well for an answer, I'm considering moving to an 13100T cpu, but as I need 8 sata ports I'm wondering what would be best from a power efficiency pov

My next try will be this board:

https://www.asrockrack.com/general/productdetail.asp?Model=W680D4ID-2T/G5/X550#Specifications

 

10G and 8 SATA ports and 2 nvme onboard + one free oculink for a third nvme or asm1166 m.2 adapter + 4 RAM Slots (which is rare for this small form factor). Should be available next month. I think it will be above 500 € 🙈

 

This is my only option as my rack is too small for mATX. But if it later supports the 48GB dimms.. 192GB RAM!

 

Note: The T-CPU is not really needed. You can change PL1, PL2 and the base multiplicator usually in all BIOS'es, so you can convert a K-CPU easily into a T-CPU. 

 

Note2: The 13th gen has only efficient Raptor Cove Cores if you buy an i5 K-CPU or any i7 or i9 CPU. So the i3 or smaller i5s are all 12th gen (with higher clocks). Regarding my research Raptor Cove is at least 17% more efficient than Golden Cove (German):

https://www.computerbase.de/forum/threads/intel-core-i5-13500-im-test-der-i5-mit-14-kernen-im-clinch-mit-ryzen-7000.2129425/page-9#post-28048507

 

 

[hot22shot]

29 minutes ago, mgutt said:

My next try will be this board:

https://www.asrockrack.com/general/productdetail.asp?Model=W680D4ID-2T/G5/X550#Specifications

 

10G and 8 SATA ports and 2 nvme onboard + one free oculink for a third nvme or asm1166 m.2 adapter + 4 RAM Slots (which is rare for this small form factor). Should be available next month. I think it will be above 500 € 🙈

 

This is way overbudget for me 😁. I'm looking at consumer oriented mobo, like this one : ASRock Z690 Pro RS ATX 1700 Socket 4 DDR4 : 8 SATA, 2.5GB network card and I can use my existing DDR4 RAM.

But as I have an existing 2.5GB PCIE Card and a LSI 8 SATA controller I wonder if buying a small/cheaper mobo could be an option.

 

29 minutes ago, mgutt said:

Note: The T-CPU is not really needed. You can change PL1, PL2 and the base multiplicator usually in all BIOS'es, so you can convert a K-CPU easily into a T-CPU. 

 

Interesting, but a K-CPU price is way higher and I don't need the extra power as I would transfer my Gaming VM to baremetal on my existing setup (5800X) with a WOL setup.

 

29 minutes ago, mgutt said:

Note2: The 13th gen has only efficient Raptor Cove Cores if you buy an i5 K-CPU or any i7 or i9 CPU. So the i3 or smaller i5s are all 12th gen (with higher clocks). Regarding my research Raptor Cove is at least 17% more efficient than Golden Cove (German):

https://www.computerbase.de/forum/threads/intel-core-i5-13500-im-test-der-i5-mit-14-kernen-im-clinch-mit-ryzen-7000.2129425/page-9#post-28048507

 

So according to you there're two options :

• A small i3/i5 Golden Cove because Raptor Cove is basically the same with higher speed.
• Or a mid-market i5 Raptor Cove that has more efficient core.

Is that right ?

 

 

 

[mgutt]

42 minutes ago, hot22shot said:

A small i3/i5 Golden Cove because Raptor Cove is basically the same with higher speed.

It can't be the same as no i3 Raptor Cove exists?!

 

43 minutes ago, hot22shot said:

• Or a mid-market i5 Raptor Cove that has more efficient core.

Note: Only i5-13600K and i5-13600KF are Raptor Cove. Not sure if you categorize them as "mid-market" ^^

 

44 minutes ago, hot22shot said:

Is that right ?

My main hint was: If you want to buy an i3 13th, than you can buy an i3 12th gen instead as those are the same.

 

47 minutes ago, hot22shot said:

Interesting, but a K-CPU price is way higher and I don't need the extra power

If you stick to i3, than you could buy the i3-12100 or i3-13100 instead. Every CPU can be throttled to a T-CPU.

[hot22shot]

19 hours ago, mgutt said:

If you stick to i3, than you could buy the i3-12100 or i3-13100 instead. Every CPU can be throttled to a T-CPU.

 

Ok I went for the 13100 and the ASRock Z690 PRO RS I mentionned before. I'll get back when I got all the parts for my new server (I also order a new case, an Aerocool Cipher). But hopefully it will be a lot less than it is actually (83W).