Making my server more power-efficient

[mathgoy]

Dear Unraid enjoyers,

I am once again asking for your help and wisdom!

 

The purpose of this post is to figure out how I can come with a more power-efficient hardware that will still be suitable for my use case.

 

My current setup is the following:

• i9-11900k
• Gigabyte z590 Aorus Ultra
• 64GB of ram
• 9x HDD array
• 2x NVME for cache
• 1x SAS card
• 1x 10GBe card
• 1x Coral m.2
• Corsair CX650M

 

My use case is the following:

• About 30 docker containers running
• Main applications are Nextcloud, Plex, Immich, Frigate, The Arr suite and few other services (reverse proxy, zigbee2mqtt, nodered...)
• Plex transcodes up to 4 4k streams in parallel
• 1 VM (Home assistant)

 

It's been working just fine for 3 years but I now find it too power hungry for my test. It also looks WAY overkill for my use case (see below).

The whole things draws 200W on average and idle at 140W (at best).

I've been using Powertop to optimze the whole thing without a lot of success. I guess my PCIE cards are preventing the system to reach higher C-Levels.

 

So here are my questions:

1. Is it normal to draw 200W on average with this setup?
2. Is there any way I can draw less power without changing the hardware?
3. Should I have to change my hardware, which ones of the following platforms would drive the most significant drop in power consumption and still be powerful enough for my use case:
   • Config A: N305-based board (probably th emost power-efficient configuration in my list but is it suitable with my use-case)
   • Config B: 7840Hs-based board (wondering about the compatibility with unraid and hardware transcoding) 
   • Congig C: i5-13500 + ATX Board (I have no doubt this will be powerful enough but I am wondering if it will be power-efficient)
4. Any other ideas?

[Vr2Io]

What device you use to take power usage figure ?

Edited April 1 by Vr2Io

[mathgoy]

Just now, Vr2Io said:

What device you use to take power usage figure ?

I am using 2 devices:
1) a plug with direct reading on it

2) a smartplug (TP-Link)

 

both give roughly the same figures (+/- 3%)

[Kilrah]

Make sure C-states are enabled in BIOS, "max performance" features like MCE aren't enabled as it's often the default for "gaming" setups.

You could also underclock/power limit.

Edited April 1 by Kilrah

[mathgoy]

13 minutes ago, Kilrah said:

Make sure C-states are enabled in BIOS, "max performance" features like MCE aren't enabled as it's often the default for "gaming" setups.

You could also underclock/power limit.

I will check that first

[_cjd_]

Start here:

Do your drives spin down? What 10Gbe, and rj45 or sfp+/dac? Swap the SAS card for a simple ASM1166 sata adapter if your drives are sata; good ~10w. Switch from VM to docker for HA and you might see some improvement. For the rest, you've got to start in the bios and powertop stuff.

For reference, my system idles around 45w mean (absolute min logged ~38w) - 12 dockers (including home assistant) 5hdd, 1nvme, 5ssd, m.2 asm1166>sata adapter, dual 10gb (sfp+/copper dac), dual 1gbe, ipmi, external usb gadget connected to my power meter, 32gb mem/amd 5600x/x570d4u. I dropped ~12w net moving to the sata adapter in place of HBA card. Stopping home assistant is good for 4-5w.

[mathgoy]

1 hour ago, _cjd_ said:

Do your drives spin down? What 10Gbe, and rj45 or sfp+/dac? Swap the SAS card for a simple ASM1166 sata adapter if your drives are sata; good ~10w. Switch from VM to docker for HA and you might see some improvement. For the rest, you've got to start in the bios and powertop stuff.

For reference, my system idles around 45w mean (absolute min logged ~38w) - 12 dockers (including home assistant) 5hdd, 1nvme, 5ssd, m.2 asm1166>sata adapter, dual 10gb (sfp+/copper dac), dual 1gbe, ipmi, external usb gadget connected to my power meter, 32gb mem/amd 5600x/x570d4u. I dropped ~12w net moving to the sata adapter in place of HBA card. Stopping home assistant is good for 4-5w.

 

I did follow the Powertop topic to improve the C-State but couldnt notics any major improvment. 
Maybe may PCIE cards are preventing the system to reach high C-Levels.

I will certainly give it another try. 

 

My 10GBe card is a Mellanox ConnectX (MT26448) (sfp+)

I didn't expect the SAS card (an LSI SAS2008) to be so power hungry. I will look into the ASM1166 to replace it.

 

Can I ask you what kind of MB and CPU you have?

[Kilrah]

2 hours ago, mathgoy said:

Maybe may PCIE cards are preventing the system to reach high C-Levels.

Install the corefreq plugin to check, it's much more readable than powertop.

 

Also I believe Frigate is "expensive" on ressources, so "idle" may not be much idle if it's constantly running. Could try stopping things to see how much they contribute to total draw.

[mathgoy]

1 hour ago, Kilrah said:

Install the corefreq plugin to check, it's much more readable than powertop.

 

Also I believe Frigate is "expensive" on ressources, so "idle" may not be much idle if it's constantly running. Could try stopping things to see how much they contribute to total draw.

Thanks mate.
Very valid points, I will check that

[_cjd_]

6 hours ago, mathgoy said:

 

I did follow the Powertop topic to improve the C-State but couldnt notics any major improvment. 
Maybe may PCIE cards are preventing the system to reach high C-Levels.

I will certainly give it another try. 

 

My 10GBe card is a Mellanox ConnectX (MT26448) (sfp+)

I didn't expect the SAS card (an LSI SAS2008) to be so power hungry. I will look into the ASM1166 to replace it.

 

Can I ask you what kind of MB and CPU you have?

 

I noted briefly in passing: AMD Ryzen 5600X, ASRock Rack X570D4U motherboard. Memory is ECC UDIMM at 3200.

Should also note I have 11 fans in total, though all running pretty low speed they still add a few W. I wish I could drop them to zero at idle, but the system won't let me tune below 20%. Normal fan stuff doesn't work and I can't get iPMI control from a plugin to work so...

 

I've tried both a ConnectX 3 and Intel x710-DA2 (both dual SFP+) without a consistent difference (I seem to have to re-tune fans every reboot making it hard) but the Intel is reportedly better on an Intel motherboard. Figured it was worth a try.

 

I think you're going to need to start stopping containers and VMs one at a time and let the system idle for a bit each time - see what's actually sucking power. Same with pulling the various cards and even m.2 devices. *something* is preventing the system from idling efficiently. If all the bios settings and such are in the right place, this is the next logical step.

[Vr2Io]

11 hours ago, mathgoy said:

I am using 2 devices:
1) a plug with direct reading on it

2) a smartplug (TP-Link)

 

both give roughly the same figures (+/- 3%)

Thats great, if you take power usage from UPS then it just meaningless.

 

Like other person suggestion, check all hardware setting first.

I suggest you troubleshoot at system in idle with spin down disk, because other factor would affect the power usage a lot too.

 

Check "disable SVID" setting in BIOS

 

12 hours ago, mathgoy said:

200W on average and idle at 140W (at best).

 

I would expect idle should under 100w. If base on my builds ( no special hard turning )

 

9800x , 256G ram, 16 disks, 10gNIC, LSI-3008, expander backplane, GT710, idle at 100w

9700k, 64G ram, 2 disks, 10gNIC,  2 NVMe, idle at 30w

5600h , 64G ram, 2 NVMe, 12 docker, average loading 20w, idle may be 15w

 

Edited April 2 by Vr2Io

[mathgoy]

thanks for yor help!

 

So i tweaked my Bios and I can now read an average of 110W after one hour. 

Mostly power related featured that were set to Auto and switched them to enable.

There is some improvement.

 

I can see this in the Corefreq plugin but am not sure what I shoudl inderstand

 

Next step will be to power dockers on and off to identify the power hungry one.
When it comes to the PCIE card it will be more difficult to compare apple to apple since I cannot start my array without my SAS card.

I will still order a ASM1166 sata adapter to give it a try.

 

 

 

[Kilrah]

1 hour ago, mathgoy said:

I can see this in the Corefreq plugin but am not sure what I shoudl inderstand

Shows low effective clock and high C7 residency which is good, still a bit weird that there's no C2-3-4-6, might be something else to enable in BIOS.

You can press space and show the Power page, bottom will have package power. If that's low that'd mean there isn't much to save on the CPU and it's the rest of the HW.

 

Oh and remembering now, install the Tips and Tweaks plugin and set governor to powersave.

 

Edited April 2 by Kilrah

[mathgoy]

3 hours ago, Kilrah said:

Shows low effective clock and high C7 residency which is good, still a bit weird that there's no C2-3-4-6, might be something else to enable in BIOS.

You can press space and show the Power page, bottom will have package power. If that's low that'd mean there isn't much to save on the CPU and it's the rest of the HW.

 

Oh and remembering now, install the Tips and Tweaks plugin and set governor to powersave.

 

thanks that is very helpful.

I have a parity build running at the moment so my CPU is busy.
I will check that as soon as it's done

 

In Tips and Tweaks my governor was on performance...

[mathgoy]

On 4/2/2024 at 7:05 PM, Kilrah said:

Shows low effective clock and high C7 residency which is good, still a bit weird that there's no C2-3-4-6, might be something else to enable in BIOS.

You can press space and show the Power page, bottom will have package power. If that's low that'd mean there isn't much to save on the CPU and it's the rest of the HW.

 

Oh and remembering now, install the Tips and Tweaks plugin and set governor to powersave.

 

So I had a look on the Power section of corefreq while my server was idling and I was a bit surprised!

 

The package power would never go below 35W and would mostly oscillate between 45 and 65W.

 

As a comparison, a friend of I who has roughly the same VM and dockers as mine is idline at less than 5W with a i5-13500!

 

I double checked my bios and C-states are enabled. 
governor is set on powersave.

 

Next step will be to switch off some containers!