I7 13700K and ASUS PRO WS W680-ACE IPMI Optimizations

[firstTimer]

Hi guys, after long time and plenty of researches I finally built a new NAS. This is a system to built to last as much as it can with a balance between power and efficiency in mind. 

 

Here there is a list of "hardware" requirements I made before building.

 

• Quicksync (de facto, is the encoder/decoder for Plex)
• More than 1 RJ45 port
• IPMI for "wireless" monitoring/configuring
• Plenty of cores for VMs/containers
• Enough M.2 slots (in this MB there are 3 in total)

 

After giving the main requirements points I will illustrate the build:

 

• CPU: Intel 13700K
• Case: Silverstone RM41-H08
• Cooler: noctua nh-d12l
• MB: ASUS PRO WS W680-ACE IPMI
• PSU: Corsair RM850i
• RAM: 2x32GB DDR5 Crucial @4000Mhz 1.1V
• SSD: 2x2TB Crucial P3 + 1x1TB Sabrent
• HDD: A mix of 6 seagate Ironwolf and barracuda from 8TB to 1 ancient 500GB and 1 SATA no brand SSD
• GPU: ASUS GTX 1050 2GB (with P8 state enabled)
• Mix: 2x5bay hotswap cages

 

Latest MB bios --> 2703

 

Now the good stuff about optimising this CPU:

PS: This is STILL a work-in-progress

 

first section is the BIOS side:

 

I am trying to be as balanced/flexible as possible so that the CPU delivers good performance when needed, but always with energy efficiency in mind.

As a starter I based myself on the specs of the cousin of this CPU the i7 13700 T model.

From its cousin we are going to apply the same maximum frequency of both E-core and P-core 4.9 GHz for P-Cores and 3.6 GHz for E-cores. (screenshot 1)

Since the cousin has a drop in the TDP (min 35W and max is 106W) I will apply the same (screenshot 2)
Next thing we want to do is to enable the C-states so that the CPU can drop to lower consumption when it is in idle state. (screenshot 3)

 

 

 

 

Other BIOS settings to experiment with:

turbo boost max 3.0: enabled
intel speedstep: disabled
intel speedshift: enabled (speedshift is an improved version of speedstep)
intel adaptive boost tech: enabled

 

Second section  Unraid side:

Tips and tweaks plugin

Turbo mode is still enabled and only powerSave or performance avaiable --> powersave is applied

User script:

I apply a non-optimised version of the script for powertop -->

powertop --auto-tune &>/dev/null

 

 

------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

 

That's it for the first post. I will keep this thread updated with new discoveries because I want to apply some undervolt and let's see how far can this CPU drop in consumption

Edited September 21, 2023 by firstTimer

[venicenerd]

This is FANTASTIC!! By any chance could you shed some light on the IOMMU groupings of this board? 
 

Will it be possible to pass the PCI 5.0x16, one of the PCI 4.0 and the main M2 drive to a VM separately? I would like to bind those to the VM but non of the documentation has block diagrams.

[Daniel15]

I had an issue with Tips and Tweaks where I  set the CPU governor to "power save" and enabled turbo, but it didn't actually enable turbo. If you're not seeing the CPU hit turbo speeds, you need to explicitly set the max scaling frequency to the max CPU frequency:

 

for x in /sys/devices/system/cpu/cpu*/cpufreq/; do cat $x/cpuinfo_max_freq > $x/scaling_max_freq; done

 

You should also set the the energy/performance preference for the hardware-managed P-states (HWP) to "balance_power". This is not currently exposed by the Tips and Tweaks plugin:

 

for x in /sys/devices/system/cpu/cpu*/cpufreq/; do echo balance_power > $x/energy_performance_preference; done

 

This'll make it less likely to turbo (and will only turbo individual cores as needed) and dropped power consumption another 2-3W for me.

 

(you can combine both into a single loop if you'd like)

 

Add these to /boot/config/go to make them persist across reboots.

 

 

Edited September 20, 2023 by Daniel15

[firstTimer]

@Daniel15 Thanks for reporting this. At first I would like to consolidate the BIOS side and then keep fine tuning on Unraid side. The frequencies seem to be good though (4.6 GHz for Core 0 (P-Core) and an example of an E-Core that maxed at 3.6 GHz)

 

Anyway actual consumption:

at startup: 120W (6 HDDs + 3 NVME + 1 SSD sata + GPU + IPMI)

idle: between 80W and 60W (with Unraid dashboard opened)

Let's see how it will behave during the night when every client etc is closed

 

PS:

I noticed that reserving (core isolation) that last 4 E-Core (20, 21, 22, 23) to Unraid led them to stay maxed out at 3.6GHz so for the moment I left Unraid to use all cores.

 

 

 

Edited September 21, 2023 by firstTimer

[firstTimer]

1 hour ago, venicenerd said:

This is FANTASTIC!! By any chance could you shed some light on the IOMMU groupings of this board? 
 

Will it be possible to pass the PCI 5.0x16, one of the PCI 4.0 and the main M2 drive to a VM separately? I would like to bind those to the VM but non of the documentation has block diagrams.

Sorry I am not into IOMMU mapping at the moment

[JimmyGerms]

@firstTimerCould you post images of all your BIOS settings for the power saving options. I cannot get this to work, unraid crashes with powertop after a few minutes. I'm wondering if I got a bunk bios setting.

 

P.S. This is an awesome write-up!

[venicenerd]

2 hours ago, firstTimer said:

Sorry I am not into IOMMU mapping at the moment

Could you post images of your tools -> system devices output from Unraid? Like my screenshots attached? I should be able to gather some information from that I think. 
 

Thank you!
 

 

[firstTimer]

4 hours ago, venicenerd said:

Could you post images of your tools -> system devices output from Unraid? Like my screenshots attached? I should be able to gather some information from that I think. 
 

Thank you!
 

 

Here it is:

The Nvidia GTX 1050 is mounted on the purple PCIE and IPMI is installed on the yellow one. NVME (starting from top to bottom, e.g from closest to the PCIE x1 slot) are the Crucial P3 mounted on first (with passive cooler) and second M.2 slots and the sabrent is mounted on the third one (near GTX 1050)

[firstTimer]

I post a little estimation of the wattage of each component (trying to be close enough):

• HDDs: 1w each (spin down) = 6w
• Sata SSD (idle): 0.5w
• NVME SSDs: 0.5w each (idle) = 1.5w
• HDD cages: each 5 blue led (not kept into consideration due to low power) + 80mm fan = 4 w
• Fans: 3x80mm = 6w
• IPMI: 6w
• Cooler fan: 120mm = 4w
• GPU: 1050 (P8 state) = 6-8w

total of NON CPU consumption: 34w

Idle reading at the moment from smart plug = 56w

CPU consumption = 22w

Edited September 21, 2023 by firstTimer

[firstTimer]

7 hours ago, JimmyGerms said:

@firstTimerCould you post images of all your BIOS settings for the power saving options. I cannot get this to work, unraid crashes with powertop after a few minutes. I'm wondering if I got a bunk bios setting.

 

P.S. This is an awesome write-up!

 

Do you have the latest bios? 2703?

 

Sure, I just need some time to do the screenshots but I will do that 👍

Edited September 21, 2023 by firstTimer

[venicenerd]

1 hour ago, firstTimer said:

Here it is:

The Nvidia GTX 1050 is mounted on the purple PCIE and IPMI is installed on the yellow one. NVME (starting from top to bottom, e.g from closest to the PCIE x1 slot) are the Crucial P3 mounted on first (with passive cooler) and second M.2 slots and the sabrent is mounted on the third one (near GTX 1050)

 

Thank you so much!!! Seems like it’s bad news for me, though. 
 

Appears like all of the SATA and M2 devices are all in group 4, right?

 

I can’t really spot your Crucial P3 M2, though. Is it part of group 4 as well?

[firstTimer]

21 minutes ago, venicenerd said:

 

Thank you so much!!! Seems like it’s bad news for me, though. 
 

Appears like all of the SATA and M2 devices are all in group 4, right?

 

I can’t really spot your Crucial P3 M2, though. Is it part of group 4 as well?

Yes they are all in the group 4, last three entries in the group are the NVME. I never tried to pass a whole SSD to a VM, but... can't you just have an ssd as an unassigned device and pass it through in that way?

Edited September 21, 2023 by firstTimer

[venicenerd]

18 minutes ago, firstTimer said:

Yes they are all in the group 4, last three entries in the group are the NVME. I never tried to pass a whole SSD to a VM, but... can't you just have an ssd as an unassigned device and pass it through in that way?

Dang it. That makes this board a little less desirable for me. 

 

yes you can but for gaming it is best to pass the bare metal m2 card for maximum speed. It’s quite a bit faster that way. 
 

I’m actually surprised how limited the IOMMU groups are on this board compared to my current Asus board. 
 

TBH I’m also confused why all the M2’s are in the same group. I thought the first M2 slot was CPU direct and the others on the chipset. I thought that for sure would make them separate groups. But then again I really don’t know too much about IOMMU and how it all works. 

 

 

[firstTimer]

1 minute ago, venicenerd said:

Dang it. That makes this board a little less desirable for me. 

 

yes you can but for gaming it is best to pass the bare metal m2 card for maximum speed. It’s quite a bit faster that way. 
 

I’m actually surprised how limited the IOMMU groups are on this board compared to my current Asus board. 
 

TBH I’m also confused why all the M2’s are in the same group. I thought the first M2 slot was CPU direct and the others on the chipset. I thought that for sure would make them separate groups. But then again I really don’t know too much about IOMMU and how it all works. 

 

 

Yeah I am not expert at all when talking about IOMMU. What I know is that you could try to isolate the devices "by hand, but I don't know how "properly" it will work out.

[venicenerd]

12 minutes ago, firstTimer said:

Yeah I am not expert at all when talking about IOMMU. What I know is that you could try to isolate the devices "by hand, but I don't know how "properly" it will work out.

Same. My only hope is to find someone with this board that is doing something similar. I was really set to buy this one until just now. Lol 

[Daniel15]

9 hours ago, venicenerd said:

yes you can but for gaming it is best to pass the bare metal m2 card for maximum speed. It’s quite a bit faster that way. 

Honestly I really doubt you'd actually notice the difference in performance between passing through the entire disk as a block device (ie adding /dev/disk/by-id/..... as a vdisk for the VM) vs passing through the M.2 controller as a PCIe device. A block device is already pretty low-level. Even a PCIe Gen 3 NVMe drive is usually sufficient for gaming, and passing through a Gen 4 NVMe drive as a block device would easily be faster than that.

Edited September 21, 2023 by Daniel15

[firstTimer]

I tried to perform a first cineBench benchmark to test mostly stability and observed the behaviour during the benchmark.

Environment:

Windows 11 VM with all 24 cores dedicated

Results:

 

Single Core performance is pretty similar to i7 13700T

Multi Core is basically half of the original i7 13700T counterpart (no good)

Observations:

During tests I checked netData and especially during multiCore (between green bars) bench the clock of all cores dropped and stabilised around 2.2 GHz (I guess because of the low TDP)

Single Core test (between blue bars) returned to a stable and much better clock

 

 

 

Edited September 21, 2023 by firstTimer

[firstTimer]

22 hours ago, JimmyGerms said:

@firstTimerCould you post images of all your BIOS settings for the power saving options. I cannot get this to work, unraid crashes with powertop after a few minutes. I'm wondering if I got a bunk bios setting.

 

P.S. This is an awesome write-up!

I did not forget about you. I am trying to fix an issue that came up. Basically Unraid works with not strange or whatever, the only I noticed is that when I reboot Unraid AND I don't access to the BIOS then it just does not boot the USB key. So as soon as I fix this I will make a BIG post with some photos

[JimmyGerms]

Sounds great! Also, I posted in another thread about the multi thread performance. I’m getting the same thing if I mess with any voltage or load line with this board. My performance is cut nearly in half. My bios is up to date as well. I’m having a heck of a time under-volting and keeping similar performance as default auto. 

I tried the other micro code setting thinking it was the whole intel disabling under-volting deal but still no go. 

[firstTimer]

13 hours ago, JimmyGerms said:

Sounds great! Also, I posted in another thread about the multi thread performance. I’m getting the same thing if I mess with any voltage or load line with this board. My performance is cut nearly in half. My bios is up to date as well. I’m having a heck of a time under-volting and keeping similar performance as default auto. 

I tried the other micro code setting thinking it was the whole intel disabling under-volting deal but still no go. 

So I got my head around the concepts of PL1 and PL2 parameters aka Short and Long duration Package Power Limit and in the BIOS. With PL1 we have the long duration, so it means for jobs where the cpu will take a long time to do it will apply that power limit to the cpu package. With PL2, if we have a short job on cpu that lasts more than a couple of seconds if I remember then the power limit kicks in and drops the power package of the cpu to PL1.

That's why in the first benchmarks I made, during multi-core I saw frequencies of all cores around 2.5GHz, basically the PL1 was enforced. On the other hand during SingleCore test instead, I got those allowed bursts in clocks  due to PL2 but that were immediately dropped and upped very quickly because PL1 was enforcing its limit.

 

If we apply the PL1 of 35W and PL2 to 106W as the cousin 13700T it means that for most of the time our cpu will be limited to 35W for long jobs but bursts to 106W (maybe i will up at 120W) are allowed. I think that in the case of Unraid and most users that's a good balance, so that you will have a very low powered cpu, very cool and silent most of the time with occasional (single core) bursts when needed.

 

To check my theory I applied a PL1 = to PL2 at 106w and the multi-core benchmark improved a lot from 600/700 to 1084 (always in a Win 11 VM)

Starting from a default 2703 Bios version and try to apply these parameters:

1. Ai Tweaker:
   1. Asus Performance enhancement 3.0: Disabled
   2. Performance Core Ratio: By Core Usage and for all P-Core apply 49
   3. Efficiency Core Ratio: By Core Usage and for all E-Core apply 36
   4. Max. CPU Cache Ratio: 38
2. Ai Tweaker --> DIGI+ VRM
   1. Cpu Input Voltage Load-Line: Level 2
   2. Sync ACDC Loadline with VRM: enabled
3. Ai Tweaker --> Internal CPU Power Management
   1. Long Duration Power Package Limit: 35
   2. Package Power Time Window: 64
   3. Short Duration Power Package Limit: 120
4. Advanced --> Cpu Configuration --> CPU Power Management Control
   1. Intel Speedstep: disabled
   2. Intel Speedshift: enabled
   3. Intel Turbo boost Max Tech: enabled

Edited September 22, 2023 by firstTimer

[casperse]

On 9/22/2023 at 6:38 PM, firstTimer said:

So I got my head around the concepts of PL1 and PL2 parameters aka Short and Long duration Package Power Limit and in the BIOS. With PL1 we have the long duration, so it means for jobs where the cpu will take a long time to do it will apply that power limit to the cpu package. With PL2, if we have a short job on cpu that lasts more than a couple of seconds if I remember then the power limit kicks in and drops the power package of the cpu to PL1.

That's why in the first benchmarks I made, during multi-core I saw frequencies of all cores around 2.5GHz, basically the PL1 was enforced. On the other hand during SingleCore test instead, I got those allowed bursts in clocks  due to PL2 but that were immediately dropped and upped very quickly because PL1 was enforcing its limit.

 

If we apply the PL1 of 35W and PL2 to 106W as the cousin 13700T it means that for most of the time our cpu will be limited to 35W for long jobs but bursts to 106W (maybe i will up at 120W) are allowed. I think that in the case of Unraid and most users that's a good balance, so that you will have a very low powered cpu, very cool and silent most of the time with occasional (single core) bursts when needed.

 

To check my theory I applied a PL1 = to PL2 at 106w and the multi-core benchmark improved a lot from 600/700 to 1084 (always in a Win 11 VM)

Starting from a default 2703 Bios version and try to apply these parameters:

1. Ai Tweaker:
   1. Asus Performance enhancement 3.0: Disabled
   2. Performance Core Ratio: By Core Usage and for all P-Core apply 49
   3. Efficiency Core Ratio: By Core Usage and for all E-Core apply 36
   4. Max. CPU Cache Ratio: 38
2. Ai Tweaker --> DIGI+ VRM
   1. Cpu Input Voltage Load-Line: Level 2
   2. Sync ACDC Loadline with VRM: enabled
3. Ai Tweaker --> Internal CPU Power Management
   1. Long Duration Power Package Limit: 35
   2. Package Power Time Window: 64
   3. Short Duration Power Package Limit: 120
4. Advanced --> Cpu Configuration --> CPU Power Management Control
   1. Intel Speedstep: disabled
   2. Intel Speedshift: enabled
   3. Intel Turbo boost Max Tech: enabled

 

Great write up!
I just got the MB but with a Bios: v. 3101
And I cant find a way to set the (1-2) AI Tweaker > Efficiency Core Ratio: By Core Usage and for all E-Core apply 36
Anymore instead I have this, and its not possible to input 36

 

IMPORTANT: I just installed the GIGABYTE - AORUS Gen4 AIC Adaptor, PCIe 4.0 GC-4XM2G4 in the 8x bifurcation slot and it dosent work on this MB, no matter what I do, even setting it to 16x and removing all card it only sees 1 Nvme 😞
IF ANYONE HERE have a Nvme expansion card that works on this Motherboard then please write me! 🙂

 

 

Edited January 13 by casperse

[firstTimer]

On 1/12/2024 at 12:50 PM, casperse said:

 

Great write up!
I just got the MB but with a Bios: v. 3101
And I cant find a way to set the (1-2) AI Tweaker > Efficiency Core Ratio: By Core Usage and for all E-Core apply 36
Anymore instead I have this, and its not possible to input 36

 

IMPORTANT: I just installed the GIGABYTE - AORUS Gen4 AIC Adaptor, PCIe 4.0 GC-4XM2G4 in the 8x bifurcation slot and it dosent work on this MB, no matter what I do, even setting it to 16x and removing all card it only sees 1 Nvme 😞
IF ANYONE HERE have a Nvme expansion card that works on this Motherboard then please write me! 🙂

 

 

I still am on the previous bios, 2703 if I remember correctly. I could update the bios and give it a check, but did you have to have a Windows 11 on a USB key and update first the intel Engine? On the asus support site it was said to first update the Intel engine and then update the bios accordingly? WHat did you do to update the bios? And as a little side note, I "decreased" my CPU to an intel i5-13400.

Edited January 15 by firstTimer

[casperse]

14 hours ago, firstTimer said:

I still am on the previous bios, 2703 if I remember correctly. I could update the bios and give it a check, but did you have to have a Windows 11 on a USB key and update first the intel Engine? On the asus support site it was said to first update the Intel engine and then update the bios accordingly? WHat did you do to update the bios? And as a little side note, I "decreased" my CPU to an intel i5-13400.

No and Yes - did the Bios update first and then booted a windows 11 to update with the "MEUpdateTool" it worked but I did it in the wrong order.

Why the i5? My thought is that with a platform that is end of live now (This socket will not get a new CPU) I might as Weel "future proof it" and with the tweaks I currently have temp at 36-52 C doing Intel burning test tool - haven't measured the power usage yet. Passmark CPU gave me around 30.000 and the official number is above 60.000 so I might turn it up a bit. (Hmm my existing server had a passmark score of 13586 so Maybe I should just leave it as it is :-) )
 

[firstTimer]

11 hours ago, casperse said:

No and Yes - did the Bios update first and then booted a windows 11 to update with the "MEUpdateTool" it worked but I did it in the wrong order.

Why the i5? My thought is that with a platform that is end of live now (This socket will not get a new CPU) I might as Weel "future proof it" and with the tweaks I currently have temp at 36-52 C doing Intel burning test tool - haven't measured the power usage yet. Passmark CPU gave me around 30.000 and the official number is above 60.000 so I might turn it up a bit. (Hmm my existing server had a passmark score of 13586 so Maybe I should just leave it as it is :-) )
 

Ok, when I have some time (probably next weeks) I will update the bios and give it a try. Actually I downgraded because I thought to spare much more energy but in the end, the difference is negligible, but when I found it out, it was too late. At the moment my system is idling at 45w so 10-15w less than before

Edited January 16 by firstTimer

[engin33rh3r3]

On 1/12/2024 at 5:50 AM, casperse said:

 

Great write up!
I just got the MB but with a Bios: v. 3101
And I cant find a way to set the (1-2) AI Tweaker > Efficiency Core Ratio: By Core Usage and for all E-Core apply 36
Anymore instead I have this, and its not possible to input 36

 

IMPORTANT: I just installed the GIGABYTE - AORUS Gen4 AIC Adaptor, PCIe 4.0 GC-4XM2G4 in the 8x bifurcation slot and it dosent work on this MB, no matter what I do, even setting it to 16x and removing all card it only sees 1 Nvme 😞
IF ANYONE HERE have a Nvme expansion card that works on this Motherboard then please write me! 🙂

Did you find a way to make the NVME expansion card work? Very interested.