[13600k Idle States]

10 hours ago, Syaoran68 said:

iono i feel like every manufacturer will have something... in the end its not THAT bad.. i'm on a fully stock 13600k with 5.2ghz boost running anywhere between 60-80 watts of power with like 25+ docker containers.. (one is a project zomboid container that runs constantly..) and 4 drives(1 drive is spun down most of the time) + 1 NVME so i'm not too concerned.. Even with like 6-10 plex streams it'll peak at like 110 for a second as it loads then drop back down to 60/70watts. I guess i can't complain haha

thanks for the help though!

Yeah every manufacturer will have it's quirks. The other ATX mobo I tried was an AsRock. Their power saving features in the BIOS isn's as robust compared to ASUS IMO. But for power saving, it's not manufacturer dependent but more like the board size/feature set. My theory is that ATX just have so many wiring and intricacies compared to ITX and mATX. I haven't tried an ATX H series mobo though so who knows, but yes I agree, it's not THAT bad hence why I haven't bought another mATX mobo lol. Besides like I mentioned, Z-Wave is somehow limiting my CStates.

 

And np!

[13600k Idle States]

On 5/19/2023 at 12:27 PM, Syaoran68 said:

how is your experience on that TUF series board? i managed to get power usage down to about 72 watts when full idle with 20 dockers alive and 4 HDs connected 1 spun down. had to update to the new 6.12 RCs to see if the new kernel would make a difference on the C state but so far nothing.. 

As a motherboard it's fine. I may use it for my gaming PC instead, but I do think if you want max power savings, I would not recommend it or any other ATX motherboards.

[13600k Idle States]

From my experience (I tried 3 Z790 mobos now), only the Asus Prime Z790m Plus D4 was able to achieve C10 states. My theory is full size ATX motherboards have a lot more features/circuitry going on compared to mATX and ITX mobos. In the end, it didn't matter and I ended up settling with the Tuf Z790 Plus Wifi D4 because zwave-js-ui was preventing deeper C States. I'm also thinking of putting a dedicated GPU.

[unRAID plugin for iGPU SR-IOV support]

On 5/1/2023 at 8:57 PM, FEILUAN said:

The plugin cannot work on 6.12.0-rc5

The installation run failed

----------Downloading i915-sriov module  for kernel v6.1.27---------- -

 

--------This could take some time, please don't close this window!-----------

--------Can't download i915-sriov module Package for kernel v6.1.27----------

- plugin: run failed: '/bin/bash' returned 1 Executing hook script: post_plugin_checks

Mine installed just fine, but this plugin is causing incompatibility issues with intel_gpu_top

[error using intel_gpu_top]

10 hours ago, ich777 said:

Please uninstall the SRIOV plugin, reboot and it will work.

 

SRIOV is in a experimental state and not officially supported by the Kernel:

Source: https://github.com/strongtz/i915-sriov-dkms/blob/master/README.md

Can confirm it worked after uninstalling SRIOV plugin

[error using intel_gpu_top]

8 hours ago, bubbadk said:

Wonder if the programmers of unraid knows about this. I dread everytime I need to reboot on what had been broken.. I'm on 6.11 and don't have the nerves to upgrade. 

I had intel_gpu_top running fine for awhile, but I was doing a lot of hardware and software all at once and I'm not sure which one broke it. I did took of the GPU though and intel_gpu_top is still not working. @ich777 may be able to point us to the right direction.

[error using intel_gpu_top]

37 minutes ago, bubbadk said:

I reinstalled unraid because I thought it was a one time incident. But yesterday I rebooted unraid, and the error was there again. Segment fault. 

 

Hw transcode still works though.. But not Intel_gpu_top 

Yeah I am facing this issue as well. HW transcode wasn't working for me because I had my 1050ti on it. I removed it, and now HW transcode works again.. I am on RC5 btw.

[error using intel_gpu_top]

On 4/26/2023 at 9:46 PM, bubbadk said:

hi gang.

 

Nevermind. made a complete reinstall

 

Im on a intel nuc generation 11 i7

 

the first time i installed intel gpu top everything worked fine. Then i rebooted and i get ithis error.

 

as you can see i have /dev/dri and everything is thee. but i can't seem to getting it work

 

 

root@Tower:~# intel_gpu_top
Failed to detect engines! (No such file or directory)
(Kernel 4.16 or newer is required for i915 PMU support.)
Segmentation fault
root@Tower:~# cd /dev/dri
root@Tower:/dev/dri# ls
by-path/  card0  card1  card2  renderD128  renderD129  renderD130
root@Tower:/dev/dri#

 

 

the picture is from fresh install without the first boot

reinstalled unRAID or intel_gpu_top?

[Reduce power consumption with powertop]

On 4/29/2023 at 12:33 PM, TheLinuxGuy said:

 

Yeah I faced problems with proxmox GUI installer; I had to do CTRL+ALT+F2 to get console then add some forced configuration override to properly detect the iGPU.

Here's the guide for the error (embedded framebuffer): https://forum.proxmox.com/threads/generic-solution-when-install-gets-framebuffer-mode-fails.111577/

 

Speaking of unRAID version, what were you using before? --- Stable was the version I was using previously during my tests.

 

I do confirm, unRAID C10 is unstable with this board I do not know why that is... Proxmox with kernel 6.0 is stable and will 90%+ stick in C10 with no vms running.

I made a VM inside proxmox to boot my unraid RC4 usb and it worked, I only played with it for like an hour last night but from my early observations once unraid VM boots and no longer uses resources - the parent proxmox host will keep C10 stable - until you start fiddling around unraid web interface to do stuff then it cycles thru higher C states (but thats expected I think).

 

I'm leaning towards having proxmox as my base OS and unraid as the VM with passthru for the SATA controller (since proxmox OS is installed on 2TB nvme) but will do more testing this evening.

Thanks for the info!

[Reduce power consumption with powertop]

11 hours ago, TheLinuxGuy said:

 

I have achieved C10 (~5 watt) on the Asus Prime Z790M-Plus D4... but there's a catch. I got there by installing Proxmox + kernel 6.xx + powertop 2.15 (from source) + scaling governor set to powersave.

 

 

root@dom:/# uname -mrs
Linux 6.0.19-edge x86_64
root@dom:/# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"

 

I wonder if unraid's kernel was newer if we would be able to get to C10 natively as I am seeing its possible here... 

Also I confirm that Intel X710-DA2 C10 is also achieved under Debian 11/Proxmox kernel 6.0 - the power consumption with 10GB x710 is ~8.7 watt idle.

 

*edit... decided to test rc4 after reading it uses kernel 6.1* results are very close (proxmox is more efficient by 3 watt for some reason - I think C10 is stable in proxmox since unraid is 40% at C10 20% at C8 and random stuff in other states) - 12 watt idle with X710-DA2 connected on unraid 6.12-rc4

 

root@Tower:~# cat /etc/unraid-version
version="6.12.0-rc4.1"
root@Tower:~# uname -mrs
Linux 6.1.26-Unraid x86_64

I was actually going to go Proxmox route before unRAID but I couldn't get the installer working. Mine was just getting stuck at the installer.. Got a guide somewhere?

 

Speaking of unRAID version, what were you using before? My first install with the Asus Prime Z790M-P D4 was RC2. I was able to reach 5sh watts at C10 but it's not constant and was more like 8w on average.

 

Man I wish I got the Intel NIC instead since those 8w with it installed is good! My thought with the CX4 is that it's SFP28 and it's more future proof in the event I upgrade my switch in the future.

 

Also more data points. I decided to try the be  quiet! Pure Power M 850W PSU because I'm think of using my unRAID server to host a gaming PC VM. To my surprise, idle power is the same (if not lower since I added an extra fan) as the PICO PSU. Dark Power 13 850W may probably even more efficient but I don't want to spend more $$ lol.

[Reduce power consumption with powertop]

55 minutes ago, mgutt said:

It seems no 600/700 chipset mobo reaches C9/C10. In the meantime I got feedback for multiple B660 and B760 and all of them stuck in C8. Some of them even reach only C3 or C6 🤨

I think only ITX and mATX boards only though. I've tried 3 Z790 mobos, and only the Asus Prime Z790M-P D4 was able to achieve C10. Both my Asrock and Tuf ATX motherboards were only able to achieve C6 max.

[Reduce power consumption with powertop]

15 hours ago, TheLinuxGuy said:

 

Thank you for confirming that you rarely saw the system go below C8. This is probably as good as it gets with this board, I am happy with it. My other B660M build eats 30 watts idle while this one is 1/3 - same cpu generation too, different chipset and board manufacturer (AsRock vs. Asus)

 

These should be the settings an owner of an Asus Prime Z790M-Plus D4 needs to set to match my power measurements below (unraid usb plugged in + 2TB NVME drive - hdmi + kb/mouse unplugged).

 

I'm still lost on where to set the CPU multiplier in this asus bios (but did set PL1 and PL2 to 35w) - something to look at this weekend I guess when the intel x710 arrives in the mail.

 

*edit note: apparently summary screenshots below are not showing but there are some settings in BIOS with the option "Max battery" and also noticed the scroll window hid my other APM settings where I am forcing L1 on everything plus S4+S5 and CPU deep c-states...

I honestly gave up on achieving the deepest C-States.. Even if I get the X710, once I run my Home Assistant and zwavejsui container, C-States immediately drop to C2. I guess the Zooz Zwave dongle probably is the culprit, not necessarily the zwavejsui, but I do use quite a few ZWave devices so I can't just drop ZWave.. Oh well.

[Reduce power consumption with powertop]

Well unfortunately @TheLinuxGuy, I don't have that mobo anymore. I broke it while I was reinstalling my CX4.... LOL. The mobo that I originally got to replace the Asus Prime Z790M-Plus D4, wasn't letting me go into deeper CStates, so I returned that for an Asus Tuf Z790-Plus D4, which also wasn't letting me go into deeper states. So yeah full size ATX mobo's are no go, especially for Z790. Anyways, those seem right.. I have a 13600k so I'm not sure if that's enough variance... What I do remember though is I set Multicore enhancement to disabled, and there was a setting for "Best Case Scenario" (i forget which one it was...). Anything motherboard vendor specific settings was basically turned off, and I used mostly Intel's default settings. I also capped my PL1 to 35W and PL2 at 125W, and unplugged my HDMI cable.

 

Forgot to mention that I was only able to achieve C10 when I let it sit long enough, but in reality, it was mostly at C8, similar to what you have. Oh and for ASPM settings, I used C10 as max, not Auto or CPU Default.

 

More edits: Install corefreq-cli, it should show you frequencies at faster poll rate.

[ACPI Error fills up syslog fast]

On 4/13/2023 at 6:44 AM, ezek1el3000 said:

I removed the GPU Statistics Plugin by b3rs3rk and the error doesn't pop up anymore in my syslog. Syslog got spammed while i was on the Dashboard. On every other Tab, no errors. So I thought if PEG1 is mentioned in the error message then it should have something to do with the GPU.

I was having an:

 

unRAID kernel: ACPI BIOS Error (bug): Failure creating named object [\_SB.PC00.PEG1.PEGP._DSM.USRG], AE_ALREADY_EXISTS (20220331/dsfield-184)

 

Removing the GPU Statistics Plugin solved that bug as well.

[HOW TO: Using SR-IOV in UnRAID with 1Gb/10Gb/40Gb network interface cards (NICs)]

Ok, I found another way of doing this. Not sure if it's better or worse, but it works.

 

# create and reload udev-rules 
cat << UDEVRULES >> /etc/udev/rules.d/99-sriov.rules
ACTION=="add", SUBSYSTEM=="net", ATTRS{vendor}=="0x15b3", ATTRS{device}=="0x1015", ATTR{device/sriov_drivers_autoprobe}="0", ATTR{device/sriov_numvfs}="8"
UDEVRULES

udevadm control --reload
udevadm trigger --action=add

 

References:

1. How to use the mlx5_core driver with Mellanox ConnectX-4 Lx in Debian? | ServeTheHome Forums

2. https://forums.unraid.net/topic/109081-udev-regeln-in-unraid/?do=findComment&comment=996973

 

[HOW TO: Using SR-IOV in UnRAID with 1Gb/10Gb/40Gb network interface cards (NICs)]

Bumping this thread to post an update. I have a Connectx-4 LX so the first method I tried was @jortan which is:

 

1. Check driver: mlx5_core

2. Created a file: /boot/config/modprobe.d/mlx5_core.conf

3. added: options mlx5_core num_vfs=8

However, I get an error with mlx5_core num_vfs config not found (something like that) when I reboot unraid.

 

So what I ended up doing is a hybrid of @BVD and @jortan

 

1. I first did this:

 

On 2/28/2021 at 11:30 AM, BVD said:

Driver / Device specific steps

 

The following is the most generic option, and should work for most UnRAID deployments that contain SR-IOV supporting NICs, going back to around 6.4, but I would recommend no lower than 6.8.2 if you're working with any device using the i40e driver (save yourself the pain and upgrade!):

 

1. Open your terminal and edit the go file

   nano /boot/config/go

2. Add the following line to the bottom, specifying the number of vf's to create for this interface, replacing my ID (17.:0.1) with your own - I chose 4 per interface:

   echo 4 > /sys/bus/pci/devices/0000:17:00.3/sriov_numvfs
   echo 4 > /sys/bus/pci/devices/0000:17:00.3/sriov_numvfs

3. Hit 'Ctrl+x' , then 'Y', then Enter (just following the on screen prompts to save the file), and it's time for a reboot (one of the joys of UnRAID!)
    
4. Now that your system is back up and running, head to the system devices screen (Tools -> System Devices) - you should see something pretty:
   
   

 

2. then:

 

On 3/4/2021 at 6:56 AM, jortan said:

Bind to vfio on startup

 

Option 1 - unRAID GUI

 

In unRAID Tools \ System Devices, select all the "Virtual Function" devices and click Bind Selected to vfio at boot

 

 

If we do nothing else, this will fail because vfio-pci script is run before device drivers are loaded as noted here

 

Tell unRAID to re-run the vfio-pci script again (but after device drivers have loaded) by calling it in the /boot/config/go file:

 

# Relaunch vfio-pci script to bind virtual function adapters that didn't exist at boot time
/usr/local/sbin/vfio-pci >>/var/log/vfio-pci

 

Reboot.

 

If you check Tools \ System Devices \ View vfio logs, you should see the first run where the bindings failed, and then a second run where the bindings succeeded.

 

So now I have this: 

 

 

I still have to play with the Permanent MAC Address but I'm just glad it worked after spending hours on this.

[Reduce power consumption with powertop]

Quick update... I ended up bricking my motherboard LOL! 🤣. Are mATX boards really that more power efficient vs ATX? I'm about to go to Micro Center to get a new mobo.

 

EDIT: I will just go with an ASRock Z790 PG Lightning. The PCIEx1 slot I think is notched so therefore I can just install my CX4 in there if it is.. Modding my Asus mobo to make the CX4 fit was a BAD idea lolol.

EDIT 2: My idle with the new motherboard is the same despite it being larger in size and have more features (ie 2.5G port, which is plugged in). The CX4 is plugged in a PCIEx1 slot, and bandwidth is limited. I'll play more with this motherboard connections. I may end up just plugging it back to a x16 slot (or get those Intel NIC).

[Reduce power consumption with powertop]

7 hours ago, mgutt said:

This is a huge difference. Much more than only resulting from bad c-states. 

 

I've tested X550 and X710:

https://forums.unraid.net/topic/105909-mein-10-zoll-server/?do=findComment&comment=1082756

 

If possible you should use an Intel card with SFP+ and DAC cables. Those are the most power efficient. If your board has an open X1 Slot, I would use this. One lane is almost enough for 10G and less lanes save even more energy. Or you try for tape the additional pins of the card to throttle it to X1. You could try the same for your current card.

 

 

Oh I haven't thought about that.. I'll try the bottom slot.. If it still doesn't idle efficiently, I'll just get one of those Intel NIC you mentioned.

 

6 hours ago, TheLinuxGuy said:

 

wow impressive. I have the 13600k on a different board B660M chipset and the most I was able to squeeze was 40 watt idle. I think I am going to buy this motherboard to see if the newer chipset makes a difference.

I was going to build my second NAS anyway and need a board like this  

I was pretty surprised myself when I achieved that. The only other computers in my house that can achieve that on idle are my Apple Silicon Macs

[Reduce power consumption with powertop]

Long time lurker, and I have been reading this thread for some time now. I was able to get a deep idle power of around 7W - 8W with the following build. The 8W was achieved with the array running, but with no active Docker containers. With 2 running containers (Plex and HomeBridge) the idle power is about 20W. If I remove the other drives I'm not using, I'm sure I can lower the deep idle wattage even more.

 

MB: Asus Prime Z790M-Plus

CPU: Intel 13600K

CPU cooler: Noctua NH-D15 (1 fan)

RAM: CORSAIR Vengeance LPX 32GB (2PK x 16GB) 3.2GHz

PSU: RGEEK 12V 300W Pico ATX

NVME 1: Crucial P3 4TB

NVME 2: Samsung 980 Pro 500GB

NVME 3: Inland Performance 1TB

SSD 1: SanDisk Ultra II 960GB

Boot drive: Kingston DataTraveler 2.0

Ethernet: Built-in 1Gbit

Case fans: 2

Idle: ~8W (C10)

Idle with running Docker container: ~20w (C6)

Idle with Connectx-4 LX and Docker containers running: 38w (C2)

 

All is well thanks to following this guide, but how the heck do I let my CX4 go into deeper C-State? It's plugged to my Mac using a DAC and should be inactive when my Mac is off..

 

@mgutt

 

Quote

PCIe cards

Some PCIe cards prevent states other than C0 or C3 (Intel). As an example my Qnap Aquantia 10G card allowed only C3 while all my Intel 10G cards allowed C7. Without any installed card I'm able to reach C10.

 

Which Intel card? X520??