I have spent the past weeks trying to optimize the power consumption of my server, and I have finally had success.
I need to preface this, I am not using UnRaid, but I have been lurking in this thread for a long time, and wanted to share my results with the community. I am using Debian, so I believe it should be transferable.
Relevant components:
Intel i5 13500
ASRock B760 Pro
LSI 9400-16i
A bunch of HDDs
NVME drive
Realtek 2.5G NIC
What I did
I will do my best to remember everything, as there has been a lot of trial and error, so don't consider this a guide, but rather a collection of things I had to do to finally solve this.
Enable every power saving feature you can in the bios, ensure all the C-states are enabled, intel speedstep, etc.
Enable ASPM in bios (L1 is max power saving, L0 least).
For my case, I had to edit my BIOS using SCEWIN. Using this tool, I was able to disable Multi-VC on all root ports (see here) and also enable further power saving functionality relevant only to ASRock (see here)
Avoid the use of any PCI-E directly connected to the CPU. This means using the NVME slot connected to the chipset, as well as the PCI-E x4 slot for my HBA (not the top x16 slot). This does limit the overall bandwidth of the HBA to 4GB/s instead of 8GB/s, but is still enough for spinning rust.
Ensure ASPM is supported by your PCI-E devices. The LSI 9400 does support ASPM L1.
For me, ASPM was disabled out of the gate when I booted my system. I used the following script to forcefully enable ASPM on the ports where it was not enabled. I had to enable it for both my LSI card AND the root ports themselves (as both were disabled when the LSI card was used). What this means is that the root complex and endpoint should be the same when editing the ASPM level of the port, while you change endpoint to target your LSI card as necessary.
I am not using the R8125 driver, as that is not working with the latest kernel. Instead I use the default R8169 driver, but explicitly enabled ASPM by using the following line:
echo 1 > /sys/bus/pci/devices/0000:04:00.0/link/l1_aspm
Run powertop (see below) or any equivalent script to set the basic power saving features.
powertop --quiet --auto-tune
Additionally, you can save a little bit of power through some additional commands:
echo "powersupersave" > /sys/module/pcie_aspm/parameters/policy;
# See https://wiki.archlinux.org/title/CPU_frequency_scaling for details
cpupower frequency-set -g powersave
cpupower idle-set -E
cpupower set -b 15
echo "balance_power" | tee /sys/devices/system/cpu/cpu*/cpufreq/energy_performance_preference
Disabled onboard sata controller (everything goes through the HBA)
All of this has lead to the final culmination of my system finally hitting C10 pkg states.
Ensure that whatever you plug into your PCI-E ports support ASPM. Older LSI cards typically do not support ASPM, which means you're out of luck.
The following guide helped me a lot in debugging and gaining a broader understanding into this whole mess. If you don't want to read the whole thing, at least use the following commands to check if your LSI card does support ASPM:
> sudo apt-get install acpica-tools
> sudo acpidump -b -n FACP
> iasl -d facp.dat
Then look for the line that says PCIe ASPM Not Supported (V4). If that says 1, you're out of luck. If it says 0, you're good to go.
I hope this helps somebody out there.