October 15, 20214 yr ok, thank you. I will try this. With ubuntu on NVME there was no c7, but ubuntu on sata and nvme as second drive it works with 3,9-4,2W idle In BIOS Settings is nothing with ASPM Edited October 15, 20214 yr by TraumG
October 15, 20214 yr ^Now it chage to C3. Long way to c7. Next step? another point is, that the averaage clock in powertop ist 3,6 GHz. In Ubunte there where lower clocks. IN "cpufreq-info" it shows 3,6 Ghz7 , too. Spoiler Edited October 15, 20214 yr by TraumG
October 15, 20214 yr Author 2 hours ago, TraumG said: another point is, that the averaage clock in powertop ist 3,6 GHz. You could try to set Powersave with the tips and tweaks Plugin, which reduces the frequency, but it should not really help with the package c states. The most important part is ASPM.
October 15, 20214 yr I tried Powersave, but there was no change in c-state. Power consumtion may be a litte bit. 0,5W or so...
October 15, 20214 yr Author Did you try to obtain the aspm status of your devices? lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM )' Every device with "ASPM disabled" is your enemy 😉
October 16, 20214 yr HOw can I copy the hole output? It copies only one line. Edited October 16, 20214 yr by TraumG
October 16, 20214 yr I have two enemies: The other 100% are no Problem? Ist 8169 the wrong driver? See https://www.linuxquestions.org/questions/linux-hardware-18/realtek-rtl8111-8168-8411-ethernet-controller-r8168-driver-install-r8169-driver-doesn't-work-4175641982/ I did use the ethernet for more than web interface, but I think there are no stability problems. Edited October 16, 20214 yr by TraumG
October 16, 20214 yr Author On 10/16/2021 at 9:48 AM, TraumG said: HOw can I copy the hole output? Are you using the web terminal? Select everything, right mouse click and copy. Or use this command to write the output to a file and download it from your USB flash drive: lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM )' > /boot/logs/aspm.log
October 16, 20214 yr Author 6 hours ago, TraumG said: I have two enemies: The other 100% are no Problem? Ist 8169 the wrong driver? I don't understand what you are talking about. ^^
October 16, 20214 yr 17 hours ago, mgutt said: lspci -vv | grep -B30 'ASPM.*abled' Every device with "ASPM disabled" is your enemy 😉 The both red marked ones had the result "disabled"
October 18, 20214 yr Author On 10/16/2021 at 10:56 AM, TraumG said: 8169 the wrong driver? Which driver uses Ubuntu?
October 19, 20214 yr Looks like 8168 Spoiler Tino, [20.10.21 00:46] 00:1d.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #12 (rev f0) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin D routed to IRQ 123 Bus: primary=00, secondary=02, subordinate=02, sec-latency=0 I/O behind bridge: 00003000-00003fff [size=4K] Memory behind bridge: 8f800000-8f8fffff [size=1M] Prefetchable memory behind bridge: 0000006000000000-00000060000fffff [size=1M] Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0 ExtTag- RBE+ DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+ RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 128 bytes DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend- LnkCap: Port #12, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <16us ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+ LnkCtl: <font color="#CC0000"><b>ASPM L1 Enabled; RCB 64 bytes Disabled</b></font>- CommClk+ <font color="#06989A">--</font> Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Region 0: Memory at fe010000 (32-bit, non-prefetchable) [size=4K] 01:00.0 Non-Volatile memory controller: Sandisk Corp Device 5009 (rev 01) (prog-if 02 [NVM Express]) Subsystem: Sandisk Corp Device 5009 Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 16 NUMA node: 0 Region 0: Memory at 8f900000 (64-bit, non-prefetchable) [size=16K] Region 4: Memory at 8f904000 (64-bit, non-prefetchable) [size=256] Capabilities: [80] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [90] MSI: Enable- Count=1/32 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [b0] MSI-X: Enable+ Count=17 Masked- Vector table: BAR=0 offset=00002000 PBA: BAR=4 offset=00000000 Capabilities: [c0] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <1us, L1 unlimited ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+ RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset- MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <8us ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: <font color="#CC0000"><b>ASPM L1 Enabled; RCB 64 bytes Disabled</b></font>- CommClk+ pcilib: sysfs_read_vpd: read failed: Input/output error <font color="#06989A">--</font> PortCommonModeRestoreTime=32us PortTPowerOnTime=10us L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- T_CommonMode=0us LTR1.2_Threshold=81920ns L1SubCtl2: T_PwrOn=10us Kernel driver in use: nvme Kernel modules: nvme Tino, [20.10.21 00:46] 02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c) Subsystem: Fujitsu Technology Solutions RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 19 Region 0: I/O ports at 3000 [size=256] Region 2: Memory at 8f800000 (64-bit, non-prefetchable) [size=4K] Region 4: Memory at 6000000000 (64-bit, prefetchable) [size=16K] Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [70] Express (v2) Endpoint, MSI 01 DevCap: MaxPayload 128 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 10.000W DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq- RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 128 bytes, MaxReadReq 4096 bytes DevSta: CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: <font color="#CC0000"><b>ASPM L1 Enabled; RCB 64 bytes Disabled</b></font>- CommClk+ <font color="#4E9A06"><b>t@pfeil</b></font>:<font color="#3465A4"><b>~</b></font>$ </pre> Edited October 19, 20214 yr by TraumG
November 14, 20214 yr Author Only as an information: I found a better command to get the ASPM status of supporting devices: lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM )' Output:
November 14, 20214 yr Author Unraid uses the following ASPM kernel settings (which are normal): grep ASPM /usr/src/linux-5.10.28-Unraid/.config CONFIG_PCIEASPM=y CONFIG_PCIEASPM_DEFAULT=y # CONFIG_PCIEASPM_POWERSAVE is not set # CONFIG_PCIEASPM_POWER_SUPERSAVE is not set # CONFIG_PCIEASPM_PERFORMANCE is not set
January 1, 20224 yr Community Expert On 11/14/2021 at 6:38 PM, mgutt said: Only as an information: I found a better command to get the ASPM status of supporting devices: lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM )' root@Unraid-1:~# lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM )' 00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 05) (prog-if 00 [Normal decode]) LnkCap: Port #2, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <256ns, L1 <8us LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ 00:1b.0 PCI bridge: Intel Corporation Comet Lake PCI Express Root Port #21 (rev f0) (prog-if 00 [Normal decode]) LnkCap: Port #21, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <1us, L1 <16us LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+ 00:1d.0 PCI bridge: Intel Corporation Comet Lake PCI Express Root Port #9 (rev f0) (prog-if 00 [Normal decode]) LnkCap: Port #9, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s unlimited, L1 <16us LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk- pcilib: sysfs_read_vpd: read failed: No such device 01:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03) LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s <64ns LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ 02:00.0 PCI bridge: ASMedia Technology Inc. ASM1184e 4-Port PCIe x1 Gen2 Packet Switch (prog-if 00 [Normal decode]) LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 unlimited LnkCtl: ASPM L0s L1 Enabled; Disabled- CommClk+ 03:01.0 PCI bridge: ASMedia Technology Inc. ASM1184e 4-Port PCIe x1 Gen2 Packet Switch (prog-if 00 [Normal decode]) LnkCap: Port #1, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 unlimited LnkCtl: ASPM L1 Enabled; Disabled- CommClk+ 03:03.0 PCI bridge: ASMedia Technology Inc. ASM1184e 4-Port PCIe x1 Gen2 Packet Switch (prog-if 00 [Normal decode]) LnkCap: Port #3, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 unlimited LnkCtl: ASPM L0s L1 Enabled; Disabled- CommClk- 03:05.0 PCI bridge: ASMedia Technology Inc. ASM1184e 4-Port PCIe x1 Gen2 Packet Switch (prog-if 00 [Normal decode]) LnkCap: Port #5, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 unlimited LnkCtl: ASPM L1 Enabled; Disabled- CommClk+ 03:07.0 PCI bridge: ASMedia Technology Inc. ASM1184e 4-Port PCIe x1 Gen2 Packet Switch (prog-if 00 [Normal decode]) LnkCap: Port #7, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 unlimited LnkCtl: ASPM L1 Enabled; Disabled- CommClk+ 04:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch (rev c1) (prog-if 00 [Normal decode]) LnkCap: Port #1, Speed 16GT/s, Width x8, ASPM L1, Exit Latency L1 <64us LnkCtl: ASPM L1 Enabled; Disabled- CommClk+ 05:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch (prog-if 00 [Normal decode]) LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us LnkCtl: ASPM L0s L1 Enabled; Disabled- CommClk+ 06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 23 [Radeon RX 6600/6600 XT/6600M] (rev c1) (prog-if 00 [VGA controller]) LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+ 06:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT] LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+ 08:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch (rev c1) (prog-if 00 [Normal decode]) LnkCap: Port #5, Speed 16GT/s, Width x8, ASPM L1, Exit Latency L1 <64us LnkCtl: ASPM L1 Enabled; Disabled- CommClk+ 09:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch (prog-if 00 [Normal decode]) LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us LnkCtl: ASPM L0s L1 Enabled; Disabled- CommClk+ 0a:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 23 [Radeon RX 6600/6600 XT/6600M] (rev c1) (prog-if 00 [VGA controller]) LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+ 0a:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT] LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+ 0b:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch (rev c1) (prog-if 00 [Normal decode]) LnkCap: Port #7, Speed 16GT/s, Width x8, ASPM L1, Exit Latency L1 <64us LnkCtl: ASPM L1 Enabled; Disabled- CommClk+ 0c:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch (prog-if 00 [Normal decode]) LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us LnkCtl: ASPM L0s L1 Enabled; Disabled- CommClk+ 0d:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 23 [Radeon RX 6600/6600 XT/6600M] (rev c1) (prog-if 00 [VGA controller]) LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+ 0d:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT] LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+
July 30, 20223 yr Sorry to bring a thread back from the dead; I decided that I wanted to spend time trying to minimize idle power-usage on my Supermicro Server (X9DRi-LN4F+ motherboard). I followed most of the advice here; have a handful of devices that ASPM is still disabled for here; hoping I could get some suggestions on how to address them: root@Tower:~# lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM )' 00:00.0 Host bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 DMI2 (rev 04) LnkCap: Port #0, Speed 2.5GT/s, Width x4, ASPM L1, Exit Latency L1 <16us LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk- 00:01.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1a (rev 04) (prog-if 00 [Normal decode]) LnkCap: Port #0, Speed 2.5GT/s, Width x4, ASPM L1, Exit Latency L1 <16us LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk- 00:01.1 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1b (rev 04) (prog-if 00 [Normal decode]) LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <16us LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ 00:02.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2a (rev 04) (prog-if 00 [Normal decode]) LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L1, Exit Latency L1 <16us LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ 00:03.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3a (rev 04) (prog-if 00 [Normal decode]) LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L1, Exit Latency L1 <16us LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ 00:11.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Virtual Root Port (rev 06) (prog-if 00 [Normal decode]) LnkCap: Port #17, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+ 00:1c.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 1 (rev b6) (prog-if 00 [Normal decode]) LnkCap: Port #1, Speed 5GT/s, Width x4, ASPM L1, Exit Latency L1 <4us LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ pcilib: sysfs_read_vpd: read failed: No such device 02:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05) LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L0s, Exit Latency L0s <64ns LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ 03:00.0 VGA compatible controller: NVIDIA Corporation TU104GL [Quadro RTX 4000] (rev a1) (prog-if 00 [VGA controller]) LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <16us LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ 03:00.1 Audio device: NVIDIA Corporation TU104 HD Audio Controller (rev a1) LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+ 03:00.2 USB controller: NVIDIA Corporation TU104 USB 3.1 Host Controller (rev a1) (prog-if 30 [XHCI]) LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <16us LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ 03:00.3 Serial bus controller: NVIDIA Corporation TU104 USB Type-C UCSI Controller (rev a1) LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <16us LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ 04:00.0 VGA compatible controller: NVIDIA Corporation GP106GL [Quadro P2000] (rev a1) (prog-if 00 [VGA controller]) LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ 04:00.1 Audio device: NVIDIA Corporation GP106 High Definition Audio Controller (rev a1) LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ 05:00.0 Serial Attached SCSI controller: Intel Corporation C602 chipset 4-Port SATA Storage Control Unit (rev 06) LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+ 06:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <32us LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ 06:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <32us LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ 06:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <32us LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ 06:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01) LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <32us LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ 80:00.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port in DMI2 Mode (rev 04) (prog-if 00 [Normal decode]) LnkCap: Port #8, Speed 2.5GT/s, Width x48, ASPM L1, Exit Latency L1 <16us LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk- 80:01.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1a (rev 04) (prog-if 00 [Normal decode]) LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <16us LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ 80:02.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2a (rev 04) (prog-if 00 [Normal decode]) LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L1, Exit Latency L1 <16us LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk- 80:03.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3a (rev 04) (prog-if 00 [Normal decode]) LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L1, Exit Latency L1 <16us LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ 82:00.0 Non-Volatile memory controller: Intel Corporation SSD 660P Series (rev 03) (prog-if 02 [NVM Express]) LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <8us LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ 84:00.0 VGA compatible controller: NVIDIA Corporation TU104GL [Quadro RTX 4000] (rev a1) (prog-if 00 [VGA controller]) LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ 84:00.1 Audio device: NVIDIA Corporation TU104 HD Audio Controller (rev a1) LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ 84:00.2 USB controller: NVIDIA Corporation TU104 USB 3.1 Host Controller (rev a1) (prog-if 30 [XHCI]) LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ 84:00.3 Serial bus controller: NVIDIA Corporation TU104 USB Type-C UCSI Controller (rev a1) LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ Is this realistically the best that can be done with respect to ASPM enabling or is there further room for improvement?
July 30, 20223 yr Have you already done the usual solutions like bios setup and kernel boot options? If your system is setup incorrectly, it will not work. If you have everything, then you should check Ubuntu first, because Ubuntu does not have the unknown ASPM bug which Unraid has for my hardware. If you can get ASPM enabled on Ubuntu without hacks by using only bios and kernel boot options, then we will use your result to push forward. It will provide more evidence to help Unraid team to identify the cause of the issue. And with the Ubuntu test you could see how much power saving is actually possible. If there isn't any, I wouldn't go forward. And I would only recommend the ASPM hack (forced activation, which does bypass any compatibility checks) if you did confirm that other operating systems can enable it, then bypassing logic is a bit less dangerous. Edited July 30, 20223 yr by Falcosc
July 30, 20223 yr 2 minutes ago, Falcosc said: Have you already done the usual solutions like bios setup and kernel boot options? If your system is setup incorrectly, it will not work. If you have everything, then you should check Ubuntu first, because Ubuntu does not have the unknown ASPM bug which Unraid has for my hardware. If you can get ASPM enabled on Ubuntu without hacks, then we will use your result to push forward. It will provide more evidence to help Unraid team to identify the cause of the issue. And I would only recommend the ASPM hack (forced activation, which does bypass any compatibility checks) if you did confirm that other operating systems can enable it, then bypassing logic is a bit less dangerous. I set my BIOS ASPM option to "auto", here is the blurb in the manual about the options. I didn't actually see the Force L0 option. I also added pcie_aspm=force to the list of options in my /boot/syslinux/syslinux.cfg file root@Tower:~# cat /boot/syslinux/syslinux.cfg default menu.c32 menu title Lime Technology, Inc. prompt 0 timeout 50 label Unraid OS menu default kernel /bzimage append intel_iommu=on rd.driver.pre=vfio-pci video=vesafb:off,efifb:off isolcpus=4-9,24-29 vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot pcie_aspm=force label Unraid OS GUI Mode kernel /bzimage append isolcpus=4-9,24-29 vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot,/bzroot-gui label Unraid OS Safe Mode (no plugins, no GUI) kernel /bzimage append initrd=/bzroot unraidsafemode label Unraid OS GUI Safe Mode (no plugins) kernel /bzimage append initrd=/bzroot,/bzroot-gui unraidsafemode label Memtest86+ kernel /memtest Before the above two options, all entries when I did lspci showed ASPM disabled, with the above changes, only a handful of entries (primarily PCI bridges and my HBA) show ASPM as being disabled. I have no problem trying ubuntu; server is a bit tough to access but I'll make a go of it. Would the idea be to boot off of a liveUSB and then check ASPM status?
July 30, 20223 yr Yes, liveUSB does work fine. On my hardware, it did immediately work in ubuntu without having any boot options. After figuring this out, I did manually set the ASPM option in the registers via a startup script which is executed after unraid config did decide that it is not save/not supported to enable it. Here is the documentation which I did use to set the flags, it isn't the reference to the real specification, more a how-to https://wireless.wiki.kernel.org/en/users/documentation/aspm#enabling_aspm_with_setpci
July 30, 20223 yr @ogiif Ubuntu works, I encourage you to create a new bug for this issue. You can have a look at to see which kind of information are needed for evidence. But if Ubuntu doesn't work for your devices, then you have a real setup issue. There could a specific hardware combination which does prevent it or a configuration issue. For example, your LSI SAS2308 PCI-Express Fusion-MPT SAS-2 doesn't have L1 support at all (LnkCap). The 4 Port SATA on the other hand have it. Have an eye on that, this one device could prevent your CPU from sleep. And even if 9 of 10 links are sleeping, you won't see much power savings as long as a single device keeps your CPU busy on the bus. For that reason, I recommend checking power consumption on Ubuntu. You don't need to spend time on ASPM for the supported devices, as long as you have at least one thing which will not support it. I don't know if L0s is enough to see significant changes in the power consumption, I only know about the huge effect of a proper sleeping CPU after having everything in the deeper L1 sleep.
July 30, 20223 yr 12 minutes ago, Falcosc said: @ogiif Ubuntu works, I encourage you to create a new bug for this issue. You can have a look at to see which kind of information are needed for evidence. But if Ubuntu doesn't work for your devices, then you have a real setup issue. There could a specific hardware combination which does prevent it or a configuration issue. For example, your LSI SAS2308 PCI-Express Fusion-MPT SAS-2 doesn't have L1 support at all (LnkCap). The 4 Port SATA on the other hand have it. Have an eye on that, this one device could prevent your CPU from sleep. And even if 9 of 10 links are sleeping, you won't see much power savings as long as a single device keeps your CPU busy on the bus. For that reason, I recommend checking power consumption on Ubuntu. You don't need to spend time on ASPM for the supported devices, as long as you have at least one thing which will not support it. I don't know if L0s is enough to see significant changes in the power consumption, I only know about the huge effect of a proper sleeping CPU after having everything in the deeper L1 sleep. I'll give the LiveUSB a try in a bit and report back (might be a day or so, the garage where my server resides is absolutely cooking right now, and I don't really want to spend much time there). Regarding the HBA, is there a known HBA that can be flashed to IT mode that supports L1 support?
July 30, 20223 yr Check first what happens with the CPU state after enabling L0s before investigating this. And then compare the power draw without this card to see if it is worth to hunt for. With Ubunutu I was able to reach C8 state in powertop which resulted in a major power draw change of 4W. And don't forget to execute autotune with powertop
July 31, 20223 yr 5 hours ago, Falcosc said: Check first what happens with the CPU state after enabling L0s before investigating this. And then compare the power draw without this card to see if it is worth to hunt for. With Ubunutu I was able to reach C8 state in powertop which resulted in a major power draw change of 4W. And don't forget to execute autotune with powertop I don't think my E5-2680v2 processors can reach C8 state, would love to go into C7 tho!
January 1, 20233 yr I'm chasing some similar issues to this on my unraid setup 00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge (prog-if 00 [Normal decode]) LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L1, Exit Latency L1 <64us LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ 00:02.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne PCIe GPP Bridge (prog-if 00 [Normal decode]) LnkCap: Port #2, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <64us LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ 00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus (prog-if 00 [Normal decode]) LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ 00:08.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus (prog-if 00 [Normal decode]) LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ 01:00.0 VGA compatible controller: NVIDIA Corporation GP107GL [Quadro P620] (rev a1) (prog-if 00 [VGA controller]) LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <16us LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ 01:00.1 Audio device: NVIDIA Corporation GP107GL High Definition Audio Controller (rev a1) LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+ 02:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 43bc (rev 02) (prog-if 30 [XHCI]) LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 unlimited LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ 02:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] Device 43b8 (rev 02) (prog-if 01 [AHCI 1.0]) LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 unlimited LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ 02:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b3 (rev 02) (prog-if 00 [Normal decode]) LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 unlimited LnkCtl: ASPM Disabled; Disabled- CommClk+ 03:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02) (prog-if 00 [Normal decode]) LnkCap: Port #4, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 unlimited LnkCtl: ASPM Disabled; Disabled- CommClk- 03:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02) (prog-if 00 [Normal decode]) LnkCap: Port #5, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 unlimited LnkCtl: ASPM L0s Enabled; Disabled- CommClk+ 03:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02) (prog-if 00 [Normal decode]) LnkCap: Port #6, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 unlimited LnkCtl: ASPM L0s L1 Enabled; Disabled- CommClk+ 03:07.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02) (prog-if 00 [Normal decode]) LnkCap: Port #7, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 unlimited LnkCtl: ASPM Disabled; Disabled- CommClk+ 05:00.0 Network controller: Realtek Semiconductor Co., Ltd. RTL8192EE PCIe Wireless Network Adapter LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us LnkCtl: ASPM L0s Enabled; RCB 64 bytes, Disabled- CommClk+ pcilib: sysfs_read_vpd: read failed: No such device 06:00.0 SATA controller: ASMedia Technology Inc. ASM1166 Serial ATA Controller (rev 02) (prog-if 01 [AHCI 1.0]) LnkCap: Port #0, Speed 8GT/s, Width x2, ASPM L0s L1, Exit Latency L0s <4us, L1 <64us LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+ 07:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c) LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ 08:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cezanne [Radeon Vega Series / Radeon Vega Mobile Series] (rev c9) (prog-if 00 [VGA controller]) LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ 08:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Renoir Radeon High Definition Audio Controller LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ 08:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ 08:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne USB 3.1 (prog-if 30 [XHCI]) LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ 08:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne USB 3.1 (prog-if 30 [XHCI]) LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ 09:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 81) (prog-if 01 [AHCI 1.0]) LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ 09:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 81) (prog-if 01 [AHCI 1.0]) LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ every device has L0s and L1 available but nearly all of the AMD chipset ones show as disabled Powertop only shows C1,C2,C3 - nothing else I've tried everything I could think of - forced ASPM on, set all the energy saving settings in bios My idling power usage of the server is around 90w 😢 System is a Gigabyte A320 with a Ryzen 5600G Edited January 1, 20233 yr by PhilBarker
January 1, 20233 yr Author 22 minutes ago, PhilBarker said: I'm chasing some similar issues Do you? Can't see your (better) result from a different Linux distribution?!
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.