Why is ASPM disabled in Unraid, while it's enabled in Ubuntu?


Recommended Posts

I have two enemies:

 

The other 100% are no Problem?

 

Ist 8169 the wrong driver?

See https://www.linuxquestions.org/questions/linux-hardware-18/realtek-rtl8111-8168-8411-ethernet-controller-r8168-driver-install-r8169-driver-doesn't-work-4175641982/

 

I did use the ethernet for more than web interface, but I think there are no stability problems.

 

aspm_disabled.png

 

RTL8111E.png

Edited by TraumG
Link to comment
On 10/16/2021 at 9:48 AM, TraumG said:

HOw can I copy the hole output?

Are you using the web terminal? Select everything, right mouse click and copy. Or use this command to write the output to a file and download it from your USB flash drive:

lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM )' > /boot/logs/aspm.log

 

Link to comment

Looks like 8168

 

Spoiler

Tino, [20.10.21 00:46]
00:1d.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #12 (rev f0) (prog-if 00 [Normal decode])
  Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
  Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
  Latency: 0, Cache Line Size: 64 bytes
  Interrupt: pin D routed to IRQ 123
  Bus: primary=00, secondary=02, subordinate=02, sec-latency=0
  I/O behind bridge: 00003000-00003fff [size=4K]
  Memory behind bridge: 8f800000-8f8fffff [size=1M]
  Prefetchable memory behind bridge: 0000006000000000-00000060000fffff [size=1M]
  Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
  BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
    PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
  Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
    DevCap:  MaxPayload 256 bytes, PhantFunc 0
      ExtTag- RBE+
    DevCtl:  CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
      RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
      MaxPayload 128 bytes, MaxReadReq 128 bytes
    DevSta:  CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
    LnkCap:  Port #12, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <16us
      ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
    LnkCtl:  <font color="#CC0000"><b>ASPM L1 Enabled; RCB 64 bytes Disabled</b></font>- CommClk+
<font color="#06989A">--</font>
  Control: I/O- Mem+ BusMaster- SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
  Status: Cap- 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast &gt;TAbort- &lt;TAbort- &lt;MAbort- &gt;SERR- &lt;PERR- INTx-
  Region 0: Memory at fe010000 (32-bit, non-prefetchable) [size=4K]

01:00.0 Non-Volatile memory controller: Sandisk Corp Device 5009 (rev 01) (prog-if 02 [NVM Express])
  Subsystem: Sandisk Corp Device 5009
  Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
  Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast &gt;TAbort- &lt;TAbort- &lt;MAbort- &gt;SERR- &lt;PERR- INTx-
  Latency: 0, Cache Line Size: 64 bytes
  Interrupt: pin A routed to IRQ 16
  NUMA node: 0
  Region 0: Memory at 8f900000 (64-bit, non-prefetchable) [size=16K]
  Region 4: Memory at 8f904000 (64-bit, non-prefetchable) [size=256]
  Capabilities: [80] Power Management version 3
    Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
    Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
  Capabilities: [90] MSI: Enable- Count=1/32 Maskable- 64bit+
    Address: 0000000000000000  Data: 0000
  Capabilities: [b0] MSI-X: Enable+ Count=17 Masked-
    Vector table: BAR=0 offset=00002000
    PBA: BAR=4 offset=00000000
  Capabilities: [c0] Express (v2) Endpoint, MSI 00
    DevCap:  MaxPayload 512 bytes, PhantFunc 0, Latency L0s &lt;1us, L1 unlimited
      ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
    DevCtl:  CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
      RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
      MaxPayload 256 bytes, MaxReadReq 512 bytes
    DevSta:  CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
    LnkCap:  Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 &lt;8us
      ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
    LnkCtl:  <font color="#CC0000"><b>ASPM L1 Enabled; RCB 64 bytes Disabled</b></font>- CommClk+
pcilib: sysfs_read_vpd: read failed: Input/output error
<font color="#06989A">--</font>
        PortCommonModeRestoreTime=32us PortTPowerOnTime=10us
    L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
         T_CommonMode=0us LTR1.2_Threshold=81920ns
    L1SubCtl2: T_PwrOn=10us
  Kernel driver in use: nvme
  Kernel modules: nvme

Tino, [20.10.21 00:46]
02:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
  Subsystem: Fujitsu Technology Solutions RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
  Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
  Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast &gt;TAbort- &lt;TAbort- &lt;MAbort- &gt;SERR- &lt;PERR- INTx-
  Latency: 0, Cache Line Size: 64 bytes
  Interrupt: pin A routed to IRQ 19
  Region 0: I/O ports at 3000 [size=256]
  Region 2: Memory at 8f800000 (64-bit, non-prefetchable) [size=4K]
  Region 4: Memory at 6000000000 (64-bit, prefetchable) [size=16K]
  Capabilities: [40] Power Management version 3
    Flags: PMEClk- DSI- D1+ D2+ AuxCurrent=375mA PME(D0+,D1+,D2+,D3hot+,D3cold+)
    Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
  Capabilities: [50] MSI: Enable- Count=1/1 Maskable- 64bit+
    Address: 0000000000000000  Data: 0000
  Capabilities: [70] Express (v2) Endpoint, MSI 01
    DevCap:  MaxPayload 128 bytes, PhantFunc 0, Latency L0s &lt;512ns, L1 &lt;64us
      ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 10.000W
    DevCtl:  CorrErr- NonFatalErr- FatalErr- UnsupReq-
      RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop-
      MaxPayload 128 bytes, MaxReadReq 4096 bytes
    DevSta:  CorrErr+ NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
    LnkCap:  Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 &lt;64us
      ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
    LnkCtl:  <font color="#CC0000"><b>ASPM L1 Enabled; RCB 64 bytes Disabled</b></font>- CommClk+
<font color="#4E9A06"><b>t@pfeil</b></font>:<font color="#3465A4"><b>~</b></font>$

</pre>

 

Edited by TraumG
Link to comment
  • 4 weeks later...
  • 1 month later...
On 11/14/2021 at 6:38 PM, mgutt said:

Only as an information: I found a better command to get the ASPM status of supporting devices:

lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM )'

 

 

root@Unraid-1:~# lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM )' 00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 05) (prog-if 00 [Normal decode])
                LnkCap: Port #2, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <256ns, L1 <8us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
00:1b.0 PCI bridge: Intel Corporation Comet Lake PCI Express Root Port #21 (rev f0) (prog-if 00 [Normal decode])
                LnkCap: Port #21, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <1us, L1 <16us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+
00:1d.0 PCI bridge: Intel Corporation Comet Lake PCI Express Root Port #9 (rev f0) (prog-if 00 [Normal decode])
                LnkCap: Port #9, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s unlimited, L1 <16us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk-
pcilib: sysfs_read_vpd: read failed: No such device
01:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
                LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s <64ns
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
02:00.0 PCI bridge: ASMedia Technology Inc. ASM1184e 4-Port PCIe x1 Gen2 Packet Switch (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 unlimited
                LnkCtl: ASPM L0s L1 Enabled; Disabled- CommClk+
03:01.0 PCI bridge: ASMedia Technology Inc. ASM1184e 4-Port PCIe x1 Gen2 Packet Switch (prog-if 00 [Normal decode])
                LnkCap: Port #1, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 unlimited
                LnkCtl: ASPM L1 Enabled; Disabled- CommClk+
03:03.0 PCI bridge: ASMedia Technology Inc. ASM1184e 4-Port PCIe x1 Gen2 Packet Switch (prog-if 00 [Normal decode])
                LnkCap: Port #3, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 unlimited
                LnkCtl: ASPM L0s L1 Enabled; Disabled- CommClk-
03:05.0 PCI bridge: ASMedia Technology Inc. ASM1184e 4-Port PCIe x1 Gen2 Packet Switch (prog-if 00 [Normal decode])
                LnkCap: Port #5, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 unlimited
                LnkCtl: ASPM L1 Enabled; Disabled- CommClk+
03:07.0 PCI bridge: ASMedia Technology Inc. ASM1184e 4-Port PCIe x1 Gen2 Packet Switch (prog-if 00 [Normal decode])
                LnkCap: Port #7, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 unlimited
                LnkCtl: ASPM L1 Enabled; Disabled- CommClk+
04:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch (rev c1) (prog-if 00 [Normal decode])
                LnkCap: Port #1, Speed 16GT/s, Width x8, ASPM L1, Exit Latency L1 <64us
                LnkCtl: ASPM L1 Enabled; Disabled- CommClk+
05:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM L0s L1 Enabled; Disabled- CommClk+
06:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 23 [Radeon RX 6600/6600 XT/6600M] (rev c1) (prog-if 00 [VGA controller])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+
06:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT]
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+
08:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch (rev c1) (prog-if 00 [Normal decode])
                LnkCap: Port #5, Speed 16GT/s, Width x8, ASPM L1, Exit Latency L1 <64us
                LnkCtl: ASPM L1 Enabled; Disabled- CommClk+
09:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM L0s L1 Enabled; Disabled- CommClk+
0a:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 23 [Radeon RX 6600/6600 XT/6600M] (rev c1) (prog-if 00 [VGA controller])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+
0a:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT]
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+
0b:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Upstream Port of PCI Express Switch (rev c1) (prog-if 00 [Normal decode])
                LnkCap: Port #7, Speed 16GT/s, Width x8, ASPM L1, Exit Latency L1 <64us
                LnkCtl: ASPM L1 Enabled; Disabled- CommClk+
0c:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] Navi 10 XL Downstream Port of PCI Express Switch (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM L0s L1 Enabled; Disabled- CommClk+
0d:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Navi 23 [Radeon RX 6600/6600 XT/6600M] (rev c1) (prog-if 00 [VGA controller])
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+
0d:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Navi 21 HDMI Audio [Radeon RX 6800/6800 XT / 6900 XT]
                LnkCap: Port #0, Speed 16GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+

 

Link to comment
  • 6 months later...

Sorry to bring a thread back from the dead; I decided that I wanted to spend time trying to minimize idle power-usage on my Supermicro Server (X9DRi-LN4F+ motherboard).

 

I followed most of the advice here; have a handful of devices that ASPM is still disabled for here; hoping I could get some suggestions on how to address them:

 

root@Tower:~# lspci -vv | awk '/ASPM/{print $0}' RS= | grep --color -P '(^[a-z0-9:.]+|ASPM )'
00:00.0 Host bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 DMI2 (rev 04)
                LnkCap: Port #0, Speed 2.5GT/s, Width x4, ASPM L1, Exit Latency L1 <16us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
00:01.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1a (rev 04) (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 2.5GT/s, Width x4, ASPM L1, Exit Latency L1 <16us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
00:01.1 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1b (rev 04) (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <16us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
00:02.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2a (rev 04) (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L1, Exit Latency L1 <16us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
00:03.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3a (rev 04) (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L1, Exit Latency L1 <16us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
00:11.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Virtual Root Port (rev 06) (prog-if 00 [Normal decode])
                LnkCap: Port #17, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+
00:1c.0 PCI bridge: Intel Corporation C600/X79 series chipset PCI Express Root Port 1 (rev b6) (prog-if 00 [Normal decode])
                LnkCap: Port #1, Speed 5GT/s, Width x4, ASPM L1, Exit Latency L1 <4us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
pcilib: sysfs_read_vpd: read failed: No such device
02:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)
                LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L0s, Exit Latency L0s <64ns
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
03:00.0 VGA compatible controller: NVIDIA Corporation TU104GL [Quadro RTX 4000] (rev a1) (prog-if 00 [VGA controller])
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <16us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
03:00.1 Audio device: NVIDIA Corporation TU104 HD Audio Controller (rev a1)
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+
03:00.2 USB controller: NVIDIA Corporation TU104 USB 3.1 Host Controller (rev a1) (prog-if 30 [XHCI])
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <16us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
03:00.3 Serial bus controller: NVIDIA Corporation TU104 USB Type-C UCSI Controller (rev a1)
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <16us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
04:00.0 VGA compatible controller: NVIDIA Corporation GP106GL [Quadro P2000] (rev a1) (prog-if 00 [VGA controller])
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
04:00.1 Audio device: NVIDIA Corporation GP106 High Definition Audio Controller (rev a1)
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
05:00.0 Serial Attached SCSI controller: Intel Corporation C602 chipset 4-Port SATA Storage Control Unit (rev 06)
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+
06:00.0 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
                LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <32us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
06:00.1 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
                LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <32us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
06:00.2 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
                LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <32us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
06:00.3 Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
                LnkCap: Port #0, Speed 5GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <4us, L1 <32us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
80:00.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port in DMI2 Mode (rev 04) (prog-if 00 [Normal decode])
                LnkCap: Port #8, Speed 2.5GT/s, Width x48, ASPM L1, Exit Latency L1 <16us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
80:01.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 1a (rev 04) (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 8GT/s, Width x8, ASPM L1, Exit Latency L1 <16us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
80:02.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 2a (rev 04) (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L1, Exit Latency L1 <16us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-
80:03.0 PCI bridge: Intel Corporation Xeon E7 v2/Xeon E5 v2/Core i7 PCI Express Root Port 3a (rev 04) (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L1, Exit Latency L1 <16us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
82:00.0 Non-Volatile memory controller: Intel Corporation SSD 660P Series (rev 03) (prog-if 02 [NVM Express])
                LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <8us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
84:00.0 VGA compatible controller: NVIDIA Corporation TU104GL [Quadro RTX 4000] (rev a1) (prog-if 00 [VGA controller])
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
84:00.1 Audio device: NVIDIA Corporation TU104 HD Audio Controller (rev a1)
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
84:00.2 USB controller: NVIDIA Corporation TU104 USB 3.1 Host Controller (rev a1) (prog-if 30 [XHCI])
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
84:00.3 Serial bus controller: NVIDIA Corporation TU104 USB Type-C UCSI Controller (rev a1)
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+

 

Is this realistically the best that can be done with respect to ASPM enabling or is there further room for improvement?

Link to comment

Have you already done the usual solutions like bios setup and kernel boot options? If your system is setup incorrectly, it will not work.

 

If you have everything, then you should check Ubuntu first, because Ubuntu does not have the unknown ASPM bug which Unraid has for my hardware.

If you can get ASPM enabled on Ubuntu without hacks by using only bios and kernel boot options, then we will use your result to push forward. It will provide more evidence to help Unraid team to identify the cause of the issue.

And with the Ubuntu test you could see how much power saving is actually possible. If there isn't any, I wouldn't go forward.

 

And I would only recommend the ASPM hack (forced activation, which does bypass any compatibility checks) if you did confirm that other operating systems can enable it, then bypassing logic is a bit less dangerous.

Edited by Falcosc
Link to comment
2 minutes ago, Falcosc said:

Have you already done the usual solutions like bios setup and kernel boot options? If your system is setup incorrectly, it will not work.

 

If you have everything, then you should check Ubuntu first, because Ubuntu does not have the unknown ASPM bug which Unraid has for my hardware.

If you can get ASPM enabled on Ubuntu without hacks, then we will use your result to push forward. It will provide more evidence to help Unraid team to identify the cause of the issue.

 

And I would only recommend the ASPM hack (forced activation, which does bypass any compatibility checks) if you did confirm that other operating systems can enable it, then bypassing logic is a bit less dangerous.

 

I set my BIOS ASPM option to "auto", here is the blurb in the manual about the options.  I didn't actually see the Force L0 option.

 

image.thumb.png.41073835155644aa44b331f238f5f6a3.png

 

I also added pcie_aspm=force to the list of options in my /boot/syslinux/syslinux.cfg file

 

root@Tower:~# cat /boot/syslinux/syslinux.cfg
default menu.c32
menu title Lime Technology, Inc.
prompt 0
timeout 50
label Unraid OS
  menu default
  kernel /bzimage
  append intel_iommu=on rd.driver.pre=vfio-pci video=vesafb:off,efifb:off isolcpus=4-9,24-29 vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot pcie_aspm=force
label Unraid OS GUI Mode
  kernel /bzimage
  append isolcpus=4-9,24-29 vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot,/bzroot-gui
label Unraid OS Safe Mode (no plugins, no GUI)
  kernel /bzimage
  append initrd=/bzroot unraidsafemode
label Unraid OS GUI Safe Mode (no plugins)
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui unraidsafemode
label Memtest86+
  kernel /memtest

 

 

Before the above two options, all entries when I did lspci showed ASPM disabled, with the above changes, only a handful of entries (primarily PCI bridges and my HBA) show ASPM as being disabled.

 

I have no problem trying ubuntu; server is a bit tough to access but I'll make a go of it.  Would the idea be to boot off of a liveUSB and then check ASPM status?

Link to comment

Yes, liveUSB does work fine. On my hardware, it did immediately work in ubuntu without having any boot options.

After figuring this out, I did manually set the ASPM option in the registers via a startup script which is executed after unraid config did decide that it is not save/not supported to enable it.

Here is the documentation which I did use to set the flags, it isn't the reference to the real specification, more a how-to https://wireless.wiki.kernel.org/en/users/documentation/aspm#enabling_aspm_with_setpci

Link to comment

@ogiif Ubuntu works, I encourage you to create a new bug for this issue. You can have a look at

to see which kind of information are needed for evidence.

 

But if Ubuntu doesn't work for your devices, then you have a real setup issue. There could a specific hardware combination which does prevent it or a configuration issue.

For example, your LSI SAS2308 PCI-Express Fusion-MPT SAS-2 doesn't have L1 support at all (LnkCap). The 4 Port SATA on the other hand have it. Have an eye on that, this one device could prevent your CPU from sleep. And even if 9 of 10 links are sleeping, you won't see much power savings as long as a single device keeps your CPU busy on the bus.

 

For that reason, I recommend checking power consumption on Ubuntu. You don't need to spend time on ASPM for the supported devices, as long as you have at least one thing which will not support it. I don't know if L0s is enough to see significant changes in the power consumption, I only know about the huge effect of a proper sleeping CPU after having everything in the deeper L1 sleep.

Link to comment
12 minutes ago, Falcosc said:

@ogiif Ubuntu works, I encourage you to create a new bug for this issue. You can have a look at

to see which kind of information are needed for evidence.

 

But if Ubuntu doesn't work for your devices, then you have a real setup issue. There could a specific hardware combination which does prevent it or a configuration issue.

For example, your LSI SAS2308 PCI-Express Fusion-MPT SAS-2 doesn't have L1 support at all (LnkCap). The 4 Port SATA on the other hand have it. Have an eye on that, this one device could prevent your CPU from sleep. And even if 9 of 10 links are sleeping, you won't see much power savings as long as a single device keeps your CPU busy on the bus.

 

For that reason, I recommend checking power consumption on Ubuntu. You don't need to spend time on ASPM for the supported devices, as long as you have at least one thing which will not support it. I don't know if L0s is enough to see significant changes in the power consumption, I only know about the huge effect of a proper sleeping CPU after having everything in the deeper L1 sleep.

 

I'll give the LiveUSB a try in a bit and report back (might be a day or so, the garage where my server resides is absolutely cooking right now, and I don't really want to spend much time there).

 

Regarding the HBA, is there a known HBA that can be flashed to IT mode that supports L1 support?

Link to comment

Check first what happens with the CPU state after enabling L0s before investigating this. And then compare the power draw without this card to see if it is worth to hunt for.

 

With Ubunutu I was able to reach C8 state in powertop which resulted in a major power draw change of 4W.

 

And don't forget to execute autotune with powertop

Link to comment
5 hours ago, Falcosc said:

Check first what happens with the CPU state after enabling L0s before investigating this. And then compare the power draw without this card to see if it is worth to hunt for.

 

With Ubunutu I was able to reach C8 state in powertop which resulted in a major power draw change of 4W.

 

And don't forget to execute autotune with powertop

 

I don't think my E5-2680v2 processors can reach C8 state, would love to go into C7 tho!

 

image.thumb.png.59cc985d6dc021408a0f4940603adff1.png

Link to comment
  • 5 months later...

I'm chasing some similar issues to this on my unraid setup

 

00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir PCIe GPP Bridge (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L1, Exit Latency L1 <64us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
00:02.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne PCIe GPP Bridge (prog-if 00 [Normal decode])
                LnkCap: Port #2, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <64us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
00:08.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Renoir Internal PCIe GPP Bridge to Bus (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
01:00.0 VGA compatible controller: NVIDIA Corporation GP107GL [Quadro P620] (rev a1) (prog-if 00 [VGA controller])
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <16us
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
01:00.1 Audio device: NVIDIA Corporation GP107GL High Definition Audio Controller (rev a1)
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <512ns, L1 <4us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+
02:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] Device 43bc (rev 02) (prog-if 30 [XHCI])
                LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 unlimited
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
02:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] Device 43b8 (rev 02) (prog-if 01 [AHCI 1.0])
                LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 unlimited
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
02:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Device 43b3 (rev 02) (prog-if 00 [Normal decode])
                LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <2us, L1 unlimited
                LnkCtl: ASPM Disabled; Disabled- CommClk+
03:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02) (prog-if 00 [Normal decode])
                LnkCap: Port #4, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 unlimited
                LnkCtl: ASPM Disabled; Disabled- CommClk-
03:05.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02) (prog-if 00 [Normal decode])
                LnkCap: Port #5, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 unlimited
                LnkCtl: ASPM L0s Enabled; Disabled- CommClk+
03:06.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02) (prog-if 00 [Normal decode])
                LnkCap: Port #6, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 unlimited
                LnkCtl: ASPM L0s L1 Enabled; Disabled- CommClk+
03:07.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 300 Series Chipset PCIe Port (rev 02) (prog-if 00 [Normal decode])
                LnkCap: Port #7, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 unlimited
                LnkCtl: ASPM Disabled; Disabled- CommClk+
05:00.0 Network controller: Realtek Semiconductor Co., Ltd. RTL8192EE PCIe Wireless Network Adapter
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <512ns, L1 <64us
                LnkCtl: ASPM L0s Enabled; RCB 64 bytes, Disabled- CommClk+
pcilib: sysfs_read_vpd: read failed: No such device
06:00.0 SATA controller: ASMedia Technology Inc. ASM1166 Serial ATA Controller (rev 02) (prog-if 01 [AHCI 1.0])
                LnkCap: Port #0, Speed 8GT/s, Width x2, ASPM L0s L1, Exit Latency L0s <4us, L1 <64us
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk+
07:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
08:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Cezanne [Radeon Vega Series / Radeon Vega Mobile Series] (rev c9) (prog-if 00 [VGA controller])
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
08:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Renoir Radeon High Definition Audio Controller
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
08:00.2 Encryption controller: Advanced Micro Devices, Inc. [AMD] Family 17h (Models 10h-1fh) Platform Security Processor
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
08:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne USB 3.1 (prog-if 30 [XHCI])
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
08:00.4 USB controller: Advanced Micro Devices, Inc. [AMD] Renoir/Cezanne USB 3.1 (prog-if 30 [XHCI])
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
09:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 81) (prog-if 01 [AHCI 1.0])
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
09:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 81) (prog-if 01 [AHCI 1.0])
                LnkCap: Port #0, Speed 8GT/s, Width x16, ASPM L0s L1, Exit Latency L0s <64ns, L1 <1us
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+

 

every device has L0s and L1 available but nearly all of the AMD chipset ones show as disabled

Powertop only shows C1,C2,C3 - nothing else

 

I've tried everything I could think of - forced ASPM on, set all the energy saving settings in bios
My idling power usage of the server is around 90w 😢

 

System is a Gigabyte A320 with a Ryzen 5600G

Edited by PhilBarker
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.