Why is ASPM disabled in Unraid, while it's enabled in Ubuntu?


Recommended Posts

A German Unraid user posted in a different German Community that he had the problem that his server booted with ASPM disabled as long he used Unraid, but it was enabled if he booted Ubuntu:

 

Unraid:

image.png.625d49809a36bb7eec4de8ae97e776a0.png

 

Ubuntu:

image.png.c824f78d0a0f056ccf2070173a5c5228.png

 

He even tried to boot with the kernel option "pcie_aspm=force", but it didn't help.

 

I tried to verify this, but my Unraid server boots with ASPM enabled?!

lspci -vvvnnPPDq | grep -B 30 ':[[:space:]]ASPM'
...
0000:00:1c.5/03:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03)
        Subsystem: Gigabyte Technology Co., Ltd I210 Gigabit Network Connection [1458:e000]
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 17
        IOMMU group: 11
        Region 0: Memory at a1200000 (32-bit, non-prefetchable) [size=1M]
        Region 2: I/O ports at 3000 [size=32]
        Region 3: Memory at a1300000 (32-bit, non-prefetchable) [size=16K]
        Expansion ROM at a1100000 [disabled] [size=1M]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME-
        Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+
                Address: 0000000000000000  Data: 0000
                Masking: 00000000  Pending: 00000000
        Capabilities: [70] MSI-X: Enable+ Count=5 Masked-
                Vector table: BAR=3 offset=00000000
                PBA: BAR=3 offset=00002000
        Capabilities: [a0] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
                LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <16us
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+

 

What could be the reason for this different behavior between his and mine setup and why is it for him different with Ubuntu?

Link to comment
On 7/29/2021 at 10:15 AM, mgutt said:

I tried to verify this, but my Unraid server boots with ASPM enabled?!

 

I believe you have (or had) a j5005?

With my j5005 I have the same problem as the German user.

 

ASPM is enabled in the BIOS and apparently supported:

root@NAS:~# dmesg | grep -i aspm
[    0.454584] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]

 

But disabled on all devices:

root@NAS:~# lspci -vv | grep 'ASPM.*abled'
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
pcilib: sysfs_read_vpd: read failed: Input/output error
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+

 

  • Thanks 1
Link to comment
1 hour ago, CS01-HS said:

I believe you have (or had) a j5005?

I don't, because I do not have the problem 😉 I have the C246N-WU2.

 

But I have access to a J5005 server and you are right. It defaults to disabled. Did you try "pcie_aspm=force"?

 

PS Ironically the C246N-WU2 returns no PCIe ASPM support, although it works flawlessly (without any changes made to the kernel boot options):

dmesg | grep -i aspm
[    0.257536] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it
[    0.536495] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
[    0.536680] acpi PNP0A08:00: FADT indicates ASPM is unsupported, using BIOS configuration

 

 

Link to comment
3 minutes ago, mgutt said:

I have the C246N-WU2

 

Very nice. I keep my fingers crossed for a W580 m-itx. 

 

2 minutes ago, mgutt said:

Did you try "pcie_aspm=force"

 

Yes. No difference as far as I can tell:

 

root@NAS:~# dmesg | grep -i aspm
[    0.000000] Command line: BOOT_IMAGE=/bzimage initrd=/bzroot mitigations=off intel_iommu=on,igfx_off pcie_aspm=force
[    0.103794] Kernel command line: BOOT_IMAGE=/bzimage initrd=/bzroot mitigations=off intel_iommu=on,igfx_off pcie_aspm=force
[    0.103948] PCIe ASPM is forcibly enabled
[    0.454763] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
root@NAS:~# lspci -vv | grep ASPM
LnkCap: Port #3, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us
ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
LnkCap: Port #4, Speed 5GT/s, Width x1, ASPM not supported
ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
LnkCap: Port #5, Speed 5GT/s, Width x1, ASPM not supported
ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
L1SubCtl1: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+
LnkCap: Port #6, Speed 5GT/s, Width x1, ASPM not supported
ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1-
pcilib: sysfs_read_vpd: read failed: Input/output error
LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s <64ns
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <4us, L1 unlimited
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us
ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+
L1SubCtl1: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+
LnkCap: Port #1, Speed 5GT/s, Width x1, ASPM not supported
ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+
root@NAS:~#

 

Link to comment
3 minutes ago, CS01-HS said:

Very nice. I keep my fingers crossed for a W580 m-itx. 

 

Yes, really sad that they did not release a W480N ITX and W580 is nothing I would buy for Unraid (no Legacy boot and no GVT-g support).

 

5 minutes ago, CS01-HS said:

No difference as far as I can tell:

😢 

 

Did you try Ubuntu?

  • Like 1
Link to comment
12 minutes ago, CS01-HS said:

Could it be a difference in kernels?

Maybe. I found out that the user already reported it as a bug.

 

As he used Ubuntu 21.04 it should be Kernel 5.11. Unraid uses 5.10.28. @ich777 Do you have an Unraid beta build with Kernel 5.11 so I could test and compare it with Ubuntu?

 

Maybe I find the time to test the J5005 with the older Unraid version and upgrade Ubuntu 20.04 to Kernel 5.10 etc as well.

  • Thanks 1
Link to comment
5 minutes ago, mgutt said:

And did you enable ASPM (L1) in the BIOS?

Can't reboot the server currently, I think so.

Have updated the BIOS two weeks ago and normaly I enabled it by default, but it could be also possible that I've missed it that time, will report back tomorrow.

  • Thanks 1
Link to comment

The power of mgutt. Why does your topic attract so many people, while I had to wait days until anybody even downloaded one of my logfiles :D

 

I bet the logfile downloads are only related to your magic thread ;)

Thank you for this attention.

Anyway, If nobody looks into bug reports, then I can post updates here:

 

Unraid's ASPM management is not totally broken on my platform. Devices connected to CPU via the x16 Slot do automatically get ASPM Enabled like expected, even without boot option. Only devices connected to my Chipset have this issue.

0000:00:1c.0 PCI bridge [0604]: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #7 [8086:a116] (rev f1) (prog-if 00 [Normal decode])

 

I don't know why Ubunutu doesn't have this issue. I tested the old ubuntu-20.04.2.0 with Kernel 5.8.

 

I can do any test you like to have because I didn't clean up my test bench yet and still evaluating. New Unraid Builds, other Ubuntu Versions, Old Ubuntu with manual Kernel update. Just tell me what you need.

 

@mgutt maybe link the bug report in your topic opening post because the test setup description is detailed enough to see that there is no setup issue and really a difference between Ubuntu and Unraid to hunt for.

Edited by Falcosc
  • Like 1
Link to comment
  • 2 weeks later...
On 7/31/2021 at 9:40 PM, Falcosc said:

I don't know why Ubunutu doesn't have this issue.

Maybe I found something interesting. Ubuntu changes some settings like the CPU governor after boot:

https://askubuntu.com/questions/1021748/set-cpu-governor-to-performance-in-18-04/1085719#1085719

 

So I did a little bit more research and I found those configurations, too:

CONFIG_PCIEASPM=y
CONFIG_PCIEASPM_DEBUG=y
CONFIG_PCIEASPM_DEFAULT=y

 

Description for CONFIG_PCIEASPM:

https://www.kernelconfig.io/config_pcieaspm

This enables OS control over PCI Express ASPM (Active State
Power Management) and Clock Power Management. ASPM supports
state L0/L0s/L1.

...

ASPM can be disabled or enabled at runtime via
/sys/module/pcie_aspm/parameters/policy

 

Are we able to pass kernel configurations through the syslinux.cfg or is this something which needs to be added through @limetech  inside the unraid kernel?

 

And I found something else. The J5005 server allows setting the ASPM policy which has ASPM disabled on all pci devices but it shows no impact:

echo -n powersave > /sys/module/pcie_aspm/parameters/policy
cat /sys/module/pcie_aspm/parameters/policy
default performance [powersave] powersupersave
lspci -vv | grep ASPM
                LnkCap: Port #3, Speed 5GT/s, Width x1, ASPM not supported
                        ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
                LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk-

 

And my C246 server disallows it which has ASPM enabled on all pci devices:

echo -n powersave > /sys/module/pcie_aspm/parameters/policy
bash: echo: write error: Operation not permitted
lspci -vv | grep ASPM
                LnkCap: Port #17, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us
                        ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
                LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk-

 

 

 

Link to comment

I did already test all of these 2 weeks ago:

 

 

Step 7 and Step 8

Both commands did execute without error message and value was set. But both did not affect ASPM in Unraid.

 

@mgutt Which part of your findings are not tested in the bugreport?

9 hours ago, mgutt said:

https://www.kernelconfig.io/config_pcieaspm


This enables OS control over PCI Express ASPM (Active State
Power Management) and Clock Power Management. ASPM supports
state L0/L0s/L1.

...

ASPM can be disabled or enabled at runtime via
/sys/module/pcie_aspm/parameters/policy

 

Are we able to pass kernel configurations through the syslinux.cfg or is this something which needs to be added through @limetech  inside the unraid kernel?

 

 

At which configuration are you interested in?

I'm not sure about CONFIG_PCIEASPM because ASPM is working in Unraid which means this can't be false. Only devices connected to my chipset don't get ASPM in Unraid. Ubunuto however did enable ASPM for all devices.

 

@limetech

I can do any test you like to have because I didn't clean up my test bench yet and still evaluating. New Unraid Builds, other Ubuntu Versions, Old Ubuntu with manual Kernel update. Just tell me what you need.

Link to comment
49 minutes ago, Falcosc said:

I'm not sure about CONFIG_PCIEASPM because ASPM is working in Unraid which means this can't be false

Does ASPM need CONFIG_PCIEASPM to work?

 

50 minutes ago, Falcosc said:

Only devices connected to my chipset don't get ASPM in Unraid

The J5005 does not even have a chipset and has the same behavior as your system.

Link to comment

I thought I'd toss this here, as it was driving me batty for a while. Going wild with ASPM was the reason my Unraid was crashing (webgui crash, ssh crash, samba crash, console crash...) without any syslog entries. Drove me crazy for a while, not knowing if I indeed have bad hardware.

 

I'm on H410 chipset. Current BIOS settings:

- PCI Express Native Control: Enabled
- PCIE ASPM Support: Enabled
- PCH PCIE ASPM: Enabled
- DMI ASPM: Enabled
- PCH DMI ASPM: Disabled


PCH DMI ASPM set to disabled, as this looked to be the reason for Unraid silently crashing.

 

root@Tower:~# dmesg | grep -i aspm
[    0.226470] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3]
root@Tower:~# lspci -vv | grep 'ASPM.*abled'
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
root@Tower:~# lspci -vvvnnPPDq | grep -B 30 ':[[:space:]]ASPM'
                Address: fee00258  Data: 0000
        Capabilities: [70] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [a8] SATA HBA v1.0 BAR4 Offset=00000004
        Kernel driver in use: ahci
        Kernel modules: ahci

0000:00:1c.0 PCI bridge [0604]: Intel Corporation Device [8086:a394] (rev f0) (prog-if 00 [Normal decode])
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 122
        IOMMU group: 5
        Bus: primary=00, secondary=01, subordinate=01, sec-latency=0
        I/O behind bridge: 0000f000-00000fff [disabled]
        Memory behind bridge: b1000000-b10fffff [size=1M]
        Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [disabled]
        Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR-
        BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B-
                PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn-
        Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00
                DevCap: MaxPayload 256 bytes, PhantFunc 0
                        ExtTag- RBE+
                DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop-
                        MaxPayload 256 bytes, MaxReadReq 128 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend-
                LnkCap: Port #5, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <16us
                        ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
--
        Kernel driver in use: e1000e
        Kernel modules: e1000e

0000:00:1c.0/01:00.0 Non-Volatile memory controller [0108]: Sandisk Corp WD Blue SN550 NVMe SSD [15b7:5009] (rev 01) (prog-if 02 [NVM Express])
        Subsystem: Sandisk Corp WD Blue SN550 NVMe SSD [15b7:5009]
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Interrupt: pin A routed to IRQ 16
        NUMA node: 0
        IOMMU group: 5
        Region 0: Memory at b1000000 (64-bit, non-prefetchable) [size=16K]
        Region 4: Memory at b1004000 (64-bit, non-prefetchable) [size=256]
        Capabilities: [80] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [90] MSI: Enable- Count=1/32 Maskable- 64bit+
                Address: 0000000000000000  Data: 0000
        Capabilities: [b0] MSI-X: Enable+ Count=17 Masked-
                Vector table: BAR=0 offset=00002000
                PBA: BAR=4 offset=00000000
        Capabilities: [c0] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <1us, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W
                DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+
                        RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset-
                        MaxPayload 256 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <8us
                        ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+
                LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+
root@Tower:~# 

 

Link to comment
  • 2 months later...

I have a similar Problem. Power top goes only to packgae C2 withe autotune. That means 8-9W idle Fujitsu 3674

 If I use Ubuntu Stick it goes to C7 package and  I have only 3,9-4,3W @idle. Half Idle Power consumtion is a very big difference.

So what can I do to reach c7 also in unraid?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.