mgutt Posted July 29 Share Posted July 29 A German Unraid user posted in a different German Community that he had the problem that his server booted with ASPM disabled as long he used Unraid, but it was enabled if he booted Ubuntu: Unraid: Ubuntu: He even tried to boot with the kernel option "pcie_aspm=force", but it didn't help. I tried to verify this, but my Unraid server boots with ASPM enabled?! lspci -vvvnnPPDq | grep -B 30 ':[[:space:]]ASPM' ... 0000:00:1c.5/03:00.0 Ethernet controller [0200]: Intel Corporation I210 Gigabit Network Connection [8086:1533] (rev 03) Subsystem: Gigabyte Technology Co., Ltd I210 Gigabit Network Connection [1458:e000] Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 17 IOMMU group: 11 Region 0: Memory at a1200000 (32-bit, non-prefetchable) [size=1M] Region 2: I/O ports at 3000 [size=32] Region 3: Memory at a1300000 (32-bit, non-prefetchable) [size=16K] Expansion ROM at a1100000 [disabled] [size=1M] Capabilities: [40] Power Management version 3 Flags: PMEClk- DSI+ D1- D2- AuxCurrent=0mA PME(D0+,D1-,D2-,D3hot+,D3cold+) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=1 PME- Capabilities: [50] MSI: Enable- Count=1/1 Maskable+ 64bit+ Address: 0000000000000000 Data: 0000 Masking: 00000000 Pending: 00000000 Capabilities: [70] MSI-X: Enable+ Count=5 Masked- Vector table: BAR=3 offset=00000000 PBA: BAR=3 offset=00002000 Capabilities: [a0] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <512ns, L1 <64us ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+ RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset- MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend- LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <2us, L1 <16us ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ What could be the reason for this different behavior between his and mine setup and why is it for him different with Ubuntu? Quote Link to comment
JorgeB Posted July 30 Share Posted July 30 Can't add much other than it's also enable for me in Unraid, as long as it's enable in the board BIOS. Quote Link to comment
mgutt Posted July 30 Author Share Posted July 30 Is there a command to check through unRAID if it's enabled by the BIOS? Quote Link to comment
CS01-HS Posted July 30 Share Posted July 30 On 7/29/2021 at 10:15 AM, mgutt said: I tried to verify this, but my Unraid server boots with ASPM enabled?! I believe you have (or had) a j5005? With my j5005 I have the same problem as the German user. ASPM is enabled in the BIOS and apparently supported: root@NAS:~# dmesg | grep -i aspm [ 0.454584] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3] But disabled on all devices: root@NAS:~# lspci -vv | grep 'ASPM.*abled' LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ pcilib: sysfs_read_vpd: read failed: Input/output error LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ 1 Quote Link to comment
mgutt Posted July 30 Author Share Posted July 30 1 hour ago, CS01-HS said: I believe you have (or had) a j5005? I don't, because I do not have the problem 😉 I have the C246N-WU2. But I have access to a J5005 server and you are right. It defaults to disabled. Did you try "pcie_aspm=force"? PS Ironically the C246N-WU2 returns no PCIe ASPM support, although it works flawlessly (without any changes made to the kernel boot options): dmesg | grep -i aspm [ 0.257536] ACPI FADT declares the system doesn't support PCIe ASPM, so disable it [ 0.536495] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3] [ 0.536680] acpi PNP0A08:00: FADT indicates ASPM is unsupported, using BIOS configuration Quote Link to comment
CS01-HS Posted July 30 Share Posted July 30 3 minutes ago, mgutt said: I have the C246N-WU2 Very nice. I keep my fingers crossed for a W580 m-itx. 2 minutes ago, mgutt said: Did you try "pcie_aspm=force" Yes. No difference as far as I can tell: root@NAS:~# dmesg | grep -i aspm [ 0.000000] Command line: BOOT_IMAGE=/bzimage initrd=/bzroot mitigations=off intel_iommu=on,igfx_off pcie_aspm=force [ 0.103794] Kernel command line: BOOT_IMAGE=/bzimage initrd=/bzroot mitigations=off intel_iommu=on,igfx_off pcie_aspm=force [ 0.103948] PCIe ASPM is forcibly enabled [ 0.454763] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3] root@NAS:~# lspci -vv | grep ASPM LnkCap: Port #3, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+ L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- LnkCap: Port #4, Speed 5GT/s, Width x1, ASPM not supported ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+ L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- LnkCap: Port #5, Speed 5GT/s, Width x1, ASPM not supported ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+ L1SubCtl1: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ LnkCap: Port #6, Speed 5GT/s, Width x1, ASPM not supported ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+ L1SubCtl1: PCI-PM_L1.2- PCI-PM_L1.1- ASPM_L1.2- ASPM_L1.1- pcilib: sysfs_read_vpd: read failed: Input/output error LnkCap: Port #0, Speed 5GT/s, Width x8, ASPM L0s, Exit Latency L0s <64ns ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp- LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ LnkCap: Port #0, Speed 5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s <4us, L1 unlimited ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp- LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ LnkCap: Port #0, Speed 2.5GT/s, Width x1, ASPM L0s L1, Exit Latency L0s unlimited, L1 <64us ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ L1SubCap: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ L1_PM_Substates+ L1SubCtl1: PCI-PM_L1.2+ PCI-PM_L1.1+ ASPM_L1.2+ ASPM_L1.1+ LnkCap: Port #1, Speed 5GT/s, Width x1, ASPM not supported ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp- LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk+ root@NAS:~# Quote Link to comment
mgutt Posted July 30 Author Share Posted July 30 3 minutes ago, CS01-HS said: Very nice. I keep my fingers crossed for a W580 m-itx. Yes, really sad that they did not release a W480N ITX and W580 is nothing I would buy for Unraid (no Legacy boot and no GVT-g support). 5 minutes ago, CS01-HS said: No difference as far as I can tell: 😢 Did you try Ubuntu? 1 Quote Link to comment
CS01-HS Posted July 30 Share Posted July 30 5 minutes ago, mgutt said: Did you try Ubuntu? I have not. Maybe when I get more time. Could it be a difference in kernels? Maybe 6.10 will fix it. Quote Link to comment
mgutt Posted July 30 Author Share Posted July 30 12 minutes ago, CS01-HS said: Could it be a difference in kernels? Maybe. I found out that the user already reported it as a bug. As he used Ubuntu 21.04 it should be Kernel 5.11. Unraid uses 5.10.28. @ich777 Do you have an Unraid beta build with Kernel 5.11 so I could test and compare it with Ubuntu? Maybe I find the time to test the J5005 with the older Unraid version and upgrade Ubuntu 20.04 to Kernel 5.10 etc as well. 1 Quote Link to comment
ich777 Posted July 30 Share Posted July 30 2 hours ago, mgutt said: @ich777 Do you have an Unraid beta build with Kernel 5.11 so I could test and compare it with Ubuntu? Currently on Kernel v5.13.6 and also disabled, attached the output: output.txt Quote Link to comment
mgutt Posted July 30 Author Share Posted July 30 26 minutes ago, ich777 said: and also disabled, You own the J5005 as well?! Quote Link to comment
ich777 Posted July 30 Share Posted July 30 13 minutes ago, mgutt said: You own the J5005 as well?! Sorry didn't go through the whole thread, I only own a J4205-ITX, the above output was from my i5-10600 on a ASUS STRIX Z490-E GAMING. Quote Link to comment
mgutt Posted July 30 Author Share Posted July 30 14 minutes ago, ich777 said: the above output was from my i5-10600 on a ASUS STRIX Z490-E GAMING. And did you enable ASPM (L1) in the BIOS? Quote Link to comment
ich777 Posted July 30 Share Posted July 30 5 minutes ago, mgutt said: And did you enable ASPM (L1) in the BIOS? Can't reboot the server currently, I think so. Have updated the BIOS two weeks ago and normaly I enabled it by default, but it could be also possible that I've missed it that time, will report back tomorrow. 1 Quote Link to comment
Falcosc Posted July 31 Share Posted July 31 (edited) The power of mgutt. Why does your topic attract so many people, while I had to wait days until anybody even downloaded one of my logfiles I bet the logfile downloads are only related to your magic thread Thank you for this attention. Anyway, If nobody looks into bug reports, then I can post updates here: Unraid's ASPM management is not totally broken on my platform. Devices connected to CPU via the x16 Slot do automatically get ASPM Enabled like expected, even without boot option. Only devices connected to my Chipset have this issue. 0000:00:1c.0 PCI bridge [0604]: Intel Corporation 100 Series/C230 Series Chipset Family PCI Express Root Port #7 [8086:a116] (rev f1) (prog-if 00 [Normal decode]) I don't know why Ubunutu doesn't have this issue. I tested the old ubuntu-20.04.2.0 with Kernel 5.8. I can do any test you like to have because I didn't clean up my test bench yet and still evaluating. New Unraid Builds, other Ubuntu Versions, Old Ubuntu with manual Kernel update. Just tell me what you need. @mgutt maybe link the bug report in your topic opening post because the test setup description is detailed enough to see that there is no setup issue and really a difference between Ubuntu and Unraid to hunt for. Edited July 31 by Falcosc 1 Quote Link to comment
mgutt Posted August 9 Author Share Posted August 9 On 7/31/2021 at 9:40 PM, Falcosc said: I don't know why Ubunutu doesn't have this issue. Maybe I found something interesting. Ubuntu changes some settings like the CPU governor after boot: https://askubuntu.com/questions/1021748/set-cpu-governor-to-performance-in-18-04/1085719#1085719 So I did a little bit more research and I found those configurations, too: CONFIG_PCIEASPM=y CONFIG_PCIEASPM_DEBUG=y CONFIG_PCIEASPM_DEFAULT=y Description for CONFIG_PCIEASPM: https://www.kernelconfig.io/config_pcieaspm This enables OS control over PCI Express ASPM (Active State Power Management) and Clock Power Management. ASPM supports state L0/L0s/L1. ... ASPM can be disabled or enabled at runtime via /sys/module/pcie_aspm/parameters/policy Are we able to pass kernel configurations through the syslinux.cfg or is this something which needs to be added through @limetech inside the unraid kernel? And I found something else. The J5005 server allows setting the ASPM policy which has ASPM disabled on all pci devices but it shows no impact: echo -n powersave > /sys/module/pcie_aspm/parameters/policy cat /sys/module/pcie_aspm/parameters/policy default performance [powersave] powersupersave lspci -vv | grep ASPM LnkCap: Port #3, Speed 5GT/s, Width x1, ASPM not supported ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+ LnkCtl: ASPM Disabled; RCB 64 bytes, Disabled- CommClk- And my C246 server disallows it which has ASPM enabled on all pci devices: echo -n powersave > /sys/module/pcie_aspm/parameters/policy bash: echo: write error: Operation not permitted lspci -vv | grep ASPM LnkCap: Port #17, Speed 8GT/s, Width x4, ASPM L0s L1, Exit Latency L0s <1us, L1 <4us ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+ LnkCtl: ASPM L0s L1 Enabled; RCB 64 bytes, Disabled- CommClk- Quote Link to comment
Falcosc Posted August 10 Share Posted August 10 I did already test all of these 2 weeks ago: Step 7 and Step 8 Both commands did execute without error message and value was set. But both did not affect ASPM in Unraid. @mgutt Which part of your findings are not tested in the bugreport? 9 hours ago, mgutt said: https://www.kernelconfig.io/config_pcieaspm This enables OS control over PCI Express ASPM (Active State Power Management) and Clock Power Management. ASPM supports state L0/L0s/L1. ... ASPM can be disabled or enabled at runtime via /sys/module/pcie_aspm/parameters/policy Are we able to pass kernel configurations through the syslinux.cfg or is this something which needs to be added through @limetech inside the unraid kernel? At which configuration are you interested in? I'm not sure about CONFIG_PCIEASPM because ASPM is working in Unraid which means this can't be false. Only devices connected to my chipset don't get ASPM in Unraid. Ubunuto however did enable ASPM for all devices. @limetech I can do any test you like to have because I didn't clean up my test bench yet and still evaluating. New Unraid Builds, other Ubuntu Versions, Old Ubuntu with manual Kernel update. Just tell me what you need. Quote Link to comment
mgutt Posted August 10 Author Share Posted August 10 49 minutes ago, Falcosc said: I'm not sure about CONFIG_PCIEASPM because ASPM is working in Unraid which means this can't be false Does ASPM need CONFIG_PCIEASPM to work? 50 minutes ago, Falcosc said: Only devices connected to my chipset don't get ASPM in Unraid The J5005 does not even have a chipset and has the same behavior as your system. Quote Link to comment
Ver7o Posted August 10 Share Posted August 10 I thought I'd toss this here, as it was driving me batty for a while. Going wild with ASPM was the reason my Unraid was crashing (webgui crash, ssh crash, samba crash, console crash...) without any syslog entries. Drove me crazy for a while, not knowing if I indeed have bad hardware. I'm on H410 chipset. Current BIOS settings: - PCI Express Native Control: Enabled - PCIE ASPM Support: Enabled - PCH PCIE ASPM: Enabled - DMI ASPM: Enabled - PCH DMI ASPM: Disabled PCH DMI ASPM set to disabled, as this looked to be the reason for Unraid silently crashing. root@Tower:~# dmesg | grep -i aspm [ 0.226470] acpi PNP0A08:00: _OSC: OS supports [ExtendedConfig ASPM ClockPM Segments MSI HPX-Type3] root@Tower:~# lspci -vv | grep 'ASPM.*abled' LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ root@Tower:~# lspci -vvvnnPPDq | grep -B 30 ':[[:space:]]ASPM' Address: fee00258 Data: 0000 Capabilities: [70] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot+,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [a8] SATA HBA v1.0 BAR4 Offset=00000004 Kernel driver in use: ahci Kernel modules: ahci 0000:00:1c.0 PCI bridge [0604]: Intel Corporation Device [8086:a394] (rev f0) (prog-if 00 [Normal decode]) Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 122 IOMMU group: 5 Bus: primary=00, secondary=01, subordinate=01, sec-latency=0 I/O behind bridge: 0000f000-00000fff [disabled] Memory behind bridge: b1000000-b10fffff [size=1M] Prefetchable memory behind bridge: 00000000fff00000-00000000000fffff [disabled] Secondary status: 66MHz- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort+ <SERR- <PERR- BridgeCtl: Parity- SERR+ NoISA- VGA- VGA16+ MAbort- >Reset- FastB2B- PriDiscTmr- SecDiscTmr- DiscTmrStat- DiscTmrSERREn- Capabilities: [40] Express (v2) Root Port (Slot+), MSI 00 DevCap: MaxPayload 256 bytes, PhantFunc 0 ExtTag- RBE+ DevCtl: CorrErr- NonFatalErr- FatalErr- UnsupReq- RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop- MaxPayload 256 bytes, MaxReadReq 128 bytes DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr+ TransPend- LnkCap: Port #5, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <16us ClockPM- Surprise- LLActRep+ BwNot+ ASPMOptComp+ LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ -- Kernel driver in use: e1000e Kernel modules: e1000e 0000:00:1c.0/01:00.0 Non-Volatile memory controller [0108]: Sandisk Corp WD Blue SN550 NVMe SSD [15b7:5009] (rev 01) (prog-if 02 [NVM Express]) Subsystem: Sandisk Corp WD Blue SN550 NVMe SSD [15b7:5009] Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx+ Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx- Latency: 0, Cache Line Size: 64 bytes Interrupt: pin A routed to IRQ 16 NUMA node: 0 IOMMU group: 5 Region 0: Memory at b1000000 (64-bit, non-prefetchable) [size=16K] Region 4: Memory at b1004000 (64-bit, non-prefetchable) [size=256] Capabilities: [80] Power Management version 3 Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0-,D1-,D2-,D3hot-,D3cold-) Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME- Capabilities: [90] MSI: Enable- Count=1/32 Maskable- 64bit+ Address: 0000000000000000 Data: 0000 Capabilities: [b0] MSI-X: Enable+ Count=17 Masked- Vector table: BAR=0 offset=00002000 PBA: BAR=4 offset=00000000 Capabilities: [c0] Express (v2) Endpoint, MSI 00 DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <1us, L1 unlimited ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset+ SlotPowerLimit 0.000W DevCtl: CorrErr+ NonFatalErr+ FatalErr+ UnsupReq+ RlxdOrd+ ExtTag- PhantFunc- AuxPwr- NoSnoop+ FLReset- MaxPayload 256 bytes, MaxReadReq 512 bytes DevSta: CorrErr- NonFatalErr- FatalErr- UnsupReq- AuxPwr- TransPend- LnkCap: Port #0, Speed 8GT/s, Width x4, ASPM L1, Exit Latency L1 <8us ClockPM+ Surprise- LLActRep- BwNot- ASPMOptComp+ LnkCtl: ASPM L1 Enabled; RCB 64 bytes, Disabled- CommClk+ root@Tower:~# Quote Link to comment
Falcosc Posted August 10 Share Posted August 10 It is ok to have reasons why ASPM should be disabled. But it is an issue that there is no way to enable ASPM on unraid. How can I help to identify which part of Unraid is preventing ASPM to be enabled? Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.