NVME drives throwing errors, filling logs instantly. How to resolve?

[JorgeB]

Oh, sorry, I see I messed up the link, thanks.

[Civic1201]

Any ideas how this fix will work with S3 sleep mode? Will it work / might give some errors?

[JorgeB]

No sure, never used it, but it should work.

[SteSto86]

Thanks all for the help, adding the pcie_aspm=off did it for me. Now the logs are useful again, because I got like 100 lines per second on pcie errors.

[CAMBO]

On 2/6/2023 at 9:51 AM, David Bott said:

Hi... I added...

 

append initrd=/bzroot pcie_aspm=off

 

...to the UnRAID OS config area.  (Click on Flash Drive to get to the right area.)  Add it in as a new line, save and reboot.

 

I am having the same issue. How do you add this to the config area, and where is it? Thanks

[trurl]

1 hour ago, CAMBO said:

where is it?

Main - Boot Device - Flash - Syslinux Configuration

[CAMBO]

6 hours ago, trurl said:

Main - Boot Device - Flash - Syslinux Configuration

Where exactly do I input “"pcie_aspm=off "? Please see screenshot. Thanks!

[trurl]

3 hours ago, CAMBO said:

Please see screenshot.

Please see screenshot in the post you quoted above

[CAMBO]

5 hours ago, trurl said:

Please see screenshot in the post you quoted above

I added in the Unraid OS, but still having the errors, I rebooted severals times as well. I'll post the screenshots. Let me know if I did something wrong. Thanks! 

 

I am running on Version: 6.12.0-rc5

 

 

[mung0]

I had the same issue and modified the config accordingly:
append pcie_aspm=off, pcie_acs_override=downstream, multifunction initrd=/bzroot

 

Seems to work perfectly - many thanks !

 

[bombz]

On 12/17/2022 at 9:59 AM, Nuke said:

Am i change things correct?

Hello, 

 

Appreciate all the posts from everyone here, based on the errors attached:

 

Oct 21 03:08:01 UnRAID emhttpd: read SMART /dev/sdf
Oct 21 03:09:47 UnRAID kernel: pcieport 0000:00:01.0: AER: Multiple Corrected error received: 0000:00:01.0
Oct 21 03:09:47 UnRAID kernel: pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Receiver ID)
Oct 21 03:09:47 UnRAID kernel: pcieport 0000:00:01.0:   device [8086:6f02] error status/mask=00000040/00002000
Oct 21 03:09:47 UnRAID kernel: pcieport 0000:00:01.0:    [ 6] BadTLP                
Oct 21 03:16:14 UnRAID kernel: pcieport 0000:00:01.0: AER: Multiple Corrected error received: 0000:00:01.0
Oct 21 03:16:14 UnRAID kernel: pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Receiver ID)
Oct 21 03:16:14 UnRAID kernel: pcieport 0000:00:01.0:   device [8086:6f02] error status/mask=00000040/00002000
Oct 21 03:16:14 UnRAID kernel: pcieport 0000:00:01.0:    [ 6] BadTLP                
Oct 21 03:19:37 UnRAID kernel: pcieport 0000:00:01.0: AER: Multiple Corrected error received: 0000:00:01.0
Oct 21 03:19:37 UnRAID kernel: pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Receiver ID)
Oct 21 03:19:37 UnRAID kernel: pcieport 0000:00:01.0:   device [8086:6f02] error status/mask=00000040/00002000
Oct 21 03:19:37 UnRAID kernel: pcieport 0000:00:01.0:    [ 6] BadTLP                
Oct 21 03:49:51 UnRAID kernel: pcieport 0000:00:01.0: AER: Multiple Corrected error received: 0000:00:01.0
Oct 21 03:49:51 UnRAID kernel: pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Receiver ID)
Oct 21 03:49:51 UnRAID kernel: pcieport 0000:00:01.0:   device [8086:6f02] error status/mask=00000040/00002000
Oct 21 03:49:51 UnRAID kernel: pcieport 0000:00:01.0:    [ 6] BadTLP                
Oct 21 04:03:13 UnRAID kernel: pcieport 0000:00:01.0: AER: Multiple Corrected error received: 0000:00:01.0
Oct 21 04:03:13 UnRAID kernel: pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Receiver ID)
Oct 21 04:03:13 UnRAID kernel: pcieport 0000:00:01.0:   device [8086:6f02] error status/mask=00000040/00002000
Oct 21 04:03:13 UnRAID kernel: pcieport 0000:00:01.0:    [ 6] BadTLP                
Oct 21 04:07:04 UnRAID kernel: pcieport 0000:00:01.0: AER: Multiple Corrected error received: 0000:00:01.0
Oct 21 04:07:04 UnRAID kernel: pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Receiver ID)
Oct 21 04:07:04 UnRAID kernel: pcieport 0000:00:01.0:   device [8086:6f02] error status/mask=00000040/00002000
Oct 21 04:07:04 UnRAID kernel: pcieport 0000:00:01.0:    [ 6] BadTLP                
Oct 21 04:10:54 UnRAID kernel: pcieport 0000:00:01.0: AER: Multiple Corrected error received: 0000:00:01.0
Oct 21 04:10:54 UnRAID kernel: pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Receiver ID)
Oct 21 04:10:54 UnRAID kernel: pcieport 0000:00:01.0:   device [8086:6f02] error status/mask=00000040/00002000
Oct 21 04:10:54 UnRAID kernel: pcieport 0000:00:01.0:    [ 6] BadTLP                
Oct 21 04:19:07 UnRAID emhttpd: read SMART /dev/sde
Oct 21 04:19:07 UnRAID emhttpd: read SMART /dev/sdo
Oct 21 05:04:47 UnRAID kernel: pcieport 0000:00:01.0: AER: Multiple Corrected error received: 0000:00:01.0
Oct 21 05:04:47 UnRAID kernel: pcieport 0000:00:01.0: PCIe Bus Error: severity=Corrected, type=Data Link Layer, (Receiver ID)
Oct 21 05:04:47 UnRAID kernel: pcieport 0000:00:01.0:   device [8086:6f02] error status/mask=00000040/00002000

 

Added the following:

Select > Main
Select > Flash
Scroll down > Syslinux Configuration
 

Unraid OS

kernel /bzimage
append pcie_aspm=off, initrd=/bzroot pci=noaer

 

Also referenced this post regarding pci=noaer

 

Since reboot, there is no further errors. 
Appreciate the assist here everyone!
Thank you!

[evilmouse]

kernel /bzimage
append pcie_aspm=off, initrd=/bzroot pci=noaer

 

Didn't seem to work for me. still getting spammed in my log.

Any help?

[Bjorn Helgaas]

The "pcie_aspm=off" kernel parameter hides a problem.  I would really like to fix the underlying problem so the parameter isn't needed.

 

If anybody is willing to help fix it, please open a bug report at https://bugzilla.kernel.org/, product Drivers/PCI, mention the hardware platform, and attach:

• complete dmesg log (I assume this will include some Correctable Errors)
• output of "sudo lspci -vv"

Try booting with the "pcie_aspm=off" kernel parameter to see if it makes any difference. If it does, please also attach similar dmesg and lspci output for this boot.

This seems similar to https://bugzilla.kernel.org/show_bug.cgi?id=215027, which we originally thought was related to Intel VMD and/or the Samsung NVMe device you have, but I now suspect we might have an ASPM configuration problem.