Hello everyone,  

After upgrading from Unraid 6.12.9 to 6.12.13, I’ve been experiencing random disconnects where my server becomes unreachable. I’ve set up a syslog server to capture logs, but it hasn’t recorded anything unusual during the incidents.  

Nov 21 10:21:58 AIO kernel: pcieport 0000:00:01.1: AER: Corrected error message received from 0000:02:00.0
Nov 21 10:21:58 AIO kernel: nvme 0000:02:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
Nov 21 10:21:58 AIO kernel: nvme 0000:02:00.0:   device [15b7:5019] error status/mask=00000001/0000e000
Nov 21 10:21:58 AIO kernel: nvme 0000:02:00.0:    [ 0] RxErr                 
Nov 21 10:26:21 AIO kernel: pcieport 0000:00:01.1: AER: Multiple Corrected error message received from 0000:02:00.0
Nov 21 10:26:21 AIO kernel: nvme 0000:02:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
Nov 21 10:26:21 AIO kernel: nvme 0000:02:00.0:   device [15b7:5019] error status/mask=00000001/0000e000
Nov 21 10:26:21 AIO kernel: nvme 0000:02:00.0:    [ 0] RxErr                 
Nov 21 10:27:22 AIO kernel: pcieport 0000:00:01.1: AER: Multiple Corrected error message received from 0000:02:00.0
Nov 21 10:27:22 AIO kernel: nvme 0000:02:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
Nov 21 10:27:22 AIO kernel: nvme 0000:02:00.0:   device [15b7:5019] error status/mask=00000001/0000e000
Nov 21 10:27:22 AIO kernel: nvme 0000:02:00.0:    [ 0] RxErr                 
Nov 21 10:27:36 AIO kernel: pcieport 0000:00:01.1: AER: Corrected error message received from 0000:02:00.0
Nov 21 10:27:36 AIO kernel: nvme 0000:02:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
Nov 21 10:27:36 AIO kernel: nvme 0000:02:00.0:   device [15b7:5019] error status/mask=00000001/0000e000
Nov 21 10:27:36 AIO kernel: nvme 0000:02:00.0:    [ 0] RxErr                 
Nov 21 10:38:55 AIO kernel: pcieport 0000:00:01.1: AER: Corrected error message received from 0000:02:00.0
Nov 21 10:38:55 AIO kernel: nvme 0000:02:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
Nov 21 10:38:55 AIO kernel: nvme 0000:02:00.0:   device [15b7:5019] error status/mask=00000001/0000e000
Nov 21 10:38:55 AIO kernel: nvme 0000:02:00.0:    [ 0] RxErr                 
Nov 21 11:04:30 AIO kernel: pcieport 0000:00:01.1: AER: Corrected error message received from 0000:02:00.0
Nov 21 11:04:30 AIO kernel: nvme 0000:02:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
Nov 21 11:04:30 AIO kernel: nvme 0000:02:00.0:   device [15b7:5019] error status/mask=00000001/0000e000
Nov 21 11:04:30 AIO kernel: nvme 0000:02:00.0:    [ 0] RxErr                 
Nov 21 11:04:30 AIO kernel: pcieport 0000:00:01.1: AER: Corrected error message received from 0000:02:00.0
Nov 21 11:04:30 AIO kernel: nvme 0000:02:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
Nov 21 11:04:30 AIO kernel: nvme 0000:02:00.0:   device [15b7:5019] error status/mask=00000001/0000e000
Nov 21 11:04:30 AIO kernel: nvme 0000:02:00.0:    [ 0] RxErr                 
Nov 21 11:10:48 AIO kernel: pcieport 0000:00:01.1: AER: Corrected error message received from 0000:02:00.0
Nov 21 11:10:48 AIO kernel: nvme 0000:02:00.0: PCIe Bus Error: severity=Corrected, type=Physical Layer, (Receiver ID)
Nov 21 11:10:48 AIO kernel: nvme 0000:02:00.0:   device [15b7:5019] error status/mask=00000001/0000e000
Nov 21 11:10:48 AIO kernel: nvme 0000:02:00.0:    [ 0] RxErr

Part of logs

 

To further diagnose the issue, I wrote a script to ping my gateway (192.168.31.1) at regular intervals. Below are the logs from the script. The disconnect seems to have occurred around 11:20 AM:  

 

2024/11/21 11:18:02: Start monitoring 192.168.31.1  
2024/11/21 11:18:05: 192.168.31.1 is reachable  
2024/11/21 11:18:20: 192.168.31.1 is reachable  
2024/11/21 11:18:35: 192.168.31.1 is reachable  
2024/11/21 11:18:50: 192.168.31.1 is reachable  
2024/11/21 11:19:05: 192.168.31.1 is reachable  

Given that the syslog didn’t capture anything significant, I suspect this might be related to a kernel crash or perhaps a complete failure of network-related modules.  

Does anyone have advice on how I can further diagnose this issue? Are there specific kernel logs or tools that might help me narrow down the root cause?  

Thank you in advance for your insights!  

aio-diagnostics-20241121-1325.zip syslog-127.0.0.1.log

This diagnosis was generated about 12 minutes after the last forced restart after the loss of connection, and may not contain any useful information

I will try some of the methods here, which may take several days.

By the way, this random loss of connection must occur within a day. It has occurred three or four times now, with different durations.

 

Also, if there is anything valuable in my diagnostic file that could cause the loss of connection, please let me know.

This may gwt rid of the PCIe errors, to help with the log spam:

https://forums.unraid.net/topic/118286-nvme-drives-throwing-errors-filling-logs-instantly-how-to-resolve/?do=findComment&comment=1165009

 

On 11/21/2024 at 4:35 PM, JorgeB said:

This may gwt rid of the PCIe errors, to help with the log spam:

https://forums.unraid.net/topic/118286-nvme-drives-throwing-errors-filling-logs-instantly-how-to-resolve/?do=findComment&comment=1165009

 

thank you，but this is not the main issue

I downgraded to version 6.12.9, and it has been running stably for 2 days now. I've given up on upgrading to version 6.12.13

7 hours ago, RWDai said:

but this is not the main issue

I'm aware, but if it fixes the log spam it may help finding the actual problem.