Unraid 6.9.1 - random crashes every couple days while replacing drives

[MikaelTarquin]

I've been in the process of upgrading drives in my array this week, and twice now the process has been interrupted with a complete system crash. Dockers inaccessible, WebGUI unreachable, pinging the server IP address gets "Destination host unreachable".

 

I first tried cutting power to the network switch the server is on, to see if it was an issue with the ethernet connection, but that didn't help.

 

Plugging in a monitor and keyboard worked for a bit, but when I ran a Diagnostics command it sat there stuck on "diagnostics collection" for about 10-15 minutes before I CTRL+C'd out to try other commands like "ethtool" or "ifconfig". But then those other commands didn't do anything, and unplugging/replugging my keyboard made it so the keyboard was totally unrecognized by the system, forcing me to manually power down. This was in the middle of a 2+ day rebuild of a new 16TB drive replacing a 4TB drive, so it's especially unfortunate having to start that process over again. 

 

The first time this happened earlier in the week was also about halfway through a clearing of a new 16TB drive going into an empty slot.

 

I turned on the local syslog server after the first crash to try catching some record of what is happening, and was able to get these two syslog files that cover this time. I tried but couldn't get the diagnostics file to write before the crash, and I don't think a new one will be much use because it was after my reboot just now, but here it is anyway.

 

I really wish I knew what was causing so many issues. I had occasional issues with previous versions of Unraid, but have been absolutely fraught with problems since upgrading to 6.9.1.

 

Edit: I think today's crash was around 2021-05-08 12:00:00, since I noticed it around 12:15:00.

syslog-127.0.0.1.log syslog-192.168.1.91.log nnc-diagnostics-20210508-1305.zip

Edited May 8, 2021 by MikaelTarquin

[JorgeB]

The are various NMI related stalls/call traces, this can be a hardware issue or current kernel doesn't like your hardware, if it was working with a previous Unraid release you can downgrade to confirm.