Nightly array failures

[BusterBrawls]

sol-diagnostics-20210109-0946.zip

 

Greetings, 

 

I upgraded my motherboard/ram/cpu and installed a new HBA and since Unraid has been somewhat unstable.

 

I've noticed two issues:

1) Three or so times in the last 2 weeks the server will go completely unresponsive in middle of the night. The array will go offline and I'm forced to reboot with a hard reset. This happened last night.

 

2) Quite a few times in the last week I've woken up in the morning to find that Docker is running and some containers that should be running are not started. They will not start and throw a generic "server error" message. Other containers continue to run until I restart them. The only way to resolve that is to initiate a reboot from the GUI.

 

I have attached is the system diagnostics I took this morning. And I am hoping someone will see something that I obviously missed. 

 

Notes:

* I've been running Unraid for many years and have never done a clean rebuild. Maybe I have something "legacy" configured that's causing issues?

* I'll say that this MSI motherboard (C246M-WU4) is finicky. Changes to the BIOS unrelated to USB will cause the usb disk to not boot and I get stuck in a boot-loop. I can't for the life of me figure out how to get into the LSI SAS 9300-16i controller firmware at boot time.    

* I tried upgrading to 6.9.rc2 then reverting back to 6.8.3.

 

I am having a difficult time troubleshooting this one as I can't view any log files after a reboot. Tips for troubleshooting a problem like this would be helpful. Maybe someone will notice something obvious in the diagnostics that I have missed myself. 

 

Thanks for the assist. 

 

Bob

 

 

*edit for clarification

Edited January 9, 2021 by BusterBrawls

[JorgeB]

You can try this, another thing you can try it to boot the server in safe mode with all docker/VMs disable, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one.

[BusterBrawls]

That was very helpful. Thanks. I enabled the syslog server and now have a better visual on system logs.

 

What was looking like multiple bad disks, or maybe a bad HBA turned out to be a bad stick of RAM.