Complete Server Lockup 6.11.x

[fiore00713]

Hey All,

I've been trying to track down an issue that's plagued me since updating my system to 6.11

I'm running into instances where my serer is becoming completely locked up and unresponsive

Dockers appear to crash as my NGINX ping test is the first to report a problem if I'm not around

Web interface, SSH, IPMI console and local console are all a no go. The only way that I've found to recover is by a hard reset.

I've turned on syslog mirroring to the flash drive but this appears to be happening in such a way that that's not even writing

 

I've got a few dockers that are running with a eth0 interface and I had previously disabled all but my PiHole instance, following this most recent occurrence I've disabled that one for good measure too

 

Looking to see where I may have something jacked up at.

 

One addition oddity is that I've got some old network interfaces that get listed as "do not exist" during boot
I had looked at the network config file but did not find the references there, if that may be causing an issue somewhere?

 

unraid-diagnostics-20221104-1611.zip

[JorgeB]

13 hours ago, fiore00713 said:

I've turned on syslog mirroring to the flash drive but this appears to be happening in such a way that that's not even writing

Nothing there or nothing relevant?

 

One thing you can try is to boot the server in safe mode with all docker/VMs disable, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one.

 

 

[fiore00713]

On 11/5/2022 at 5:49 AM, JorgeB said:

Nothing there or nothing relevant?

 

One thing you can try is to boot the server in safe mode with all docker/VMs disable, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one.

 

 

 

Nothing there from the time when the system would have encountered an issue. I was first tipped off by a ping monitor on my NGINX and there was no corresponding logs that were mirrored to the flash drive from that time. I had logs from overnight when backups and docker updates ran, and then subsequently after I had to hard-reset the system but nothing in-between to help me track down anything.

[fiore00713]

I ran for 19 days before experiencing the issue again. I thought for a moment that it may have been oddly related to a Win11 VM I had but the lockup proved that wrong.

In reviewing the syslog it appears that there may be some info captures but I'm not entirely certain how to interpret it if anyone is able to assist in my efforts

 

unraid-diagnostics-20221207-1743.zip

[JorgeB]

Those are the diags after rebooting so not much to see.

[fiore00713]

Not sure if I should be revitalizing this thread or starting a new one. I'm still having intermittent issues with this even after completing a rebuild of the system. I DID manage to catch an error in the log window coinciding with a lockup this morning. It would appear that it may be a configuration issue, but I'm not 100% sure how to interpret this.

If anyone can make heads or tails of this and at least get me down the right path

 

As an aside, I have VMs currently disabled (disk failure, waiting on RMA)

 

I also have a replacement 10g NIC I was going to work on replacing when I had the chance, should it be a hardware issue on the NIC (one of the only things I haven't yet replaced)

 

Excerpt from the log window attached

unraid_error_capture.txt

[JorgeB]

Try switching to ipvlan (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)).