unRaid Server freezing up, how to troubleshoot?

[Kristophersson]

I've had this issue with my server freezing up for a fairly long time now. By freezing up I mean that it's utterly unresponsive. Any site hosted on it becomes unavailable, it's not visible in the network anymore, and when I connect a keyboard it doesn't get power (for example CapsLock LED won't light up).

 

So basically the only thing I can do whenever this happens is to force a shutdown by long-pressing the power button, and that's just super destructive. Are there any logs I can look at to try and identify the source of the problem?

[JorgeB]

Enable the syslog server and post that after a crash together with the complete diagnostics.

[Kristophersson]

41 minutes ago, JorgeB said:

Enable the syslog server and post that after a crash together with the complete diagnostics.

Will do, thank you

[Kristophersson]

On 5/4/2023 at 7:53 PM, JorgeB said:

Enable the syslog server and post that after a crash together with the complete diagnostics.

Okay, it's happened again. I've attached the syslog file which I copied off of the thumbdrive right after the failure. Also the (anonymized) diagnostics which I've extracted first thing after rebooting

 

I also ran memtest for about 20 minutes, and it did't find any errors

 

I have a feeling that this is something to do with my CPU. I've already swapped out the RAM and motherboard to no avail.

 

One possible fix I'm considering (apart from getting a new CPU) is rebooting the system once per day (like 2AM), and see if the error occurs again. But I'm curious as to what the pro's are able to find in my logs.

 

One interesting addage, when I one time increased the RAM assigned to a Ubuntu VM (from 8GB to 10GB, out of a complete 32GB), the error occured much sooner, like less than 24hours after rebooting.

syslog tower-diagnostics-20230507-2254.zip

[JorgeB]

Unfortunately there's nothing relevant logged, this usually points to a hardware issue, also take a look here, another thing you can try is to boot the server in safe mode with all docker/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one.

[Kristophersson]

12 hours ago, JorgeB said:

Unfortunately there's nothing relevant logged, this usually points to a hardware issue, also take a look here, another thing you can try is to boot the server in safe mode with all docker/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one.

I've done this for now:

Quote

look for "Power Supply Idle Control" (or similar) and set it to "typical current idle" (or similar

and I'm seeing if this fixes it, otherwise I'll explore other options, thank you so much for your help, I appreciate it

[Kristophersson]

On 5/8/2023 at 10:51 PM, Kristophersson said:

look for "Power Supply Idle Control" (or similar) and set it to "typical current idle" (or similar

9 days of uptime and no incidents, so far looking like this did it

[Kristophersson]

Over the weekend I upgraded to the latest unraid version (on 6.12.1 now), and it started happening like described in my original Post. I checked the BIOS config and all my changes are still present