Unraid Server randomly Crashing

[BrainGameFactory]

Hi, My unraid server crashed last week for the first time. I had to press the hardware reset and then it came back. Yesterday my Computer was lagging and I found that it was my server through the SMB shares. The Server was under constant CPU load and was really slow, but it responded. htop showed that docker used the CPU (ScreenShot). I tried everything to bring it back but had to push the reset button again. Then i disabled docker and Started the Array and thought that should at least keep it from Crashing. The automatic Parity check started. When i came from work today the server was completely locked up again. Had to press Reset and then copied the important data. 1 Hour ago it crashed again. I literally have no clue what's happening.

 

itanium-diagnostics-20220830-2213.zip

[dlandon]

If it crashed when doing a parity check, you may have a power supply problem.  Run a memory test.  Try running it in safe mode and see if i crashes.

[trurl]

 

[BrainGameFactory]

Thanks for the Quick Answers. I did some testing: Tried a different Power Supply, Deactivating Docker and deleted the Docker.img. Now Unraid runs normally when the Docker service is not enabled but when I enable Docker it crashes in a Day. Now I have no clue how to get Docker working again.

[JorgeB]

Enable the syslog server and post that after a crash, or if you have containers with a custom IP address update to v6.10.3 and switch to ipvlan, it should help (Settings -> Docker Settings -> Docker custom network type -> ipvlan (advanced view must be enabled, top right)).

 

 

[BrainGameFactory]

I have the Syslog server activated. Here is the last Syslog when the server Crashed. But I can not really tell when the Server Crashed.

Syslog Crash.txt

[JorgeB]

Not much there, are sure it was always active? Also see the ipvaln setting like mentioned above.

[trurl]

19 hours ago, BrainGameFactory said:

runs normally when the Docker service is not enabled but when I enable Docker it crashes in a Day

I wonder if you don't have some container host path in rootfs instead of actual mounted storage. If you fill up rootfs the OS no longer has any space to work in.

 

What do you get from the command line with this?

df -h /

 

[BrainGameFactory]

On 9/16/2022 at 8:44 AM, JorgeB said:

Not much there, are sure it was always active? Also see the ipvaln setting like mentioned above.

 Yes, I should always be active, to my knowledge. I enabled Docker again (i have not a single container installed) to have a new Syslog report. Strangely the server did not crash in the first 24h. But today was the Parity check, it started normally but went for some reason extreme slowly after some time. And also some CPU cores were reporting 100% load. The overall response time was really bad too. I was only able  to take a screenshot from the CPU load reporting. After that the web interface did not load at all anymore. Had to press reset again.

 

Syslog Crash2.txt

[BrainGameFactory]

On 9/16/2022 at 3:15 PM, trurl said:

What do you get from the command line with this?

df -h /

root@Itanium:~# df -h /
Filesystem      Size  Used Avail Use% Mounted on
rootfs           16G  812M   15G   6% /
root@Itanium:~# 

 

[trurl]

That looks normal, as expected since you had already rebooted. Try it again later to see if Use% increases.

[BrainGameFactory]

So the Server was running perfectly, until the Parity Check in the morning. I restarted the server and monitored the automatic Parity Check. But strangely, it completed without any problems. The server was normal afterwards. The Rootfs was also not any larger. The Parity Check was completed at around 17 p.m. I monitored the server afterward and at around 20 p.m. it happened. Some cpu cores were at 100% and the web interface wasn't working. I noticed the kernel log is getting updated. That's the only thing I am getting from the server right now.kernel.txt

[JorgeB]

There's constant crashing on the syslog, looks more like a hardware issue, one thing you can try is to boot the server in safe mode with all docker/VMs disable, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one.

[BrainGameFactory]

Today the Server Crashed while in Safe mode. Before the Crash i got the message that there are Errors on the Array. Any ideas what the Hardware issue might be ? And what should I do next ?Syslog Kernel Itanium.txt

[JorgeB]

Log if filled with call traces, difficult to look for anything else.