Hi. My Unraid server seems to be crashing. Its crashed 4 times in the last 16 hours. I've had it running flawlessly for 8 months until yesterday. Each time I have needed to reboot the server manually (long press the power button, or press the reset button on the front) to get it to boot up again.

 

First time it crashed, I restarted the server, then started the array. A hour or so later it crashed in the middle of the parity check. Restarted again, started the array, it seemed to crash almost immediately. Restart. Ran memtest for a single pass, and passed. Started the array and dumped the log, turned off all my dockers except for 4 that I use all the time (they run together) and copied over my config and USB backup to my desktop. This time it seemed to crash as soon as that copy was finished. I didn’t catch the moment to see if it was exactly the same time, but it seems close to me. 

 

I’ve been running a continuous ping test to see when it crashes. It completely drops off my network, and all drive activity stops.

 

 

I asked the Discord for help. My docker is running ipvlan for my dockers (was asked to check this from the discord), then Unraid crashed while sitting idle. I had not started the array.

 

I'm now running a longer memtest. So far, 12 hours in and its still passing. But I'm going to let it run for 24-72 hours.

 

Any thoughts?

newborg-diagnostics-20240728_1223.zip

If it started crashing out of the blue it looks more like a hardware issue, but enable the syslog server and post that after a crash, in case there's something there.

14 minutes ago, JorgeB said:

If it started crashing out of the blue it looks more like a hardware issue, but enable the syslog server and post that after a crash, in case there's something there.

A hardware problem is also what I'm leaning towards. It's hardware is just old PC gaming parts from a last build, so they all have 6+ years of use on them. 

 

I'll do the syslog in a few days. I want to let memtest run a long time first. I also wont have time with my work schedule to really mess with it for a few days anyway

Edited July 29, 20241 yr by indianazoidberg

I have a update that is probably not good, but I would like to hear what others think.

 

I have been running memtest for 48 hours. Zero errors were found. I then tried to boot into unraid, but now I unraid wont even load. Normal, or safemode. And it seems to do it in a different place when it tries to boot. I tried this around 10 times today. So I cant turn on syslog like @JorgeB recommended.

 

I have also noticed that on the monitor I've attached to my server for troubleshooting, when it stops booting, the text at the end seems to get slightly corrupted. I've attached photos showing that, and some of the other screens that its shown when its died. I've also tried removing ram sticks, and swapping them in case there is a faulty stick, but that hasn't seemed to change anything.

 

Questions:

1. Could my CPU be going bad/failed? Or is ram still the most likely issue. 

2. Is there anyway check the integrity of my usb drive? I could do this on windows, or linux (if someone could walk me through that process). I managed to pull a usb backup before this went REALLY down hill. But I would also guess if that was the case, it would probably fail to boot in a more reputable way and not seemingly randomly?

3. If this is a hardware failure, I'm probably just going to upgrade my system with a new motherboard, cpu, and ram. Unraid and the dockers it runs have become a thing that I am now using so often that its extremely inconvenient to loose access to it. Should I look into a system that supports ECC ram? Is that worth it?

 

Any other ideas before I start shooting the parts cannon at the problem?

Edited July 31, 20241 yr by indianazoidberg
Added another image

You can try booting with a new flash drive using a stock install, no key needed, if it also fails it will confirm a hardware issue.

2 hours ago, JorgeB said:

You can try booting with a new flash drive using a stock install, no key needed, if it also fails it will confirm a hardware issue.

Okay. And that won't fuck anything up in that long term?

Nope, just for boot testing.

16 hours ago, JorgeB said:

You can try booting with a new flash drive using a stock install, no key needed, if it also fails it will confirm a hardware issue.

Just tried this a few times. Still won't boot. I guess I will use this as an opportunity for an upgrade. Just sooner than I thought. I'm just going to get a new motherboard, CPU, and RAM. I'm going with the assumption that my PSUs, HBA, and backplane are all still working (the HBA shows drives connected to it on boot, and its showing everything I believe)