Server has been hanging and becoming non-responsive

[wes.crockett]

As the title says, on Saturday, the server started acting up. I could access the unraid UI but not really od anything. Rebooting resulted in nothing. I could access the terminal but I couldn't get the reboot command to work there either. CPU was pegged at 100%. I had to hard boot it.

 

Parity check passed fine after brining the array online.

 

Tonight, I came home after a few hours away and I couldn't even access the unraid UI through web browser and I couldn't ping the box (so obviously I couldn't ssh in either.) iDrac showed no issues of note (but I could reach the idrac on the box) and I couldn't wake up the box to get direct video out. Naturally, I'm leaning to a dreaded hardware issue.

 

I've attached Unraid logs... Hopefully someone more well versed than I can see a glaring issue in here?

 

Syslogs attached

kuiper-diagnostics-20230926-2310.zip

[JorgeB]

Enable the syslog server and post that after a crash.

 

[wes.crockett]

I turned it on last night. I THINK I have it set up correctly.

 

Thank you.

[wes.crockett]

7 hours ago, JorgeB said:

Enable the syslog server and post that after a crash.

 

Hey @JorgeB, I should be seeing logs to the directory I specified, correct? I'm not seeing any generated at this point even though logging is enabled.

[JorgeB]

36 minutes ago, wes.crockett said:

correct?

Correct, make sure you set the Unraid server IP in the remote server filed, it's a common mistake.

[itimpi]

1 hour ago, wes.crockett said:

Hey @JorgeB, I should be seeing logs to the directory I specified, correct? I'm not seeing any generated at this point even though logging is enabled.

It is often easier to set the “mirror to flash” option to get the output written to the ‘logs’ folder on the flash drive.

[wes.crockett]

5 hours ago, JorgeB said:

Correct, make sure you set the Unraid server IP in the remote server filed, it's a common mistake.

I'm all about the common mistakes... like this one... that I made.  Thanks.

 

4 hours ago, itimpi said:

It is often easier to set the “mirror to flash” option to get the output written to the ‘logs’ folder on the flash drive.

So this will write them to my boot drive too then (being the flash drive)? Sounds good.

[wes.crockett]

The log from the boot drive and the log on my cache drive aren't perfectly aligned, so I uploaded both... either way:

 

I came into my office this morning to find the server totally unresponsive again. Had to perform a hard reboot.

 

Logs attached.

syslogs.zip

[JorgeB]

Both logs only cover a few minutes, at what time was the crash?

[wes.crockett]

15 minutes ago, JorgeB said:

Both logs only cover a few minutes, at what time was the crash?

Unknown. Sometime between 11pm and 630ish. I came in this morning and it was unusable. It became unusable overnight.

 

I'm seeing several very similar threads. Going through them, looks like pretty identical symptoms. Hoping that, between all the threads, some light can come as to what may be causing it.

Edited September 28, 2023 by wes.crockett

[JorgeB]

35 minutes ago, JorgeB said:

Both logs only cover a few minutes

Forget that, I missed the first couple of lines, since I was expecting more activity logged before the crash:

 

Sep 27 15:12:16 Kuiper monitor: Stop running nchan processes
Sep 27 21:21:14 Kuiper webGUI: Successful login user root from 192.168.1.162
Sep 28 06:40:04 Kuiper kernel: Linux version 6.1.49-Unraid (root@Develop-612) (gcc (GCC) 12.2.0, 

 

Unfortunately there's nothing relevant logged, this usually points to a hardware issue, one thing you can try is to boot the server in safe mode with all docker/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one.

[wes.crockett]

I'm seeing several people mention downgrading and the issue going away... would this be worth pursuing to ensure it isn't the newest release resulting in the issue?

[JorgeB]

You can try but when there's nothing logged usually it's not release related, unless there's some compatibility issue with the kernel itself.

[wes.crockett]

I just ran /sbin/reboot as a user script. Server rebooted fine but array didn't come back online on its own. Is that normal behavior when running /sbin/reboot?

 

My thinking is trying nightly reboots to see if the issue persists every now and then.

 

EDIT:

I'm dumb... that just flat out reboots Linux. Is there a prebuilt script for safely rebooting Unraid in the proper order?

Edited September 28, 2023 by wes.crockett

[JorgeB]

17 minutes ago, wes.crockett said:

Is that normal behavior when running /sbin/reboot?

No.

 

17 minutes ago, wes.crockett said:

EDIT:

I'm dumb... that just flat out reboots Linux. Is there a prebuilt script for safely rebooting Unraid in the proper order?

Reboot works with Unraid, it will start a clean shutdown then reboot.

[wes.crockett]

It would help if I had auto-start enabled in disk settings.

[wes.crockett]

Ever since downgrading to 6.12.3, I have had no issues with my system.