Server becoming unresponsive multiple times a day

[eman31]

Hi,

 

Currently running 6.12.8 and the past couple of weeks started having problems with the server becoming unresponsive. Prior to this I was up running for months without issues. When it happens I can't access anything and have to hard reset the server. I've been trying to look at the previous logs and the only thing I've been seeing is that it happens right after the mover is finished. Not every time the mover runs but when it hangs, that is the last entry in the log.

 

Current hardware is:

Micro-Star International Co., Ltd. MEG X570 ACE (MS-7C35) Version 1.0

AMD Ryzen 7 3700X 8-Core @ 3600 MHz

32 GiB DDR4

 

Any help would be appreciated.

tower-diagnostics-20240229-2012.zip

[itimpi]

The syslog in the diagnostics is the RAM copy and only shows what happened since the reboot.   It could be worth enabling the syslog server to get a log that survives a reboot so we can see what happened prior to the reboot. The mirror to flash option is the easiest to set up, but if you are worried about excessive wear on the flash drive you can put your server’s address into the Remote Server field.

[eman31]

I setup remote syslog and this is what I have. Hopefully I did it correctly to capture everything. There were two times that it became unresponsive again after the mover ran where I had to hard reboot. One at 01:00 and again at 13:00 today. I did run the mover manually after the 01:00 and it ran with no issues and a few times after that as well.

syslog-192.168.50.68.log

[JorgeB]

Nothing obvious logged, make sure this has been taken care of:

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/?do=findComment&comment=819173

 

[eman31]

Thanks for the suggestions. After the last lock up yesterday, I updated my bios and changed the Power Supply Control and disabled C-States. Also ran a memory test for 24 hours to rule a RAM problem out. It came back clean so that appears to be good. I will update after running for a while to see if this fixes things.

[eman31]

No luck after all the changes I made. Went down after about 3 hours. Attaching current syslog to see if anything is relevant.

syslog-192.168.50.68-03-03-24.log

[JorgeB]

I'm afraid there's still nothing relevant.

[eman31]

I tried running in safe mode yesterday and while it lasted longer it still locked up overnight. Not sure where to take my troubleshooting to next. 

 

Edit: I just read that I should have dockers disabled in safe mode which I did not do. Will give it another try with that. 

Edited March 4 by eman31

[eman31]

I've been running in safe mode for about 72 hours with no lock ups. What's a reasonable length to go to rule out a hardware issue? Once that time has passed, do I just start turning on dockers one by one and letting it run for a while to see if one is causing the problem? 

[JorgeB]

50 minutes ago, eman31 said:

What's a reasonable length to go to rule out a hardware issue?

I would say take the time it usually was needed for a crash a double it.

[eman31]

After letting things run for a while, starting all my dockers one by one and letting them run for at least 24 hours before starting another one, it looks like the issue may be with my binhex-jellyfin application. As soon as I turn it on it eats up all my ram and pegs the cpu to 100%. I didn't let it run long enough for the system to crash but none of the others have acted like that.

Edited March 16 by eman31

[JorgeB]

Try asking for help in the support thread/discord for that container:

 

[eman31]

Rebooted out of safe mode without Jellyfin running and system locked up on me again after a couple of hours. Ran fine in safe mode for two weeks. No idea what the issue is here. I don't see anything in the syslog at the time it locked up.

syslog-192.168.50.68 - 03-17-24.log

[JorgeB]

11 hours ago, eman31 said:

Rebooted out of safe mode without Jellyfin running and system locked up on me again

Were other container still running?

[eman31]

Quote

6 hours ago, JorgeB said:

Were other container still running?

I had started all of the same containers that were running ok in safe mode. Basically everything other than Jellyfin.

 

[JorgeB]

If it doesn't crash in safe mode with the same containers running it could be a plugin.

[eman31]

5 minutes ago, JorgeB said:

If it doesn't crash in safe mode with the same containers running it could be a plugin.

I was thinking along the same lines yesterday but didn't see an obvious way to stop them and have never had a reason to before. Is there a way to turn off individual plugins like with the containers or do they need to be uninstalled?

[JorgeB]

You can rename the *.plg files, then reboot, to reinstall rename back.

[itimpi]

Just now, eman31 said:

I was thinking along the same lines yesterday but didn't see an obvious way to stop them and have never had a reason to before. Is there a way to turn off individual plugins like with the containers or do they need to be uninstalled?

There is no GUI support for disabling a plugin without uninstalling.

 

you can do it manually by renaming the relevant .plg file on the config/plugins folder on the flash drive to have a different file extension (e.g .plgx) and then rebooting..   Reversing the process re-enables the plugin.    Advantage is any downloaded files and/or settings for the plugin remain intact on the flash drive.