unraid going unresponsive

[daemian]

Hello,

 

I have been running unraid for many years. I have had issues at times, but for the last 1-2 years its been extremely stable. Then, seemingly out of nowhere, I stated having stability issues. I cannot think of any notable changes. The server will run fine for a few hours, but then the dockers crash and the GUI goes unreachable. I am not able to connect via SSH either. If i hit the local console it responds initially, and I am able to enter a username, but then i never get to enter the password (it times out after 60 sec and goes back to the username prompt.

 

After it started happening, I did try upgrading to 6.12.3, but that did not help, so i went back down to 16.12.1

 

Attached is a diagnostics taken after the most recent crash. Hoping someone can take a look and help me figure out what is causing this. Its driving me crazy!

 

TIA!

ur01.marx.local-diagnostics-20230812-2205.zip

[nerbonne]

Are you using macvlan?  If so, that caused me issues on every server until I switched to ipvlan.

 

Also your docker img being 30gb probably doesn't help, it shouldn't need to be over 20gb.  What is filling your docker image up?

Edited August 13, 2023 by nerbonne

[JorgeB]

Enable the syslog server and post that after a crash.

 

[daemian]

i did also switch from macvlan to ipvlan a while back (last time i had stability issues) and found it made a massive difference. I did double check that i am still set to ipvlan.

 

Syslog is attached from the latest crash.

 

Thanks,

syslog

[JorgeB]

Unfortunately there's nothing relevant logged, this usually points to a hardware issue, one thing you can try is to boot the server in safe mode with all docker/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one.

[daemian]

Thanks for the assistance. I ran unraid in safe mode for several days without issues, so i rebooted into "regular" mode and on a hunch started everything but my Frigate docker. Low and behold it ran for a week without issue. Yesterday i started Frigate back up and it crashed in a few hours.

So clearly its that docker causing the issue and I can try to get assistance with that specific application for root cause.

Thinking about it though and I have a followup questions - is there anything i can do to prevent that docker from crashing the entire system? For instance, can I throttle its access to resources so it cannot choke everything out (at least thats what I assume is happening)? Or are there are prevention/protection measures i can put in place?

[JorgeB]

15 minutes ago, daemian said:

is there anything i can do to prevent that docker from crashing the entire system? For instance, can I throttle its access to resources so it cannot choke everything out (at least thats what I assume is happening)?

Not sure, docker is not my main area, but someone else may be able to answer that.