Unraid array stops responding

[oliver]

Hi, suddenly today out of nowhere my unraid system is unresponsive, usually within minutes of starting the array.  This started completely out of the blue, with no hardware or software changes for a while.  Running 6.12.10

 

I've attached diagnostics but here are some observations/behaviors.  

• WEBUI is not responsive.
• Docker containers are not responsive.
• Shares are not responsive.
• VM's ARE responsive and appear to function normally.
• SSH is responsive
• During all of this, syslog continues writing, but there are no errors. 
• When I try to run `ls /mnt`, the terminal freezes - I am never able to view the mount directory.

 

Throughout all of this, syslog continues to write but I don't see any errors or anything pop up.  Everything just....freezes.  

 

This either seems to be hardware failure or something docker related.  

 

I'm leaning towards a docker container being the problem.  I noticed random port allocation/execution errors pop up once I enable docker.  Not sure which container it could be as I didn't make any changes recently.  Is it possible to disable autostart so I can go one by one?  

 

EDIT:  After doing more testing, im increasingly convinced this is some docker issue, particularly networking related.  I get random errors across all containers, that stop and go in between service restarts.  I'm using ipvlan.  

 

 

I've attached unraid logs, which should include previous logs, as I had that enabled copy to flash drive on shutdown.  
 

nas-diagnostics-20240510-1844.zip

Edited May 11 by oliver

[JorgeB]

Enable the syslog server and post that after it happens again, to see if there's something there.

[oliver]

19 minutes ago, JorgeB said:

Enable the syslog server and post that after it happens again, to see if there's something there.

 

Hi, I did have that enabled at that time via a remote syslog server, attached 

logs.txt

[JorgeB]

Unfortunately there's nothing relevant logged, if you suspect a container you can run the server with all but one enable, if it still happens try leaving a different one disabled.

[oliver]

2 minutes ago, JorgeB said:

Unfortunately there's nothing relevant logged, if you suspect a container you can run the server with all but one enable, if it still happens try leaving a different one disabled.

yeah, thanks it's running stable now although im very carefully managing containers.  the symptoms seem somewhat similar to this 

 

 

 

it's very odd.  the unraid GUI seems to not by in sync with what docker ps shows as up and running in certain scenarios.  im gonna try rebuilding all of the containers like OP there is doing.  

 

 

[oliver]

2 hours ago, JorgeB said:

Unfortunately there's nothing relevant logged, if you suspect a container you can run the server with all but one enable, if it still happens try leaving a different one disabled.

 

there is something terribly wrong with unraid.  containers with autostart marked as 'off' are starting anyways.  stop/start commands don't do what they shoud.  i can't even kill the containers when this happens, i get:

 

Error response from daemon: cannot stop container: <container-id>: tried to kill container, but did not receive an exit event

 

what would you say is the best way to start from scratch here without losing my appdata?  i do have backups.  

 

and on a secondary note, why is an errant docker container enough to completely halt the system?  im re-thinking using docker containers on unraid because they shouldn't be able to completely lock up the array in this way.

 

 

EDIT: just downgraded to 6.12.8.  even though i had been running 6.12.10 for a few days, i immediately noticed two things:

• docker is far more responsive
• the glitch with containers not respecting autostart is fixed

too soon to tell if it's permanently fixed.

 

Edited May 11 by oliver