Been running my Unraid server for a few years without issue. In the last couple of weeks I've had a few occurances of it grinding to a halt - some apps still run, but the Unraid UI is inaccessible, and even SSH is inconsistent. Apps are typically not accessible.

 

A reboot clears it for a while but then I can't grab the diags. Today it happened again, but I was able to use the GUI. Fix Common Problems indicated "Out Of Memory errors detected on your server". I was also able to grab a diag download.

 

I assume one of my containers if probably to blaim, but the system is so unresponsive that I'm struggling to troubleshoot it at the time.

server-diagnostics-20241130-1548.zip

I have not seen a out of memory error in unraid. you may have a phtom mount not wrting to the disk and wirting to the rot fs in ram.

do you have vms are you over provisoned with mutiple running?

 

Problem appears to be postgres, so likely the container that uses that needs to be limited or reconfigured.

19 hours ago, Nuuki said:

I assume one of my containers if probably to blaim, but the system is so unresponsive that I'm struggling to troubleshoot it at the time.

Probably Immich

 

You also have a container flapping, check the docker page in advanced view which container has the lowest uptime or using a port that's already in use

 

Nov 30 01:26:52 Server kernel: docker0: port 13(veth2b88528) entered blocking state
Nov 30 01:26:52 Server kernel: docker0: port 13(veth2b88528) entered disabled state
Nov 30 01:26:55 Server kernel: docker0: port 13(veth2b88528) entered blocking state
Nov 30 01:26:55 Server kernel: docker0: port 13(veth2b88528) entered forwarding state

 

23 hours ago, Michael_P said:

Quote

Probably Immich

 

It looks like the postgres container was to blame. I use that for about a dozen application - the one exception being Immich, which I have moved to its own postgres instance. I've bounced back and forth on whether to have a dbms per app stack or one shared one - up to now I've gone the shared route as I'm not using compose and it seemed more elegant, but maybe its time to split postgres out, to make troubleshooting and resource management easier.

 

23 hours ago, Michael_P said:

Quote

 

You also have a container flapping, check the docker page in advanced view which container has the lowest uptime or using a port that's already in use

 

Nov 30 01:26:52 Server kernel: docker0: port 13(veth2b88528) entered blocking state
Nov 30 01:26:52 Server kernel: docker0: port 13(veth2b88528) entered disabled state
Nov 30 01:26:55 Server kernel: docker0: port 13(veth2b88528) entered blocking state
Nov 30 01:26:55 Server kernel: docker0: port 13(veth2b88528) entered forwarding state

 

Good to know. I've rebooted since then so I can't check which container that was I don't think, but I'll keep an eye on the logs to see if it crops up again.

 

On 12/1/2024 at 10:32 AM, JorgeB said:

Problem appears to be postgres, so likely the container that uses that needs to be limited or reconfigured.

Thanks! I've added a memory limit for now, but I guess I'll need to do more digging if it starts running out of control again. As noted above its a shared postgres container, so maybe its time to run 1 per application...