In the last week my server has become unresponsive two times at the point that SSH connection through Putty wasn't responding, WebUI wasn't loading (Connection timeout), Glances webui was in timeout too, no other Docker service was responding.

The only thing that gave me a heartbeat was a ping to which the server did respond. The server did not have any recent modification, last thing was the setup of Immich a month ago.

 

Given the situation the only action I could perform was a forced shutdown 😖, upon rebooting the server resumed functionalities as normal.

 

I have Glances installed which I used in the pas to explore hungry processes, however, it's a live tool and doesn't retain historical data.

 

To troubleshoot the issue, I am trying to deactivate everything and activate things one by one, but I surely feel like I need something to retain historical data in case a new disaster occurs. What (how) can I setup to ensure Unraid logs, system metrics, disks metrics, processes, docker containers system usage etc are logged and retained across reboots\forced shutdowns?
 

For what's worth I have another machine where I could send data (or pull it from Unraid).

 

---

After disabling all the containers dockerd is still consuming a huge amount of CPU. I'm investigating if there is something corrupted with docker maybe, but I'm really going blindly into the unknown.

 

Executing from SSH `/etc/rc.d/rc.docker stop` frees up 100% of the CPU. But basically I am unable to restart it because if I execute `/etc/rc.d/rc.docker start` the dockerd process spikes again to 300% CPU usage with 0 containers running (I disabled the autostart on all of them).

 

 

 

Edit:

- Attached diagnostics zip

tower-diagnostics-20250121-1103.zip

Edited January 21, 20251 yr by rug
Attached htop

Make sure this has been taken care of

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/page/2/#findComment-819173

 

If it keeps crashing, enable the syslog server and post that after a crash.

20 hours ago, JorgeB said:

Make sure this has been taken care of

https://forums.unraid.net/topic/46802-faq-for-unraid-v6/page/2/#findComment-819173

 

If it keeps crashing, enable the syslog server and post that after a crash.

 

Thanks, I will try to address this first.

After a year it happened again, server unresponsive:

• can't SSH into it

• Web GUI not loading (times out)

• Disks i/o led blinking and fans spinning like if it was doing something

Had to force shutdown the server but this time I do have the syslog file, what should I look for to troubleshoot the issue? Can I post it on the forum or does it contain sensitive information?

I also did change c-state settings last year as recommended in the above mentioned thread, but is there anything I can check to be sure whether the issue is still related to c-states or something else? I wouldn't mind re-enabling the c-states for power saving, at the time I changed the setting "blindly" without a confirmation that this was actually my issue. AMD Ryzen 5700G.

The server is headless so I am also thinking to buy a kvm to remotely access the screen and try to reboot it from the shell as if I was with screen and keyboard attached to the server (assuming at least that works).

If it's once a year, it won't be easy to troubleshoot, but you can enable the syslog server to save to a share and see if that catches something.

9 hours ago, JorgeB said:

If it's once a year, it won't be easy to troubleshoot, but you can enable the syslog server to save to a share and see if that catches something.

I had that enabled and I do have the syslog file of the day of the recent freeze but I don't know what to look for and if it's safe to share on the forum.

By the way a change I applied to the system recently was to set the Power Mode to power saving, previously on Best performance). Maybe that re enabled low c-states?