I've been experiencing server instability for about a month or more, with what feels like escalating frequency. At this point, I suspect a hardware problem, but I would appreciate any insight about where to look next for troubleshooting.

 

Symptoms: after anywhere from 6-48 hours of uptime, server becomes unreachable.  If I can access unraid at all, the docker tab will claim to crash but some docker services may still be reachable online, (which is extremely odd). Sometimes I cannot access the web portal at all, but the server will claim to be online via Unraid Connect.

 

What I have done:

deleted my docker image and reinstalled apps

rebuilt my boot drive (after finding a boot sector error, which I had hoped was the source).

disk checks in maintenance mode for all array disks with and then without "-n" (x2)

 

 

Given the regretful regularity in which I've force-reset my system over the last month, file and docker corruptions have not been a surprise, but it's also making it challenging to identify the root cause of the problem.

 

Today I had to force reset again. I saved my diagnostics and came here to see if anyone has any suggestions. This forum has been exceedingly helpful in the past. Thank you for your time.

alexandria-diagnostics-20241004-1238.zip

As a data point, the server crashed again while I was typing this. Uptime was about 45 minutes. Accessing the NAS directly, the login prompt for non-gui was still available, but it accepted root as login,  followed by a 60 second timeout error and then popped back to login without prompting for password.

 

My server still claims to be online via unraid connect.

 

[EDIT: I attempted to shutdown via a soft reset, got the 90-second graceful notice, but 10 minutes later the message was hung on "Shutting down" and I forced reset.]

 

Edited October 4, 20241 yr by silasfelinus

Including most recent diagnotics after the aforementioned reboot.

alexandria-diagnostics-20241004-1344.zip

After two more crashes, I've removed two cache disks that were of questionable consistency and being used exclusively for docker container processes. There were a couple hundred crc errors when I removed one of the disks, and now the system is running again but with meager uptime to say anything consistent. If this doesn't fix it, I'm not currently sure where I'll look next.

Edited October 5, 20241 yr by silasfelinus

The syslog starts over after every boot, enable the syslog server and post that after a crash.

I’m probably just feeling overly optimistic, but I haven’t had another crash since the afternoon. I’ve gone longer, but I’ve got a feeling from the general responsiveness of my containers that one of the ssd drives I removed was the culprit (I suspect it was the 1tb drive that gave CRC errors on disconnect, though I would have previously wagered the older 480gb drive was likelier to have been at fault).
 

in any case, I’m marking this as solved to save people unneeded effort, and I’ll install the syslog server and report back if my hope was unfounded.

Edited October 5, 20241 yr by silasfelinus
Grammar

Yeah, I was feeling overly optimistic. The server crashed almost immediately after I typed that last message.

 

I'm now working on a theory that I was effectively DoSing myself with a personaly project that was using an old build and updating my mariadb container with too many log updates. If so, then killing the old server will have fixed  my problem. 5 hours of uptime and counting. If not, I have the server logs configured and I will update.

Edited October 6, 20241 yr by silasfelinus
DoS, not DDoS