Hi all.

 

I've been dealing with a sneaky issue for a few months now. Been trying to get support for it via the Unraid discord but given the nature of troubleshooting, I've had to catch people up every time I post there with what I've already done.

 

My server periodically becomes unresponsive. It is not reachable by webui, none of the docker container apps are reachable, it is not reachable by ssh, or by navigating the network in general. It is ping-able and tracert-able.  Previously this issue had weeks between instances, then it happened again last week, and now twice today (albeit these ones fixed themselves automatically, past cases I had to hard reboot the server). It is running 16gb of ddr4 kingston ram and running on an intel I219-V nic.

 

Here is a pastebin of the syslog where it happened earlier today. The relevant timestamps are 18:11 and 19:30.

https://pastebin.com/KZVqRDLM

 

I have also been capturing the output of running `free -h` every minute as a diagnostic tool. Here's a pastebin of the timestamped file contents for both time periods today:

18:11 - https://pastebin.com/MVyxVcKf

19:30 - https://pastebin.com/ivQ6Ssaz

 

Some items to note:

 

- I'm unsure the source of this syn flooding on port 8080 from the syslog, as this port is not open in my router, and nothing is running on port 8080 at all except for in a single docker container (but that is only internally to that container, it is mapped to a custom port in the 30,000's on unraid itself). I have had my only VM disabled for a couple weeks now to rule that out, so that's not a source either.

 

- I have previously hooked up a kb and monitor to attempt to log in during one of the instances of unresponsiveness. The server would not let me log in, as anytime I input the root username, it would timeout after 60s.

 

- The system is fully hung during the time period when it's unresponsive, as the df/free commands are also not running during that time period (as seen in the above pastebin, there is a timeskip). This combined with the above point leads me to believe it's not an issue with the NIC, but something else hardware related causing the entire system to hang.

 

- I have run a memtest for 12 hours overnight with zero errors reported. I am concerned about running for longer due to the potential for soft errors appearing as if actual errors, but if that's the recommendation, I will do it.

 

- I would like to leave a docker container process of elimination for last. I have several docker containers that I need for uni, and disabling them for what might potentially be weeks, will be extremely disruptive.

Edited May 21, 20242 yr by Mareau
Forgot to add pastebin of syslog

Enable the syslog server and post that after a crash, together with fresh diagnostics.

Hi. Thanks for the reply.

 

I must have forgotten to paste it in. I meant to post the relevant syslog for earlier today. Here it is: https://pastebin.com/KZVqRDLM

 

Regarding diagnostics, is there a way to gather them when the server is unresponsive? Even connecting a kb/monitor doesn't let me log in as it times out, due to the entire system being hung.

It just happened again. This time the syslog doesn't have any php-fpm warnings for the time period. Relevant timestamp is 21:06. https://pastebin.com/uHXHE0Ne

The diags you can post after the crash, it's mostly to see the hardware, and post the complete persistent syslog please, you can attach any files directly to the forum, no need to use external sites.

 

 

Roger that. Here's a current diagnostic and syslog. Nothing has changed on the server since the last instance of unresponsiveness 15 minutes ago.

atlas-diagnostics-20240521-2133.zip syslog-127.0.0.1.log

If I understand correctly these last two times the server recovered by itself? And the issue coincided with the 

php-fpm[10341]: [WARNING] [pool www] child 6903 exited on signal 9 (SIGKILL) after 13.940049 seconds from start

errors?

18:11 and 19:30 both had those entries. However another instance happened around 21:06 which didn't have that entry. Also, it just happened again, so I'm attaching a new diagnostics and syslog. This last one did have one of those php-fpm warnings this time. The times it's been happening today have been self-correcting, although before that it seemed to require a hard reboot. The issues before today also seemed to happen weeks apart. Today is the first time I'm noticing that it happened regularly. That may be because I'm actually constantly using my server today so I'm noticing the outage more than previously.

atlas-diagnostics-20240521-2152.zip syslog-127.0.0.1.log

Edited May 21, 20242 yr by Mareau

Those php errors can sometimes be caused by a plugin, since there's nothing else relevant logged you can try booting in safe mode with all docker containers/VMs disabled, let it run as a basic NAS, if it doesn't hang like that start turning on the other services one by one. 

How long should I give it in safe mode before coming to the conclusion that it's a plugin/docker container?

Long enough to confirm it no longer happens, I would say take the average take it takes and double it.

One last question: what would be the best way to determine when it goes unresponsive. I've been noticing it a lot today because I've been using some docker containers all day. The server also seems to remain pingable during the issue. Do you have any tips for noticing when it goes out?

In this case you would need to be using the server, since it can recover and not leave anything logged.