Hi all.
I've been dealing with a sneaky issue for a few months now. Been trying to get support for it via the Unraid discord but given the nature of troubleshooting, I've had to catch people up every time I post there with what I've already done.
My server periodically becomes unresponsive. It is not reachable by webui, none of the docker container apps are reachable, it is not reachable by ssh, or by navigating the network in general. It is ping-able and tracert-able. Previously this issue had weeks between instances, then it happened again last week, and now twice today (albeit these ones fixed themselves automatically, past cases I had to hard reboot the server). It is running 16gb of ddr4 kingston ram and running on an intel I219-V nic.
Here is a pastebin of the syslog where it happened earlier today. The relevant timestamps are 18:11 and 19:30.
https://pastebin.com/KZVqRDLM
I have also been capturing the output of running `free -h` every minute as a diagnostic tool. Here's a pastebin of the timestamped file contents for both time periods today:
18:11 - https://pastebin.com/MVyxVcKf
19:30 - https://pastebin.com/ivQ6Ssaz
Some items to note:
- I'm unsure the source of this syn flooding on port 8080 from the syslog, as this port is not open in my router, and nothing is running on port 8080 at all except for in a single docker container (but that is only internally to that container, it is mapped to a custom port in the 30,000's on unraid itself). I have had my only VM disabled for a couple weeks now to rule that out, so that's not a source either.
- I have previously hooked up a kb and monitor to attempt to log in during one of the instances of unresponsiveness. The server would not let me log in, as anytime I input the root username, it would timeout after 60s.
- The system is fully hung during the time period when it's unresponsive, as the df/free commands are also not running during that time period (as seen in the above pastebin, there is a timeskip). This combined with the above point leads me to believe it's not an issue with the NIC, but something else hardware related causing the entire system to hang.
- I have run a memtest for 12 hours overnight with zero errors reported. I am concerned about running for longer due to the potential for soft errors appearing as if actual errors, but if that's the recommendation, I will do it.
- I would like to leave a docker container process of elimination for last. I have several docker containers that I need for uni, and disabling them for what might potentially be weeks, will be extremely disruptive.