Hi All!
Hoping some of your expertise can shed some light on some issues I've been seeing with my Unraid server. I've had this issue for some time, trying to diagnose it each time, at a guess at least 6 months.
When I am experiencing the issue the symptoms are:
Web UI is unreachable - Firefox says an error occurred
SSH returns connection refused
Any host docker containers are unreachable - AdGuard Home is the only one I have and so DNS and DHCP also dies with it
Other docker containers for the most part still seem to be running fine though; their respective web UIs load fine, although error if they need to communicate with another container, and in the case of Jellyfin I can play media from my array.
Docker internal networking stops working, so inter-container communication breaks
With a monitor plugged in, I cannot log in. Typing is "root" in the username just spits out the server IP then asks for the username again
Samba also stops working
The syslog does not show anything that looks useful, other than Samba errors, which appears to be a known issue with the version in Unraid. Although, this does not appear to be a trigger. I have attached the syslog covering a period of when this last happened, earlier today, along with the diagnostics from Unraid.
Following some advice on Discord I have today run memtest to test the RAM, initially using the version bundled with Unraid I allowed it to complete 3 passes and had no issue. I then installed the latest version of memtest to a spare flash drive and run the default settings which resulted in 4 passes as well.
Since doing that, I have:
Updated the MB bios to the latest
Reseated my RAM
Disabled VT-x and VT-d in the BIOS - I saw mentions of this on the forum and since I don't need them for Docker no harm in it
Uninstalled the Intel GPU Top plugin, removed the devices from Jellyfin, and added `options i915 enable_dc=0` to `/boot/config/modprobe.d/i915.conf` I saw this was another option for those on 12th gen intel, I don't really transcode, so I'm happy to try this out.
It's now a waiting game to see if anything has improved, since it can be hours, days, weeks, or the longest I've had is just over a month before it'll trigger. But I'm just wondering if anyone has ideas or has maybe experienced this before.
The logs and graphs I've included from Midnight on the 5th to about 1pm on the 5th - the issue happened somewhere in here, syslog has nothing from 7/8am until I force restarted the server at around 12pm after noticing my DNS/DHCP has gone offline, which seems odd to me as well.
Finally, in a last ditch effort to try and get some data when this issue happens I took advantage of Docker containers that are fairly self-contained not being impacted, so I installed Netdata when I last had this issue a few days ago. This collected data even during when I was experiencing the symptoms above, I don't know if I should look for anything specific, however, some memory stats did jump out as potentially curious, most notably:
When I noticed the issue, around ~12pm (GMT) available RAM increased by about 1 - 1.5GB - odd but makes sense as services seemly stopped
Committed memory took a nose dive by about half
Memory page faults were increased, although I don't think that is a big issue from my limited understanding
Transparent Huge Pages Collapsed shot up almost 10-fold - again, not sure if that is an issue or not though
I've attached the graphs for those Netdata stats, if they help with anything, although it could just be trying to find something and focusing too much on RAM!
Many thanks in advance for anyone that made it this far 😅
gandalf-diagnostics-20240305-2003.zip
Syslog.txt