Hi All!

 

Hoping some of your expertise can shed some light on some issues I've been seeing with my Unraid server. I've had this issue for some time, trying to diagnose it each time, at a guess at least 6 months.

 

When I am experiencing the issue the symptoms are:

• Web UI is unreachable - Firefox says an error occurred
• SSH returns connection refused
• Any host docker containers are unreachable - AdGuard Home is the only one I have and so DNS and DHCP also dies with it
• Other docker containers for the most part still seem to be running fine though; their respective web UIs load fine, although error if they need to communicate with another container, and in the case of Jellyfin I can play media from my array.
• Docker internal networking stops working, so inter-container communication breaks
• With a monitor plugged in, I cannot log in. Typing is "root" in the username just spits out the server IP then asks for the username again
• Samba also stops working

 

The syslog does not show anything that looks useful, other than Samba errors, which appears to be a known issue with the version in Unraid. Although, this does not appear to be a trigger. I have attached the syslog covering a period of when this last happened, earlier today, along with the diagnostics from Unraid.

 

Following some advice on Discord I have today run memtest to test the RAM, initially using the version bundled with Unraid I allowed it to complete 3 passes and had no issue. I then installed the latest version of memtest to a spare flash drive and run the default settings which resulted in 4 passes as well.

 

Since doing that, I have:

• Updated the MB bios to the latest
• Reseated my RAM
• Disabled VT-x and VT-d in the BIOS - I saw mentions of this on the forum and since I don't need them for Docker no harm in it
• Uninstalled the Intel GPU Top plugin, removed the devices from Jellyfin, and added `options i915 enable_dc=0` to `/boot/config/modprobe.d/i915.conf` I saw this was another option for those on 12th gen intel, I don't really transcode, so I'm happy to try this out.

 

It's now a waiting game to see if anything has improved, since it can be hours, days, weeks, or the longest I've had is just over a month before it'll trigger. But I'm just wondering if anyone has ideas or has maybe experienced this before.

 

The logs and graphs I've included from Midnight on the 5th to about 1pm on the 5th - the issue happened somewhere in here, syslog has nothing from 7/8am until I force restarted the server at around 12pm after noticing my DNS/DHCP has gone offline, which seems odd to me as well.

 

Finally, in a last ditch effort to try and get some data when this issue happens I took advantage of Docker containers that are fairly self-contained not being impacted, so I installed Netdata when I last had this issue a few days ago. This collected data even during when I was experiencing the symptoms above, I don't know if I should look for anything specific, however, some memory stats did jump out as potentially curious, most notably:

• When I noticed the issue, around ~12pm (GMT) available RAM increased by about 1 - 1.5GB - odd but makes sense as services seemly stopped
• Committed memory took a nose dive by about half
• Memory page faults were increased, although I don't think that is a big issue from my limited understanding
• Transparent Huge Pages Collapsed shot up almost 10-fold - again, not sure if that is an issue or not though

 

I've attached the graphs for those Netdata stats, if they help with anything, although it could just be trying to find something and focusing too much on RAM!

 

Many thanks in advance for anyone that made it this far 😅

gandalf-diagnostics-20240305-2003.zip Syslog.txt

12 hours ago, terbe said:

since it can be hours, days, weeks, or the longest I've had is just over a month

This and nothing relevant being logged make this very difficult to diagnose, one thing you could try is to boot the server in safe mode with all docker containers/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one, but this is not great if the server can run a month before it crashes.

Yeah it has been tricky so far.

 

So far so good following the steps I mentioned before, although that doesn't nessarily mean much. I've also disabled AdGuard for now as the instability means my whole DNS/DHCP functionality goes with it.

 

My gut is telling me it's a hardware issue as I only started encountering this after upgrading from a i5 3330 to the i5 12400 I have now, plus all the other parts required for that. My gut feeling is the memory too as that is old ish memory I used to have in a gaming PC, I have a vague memory that I used to experience intermittent BSODs as well relating to memory, but I could be imagining it. Given it was a gaming rig too it would have had XMP enabled likely be warmer etc. All things that may degrade its life. Just not being able to see what is causing it is frustrating. If it happens again I may just buy some replacement RAM and try it out. I only have 16GB and they're relatively cheap at the moment, at least I would be able to maybe rule out the DIMMs themselves, or fix the issue.

 

Was hoping someone would identify a tiny nuance I have missed 😅 that would give a clear reason, I suspect that is not going to be the case though.

Will try safe mode, like you say though the intermittentness of it makes that tricky too since I could run it in safe mode for 2 weeks just fine but if I had left it a little longer it may or may not have triggered the issue.

 

The lack of syslog entries between 7am and when I rebooted seems odd to me as well, I would have expected at least something. That behaviour I see with the syslog server that I have configured now, but also if I used mirror to flash, it seems to just stop logging to the syslog.

I'm writing this far away from my server so it cannot hear me! I think I have gotten to the bottom of this now.

 

After all the above things I tried the server was up for almost 4 days before it happened again while I was using it. This time it was slightly different though:

 

* the unraid web ui would render the html but no styling or functionality

 

Everything else was the same except I was able to login via the terminal and a monitor this time. A lot of commands would return an io error. However, the syslog have me some output! Swuashfs block read errors.

 

Some research on this forum pointed me yo a possible dodgy flash drive, which is possible it is quite old. Plugged it into my windows PC and it did detect errors.

 

So I moved it over to a new flash drive and so far seems okay, been up for over a week now.

 

Thanks for the help and suggestions!