Random crashing after long term stability

[nomisco]

Looking for some pointers as to what may be causing my server to crash. At the moment, from memory, it will run for between a day and 7 to 10 days before crashing.

 

It's a headless machine, so as it's sat behind my TV, I've connected it to the HDMI port and added a keyboard to see if anything happens onscreen when it crashes. It just sits at the login prompt (untouched since boot), but that is also unresponsive to the keyboard.

 

There are plugins on the machine which may not have been on there when it was stable, but the nature of the crashing means it difficult to rule some out by removing them as I don't know which ones it could be - if any.

 

I'm a total Linux noob, so I set the log to be mirrored to the flash drive. Looking at the syslog file in the logs directory does not show any event when it crashes (I've taken the flash drive to my Windows PC to look at the file). Attached are the system diagnostics, plus the syslog file just after it crashed which I may have inadvertently deleted from the flash before rebooting the server.

 

Just to add that I've switched between the stable and RC release with the same outcome.

 

unraid-diagnostics-20211222-0932.zip syslog

Edited December 22, 2021 by nomisco

[JorgeB]

Could be this issue:

 

https://forums.unraid.net/bug-reports/prereleases/69x-610x-intel-i915-module-causing-system-hangs-with-no-report-in-syslog-r1674/?do=getNewComment&d=2&id=1674

[nomisco]

Thank you Jorge. I only use that driver for Plex transcoding but can do without. I remember adding some lines to a config file or two; but can't remember how I'd remove them.

 

I suppose a reboot of the server would be required after they're removed?

 

Update:  I removed some lines from the 'Go' file:

 

#enable module for iGPU and perms for the render device
modprobe i915
chown -R nobody:users /dev/dri
chmod -R 777 /dev/dri

 

And also the relevant lines from the Plex container.

 

I'll give this a try, and report back either way, but it may be a week! Any other suggestions in the meantime would be welcome.

Edited December 22, 2021 by nomisco
Update info

[nomisco]

Unfortunately the server crashed again in the night, just after 3AM. I've attached the log from the flash which shows just over an hour and twenty minutes of errors. The midnight spin up will be for the Plex library scan, then the disks go to sleep until an hour and ten minutes later when I get this massive block of errors until the box dies. I can't see what is causing the error, but it may be more obvious to someone here. Hopefully.

 

Or perhaps someone can suggest a more verbose logging effort if they have a suspect in mind.

 

Thanks

 

 

 

syslog.txt

[JorgeB]

Lots of nginx errors, can't really help with those, other than make sure only one browser window is open on the GUI, close any stale ones.

[nomisco]

8 minutes ago, JorgeB said:

Lots of nginx errors, can't really help with those, other than make sure only one browser window is open on the GUI, close any stale ones.

So is it possible for me to look in the running processes to see an abundance of these?

[Corruptsector]

2 Days ago a mate of mines Unraid Server became unresponsive, then this morning mine did. Same thing happened a couple of weeks ago, His server died then the next day mine did. Very odd.

 

His is running a AMD CPU and mine an intel. Both on 6.10.0-rc2

 

If it happens again what should i grab? Syslog at the moment shows nothing interesting.

 

[nomisco]

I've uninstalled the 'My Servers' plugin and so far I have a completely uneventful log, with just disk spin ups / downs.

[nomisco]

Thought it might be worth an update:

 

Now over two weeks of uptime without anything unusual in the logs. I re-enabled the i915 driver about a week ago and it's being used for Plex HW transcoding - the same as the last couple of years.

 

I may reinstall the MyServers Plugin to see what happens. I am still running 6.10 RC2.