Occasional crashing - unreachable server

[PurpleKoolAid]

Hey folks, I'm fairly new to Unraid (only a couple weeks now) and I'm occasionally getting an error and when it happens, I can't reach my Unraid server and have to reboot it.

 

The error is:

exit status 135 from user root /use/local/emhttp/plugins/dynamix/scripts/monitor &> /dev/null

Sometimes it's exit status 1.

Here are two pics of what the display on the Unraid machine looks like when it happens (I usually don't discover it until a couple hours later or whatever).

 

 

I have no idea and my searching hasn't yielded any information either - except for an old post on a slightly different issue that was never resolved.

I ran phpdbg on that monitor script and it came back with "successful compilation" so I'm at a loss of what's going on.

 

Unraid 6.12.8 - fresh install and setup.

System specs are:

I5 6600k

Gigabyte Z170X Gaming GT motherboard

16GB G-Skill 3200mhz ddr4 (memtest passed)

GTX 980

7 disks, no cache, no parity 30.4 TB free

850w psu

 

Docker list:

Krusader

binhex overseerr

binhex prowlarr

binhex radarr x 2 (anime movies and regular movies

binhex sonarr x 2 (anime and regular shows)

linuxserver Plex

linuxserver qbittorrent

 

Anyone have any ideas or starting points? Driving me crazy. It's happened three times in the last two days. And a couple times before. Been trying to find a solution before asking but have had no luck. I recently discovered syslog so I've setup remote logging with Kiwi Syslog Manager so I'll have a log next time it happens but for now I'm just waiting.

[trurl]

Attach Diagnostics to your NEXT post in this thread.

 

Setup syslog server.

[PurpleKoolAid]

Diagnostics attached.

tower-diagnostics-20240301-1441.zip

[itimpi]

The syslog in the diagnostics is the RAM copy and only shows what happened since the reboot.   It could be worth enabling the syslog server to get a log that survives a reboot so we can see what happened prior to the reboot. The mirror to flash option is the easiest to set up, but if you are worried about excessive wear on the flash drive you can put your server’s address into the Remote Server field.

[PurpleKoolAid]

I just found out about the syslog stuff earlier. I've set it up now (remote server option) but of course now that I have that going and I'm waiting for it to crash, it hasn't crashed lol.

[PurpleKoolAid]

Log is attached - looks like I have a dying disk 😭

 

The error on the screen was an I/O error, not the one from above so looks like this is a brand new issue I have to sort out asap. Yay....

SyslogCatchAll-2024-03-01.txt

[trurl]

2 hours ago, PurpleKoolAid said:

looks like I have a dying disk 

No, looks like your boot flash drive has disconnected.

[PurpleKoolAid]

Yeah I missed that 'USB disconnect' line and jumped to panic mode. Got that all sorted out now - still new and learning all this but I did buy some backup flash drives and do regular backups so all good on that front.

 

Now just waiting for the original issue to resurface again.

[PurpleKoolAid]

Okay so the error happened again and it appears to be network related. Plex. Sonarr, qbittorrent, etc were available for a while afterwards but it's all unreachable now. I took a screen grab of the log because for some reason it stopped writing to the file after the highlighted line. I found something about that 'fix common problems' plugin so I gave it a whirl but it didn't do anything other than advise of a few updates.

 

I also attached the logs from yesterday after I had rebooted after resolving that flash drive issue - towards the end it seems that something similar happened overnight.

 

The weird thing is I have no logs from the end of that file to this screenshot. This screenshot is all I have from today - I wonder if because I'm losing network I'm losing the logging as well? Makes sense I think. I've switched to local logging on a share but I'm gussing mirror to flash might be the best way to capture the issue?

 

Edited to attach a photo of what the monitor on the Unraid PC is showing before I reboot it.

 

 

SyslogCatchAll-2024-03-01.txt

Edited March 2 by PurpleKoolAid
details

[FreakyBigFoot]

I'm having something similar happen. My unraid keeps going unresponsive every couple days since I've updated to 6.12.8. I have my logs forwarded to a log server and I didn't get any logs. I've just ran a memtest86 all the way through and I'm getting 0 hardware issues. If you look around you'll see there's quite a few people having these random crash issues since 6.12.8

[trurl]

Found something recent related to your NIC, don't know if it will help or not.

 

[MassimoMx]

I have been havig a similiar issue on both 6.12.8 and 6.12.6. I even made a post for it, but for now. No luck at all https://forums.unraid.net/topic/156267-6126-random-crashesunresponsiveness-overnight/

[PurpleKoolAid]

17 hours ago, trurl said:

Found something recent related to your NIC, don't know if it will help or not.

 

My system just went down again - logs don't show anything for the past few hours and I'm not getting that same fatal ethernet error as that NIC post. Really scratching my head on this one and it's driving me insane.

 

I tried my second ethernet port but it's not even being recognized.

[PurpleKoolAid]

Just an update on this - I had read a couple days ago (can't find the post now unfortunately) but it had mentioned crashing regularly so they removed and reinstalled the Unassigned Devices plugin. I have removed the plugin (and not reinstalled it) and it may be too soon to celebrate - but I'm over 24 hours of uninterrupted uptime now.