Unraid server "freezes" at the same time every day...

[toolmanz]

All,

This is bizarre and not really sure how to approach this....

 

Every evening at exactly 11:10 pm the entire server "freezes". I can't get into the server, internet is down - there is no way to communicate with the server. I just dies - I have never seen that before.

 

There were a few updates a few days ago and that is the only thing that has changed.

 

The only solution is to reboot the server but I have to use the power switch to get it to shutdown ..... last night it triggered a parity check (previous 2 nights it did not).

 

This has been going on for 3 straight nights with no end in site.

 

Equipment:

Dell R710 2U rackmount server

Onboard 4 port NIC

23 HT’s and 46G memory

Unraid server with various dockers such as Nextcloud, NZBGet, Sonarr and Bitwarden

Server is stable and functioning as expected.

PF Sense running on a VM using 2 of the 4 internal NIC's

It has been really stable for the past 6 to 8 weeks. No sign of trouble.

 

I did look at all the logs for the dockers, all cron jobs and found nothing except:

 

Duckdns Docker Log at approx. the same time as the "outage"

Something went wrong, please check your settings Sun May 17 23:10:05 EDT 2020
[s6-init] making user provided files available at /var/run/s6/etc...exited 0.
[s6-init] ensuring user provided files have correct perms...exited 0.
[fix-attrs.d] applying ownership & permissions fixes...
[fix-attrs.d] done.
[cont-init.d] executing container initialization scripts...
[cont-init.d] 01-envfile: executing...
[cont-init.d] 01-envfile: exited 0.
[cont-init.d] 10-adduser: executing...
usermod: no changes

 

Only thing I have found remotely close to the same time.

 

I am out of ideas as to what is happening and am looking for some direction on where to look next.

 

I will peruse the jobs that Sonarr and Radarr launch but no expecting much.

 

Any and all suggestions welcome.

 

Thanks in advance. Cheers.

 

 

 

[Squid]

Set up the syslog server (https://forums.unraid.net/topic/46802-faq-for-unraid-v6/page/2/?tab=comments#comment-781601) to mirror to the flash.  Then wait and afterwards post the resulting syslog, along with a set of diagnostics

 

[toolmanz]

Ok thanks muchly. I have set up a syslog server on my windows PC and await 11:10 pm.

 

I figure that is the best way to receive and process the entries.

 

Good catch - I had no idea this was possible.

 

Thanks

 

Cheers

Edited May 18, 2020 by toolmanz

[toolmanz]

Well I have solved the issue - didn't happen last night.

I only changed a couple of things on the server and my network infrastructure - not sure what solved it.

This is what I did before the appointed hour:

1. Updated some backend software on the Unraid Server

   

 

2. I stopped my Netdata docker because of how it was running it was highly suspect.

 

3. I removed an old router from my infrastructure that has caused problems in the past: dropping internet unexpectedly; certain ports messed up access; the reason I moved to PF Sense as speeds were extremely slow over fibre internet (getting 50 mbps when I should be getting at least 500mbps); when used as main router it would glitch out regularly.

 

I think it is option 3 but not prepared to test it .....

 

I will however offer it up to the God of Routers and burn it on the alter of internet speeds mwahahaha.

 

 

Sorry I digress.

 

Lesson learned: Don't just expect that it is a software problem .... it might be hardware!

 

Cheers