Server froze while away for weekend

[Big Ry]

Current os: 6.12.3

Server built: July 2023

Hardware: 12700k (no OC) , z790, 64gb ddr5 (no xmp) , 2x 2tb Samsung 980 pro (zfs cache), 3x 12tb Seagate exos x18 (xfs array)

 

Problem:

I was away at my parents over the weekend, and I came home to my server being frozen. I accessed plex from my parents without issue on Saturday. That was the only access of my server, other than perhaps some sonarr media updates. Nothing else was running all weekend. Nothing is set to auto-update either.

 

But come this afternoon, all cpu E-cores were pegged and 1 thread of all P-cores were pegged. My cache was showing 0kb used of 0kb. Unraid said mover was running, but it shouldn't have been running at this time of day. I couldn't access logs, stop dockers, stop array, or even shut down unraid. It kept saying the array was busy. I had to hard reboot. But even after doing that, it froze again a minute after starting the array. I have it started again now, and I got the logs attached. So far it hasn't frozen again.  Any idea what these warnings mean?

[Big Ry]

Tried to update to 6.12.4. Everything seemed fine until this morning. Nvme drives got hot, and this is my log this AM...

[JorgeB]

Enable the syslog server and post that after a crash together with the diagnostics.

[Big Ry]

30 minutes ago, JorgeB said:

Enable the syslog server and post that after a crash together with the diagnostics.

I enabled local syslog writing to cache about an hour ago, but it's not writing anything to it. 

[JorgeB]

Probably not correctly configured, check the instructions again, or enable the mirror to flash drive option.

[Big Ry]

39 minutes ago, JorgeB said:

Probably not correctly configured, check the instructions again, or enable the mirror to flash drive option.

How many different ways are there to configure it?  There really aren't many options.  Anyway, I added the flash drive option, and that seems to be working. 

 

Unraid seems to only be freezing now when Dockers are running.  Sometimes the entire Docker page fails to load.  I managed to get all docker containers turned off and turned auto-start off for all of them.  I'm still getting a bunch of kernel errors though.  See attached syslog and diagnostics.

 

I had updated to 6.12.4 on Monday, and I thought that fixed the freezing issues.  So last night I installed Linuxserver Radarr (dev branch). The install seemed to go perfectly fine.  I had everything up and running and was updating a dozen or so items in my movie library overnight.  Then this morning I had the warning that the nvme cache drives got hot (both are under ASRock heat spreaders on my board) and the system froze up on me as I mentioned a few posts up.

 

Ive since reverted back to 6.12.3 and removed several plugins, but neither of these did a thing to fix the problem.  I'm still getting a bunch of errors and warnings.

funraid-diagnostics-20230906-1114.zip syslog (1)

[JorgeB]

12 minutes ago, Big Ry said:

How many different ways are there to configure it?  There really aren't many options.

There's a typical mistake users do, by not setting the remote IP when doing a local log.

 

There are multiple apps segfaulting, start by running memtest.

 

 

 

[Big Ry]

2 minutes ago, JorgeB said:

There's a typical mistake users do, by not setting the remote IP when doing a local log.

 

There are multiple apps segfaulting, start by running memtest.

 

 

 

It sounded to me like the documentation was saying remote setting were for writing to remote device.  What remote IP would you use for a local write? The server IP? And if so, why even make this a setting if it isnt going to change?

 

I built this server in July of this year using all brand new equipment.  I used 32x2 G. Skill Ripjaws S5 DDR5 sticks.  I ran Memtest on the ram before setting up unraid, and everything tested fine.  I can run memtest again, but it seems highly implausible that my brand new memory would kick the bucket in just over a month.  As you know, memtest takes quite a while to run, so I'd really rather not do it again if its going to be a waste of time.  Is there something in the logs that suggests the memory is bad?

[JorgeB]

9 minutes ago, Big Ry said:

The server IP?

Yes, it's in the instructions.

 

 

[JorgeB]

11 minutes ago, Big Ry said:

I can run memtest again, but it seems highly implausible that my brand new memory would kick the bucket in just over a month. 

That's up to you, in my experience RAM can go bad at any time.

[Big Ry]

Just now, JorgeB said:

That's up to you, in my experience RAM can go bad at any time.

Anything can go bad at any time, so the same can be said about the cpu and board.  I don't think I've ever had a ram failure in my life, at least nothing that was serious enough to cause problems. Point is, I just don't want to go shooting in the dark at this, which is why I asked if something in the logs pointed to ram being the issue. There are a lot of diagnostics that can be run on a system, but many take a very long time to run. I assumed that the errors and warnings in the logs would point to a culprit.