First of all - I think this might be my very first cry for help since I got Unraid, so be gentle to me

 

Let's start at when this was still working great: 6.12.4. I had it running for about 100 days without issues. Then randomly the parity drive failed on me. Turned unraid off, checked and re-seated the cables for the affected drive and powered back up. Then ran a memory check as well as SMART self-test. All came back clear so I had unraid simply rebuild the drive without issues. Gave it a few days and noticed that PLEX was shut down by Unraid for filling up my 32 gigs of ram according to the logs. It's never done that in the years of running unraid so I started limiting PLEX Memory allocation to 16GB but all it did was continuously giving me warnings that memory was full and unraid shut the process down. I removed the memory limitation again and the alarms in the logs stopped. I wrote it off as a fluke for the time being but figured I should mention it in there just in case it's important.

 

Since everything seemed fine at the moment I decided to finally pull the trigger and upgrade to 6.12.6. I ran it for about 5 days or so with nightly lockups. Got tired of it and decided to downgrade to 6.12.4 again but the nightly lockups continued. Stumped that it didn't fix it I started a barrage of "fixes" but none of them fruitful

To help me troubleshoot I did start to copy the syslog to flash but it doesn't record anything useful. it simply stops recording when unraid hangs. Funny thing is Unraid will continue to ping BUT nothing works - can't access GUI, can't SSH, can't access shared drives, nor dockers. I tried hooking up a monitor but also get no output there.

 

Now here are the things I've tried or have currently set:

-Upgrade BIOS to latest and greatest

-Disable all C-states (it's a Ryzen 5950x build on a 570x chipset) as well as AMD Cool 'n quiet to be on the safe side

- Power supply state is set to normal during idle

-Switched MACVLAN to IPVLAN as well as removed bonding (that was how I got to 100 days running before upgrading).

-Copied the contents of the flash drive from my previous version on 6.12.4 over in case the built-in downgrade doesn't work so well

-Ran memory test with multiple successful passes

 

It always seems to lock up between 2am and 7am, regardless of when I start Unraid. That does seem like some scheduled task is causing all this but all I can think of is either the mover (3am) or PLEX tasks (2am through 5am). I'm going to shut down PLEX tonight before I get to bed to test it out but I was hoping I can have someone look over my request in the meantime to collect some additional ideas or maybe something I completely overlooked. Thanks in advance!

tower-diagnostics-20240104-0806.zip syslog-01-2024

Unfortunately there's nothing relevant logged, this usually points to a hardware issue, downgrading back not fixing the issue also points to hardware, one thing you can try is to boot the server in safe mode with all docker/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one.

My server finally survived a night when disabling PLEX before it's maintenance starts. Now I have to dig into Plex and see what causes my server to crash during maintenance since it runs fine throughout the day with multiple 4k transcodes (I don't use HW transcode at this time). I'll update / report back once I have a solution in case someone else has the same problem.
I did read through the support forum for the linuxserver.io group and there are a few reporting plex crashes as well as extended power consumption beyond the maintenance window since the latest updates- my hunch tells me that it might use all my resources and crashes unraid or maybe it analyzes a file that is somewhat corrupt and locks everything up trying to access it? Not sure but I'll dig into it further.

Reporting back with final fix/solution

 

During the Unraid upgrade to 6.12.6 something happened to the Plex docker and it updated itself to the BETA release branch instead of the STABLE release branch. I have Plex Pass so it makes sense that I technically have access to it but I never opted in, still my docker ended up going there and that's what has seemed to cause all my issues. When I tried to manually set a version by specifying a version through repository I ended up getting an orphan image and PLEX was completely gone
Well I downloaded Plex docker again and noticed repository went from linuxservers to lscr.io but I have no idea if that just was because it was an old docker pull previously. The new docker now pulls the latest stable release automatically instead of BETA. Anyhow, I have survived 2 full nights sine the rebuild and my CPU has been hurting for a full 24h to re-analyze all the media files (55TB takes a while), so I'm sure that CPU/RAM clocking is good or would have crashed during those analyzing times.

Long story short - Issue was a bad Plex docker taking the server down. Check your Plex console and logs to see what it's doing and what version you are running - if you ended up on the 1.40 branch you on the BETA and could explain random lockups.

Thanks for this. Will be trying this theory tonight as I've been going through all of the same troubleshooting. Only difference is I'm using the hotio plex container. I DID verify that I'm on the Plex 1.40 branch however.