Randomly started locking up randomly requiring a hard reset about a week ago. After I updated my binhex-qbit docker to 5.0, not sure how/why that would be related but that was the only change I made around that time. I've been trying different things over the last few weeks.

 

i've done memtest twice. First time did two passes no errors. Second time was ran a few days after the first, did six passes and no errors on that one either.

 

Once it was stable without docker/vms running without plugins enabled I brought it up with plugins enabled, still in NAS mode. Stable for 48ish hours.

 

Enabled docker. Turned off everything but "thelounge" and "PlexMediaServer (The Offficial one)". Was stable for another 48 hours.

 

Moved into running with plugins and only those two dockers enabled, was stable for 48 hours.

 

Turned all the docker containers back on today and crashed within an hour.

 

Brought it back up, and disabled the qbit docker to see if maybe that was the cause... but still crashed within an hour.

 

Throughout all of this, could not find anything relevant in any of the syslogs (had a monitor hooked up to the tower so I could see what was in the backscroll when it crashed).

 

Curious what input I could get from y'all, as this is driving me out of mind trying to narrow down what the cause is.

 

tower-diagnostics-20241011-1453.zip syslog-previous syslog.txt

Edited October 11, 20241 yr by Ewok

12 hours ago, Ewok said:

Turned all the docker containers back on today and crashed within an hour.

Try enabling half of the containers, if the same try the other half.

Testing so far.... ran the following by together for 5+ hours and no crashes:

Prowlarr

Jackett

Sonarr Develop

Sonarr Latest

Sonarr Latest

Radarr Latest

Radar Latest

NPM

TheLounge.

 

Going to move into testing the other half which includes:

Binhex-QBit 5.0

PlexMediaServer Official

Portainer (Immich + Filegator)

 

Things that have been disabled since crash started:

5x Ark Survival Ascended Containers

Binhex Syncthing

Calibre

Calibre-web

 

I think if the second group begins crashing I'm going test running only Plex by itself and forcing hardware transcoding. I have a hunch this is what's causing the issues, but it will never crash at the same time. I forced a transcode yesterday and it ran the movie for about an hour before crashing. But this was also with other containers running.

 

If it does end up only crashing with plex running + doing transcodes in RAM.... Could this be indicative of RAM being an issue? And if so, how would I narrow down which stick?

 

I will say that I did reseat the CPU, RAM Sticks, and repasted the CPU/Cooler last night with zero change.

1 hour ago, Ewok said:

If it does end up only crashing with plex running + doing transcodes in RAM.... Could this be indicative of RAM being an issue? And if so, how would I narrow down which stick?

Yes it could be an indicator.  Test that by removing a stick (or more) and pushing it again.  Using up all of the memory during a transcode should not cause a crash if memory is good; it would just slow down the video playback.  If it crashes again on one stick of memory, swap it out for a different stick and use the same slot.  If both runs crash, try again in a different slot.  If you go through 4 tests, 2 sticks on 2 different slots and still crashing, then the problem is likely elsewhere.

Edited October 12, 20241 yr by Veah

Could something be causing you to run out of memory? Most of the times Unraid has locked up on me was when something was causing the server to completely run of of memory. Misconfigured rclone did this to me once, I didn't set a cache location so it cached to RAM.

I've been running netdata in the background. Looks like anytime the server gets into the 48gb+ usage of ram is when it locks up. So going to start putting sticks in one at a time.

19 hours ago, Ewok said:

I've been running netdata in the background. Looks like anytime the server gets into the 48gb+ usage of ram is when it locks up. So going to start putting sticks in one at a time.

How much memory do you have in total? You may be running out of memory, which will cause a lock up.

64GB.

So can't narrow it down to a single stick. Acquired some new RAM sticks to test with just in case and same issue.

 

Is there a way to narrow down what is causing the system to run out of memory?

Kept testing different setups. Deleted the docker.img and restored my containers from backup.

 

Every time I HW transcode with plex I could get it to crash.

 

I switched from transcoding to RAM to trascoding to my secondary cache drive.

 

Continues to crash when transcoding, can be anywhere from almost instantly, to a few minutes/hours in.

 

------

Step 2....

Stopped all containers and used this: https://hub.docker.com/r/progrium/stress/

Ran with various stages, stepping up from 1 CPU to maxing out all. It will reliably fail at the conclusion of EVERY test now.

 

---

I recognize this test thing is running in docker... but that shouldn't have any effect right?

 

Does this point to CPU being bad/going bad?

Swapped with a known good CPU and still crashing.

 

Everything i've done so far:

RAM: Swapped RAM with known good.

CPU: Swapped CPU with known good.

Flash Drive: Replaced with new drive, did a fresh install of UNRAID.

 

The only things left are mother board and PSU. Both of which are expensive to swap out.

If we presume memory and cpu are good and also MB and PSU cause can't test other known good, that leaves settings (in bios) or overheating.  Another stretch with all the memory tests... Maybe one of the memory slots.  Could test that by 2 tests one on ch.A and another on ch.B.

I'm at a loss.

 

It crashed once after I swapped to the known good CPU using the stress docker + a 1080p plex transcode.

 

Known good CPU in, stress docker running + three 4K transcodes being done to an nvme disk. Swapped to running all the above but transcoding to /dev/shm/... still no issues.

 

This is frustrating. I'm going to stress test for a few hours and see what happens. =/

 

Edit: Just because you mentioned temps...

Full Load:

Motherboard - 29 - 33C

CPU - 60 - 68C

 

Idle temps:

Motherboard - 25-30C

CPU - 32-40c

Edited October 25, 20241 yr by Ewok
Adding temps