Server Crash: CPU and RAM being Taken to 100%, WebUI Unresponsive Requiring Unsafe Server Hard Shutdown on 6.12.6

[Linguafoeda]

I moved my server to a new apartment / new ISP and router and now am having serious issues with with it  I moved it two days ago within a couple of hours last night, i noticed the web UI had frozen last showing the CPU pegged at 100% / temps in the 80s, the RAM usage skyrocketing from 48% to 97% and the entire GUI freezing. It required me to manually turn off the server unsafely (not sure what's called but a forced shutdown essentially). Then to make things worse, the mobo won't boot the USB device until I unplug the PSU and re-plug it in. this now happened again for 2nd time 24 hours later. I have no idea how to begin to troubleshoot this or what is causing this. I didn't have this issue 3 days ago when my server was sitting in another apartment, I'm extremely sad / frustrated given a lot of what i do is sitting on this now, unstable server. I typically run Plex, related *arr apps, downloading utilities, Windows 11 VM and some other random media-related things.

 

I started off not being able to connect with wireguard down, nginx not working (see below thread i'm having issues with) and now my CPU/RAM is getting taken to 100% within 24 hours of use causing a crash...does anyone know what is happening here? I'm running 6.12.6 which has been running for a at least a few weeks now (and again, all was fine until 3 days ago).

 

I got a notice last night from Fix Common Problems to (a) use the community apps 8125G realtek plugin so i installed that and (b) also fiddled around with something related to "macvlan" after being redirected to this link from FCP (https://docs.unraid.net/unraid-os/release-notes/6.12.4/#fix-for-macvlan-call-traces), where I did all the steps (I changed it to ipvlan, bonding was already on so no changes there, I turned bridging off, and I enabled "Host access to custom networks"). I don't have any logs from when the server crashd requiring a hard shut down, I can only see logs when i rebooted. Now IPVlan option is grayed out from my server setting too...i'm at loss of what to do.

Edited January 17 by Linguafoeda

[Linguafoeda]

I disabled "Host access to custom networks" and enabled "bridging" on eth0, and was able to re-select ipvlan for the docker settings. Could the fact that it was on macvlan be the reason it was crashing? I guess we'll see if 24 hours from now the server freezes up...

[itimpi]

7 minutes ago, Linguafoeda said:

Could the fact that it was on macvlan be the reason it was crashing?

It could have been.   Macvlan crashes are known to eventually crash the server on the Unraid 6.12.x releases.  We would need the persistent syslog produced by the syslog server so we could see what lead up to the crash to give a more informed view.

[Linguafoeda]

It just happened again when i woke up - less than 8 hours since last reboot. See attached for diagnostic file downloaded. I enabled the syslog server but it didn't output much. How do i make it output more / more verbosity?

 

 

Edited January 17 by Linguafoeda

[JorgeB]

Unfortunately there's nothing relevant logged, this usually points to a hardware issue, one thing you can try is to boot the server in safe mode with all docker/VMs disabled, let it run as a basic NAS for a few days, if it still crashes it's likely a hardware problem, if it doesn't start turning on the other services one by one.

[Linguafoeda]

it just crashed once again around 4am eastern. What could possibly be the hardware issue causing the crash / making the RAM go to 100%? This wasn't happening 4 days ago, nothing changed. How can i troubleshoot this more?

 

 

[hinduguy]

I am having a similar issue,

 

I narrowed it down to the shfs is process is using most of the ram and at some point crash the server, and it will require a reboot.  However the process starts all over again.

 

This started happening recently in the past month. 

 

I rebooted the server last night and now it up to 30.9 % ram usage (it was 15% after reboot)

 

I am on unraid version Version: 6.12.6

root     11504  0.0  0.0 208872  2292 ?        Ssl  Dec25   0:00 /usr/local/bin/shfs /mnt/user0 -disks 510 -o default_permissions,allow_other,noatime
root     11515  136 30.9 20886396 20338792 ?   Ssl  Dec25 1210:05 /usr/local/bin/shfs /mnt/user -disks 511 -o default_permissions,allow_other,noatime -o remember=0

 

I have attached the diagnostic file.  Any help would be appreciated. 

tower-diagnostics-20231226-0759.zip

[Linguafoeda]

Bump - really need help with solving this problem

[Vr2Io]

You have many ARRS application and Plex, could you disable those and monitor crash or not ? If still crash, you may need disable docker service, the aim were identify does problem come from docker.

 

Your system temperature in 6x degree, does this true or wrong sensors select ? ( It doesn't make sense as CPU temp in 7x degree only with 50% loading when system temp in 6x )

Edited January 6 by Vr2Io

[Linguafoeda]

23 hours ago, Vr2Io said:

You have many ARRS application and Plex, could you disable those and monitor crash or not ? If still crash, you may need disable docker service, the aim were identify does problem come from docker.

 

Your system temperature in 6x degree, does this true or wrong sensors select ? ( It doesn't make sense as CPU temp in 7x degree only with 50% loading when system temp in 6x )

 

I will try that in a few days. I may try roll back to 6.12.4 too. Should I just let my VM run and turn off Plex and ARRs. It would be odd that only I'm experiencing this (has it been posted elsewhere?) If it were related to one of those arr/Plex Dockers right?

[Linguafoeda]

Crashed again at 4:20am. This time i had PuTTY running TOP and it gave me the below last screenshot. So this might be plex related? Any idea how to further troubleshoot this?

 

 

I have re-followed the macvlan troubleshooting (https://docs.unraid.net/unraid-os/release-notes/6.12.4/#fix-for-macvlan-call-traces)

Edited January 17 by Linguafoeda

[Michael_P]

37 minutes ago, Linguafoeda said:

So this might be plex related?

 

Limit the Plex container to a reasonable amount of RAM and set the scheduled task to run at on the next closest hour to see it's fixed. If it is, you can change it to your regular time. If there's a media file that the scanner doesn't like, it'll cause the process to run away so limiting its RAM will restart the container instead of bringing down the system.

[Linguafoeda]

So i opened the Plex Media Scanner Matcher log file and it's absolutely filled with this line item at 4:19am

 

Jan 10, 2024 04:19:23.765 [23306795789120] WARN - [FFMPEG] - Unknown encoding

[Michael_P]

3 hours ago, Linguafoeda said:

it's absolutely filled with this line item at 4:19am

 

Well, you can try to figure out which file it's choking on, or just limit the RAM to the container (which is what I do)

[Linguafoeda]

I'd really love to figure out what's happening at Plex that is choking up server, instead of a stop-gap measure. i also noticed some errors with my music not showing properly in the web GUI anymore. Maybe i need to fix the database

[Michael_P]

19 hours ago, Linguafoeda said:

[FFMPEG] - Unknown encoding

 

This suggests its a video file that can't be properly read. If you've added media before it started acting up, start by removing those files and/or examining them, and adding them back until you find the culprit.

[Linguafoeda]

6 hours ago, Michael_P said:

 

This suggests its a video file that can't be properly read. If you've added media before it started acting up, start by removing those files and/or examining them, and adding them back until you find the culprit.

 

I posted in the plex forum to see how to trouble shoot this. I added a ton of different tv shows / movies so it would be very cumbersome to try test each and every one. Any idea of how to troubleshoot finding specifically the one that is throwing the error?

[Michael_P]

2 minutes ago, Linguafoeda said:

Any idea of how to troubleshoot finding specifically the one that is throwing the error

 

I'd start with large blocks of files, and narrow it down from there. Or maybe Tdarr's health check could work, too

[Linguafoeda]

On 1/11/2024 at 11:57 AM, Michael_P said:

 

I'd start with large blocks of files, and narrow it down from there. Or maybe Tdarr's health check could work, too

 

How do you limit Plex's RAM usage? It just crashed again last night. i feel i can safely say it's Plex causing unraid to crash, but now i want to troubleshoot and fix my plex install while i will be remotely away instead of having my server constantly crash. I also pinned the Plex container to only use CPU 4-7 / HT 12-15 

 

Edited January 12 by Linguafoeda

[Michael_P]

35 minutes ago, Linguafoeda said:

How do you limit Plex's RAM usage?

 

In the container's settings, toggle advanced view and add this into the extra parameters field (whatever amount of RAM you want to limit to, I just use 4G

 

--memory=4G

 

 

[Linguafoeda]

as an update - a Plex employee told me this is a known bug with the BETA plex version 1.40.7775 which i had been inadvertently on. I changed the VERSION variable of linuxserver container from "latest" to "docker" and it downgraded me to 1.32.8.7639. the problem has disappeared. Hallelujah