David Grenon

TLDR; Setting Credit detection to online only (no local detection) and make sure nothing is in temp folder assigned to Plex container (related to plex) solved the memory usage issue.

Ok so problem solved here and a lot of people having issues with 6.12.X might have the same root cause.
Writing this down here so it could help other people in the same boat.

Downgrading my server to 6.11.5 was the right initial thing to do to actually get some logs and pinpoint the actual issues.
UNRAID OS VERSION WAS NOT THE CAUSE. But it was way more difficult to troubleshoot with 6.12.x*

By putting Plex TMP folder to /shm instead of /tmp prevent from it to take all the memory and make the system misbehave.
 
I was then able to see with the container and server logs to see whats going on without being in the front of the server when the issue happen.

A thing that was kinda persistent is that the issue was almost all the time happening around 5h30 - 7AM in the morning.
Digging down process and ram usage I found that:
HTOP would show that couple Plex transcoder process using a lot of CPU Temp folder (/SHM) would actually fill up with couple creditDetection-(UID), etc Digging more, there is a thing called Credit detection (skip outro) that is not the same as the intro detection:
https://support.plex.tv/articles/credits-detection/
 
By default this thing is set to : both, try online first
Which mean if the Outro metadata doesn't exist on Plex DB let your server transcode the files to search for pattern or w/e magic this does.
This option by default start when media is added AND at maintenance hours: By default at 2:00 AM
End of maintenance tasks are also by default at 5AM and maybe something at 5AM was triggering end of task but wouldn't work.
Anyway, putting this option to : online only (no local detection) completely solved my issue.

This issue happened out of nowhere. Why ? because at some point, a TV show was added to my server and that one had probably no entry in the Plex db and made the task goes.

Its unfortunate that this kind of features just break the whole system.
Moral of the story, never map a container path to Unraid /tmp. Rather use /Shm.