xcsascii

It has been 10 days and I have had 8 days of uptime. Here is what I have learned and what I am doing now. If someone is having the same issue as I am this may not be the solution you are looking for.
I do not have any hardware errors/failures. The root issue on my system was the IOWAIT generated by running the Plex docker container. Based on usage this is due to library scanning for intros and credits. I can run all of my other containers without crashing, even heavier ones (including a firefox container, or delugevpn container) I still get quite a bit of IOWAIT. Specifically when running large jobs in deluge or any of the *arrs A co-worker of mine running 6.12.10 has the same processor (AMD Ryzen 5700G) he also noticed that his IOWAITs have gotten much higher. I noticed that regardless of IOWAIT, I am not seeing any real disk usage on the array. Maybe 10-20mb/s on one or two disks. the CPU on the homepage GUI shows that the CPU is running high (80%+) but htop and glances contradicts that information. Showing a single CPU core running at 100% while the remaining are much much lower.
 
As a temporary work around I moved the Plex workload over to a VM running on unraid, and it worked without issue. I then moved the workload over to Proxmox and copied the existing cache and metadata. I have since been running Plex in Proxmox pointed to unraid via NFS.
 
Based on this I think it is safe to assume that the file manipulation on media files themselves isn't the root issue. It must be the appdata share. However the disk usage is also pretty minimal. My appdata is hosted on a zfs pool with 10k sas drives. HOWEVER the mount is still /mnt/user/appdata. The appdata folder is then set to the cache as the primary folder.
 
Based on this information, I have begun the process of migrating all of the containers off unraid, and onto proxmox. I will probably use unraid as pure storage only from now on. Which is really a shame because the community apps is what pushed me to purchase a license. Oh well maybe I have just hit the magic number of files to hardware to warrant a different system for compute.
 
TLDR; Plex container was causing high IOWAIT which was causing my issues. Moving the workload off unraid solved* my issue.