Update on the issue, right before the whole unraid server becomes unresponsive I got this message inside Plex.
After this message the only way to restore unraid is to do a hard shutdown.
It doesn't crash if I don't pass the GPU device into Plex.
The hardware encoding works perfectly but it will crash/hung the whole unraid server including unraid itself after awhile.
This happens in both 6.9 and 6.10 version of unraid.
nuc-diagnostics-20211007-1101.zip