EDIT - Sorry to waste everyone's time, i looked just now and tried to transcode and plex server crashed. unraid itself has not crashed atleast but the only fix is a reboot of unraid itself.
I have been following this thread for a couple months now. I am running a 15-12500 and a asus B660 motherboard, ddr4 ram. Ive been battling the same issues with unraid crashing that everyone else has with quicksync transcoding. With the recent version 6.11 RC2 and the latest plex update, i still had issues with HW transcoding causing crashes with plex container or unraid itelf.
Two days ago I decided to turn on the monitor that i keep connected to my server, just for funsies. Since i rebooted and kept the monitor on (and it just sitting at the login prompt) i have had hw accel and hw decode and HDR tonemapping enabled without issues for two days now. Between me (local) and one remote user the server has transcoded 4k and 1080p content 8 times now without any crash or issue.
I cant account it 100% to the monitor plugged in, or if its just not my time to crash yet but I figured i would share this and see if anyone else can plug in a monitor or monitor dummy plug in to their server and see if they get any different results.
I'll update when this crashes (or if it does). I did just notice on the GPU stats plugin mine is showing power usage even though the GPU is not being used right now. Last transcode was last night, without issues. but no crash.