I've just been trying to get Nvidia hardware encoding to work. I am having apparently this exact issue. The GPU seems to start a process for encoding then drops it almost immediately. One thing I have noticed though is if I switch back and forth between original quality and a transcode a couple times, it seems to trick it into keeping the process running. However, this is very annoying and far from a solution. I'll note I've updated to Unraid 6.12.0, installed the latest Nvidia Drivers from the plugin, restarted, added the --runtime=nvidia and the nvidia devices environment variable for plex, restarted again, switched PCIe slots, somewhere said to make sure a cable is plugged into the card so I tried that, and switched to a different plex image (from plexinc to linuxserver). All to no avail.
Edit: I've also tried using the binhex/plexpass image. I am currently trying the linuxserver image with a different nvidia driver version