I just followed what's in here to first enable the NVIDIA card on my Unraid server, then to enable access to the GPU on my PLEX docker, then I followed the instructions over at PLEX forum to create a workaround to enable the NVDEC. (see this link : Instructions for the script
It's only 2 lines of codes you put in a new file called "/usr/lib/plexmediaser/Plex Transcoder".
Step 1. Go inside the docker PLEX terminal, to /usr/lib/plexmediaserver/.
Step 2. Rename the file "Plex Transcoder" to "Plex Transcoder2"
Step 3. Create a script called "Plex Transcoder" (I use nano editor, apt-get update, apt-get install nano, yes to all questions), and put this code in it :
#!/bin/sh
/usr/lib/plexmediaserver/Plex\ Transcoder2 -hwaccel nvdec "$@"
This will tell call the Plex Transcoder and enable the decode part. Since it's a workaround, it won't show it Tautulli or in Plex Dashboard because plex doesn't know it decode hardware. You can see it working by running this command from the same Terminal inside plex docker:
nvidia-smi dmon -s u
There is column for enc. and one for dec., if it's not 0, it means it's working.