Did you ever find a fix for this? Im legit having an extremely similar issue. Expect I have a 1650 Super.
GPU transcodes on all client players expect browser. If it is 'Converting', it will transcode very slow. If i set it to a specific resolution, i get the following error:
[Req#2ae/Transcode] [FFMPEG] - Failed to initialise VAAPI connection: -1 (unknown libva error).
GPU transcoding works great on all clients except browsers.