Can you pls provide some more detail of how you got passthrough to work on Windows VM initially (w.o drivers)? VM setup, specific tweaks, etc. I think we're in the same boat as in gaming is not really a big concern, and occasional desktop use is good enough. For me after setting up the vm, set the graphic to AMD Cezanne, along with its audio, the vm starts fine, but no signal comes out of HDMI output.
As for the driver issue I do remember seeing something similar to yours elsewhere, but with other setup of course. Think that's not an easy fix tho.
I did upgrade to 6.10 rc, and I can try out any ideas you have as well. btw it's (supposedly) quite easy to revert back to 6.9 as well according to tutorials, just so if you want to try it out.