So I've got my VM setup with Catalina and my gpu passthrough is working. I passed through my USB controller and my mouse skips around and my keyboard has a delay and some keys send multiple keys to the VM.
I have a W10 VM with the same GPU and USB controller and everything works properly on that.
IMG_2170.mp4