@koiril, the best way to install GPU in VM is to use VNC all the way up unto ALL drivers are installed. I have x470 based mobo and Ryzen 5 CPU with XFX RX580 GPU. I had installed it into Win10 Vm without any issues. Well, there is an issue of Error 43 that appears if I do not shutdown VM completely/safely. Other than that, it works.
Also, take a look at SpaceInvaderOne video about advanced GPU pathrough. It shows that it is necessary to passthrough both Video&Audo devices together in the same virtual Bus and Slot but use function 0 and 1 accordingly.