Okay so I've made some progress, I put my RX 480 into the server, PCI power from my PC power supply, i mean it works right.
I managed to split the video and the HDMI audio in other groups, and I deleted my old VM, and made a new one. I've found out that I can see the bios of the VM and stuff, it all shows fine until the prompt "press any key to boot from CD or DVD", after that prompt goes through, on the VNC the windows install starts, but on the monitor the image freezes and never continues. After i force stop the VM the frozen image stays there, the only way to unfreeze it is to set the Sound device in the vm to the GPU, and then it gives me the error for failed to set up group, and as the error happens, the image unfreezes and the display turns off. If I launch the VM again, it shows video on the monitor right until that point.
Any suggestions for what to do next?