So this is something that would be worth adding to the documentation somewhere, I think it's not fully documented. Read the posts below.
USE AT YOUR OWN RISK! THIS IS NOT DOCUMENTED ON PURPOSE. CAN LEAD TO DATA LOSS IF YOU CRASH YOUR SERVER.
Here are the steps I took with the nvidia-driver installed
Settings -> VM Manager -> PCIe ACS override -> Set to "Downstream"
Reboot server
Check Tools -> System Devices and make sure the GPU and it's sound card are in an IOMMU group by themselves
Run `lspci -vnn -s 01:00.0` to verify `Kernel driver in use: nvidia`.
At this stage I have `nvidia-persistenced` currently running, and made sure it's not being used by any Docker containers explicitly
Run `pidof nvidia-persistenced`
Run `kill -9 5212` - or whatever the pid is from the above command
Now we should be good to pass through the GPU to the VM
Make sure to pass through both the VM and its sound card
Switch to XML mode on the same page (top right corner)
Make sure the GPU is in the same `slot`, add `multifunction='on'` and increment the soundcard function to `0x1`.
Start the VM, install nvidia drivers and you should be good to go!
Thank you for your help!