I am also having a similar issue. ( and I am who is currently googling them and pulling their hair).
As long as the Nvidia driver is installed, I can't reboot and the first CPU that's allocated to the VM is stuck at 100%, I have to delete `-device vfio-pci` , then boot to Guest and delete the driver. Then it can boot normally.