Struggling to run even 1 Windows VM with a GPU

[Sawtaytoes]

I built a new system to run 2 Windows VMs with Unraid using 2 PCs and half of the cores and RAM in AMD Ryzen CPU, but I've spent 2 days getting a bunch of black screens.

 

The furthest I've gotten was seeing the Windows installer working via the secondary GPU in a VM all the way to the place after it installs Windows and restarts the computer. I went to sleep after that, but I've never been able to replicate it.

In the meantime, I've consistently been able to install Windows in the VM using the VNC option. That works fine. As soon as I try to use either of the 2 nVIDIA GPUs with or without the soundcard, it doesn't work.

The GPUs are the:

- nVIDIA GTX 1080 Ti (Primary)

- nVIDIA GTX 980 Ti (Secondary)

 

I know it's loading the VM because I switched back to VNC and saw the nVIDIA drivers were installed. My monitor's still blank though. When I switch to the primary GPU, the console output from Unraid becomes zoomed in and the screen shows a bunch of colorful artifacts on a black background. Using the secondary GPU, it's just blank. Nothing from the monitor.

 

I thought I could use RDP for debugging, but even after logging into my Microsoft account and enabling Remote Desktop, it doesn't work. Dunno what's going on there. Ping doesn't work either, and the firewall is correctly in Private mode.

 

I've tried just about every configuration option to see what would change:

- CPUs

- Memory

- Machine

- BIOS

- Hyper-V

- USB Controller

- IDE vs SATA for ISO

- Network Bridge and Model

- Having or not having USB Devices

 

1. I also dumped the vbios from both cards and tried using those.

2. I tried grabbing the latest vbios for each card off TechPowerUp and removing the "no VM" hex at the beginning.

3. I tried swapping the DisplayPort to different slots.
4. Tried attaching and removing USB devices.

5. Tried creating a bunch of new VMs with different or the same images.
6. Upgraded to Unraid 6.10 RC1.

 

I cannot figure out to change to legacy boot mode. I'm stuck in UEFI boot mode as far as I can see.

I'm lost. Everyone makes this look so easy.

System Specs:

- ASUS ROG Crosshair VIII Dark Hero

- AMD Ryzen 3800X

- 32GB RAM

- nVIDIA GTX 1080 Ti (Primary)

- nVIDIA GTX 980 Ti (Secondary)

- 3x250GB SATA SSDs (2xArray, 1xParity)

 

Edited August 29, 2021 by Sawtaytoes