Brutally slow Windows 10 VM boot with >2GB RAM and GPU passthrough

[Blunt]

Hi guys, I'm new to the forum and this is my first post.

 

As the title suggests, when I assign more than 2GB RAM to my Windows 10 VM it takes longer than expected to start, followed by a 'guest has not initialized the display (yet)' delay, then a long wait at the Tianocore logo screen before finally booting Windows.

 

I have 10 cores 20 threads pinned and would ideally like to assign >64GB RAM.

I'm passing through my Samsung 970 EVO PLUS NVME controller, a GTX 1660 Super and the onboard ASMedia USB 3.0 controller.

 

Full tower spec

Intel Xeon 2673v4
ASUS X99 WS/IPMI
128GB 4 x 32GB Samsung ECC (M393A4K40CB1-CRC)
Gigabyte GTX 1660 Super
EVGA GTX 1660
ASUS ROG Areion 10gb nic
2 x 1TB Samsung 970 Pro (cache pool)
1 x 1TB Samsung 970 Evo Plus (pass through drive for VM)
2 x WD 12TB (shucked from Elements)
1 x WD 14TB (shucked from Elements)
RM 650W Corsair Power Supply

 

I have tried pinning fewer cores/threads, lowering allocated RAM to 2GB and while it does seem to help it's not going to be any good in the long term.

 

Similar posts I found while googling are years old, some state slow boot (GPU passthrough with high RAM allocation) has since been fixed.

Does anyone have any ideas or suggestions?

 

I can post my VM XML later if it helps?

 

Any help is greatly appreciated.

Thanks in advance

Edited February 27, 2020 by Blunt
Typo

[testdasi]

You probably want to attach diagnostics (Tools -> Diagnostics -> attach zip file)

 

Also, up top you said you are passing through the 970 Pro but below that you said passthrough 970 Evo Plus. Which one is passed through? What method did you use (ata-id? scsi bus? PCIe device?)

 

2 things you might want to try:

• Start a new config, pass through just storage + GPU and nothing else (i.e. no USB device / controller). I have had experience with strange delays on the Tianocore stage due to USB devices not doing handshakes properly.
• Go to you BIOS and look for any kind of power saving settings and turn them off / to max performance. I haven't had X99 for a long time so things are a little vague now but I remember there was a power-saving related thing in the BIOS that caused me quite a bit of troubles with Unraid VM.

[Blunt]

Sorry, meant to say I'm passing through the 970 EVO Plus. Both 970 Pro's are being used in the cache pool.

 

I have bound the 970 EVO Plus NVME controller using the VFIO-PCI Config plugin.

 

I'll attach my diagnostics zip when I get back from work this evening, thanks.

 

I have gone through the BIOS a few times looking for anything out of place but I'll do as you've recommended and double check there are no power saving features left enabled.

 

I'll also try dropping out the USB controller and/or removing the keyboard and mouse see if that helps.

 

 

 

 

 

 

 

 

[Blunt]

Apologies for the delayed response. After fiddling in the BIOS and re-enabling power saving, the VM is now booting within 15 seconds or so from vdisk. Performance state during VM boot seems to have caused the issue. 

 

I'll need to try passing the controller through on a second VM now. Hopefully it's sorted.

 

 

[wolfNZ]

For me I went from 5 - 10min VM boot load to about 30 seconds. SeaBIOS was incredibly slow to boot the VM but once I transitioned the system to UEFI I can now boot with a usable speed.

 

The steps I took to resolve the problem were;

 

• Backup your vdisk. Reduce your problematic VM CPU core count and RAM (4 cores, 8GB worked for me). Run the following commands in the windows VM which is slow to boot. This prepares the system for UEFI booting. 

   

// The first line checks if your system is valid for UEFI
mbr2gpt /validate /allowFullOS

// The second line will convert your boot to UEFI (less than 1m)
mbr2gpt /convert /disk:0 /allowFullOS

• Enable Unraids UEFI boot under the flash device settings. 

• Enable UEFI in motherboard (leave CSM enabled).
• Reboot the server.
• Create a new VM changing SeaBIOS to OVMF and whatever RAM/CPU core count you desire. Point the vdisk to your old image.

 

Please let me know if this works