not able to run VMs over GPU passthrough


Recommended Posts

Hi Guys, I'm trying for several days to run VMs over GPU passthrough but it doesn't work...

 

I'm running Unraid 6.7.0 UEFI Mode

 

My Hardware is a

Asus X99-A/USB 3.1 + [email protected]

PCI Slot #1 - ATI Radeon HD 5570 I think (it's really old)

PCI Slot #2 - NVIDIA GALAX GTX 970 EXOC

 

I've seen I think all videos on about GPU passthrough on YT, all the amazing vids from Spaceinvader One, Linus Tech Tips and finally even b00bytrap thanks for that. With same settings and without. GPU BIOS dump file and without btw the GPU BIOS is from techpowerup GALAX or kown as KFA2 (same size and company). I've tried PCIe ACS override: Multi and off. These ACS setting in UEFI. And first install OS with VNC and switch to GPU passthrough afterwards.

 

If I passthrough the GTX970 there's always a blackscreen or a vm logo. And I don't know but as far as I can the HD 5570 is on Slot #1 but when I reboot server there is only a output on the gtx970, wth? I tested it with Manjaro + HD 5570 and there was a boot menu where I could boot into the live desk but after it loads it's stuck at a last point, I seached for that issue in the web, but it doesn't work.

 

My goal is to use my main PC as a NAS and PC. I want to use Linux+GNU Manjaro and for stuff that doesn't works I'll use Windows 10. The installation always works with VNC. The last thing I've tested was that I've installed Windows 10 with VNC, activated RDP and passthrough GTX970 and install NVIDIA Driver restarted the VM and the complete unraid server freezed.

 

I hope you guys can help me! Thanks!

 

 

Edited by Tauro
Link to comment

First and foremost, you have 2 GPUs, there is no excuse to not dumping your own BIOS. Yours is the 3rd or 4th cases I have seen of people using BIOS from techpowerup without understanding exactly what they are using. When it comes to GPU BIOS, close enough is not good enough so I can understand if people only have one GPU and can't dump their own but you have 2, there is no excuse.

 

Now your symptom of only having output on the GTX 970 suggests your motherboard is picking the 970 as the main GPU for some reasons. If I remember correctly, some Asus motherboards allow people to pick which GPU to boot from so it's most likely to be culprit. Check your BIOS setting thoroughly.

Link to comment

Ok, thank you, now I have a clue what to do next. Yes, I had put in the ATI because it didn't work with one. So, next step is to look up for the setting in uefi for preferred gpu use and dump the gtx970 bios.

Edited by Tauro
Link to comment

There aren't settings to prefer the GTX 970. I can only say PCIEX16_1 is GEN2 or Bandwidth. And there are Options for MCTP, ACS Control, DMI, ASPM Support (L1? is disabled). CSM/Fast Boot are disabled. Ahh there's a GPU Post I see that my GTX is running at x1 Native and I have to switch it to Slot 3. After this I boot to Windows (not the VM) and dump the Bios with gpu-z so I don't need a second gpu for that. I removed the nvflash header. And now I test the passthrough vm, first Windows 10.

 

EDIT: stuck at TianoCore Splashscreen comes in existing Windows 10 VM, i440fx-3.1, OVMF.

 

EDIT#2: At the first time after I delete all old vms and make a new one I see a UEFI Interactive Shell with something 'Press ESC in 1 sec. to skip startup.nsh or any other key to continue.'

 

EDIT#3: Passthrough works finally with SeaBIOS and Windows 10! But the next problem has appeared. But entire Unraid crashes randomly last action I was installing Nvidia driver... Could this be a stability issue?

Edited by Tauro
Link to comment

Yeah, Thank you for your help! I had tried it with default bios settings with virtualisation stuff on but same Post Code Error appeared [FF]. First looked up maybe a shortning or smell but nothing. I replaced graphics card and it is not. Now I am testing the RAM in MemTest86 for 2h. Pass 1/4 but 91% of the second pass and no Error I think I'll stop it soon?

 

With the last OC in windows I never had or at least really rare problems.

Edited by Tauro
Link to comment

This isn't windows land- unfortunately unraid is pretty sensitive to overclocking and ECC ram is even recommended to prevent data corruption.  I can overclock the crap out of my rig with a bare metal windows install but unraid will freak out on those same settings after I start a few VMs and dockers.  I highly recommend just getting rid of all overclocking- even if just to test.  Once you have things stable on stock clocks- depending on how much pain and suffering you can endure- you can start tweaking but really need a great way to stress test the entire machine- not just single VMs on the host.

Link to comment

I've still default UEFI settings, now with better cooling and CSM on and for now it's stable. Problem is after I start The VM there comes boot bla bla and than blackscreen even with new vm. Are there any suggestions about i440fx/Q35 and SeaBIOS/OVMF?

 

And I haven't ECC X99 doesn't support it unfortunately.

Edited by Tauro
Link to comment

Ok Thanks. I tested it and it work but the graphics make now problems. I've tried to reinstall it over fivr times. First Direct from downloaded Nvidia. Next deinstall it over manager and nvidia again with reboot etc. Than over Windows search for driver. Now I think I'll test it with seabios..

 

800x600 px isn't great ^^

Edited by Tauro
Link to comment

Like I said in a previous post. I had it already dumped in a past, ok I didn't mention that I delete the header. With SeaBIOS and i440fx it says 'Booting from Harddisk.... boot failed and than boot from DVD and than blackscreen even with one core tip from spaceinvader. Now I try SeaBIOS with Q35. EDIT: Ok same error occurred.

Edited by Tauro
Link to comment

OMG after reinstalling Windows with at least 7 times, with different Versions. I've googled about gtx 970 and there was one guy that suggests to use the onboard gpu, so I thought, maybe I should test it again with a "newer" graphics card in pci-e slot 1 because that old hd5570 doesn't show the uefi. But with the HD6850 the uefi showed up. Next step was that I reconfigure the vm config so that there isn't a bios dump in it and has the right gpu (gtx 970). After I start the vm I installed at the device manager VirtIO Serial Driver again because that was a new vm. Immediately windows were refreshing and it was 1080p. Reinstalling the newest Nvidia driver wasn't a problem now.

 

Next step is to get it working with single graphics card bacause it does only run with a speed of pci-e 8x. I'm trying to dump the bios via terminal. Maybe that will work. But next trial is to get the HD6850 with bios dump working mybe the gtx 970 has a special problem.

 

And sorry for may bad English. :)

 

EDIT#1:

I think there is a problem with the mainboard because if done the dump via terminal and both works as bad as the gpu-z export without header. Yesterday I test the HD 6850 an there wasn't even a signal with a single card. In my dads pc is still my old gtx 660 next and last card I'll test.

 

EDIT#2:

It didn't work. The best results for a Windows 10 VM as it is recommended are OVMF+i440fx. For Linux Arch worked fine OVMF+Q35. And I fixed an issu that was my fault. The ntp or clock at my VM were wrong because I forgot to change it in UNRAID.

Edited by Tauro
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.