(SOLVED) VM stuck at Tianocore with passthrough *IF* a monitor is connected to the GFX Card.

[dhstsw]

HI all,

 

Motherboard: Gigabyte B450 Aorus M (Bios with AGESA 1.0.0.3 but tried also with previous ones).
CPU: Ryzen 5 2400G

RAM: 16Gb
Unraid Version: 6.8.0-rc3

Tried cards:
Radeon RX 560 2Gb (Gigabyte Radeon™ RX 560 Gaming OC 2G)
NVidia GTX 670 2Gb.
Machine: i440fx-4.1 (tried 3.1 too).
BIOS: OVMF (with Seabios it doesn't even boot, not even from the install iso).
PCIe ACS override set as "Downstream" in VM Manager; this way i get the GPU and GPU-Audio in its own IOMMU group.

Trying to boot a Windows VM (or install one from zero).
The same goes with any VM (ubuntu and so on).

When i start the VM the Tianocore logo happears. HDDs are working for a while.
That's all.

Things i noted/did:
-My motherboard bios is setup to use the iGPU as primary; when i boot up i get the Unraid selection boot menu from the iGPU. When the selection is made and bzroot is launched, the output switches to the PCIe1 GPU.
-Even if the (already made) machine is setup to accept RPC session (remote desktop), normally working when the VM uses VNC as graphic card, if i setup the passed through GPU, remote desktop fails to connect.
-To elaborate more, after starting the VM with the passed through GPU, despite the 'normal looking' hdd activity, the VM fails to start a network connection (i can see this from the clients lists in my lan as showed by my router - VM is missing).
-For both GPU i tried configuring them WITH *and* WITHOUT the bios rom image. No differences.
-With the Radeon GPU i get the aforementioned problems (not working VM, tianocore logo). With the NVidia GPU i have the added "bonus" of having Unraid completely blocked after trying to stop the VM (Force stop, with normal stop it keeps to be 'on'). If i don't stop the VM and simply try to reboot/shutdown Unraid i obtain the same result (Unraid completely blocked). Of course, hard-restarting Unraid triggers the parity check.

So, basically i'm clueless. I noticed someone else on the forum had similar issues but didn't get any answer.

Can anybody help?

Thanks.

PS: I'll attach the syslog as soon as the parity check will end.

Edited October 23, 2019 by dhstsw

[John_M]

Did you stub the discrete GPU to prevent Unraid from seeing it?

[dhstsw]

23 minutes ago, John_M said:

Did you stub the discrete GPU to prevent Unraid from seeing it?

Probably is the only thing i forgot. Thanks.
I'll try as soon as the parity check ends.

C.

[dhstsw]

Nothing to do.

Removed the ACS support in unraid and turned it on on the motherboard BIOS.
I still have my nice IOMMU group:
 

IOMMU group 8:    

[1002:67ff] 01:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Baffin [Radeon RX 550 640SP / RX 560/560X] (rev cf)
[1002:aae0] 01:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Baffin HDMI/DP Audio [Radeon RX 550 640SP / RX 560/560X]

Stubbed the card:
 

append pci-stub.ids=1002:67ff,1002:aae0 initrd=/bzroot

 

No changes:

-During boot of Unraid the output still switches to the card output;
-Starting the VM with the card (graphic/audio), with or without rom bios files still 'hangs' at tiocore logo.

The only 'new' thing that happened was once the already made Windows10 image was damaged (caused i think by the continuous 'force stop vm'), the output of "windows can' boot, try again or press ESC to enter efi bios" went on the graphic card output.

Also, trying installing a VM from zero leads to the same result: Tiocore logo, no other output.

Here attacched the diagnostics file.
Hope someone can help.

Now, for another 17 hours parity check (yup, Unraid got frozen again).

C.

 

Edited October 23, 2019 by dhstsw

[dhstsw]

So, basically the output of the video is coming out from the DVI-D of the graphic card instead of the DP (of the same card).

Anyone knows how to solve this behaviour?

[J89eu]

Try swapping the machine to G35, that fixed the issue for me 

[dhstsw]

21 minutes ago, J89eu said:

Try swapping the machine to G35, that fixed the issue for me 

You mean Q35?

[dhstsw]

Nope.

The only solution to have it working is:

-Detach all monitor cable(s) from the GFX board;

-Start th VM;

-Wait until the VM booted;

-Attache the cables to the GFX Board;
 

If, for any reason, there's a monitor cable attacched to the GFX board before/while the VM starts it won't work.

:(

 

[testdasi]

As a starting point, are you booting Unraid itself in legacy mode i.e. NOT UEFI?

With cards that don't like being passed through (I vaguely remember RX 560 being one of them), the following seem to help:

• Boot Unraid in legacy mode - I purposely disabled UEFI when building my USB stick to ensure if it boots, it 100% boots in legacy mode.
• Set primary GPU as something other than the to-be-passed-through card in the motherboard BIOS
• Use Q35 machine type (I recommend starting a new template from scratch AND use Unraid 6.8, save you the trouble of needing the root port patch)
• Dump your own vbios and use it

Edited October 23, 2019 by testdasi

[dhstsw]

23 minutes ago, testdasi said:

As a starting point, are you booting Unraid itself in legacy mode i.e. NOT UEFI?

With cards that don't like being passed through (I vaguely remember RX 560 being one of them), the following seem to help:

• Boot Unraid in legacy mode - I purposely disabled UEFI when building my USB stick to ensure if it boots, it 100% boots in legacy mode.
• Set primary GPU as something other than the to-be-passed-through card in the motherboard BIOS
• Use Q35 machine type (I recommend starting a new template from scratch AND use Unraid 6.8, save you the trouble of needing the root port patch)
• Dump your own vbios and use it

 

I solved it while you where writing this
Well, now it's (finally) working.

Thanks everybody for the help!

BTW: should i put machine type to Q35 for every kind of VM (Linux, MacOS, etc)?

Thanks!

C.
 

 

[testdasi]

27 minutes ago, dhstsw said:

BTW: should i put machine type to Q35 for every kind of VM (Linux, MacOS, etc)?

It depends. In my own experience, any VM that needs OVMF to boot should be on Q35. I440fx works better for VM booting in SeaBIOS.

For Windows, I have found switching to Q35 machine type to resolve a lot of PCIe pass-through issues, especially the infamous error 43 for Nvidia GPU.

For MacOS, Q35 is a pre-requisite.

 

Edited October 23, 2019 by testdasi

[dhstsw]

Thanks!

[colinator]

What did you do to resolve this?

running into the same issue here....

[dhstsw]

See above post by testdasi.

[colinator]

Unfortunately that didn’t help in my case.

also tried your idea on booting the vm with no cables attached but no luck.

 

whatever I try, I can only boot windows in Safe Mode with GPU passthrough. Once I boot normally, it freezes on the tianocore boot screen, I believe at the moment the drivers are loaded and the GPU is utilized.

 

reinstalling windows works fine til the point of driver installation, then I get a black screen and restarting the VM results in same boot logo freeze.

[dhstsw]

So, youre booting Unraid in NON EFI, right?
Try to post your Tools/System Devices, that may help to better understand.