WinPro kvm black screen with edited vbios....

[frodr]

I set up the Nvidia Quadro RTX 4000 as secondary GPU, dumped the vbios and edited it. Then put it as primary with the modded vbios, but still black screen. 

 

No ACS override.

VFIO-PCI Config 

 

I really want to get this fixed, but I need help.

 

 

// Frode

 

 

 

Asus C621E Sage/BMC | dual Xeon Silver 4116 | Samsung 4 x 32GB DDR4-2933 RDIMM | 4 x Corsair Force MP510 (960GB) NvME ssd on Asus Hyper M.2 Card V2 l | 4 x WD and Seagate hdd l Sapphire Pulse Radeon RX 580 8GB as Mac WS l Quadro RTX4000 l Samsung 970 Pro in M.2 port  l Asmedia PCIe USB card

WinPro_black.rtf maxx-diagnostics-20200302-1554.zip

[testdasi]

13 minutes ago, frodr said:

I set up the Nvidia Quadro RTX 4000 as secondary GPU, dumped the vbios and edited it. Then put it as primary with the modded vbios, but still black screen. 

How did you "put it as primary"? Have you tried if it would work as secondary?

[frodr]

2 minutes ago, testdasi said:

How did you "put it as primary"? Have you tried if it would work as secondary?

I set up a new kvm and then changed from vnc to the GPU. Yes, on another kvm I set it up as secondary and it seems to be working. Trying Plex Server on it was not working well with transcoding. 

[frodr]

Does this vbios seem to be edited correctly?

 

// Frode 

TU104mod.rom

[david279]

1 hour ago, frodr said:

Does this vbios seem to be edited correctly?

 

// Frode 

TU104mod.rom 861 kB · 0 downloads

Just took a look at it and it does look edited correctly

[frodr]

Just took a look at it and it does look edited correctly

Thanks, then the vbios is clear. What can be wrong?

[frodr]

Anyone?

 

Is this problem "only" my Windows kvm`s?

 

 

// Frode

[frodr]

Any Limetech guys around?

 

Is the problem with black screen Windows kvm Nvidia GPU passthrough an Unraid issue, or most likely related to my Server? Either way it has to be solved as kvm`s is a major functionality on the Server. If it the first Iĺl be happy to get a feedback. 

 

Cheers,

 

 

Frode

[JonathanM]

16 minutes ago, frodr said:

Any Limetech guys around?

https://unraid.net/contact

[frodr]

6 minutes ago, jonathanm said:

https://unraid.net/contact

Thanks.

[jonp]

Hi there,

 

Unfortunately you have already gone down all the paths I would have suggested to get this into a working state.  You're using it as a secondary GPU (so it's not competing for the primary slot), you've dumped/edited the vbios, and properly stubbed the IOMMU group.  Unfortunately we don't have a lot of experience with Quadro cards, only GTX/RTX, so this could very well be a hardware-specific issue.  I'd try replacing the GPU with something like a GTX/RTX from EVGA and see if that works.

[frodr]

4 minutes ago, jonp said:

Hi there,

 

Unfortunately you have already gone down all the paths I would have suggested to get this into a working state.  You're using it as a secondary GPU (so it's not competing for the primary slot), you've dumped/edited the vbios, and properly stubbed the IOMMU group.  Unfortunately we don't have a lot of experience with Quadro cards, only GTX/RTX, so this could very well be a hardware-specific issue.  I'd try replacing the GPU with something like a GTX/RTX from EVGA and see if that works.

Hi,

 

Thanks for very quick feedback. Windows is seeing the Quadro GPU as a secondary GPU. First test with Plex Server in this specific setup was not so good as video stream stopped when transcoding. Can be something with the transcoding setup though. 

 

Isn't the Geforce RTX and Nvidia RTX based on the same technology? Also, I could not get a Geforce RTX 2080ti out of black screen on primary either. 

 

I have been running RealVNC Server on the kvm. I started a fresh WinPeo10 install and made as little changes as possible before adding Splashtop and passthrough the quadro card. I was able to connect with quadro as primary, and install the quadro driver. The kvm was extremely sluggish, scrolling not possible. I then set the quadro up with the original kvm and was able to connect through Splashtop. Also on this kvm the scrolling was not possible and a strange resolution. After a kvm restart or two, I also got a black screen with Splashtop. 

 

My next question is, is this most likely an Unraid issue, or an issue with my Server? If the first, it is something that Limetech will look into?

 

 

Cheers,

 

 

Frode

[jonp]

On 3/3/2020 at 4:53 PM, frodr said:

Thanks for very quick feedback. Windows is seeing the Quadro GPU as a secondary GPU.

Ok, that is likely why you're having the issue.  You need to disable VNC and use the Quadro as the primary GPU with no secondary.  If you need remote graphics access to the VM, you'll need to install a client in the guest VM such as RealVNC (or if Windows, you can use RDP) to connect.  The VNC graphics adapter Unraid can provide the VM only works when its the primary and that will cause issues with the GPU pass through in many configurations.  You should be able to install Windows with a locally attached monitor, mouse, and keyboard if you pass those devices and the GPU through to the VM.

 

On 3/3/2020 at 4:53 PM, frodr said:

Isn't the Geforce RTX and Nvidia RTX based on the same technology? Also, I could not get a Geforce RTX 2080ti out of black screen on primary either.

Yes, but individual GPU devices can function differently when it comes to virtualization and GPU pass through.  For example, there are plenty of RTX 2080 Ti cards out there that don't work, but EVGA-branded devices do.  This is because their firmware is slightly different.  This isn't to say that only EVGA devices work, but those are the ones we've personally tested and with which we've had the best and most consistent results.

 

On 3/3/2020 at 4:53 PM, frodr said:

I have been running RealVNC Server on the kvm.

I think you mean in the VM.  If you use VNC to connect to a VM, regardless of the GPU horsepower attached, the experience will be sluggish.  That is because VNC isn't accelerated by GPUs.  Splashtop might be (honestly not sure because I haven't used it before).  Microsoft RDP definitely can benefit from having a GPU available.  As far as getting a black screen with Splashtop, I can't really speak to that as I don't use it.  That said, I've used RDP in the past and had good results.

 

On 3/3/2020 at 4:53 PM, frodr said:

My next question is, is this most likely an Unraid issue, or an issue with my Server? If the first, it is something that Limetech will look into?

Its definitely not an Unraid issue.  What's important to understand is that Unraid utilizes Linux KVM for virtualization.  This means that we can only deliver the functionality that Linux itself can provide, and unfortunately when it comes to GPU pass through, there is a lot of inconsistency in what is going to work out there.  Most folks using standard hardware will be fine, but there are outliers that run into weird issues like these because of unique hardware that doesn't operate in a standard fashion.

 

Unless we see wider spread reports of this type of problem, there really isn't anything for us to look into, because we don't have the same hardware as you and can't replicate the problem.

[frodr]

44 minutes ago, jonp said:

Ok, that is likely why you're having the issue.  You need to disable VNC and use the Quadro as the primary GPU with no secondary.  If you need remote graphics access to the VM, you'll need to install a client in the guest VM such as RealVNC (or if Windows, you can use RDP) to connect.  The VNC graphics adapter Unraid can provide the VM only works when its the primary and that will cause issues with the GPU pass through in many configurations.  You should be able to install Windows with a locally attached monitor, mouse, and keyboard if you pass those devices and the GPU through to the VM.

A misunderstanding. Quadro as secondary just to download the vbios and to confirm that the kvm is working in this setup. All attempts to connect giving black screen is with quadro as primary. Black screen with RealVNS and Splashtop, RDP not tested. 

44 minutes ago, jonp said:

 

Yes, but individual GPU devices can function differently when it comes to virtualization and GPU pass through.  For example, there are plenty of RTX 2080 Ti cards out there that don't work, but EVGA-branded devices do.  This is because their firmware is slightly different.  This isn't to say that only EVGA devices work, but those are the ones we've personally tested and with which we've had the best and most consistent results.

This is important information. I will get a EVGA GPU. Any RTX 2080 ti tested and ok?

44 minutes ago, jonp said:

 

I think you mean in the VM.  If you use VNC to connect to a VM, regardless of the GPU horsepower attached, the experience will be sluggish.  That is because VNC isn't accelerated by GPUs.  Splashtop might be (honestly not sure because I haven't used it before).  Microsoft RDP definitely can benefit from having a GPU available.  As far as getting a black screen with Splashtop, I can't really speak to that as I don't use it.  That said, I've used RDP in the past and had good results.

My experience with RealVNC have fine, actually Splashtop have been more "greasy" feeling. The sluggishness was referred to when I was able to connect thru Splashtop with Quadro as primary. It was actually impossible to scroll. 

44 minutes ago, jonp said:

 

Its definitely not an Unraid issue.  What's important to understand is that Unraid utilizes Linux KVM for virtualization.  This means that we can only deliver the functionality that Linux itself can provide, and unfortunately when it comes to GPU pass through, there is a lot of inconsistency in what is going to work out there.  Most folks using standard hardware will be fine, but there are outliers that run into weird issues like these because of unique hardware that doesn't operate in a standard fashion.

 

Unless we see wider spread reports of this type of problem, there really isn't anything for us to look into, because we don't have the same hardware as you and can't replicate the problem.

Understand. 

[frodr]

Last update.

 

Good news!

 

With Microsoft RDP and Quadro as primary I am able to connect. It seems to be working ok, just 10 min. testing...

 

One question. I see two Display Adapters, RTX 4000 and MRDP. Is this correct? 

 

 

Cheers,

 

 

Frode

[frodr]

No sigar....

 

Running Passmark benchmark and Nvidia Control Panel shows some issues.

[testdasi]

Reinstall drivers? Maybe silly question but did you install the Quadro driver? I don't think it's the same as the consumer line.

[frodr]

Reinstall drivers? Maybe silly question but did you install the Quadro driver? I don't think it's the same as the consumer line.

Done that. Yes, installed quadro drivers.

[frodr]

6 hours ago, jonp said:

I think you mean in the VM.  If you use VNC to connect to a VM, regardless of the GPU horsepower attached, the experience will be sluggish.  That is because VNC isn't accelerated by GPUs.  Splashtop might be (honestly not sure because I haven't used it before).  Microsoft RDP definitely can benefit from having a GPU available.  As far as getting a black screen with Splashtop, I can't really speak to that as I don't use it.  That said, I've used RDP in the past and had good results.

With RDP it creates another Display Adapter. To my understanding RDP overrides xml settings, putting HW-GPU as secondary. Is this the case?

 

In the setup (pictures) I put VNC as primary, nothing as secondary. Above picture is connection thru RealVNC, the lower picture is with RDP

 

Edited March 5, 2020 by frodr

[jonp]

Rdp can lean on your add on gpu to accelerate things like video. Have you tried YouTube through the rdp session?

Sent from my Pixel 3 XL using Tapatalk

[frodr]

14 hours ago, jonp said:

Rdp can lean on your add on gpu to accelerate things like video. Have you tried YouTube through the rdp session?

Sent from my Pixel 3 XL using Tapatalk
 

Yes, I got sound thru RDP.

 

Nvidia Control Panel error message says Nvidia GPU not running. Which seems correct as RDP is running on a RDP specific Display Adapter. 

[jonp]

Some articles that may assist you:

 

https://getadmx.com/?Category=Windows_10_2016&Policy=Microsoft.Policies.TerminalServer::TS_SERVER_AVC_HW_ENCODE_PREFERRED

https://blog.tedd.no/2011/04/28/optimizing-rdp-for-casual-use-windows-7-sp1-remotefx/

 

In short, I believe you can achieve hardware acceleration for your RDP connection, regardless of what display adapter shows up.

 

What's even more important than that was how the experience was through RDP.  Was it sluggish scrolling or did it perform up to expectations?  You're not going to get 3D Acceleration through Splashtop or VNC, but there is a possibilty with RemoteFX over RDP.  That said, a lot of this is beyond Unraid and more Microsoft-specific.

[frodr]

I connected monitor directly to the Quadro card, and Nvidia Control Panel starts up fine. Passmark benchmark runs without error message. 

 

It seems as the kvm runs fine. I will test more during the week-end. 

 

Thank you very much, Limetech. You stepped up when I needed you. Even though you didn't do anything specific, your advice pushed me in the right direction. Now we are a little bit wiser regarding Quadro RTX in kvm. I will try to set up RDP as to use hw GPU, but that's secondary. 

 

Cheers,

 

 

Frode

[frodr]

I have tested a game and it works fine. No error messages. Here is a short summary for future use:

 

• When passthrough Nvidia RTX 4000 in a Windows 10 Pro kvm I got a black screen when connecting remotely, Splashtop Desktop Personal and RealVNC. 
• Kvm setup based on SIO`s video tutorials. Running VFIO-PCI Config (Plugin in the App Store).
• First adding the Quadro card as 2nd graphics card and dumped the vbios using techpowerup.com`s app GPU-Z.
• Edited the vbios according SIO`s video.
• Passthrough the edited vbios.
• At this moment it was still black screen with both Splashtop and RealVNC. 
• Recommended to try Microsoft Remote Desktop by Limetech guys.
• Microsoft adds a Display Adapter that RDP uses to remotely connect. Starting i.e. Nvidia Control Panel remote creates an error message saying no GPU in use. Setting the RDP to run hardware accelerated GPU solves this. 

 

 

How to enable GPU support in RDP:

 

Open the Edit Group Policy tool from Control Panel or use the Windows Search dialog (Windows Key + R, then type in gpedit.msc)

Browse to: Local Computer Policy\Computer Configuration\Administrative Templates\Windows Components\Remote Desktop Services\Remote Desktop Session Host\Remote Session Environment 

Then enable “Use the hardware default graphics adapter for all Remote Desktop Services sessions”

 

Cheers,

 

 

Frode

 

 

Asus C621E Sage/BMC | dual Xeon Silver 4116 | Samsung 4 x 32GB DDR4-2933 RDIMM | 4 x Corsair Force MP510 (960GB) NvME ssd on Asus Hyper M.2 Card V2 l | 4 x WD and Seagate hdd l Sapphire Pulse Radeon RX 580 8GB as Mac WS l ASUS GeForce RTX 2080 Ti ROG Strix Advance in a gaming kvm (temporary out the box) l Quadro RTX4000 for gaming and Plex kvm l Samsung 970 Pro in M.2 port for gaming kvm  l Asmedia PCIe USB card