Hi,

I'm trying to run 3 Windows VM at the same time each one with it's own dedicated GPU but GPUs on the same "Quick switch" slots of the motherboard do glitches (white bars, image flickering, etc). When running only 2 VM with the GPUs on different "Quick switch" all works fine. I tried to change GPU's PCIe slots, update Windows drivers, reboot but the results are the same.

How can I fix those glitches?

 

My hardware is:

CPU: i7 6700K (no overclocking)

Motherboard: Asus Z170 WS

GPUs: Asus Nvidia 660; Asus Nvidia 730; Asus Nvidia 720;

 

Any sugestion will be appreciated

Best regards

Hi,

I'm trying to run 3 Windows VM at the same time each one with it's own dedicated GPU but GPUs on the same "Quick switch" slots of the motherboard do glitches (white bars, image flickering, etc). When running only 2 VM with the GPUs on different "Quick switch" all works fine. I tried to change GPU's PCIe slots, update Windows drivers, reboot but the results are the same.

How can I fix those glitches?

 

My hardware is:

CPU: i7 6700K (no overclocking)

Motherboard: Asus Z170 WS

GPUs: Asus Nvidia 660; Asus Nvidia 730; Asus Nvidia 720;

 

Any sugestion will be appreciated

Best regards

 

What is a "quick switch" exactly?  This sounds like an issue specific to that.

Thanks for the reply,

 

I attached the Asus Z170 WS block diagram, as you can see on it, the slots 1 and 2 are on the same "Quick Switch", also 4 and 5.

 

I see that on the motherboard used on "7 Gamers 1 CPU" (Asus Z10PE-D8 WS), each PCIe slot of the CPU 1 has it own dedicated 8 lanes, without "quick switch" simulating more lanes than real so I think that's the problem but I don't know if it's fixable or if it's something else.

 

Maybe because from the CPU on the Asus Z170 WS comes out 16 lanes to the slots, those are the real maximum available to them. I'm newbie virtualizing hardware

I'm looking into this, but these quick switches are foreign territory to me and we don't have a system like that in house to test with.

It's possible to have 4 VM with their own GPU with the 1151 socket? What motherboard can do it (for example)? (or 3 VM if not)

 

Also it's possible to have 7 VM with their own GPU with only one 2011-3 socket? What motherboard?

 

I though I could get 4 VM on Asus Z170 WS but it looks not possible.

A quick update on this.  I spoke with someone else on this subject to get some more insight, but I think I'm left with more questions than answers.  In short, Skylake CPUs offer no support for ACS, which is what isolates PCIe devices into their own IOMMU groups nicely.  While previous generation CPUs from Intel could sometimes do this without needing ACS (such as my i7-4770k), the Skylake CPUs tend to group devices together.  The ACS override is designed to deal with that, but it is experimental and may cause issues (like what you're seeing).  Don't expect this to be resolved either.  The ACS override patch will never be merged into the upstream, so getting support for it is out of the question.  For those that it works for, great, for those that it doesn't, the answer is to not use it.

 

That said, my contact did have an interesting question:

 

I would wonder if the testing he's done to associate the

issue with the quickswitch is really just experiencing the issue due to

the memory layout of 2 cards vs 3 cards.  For instance, it may not be

supported by the BIOS, but can they make a 2-card configuration in

slots 1 & 2, slots 3 & 4 not loaded, that works correctly?

 

Maybe give that a shot?

 

It's possible to have 4 VM with their own GPU with the 1151 socket? What motherboard can do it (for example)? (or 3 VM if not)

 

Possible?  Maybe.  What motherboard?  I have no idea.  You're treading into foreign / unknown territory here.

 

Also it's possible to have 7 VM with their own GPU with only one 2011-3 socket? What motherboard?

 

I though I could get 4 VM on Asus Z170 WS but it looks not possible.

 

Have you seen the videos from Linus Tech Tips?  He did a dual and a 7 GPU setup.  The 7 GPU setup was with a 2011 socket.  Look up 7 gamers 1 CPU as a reference.  Just know that we don't recommend AMD GPUs.

Just know that we don't recommend AMD GPUs.

 

That's not fair, how about "we don't recommend Fiji or Hawaii based AMD GPU's at this time"?

Us Bonairre and Pitcairn's don't have these issues AND can be used for a VM as the only installed card. 

(I actually like Nvidia better, as I have never had an AMD card that did not die or had awful, awful, driver support).

 

Had to add (no I did not make this, nor endorse the having of 8 arms):

Just know that we don't recommend AMD GPUs.

 

That's not fair, how about "we don't recommend Fiji or Hawaii based AMD GPU's at this time"?

Us Bonairre and Pitcairn's don't have these issues AND can be used for a VM as the only installed card. 

(I actually like Nvidia better, as I have never had an AMD card that did not die or had awful, awful, driver support).

 

Had to add (no I did not make this, nor endorse the having of 8 arms):

I stand by my comment. I have seen tons of issues with all sorts of AMD GPUs. The bottom line is they just haven't had any sister products really designed for the use case with VMs and therefore were able to, quite frankly, cut corners on lots of hardware design choices that don't suit virtualization. While NVIDIA GPUs may have issues too, they are far less prone to them than their AMD counterparts because their sister cards based on the equivalent chipsets are designed to run within virtualized environments. So because they have taken the virtual-friendly approach in hardware design (like with bus resets), we like them better.

 

AMD is literally a gamble when it comes to passing through a GPU. I think with Linux based guests, it might be better, but with Windows, half the time just loading the AMD drivers cause issues.

 

Thanks for all replies, they are quite interesting!

 

Have you seen the videos from Linus Tech Tips?  He did a dual and a 7 GPU setup.  The 7 GPU setup was with a 2011 socket.  Look up 7 gamers 1 CPU as a reference.  Just know that we don't recommend AMD GPUs.

 

@jonp I seen 7 gamers 1 CPU but they use 2 CPU instead  . Just to confirm, 2011-v3 motherboards with 1 "real" CPU usually support to run 7 VM at same time each one with its own GPU? It's common support at least 4?

 

I'm planning to buy another motherboard and a CPU. This time I hope to get what I expect x)

 

The Asus Z170 WS virtualizes well, I can passthrough 2 GPU, 1 PCIe USB (maybe 2) and other internall things but I want to get al least 3 GPU powered VM 

I stand by my comment. I have seen tons of issues with all sorts of AMD GPUs. The bottom line is they just haven't had any sister products really designed for the use case with VMs and therefore were able to, quite frankly, cut corners on lots of hardware design choices that don't suit virtualization. While NVIDIA GPUs may have issues too, they are far less prone to them than their AMD counterparts because their sister cards based on the equivalent chipsets are designed to run within virtualized environments. So because they have taken the virtual-friendly approach in hardware design (like with bus resets), we like them better.

 

AMD is literally a gamble when it comes to passing through a GPU. I think with Linux based guests, it might be better, but with Windows, half the time just loading the AMD drivers cause issues.

 

Fair points, however responding to the support requests and issues here on the forum there seems to be a far higher number of Nvidia people complaining about issues than AMD ones.

Now this may also be that Nvidia is currently the "better buy" (I do not know, but is a likely thing), however price wise AMD remains competitive in most price points.

 

Also, AMD has never (in my knowledge) released a software update that (basically) broke virtualizing their GPU, causing the "error 43" message.

Then the decision to use the older software and remain with hyper-v, or disable it and use the newest. Now there is clearly a work around for this, however this is very recent.

Even though they acted like "we don't know what you're talking about, but we don't plan to fix it", it sure seems intentional and they have a product that directly supports that kind of use-case.

 

I sound like a fanboi, but I'm honestly not, in any way, to almost anything I purchase.

I had an R260X that worked extremely well, it had to be RMA's recently (good 'ol AMD), I recently purchased a R370 OC for testing, and usage until mine comes back.

I've had zero issues with this card, plug and play.

Many users here use the 6450 (a reasonably old card) without any problems also.

On that same token I have 3 GT 720's that are fantastic, take anything I throw at them.

 

I commend the recommendation for a processor with ACS as it is absolutely for exactly what we're trying to do, however I think this one is a bit biased (but I don't get support emails/PM's like you likely get a LOT of). Anyhow, just having a friendly conversation.

Thanks for all replies, they are quite interesting!

Just to confirm, 2011-v3 motherboards with 1 "real" CPU usually support to run 7 VM at same time each one with its own GPU? It's common support at least 4?

 

You're limited (to a point) with the design of the CPU and the # of PCI Express lanes coming from the CPU directly.

Without ACS support there is also no specific isolation, and is really dependent on how the motherboard manufacturer decides to "wire" the PCIe slots.

The Z97 chipset - for instance and an i7-4790k (a common choice) has 16 lanes directly from the CPU, no ACS support, and another 8 lanes from the chipset itself.

X99 chipset  is also limited to the same 8 lanes from the chipset, however the CPU has either 28 lanes (5820k) or 40 lanes (5930k, 5960X).

 

Most of the time (like always) the PCIe 16X slots are wired directly to the CPU. If we only have 16 lanes, then they share that bus. Without isolation between them directly, it is very likely they end up in the same IOMMU grouping.

 

So if the CPU has more of these lanes, it is very likely the MB will have more 16x slots, and the more we have, the more video cards we can run.

Add to that ACS, and you're pretty much guaranteed that they will be isolated, and assignable as needed.

 

Now the Z170 chipset (Skylake) is a more advanced chipset in ways, in that it's connection is a faster DMI 3.0, which also has 20 pci lanes from the chipset (used for 1X slots, Wifi, extra SATA, sound, etc...).

However a 6600 or 6700k only have the same 16 PCI lanes from the CPU, which again is primarily wired to the PCIe 16X slots.

 

Since a lot of us use "gaming" motherboards (speaking X99 here), we get 4 16X slots (not wired per say, typically some way of 40 total ex:16/16/4/4, 16/8/8/8).

However if the board is designed more for a server, you will find that some have more 16X length slots, wired as 8/8/8/8/8 which would support 5 GPU's, all operating at 8x PCIe speed.

You can also just buy specialty cards made for 1x slots, which will also work great for basic usage/HD, but limited for gaming use (ie: not recommended).

As explained before, you are limited in the number is PCIe lanes on many CPU/chipsets. If you want to run 4 GPU's effectively, and not spend 800+ bucks on a single CPU with 40 PCIe lanes, have a look here: https://lime-technology.com/forum/index.php?topic=46077.0

 

I picked up the CPU's, an asRock mobo and 64GB RAM for a few hundred bucks, and have currently got 3 VM's up with dedicated GPU's. When the GF moves in(shudder), Ill be looking to add a fourth.

Wow awesome replies. I invested months trying to understand the hardware passthrough on internet without success, here in just few replies I learned what I need

 

Another doubt that I have is that if on Xeon computers it's fine to share a thread to multiple VMs running at te same time. I get BSOD on Windows when I play CPU-GPU intensive games on both VMs at the same time but it works fine if they are not shared.

 

Thanks!!

Wow awesome replies. I invested months trying to understand the hardware passthrough on internet without success, here in just few replies I learned what I need

 

Another doubt that I have is that if on Xeon computers it's fine to share a thread to multiple VMs running at te same time. I get BSOD on Windows when I play CPU-GPU intensive games on both VMs at the same time but it works fine if they are not shared.

 

Thanks!!

 

I share some cores and have yet to have an issue.