GPU and USB Passthrough breaking network

[Brad the Beast]

I'm having a really bad time trying to get all my stuff to work so bear with me. I have a Windows 10 VM that I'm trying to get working. I have an Intel X710-T4 passed through to a Windows 10 VM. When I pass through either the USB controller or the Titan RTX, then the networking works fine. But when I pass through both the GPU and the USB controller the networking breaks. I can see that the link is up and the NIC is detected correctly but the VM can't transmit any data. Sometimes I get the attached message and the NIC shows up with a "#2" at the end of the name in Device Manager. I haven't touched the NIC passthrough so I don't understand what is happening. Assistance would be greatly appreciated. 

 

Hardware:

Asus Z11PG-D24 Motherboard

2x Intel Xeon Gold 6238

256GB DDR4

2x 2TB Samsung 970 Evo Plus

2x Nvidia Titan RTX

2x Nvidia Tesla V100

Intel X710-T4 NIC

StarTech 4-port USB 3 PCIe Card (PEXUSB3S44V)

Unraid 6.8.3

Edited December 11, 2020 by Brad the Beast

[jonp]

What if you don't pass through the NIC.  If you just pass through the USB/GPU, does everything work fine?  If so, why did you want to pass through the NIC direct to the VM anyway?

[Brad the Beast]

1 hour ago, jonp said:

What if you don't pass through the NIC.  If you just pass through the USB/GPU, does everything work fine?  If so, why did you want to pass through the NIC direct to the VM anyway?

I want each VM to have it's own 10Gbps connection. Most of my machines don't have onboard 10Gbps. 

[Brad the Beast]

So upon talking with our network team, we've made a discovery. When the VM's are up, the switches report input errors whenever I try to do anything like ping the gateway of the VLAN. When the VM's are off, the input errors stop. 

Edited December 15, 2020 by Brad the Beast

[Brad the Beast]

@jonpAny ideas? This is a time sensitive issue. 

[Brad the Beast]

So I've figured something out. The networking works fine with the GPU and USB controller passed through. 

Normal behavior: VNC as primary graphics and GPU as secondary graphics

Strange behavior: GPU as primary graphics.

When I set the GPU to be the primary graphics card, Windows detects the NIC as if it were a different device even though the NIC has the same MAC address. The NIC shows up in device manager as the correct model but it has a #2 or #3 appended to the end. Windows also dumps the static IP I set because it thinks it has a different NIC. 

 

If I try to change it back to VNC as primary and GPU as secondary, the whole host crashes with a machine check exception. 

 

So when I change the primary graphics from VNC to the GPU, Unraid changes something which causes Windows not to detect the NIC correctly. Any ideas?

Edited December 30, 2020 by Brad the Beast

[jonp]

Hi Brad and apologies for the delay in getting back to you.  The holidays got away from me a bit.  This is really perplexing.  There was one other user who reported this as well, but unfortunately I don't have any insights on what might be causing it.  Seems like an issue with upstream Linux/VFIO/QEMU.  Have you tried 6.9-rc to see if the newer kernel has any impact?

[Brad the Beast]

@jonp,

 

Thanks for getting back to me. After updating to 6.9-rc2, the same issues still persist. It seems to only affect my Windows 10 VMs. I also tried the latest version of VirtIO.