Unraid Windows 10 VM 2 GPUs RTX 3070 and 1650 Fail

[Freelancer83]

Hello friends,
I've been trying to install a Windows 10 VM in Unraid for days. Unfortunately it always fails with an error message.
In my system I have as CPU 1950 Threadripper, 4 HDDs a 12 TB, Samsung 512 GB SSD as cache storage, 2 graphics cards, MSI Geforce 1650 and RTX 3070 3xOC Ventus.
I just reinstalled Unraid and so far only installed the Docker, nothing else has been done yet.

OS Install ISO: /mnt/user/isos/Win10_21H1_German_x64.iso
VirtIO Drivers ISO: /mnt/user/isos/virtio-win-0.1.189-1.iso
 Graphics Card: NVIDIA GeForce RTX 3070 (06.00.0)
 Graphics ROM BIOS: /mnt/disk1/Temps/MSI.RTX3070.8192.201012.rom

2nd Graphics Card: NVIDIA GeForce GTX 1650 (41:00:0)
2. Graphics ROM BIOS:   /mnt/disk1/Temps/MSI.GTX1650.4096.190219.rom

Sound Card: None
Network Bridge: br0
Network Model: virtio-net

When I click on "Create" I get this error message:
 
VM creation error

internal error: qemu unexpectedly closed the monitor: 2021-06-23T08:20:23.097159Z qemu-system-x86_64: -device vfio-pci,host=0000:06:00.0,id=hostdev0,bus=pci.0,addr=0x5,romfile=/mnt/disk1/Temps/MSI.RTX3070.8192.201012.rom: vfio 0000:06:00.0: group 0 is not viable
Please ensure all devices within the iommu_group are bound to their vfio bus driver.

I've been working and trying for days, but I can't really get started creating a Windows 10 VM where I can pass the 3070 through. A VM with the GPU 1650 but works....

Can you help me with this please?

 

LIBVIRT LOG:

 

 

2021-06-23 09:00:55.666+0000: 32438: error : qemuMonitorIORead:490 : Unable to read from monitor: Connection reset by peer
2021-06-23 09:00:55.666+0000: 32438: error : qemuProcessReportLogError:2097 : internal error: qemu unexpectedly closed the monitor: 2021-06-23T09:00:55.642595Z qemu-system-x86_64: -device vfio-pci,host=0000:06:00.0,id=hostdev0,bus=pci.0,addr=0x5: vfio 0000:06:00.0: group 0 is not viable

 

 

Edited June 23, 2021 by Freelancer83

[ghost82]

You need to passthrough all the components of the gpu, for sure the audio part, and maybe also usb controllers (if any).

Put that in the same bus, slot, each with a different function, in a multifunction device (N.B.: unraid fails from the vm setting gui to properly set correct addresses).

[Freelancer83]

Thanks a thousand for the reply!!!👍
What I want to ask what you mean by "bus slots" and what "unraid multifunction devices"?
And what is the best way to proceed to be able to fix the error?

[Freelancer83]

It's done, I found the solution....
Also for others who have or will have the problem:
The solution was: In the settings in the "VM Manager" the item PCIe ACS override to "Both" to click. Then save and restart the PC!

After that set up a VM!

But still many many thanks to ghost82 for the answer!👍   T H A N K S for Help!!!!

[ghost82]

1 hour ago, Freelancer83 said:

What I want to ask what you mean by "bus slots" and what "unraid multifunction devices"?
And what is the best way to proceed to be able to fix the error?

First check your iommu groups: you need all of the components of your gpu device in a iommu group, or each in its own iommu group, without other devices.

Take the following picture as an example:

 

 

Let's consider iommu group 29, which has the vga (video component) of the gpu (10de:2484) and the audio component (10de:228b)

Both the audio and vga components are in the same physical device, the gpu.

You need to passthrough all the components to make a proper passthrough, so you need to put a check (i.e. bound to vfio) both for the vga and for the audio components.

Some gpus have also usb (typically usb-c) port(s): this means that the gpu will also have an usb controller, which must also be put in passthrough mode.

Note that if you have other devices in the same iommu group different than your gpu you should use acs override to split the iommu groups.

 

Once the components are bounded to vfio set your vm in unraid (gui): select the gpu (as I think you already did), choose the vbios.rom (if you want and if it's needed) and select the audio too (choose the audio component of the gpu, see above).

 

Now switch to the xml view mode (advanced mode): you should have something like this (considering the vga and audio parts):

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x41' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/disk1/Temps/MSI.GTX1650.4096.190219.rom'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x41' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </hostdev>

Note that the addresses in <source></source> are referred to the host (your computer), and from your log the video of the 1650 is at 0000:41:00.0 (domain:bus:slot.function), the audio part is usually at 0000:41:00.1 (domain:bus:slot.function).

Note how the host manages addresses of the 2 parts: same domain (0000), same bus (41), same slot (00), different function (0 for the vga, 1 for the audio).

If you set the vm in the unraid gui it fails to properly set the target addresses: in this example (numbers may change in your case) it puts the video part at 0000:05:00.0 (domain:bus:slot.function) and the audio at 0000:06:00.0 (domain:bus:slot.function).

In other words video and audio are on different buses, which is incorrect.

You need to manually correct this, by changing the target addresses: in the case of this example, this will be:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x41' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/disk1/Temps/MSI.GTX1650.4096.190219.rom'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x41' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x1'/>
    </hostdev>

Now you can see the target addresses:

video is at 0000:05:00.0 (and it's a multifunction device)

audio si at 0000:05:00.1

 

Having audio and video on different buses may cause unexpected behaviors in the vm.

Edited June 23, 2021 by ghost82