[[SOLVED] 2060 super single GPU passthrough troubleshooting]

Unraid 6.8.3

Motherboard: MSI B450M Mortar Max

CPU: AMD 3600x

GPU: Nvidia 2060 super

RAM: 64GB

 

I'm trying to passthrough a single GPU to a Windows 10 guest. I tried every possibility solution I can find on the internet. I think I'm pretty near to success but there is something that I still can't fix.

 

First, I enabled IOMMU and SVM in BIOS and Checked in Unraid system information that.

HVM: Enabled
IOMMU: Enabled

After setting PCIe ACS override to multi function, the IOMMU group for my gpu is now, 

(Downstream makes my unraid unbootable. I don't know why.)

IOMMU group 23:	[10de:1f06] 26:00.0 VGA compatible controller: NVIDIA Corporation TU106 [GeForce RTX 2060 SUPER] (rev a1)
IOMMU group 24:	[10de:10f9] 26:00.1 Audio device: NVIDIA Corporation TU106 High Definition Audio Controller (rev a1)
IOMMU group 25:	[10de:1ada] 26:00.2 USB controller: NVIDIA Corporation TU106 USB 3.1 Host Controller (rev a1)
IOMMU group 26:	[10de:1adb] 26:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU106 USB Type-C UCSI Controller (rev a1)

To ensure that GPU is not occupied by Unraid, I inserted pci-stub into syslinux: 

kernel /bzimage
append pcie_acs_override=multifunction isolcpus=0-3,6-9 pci-stub.ids=10de:1f06,10de:10f9,10de:1ada,10de:1adb initrd=/bzroot

I tried to download a 2060 super rom and remove the header using hex editor.  The configure of a new Windows 10 machine. 

Set the XML to multifunction to 'on' and change the virtual devices to the same slot with corresponding functions.

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x26' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/isos/2060s.rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x26' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x26' slot='0x00' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x2'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x26' slot='0x00' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x3'/>
    </hostdev>

After all these work, I login Windows 10 from server from RDP and installed the device driver. Windows is detecting the device. 

However, Nvidia driver is not detecting the gpu! Pretty wired!

 

Here are some error informations the might be useful.

VM-> Windows 10 -> Log I can get Qemu log with full of errors like,

2020-06-04T15:14:05.725314Z qemu-system-x86_64: vhost_region_add_section:Section rounded to 92206000 prior to previous 92206010
2020-06-04T15:14:05.726742Z qemu-system-x86_64: vhost_region_add_section:Section rounded to 92206000 prior to previous 92206010
2020-06-04T15:14:05.727011Z qemu-system-x86_64: vhost_region_add_section:Section rounded to 92206000 prior to previous 92206010
2020-06-04T15:14:05.727345Z qemu-system-x86_64: vhost_region_add_section:Section rounded to 92206000 prior to previous 92206010
2020-06-04T15:14:05.727888Z qemu-system-x86_64: vhost_region_add_section:Section rounded to 92206000 prior to previous 92206010
2020-06-04T15:14:05.728086Z qemu-system-x86_64: vhost_region_add_section:Section rounded to 92206000 prior to previous 92206010
2020-06-04T15:14:05.728246Z qemu-system-x86_64: vhost_region_add_section:Section rounded to 92206000 prior to previous 92206010

In Windows 10 Devices manager there is an unknown device and virtio-win does not contain its driver.

 

I have worked three days on passing through the GPU. Any advice/idea is appreciated. 

If you need detailed information please tell me.

Thanks in advance.

 

~~~~~~

I think I have solved this problem by buying another graphic card for Unraid.