Unable to pass-through graphics board to Win10-VM

[ghost82]

3 minutes ago, Nebrius said:

What does pci=nommconf do?

It disables memory mapped PCI configuration registers, so the kernel can't read these registers.

 

6 minutes ago, Nebrius said:

What do I need this for or can I leave it out?

I don't know, you added it manually, not me

 

Usually added for bugged bioses/bugged pcie devices.

 

If your system runs well, without any flooding error in the logs you are good without it.

[Nebrius]

Ok, thank you.

 

I want to try, if the gpu is also working in the former pci slot. Therefore I have to adjust the bus addresses in the xml file. As I understood this information is in the <hostdev></hostdev> part of the xml, in detail in the <address type='pci' domain='0x0000' bus='0x0X' ....> tag, is it? So I only would have to change the number by "bus"? The current working xml (2. pci slot) looks like this:

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/isos/vbios/Geforce_RTX2070.rom'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x04' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x04' slot='0x00' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x2'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x04' slot='0x00' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x3'/>
    </hostdev>    
    

 

Based on our conversation here, I adjusted the xml for working in the 1. pci slot:

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/isos/vbios/Geforce_RTX2070.rom'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x04' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x04' slot='0x00' function='0x2'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x00' function='0x2'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x04' slot='0x00' function='0x3'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x3'/>
    </hostdev>    
    

 

Can I give it a try? Am i on the right track?

 

 

 

 

 

 

 

[ghost82]

34 minutes ago, Nebrius said:

As I understood this information is in the <hostdev></hostdev> part of the xml

There are two addresses for each component of the gpu, one is the source address, the other is the target address.

Your gpu has 4 components, video, audio, usb controller and serial bus controller.

Each component is inside a hostdev block.

Let's take as an example this hostdev block:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/isos/vbios/Geforce_RTX2070.rom'/>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>

 

This is the block of the video component of the gpu.

The source address:

source address, specified inside the <source></source> block is the address of the video component seen by unraid.

Where you can find it?

For example with the lspci command, or in unraid diagnostics in the /system/lspci.txt file, or even in the unraid gui.

In one of your diagnostics you had in the lspci.txt file:

 

04:00.0 VGA compatible controller [0300]: NVIDIA Corporation TU106 [GeForce RTX 2070] [10de:1f02] (rev a1)
	Subsystem: Gigabyte Technology Co., Ltd Device [1458:3fda]
	Kernel driver in use: vfio-pci
04:00.1 Audio device [0403]: NVIDIA Corporation TU106 High Definition Audio Controller [10de:10f9] (rev a1)
	Subsystem: Gigabyte Technology Co., Ltd Device [1458:3fda]
	Kernel driver in use: vfio-pci
04:00.2 USB controller [0c03]: NVIDIA Corporation TU106 USB 3.1 Host Controller [10de:1ada] (rev a1)
	Subsystem: Gigabyte Technology Co., Ltd Device [1458:3fda]
	Kernel driver in use: vfio-pci
04:00.3 Serial bus controller [0c80]: NVIDIA Corporation TU106 USB Type-C UCSI Controller [10de:1adb] (rev a1)
	Subsystem: Gigabyte Technology Co., Ltd Device [1458:3fda]
	Kernel driver in use: vfio-pci

 

VGA: video component of the gpu; its source address is 04:00.0 (bus:slot.function), meaning the video component of the gpu is at the source address bus=0x04, slot=0x00, function=0x00.

Audio device: audio component of the gpu; its source address is 04:00.1 (bus:slot.function), meaning the audio component of the gpu is at the source address bus=0x04, slot=0x00, function=0x01.

USB controller: usb controller component of the gpu; its source address is 04:00.2 (bus:slot.function), meaning the usb controller component of the gpu is at the source address bus=0x04, slot=0x00, function=0x02.

Serial bus controller: serial bus controller component of the gpu; its source address is 04:00.3 (bus:slot.function), meaning the serial bus controller component of the gpu is at the source address bus=0x04, slot=0x00, function=0x03.

 

Back to the xml code, as you can see, for the video component you set the source address to this:

      <source>
        <address domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
      </source>

which is correct.

 

Target address:

target address, specified outside the <source></source> block is the address of the video component seen by the vm.

 

      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0' multifunction='on'/>

 

In this example in that block you are saying: unraid, pick the pcie device that you have at 04:00.0, attach it to vfio driver and put it in a vm at address 05:00.0.

 

Note that in the target address line you have also multifunction='on': it means that the whole gpu is a multifunction device made of different components.

A multifunction device is a device with different components, each having the same bus, same slot and different function.

 

The source device (gpu):

04:00.0

04:00.1

04:00.2

04:00.3

 

is multifunction, all the components are at bus 0x04, slot 0x00, but each has a different function, 0x00, 0x01, 0x02, 0x03.

 

So, in the xml, you specified multifunction='on' only in the video component, and you change bus/slot/function of the other components so to have the same bus, same slot and different function.

 

In the target vm the components of the gpu are at:

05:00.0

05:00.1

05:00.2

05:00.3

 

In other words the source-->target map is:

04:00.0 --> 05:00.0

04:00.1 --> 05:00.1

04:00.2 --> 05:00.2

04:00.3 --> 05:00.3

Edited February 2, 2022 by ghost82

[ghost82]

30 minutes ago, Nebrius said:

Can I give it a try? Am i on the right track?

All this to say...no it's wrong

Understand the logic behind this and it will be easy! I had issues too in understanding this at the beginning.

 

Starting from the working xml, all you have to do is to change all the sources addresses, which will probably be:

01:00.0, 01:00.1, 01:00.2, 01:00.3 instead of 04:00.0, 04:00.1, 04:00.2, 04:00.3

Edited February 2, 2022 by ghost82

[Nebrius]

Thank you very much for the wonderful explanation, now even I understand. With the knowledge I have my graphics card now also run at desired slot. 😍
How can I show my appreciation?

[ghost82]

happy to read that you solved. A thank you, which you already said, is more than enough for me.