Primary RTX 3080 does not appear in the VM's graphic card dropdown (GPU passthrough)

[Bearbite]

Hi all,

 

I'm new with Unraid, and I love it, but I have a problem with my primary GPU.

 

I have a server with 2 GPUS. Primary is the RTX 3080 secondary is a GTX 1650. Unraid starts well without GUI mode, everything is fine, but in the VM's dropdown I can select only the GTX 1650. The RTX 3080 does not appear in the list. --I used the 1st PCIe X16 slot for the RTX 3080 and the card is working well if I connect a monitor directly it, I can see the Unraid's command line messages. 

 

So I tried to list my devices in the tools / system device but I see only groups to ID 30. IOMMU group 30. BUT...

 

When I run the lspci -nn from the command line, i see AND I CAN SEE my card with group ID 43.00.0:

 

43:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA102 [GeForce RTX 3080] [10de:2206] (rev a1)
43:00.1 Audio device [0403]: NVIDIA Corporation GA102 High Definition Audio Controller [10de:1aef] (rev a1)

 

I read many forum topics but non of them helped.

 

My Setup:

 

Asus Zenith Extreme mobo

Threadripper 2950x

96Gb RAM

Primary: ASUS RTX 3080 (EKWB)

Secondary: GTX 1650

5,5TB m2. SSD

1 TB SSD

4TB HDD

 

Can anybody help me? What did I wrong?

 

Thank you,

[ghost82]

If you know the source address of the gpu (and you know it: 43:00.0 and 43:00.1) you can always set it manually in the advanced xml view:

 

      <hostdev mode='subsystem' type='pci' managed='yes'>
        <driver name='vfio'/>
        <source>
          <address domain='0x0000' bus='0x043' slot='0x00' function='0x0'/>
        </source>
        <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0' multifunction='on'/>
      </hostdev>
      <hostdev mode='subsystem' type='pci' managed='yes'>
        <driver name='vfio'/>
        <source>
          <address domain='0x0000' bus='0x043' slot='0x00' function='0x1'/>
        </source>
        <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x1'/>
      </hostdev>

 

Note that the target addresses 03:00.0 and 03:00.1 are examples, you need a pci-root-port with 'index=3', and no device attached to bus 3.

Edited February 10, 2022 by ghost82

[Squid]

Reason that you're not seeing it is because Unraid has already grabbed it for it's own purposes.  You need to isolate it from the OS (Tools - System Devices) to easily pass it through.  (And set the 1650 to be the primary in the BIOS so that Unraid uses it instead)

[Bearbite]

Thank you @ghost82 for your fast reply. How can I use this XML? Where can I set these values? From the admin side? Thx

[ghost82]

Follow squid comment, attach to vfio at boot the 3080, either set the other gpu as primary and/or be prepared to use an external device to connect via ethernet to unraid to manage it.

Once isolated as he said, you should see it in the dropdown menu.

[Bearbite]

3 minutes ago, Squid said:

Reason that you're not seeing it is because Unraid has already grabbed it for it's own purposes.  You need to isolate it from the OS (Tools - System Devices) to easily pass it through.  (And set the 1650 to be the primary in the BIOS so that Unraid uses it instead)

 

Thank you, but the RTX 3080 does not in the list under Tools - System Devices. I can see only the GTX 1650.

And another question. how can you set (change) the primary GPU in the Bios? I don't remember the settings for it.

[Bearbite]

So here is the short list:

 

- My BIOS can see my RTX 3080

- The RTX 3080 works well because I see terminal when and after the Unraid OS started

- The Unraid's os can see it within the the terminal after the command lspci -nn 

- The Dropdown in the VM GPU passthrough section not listed it on the WEB GUI

- The Tools / System devices not listed the RTX 3080 only the GTX 1650 on the web GUI

 

[Squid]

3 minutes ago, Bearbite said:

- The Tools / System devices not listed the RTX 3080 only the GTX 1650 on the web GUI

 

Post your diagnostics

[Bearbite]

Thank you Squid I attached it.

bishop-diagnostics-20220210-1824.zip

[ghost82]

You should have something like this in your bios:

Advanced/AMD PBS/Enumerate all IOMMU in IVRS

 

Change that setting (ENABLE IT) and see if all iommu groups are enumerated

 

(image taken from internet)

 

As you can see, if it's disabled, only devices attached to the primary cpu die will be mapped.

If enabled, devices attached to both dies should be enumerated, in fact now, your iommu groups lack some more devices, not only that gpu.

Edited February 11, 2022 by ghost82

[Bearbite]

Wooow, thank you @ghost82 You are my hero Yes, this was the problem. Sorry, I didn't know this feature of my BIOS.

I set it to enable and all of my devices appears under tools / system devices.

Thank you again