Cant Start VM with GPU, IOMMU problems

[Togay]

I get this Message every time when i start the VM with the GPU. I tried everything, making a new unraid Stick, changing the GPU to another Pcie Slot, BIOS Clean, BIOS Update, changing IOMMU Settings. I Attached the Diagnostics. The CPU and Mainboard are Supporting VT-D and VT-X

 

Setup:

Xeon X5690

Nvidia GTX 660

MSI X58 Pro-e Mainboard

24Gig Ram

tower-diagnostics-20220308-2330.zip

[ghost82]

vfio_iommu_type1_attach_group: No interrupt remapping support.  Use the module param "allow_unsafe_interrupts" to enable VFIO IOMMU support on this platform

So I would try to add to syslinux config:

vfio_iommu_type1.allow_unsafe_interrupts=1

 

--->

For unraid without gui it becomes:

append vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot

 

It is also recommended to attach to vfio at boot iommu groups 14 and 15, gpu video and audio.

[Togay]

I did what you said and also made an new Unraid Stick and now it wont even Boot anymore...

[ghost82]

On 3/10/2022 at 10:34 PM, Togay said:

and now it wont even Boot anymore

Allow unsafe interrupts should not prevent the os to load...

 

On 3/10/2022 at 10:34 PM, Togay said:

and also made an new Unraid Stick

Maybe you messed with uefi/legacy bios?

Check the mode unraid is booting (uefi vs legacy bios) and check the bios options (csm disabled vs csm enabled).

[Ladousek]

Hello all, 

Can I ask you for help? I cannot put my graphic card in to one IOMMU group to be able pass through it to VM. VGA and sound controller are still in other groups. I tried setup in "PCIe ACS override" all options (downstream, both, multi-function) and graphic card is still in 2 groups, and I am not able to put it on VM.

 

I tried also code it in Syslinux Configuration as pcie_acs_override=id:1002:743f and no influence. 

Do you have any idea?

 

[ghost82]

12 hours ago, Ladousek said:

Can I ask you for help? I cannot put my graphic card in to one IOMMU group to be able pass through it to VM. VGA and sound controller are still in other groups.

It's ok, what you don't want is to have other devices you don't want to passthrough in the same iommu group as your gpu video or gpu audio.

So these cases are ok:

1. gpu audio and video are in the same iommu group, without any other device that you don't want to passthrough to the vm

2. gpu audio and video are in different iommu groups, without any other device that you don't want to passthrough to the vm

3. gpu audio and video are in the same iommu group, together with other devices that you want to passthrough to the vm

4. gpu audio and video are in different iommu groups, together with other devices that you want to passthrough to the vm

 

In your case (case: 2) simply bound to vfio iommu groups 17 and 18.

 

12 hours ago, Ladousek said:

I tried also code it in Syslinux Configuration as pcie_acs_override=id:1002:743f

Do not use syslinux config. use only unraid panel.

[Ladousek]

9 hours ago, ghost82 said:

In your case (case: 2) simply bound to vfio iommu groups 17 and 18.

 

 

I tried but still VM does not see graphic card. Could you take a look at my config?

• Adding Graphic and sound card

• Binding 2 groups

• Setup

• WIN 10 sees Basic Graphic adapter only

[ghost82]

2 minutes ago, Ladousek said:

Could you take a look at my config?

Sure but you need to attach diagnostics, did you install the graphics driver inside the vm?

[Ladousek]

I am sending diagnostics.

Drivers are not installed because at VNC mode "no hardware found" so I cannot. When I switch to AMD at VM setup I cannot try to install AMD drivers because probably due to limited capacity of basic adapter I cannot run installer. The whole screen gets black every time.

tower-diagnostics-20220530-1819.zip

[ghost82]

13 hours ago, Ladousek said:

Sorry to ask, but are you sure that this graphic adapter is the qxl/vnc one?

I'm asking for 2 reasons:

1. in your Windows 10 vm there is only the amd gpu defined, no qxl/vnc

2. when you define qxl/vnc virtual gpu it shouldn't appear with the yellow mark on it

 

Every passed through gpu without its drivers will appear as a basic graphic microsoft adapter because the microsoft driver is attached to it: this can work as a basic adapter or not (as in your case); if it doesn't work you should install the drivers of that gpu.

 

Double check the property of that graphic adapter and see what it is and why it's not running properly, what error outputs?

 

Gpu is isolated (OK) and in its iommu groups (OK), it seems not in use by the host (OK), if it's still doesn't work, change the settings in the xml to it becomes a multifunction device, from this:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x28' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x28' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>

to this:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x28' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x28' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x1'/>
    </hostdev>

 

You may need to pass also the vbios, dump it and pass the rom file too.

 

You may need to change (re-create) the machine type, from i440fx to pc-q35 for better passthrough compatibility.

[Ladousek]

Hi, I installed todays unraid update, implemented your code and now I could install GPU drivers 🙂 Thank you very much for your help.

Win are working fine now. Next step I try to run macos with graphic. 

 

[ghost82]

7 minutes ago, Ladousek said:

Next step I try to run macos with graphic. 

Note that this should be a new 6500 xt and unfortunately there is still no driver in mac os, so it wont work. Maybe in future versions of mac os the driver will be added.