GPU Passthrough in Unraid not working after driver install

[AKSAdrian]

Hello,

 

I am trying to get GPU passthrough to work. The Windows 10 VM works with no issues up until I install the Nvidia drivers, the system crashes during install and then goes into a boot loop until it eventually loads recovery. If I switch the GPU back to VNC, the system boots normally and i am able to remove the driver.

 

I'm running Unraid version 6.9.2, the reason for this is I thought maybe this would work better on an older version of Unraid since every YouTube video about it they are running something like 6.7 or 6.8 but unforturntly these versions aren't downloadable anymore so I can't try it. I did try this on the latest version 6.10.3 but exactly the same problem.

 

My hardware is as follows: Motherboard: Gigabyte X99-UD3P-CF, CPU: Intel Xeon E5-2620 V3, Memory: 96GB (3x32gb), GPU 1: GTX 650 Ti (Old and terrible but this is just to boot Unraid), GPU 2: GT 1030 (This is what I am trying to passthrough to Windows.), GPU 3: GT 1030 (I was hoping to have 2 VMs running.).

Intel VT-D is enabled in the BIOS.

 

On the Windows 10 VM Event Viewer tells me nothing helpful, it's just logged the crashes and unexpected shutdowns with no useful information.

I've also tried older driver versions because apperently the newer ones refuse to start up if it detects that it's running in a virtulized enviroment, but this didn't change anything either.

 

I have tried many different configurations for the VM. 

I tried it with Hyper-V set to No and changing the maching type to Q35. I've also tried different GPU rom files which didn't solve anything.

 

I've also gone into system devices, selected the GPU and it's audio and binded it to VFIO.

 

Hopefully everything I've wrote here made sense, any help with this would be much appreciated because I just don't know what else to do.

Thanks.

[ghost82]

Hi,

you need to setup the gpu as multifunction in the vm, it should be isolated (bound to vfio), allow unsafe interrupts should be enabled in unraid, newest drivers should be installed, modification to syslinux config may be required (es: video=efifb:off), q35+ovmf should be preferred, video rom should be dumped from your gpu and not downloaded somewhere.

 

5 hours ago, AKSAdrian said:

I've also tried older driver versions because apperently the newer ones refuse to start up if it detects that it's running in a virtulized enviroment

It is the opposite, from v465 drivers should work as expected inside a vm, so the newest drivers should be preferred.

 

diagnostics may help to identify the issue.

[AKSAdrian]

6 hours ago, ghost82 said:

Hi,

you need to setup the gpu as multifunction in the vm, it should be isolated (bound to vfio), allow unsafe interrupts should be enabled in unraid, newest drivers should be installed, modification to syslinux config may be required (es: video=efifb:off), q35+ovmf should be preferred, video rom should be dumped from your gpu and not downloaded somewhere.

 

It is the opposite, from v465 drivers should work as expected inside a vm, so the newest drivers should be preferred.

 

diagnostics may help to identify the issue.

 

Hello,

 

It looks like it still doesn't want to work. I've done everything you said and it still crashes while installing the latest drivers from Nvidia and going into a boot loop. 

 

I have set the GPU as multifunction in the xml for the VM, here is what I changed it to, let me know if this is correct.

<hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/isos/vbios/gt 1030.rom'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>

 

The GPU is bounnd to vfio.

I dumped the video rom from my GPU using this video.

I enabled allow unsafe interrupts.

I modified the syslinux config, I have also pasted that below for you to check out.

  kernel /bzimage
  append vfio_iommu_type1.allow_unsafe_interrupts=1 pcie_acs_override=downstream,multifunction initrd=/bzroot video=efifb:off

 

I have also attached a diagnostics zip file.

 

Thanks for taking the time to help me out, it means a lot! Let me know what I should do next.

tower-diagnostics-20220624-0517.zip

[ghost82]

2 hours ago, AKSAdrian said:

I have set the GPU as multifunction in the xml for the VM

Multifunction is wrong, the audio must be in the same bus, same slot, different function, so change to this:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/isos/vbios/gt 1030.rom'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x1'/>
    </hostdev>

 

Try..other than that I could think about the vbios, can you share it to have a look?

Edited June 24, 2022 by ghost82

[AKSAdrian]

11 hours ago, ghost82 said:

Multifunction is wrong, the audio must be in the same bus, same slot, different function, so change to this:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/isos/vbios/gt 1030.rom'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x1'/>
    </hostdev>

 

Try..other than that I could think about the vbios, can you share it to have a look?

 

Hello,

 

I changed the multifunction to what you sent above and it hasn't changed anything, it still crashes while installing the driver.

I have attached the vbios for you.

 

Thanks.

 

gt 1030.rom