(Solved) Another GPU Passthrough Thread

[Protheus]

Hi,

 

my Unraid-Server is now on and productive (thanks for the recent help).

 

But now i thought "why not place my plex VM onto this?" (please, i don't want to discus plex vm vs. plex docker)

So i migrated the VM from esxi to my unraid.

 

Then i said "passthrough my old NVidia GTX 560 Ti (for transcoding).

But if i boot the VM the screen remains black. If i switch to VNC output the VM is booting and plex is working.

 

System:

M/B: MSI Z97 GAMING 7 (MS-7916) Version 1.0 - s/n: To be filled by O.E.M.

BIOS: American Megatrends Inc. Version V1.12. Dated: 02/16/2016

CPU: Intel® Core™ i7-4790K CPU @ 4.00GHz

HVM: Enabled

IOMMU: Enabled

 

Unraid boot is from integrated HDMI (The Intel Xeon one in the following picture)

 

 

The Nvidia Card is in PCIe 1 Slot (but Multi-GPU Output (integrated AND the PCIe one) is activated on the board)

And Boot is set to the integrated.

 

Here is my VM Configuration:

 

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>Plex</name>
  <uuid>20ba90ef-cb70-1f00-cd24-f8a6eca5cf2f</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Debian" icon="debian.png" os="debian"/>
  </metadata>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>2</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='2'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-4.2'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/20ba90ef-cb70-1f00-cd24-f8a6eca5cf2f_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='1' threads='2'/>
    <cache mode='passthrough'/>
  </cpu>
  <clock offset='utc'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/Virtualization/_VMs/Plex/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0x13'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0x14'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
    </controller>
    <controller type='pci' index='7' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </controller>
    <controller type='pci' index='8' model='pci-bridge'>
      <model name='pci-bridge'/>
      <target chassisNr='8'/>
      <address type='pci' domain='0x0000' bus='0x07' slot='0x01' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0xa'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:b8:10:8d'/>
      <source bridge='br1.1'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>

I have no more ideas and i googled a lot of time and find way more threads to this (but mostly with more newer grafics cards)

I hope i can find some help.

 

Thanks.

Protheus

 

P.S: Forgot Unraid version: 6.8.3

 

Edited January 3, 2021 by Protheus
Forgot Version

[Protheus]

UPDATE:

 

OK, since i thought, a GTX 560 ti is a little bit weak for transcoding i bought a new Gigabyte GTX 1650 D6 OC 4G.

But the Problem still exists.

 

I've looked the two relevant SpaceInvaderOne Videos

 

and read various Posts

(most relevant):  https://mediaserver8.blogspot.com/2020/07/problems-passing-through-gpu-to-unraid.html

 

I've tried every possible Variation/Solution:

- Switched the GPU from PCIe-Slot1 to PCIe-Slot2

- Activated PCIe ACS Override Downstream/Multifunction/Both (my Bios have no clear Option for this)

(Now it looks like that)

 

 

Yes, i passed through all of the device Parts

 

I've created 4 VMs in various Configurations

OVMF-i44fx

ovmf-q35

seabios-i440fx

seabios-q35

(XML attached):

Plex-seabios-q35.xmlPlex-seabios-i440fx.xmlPlex-OVMF-q35.xmlPlex-OVMF-i440fx.xml

 

- I've set the multifunction Parameter in the XML and corrected the BUS/SLOT/FUNCTION

- I've downloaded the vbios (and corrected it like descibed in SIOs Video) and tried it with and without

 

... but nothing works. Can someone show me my tiny little mistake i did? It cant be that hard to pass through a modern GPU.

 

Here is my System:

 

M/B: MSI Z97 GAMING 7 (MS-7916) Version 1.0 - s/n: To be filled by O.E.M.

BIOS: American Megatrends Inc. Version V1.12. Dated: 02/16/2016

CPU: Intel® Core™ i7-4790K CPU @ 4.00GHz

HVM: Enabled

IOMMU: Enabled

Cache: 256 KiB, 1024 KiB, 8192 KiB

Memory: 32 GiB DDR3 (max. installable capacity 32 GiB)

Network: eth0: 1000 Mbps, full duplex, mtu 1500
 eth1: 1000 Mbps, full duplex, mtu 1500

Kernel: Linux 4.19.107-Unraid x86_64

OpenSSL: 1.1.1d

 

Please help someone

 

Greetings and a happy and healthy new Year.

Protheus

[Protheus]

UPDATE 2:

 

i found a workaround for myself -> Update to Unraid 6.9 RC2

After the Upgrade i've got an Output.

 

After that i struggled a little bit with boot from ISO (GPU passthrough blocked boot from ISO with a OVMF VM).

So i switched to VNC -> booted Veeam Linux Recovery ISO -> Recovered my ESXi Plex VM (Bare Metal) -> Switched to GPU Pasthrough -> and BINGO - Its working

After some linux driver improvements and nvenc patching my plex VM can hardware-transcode (that was one of the Goals)

 

So conclusion:

- The whole migration project came with many running (between my rooms), shouting and reading. But in the end it worked.

- After the unraid upgrade the web-gui is fast as hell and really responsive.

- I think Unraid is the best NAS System for my needs and i will stay with it (bought Pro)

 

Thanks to limetech for this nice piece of software, SpaceInvaderOne for his really understandable Videos (without that i could never realize this) and the community (Addons and some hints)

 

Greetings