unable to passthrough primary gpu

[Miraqulix]

Hi all,

 

I'm running unraid 6.6.6 on an AMD Threadripper, with an ASUS Motherboard. I have two identical ROG Strix RX VEGA64 graphic cards in my System. I'm trying to pass through both GPU's to different VM's. 

 

I followed the (spaceinvader one) guide on how to get the vbios / rom from the graphic card.

 

If I use the secondary card in the VM's the VM boot and use the card as aspected. But when i change one of the VM's to the primary card it won't start properly. I try to start a Windows 10 Pro VM.

 

The Unraid Log get flooded with the following message:

Jan 19 20:14:18 blackhole kernel: vfio-pci 0000:45:00.0: BAR 0: can't reserve [mem 0x80000000-0x8fffffff 64bit pref]
Jan 19 20:14:18 blackhole kernel: vfio-pci 0000:45:00.0: BAR 0: can't reserve [mem 0x80000000-0x8fffffff 64bit pref]
Jan 19 20:14:18 blackhole kernel: vfio-pci 0000:45:00.0: BAR 0: can't reserve [mem 0x80000000-0x8fffffff 64bit pref]
Jan 19 20:14:18 blackhole kernel: vfio-pci 0000:45:00.0: BAR 0: can't reserve [mem 0x80000000-0x8fffffff 64bit pref]
Jan 19 20:14:18 blackhole kernel: vfio-pci 0000:45:00.0: BAR 0: can't reserve [mem 0x80000000-0x8fffffff 64bit pref]
Jan 19 20:14:18 blackhole kernel: vfio-pci 0000:45:00.0: BAR 0: can't reserve [mem 0x80000000-0x8fffffff 64bit pref]
Jan 19 20:14:18 blackhole kernel: vfio-pci 0000:45:00.0: BAR 0: can't reserve [mem 0x80000000-0x8fffffff 64bit pref]
Jan 19 20:14:18 blackhole kernel: vfio-pci 0000:45:00.0: BAR 0: can't reserve [mem 0x80000000-0x8fffffff 64bit pref]
Jan 19 20:14:18 blackhole kernel: vfio-pci 0000:45:00.0: BAR 0: can't reserve [mem 0x80000000-0x8fffffff 64bit pref]
Jan 19 20:14:18 blackhole kernel: vfio-pci 0000:45:00.0: BAR 0: can't reserve [mem 0x80000000-0x8fffffff 64bit pref]

and the VM Log gets flooded with:

2019-01-19T19:14:18.863836Z qemu-system-x86_64: vfio_region_write(0000:45:00.0:region0+0xfbec7e8, 0x600000136408077,8) failed: Device or resource busy
2019-01-19T19:14:18.863845Z qemu-system-x86_64: vfio_region_write(0000:45:00.0:region0+0xfbec7f0, 0x600000137f09077,8) failed: Device or resource busy
2019-01-19T19:14:18.863855Z qemu-system-x86_64: vfio_region_write(0000:45:00.0:region0+0xfbec7f8, 0x60000013640a077,8) failed: Device or resource busy

 

My xml:

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>W10</name>
  <uuid>1dca3d1d-2ff0-0c58-b290-91475a991787</uuid>
  <description>Games VM</description>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>4194304</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>2</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='15'/>
    <vcpupin vcpu='1' cpuset='31'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-3.0'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/1dca3d1d-2ff0-0c58-b290-91475a991787_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='2' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/W10/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.160-1.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:f7:7c:53'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x45' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/Software/Unraid/vbios/ASUS-VEGA64-VBIOS.direct.dump'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>

At this time I'm at the stage where I get the Bios Boot Logo on the screen. But nothing else and the logs get flooded with the provided error messages. 

 

If you need more Information or have any hint's let me know. 

 

Thanks a lot and regards

Miraqulix

[Maor]

Do you run Unraid in GUI or CMD mode? Clearly you must have missed something.

 

try 

echo 0 > /sys/class/vtconsole/vtcon0/bind
echo 0 > /sys/class/vtconsole/vtcon1/bind
echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind

https://forums.unraid.net/topic/56049-video-guide-how-to-easily-passthough-a-nvidia-gpu-as-primary-without-dumping-your-own-vbios/?page=2&tab=comments#comment-644054

Edited January 19, 2019 by Maor

[Miraqulix]

He Maor,

 

thanks for the reply. I saw this post and tested it yesterday. I didn't work for me. Today I tried it again and I manage to the primary graphics card up with these commands. 

 

I'm not sure what blocked me yesterday. Maybe I had fun with the AMD reset bug or something similar. I test some more cases and hopefully can narrow down my issue from yesterday. 

 

To answer the question: I tried both GUI and cmd mode. 

Edited January 20, 2019 by Miraqulix

[fakh2006]

On 1/19/2019 at 11:55 PM, Maor said:

Do you run Unraid in GUI or CMD mode? Clearly you must have missed something.

 

try 

echo 0 > /sys/class/vtconsole/vtcon0/bind
echo 0 > /sys/class/vtconsole/vtcon1/bind
echo efi-framebuffer.0 > /sys/bus/platform/drivers/efi-framebuffer/unbind

https://forums.unraid.net/topic/56049-video-guide-how-to-easily-passthough-a-nvidia-gpu-as-primary-without-dumping-your-own-vbios/?page=2&tab=comments#comment-644054

I have a similar problem, hope someone can help, I have one card 2070 GTX and it shows in system devices but not working, i tried the above command line 1 and line 3 and started the VM and i was able to see my windows and the graphic card was recognized, when I rebooted the unraid server it went back as before ? anyone has a clue what is the cause of this. knowing that I setup the same before and it worked but it was a test run so had to reformat everything and do it all over, and I am following all the guides from spaceinvader.

[GHunter]

You have to run them on every reboot of unRaid. You can use the User Script plugin to do it for you or run them from your go file.

[fakh2006]

Thank you, but why did it work first time without scripts ? any idea ?

[Hoopster]

1 hour ago, fakh2006 said:

Thank you, but why did it work first time without scripts ? any idea ?

Because the unRAID OS loads in and runs from RAM.  Any changes you make work fine until you reboot and then are lost since the changes were only made in RAM.

 

Anything you want to be permanent and survive a reboot must be in the go file on the flash drive or run as a script through the User Scripts plugin.

Edited January 20, 2019 by Hoopster