Jump to content
SpaceInvaderOne

**VIDEO GUIDE** How to easily passthough a Nvidia GPU as primary without dumping your own vbios!

50 posts in this topic Last Reply

Recommended Posts

Hi, guys. I was frustrated as to why we couldn't use the techpowerup vbios files in our VMs. So I have discovered it is due 

to a header put there for later Nvidia cards which isn't compatible with KVM. (the header is for the NVflash programme)

 

So in this video, you will see how to pass through a Nvidia GPU as primary in a KVM virtual machine on unRAID by
removing the header from a vbios dumped by GPU-z/ Techpowerup making it compatible with a KVM virtual machine.


This is very useful as techpowerup.com has an extensive database of GPU vbioses. Also, should you want to dump your own previously
 you had to have 2 GPUs and 2 PCIe slots to do this. Well now you can do it using GPU-Z easily then convert it for KVM.

 

How to easily passthough a Nvidia GPU as primary without dumping your own vbios!

 

 

Edited by gridrunner

Share this post


Link to post

Awesome digging! This is going to help a lot of those with only one GPU :)

Share this post


Link to post

1)

I have a GTX560Ti  that works with my OSX VM.  It doesnt work with Windows. no matter what I throw at it including the dumped bios.

does the above mean that after removing the header of the bios file it should work with Windows VM ? (I tried the bios from GPUZ and from command line as well)

 

2)

<rom bar='on' file='/mnt/user/isos/vbios.rom'/> 

can you guys explain what exactly happens when I include the above in the XML?

does it mean that the internal bios of the GPU no longer used? 

does this action have the same impact on OSX as in Windows VMs?  is it OS agnostic and good for any type of VM ?  

 

3)

I still have some strange behaviour of this 560Ti card in the OSX.   I am using the original driver of OSX without installing nvidia webdrivers.  

when I update the OSX, sometimes the card stops to be recognized. all I need to do in that case is switch the power to the unRAID server and start it again.  full reboot incl power.

any thoughts why this happens?  any past experience like this ?

 

Share this post


Link to post

Worked perfect for me, just had to deal with some pesky unrelated iommu issues, then it all worked like magic!

Share this post


Link to post
51 minutes ago, tjb_altf4 said:

Worked perfect for me, just had to deal with some pesky unrelated iommu issues, then it all worked like magic!

Great glad it worked :D

Share this post


Link to post
23 hours ago, dadarara said:

1)

I have a GTX560Ti  that works with my OSX VM.  It doesnt work with Windows. no matter what I throw at it including the dumped bios.

does the above mean that after removing the header of the bios file it should work with Windows VM ? (I tried the bios from GPUZ and from command line as well)

 

2)

<rom bar='on' file='/mnt/user/isos/vbios.rom'/> 

can you guys explain what exactly happens when I include the above in the XML?

does it mean that the internal bios of the GPU no longer used? 

does this action have the same impact on OSX as in Windows VMs?  is it OS agnostic and good for any type of VM ?  

 

3)

I still have some strange behaviour of this 560Ti card in the OSX.   I am using the original driver of OSX without installing nvidia webdrivers.  

when I update the OSX, sometimes the card stops to be recognized. all I need to do in that case is switch the power to the unRAID server and start it again.  full reboot incl power.

any thoughts why this happens?  any past experience like this ?

 

 

 Hi @dadarara

To answer questions

1.

You say it doesn't work in windows. I am guessing that your osx might be seabios and windows ovmf (as it is by default in the templates)

Your GPU probably doesn't support UEFI so will not work with ovmf.

Now you could do one of the following

a/ Just use seabios.

b/ Look through techpowerup database and try and find a UEFI vbios dump. I have in my second server, an r9 270x which isn't UEFI compatible.

So i downloaded a vbios with UEFI in it and passed that through and now it works fine in ovmf (no need to flash the card)

 

2.

That brings me onto your second question. Yes, it uses the ROM file passed through in the XML instead of the one on the card.

The rombar=on stops the rom being used and then uses the one specified after.

But you don't need to put the rombar=on in, as it is on by default when you specify a ROM file. (that was only necessary on old versions of qemu

If no rom bar is specified, the qemu default will be used (older versions of qemu used a default of "off", while newer qemus have a default of "on")  https://libvirt.org/formatdomain.html

 

3. Not sure why you get those issues in osx. Maybe the card isn't resetting correctly hence the need to reboot the whole server.

 

So overall if it were me I would try finding an UEFI vbios and pass it through.

 

Share this post


Link to post

Awesome... I have a single GPU on my system.  I hope this works for me.  I just need to find some time to myself to be able to try it out.

 

Right now on my unRAID machine I have two monitors set up (because it used to be a desktop Linux machine).

I'd like to utilize this VM and GPU passthrough to have it act like a desktop machine once again.

Let me ask a question... right now I see a Linux console on one of the monitors.  What will happen if I follow this guide?  I'll boot unRAID and unRAID will utilize the graphics card until I start my VM?... then what... it hijacks it?

Share this post


Link to post

Yes unRAID will use the GPU for console output. However when you start the VM the VM started will then take over the GPU and use it for its output.

When the VM is shut down you unRAID will not then take it back (unless you reboot the server)  If you start the VM again or another VM using that card, the VM will again output from it.

Its not a problem for most loosing the console locally as you can manage the server from the GUI or by ssh.

Share this post


Link to post

I followed the guide.  This is what I get when I try to start my VM...

 

Execution error

internal error: qemu unexpectedly closed the monitor: 2017-04-03T14:50:39.777467Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x4,romfile=/mnt/cache/domains/vbios/EVGA.GTX650Ti.1024.120919.dump: vfio: failed to set iommu for container: Operation not permitted
2017-04-03T14:50:39.777494Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x4,romfile=/mnt/cache/domains/vbios/EVGA.GTX650Ti.1024.120919.dump: vfio: failed to setup container for group 13
2017-04-03T14:50:39.777501Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x4,romfile=/mnt/cache/domains/vbios/EVGA.GTX650Ti.1024.120919.dump: vfio: failed to get group 13
2017-04-03T14:50:39.777514Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x4,romfile=/mnt/cache/domains/vbios/EVGA.GTX650Ti.1024.120919.dump: Device initialization failed

Share this post


Link to post
1 hour ago, eric.frederich said:

I followed the guide.  This is what I get when I try to start my VM...

 

Execution error

internal error: qemu unexpectedly closed the monitor: 2017-04-03T14:50:39.777467Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x4,romfile=/mnt/cache/domains/vbios/EVGA.GTX650Ti.1024.120919.dump: vfio: failed to set iommu for container: Operation not permitted
2017-04-03T14:50:39.777494Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x4,romfile=/mnt/cache/domains/vbios/EVGA.GTX650Ti.1024.120919.dump: vfio: failed to setup container for group 13
2017-04-03T14:50:39.777501Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x4,romfile=/mnt/cache/domains/vbios/EVGA.GTX650Ti.1024.120919.dump: vfio: failed to get group 13
2017-04-03T14:50:39.777514Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x4,romfile=/mnt/cache/domains/vbios/EVGA.GTX650Ti.1024.120919.dump: Device initialization failed

 

seems like you have a problem with your iommu groups. Please post your pci devices and iommu groups. I guess you have something in group 13 other than your GPU.

Share this post


Link to post

From "tools" -> "system devices"

 

IOMMU group 0
    [1002:5a14] 00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD9x0/RX980 Host Bridge (rev 02)
IOMMU group 1
    [1002:5a16] 00:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890/RD9x0/RX980 PCI to PCI bridge (PCI Express GFX port 0)
IOMMU group 2
    [1002:4390] 00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [IDE mode] (rev 40)
IOMMU group 3
    [1002:4397] 00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
    [1002:4396] 00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
IOMMU group 4
    [1002:4397] 00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
    [1002:4396] 00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
IOMMU group 5
    [1002:4385] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller (rev 42)
IOMMU group 6
    [1002:439c] 00:14.1 IDE interface: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 IDE Controller (rev 40)
IOMMU group 7
    [1002:4383] 00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia (Intel HDA) (rev 40)
IOMMU group 8
    [1002:439d] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller (rev 40)
IOMMU group 9
    [1002:4384] 00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge (rev 40)
IOMMU group 10
    [1002:4399] 00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller
IOMMU group 11
    [1002:43a0] 00:15.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 0)
    [1002:43a1] 00:15.1 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 1)
    [1002:43a2] 00:15.2 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB900 PCI to PCI bridge (PCIE port 2)
    [1b21:1042] 04:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller
    [10ec:8168] 05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 07)
IOMMU group 12
    [1002:4397] 00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller
    [1002:4396] 00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller
IOMMU group 13
    [10de:11c6] 01:00.0 VGA compatible controller: NVIDIA Corporation GK106 [GeForce GTX 650 Ti] (rev a1)
    [10de:0e0b] 01:00.1 Audio device: NVIDIA Corporation GK106 HDMI Audio Controller (rev a1)

 

Share this post


Link to post

I have the exact same problem at this very moment. Just like yours my graphics audio controller is showing up in the same IOMMU group as the graphics card.

Share this post


Link to post

Does that not normally happen?  Do you normally see them in separate groups?  I think I see them in the same group in videos online and it's working.

Share this post


Link to post
59 minutes ago, eric.frederich said:

Does that not normally happen?  Do you normally see them in separate groups?  I think I see them in the same group in videos online and it's working.

Yes its fine to have the audio and video in the same group as the are the same device.

Just make sure you pass through both the video and audio part in the vm.

 

 

Try changing your sys linux config file to add vfio_iommu_type1.allow_unsafe_interrupts=1

 

after the append part

 

so the config file it would look like this.

default /syslinux/menu.c32
menu title Lime Technology, Inc.
prompt 0
timeout 50
label unRAID OS
  menu default
  kernel /bzimage
append vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot  
label unRAID OS GUI Mode
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui
label unRAID OS Safe Mode (no plugins, no GUI)
  kernel /bzimage
  append initrd=/bzroot unraidsafemode
label Memtest86+
  kernel /memtest

Share this post


Link to post
13 hours ago, gridrunner said:

Try changing your sys linux config file to add vfio_iommu_type1.allow_unsafe_interrupts=1

 

 

This seemed to work in that I can now launch the VM.

It took over the console that was displayed and I even saw my Linux Mint startup splash screen.

After that I got some X errors and was unable to do anything graphically, it didn't fall back to a terminal either.

 

I figured maybe when I installed the VM under VNC it didn't know about the graphics card, so I have tried to set up a new VM with the graphics card there the whole time.

Now again (after a reboot) it takes over the unRAID terminal but instead I see nothing.  I log in from another machine via ssh and I see some qemu process taking 100% cpu.  Nothing happens, I have waited a while.

 

Maybe I should try Ubuntu instead of Mint since it's more mainstream?  Maybe I should install the OS first under VNC then switch?  What should I do?

Share this post


Link to post
47 minutes ago, eric.frederich said:

 

This seemed to work in that I can now launch the VM.

It took over the console that was displayed and I even saw my Linux Mint startup splash screen.

After that I got some X errors and was unable to do anything graphically, it didn't fall back to a terminal either.

 

I figured maybe when I installed the VM under VNC it didn't know about the graphics card, so I have tried to set up a new VM with the graphics card there the whole time.

Now again (after a reboot) it takes over the unRAID terminal but instead I see nothing.  I log in from another machine via ssh and I see some qemu process taking 100% cpu.  Nothing happens, I have waited a while.

 

Maybe I should try Ubuntu instead of Mint since it's more mainstream?  Maybe I should install the OS first under VNC then switch?  What should I do?

 

The linux flavour will not make a difference. But try using seabios instead of OVMF or vice versa.

Also try machine type i440 and q35.

Failing that I am not sure what else that you can try. Try dropping limetech support an email they may be able to suggest a solution :D

Share this post


Link to post

This worked great for me with two GTX 1080s! I was able to previously pass through only the one that was not in the top PCI-E slot but now I can use both on the same machine and power my monitor on the primary one (which before starting the VM displays the UNRAID terminal). Thank you!

Share this post


Link to post

Doen't work for me, what did i wrong ? 

 

    <hostdev mode='subsystem' type='pci' managed='yes' xvga='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/ISOs/980ti.dump'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>

 

Bildschirmfoto 2017-04-14 um 23.31.16.png

 

 

Now it works for me too, i only need to wait like 3-5 min. after starting the VM

 

Edited by unknown
Now it works

Share this post


Link to post
10 hours ago, unknown said:

Doen't work for me, what did i wrong ? 

 


    <hostdev mode='subsystem' type='pci' managed='yes' xvga='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/ISOs/980ti.dump'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>

 

Bildschirmfoto 2017-04-14 um 23.31.16.png

 

 

Now it works for me too, i only need to wait like 3-5 min. after starting the VM

 

 

Wow, so you are having to wait that long for boot up. It shouldn't take so long

You may try installing the tips and tweaks plugin and set your CPU governor to performance and the intel turbo boost to enable.

(also other useful tweaks there too) and see if that may help

Share this post


Link to post
5 hours ago, unknown said:

Intel Turbo is enable. Where can i find there tweak plugins or what do i have to change ?

You can download a plugin called tips and tweaks and make some changes there too.  Download from community applications.

58f2339ba1b15_ScreenShot2017-04-15at15_50_13.png.bcf941119f84309cc91cf1afa15a5290.png

Share this post


Link to post
On 4/4/2017 at 10:58 AM, gridrunner said:

The linux flavour will not make a difference. But try using seabios instead of OVMF or vice versa.

Also try machine type i440 and q35.

 

Sorry, been away on vacation.  I switched to seabios and it seemed to work.  I installed Linux Mint and have both monitors working now.

Of course now it's easter I need to leave the house for the day, no time to play with it.

 

I did it with both the graphics and HDMI audio since I heard you have to have everything within the IOMMU group mapped in.  I'd like to try to use my onboard audio since my monitors don't have a line-out for the HDMI.  Will this stop working if I don't have the HDMI audio mapped in?... Is it possible to have both onboard and HDMI mapped into the VM?  The configuration page is a drop-down list so I can only select one, but maybe by editing the XML itself?

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.