March 26, 20179 yr Hi, guys. I was frustrated as to why we couldn't use the techpowerup vbios files in our VMs. So I have discovered it is due to a header put there for later Nvidia cards which isn't compatible with KVM. (the header is for the NVflash programme) So in this video, you will see how to pass through a Nvidia GPU as primary in a KVM virtual machine on unRAID by removing the header from a vbios dumped by GPU-z/ Techpowerup making it compatible with a KVM virtual machine. This is very useful as techpowerup.com has an extensive database of GPU vbioses. Also, should you want to dump your own previously you had to have 2 GPUs and 2 PCIe slots to do this. Well now you can do it using GPU-Z easily then convert it for KVM. How to easily passthough a Nvidia GPU as primary without dumping your own vbios! Edited March 26, 20179 yr by gridrunner
March 26, 20179 yr 1) I have a GTX560Ti that works with my OSX VM. It doesnt work with Windows. no matter what I throw at it including the dumped bios. does the above mean that after removing the header of the bios file it should work with Windows VM ? (I tried the bios from GPUZ and from command line as well) 2) <rom bar='on' file='/mnt/user/isos/vbios.rom'/> can you guys explain what exactly happens when I include the above in the XML? does it mean that the internal bios of the GPU no longer used? does this action have the same impact on OSX as in Windows VMs? is it OS agnostic and good for any type of VM ? 3) I still have some strange behaviour of this 560Ti card in the OSX. I am using the original driver of OSX without installing nvidia webdrivers. when I update the OSX, sometimes the card stops to be recognized. all I need to do in that case is switch the power to the unRAID server and start it again. full reboot incl power. any thoughts why this happens? any past experience like this ?
March 27, 20179 yr Worked perfect for me, just had to deal with some pesky unrelated iommu issues, then it all worked like magic!
March 27, 20179 yr Author 51 minutes ago, tjb_altf4 said: Worked perfect for me, just had to deal with some pesky unrelated iommu issues, then it all worked like magic! Great glad it worked
March 27, 20179 yr Author 23 hours ago, dadarara said: 1) I have a GTX560Ti that works with my OSX VM. It doesnt work with Windows. no matter what I throw at it including the dumped bios. does the above mean that after removing the header of the bios file it should work with Windows VM ? (I tried the bios from GPUZ and from command line as well) 2) <rom bar='on' file='/mnt/user/isos/vbios.rom'/> can you guys explain what exactly happens when I include the above in the XML? does it mean that the internal bios of the GPU no longer used? does this action have the same impact on OSX as in Windows VMs? is it OS agnostic and good for any type of VM ? 3) I still have some strange behaviour of this 560Ti card in the OSX. I am using the original driver of OSX without installing nvidia webdrivers. when I update the OSX, sometimes the card stops to be recognized. all I need to do in that case is switch the power to the unRAID server and start it again. full reboot incl power. any thoughts why this happens? any past experience like this ? Hi @dadarara To answer questions 1. You say it doesn't work in windows. I am guessing that your osx might be seabios and windows ovmf (as it is by default in the templates) Your GPU probably doesn't support UEFI so will not work with ovmf. Now you could do one of the following a/ Just use seabios. b/ Look through techpowerup database and try and find a UEFI vbios dump. I have in my second server, an r9 270x which isn't UEFI compatible. So i downloaded a vbios with UEFI in it and passed that through and now it works fine in ovmf (no need to flash the card) 2. That brings me onto your second question. Yes, it uses the ROM file passed through in the XML instead of the one on the card. The rombar=on stops the rom being used and then uses the one specified after. But you don't need to put the rombar=on in, as it is on by default when you specify a ROM file. (that was only necessary on old versions of qemu If no rom bar is specified, the qemu default will be used (older versions of qemu used a default of "off", while newer qemus have a default of "on") https://libvirt.org/formatdomain.html 3. Not sure why you get those issues in osx. Maybe the card isn't resetting correctly hence the need to reboot the whole server. So overall if it were me I would try finding an UEFI vbios and pass it through.
April 3, 20179 yr Awesome... I have a single GPU on my system. I hope this works for me. I just need to find some time to myself to be able to try it out. Right now on my unRAID machine I have two monitors set up (because it used to be a desktop Linux machine). I'd like to utilize this VM and GPU passthrough to have it act like a desktop machine once again. Let me ask a question... right now I see a Linux console on one of the monitors. What will happen if I follow this guide? I'll boot unRAID and unRAID will utilize the graphics card until I start my VM?... then what... it hijacks it?
April 3, 20179 yr Author Yes unRAID will use the GPU for console output. However when you start the VM the VM started will then take over the GPU and use it for its output. When the VM is shut down you unRAID will not then take it back (unless you reboot the server) If you start the VM again or another VM using that card, the VM will again output from it. Its not a problem for most loosing the console locally as you can manage the server from the GUI or by ssh.
April 3, 20179 yr I followed the guide. This is what I get when I try to start my VM... Execution error internal error: qemu unexpectedly closed the monitor: 2017-04-03T14:50:39.777467Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x4,romfile=/mnt/cache/domains/vbios/EVGA.GTX650Ti.1024.120919.dump: vfio: failed to set iommu for container: Operation not permitted 2017-04-03T14:50:39.777494Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x4,romfile=/mnt/cache/domains/vbios/EVGA.GTX650Ti.1024.120919.dump: vfio: failed to setup container for group 13 2017-04-03T14:50:39.777501Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x4,romfile=/mnt/cache/domains/vbios/EVGA.GTX650Ti.1024.120919.dump: vfio: failed to get group 13 2017-04-03T14:50:39.777514Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x4,romfile=/mnt/cache/domains/vbios/EVGA.GTX650Ti.1024.120919.dump: Device initialization failed
April 3, 20179 yr Author 1 hour ago, eric.frederich said: I followed the guide. This is what I get when I try to start my VM... Execution error internal error: qemu unexpectedly closed the monitor: 2017-04-03T14:50:39.777467Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x4,romfile=/mnt/cache/domains/vbios/EVGA.GTX650Ti.1024.120919.dump: vfio: failed to set iommu for container: Operation not permitted 2017-04-03T14:50:39.777494Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x4,romfile=/mnt/cache/domains/vbios/EVGA.GTX650Ti.1024.120919.dump: vfio: failed to setup container for group 13 2017-04-03T14:50:39.777501Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x4,romfile=/mnt/cache/domains/vbios/EVGA.GTX650Ti.1024.120919.dump: vfio: failed to get group 13 2017-04-03T14:50:39.777514Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x4,romfile=/mnt/cache/domains/vbios/EVGA.GTX650Ti.1024.120919.dump: Device initialization failed seems like you have a problem with your iommu groups. Please post your pci devices and iommu groups. I guess you have something in group 13 other than your GPU.
April 3, 20179 yr From "tools" -> "system devices" IOMMU group 0 [1002:5a14] 00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD9x0/RX980 Host Bridge (rev 02) IOMMU group 1 [1002:5a16] 00:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890/RD9x0/RX980 PCI to PCI bridge (PCI Express GFX port 0) IOMMU group 2 [1002:4390] 00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [IDE mode] (rev 40) IOMMU group 3 [1002:4397] 00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4396] 00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller IOMMU group 4 [1002:4397] 00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4396] 00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller IOMMU group 5 [1002:4385] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller (rev 42) IOMMU group 6 [1002:439c] 00:14.1 IDE interface: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 IDE Controller (rev 40) IOMMU group 7 [1002:4383] 00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia (Intel HDA) (rev 40) IOMMU group 8 [1002:439d] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller (rev 40) IOMMU group 9 [1002:4384] 00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge (rev 40) IOMMU group 10 [1002:4399] 00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller IOMMU group 11 [1002:43a0] 00:15.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 0) [1002:43a1] 00:15.1 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 1) [1002:43a2] 00:15.2 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB900 PCI to PCI bridge (PCIE port 2) [1b21:1042] 04:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller [10ec:8168] 05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 07) IOMMU group 12 [1002:4397] 00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4396] 00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller IOMMU group 13 [10de:11c6] 01:00.0 VGA compatible controller: NVIDIA Corporation GK106 [GeForce GTX 650 Ti] (rev a1) [10de:0e0b] 01:00.1 Audio device: NVIDIA Corporation GK106 HDMI Audio Controller (rev a1)
April 3, 20179 yr I have the exact same problem at this very moment. Just like yours my graphics audio controller is showing up in the same IOMMU group as the graphics card.
April 3, 20179 yr Does that not normally happen? Do you normally see them in separate groups? I think I see them in the same group in videos online and it's working.
April 4, 20179 yr Author 59 minutes ago, eric.frederich said: Does that not normally happen? Do you normally see them in separate groups? I think I see them in the same group in videos online and it's working. Yes its fine to have the audio and video in the same group as the are the same device. Just make sure you pass through both the video and audio part in the vm. Try changing your sys linux config file to add vfio_iommu_type1.allow_unsafe_interrupts=1 after the append part so the config file it would look like this. default /syslinux/menu.c32 menu title Lime Technology, Inc. prompt 0 timeout 50 label unRAID OS menu default kernel /bzimage append vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot label unRAID OS GUI Mode kernel /bzimage append initrd=/bzroot,/bzroot-gui label unRAID OS Safe Mode (no plugins, no GUI) kernel /bzimage append initrd=/bzroot unraidsafemode label Memtest86+ kernel /memtest
April 4, 20179 yr 13 hours ago, gridrunner said: Try changing your sys linux config file to add vfio_iommu_type1.allow_unsafe_interrupts=1 This seemed to work in that I can now launch the VM. It took over the console that was displayed and I even saw my Linux Mint startup splash screen. After that I got some X errors and was unable to do anything graphically, it didn't fall back to a terminal either. I figured maybe when I installed the VM under VNC it didn't know about the graphics card, so I have tried to set up a new VM with the graphics card there the whole time. Now again (after a reboot) it takes over the unRAID terminal but instead I see nothing. I log in from another machine via ssh and I see some qemu process taking 100% cpu. Nothing happens, I have waited a while. Maybe I should try Ubuntu instead of Mint since it's more mainstream? Maybe I should install the OS first under VNC then switch? What should I do?
April 4, 20179 yr Author 47 minutes ago, eric.frederich said: This seemed to work in that I can now launch the VM. It took over the console that was displayed and I even saw my Linux Mint startup splash screen. After that I got some X errors and was unable to do anything graphically, it didn't fall back to a terminal either. I figured maybe when I installed the VM under VNC it didn't know about the graphics card, so I have tried to set up a new VM with the graphics card there the whole time. Now again (after a reboot) it takes over the unRAID terminal but instead I see nothing. I log in from another machine via ssh and I see some qemu process taking 100% cpu. Nothing happens, I have waited a while. Maybe I should try Ubuntu instead of Mint since it's more mainstream? Maybe I should install the OS first under VNC then switch? What should I do? The linux flavour will not make a difference. But try using seabios instead of OVMF or vice versa. Also try machine type i440 and q35. Failing that I am not sure what else that you can try. Try dropping limetech support an email they may be able to suggest a solution
April 12, 20179 yr This worked great for me with two GTX 1080s! I was able to previously pass through only the one that was not in the top PCI-E slot but now I can use both on the same machine and power my monitor on the primary one (which before starting the VM displays the UNRAID terminal). Thank you!
April 14, 20179 yr Doen't work for me, what did i wrong ? <hostdev mode='subsystem' type='pci' managed='yes' xvga='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <rom file='/mnt/user/ISOs/980ti.dump'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </hostdev> Now it works for me too, i only need to wait like 3-5 min. after starting the VM Edited April 14, 20179 yr by unknown Now it works
April 15, 20179 yr Author 10 hours ago, unknown said: Doen't work for me, what did i wrong ? <hostdev mode='subsystem' type='pci' managed='yes' xvga='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <rom file='/mnt/user/ISOs/980ti.dump'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </hostdev> Now it works for me too, i only need to wait like 3-5 min. after starting the VM Wow, so you are having to wait that long for boot up. It shouldn't take so long You may try installing the tips and tweaks plugin and set your CPU governor to performance and the intel turbo boost to enable. (also other useful tweaks there too) and see if that may help
April 15, 20179 yr Intel Turbo is enable. Where can i find there tweak plugins or what do i have to change ?
April 15, 20179 yr Author 5 hours ago, unknown said: Intel Turbo is enable. Where can i find there tweak plugins or what do i have to change ? You can download a plugin called tips and tweaks and make some changes there too. Download from community applications.
April 16, 20179 yr On 4/4/2017 at 10:58 AM, gridrunner said: The linux flavour will not make a difference. But try using seabios instead of OVMF or vice versa. Also try machine type i440 and q35. Sorry, been away on vacation. I switched to seabios and it seemed to work. I installed Linux Mint and have both monitors working now. Of course now it's easter I need to leave the house for the day, no time to play with it. I did it with both the graphics and HDMI audio since I heard you have to have everything within the IOMMU group mapped in. I'd like to try to use my onboard audio since my monitors don't have a line-out for the HDMI. Will this stop working if I don't have the HDMI audio mapped in?... Is it possible to have both onboard and HDMI mapped into the VM? The configuration page is a drop-down list so I can only select one, but maybe by editing the XML itself?
Archived
This topic is now archived and is closed to further replies.