SpaceInvaderOne Posted March 26, 2017 Share Posted March 26, 2017 (edited) Hi, guys. I was frustrated as to why we couldn't use the techpowerup vbios files in our VMs. So I have discovered it is due to a header put there for later Nvidia cards which isn't compatible with KVM. (the header is for the NVflash programme) So in this video, you will see how to pass through a Nvidia GPU as primary in a KVM virtual machine on unRAID by removing the header from a vbios dumped by GPU-z/ Techpowerup making it compatible with a KVM virtual machine. This is very useful as techpowerup.com has an extensive database of GPU vbioses. Also, should you want to dump your own previously you had to have 2 GPUs and 2 PCIe slots to do this. Well now you can do it using GPU-Z easily then convert it for KVM. How to easily passthough a Nvidia GPU as primary without dumping your own vbios! Edited March 26, 2017 by gridrunner 7 Quote Link to comment
saarg Posted March 26, 2017 Share Posted March 26, 2017 Awesome digging! This is going to help a lot of those with only one GPU Quote Link to comment
dadarara Posted March 26, 2017 Share Posted March 26, 2017 1) I have a GTX560Ti that works with my OSX VM. It doesnt work with Windows. no matter what I throw at it including the dumped bios. does the above mean that after removing the header of the bios file it should work with Windows VM ? (I tried the bios from GPUZ and from command line as well) 2) <rom bar='on' file='/mnt/user/isos/vbios.rom'/> can you guys explain what exactly happens when I include the above in the XML? does it mean that the internal bios of the GPU no longer used? does this action have the same impact on OSX as in Windows VMs? is it OS agnostic and good for any type of VM ? 3) I still have some strange behaviour of this 560Ti card in the OSX. I am using the original driver of OSX without installing nvidia webdrivers. when I update the OSX, sometimes the card stops to be recognized. all I need to do in that case is switch the power to the unRAID server and start it again. full reboot incl power. any thoughts why this happens? any past experience like this ? Quote Link to comment
RobJ Posted March 27, 2017 Share Posted March 27, 2017 Added to Guides and Videos ! Quote Link to comment
tjb_altf4 Posted March 27, 2017 Share Posted March 27, 2017 Worked perfect for me, just had to deal with some pesky unrelated iommu issues, then it all worked like magic! Quote Link to comment
SpaceInvaderOne Posted March 27, 2017 Author Share Posted March 27, 2017 51 minutes ago, tjb_altf4 said: Worked perfect for me, just had to deal with some pesky unrelated iommu issues, then it all worked like magic! Great glad it worked Quote Link to comment
SpaceInvaderOne Posted March 27, 2017 Author Share Posted March 27, 2017 23 hours ago, dadarara said: 1) I have a GTX560Ti that works with my OSX VM. It doesnt work with Windows. no matter what I throw at it including the dumped bios. does the above mean that after removing the header of the bios file it should work with Windows VM ? (I tried the bios from GPUZ and from command line as well) 2) <rom bar='on' file='/mnt/user/isos/vbios.rom'/> can you guys explain what exactly happens when I include the above in the XML? does it mean that the internal bios of the GPU no longer used? does this action have the same impact on OSX as in Windows VMs? is it OS agnostic and good for any type of VM ? 3) I still have some strange behaviour of this 560Ti card in the OSX. I am using the original driver of OSX without installing nvidia webdrivers. when I update the OSX, sometimes the card stops to be recognized. all I need to do in that case is switch the power to the unRAID server and start it again. full reboot incl power. any thoughts why this happens? any past experience like this ? Hi @dadarara To answer questions 1. You say it doesn't work in windows. I am guessing that your osx might be seabios and windows ovmf (as it is by default in the templates) Your GPU probably doesn't support UEFI so will not work with ovmf. Now you could do one of the following a/ Just use seabios. b/ Look through techpowerup database and try and find a UEFI vbios dump. I have in my second server, an r9 270x which isn't UEFI compatible. So i downloaded a vbios with UEFI in it and passed that through and now it works fine in ovmf (no need to flash the card) 2. That brings me onto your second question. Yes, it uses the ROM file passed through in the XML instead of the one on the card. The rombar=on stops the rom being used and then uses the one specified after. But you don't need to put the rombar=on in, as it is on by default when you specify a ROM file. (that was only necessary on old versions of qemu If no rom bar is specified, the qemu default will be used (older versions of qemu used a default of "off", while newer qemus have a default of "on") https://libvirt.org/formatdomain.html 3. Not sure why you get those issues in osx. Maybe the card isn't resetting correctly hence the need to reboot the whole server. So overall if it were me I would try finding an UEFI vbios and pass it through. Quote Link to comment
andrewraynor Posted March 28, 2017 Share Posted March 28, 2017 worked great for me and thanks for all your guides Quote Link to comment
eric.frederich Posted April 3, 2017 Share Posted April 3, 2017 Awesome... I have a single GPU on my system. I hope this works for me. I just need to find some time to myself to be able to try it out. Right now on my unRAID machine I have two monitors set up (because it used to be a desktop Linux machine). I'd like to utilize this VM and GPU passthrough to have it act like a desktop machine once again. Let me ask a question... right now I see a Linux console on one of the monitors. What will happen if I follow this guide? I'll boot unRAID and unRAID will utilize the graphics card until I start my VM?... then what... it hijacks it? Quote Link to comment
SpaceInvaderOne Posted April 3, 2017 Author Share Posted April 3, 2017 Yes unRAID will use the GPU for console output. However when you start the VM the VM started will then take over the GPU and use it for its output. When the VM is shut down you unRAID will not then take it back (unless you reboot the server) If you start the VM again or another VM using that card, the VM will again output from it. Its not a problem for most loosing the console locally as you can manage the server from the GUI or by ssh. Quote Link to comment
eric.frederich Posted April 3, 2017 Share Posted April 3, 2017 I followed the guide. This is what I get when I try to start my VM... Execution error internal error: qemu unexpectedly closed the monitor: 2017-04-03T14:50:39.777467Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x4,romfile=/mnt/cache/domains/vbios/EVGA.GTX650Ti.1024.120919.dump: vfio: failed to set iommu for container: Operation not permitted 2017-04-03T14:50:39.777494Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x4,romfile=/mnt/cache/domains/vbios/EVGA.GTX650Ti.1024.120919.dump: vfio: failed to setup container for group 13 2017-04-03T14:50:39.777501Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x4,romfile=/mnt/cache/domains/vbios/EVGA.GTX650Ti.1024.120919.dump: vfio: failed to get group 13 2017-04-03T14:50:39.777514Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x4,romfile=/mnt/cache/domains/vbios/EVGA.GTX650Ti.1024.120919.dump: Device initialization failed Quote Link to comment
SpaceInvaderOne Posted April 3, 2017 Author Share Posted April 3, 2017 1 hour ago, eric.frederich said: I followed the guide. This is what I get when I try to start my VM... Execution error internal error: qemu unexpectedly closed the monitor: 2017-04-03T14:50:39.777467Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x4,romfile=/mnt/cache/domains/vbios/EVGA.GTX650Ti.1024.120919.dump: vfio: failed to set iommu for container: Operation not permitted 2017-04-03T14:50:39.777494Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x4,romfile=/mnt/cache/domains/vbios/EVGA.GTX650Ti.1024.120919.dump: vfio: failed to setup container for group 13 2017-04-03T14:50:39.777501Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x4,romfile=/mnt/cache/domains/vbios/EVGA.GTX650Ti.1024.120919.dump: vfio: failed to get group 13 2017-04-03T14:50:39.777514Z qemu-system-x86_64: -device vfio-pci,host=01:00.0,id=hostdev0,bus=pci.2,addr=0x4,romfile=/mnt/cache/domains/vbios/EVGA.GTX650Ti.1024.120919.dump: Device initialization failed seems like you have a problem with your iommu groups. Please post your pci devices and iommu groups. I guess you have something in group 13 other than your GPU. Quote Link to comment
eric.frederich Posted April 3, 2017 Share Posted April 3, 2017 From "tools" -> "system devices" IOMMU group 0 [1002:5a14] 00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD9x0/RX980 Host Bridge (rev 02) IOMMU group 1 [1002:5a16] 00:02.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] RD890/RD9x0/RX980 PCI to PCI bridge (PCI Express GFX port 0) IOMMU group 2 [1002:4390] 00:11.0 SATA controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 SATA Controller [IDE mode] (rev 40) IOMMU group 3 [1002:4397] 00:12.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4396] 00:12.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller IOMMU group 4 [1002:4397] 00:13.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4396] 00:13.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller IOMMU group 5 [1002:4385] 00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 SMBus Controller (rev 42) IOMMU group 6 [1002:439c] 00:14.1 IDE interface: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 IDE Controller (rev 40) IOMMU group 7 [1002:4383] 00:14.2 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 Azalia (Intel HDA) (rev 40) IOMMU group 8 [1002:439d] 00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 LPC host controller (rev 40) IOMMU group 9 [1002:4384] 00:14.4 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SBx00 PCI to PCI Bridge (rev 40) IOMMU group 10 [1002:4399] 00:14.5 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI2 Controller IOMMU group 11 [1002:43a0] 00:15.0 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 0) [1002:43a1] 00:15.1 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB700/SB800/SB900 PCI to PCI bridge (PCIE port 1) [1002:43a2] 00:15.2 PCI bridge: Advanced Micro Devices, Inc. [AMD/ATI] SB900 PCI to PCI bridge (PCIE port 2) [1b21:1042] 04:00.0 USB controller: ASMedia Technology Inc. ASM1042 SuperSpeed USB Host Controller [10ec:8168] 05:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 07) IOMMU group 12 [1002:4397] 00:16.0 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB OHCI0 Controller [1002:4396] 00:16.2 USB controller: Advanced Micro Devices, Inc. [AMD/ATI] SB7x0/SB8x0/SB9x0 USB EHCI Controller IOMMU group 13 [10de:11c6] 01:00.0 VGA compatible controller: NVIDIA Corporation GK106 [GeForce GTX 650 Ti] (rev a1) [10de:0e0b] 01:00.1 Audio device: NVIDIA Corporation GK106 HDMI Audio Controller (rev a1) 1 Quote Link to comment
aekstrom Posted April 3, 2017 Share Posted April 3, 2017 I have the exact same problem at this very moment. Just like yours my graphics audio controller is showing up in the same IOMMU group as the graphics card. Quote Link to comment
eric.frederich Posted April 3, 2017 Share Posted April 3, 2017 Does that not normally happen? Do you normally see them in separate groups? I think I see them in the same group in videos online and it's working. Quote Link to comment
SpaceInvaderOne Posted April 4, 2017 Author Share Posted April 4, 2017 59 minutes ago, eric.frederich said: Does that not normally happen? Do you normally see them in separate groups? I think I see them in the same group in videos online and it's working. Yes its fine to have the audio and video in the same group as the are the same device. Just make sure you pass through both the video and audio part in the vm. Try changing your sys linux config file to add vfio_iommu_type1.allow_unsafe_interrupts=1 after the append part so the config file it would look like this. default /syslinux/menu.c32 menu title Lime Technology, Inc. prompt 0 timeout 50 label unRAID OS menu default kernel /bzimage append vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot label unRAID OS GUI Mode kernel /bzimage append initrd=/bzroot,/bzroot-gui label unRAID OS Safe Mode (no plugins, no GUI) kernel /bzimage append initrd=/bzroot unraidsafemode label Memtest86+ kernel /memtest Quote Link to comment
eric.frederich Posted April 4, 2017 Share Posted April 4, 2017 13 hours ago, gridrunner said: Try changing your sys linux config file to add vfio_iommu_type1.allow_unsafe_interrupts=1 This seemed to work in that I can now launch the VM. It took over the console that was displayed and I even saw my Linux Mint startup splash screen. After that I got some X errors and was unable to do anything graphically, it didn't fall back to a terminal either. I figured maybe when I installed the VM under VNC it didn't know about the graphics card, so I have tried to set up a new VM with the graphics card there the whole time. Now again (after a reboot) it takes over the unRAID terminal but instead I see nothing. I log in from another machine via ssh and I see some qemu process taking 100% cpu. Nothing happens, I have waited a while. Maybe I should try Ubuntu instead of Mint since it's more mainstream? Maybe I should install the OS first under VNC then switch? What should I do? Quote Link to comment
SpaceInvaderOne Posted April 4, 2017 Author Share Posted April 4, 2017 47 minutes ago, eric.frederich said: This seemed to work in that I can now launch the VM. It took over the console that was displayed and I even saw my Linux Mint startup splash screen. After that I got some X errors and was unable to do anything graphically, it didn't fall back to a terminal either. I figured maybe when I installed the VM under VNC it didn't know about the graphics card, so I have tried to set up a new VM with the graphics card there the whole time. Now again (after a reboot) it takes over the unRAID terminal but instead I see nothing. I log in from another machine via ssh and I see some qemu process taking 100% cpu. Nothing happens, I have waited a while. Maybe I should try Ubuntu instead of Mint since it's more mainstream? Maybe I should install the OS first under VNC then switch? What should I do? The linux flavour will not make a difference. But try using seabios instead of OVMF or vice versa. Also try machine type i440 and q35. Failing that I am not sure what else that you can try. Try dropping limetech support an email they may be able to suggest a solution Quote Link to comment
nrosario Posted April 8, 2017 Share Posted April 8, 2017 Thank you for the guide it help me overcome my issues as well. Regards Quote Link to comment
NNK Posted April 12, 2017 Share Posted April 12, 2017 This worked great for me with two GTX 1080s! I was able to previously pass through only the one that was not in the top PCI-E slot but now I can use both on the same machine and power my monitor on the primary one (which before starting the VM displays the UNRAID terminal). Thank you! Quote Link to comment
unknown Posted April 14, 2017 Share Posted April 14, 2017 (edited) Doen't work for me, what did i wrong ? <hostdev mode='subsystem' type='pci' managed='yes' xvga='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <rom file='/mnt/user/ISOs/980ti.dump'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </hostdev> Now it works for me too, i only need to wait like 3-5 min. after starting the VM Edited April 14, 2017 by unknown Now it works Quote Link to comment
SpaceInvaderOne Posted April 15, 2017 Author Share Posted April 15, 2017 10 hours ago, unknown said: Doen't work for me, what did i wrong ? <hostdev mode='subsystem' type='pci' managed='yes' xvga='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <rom file='/mnt/user/ISOs/980ti.dump'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/> </hostdev> Now it works for me too, i only need to wait like 3-5 min. after starting the VM Wow, so you are having to wait that long for boot up. It shouldn't take so long You may try installing the tips and tweaks plugin and set your CPU governor to performance and the intel turbo boost to enable. (also other useful tweaks there too) and see if that may help Quote Link to comment
unknown Posted April 15, 2017 Share Posted April 15, 2017 Intel Turbo is enable. Where can i find there tweak plugins or what do i have to change ? Quote Link to comment
SpaceInvaderOne Posted April 15, 2017 Author Share Posted April 15, 2017 5 hours ago, unknown said: Intel Turbo is enable. Where can i find there tweak plugins or what do i have to change ? You can download a plugin called tips and tweaks and make some changes there too. Download from community applications. Quote Link to comment
eric.frederich Posted April 16, 2017 Share Posted April 16, 2017 On 4/4/2017 at 10:58 AM, gridrunner said: The linux flavour will not make a difference. But try using seabios instead of OVMF or vice versa. Also try machine type i440 and q35. Sorry, been away on vacation. I switched to seabios and it seemed to work. I installed Linux Mint and have both monitors working now. Of course now it's easter I need to leave the house for the day, no time to play with it. I did it with both the graphics and HDMI audio since I heard you have to have everything within the IOMMU group mapped in. I'd like to try to use my onboard audio since my monitors don't have a line-out for the HDMI. Will this stop working if I don't have the HDMI audio mapped in?... Is it possible to have both onboard and HDMI mapped into the VM? The configuration page is a drop-down list so I can only select one, but maybe by editing the XML itself? Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.