GPU Pass Through Using VFIO-PCI) - SUCCESS WITH Windows, SteamOS, and MORE!

jonp · July 1, 2014

All,

Apologies for not posting this sooner, but wanted to give you guys a heads up on this. We've been working hard at improving this feature as we think it adds a core value to our solution (combining the best of both worlds: desktop and server). First and foremost, our latest XML file (anything in between  is a comment):

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
<name>win81-r9290</name>

<uuid>554cbf6b-aa75-4044-b1b3-c1005bea6064</uuid>



<memory unit='KiB'>8388608</memory>

<currentMemory unit='KiB'>8388608</currentMemory>

<memoryBacking>

<nosharepages/>

<locked/>

</memoryBacking>



<vcpu placement='static'>4</vcpu>

<cpu mode='host-passthrough'>

<topology sockets='1' cores='4' threads='1'/>

</cpu>

<os>

<type arch='x86_64' machine='q35'>hvm</type>

<loader>/usr/share/qemu/bios-256k.bin</loader>

<boot dev='cdrom'/>

</os>

<features>

<acpi/>

<apic eoi='on'/>

</features>

<clock offset='localtime'>

<timer name='rtc' tickpolicy='catchup'/>

<timer name='pit' tickpolicy='delay'/>

<timer name='hpet' present='yes'/>

</clock>

<on_poweroff>destroy</on_poweroff>

<on_reboot>restart</on_reboot>

<on_crash>destroy</on_crash>

<devices>



<disk type='file' device='disk'>

<driver name='qemu' type='qcow2' cache='none' io='native'/>

<source file='/mnt/cache/path/to/image.qcow'/>

<target dev='vdc' bus='virtio'/>

</disk>



<disk type='file' device='cdrom'>

<source file='/mnt/cache/isos/virtio.iso'/>

<target dev='hdb' bus='ide'/>

<address type='drive' controller='0' bus='1' unit='0'/>

</disk>

<controller type='usb' index='0' model='ich9-ehci1'>

<address type='pci' domain='0' bus='0' slot='4' function='7'/>

</controller>

<controller type='usb' index='0' model='ich9-uhci1'>

<master startport='0'/>

<address type='pci' domain='0' bus='0' slot='4' function='0' multifunction='on'/>

</controller>

<controller type='scsi' index='0' model='virtio-scsi'/>



<interface type='bridge'>

<model type='virtio'/>

<source bridge='xenbr0'/>

</interface>

<controller type='pci' index='0' model='pcie-root'/>

<controller type='pci' index='1' model='dmi-to-pci-bridge'>

<address type='pci' domain='0x0000' bus='0x00' slot='0x1d' function='0x0'/>

</controller>

<controller type='pci' index='2' model='pci-bridge'>

<address type='pci' domain='0x0000' bus='0x01' slot='0x01' function='0x0'/>

</controller>

<controller type='sata' index='0'>

<address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>

</controller>

<controller type='virtio-serial' index='0'>

<address type='pci' domain='0x0000' bus='0x02' slot='0x05' function='0x0'/>

</controller>

<controller type='usb' index='0' model='ich9-ehci1'>

<address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x7'/>

</controller>

<controller type='usb' index='0' model='ich9-uhci1'>

<master startport='0'/>

<address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x0' multifunction='on'/>

</controller>

<controller type='usb' index='0' model='ich9-uhci2'>

<master startport='2'/>

<address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x1'/>

</controller>

<controller type='usb' index='0' model='ich9-uhci3'>

<master startport='4'/>

<address type='pci' domain='0x0000' bus='0x02' slot='0x02' function='0x2'/>

</controller>



<hostdev mode='subsystem' type='usb'>

<source>

<vendor id='0x046d'/>

<product id='0xc52b'/>

</source>

</hostdev>

<emulator>/usr/bin/qemu-system-x86_64</emulator>

<input type='mouse' bus='ps2'/>

<memballoon model='virtio'>

<address type='pci' domain='0x0000' bus='0x02' slot='0x06' function='0x0'/>

</memballoon>

</devices>



<qemu:commandline>

<qemu:arg value='-device'/>

<qemu:arg value='ioh3420,bus=pcie.0,addr=1c.0,multifunction=on,port=1,chassis=1,id=root.1'/>

<qemu:arg value='-device'/>



<qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on'/>

<qemu:arg value='-device'/>



<qemu:arg value='vfio-pci,host=01:00.1,bus=root.1,addr=00.1'/>

</qemu:commandline>

</domain>

Now there is more to this magic than just this. You will need to edit your qemu.conf file located under /etc/libvirt/qemu.conf and add items to your cgroups there. Once you have edited your qemu.conf file, I would also copy that file somewhere off to the side as your edits will NOT persist after a host reboot. Remember to restart libvirt after editing. /etc/rc.d/rc.libvirt restart.

You will also need the vfio-bind script in order to bind your PCI devices to the vfio-pci driver.

You do NOT need to do a pci-stub with unRAID as we do not bind any GPU to the host anyway.

We have this working with SteamOS, Windows (7 and 8.1), Ubuntu 14.04, XBMCbuntu (13.0), and a few other operating systems already. Other operating systems should work as well.

NOTE: NVIDIA GRAPHICS CARDS IN BETA 6 WILL NOT WORK WITH THE LATEST NVIDIA DRIVER...yet... ;-)

I will be updating this thread over time with more details and *GASP* VIDEOS! That's right folks, Lime Tech is going to start producing video content to showcase how you can take advantage of new and powerful features and one of the first will be a demo of GPU pass through to a variety of guest VMs.

Keep in mind, we are very focused on development right now to get 6.0 out the door, but we wanted to share this research with you guys to help you along your journey's to combining the best of both worlds: Desktop and Server!

peter_sm · July 1, 2014

Great progress!

To make it more less lines, you can remove all lines started with

<address type=

These will be generated when starting machine, I assume you have done xmldump of the VM?

I also get some error with these 2 lines

<controller type='usb' index='0' model='ich9-ehci1'>
    </controller> 

   <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
    </controller>

Removed these, all started up as I expect.

//Peter

JustinChase · July 1, 2014

You know that I'm pretty motivated to get GPU passthru working, but I'm not willing to go thru all of that.

Hopefully a new beta will come soon and remove the need for all this hoop-jumping.

Until the, no TV or Movies for my family

jonp · July 1, 2014

You know that I'm pretty motivated to get GPU passthru working, but I'm not willing to go thru all of that.

Hopefully a new beta will come soon and remove the need for all this hoop-jumping.

Until the, no TV or Movies for my family

Most definitely. There are LOTS of hoops to jump through right now, but when you have it working, you get AMAZING results. In addition, making this work on ANY hardware is a trick that will take more time. So far we have this working on 3 motherboards (1 supermicro, 2 consumer-grade desktop boards), but we have had a problem getting it to work with another board. In addition, specific cards may work while others don't. We have NOT been able to get a GTX 550 ti we have to work and survive a reboot.

We also haven't tested SLI capabilities, but I don't think that's as critical at this very moment.

To be clear, our primary goal with this isn't for gaming, but rather, allowing folks to do HDMI out of the box storing their content with RAW IO to the content its streaming. Imagine blu ray rips being played via HDMI with no network involvement whatsoever (virtually or physically).

I hope the future videos will make our goals here more apparent along with the possibilities!!

jonp · July 1, 2014

Great progress!

To make it more less lines, you can remove all lines started with
<address type=
These will be generated when starting machine, I assume you have done xmldump of the VM?

I also get some error with these 2 lines
<controller type='usb' index='0' model='ich9-ehci1'>
    </controller> 

   <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
    </controller>
Removed these, all started up as I expect.

//Peter

Thanks Peter! Yeah we made this XML from a smattering of various XML files we've found referenced online. Making this work in libvirt has been a priority because manual QEMU invocation would be a royal pain to manage. Another setting we found not required today is the SMBIOS. Shouldn't be necessary.

JustinChase · July 1, 2014

Thanks for the input. Do you have iGPU working with this? That's my main goal right now. Well, I was considering putting my 550Ti into the server, but since you don't have that working right now, that's a no go.

I have decided to dedicate 1 hour more in trying to implement all you've described, mainly because I feel like I'm probably close already, and I have the windows7 image in qcow2 format already created and updated.

I'll post back with any issues, or success I have.

JustinChase · July 1, 2014

Okay, can one of you 2 post the current, known to be working xml, and I'll start with that one.

thanks again.

peter_sm · July 1, 2014

Jomp, you could also move away from IDE to Sata for the CDROM. I think there would be some performance boost. See my config in my KVM thread for w8 pass trough.

Sent from my iPad using Tapatalk HD

JustinChase · July 1, 2014

Yeah, never mind. unRAID stopped responding, so I hard booted, now it seems my motherboard died. It won't even POST now.

I think I may just throw it all in the trash, and forget computers even exist!

jonp · July 1, 2014

Couple quick comments on this stuff:

1) Definitely will be changing IDE to SATA on CDROM. Again, this was a big cut-copy-and-paste job. That said, not sure if you'll see a performance boost. This is all "virtual" in there so unless you're using VIRTIO, it really shouldn't impact performance, but that's just my theory. We saw Win81 installs using IDE go REALLY fast.

2) With respect to getting iGPU working, we have NOT been successful with that yet. The i915 patch that people refer to DOES NOT APPLY TO UNRAID! Same thing goes for using pci-stub or any of that to "hide" the device from unRAID. unRAID does not claim devices during its boot process and doesn't need to for GPUs since there isn't a GUI anyway. iGPU pass through is something we hope to see resolved too, but right now, we've only been able to test this with 3 different test systems and none of them worked. While the display did get taken away from the console, it never started showing the guest, even though the guest was "running". Maybe the community can test this more.

JustinChase · July 2, 2014

Well, rebooting unRAID was unsuccessful again. I just spent about an hour dicking around, and have determined that my HD5550 is making my motherboard unhappy. If I remove it, i can boot, if I put it back, i can boot once, but not reboot. So, I'm retiring this card. The only other video cards I have are my Radeon X600, which is a really old, low quality card, which unRAID doesn't even recognize, and a GeForce GTX 550 Ti, which seems is one that jonp has been unsuccessful in getting to work with KVM passthru.

So, I guess I'm not going to be able to try anymore on GPU passthru.

Hopefully, either the 550 Ti, or, preferably, the iGPU will start working for passthru.

I suspect, I'll end up rebuilding the server with my old board, and go back to not bothering with GPU passthru, and just have a separate HTPC. Sad, because I've spent dozens, maybe a hundred hours on trying to get this to work, and a few hundred dollars on new parts, and in the end, it just doesn't work.

Perhaps I'll keep pushing my Linux knowledge and figure out how to move away from unRAID to a distro that works for passthru, like others seem to have successfully done.

I'm genuinely upset about all this.

Abnorm · July 7, 2014

Well, rebooting unRAID was unsuccessful again. I just spent about an hour dicking around, and have determined that my HD5550 is making my motherboard unhappy. If I remove it, i can boot, if I put it back, i can boot once, but not reboot. So, I'm retiring this card. The only other video cards I have are my Radeon X600, which is a really old, low quality card, which unRAID doesn't even recognize, and a GeForce GTX 550 Ti, which seems is one that jonp has been unsuccessful in getting to work with KVM passthru.

So, I guess I'm not going to be able to try anymore on GPU passthru.

Hopefully, either the 550 Ti, or, preferably, the iGPU will start working for passthru.

I suspect, I'll end up rebuilding the server with my old board, and go back to not bothering with GPU passthru, and just have a separate HTPC. Sad, because I've spent dozens, maybe a hundred hours on trying to get this to work, and a few hundred dollars on new parts, and in the end, it just doesn't work.

Perhaps I'll keep pushing my Linux knowledge and figure out how to move away from unRAID to a distro that works for passthru, like others seem to have successfully done.

I'm genuinely upset about all this.

So .. you're upset that a beta feature in UnRaid (that is pretty much borderline experimental) doesn't work

Have you tried running XEN as your hypervisor instead? Unraid 6b5 (yes beta 5) works fine with this. If you get beta 5 working, beta 6 should be a walk in the park (xen cfg vs libvirt).

It just seems like a simpler approach (vs KVM), and there is more information available on the forum too.

My primary computer is now a virtualized windows 8.1 on a dedicated SSD with a passed through AMD/ATI 7950 GPU and a dedicated USB controller card.

Although it will crash now and again.. considering this is somewhat highly experimental who cares, it works "enough".

Parallel to this I'm also running a Archlinux VM for my sql media database.

Keep in mind, these are experimental features. Nothing will just work out of the box, if you're outside your comfort zone in getting your hands dirty, virtualization is prolly not for you (until a one-click-solution is available at least).

Since you have some hardware you're not sure if works or not, gpu wise, ALWAYS use AMD/ATI, since Nvidia can and will be problematic (not impossible, but it's more work).

It's not because Nvidia sucks, they usually just do not have a way to "trick" your gpu in thinking it has been turned off (in the virtual space that is, like i.e your VM crashes and you need to restart the VM = result entire domU crashes).

Another thing to note about Nvidia, you have to reflash the card to a quaddro card (a CAD card), which sucks balls, as the performance you'd want (for gaming I presume) will be virtually (pun intended) non-existing.

Only way to figure this out, is learning it, I got tons of pointers from this part of the forum; http://lime-technology.com/forum/index.php?board=52.0 for XEN not KVM btw..

Just to be clear, I'm not trying to be an ass towards you, all I'm saying is, it's not UnRAID as a distro that is the problem here. If you've spent around 100 hours on getting it to work.. you have to have gotten some faulty instructions somewhere or bought incompatible/defective hardware.

Oh and you mentioned you could not get past POST when hardbooting ? remove your powercord, press the on/off button on your server, stick it back in, boot your server - magic (it's because your card hasn't reset correctly after your server crashed)

piotrasd · September 13, 2014

Hi @jonp

Could you now after released beta9 give some more light on Openelec over KVM with GPU, some example XML configs etc.

I will be very grateful

m4f1050 · June 18, 2015

What is the best GPU you can get VM Manager to passthrough? (i.e. HD-6450, GTX-780, etc etc?)

I tried my GTS-450 (converted to Quadro 2000 for Xen) and it will not display anything at all. Am I missing something or is this card not compatible? Would it be compatible if I switch it back to GTS-450? Need help with GPU Pass Through.

Thanks!

dlandon · June 18, 2015

What is the best GPU you can get VM Manager to passthrough? (i.e. HD-6450, GTX-780, etc etc?)

I tried my GTS-450 (converted to Quadro 2000 for Xen) and it will not display anything at all. Am I missing something or is this card not compatible? Would it be compatible if I switch it back to GTS-450? Need help with GPU Pass Through.

Thanks!

Are you booting in OVMF? What OS are you installing?

m4f1050 · June 18, 2015

Not sure about OVMF (is there a setting? I have recently upgraded to unRAID 6.0.0 (usb stick) and booting from it)

OS is Windows 8, already installed, but when I pick video card it stays blank screen.

EDIT: I had a GTS-450 (converted to Quadro 2000) working on unRAID 6.0-beta14b and Xen 4.4 but it will not work on QEMU/KVM in unRAID 6.0.0

jonp · June 18, 2015

You need to read the wiki guides. There are numerous suggestions on how to improve your chance of success.

m4f1050 · June 18, 2015

What is the fastest GPU that someone has gotten to work with KVM/QEMU passthrough on unRAID 6.0.0? I need a starting point, I don't want to try with a hardware someone hasn't yet proven it to work.

dlandon · June 18, 2015

What is the fastest GPU that someone has gotten to work with KVM/QEMU passthrough on unRAID 6.0.0? I need a starting point, I don't want to try with a hardware someone hasn't yet proven it to work.

I have an XFX AMD 5450 passed through to a Windows 8.1 VM that is booting in OVFM. My system hardware is in my signature.

bungee91 · June 18, 2015

What is the fastest GPU that someone has gotten to work with KVM/QEMU passthrough on unRAID 6.0.0? I need a starting point, I don't want to try with a hardware someone hasn't yet proven it to work.

While there are certainly more, this should be helpful (scroll down a little)

http://lime-technology.com/wiki/index.php/UnRAID_Manual_6#Lime_Technology_Tested_Components

Also, this is helpful (also in the wiki) https://docs.google.com/spreadsheets/d/1LnGpTrXalwGVNy0PWJDURhyxa3sgqkGXmvNCIvIMenk/edit#gid=0

net2wire · June 18, 2015

What is the fastest GPU that someone has gotten to work with KVM/QEMU passthrough on unRAID 6.0.0? I need a starting point, I don't want to try with a hardware someone hasn't yet proven it to work.

I have an XFX AMD 5450 passed through to a Windows 8.1 VM that is booting in OVFM. My system hardware is in my signature.

+1 on the XFX AMD 5450. Windows 7 Ultimate VM booting in SeaBIOS.

m4f1050 · June 19, 2015

IDK if I want an HD 5450, got my kid wanting to play Minecraft with him, don't want to be lagging...

Went with GTX 780, on Amazon I had $75 in gift, cost me like $176... Not too bad. Cheaper than the 650 Ti and beats it by far.

net2wire · June 19, 2015

IDK if I want an HD 5450, got my kid wanting to play Minecraft with him, don't want to be lagging...

Yeah that's understandable. My Windows 7 VM + 5450 is mainly used for security surveillance with two displays.

jonp · June 19, 2015

IDK if I want an HD 5450, got my kid wanting to play Minecraft with him, don't want to be lagging...

Went with GTX 780, on Amazon I had $75 in gift, cost me like $176... Not too bad. Cheaper than the 650 Ti and beats it by far.

The 780 is the same card I have. That thing is a beast and you got a helluva deal.

m4f1050 · June 24, 2015

I went with the HD 6450, read it worked very nice with the only game my son plays (Minecraft) and it was only $33 :-)

My Win8 KVM now works perfect. Thanks!

GPU Pass Through Using VFIO-PCI) - SUCCESS WITH Windows, SteamOS, and MORE!

Recommended Posts

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Link to comment

Join the conversation