Intel IGD (integrated graphics) Pass Through Support - March 2016 Update


Recommended Posts

Hi all,

 

It's been a while since I chimed in on this topic and given some recent updates / patches from the QEMU team and some testing I've been performing, I thought it was about time I provide an update.

 

Background

Many Intel CPUs feature support for an on-die GPU. We refer to this as an Intel IGD (Integrated Graphics Device).  This essentially means being able to supply and power graphics for a system without needing a discrete GPU (either soldered onto the chipset like with many AMD systems and their Radeon graphics chips or as a PCIe device).  However, up and through this point, unRAID has not supported assigning that GPU to a virtual machine in the same way we support it for PCIe add-on GPUs such as AMD and NVIDIA.  However, that will be changing in the future.

 

Why is IGD pass through to VMs important?

While discrete GPUs can provide a powerful performance boost to 3D applications (making them ideal for gaming), they do require an additional investment above and beyond the CPU/motherboard, increase the demand for power, generate more heat, and require you to supply an enclosure (case) that is large enough to house it effectively.  If all you want is a VM with a local desktop for basic 2D applications (office / productivity / browser), an IGD has more than enough horsepower to satisfy those tasks.

 

A powerful use-case could be to build a small 4-bay NAS for the living room that has a CPU with both an IGD and Intel VT-d support.  This small form-factor system could be placed in the living room and a video connection could be made from the IGD directly to the main TV, while also using the NAS for streaming media to other devices over the LAN.  The VMs that system could run would include both OpenELEC (for playing media) and SteamOS (for streaming games from another PC).

 

Why doesn't IGD pass through work today?

In short, IGDs are not like your atypical discrete GPU.  They behave very differently and therefore require a lot of special coding to pass them through to VMs.

 

Are there any efforts to support IGD pass through to VMs?

There is an entire project dedicated towards the use of IGDs with virtual machines:  Intel GVT.  This project is hard at work to bring multiple benefits to users.  From the projects dedicated site:

 

Intel® Graphics Virtualization Technology (Intel® GVT) encompasses three virtualization technologies, which use "-s, -d, -g" modifiers to differentiate, as follows:
  • Intel® Graphics Virtualization Technology –d (Intel® GVT –d): vDGA: virtual dedicated graphics acceleration (one VM to one physical GPU)
  • Intel® Graphics Virtualization Technology –s (Intel® GVT -s): vSGA: virtual shared graphics acceleration (multiple VMs to one physical GPU)
  • Intel® Graphics Virtualization Technology –g (Intel® GVT -g): vGPU: virtual graphics processing unit (multiple VMs to one physical GPU)

Xen is currently supported on Intel Processor Graphics (a.k.a. XenGT); [glow=red,2,300]experimental code is in development that is intended to support iGVT-g running on a KVM hypervisor (a.k.a KVMGT).[/glow]

 

You can read more about the project on it's official site:  https://01.org/igvt-g

 

What hardware will be supported?

Initial support will be for Intel Haswell and Broadwell CPUs only

 

But wait, what about my shiny new Skylake?!

Sorry but no, it doesn't appear that there are any plans to support Skylake at this time, but that may change in the future.  Note that Skylake CPUs are not recommended for VMs with GPU pass through (harder to isolate the IOMMU groups; no support for the ACS override). 

 

What about my Ivy-Bridge or <INSERT OLDER CPU HERE>?

Nope.

 

What is the current status of IGD pass through?

Back in December, a patch was released for QEMU to add support for passing through Intel IGDs to virtual machines.  This patch has yet to be merged in the stable branch for QEMU, but we at Lime Tech built a special version of QEMU to include this patch and tested it ourselves.  In short...[glow=red,2,300]it works[/glow].

 

However, it is not yet ready for prime time inclusion with unRAID.  First and foremost, there is no libvirt support for this at all yet, which means to do it we have to invoke QEMU manually, which is a huge problem from a manageability standpoint.  We also need to thoroughly test this (with various hardware and VM configurations).  And lastly, we need to do the logic coding on VM manager to then make it as simple as just allowing folks to select the IGD from the graphics device list when adding/editing a VM.

 

So???  When???

Supporting IGD pass through to VMs should be an achievable objective for us in 2016.  I will be able to narrow down that time-line sometime in the next 1-2 months as that will be the earliest we will be able to dedicate significant R&D time towards this item.

Link to comment

Super news Jon! My Q25 might have some more use cases now.

 

Is the performance of the iGPU in a VM environment suitable for something like Kodi or Plex? If so I now have a good reason to buy a i5 :)

 

It should be no different than in a physical environment, but again, limited testing.  To be perfectly honest, the way I had to test this was to use our OpenELEC VM.  I took the XML from that VM and ran it through a parser that libvirt provides to convert it to a QEMU command string, then I just modified the command string to pass it the additional items needed for IGD pass through and fired it up (no KB/mouse assigned yet).  The monitor lit up with OE running on it and that was good enough for me to say "test successful."

Link to comment

Thanks for this update. Your work on this is what convinced me to purchase unRAID. I really wanted a system that supported GPU passthrough out of the box (I don't really use unRAID as a NAS server, but rather a VM and docker machine). I just had such a hard time getting proxmox to work with GPU passthrough that having your software work out of the box was heaven sent. Thank you for all you do to get the latest and greatest features to the KVM hypervisor working in unRAID. It's definitely earning you sales.

Link to comment

But wait, what about my shiny new Skylake?!

Sorry but no, it doesn't appear that there are any plans to support Skylake at this time, but that may change in the future.  Note that Skylake CPUs are not recommended for VMs with GPU pass through (harder to isolate the IOMMU groups; no support for the ACS override). 

 

Are you sure ACS overide not supported in Skylake CPUs,  Jon?  I have Skylake (hardware in sig) and ACS works for me

 

 

This is my iommu groups with ACS override. Note my 2 gpus in separate groups 11 and 12

 

PCI Devices

00:00.0 Host bridge: Intel Corporation Sky Lake Host Bridge/DRAM Registers (rev 07)
00:01.0 PCI bridge: Intel Corporation Sky Lake PCIe Controller (x16) (rev 07)
00:01.1 PCI bridge: Intel Corporation Sky Lake PCIe Controller (x8) (rev 07)
00:02.0 VGA compatible controller: Intel Corporation Sky Lake Integrated Graphics (rev 06)
00:14.0 USB controller: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller (rev 31)
00:14.2 Signal processing controller: Intel Corporation Sunrise Point-H Thermal subsystem (rev 31)
00:16.0 Communication controller: Intel Corporation Sunrise Point-H CSME HECI #1 (rev 31)
00:17.0 SATA controller: Intel Corporation Device a102 (rev 31)
00:1b.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Root Port #17 (rev f1)
00:1c.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #1 (rev f1)
00:1d.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #9 (rev f1)
00:1f.0 ISA bridge: Intel Corporation Sunrise Point-H LPC Controller (rev 31)
00:1f.2 Memory controller: Intel Corporation Sunrise Point-H PMC (rev 31)
00:1f.3 Audio device: Intel Corporation Sunrise Point-H HD Audio (rev 31)
00:1f.4 SMBus: Intel Corporation Sunrise Point-H SMBus (rev 31)
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (2) I219-V (rev 31)
01:00.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 960] (rev a1)
01:00.1 Audio device: NVIDIA Corporation Device 0fba (rev a1)
02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Caicos [Radeon HD 6450/7450/8450 / R5 230 OEM]
02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6400 Series]
04:00.0 USB controller: ASMedia Technology Inc. Device 1242
IOMMU Groups

Warning: Your system has booted with the PCIe ACS Override setting enabled. The below list doesn't not reflect the way IOMMU would naturally group devices. To see natural IOMMU groups for your hardware, go to the VM Settings page and set the PCIe ACS Override setting to No.

/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/2/devices/0000:00:01.1
/sys/kernel/iommu_groups/3/devices/0000:00:02.0
/sys/kernel/iommu_groups/4/devices/0000:00:14.0
/sys/kernel/iommu_groups/4/devices/0000:00:14.2
/sys/kernel/iommu_groups/5/devices/0000:00:16.0
/sys/kernel/iommu_groups/6/devices/0000:00:17.0
/sys/kernel/iommu_groups/7/devices/0000:00:1b.0
/sys/kernel/iommu_groups/8/devices/0000:00:1c.0
/sys/kernel/iommu_groups/8/devices/0000:04:00.0
/sys/kernel/iommu_groups/9/devices/0000:00:1d.0
/sys/kernel/iommu_groups/10/devices/0000:00:1f.0
/sys/kernel/iommu_groups/10/devices/0000:00:1f.2
/sys/kernel/iommu_groups/10/devices/0000:00:1f.3
/sys/kernel/iommu_groups/10/devices/0000:00:1f.4
/sys/kernel/iommu_groups/10/devices/0000:00:1f.6
/sys/kernel/iommu_groups/11/devices/0000:01:00.0
/sys/kernel/iommu_groups/11/devices/0000:01:00.1
/sys/kernel/iommu_groups/12/devices/0000:02:00.0
/sys/kernel/iommu_groups/12/devices/0000:02:00.1

 

now this is my iommu groups wothout acs override. Note my 2 gpus now in group 1

 

PCI Devices

00:00.0 Host bridge: Intel Corporation Sky Lake Host Bridge/DRAM Registers (rev 07)
00:01.0 PCI bridge: Intel Corporation Sky Lake PCIe Controller (x16) (rev 07)
00:01.1 PCI bridge: Intel Corporation Sky Lake PCIe Controller (x8) (rev 07)
00:02.0 VGA compatible controller: Intel Corporation Sky Lake Integrated Graphics (rev 06)
00:14.0 USB controller: Intel Corporation Sunrise Point-H USB 3.0 xHCI Controller (rev 31)
00:14.2 Signal processing controller: Intel Corporation Sunrise Point-H Thermal subsystem (rev 31)
00:16.0 Communication controller: Intel Corporation Sunrise Point-H CSME HECI #1 (rev 31)
00:17.0 SATA controller: Intel Corporation Device a102 (rev 31)
00:1b.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Root Port #17 (rev f1)
00:1c.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #1 (rev f1)
00:1d.0 PCI bridge: Intel Corporation Sunrise Point-H PCI Express Root Port #9 (rev f1)
00:1f.0 ISA bridge: Intel Corporation Sunrise Point-H LPC Controller (rev 31)
00:1f.2 Memory controller: Intel Corporation Sunrise Point-H PMC (rev 31)
00:1f.3 Audio device: Intel Corporation Sunrise Point-H HD Audio (rev 31)
00:1f.4 SMBus: Intel Corporation Sunrise Point-H SMBus (rev 31)
00:1f.6 Ethernet controller: Intel Corporation Ethernet Connection (2) I219-V (rev 31)
01:00.0 VGA compatible controller: NVIDIA Corporation GM206 [GeForce GTX 960] (rev a1)
01:00.1 Audio device: NVIDIA Corporation Device 0fba (rev a1)
02:00.0 VGA compatible controller: Advanced Micro Devices, Inc. [AMD/ATI] Caicos [Radeon HD 6450/7450/8450 / R5 230 OEM]
02:00.1 Audio device: Advanced Micro Devices, Inc. [AMD/ATI] Caicos HDMI Audio [Radeon HD 6400 Series]
04:00.0 USB controller: ASMedia Technology Inc. Device 1242
IOMMU Groups

/sys/kernel/iommu_groups/0/devices/0000:00:00.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.0
/sys/kernel/iommu_groups/1/devices/0000:00:01.1
/sys/kernel/iommu_groups/1/devices/0000:01:00.0
/sys/kernel/iommu_groups/1/devices/0000:01:00.1
/sys/kernel/iommu_groups/1/devices/0000:02:00.0
/sys/kernel/iommu_groups/1/devices/0000:02:00.1
/sys/kernel/iommu_groups/2/devices/0000:00:02.0
/sys/kernel/iommu_groups/3/devices/0000:00:14.0
/sys/kernel/iommu_groups/3/devices/0000:00:14.2
/sys/kernel/iommu_groups/4/devices/0000:00:16.0
/sys/kernel/iommu_groups/5/devices/0000:00:17.0
/sys/kernel/iommu_groups/6/devices/0000:00:1b.0
/sys/kernel/iommu_groups/7/devices/0000:00:1c.0
/sys/kernel/iommu_groups/7/devices/0000:04:00.0
/sys/kernel/iommu_groups/8/devices/0000:00:1d.0
/sys/kernel/iommu_groups/9/devices/0000:00:1f.0
/sys/kernel/iommu_groups/9/devices/0000:00:1f.2
/sys/kernel/iommu_groups/9/devices/0000:00:1f.3
/sys/kernel/iommu_groups/9/devices/0000:00:1f.4
/sys/kernel/iommu_groups/9/devices/0000:00:1f.6

Link to comment

Any thoughts on @gridrunner comment @jonp?

Interesting, but doesn't change our recommendations. When Linus did the gaming + NAS video (not 2 gamers or 7), he had a skylake system and the ACS override didn't work for him. Luck of the draw on the hardware I guess.

 

Certainly doesn't change anything with respect to igd pass through support.

Link to comment

Any thoughts on @gridrunner comment @jonp?

Interesting, but doesn't change our recommendations. When Linus did the gaming + NAS video (not 2 gamers or 7), he had a skylake system and the ACS override didn't work for him. Luck of the draw on the hardware I guess.

 

Certainly doesn't change anything with respect to igd pass through support.

 

So, Jon, any comments on when us, the great unwashed, will be able to have a play with this.....  ;)

Link to comment

Any thoughts on @gridrunner comment @jonp?

Interesting, but doesn't change our recommendations. When Linus did the gaming + NAS video (not 2 gamers or 7), he had a skylake system and the ACS override didn't work for him. Luck of the draw on the hardware I guess.

 

Certainly doesn't change anything with respect to igd pass through support.

 

So, Jon, any comments on when us, the great unwashed, will be able to have a play with this.....  ;)

Eric sent this to us last night suggesting this is the current feel from the community about 6.2:

 

NeighboringAmpleBlacklab.gif

 

Yes, we know everyone is excited ;-)

 

I have been going through final upgrade testing this past week to make sure folks won't have any rude surprises and this weekend will be spent writing up upgrade instructions. That's a pretty good sign that we are close.

Link to comment

Any thoughts on @gridrunner comment @jonp?

Interesting, but doesn't change our recommendations. When Linus did the gaming + NAS video (not 2 gamers or 7), he had a skylake system and the ACS override didn't work for him. Luck of the draw on the hardware I guess.

 

Certainly doesn't change anything with respect to igd pass through support.

 

So, Jon, any comments on when us, the great unwashed, will be able to have a play with this.....  ;)

Eric sent this to us last night suggesting this is the current feel from the community about 6.2:

 

NeighboringAmpleBlacklab.gif

 

Yes, we know everyone is excited ;-)

 

I have been going through final upgrade testing this past week to make sure folks won't have any rude surprises and this weekend will be spent writing up upgrade instructions. That's a pretty good sign that we are close.

Look at the goodies you're dangling in front of our noses! No wonder we're excited  :P

Link to comment

I'm just grateful they're not the dog's danglies in front of me........  ;D

 

Saarg: In English, the phrase the "dogs danglies" (or a variety of this) is meant to mean something is very good.......

 

So your post made me laugh.  ;D

I thought it was dogs bollocks?

Link to comment

I'm just grateful they're not the dog's danglies in front of me........  ;D

 

Saarg: In English, the phrase the "dogs danglies" (or a variety of this) is meant to mean something is very good.......

 

So your post made me laugh.  ;D

I thought it was dogs bollocks?

I thought it was Sandra Bollocks  ;D

Link to comment

I'm just grateful they're not the dog's danglies in front of me........  ;D

 

Saarg: In English, the phrase the "dogs danglies" (or a variety of this) is meant to mean something is very good.......

 

So your post made me laugh.  ;D

I thought it was dogs bollocks?

 

Or the "Mutt's nuts" depends on whether you're in polite company or not... bollocks would be considered profanity by some...  ::)

Link to comment

Yes, we know everyone is excited ;-)

 

I have been going through final upgrade testing this past week to make sure folks won't have any rude surprises and this weekend will be spent writing up upgrade instructions. That's a pretty good sign that we are close.

 

Its only because you have been saying 6.2 is soon™ and whenever people have issues that 6.2 fixes those issues, since November.

We've fixed quite a few issues with incremental 6.1 releases too, just an FYI. Regardless, we are in the final stretch now.

Link to comment

Woot! 6.2 is Final Stretch!

 

Guess we can stop sharpening our pitchforks and stoking the fires then archedraft?  :-[

 

Was quite looking forward to that...

 

 

I'm guessing mid 2016 and 2 (maybe 3) unRAID forum uprisings...

 

Indeed... I suppose if 6.2 is released this month my prediction will have been wrong. As far as the uprisings, it would seem that this forum has become a little more classier than the 6.0 beta days.

Link to comment

 

Woot! 6.2 is Final Stretch!

 

Guess we can stop sharpening our pitchforks and stoking the fires then archedraft?  :-[

 

Was quite looking forward to that...

 

 

I'm guessing mid 2016 and 2 (maybe 3) unRAID forum uprisings...

 

Indeed... I suppose if 6.2 is released this month my prediction will have been wrong. As far as the uprisings, it would seem that this forum has become a little more classier than the 6.0 beta days.

<cough>grumpy<cough>
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.