Intel GVT-g support, multiple virtual intel iGPUs (we already have GVT-d)


scorcho99

Recommended Posts

So I've been playing around and it seems 6.6.7 at least already supports GVT-d (direct, single device passthrough). I have it working on my coffeelake system and its been solid through my basic testing phase.

 

My request is that the additional components needed for GVT-g (mediated passthrough, the virtual device has 'slices' taken off that can be given to multiple different VMs) be included in unraid. I looked around and although I think coffeelake will require a 5 series kernel, I haven't seen anyone uses this even with sky or kabylake. I believe everyone talking about GVT and Intel iGPU passthrough is talking about the single device mode. If I'm wrong about this I'd love to hear some one has this working.

 

I wouldn't even request support actually (although when things are more mature with this that would be nice), just include the modules required in the Intel guide: https://github.com/intel/gvt-linux/wiki/GVTg_Setup_Guide

Having the components to try would be a good start.

 

kvmgt

xengt (probably not needed for unraid)

vfio-iommu-type1

vfio-mdev

  • Like 1
Link to comment
  • 2 months later...

For more detail on what is required, here is a thread where it was added to Solus.

https://dev.getsol.us/T6812

 

Kernel options:

CONFIG_DRM_I915_GVT

CONFIG_DRM_I915_GVT_KVMGT

CONFIG_VFIO_MDEV

CONFIG_VFIO_MDEV_DEVICE

 

And a snippet from the developer on how to build:

https://github.com/intel/gvt-linux/issues/75#issuecomment-468122607



Could you confirm the kernel boot configuration and the mkinitramfs rule?
you can find the detailed information in our setup guide:
3.2 Build Kernel
3.2.1 Build the initrd (initial ramdisk)
Use Ubuntu as example, modify /etc/initramfs-tools/modules, like below:

kvmgt

xengt

vfio-iommu-type1

vfio-mdev

For Fedora or RHEL you should make the similar changes in "dracut".

3.2.2 Build Kernel Source
git clone https://github.com/intel/gvt-linux.git

cd gvt-linux

git checkout gvt-stable-4.17

echo ""|make oldconfig

Then make sure to enable CONFIG_DRM_I915_GVT, CONFIG_DRM_I915_GVT_KVMGT and CONFIG_DRM_I915_GVT_XENGT in ".config", which depends on CONFIG_VFIO_MDEV and CONFIG_VFIO_MDEV_DEVICE.

make -j8 && make modules_install && make install

Edited by scorcho99
Link to comment
  • 3 weeks later...

So, I built a custom unraid 6.8.0-rc7 kernel adding support for these modules which was a first for me. I ended up relying on the Unraid-DVB scripts to do a lot of the heavy lifting on that front.

 

After struggling with some configuration problems for the VM I now have GVT-g working on Unraid. At least, its working as well as it worked on Ubuntu. Lots more to investigate. But as a proof of concept, I think its fair to say that the kernel options listed above are the only thing really blocking this from working today.

  • Like 1
Link to comment

Surprised there is not more activity from others in this thread, it would be amazing to be able to virtualise integrated graphics in the same way as the rest of the cpu and have multiple vms benefit from it.  I would ask you for more details to try myself but I didn't think something like this would be possible and now I am running xeons.

Link to comment
9 minutes ago, flaggart said:

Surprised there is not more activity from others in this thread, it would be amazing to be able to virtualise integrated graphics in the same way as the rest of the cpu and have multiple vms benefit from it.  I would ask you for more details to try myself but I didn't think something like this would be possible and now I am running xeons.

Yeah, it doesn't seem like there is a lot of interest I guess. That's kind of why I set out to try it myself. I actually didn't think I was going to get it to work at all, much less transfer the setup to unraid. I'd given up on the idea since coffeelake support was initially not going to happen and then finally showed up in 5.1 kernel.

 

Next step is to mod my bios to support increased aperture sizes. That's a large problem for anyone running this on consumer motherboards, the mediated GPUs take their slice from the GPU aperture pie, not the shared variable memory. While changing the aperture size is a supported option, most motherboards seem to just lock it to 256MB. This means I can only create 2 of the smallest type of virtual GPU at the moment.

  • Like 1
Link to comment
  • 2 weeks later...

Another component I've found that is missing is unraid's qemu appears to not be compiled with opengl support. This isn't required to make gvt-g function, but there is a dmabuf function that requires it to link the VM display to the mediated card. You can use a client installed vnc, rdp...maybe looking glass, instead but its a real QoL improvement that isn't there.

 

That said, given how unraid is built its not clear to me if openGL would actually work. The i915 driver must be loaded for this to work, but that might not be the only backend piece missing.

  • Like 1
Link to comment

LT has officially mentioned that 5.x kernel won't be in 6.8.x due to this bug:

 

So you will have to wait for 6.9.0 at the earliest. (coming soon™)

 

As to why there's limited interest, I can venture to speculate.

  • Support for the tech itself is still limited. AFAIK, it needs a few hoops to work with Ubuntu 18.04 (LTS) and need lowest 19.04 for full out of the box support.
  • AMD has been kicking Intel in the groin lately so asking for an Intel-only feature probably won't get as much buzz.
  • Nvidia support is limited to Quadro RTX (and Tesla!, who has the dough?) so again, probably won't get as much buzz.

TL;DR: a very niche bleeding edge feature that LT probably will only include out of convenience.

 

Link to comment
  • 5 months later...
On 1/3/2020 at 12:43 PM, scorcho99 said:

Yeah, it doesn't seem like there is a lot of interest I guess. That's kind of why I set out to try it myself. I actually didn't think I was going to get it to work at all, much less transfer the setup to unraid. I'd given up on the idea since coffeelake support was initially not going to happen and then finally showed up in 5.1 kernel.

 

Next step is to mod my bios to support increased aperture sizes. That's a large problem for anyone running this on consumer motherboards, the mediated GPUs take their slice from the GPU aperture pie, not the shared variable memory. While changing the aperture size is a supported option, most motherboards seem to just lock it to 256MB. This means I can only create 2 of the smallest type of virtual GPU at the moment.

I stumbled in here wondering about slicing GPU compute for unraid like theyu do for enterprised virtualized cards, and wondering if its possible in unraid.

Link to comment
  • 4 weeks later...

Is there any news about GVT-G on unraid? I've upgraded my home server and want to consolidate all services into vms running off of it. I like the look of unraid but lack of official support for GVT-G would be a deal-breaker for me (need it for GPU assisted trans-coding on media and cctv servers). If I would need to re-complie the kernel myself at every update I may as well just roll my own solution with proxmox.

On 1/3/2020 at 5:43 PM, scorcho99 said:

Next step is to mod my bios to support increased aperture sizes. That's a large problem for anyone running this on consumer motherboards, the mediated GPUs take their slice from the GPU aperture pie, not the shared variable memory. While changing the aperture size is a supported option, most motherboards seem to just lock it to 256MB. This means I can only create 2 of the smallest type of virtual GPU at the moment.

@scorcho99 - you probably worked this out yourself since you last posted, but it isn't necessary to flash a modded bios just to increase the GPU aperture. If the Aperture Size option is hidden in your bios you can probably still modify it from a UEFI shell. Take a look at these instructions in the Opencore documentation

 

The opencore people are interested in changing a different setting (CFG Lock) just substitute "Aperture" for every instance of "CFG Lock" in the instructions, the process is the same.

 

For example, following the instructions from opencore, I found this entry in the file created by running ifrextract on my bios setup.bin:

0x3A140 One Of: Aperture Size, VarStoreInfo (VarOffset/VarName): 0x2B9, VarStore: 0x1, QuestionId: 0x2759, Size: 1, Min: 0x0, Max 0x1F, Step: 0x0 {05 91 AC 07 AD 07 59 27 01 00 B9 02 14 10 00 1F 00}
0x3A151 One Of Option: 128MB, Value (8 bit): 0x0 {09 07 AE 07 00 00 00}
0x3A158 One Of Option: 256MB, Value (8 bit): 0x1 (default) {09 07 AF 07 30 00 01}
0x3A15F One Of Option: 512MB, Value (8 bit): 0x3 {09 07 B0 07 00 00 03}
0x3A166 One Of Option: 1024MB, Value (8 bit): 0x7 {09 07 B1 07 00 00 07}
0x3A16D One Of Option: 2048MB, Value (8 bit): 0xF {09 07 B2 07 00 00 0F}
0x3A174 One Of Option: 4096MB, Value (8 bit): 0x1F {09 07 B3 07 00 00 1F}
0x3A17B End One Of {29 02}

So to increase the aperture on my system to 1024MB I issued the following command in the modified grub UEFI shell:

setup_var 0x2B9 0x7

NOTE: this is BIOS specific, the settings NVRAM memory address will be different for every motherboard, possibly even changing between BIOS versions, so you need to follow the full instructions in the opencore documentation, just copy/pasting the above command I used for my board will not work and could leave your system unable to boot.

Edited by drbob
Link to comment
16 minutes ago, drbob said:

Is there any news about GVT-G on unraid? I've upgraded my home server and want to consolidate all services into vms running off of it. I like the look of unraid but lack of official support for GVT-G would be a deal-breaker for me (need it for GPU assisted trans-coding on media and cctv servers). If I would need to re-complie the kernel myself at every update I may as well just roll my own solution with proxmox.

@scorcho99 - you probably worked this out yourself since you last posted, but it isn't necessary to flash a modded bios just to increase the GPU aperture. If the Aperture Size option is hidden in your bios you can probably still modify it from a UEFI shell. Take a look at these instructions in the Opencore documentation

 

The opencore people are interested in changing a different setting (CFG Lock) just substitute "Aperture" for every instance of "CFG Lock" in the instructions, the process is the same.

 

For example, following the instructions from opencore, I found this entry in the file created by running ifrextract on my bios setup.bin:


0x3A140 One Of: Aperture Size, VarStoreInfo (VarOffset/VarName): 0x2B9, VarStore: 0x1, QuestionId: 0x2759, Size: 1, Min: 0x0, Max 0x1F, Step: 0x0 {05 91 AC 07 AD 07 59 27 01 00 B9 02 14 10 00 1F 00}
0x3A151 One Of Option: 128MB, Value (8 bit): 0x0 {09 07 AE 07 00 00 00}
0x3A158 One Of Option: 256MB, Value (8 bit): 0x1 (default) {09 07 AF 07 30 00 01}
0x3A15F One Of Option: 512MB, Value (8 bit): 0x3 {09 07 B0 07 00 00 03}
0x3A166 One Of Option: 1024MB, Value (8 bit): 0x7 {09 07 B1 07 00 00 07}
0x3A16D One Of Option: 2048MB, Value (8 bit): 0xF {09 07 B2 07 00 00 0F}
0x3A174 One Of Option: 4096MB, Value (8 bit): 0x1F {09 07 B3 07 00 00 1F}
0x3A17B End One Of {29 02}

So to increase the aperture on my system to 1024MB I issued the following command in the modified grub UEFI shell:


setup_var 0x2B9 0x7

NOTE: this is BIOS specific, the settings NVRAM memory address will be different for every motherboard, possibly even changing between BIOS versions, so you need to follow the full instructions in the opencore documentation, just copy/pasting the above command I used for my board will not work and could leave your system unable to boot.

Good tip on the uefi shell. I think I actually had to do that when I installed the modded bios to disable write protection or something, I didn't get that far but I did wonder at the time if I could have just used the shell to change the option. I don't know much about bios modding (more than I did now) A guy on win-raid forums created the bios mod for me and I sent him a amazon card as thanks.

 

Anyway, I ultimately decided against using GVT-g, but it was a fun experiment...required me to figure out a lot of pieces I didn't know anything about. It was a death of a thousand cuts for final purpose though. In addition to having pretty weak performance potential, it didn't support old Windows OSes, without openGL in unraid I couldn't get it to display in spice, it didn't support saving VM states and I believe I had a fair amount of trouble with it in linux guests? I can't remember. And there is (currently) no (straightforward) way to direct output to another display which I wanted. It just wasn't a great fit for any of the use cases I'd dreamed up for it. Maybe after it bakes a little longer.

 

I ended up selling my Intel hardware and went with a Ryzen setup for more cores. I ended up deciding that a mining riser with a bunch of zerocore supporting AMD graphics cards was more flexible even if it is a big cabling mess. Only thing I feel like I'm missing out on is quicksync which is pretty good.

 

 

Link to comment

Interesting to hear your experience, it underlines how unraid would need to officially support GVT-G for it to be worthwhile for me. 

20 minutes ago, scorcho99 said:

Only thing I feel like I'm missing out on is quicksync which is pretty good.

Funny how our use cases differ, my reason for wanting GVT-G is all about having quicksync in mulitple VMs, with no need for any direct display output!

Edited by drbob
  • Like 1
Link to comment
  • 6 months later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.