Intel iGPU SR-IOV on 12th Gen Alderlake UHD770 /GT1


Recommended Posts

Hi all,

 

I'm trying to use the SR-IOV feature of the iGPU of the intel 12600k (VGA function is device 0000:00:02.0).

 

I'm on 6.10RC2 using an intel 12600k on a MSI Z690 PRO A

 

This is my /boot/config/go

#!/bin/bash
# Start the Management Utility
/usr/local/sbin/emhttp &
echo 3 > /sys/bus/pci/devices/0000:00:02.0/sriov_numvfs
# Relaunch vfio-pci script to bind virtual function adapters that didn't exist at boot time
/usr/local/sbin/vfio-pci >>/var/log/vfio-pci

 

This is my /boot/syslinux/syslinux.cfg

default menu.c32
menu title Lime Technology, Inc.
prompt 0
timeout 50
label Unraid OS
  menu default
  kernel /bzimage i915.force_probe=4680 vfio-pci.enable_sriov=1
  append isolcpus=0-11 video=efifb:off initrd=/bzroot intel_iommu=on
label Unraid OS GUI Mode
  kernel /bzimage
  append isolcpus=0-11 initrd=/bzroot,/bzroot-gui
label Unraid OS Safe Mode (no plugins, no GUI)
  kernel /bzimage
  append initrd=/bzroot unraidsafemode
label Unraid OS GUI Safe Mode (no plugins)
  kernel /bzimage
  append initrd=/bzroot,/bzroot-gui unraidsafemode
label Memtest86+
  kernel /memtest

 

 

The log from the vfio-pci binding 

Loading config from /boot/config/vfio-pci.cfg
BIND=0000:00:02.0|8086:4680 0000:00:02.1|8086:4680 0000:00:02.2|8086:4680 0000:00:02.3|8086:4680
---
Processing 0000:00:02.0 8086:4680
Vendor:Device 8086:4680 found at 0000:00:02.0

IOMMU group members (sans bridges):
/sys/bus/pci/devices/0000:00:02.0/iommu_group/devices/0000:00:02.0

Binding...
Successfully bound the device 8086:4680 at 0000:00:02.0 to vfio-pci
---
Processing 0000:00:02.1 8086:4680
Error: Device 0000:00:02.1 does not exist, unable to bind device
---
Processing 0000:00:02.2 8086:4680
Error: Device 0000:00:02.2 does not exist, unable to bind device
---
Processing 0000:00:02.3 8086:4680
Error: Device 0000:00:02.3 does not exist, unable to bind device
---
vfio-pci binding complete

Devices listed in /sys/bus/pci/drivers/vfio-pci:
lrwxrwxrwx 1 root root 0 Jan 12 21:51 0000:00:02.0 -> ../../../../devices/pci0000:00/0000:00:02.0

Loading config from /boot/config/vfio-pci.cfg
BIND=0000:00:02.0|8086:4680 0000:00:02.1|8086:4680 0000:00:02.2|8086:4680 0000:00:02.3|8086:4680
---
Processing 0000:00:02.0 8086:4680
Vendor:Device 8086:4680 found at 0000:00:02.0

IOMMU group members (sans bridges):
/sys/bus/pci/devices/0000:00:02.0/iommu_group/devices/0000:00:02.0

Binding...
0000:00:02.0 already bound to vfio-pci
Successfully bound the device 8086:4680 at 0000:00:02.0 to vfio-pci
---
Processing 0000:00:02.1 8086:4680
Vendor:Device 8086:4680 found at 0000:00:02.1

IOMMU group members (sans bridges):
/sys/bus/pci/devices/0000:00:02.1/iommu_group/devices/0000:00:02.1

Binding...
0000:00:02.1 already bound to vfio-pci
Successfully bound the device 8086:4680 at 0000:00:02.1 to vfio-pci
---
Processing 0000:00:02.2 8086:4680
Vendor:Device 8086:4680 found at 0000:00:02.2

IOMMU group members (sans bridges):
/sys/bus/pci/devices/0000:00:02.2/iommu_group/devices/0000:00:02.2

Binding...
0000:00:02.2 already bound to vfio-pci
Successfully bound the device 8086:4680 at 0000:00:02.2 to vfio-pci
---
Processing 0000:00:02.3 8086:4680
Vendor:Device 8086:4680 found at 0000:00:02.3

IOMMU group members (sans bridges):
/sys/bus/pci/devices/0000:00:02.3/iommu_group/devices/0000:00:02.3

Binding...
0000:00:02.3 already bound to vfio-pci
Successfully bound the device 8086:4680 at 0000:00:02.3 to vfio-pci
---
vfio-pci binding complete

Devices listed in /sys/bus/pci/drivers/vfio-pci:
lrwxrwxrwx 1 root root 0 Jan 12 13:51 0000:00:02.0 -> ../../../../devices/pci0000:00/0000:00:02.0
lrwxrwxrwx 1 root root 0 Jan 12 13:52 0000:00:02.1 -> ../../../../devices/pci0000:00/0000:00:02.1
lrwxrwxrwx 1 root root 0 Jan 12 13:52 0000:00:02.2 -> ../../../../devices/pci0000:00/0000:00:02.2
lrwxrwxrwx 1 root root 0 Jan 12 13:52 0000:00:02.3 -> ../../../../devices/pci0000:00/0000:00:02.3

 

So the virtual function seem to be established end binded to vfio-pci (Kernel driver in use is vfio-pci):

IOMMU.thumb.PNG.64f6885b63446b8d4df86ff5b29660c1.PNG

 

The devices are also showing in the GPU DropDown in the VM-Config:

VMconf.PNG.e25ceb44be3b93cf7afc22c0c4dfa675.PNG

 

When I try to start the VM with any VF other than 00:02.0 I get the following error-message:

internal error: qemu unexpectedly closed the monitor: 2022-01-12T22:18:26.174166Z qemu-system-x86_64: -device vfio-pci,host=0000:00:02.1,id=hostdev0,bus=pci.0,addr=0x2.0x1: vfio 0000:00:02.1: error getting device from group 14: Permission denied Verify all devices in group 14 are bound to vfio- or pci-stub and not already in use

 

But I guess this error message is generic - in DMESG I get the following error after starting the VM, which I think is the real error: 

[  640.917885] vfio-pci 0000:00:02.3: VF token required to access device

 

By god I tried my best to find anything about that "VF token" thing, but I wasn't able to find much.

Does anyone have a clue how get around that error?

Edited by Acidcliff
Forgot to mention my Unraid Version
  • Thanks 1
Link to comment
  • 1 month later...
On 3/12/2022 at 2:04 PM, monkiq said:

unraid 6.10 rc3 use the new linux kernel 5.15.27, will the new version support the sr-iov ?

 

Would love to know this. I have a 11th gen on the way, and it would be awesome to be able to use the HDMI output for a VM, but still use the GPU for hardware transcoding in Plex.

 

I'm running RC3 currently, but on an AMD system

Link to comment
  • 2 months later...

What do you guys make of this comment?
 

https://github.com/intel/gvt-linux/issues/190#issuecomment-1139698936

Quote

You can check the linux-intel-lts repo, it is also maintained by Intel,
You can check this tag, https://github.com/intel/linux-intel-lts/releases/tag/lts-v5.10.100-yocto-220223T232711Z which is support Alder Lake platform GPU SRIOV.

Edited by smurofdiarnu
  • Like 2
Link to comment

Looking at the code and commits to me it seems that Intel's LTS Kernel 5.10 has added SR-IOV support for the i915 driver. (Although nothing happened to this after the commit in on 24th Feb). Intels 5.15 LTS Kernel does not have it though. It's also not in the 5.18 Linux Kernel.

 

Not sure in what state the updated SR-IOV driver in 5.10 is.

 

I'm no Linux Pro so I'm also not sure if and how it is possible to compile the adapted driver and add it to the unraid kernel. But if it would be possible it would definitelly be cool :D

  • Upvote 2
Link to comment
  • 1 month later...
On 5/31/2022 at 5:15 PM, Acidcliff said:

Looking at the code and commits to me it seems that Intel's LTS Kernel 5.10 has added SR-IOV support for the i915 driver. (Although nothing happened to this after the commit in on 24th Feb). Intels 5.15 LTS Kernel does not have it though. It's also not in the 5.18 Linux Kernel.

 

Not sure in what state the updated SR-IOV driver in 5.10 is.

 

I'm no Linux Pro so I'm also not sure if and how it is possible to compile the adapted driver and add it to the unraid kernel. But if it would be possible it would definitelly be cool :D

Hello Acidcliff, I built a kernel with the IOV code in intel LTS Kernel 5.15. It looks fine to run but I have no 12th CPU to test the i915 SR-IOV ability.

Would you please try it? It was built for unraid 6.10.3, please just overwrite these files to the flash drive.

Please backup original 4 files, copying them back will revert the change.

https://drive.google.com/drive/folders/1bAnedRHWaz7QGQAkO7xWohhnjDxSfbqS?usp=sharing

It is just experiment, may not work. Maybe the code is not the only condition.

Not working!!!

Edited by jinlife
  • Like 1
Link to comment
On 7/2/2022 at 1:06 PM, jinlife said:

Hello Acidcliff, I built a kernel with the IOV code in intel LTS Kernel 5.15. It looks fine to run but I have no 12th CPU to test the i915 SR-IOV ability.

Would you please try it? It was built for unraid 6.10.3, please just overwrite these files to the flash drive.

Please backup original 4 files, copying them back will revert the change.

https://drive.google.com/drive/folders/1bAnedRHWaz7QGQAkO7xWohhnjDxSfbqS?usp=sharing

It is just experiment, may not work. Maybe the code is not the only condition.

Anyone has tried ?

Link to comment
On 7/2/2022 at 7:06 AM, jinlife said:

Hello Acidcliff, I built a kernel with the IOV code in intel LTS Kernel 5.15. It looks fine to run but I have no 12th CPU to test the i915 SR-IOV ability.

Would you please try it? It was built for unraid 6.10.3, please just overwrite these files to the flash drive.

Please backup original 4 files, copying them back will revert the change.

https://drive.google.com/drive/folders/1bAnedRHWaz7QGQAkO7xWohhnjDxSfbqS?usp=sharing

It is just experiment, may not work. Maybe the code is not the only condition.

I think I will be able to test it on the weekend!

Link to comment
On 7/2/2022 at 6:06 AM, jinlife said:

Hello Acidcliff, I built a kernel with the IOV code in intel LTS Kernel 5.15. It looks fine to run but I have no 12th CPU to test the i915 SR-IOV ability.

Would you please try it? It was built for unraid 6.10.3, please just overwrite these files to the flash drive.

Please backup original 4 files, copying them back will revert the change.

https://drive.google.com/drive/folders/1bAnedRHWaz7QGQAkO7xWohhnjDxSfbqS?usp=sharing

It is just experiment, may not work. Maybe the code is not the only condition.

Are you able to advise which kernel values you may have changed?

Link to comment
14 hours ago, SimonF said:

Are you able to advise which kernel values you may have changed?

I just built the whole kernel from intel LTS Kernel 5.15 which has the SR-IOV code. It may be not stable, but we could find out whether the SR-IOV code works fine or it is just earlier stage code.

  • Thanks 1
Link to comment
On 7/2/2022 at 1:06 PM, jinlife said:

Hello Acidcliff, I built a kernel with the IOV code in intel LTS Kernel 5.15. It looks fine to run but I have no 12th CPU to test the i915 SR-IOV ability.

Would you please try it? It was built for unraid 6.10.3, please just overwrite these files to the flash drive.

Please backup original 4 files, copying them back will revert the change.

https://drive.google.com/drive/folders/1bAnedRHWaz7QGQAkO7xWohhnjDxSfbqS?usp=sharing

It is just experiment, may not work. Maybe the code is not the only condition.

It seems that only branches with yocto fields in the tags have SR-IOV related code, but I checked and it seems that only linux510 tags have yocto fields, did you port the i915 SR-IOV patch for linux510 to linux515?

Link to comment
  • 3 weeks later...
On 7/10/2022 at 6:13 PM, Da_iaji said:

It seems that only branches with yocto fields in the tags have SR-IOV related code, but I checked and it seems that only linux510 tags have yocto fields, did you port the i915 SR-IOV patch for linux510 to linux515?

There are 2 branches of 5.15 have the SR-IOV code. I guess you need to check again. 

Anyway, no one test it yet. I guess not many people need it.

Edited by jinlife
Link to comment
On 7/26/2022 at 1:00 PM, jinlife said:

There are 2 branches of 5.15 have the SR-IOV code. I guess you need to check again. 

Anyway, no one test it yet. I guess not many people need it.

I think people are scared to test, I know I am. I have 1 Unraid server and if it goes down then that's my 1 server. Still want SR-IOV to work though 

Link to comment
On 7/10/2022 at 7:13 PM, Da_iaji said:

It seems that only branches with yocto fields in the tags have SR-IOV related code, but I checked and it seems that only linux510 tags have yocto fields, did you port the i915 SR-IOV patch for linux510 to linux515?

 

If you checkout the 5.15/linux branch then it does show some SRIOV commits related to i915:

git log --author='intel.com' --all --grep='SRIOV'

but then again it's hard to tell whether it includes the latest work or not...

Edited by smurofdiarnu
Link to comment
On 7/28/2022 at 6:36 AM, eatoff said:

I'm not that knowledgeable about Linux, but is the SR-IOV compatibility built into the new 6.11 RC1?

Unraid 6.11 RC1 is based on Linux 5.18 branch but intel only has 5.15 as the latest one for now.

There are many customized codes in intel's repository, I would say let's wait for intel to update it.

Edited by jinlife
Link to comment
  • 3 weeks later...
  • 1 month later...
  • 1 month later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.