Acidcliff Posted January 12, 2022 Share Posted January 12, 2022 (edited) Hi all, I'm trying to use the SR-IOV feature of the iGPU of the intel 12600k (VGA function is device 0000:00:02.0). I'm on 6.10RC2 using an intel 12600k on a MSI Z690 PRO A This is my /boot/config/go #!/bin/bash # Start the Management Utility /usr/local/sbin/emhttp & echo 3 > /sys/bus/pci/devices/0000:00:02.0/sriov_numvfs # Relaunch vfio-pci script to bind virtual function adapters that didn't exist at boot time /usr/local/sbin/vfio-pci >>/var/log/vfio-pci This is my /boot/syslinux/syslinux.cfg default menu.c32 menu title Lime Technology, Inc. prompt 0 timeout 50 label Unraid OS menu default kernel /bzimage i915.force_probe=4680 vfio-pci.enable_sriov=1 append isolcpus=0-11 video=efifb:off initrd=/bzroot intel_iommu=on label Unraid OS GUI Mode kernel /bzimage append isolcpus=0-11 initrd=/bzroot,/bzroot-gui label Unraid OS Safe Mode (no plugins, no GUI) kernel /bzimage append initrd=/bzroot unraidsafemode label Unraid OS GUI Safe Mode (no plugins) kernel /bzimage append initrd=/bzroot,/bzroot-gui unraidsafemode label Memtest86+ kernel /memtest The log from the vfio-pci binding Loading config from /boot/config/vfio-pci.cfg BIND=0000:00:02.0|8086:4680 0000:00:02.1|8086:4680 0000:00:02.2|8086:4680 0000:00:02.3|8086:4680 --- Processing 0000:00:02.0 8086:4680 Vendor:Device 8086:4680 found at 0000:00:02.0 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:00:02.0/iommu_group/devices/0000:00:02.0 Binding... Successfully bound the device 8086:4680 at 0000:00:02.0 to vfio-pci --- Processing 0000:00:02.1 8086:4680 Error: Device 0000:00:02.1 does not exist, unable to bind device --- Processing 0000:00:02.2 8086:4680 Error: Device 0000:00:02.2 does not exist, unable to bind device --- Processing 0000:00:02.3 8086:4680 Error: Device 0000:00:02.3 does not exist, unable to bind device --- vfio-pci binding complete Devices listed in /sys/bus/pci/drivers/vfio-pci: lrwxrwxrwx 1 root root 0 Jan 12 21:51 0000:00:02.0 -> ../../../../devices/pci0000:00/0000:00:02.0 Loading config from /boot/config/vfio-pci.cfg BIND=0000:00:02.0|8086:4680 0000:00:02.1|8086:4680 0000:00:02.2|8086:4680 0000:00:02.3|8086:4680 --- Processing 0000:00:02.0 8086:4680 Vendor:Device 8086:4680 found at 0000:00:02.0 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:00:02.0/iommu_group/devices/0000:00:02.0 Binding... 0000:00:02.0 already bound to vfio-pci Successfully bound the device 8086:4680 at 0000:00:02.0 to vfio-pci --- Processing 0000:00:02.1 8086:4680 Vendor:Device 8086:4680 found at 0000:00:02.1 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:00:02.1/iommu_group/devices/0000:00:02.1 Binding... 0000:00:02.1 already bound to vfio-pci Successfully bound the device 8086:4680 at 0000:00:02.1 to vfio-pci --- Processing 0000:00:02.2 8086:4680 Vendor:Device 8086:4680 found at 0000:00:02.2 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:00:02.2/iommu_group/devices/0000:00:02.2 Binding... 0000:00:02.2 already bound to vfio-pci Successfully bound the device 8086:4680 at 0000:00:02.2 to vfio-pci --- Processing 0000:00:02.3 8086:4680 Vendor:Device 8086:4680 found at 0000:00:02.3 IOMMU group members (sans bridges): /sys/bus/pci/devices/0000:00:02.3/iommu_group/devices/0000:00:02.3 Binding... 0000:00:02.3 already bound to vfio-pci Successfully bound the device 8086:4680 at 0000:00:02.3 to vfio-pci --- vfio-pci binding complete Devices listed in /sys/bus/pci/drivers/vfio-pci: lrwxrwxrwx 1 root root 0 Jan 12 13:51 0000:00:02.0 -> ../../../../devices/pci0000:00/0000:00:02.0 lrwxrwxrwx 1 root root 0 Jan 12 13:52 0000:00:02.1 -> ../../../../devices/pci0000:00/0000:00:02.1 lrwxrwxrwx 1 root root 0 Jan 12 13:52 0000:00:02.2 -> ../../../../devices/pci0000:00/0000:00:02.2 lrwxrwxrwx 1 root root 0 Jan 12 13:52 0000:00:02.3 -> ../../../../devices/pci0000:00/0000:00:02.3 So the virtual function seem to be established end binded to vfio-pci (Kernel driver in use is vfio-pci): The devices are also showing in the GPU DropDown in the VM-Config: When I try to start the VM with any VF other than 00:02.0 I get the following error-message: internal error: qemu unexpectedly closed the monitor: 2022-01-12T22:18:26.174166Z qemu-system-x86_64: -device vfio-pci,host=0000:00:02.1,id=hostdev0,bus=pci.0,addr=0x2.0x1: vfio 0000:00:02.1: error getting device from group 14: Permission denied Verify all devices in group 14 are bound to vfio- or pci-stub and not already in use But I guess this error message is generic - in DMESG I get the following error after starting the VM, which I think is the real error: [ 640.917885] vfio-pci 0000:00:02.3: VF token required to access device By god I tried my best to find anything about that "VF token" thing, but I wasn't able to find much. Does anyone have a clue how get around that error? Edited January 12, 2022 by Acidcliff Forgot to mention my Unraid Version 1 Quote Link to comment
BVD Posted January 13, 2022 Share Posted January 13, 2022 The kernel driver isnt ready for 12th gen at least as far as gpu partitioning is concerned. Haven't tried it with bleeding edge kernels yet or anything, but at least as of 5.13, the latest "clean/stable" iGPU supported is 10th gen. Quote Link to comment
monkiq Posted March 12, 2022 Share Posted March 12, 2022 unraid 6.10 rc3 use the new linux kernel 5.15.27, will the new version support the sr-iov ? Quote Link to comment
eatoff Posted March 18, 2022 Share Posted March 18, 2022 On 3/12/2022 at 2:04 PM, monkiq said: unraid 6.10 rc3 use the new linux kernel 5.15.27, will the new version support the sr-iov ? Would love to know this. I have a 11th gen on the way, and it would be awesome to be able to use the HDMI output for a VM, but still use the GPU for hardware transcoding in Plex. I'm running RC3 currently, but on an AMD system Quote Link to comment
DesertCookie Posted May 30, 2022 Share Posted May 30, 2022 On 3/18/2022 at 4:03 AM, eatoff said: Would love to know this. I have a 11th gen on the way [...] I have an 11600K sitting in my system right now and just realized I can't use GVT-g. I'm really interested in this too. Quote Link to comment
Zonediver Posted May 30, 2022 Share Posted May 30, 2022 As mentioned: The latest "clean/stable" iGPU supported is 10th gen... Quote Link to comment
smurofdiarnu Posted May 30, 2022 Share Posted May 30, 2022 (edited) What do you guys make of this comment? https://github.com/intel/gvt-linux/issues/190#issuecomment-1139698936 Quote You can check the linux-intel-lts repo, it is also maintained by Intel, You can check this tag, https://github.com/intel/linux-intel-lts/releases/tag/lts-v5.10.100-yocto-220223T232711Z which is support Alder Lake platform GPU SRIOV. Edited August 1, 2022 by smurofdiarnu 2 Quote Link to comment
Acidcliff Posted May 31, 2022 Author Share Posted May 31, 2022 Looking at the code and commits to me it seems that Intel's LTS Kernel 5.10 has added SR-IOV support for the i915 driver. (Although nothing happened to this after the commit in on 24th Feb). Intels 5.15 LTS Kernel does not have it though. It's also not in the 5.18 Linux Kernel. Not sure in what state the updated SR-IOV driver in 5.10 is. I'm no Linux Pro so I'm also not sure if and how it is possible to compile the adapted driver and add it to the unraid kernel. But if it would be possible it would definitelly be cool 2 Quote Link to comment
jinlife Posted July 2, 2022 Share Posted July 2, 2022 (edited) On 5/31/2022 at 5:15 PM, Acidcliff said: Looking at the code and commits to me it seems that Intel's LTS Kernel 5.10 has added SR-IOV support for the i915 driver. (Although nothing happened to this after the commit in on 24th Feb). Intels 5.15 LTS Kernel does not have it though. It's also not in the 5.18 Linux Kernel. Not sure in what state the updated SR-IOV driver in 5.10 is. I'm no Linux Pro so I'm also not sure if and how it is possible to compile the adapted driver and add it to the unraid kernel. But if it would be possible it would definitelly be cool Hello Acidcliff, I built a kernel with the IOV code in intel LTS Kernel 5.15. It looks fine to run but I have no 12th CPU to test the i915 SR-IOV ability. Would you please try it? It was built for unraid 6.10.3, please just overwrite these files to the flash drive. Please backup original 4 files, copying them back will revert the change. https://drive.google.com/drive/folders/1bAnedRHWaz7QGQAkO7xWohhnjDxSfbqS?usp=sharing It is just experiment, may not work. Maybe the code is not the only condition. Not working!!! Edited December 14, 2022 by jinlife 1 Quote Link to comment
joykingdom Posted July 5, 2022 Share Posted July 5, 2022 On 7/2/2022 at 1:06 PM, jinlife said: Hello Acidcliff, I built a kernel with the IOV code in intel LTS Kernel 5.15. It looks fine to run but I have no 12th CPU to test the i915 SR-IOV ability. Would you please try it? It was built for unraid 6.10.3, please just overwrite these files to the flash drive. Please backup original 4 files, copying them back will revert the change. https://drive.google.com/drive/folders/1bAnedRHWaz7QGQAkO7xWohhnjDxSfbqS?usp=sharing It is just experiment, may not work. Maybe the code is not the only condition. Anyone has tried ？ Quote Link to comment
Acidcliff Posted July 5, 2022 Author Share Posted July 5, 2022 On 7/2/2022 at 7:06 AM, jinlife said: Hello Acidcliff, I built a kernel with the IOV code in intel LTS Kernel 5.15. It looks fine to run but I have no 12th CPU to test the i915 SR-IOV ability. Would you please try it? It was built for unraid 6.10.3, please just overwrite these files to the flash drive. Please backup original 4 files, copying them back will revert the change. https://drive.google.com/drive/folders/1bAnedRHWaz7QGQAkO7xWohhnjDxSfbqS?usp=sharing It is just experiment, may not work. Maybe the code is not the only condition. I think I will be able to test it on the weekend! Quote Link to comment
jinlife Posted July 8, 2022 Share Posted July 8, 2022 On 7/6/2022 at 4:06 AM, Acidcliff said: I think I will be able to test it on the weekend! Awesome, thanks for trying. Looking forward to the result, otherwise, there is still long way to go. Quote Link to comment
smurofdiarnu Posted July 8, 2022 Share Posted July 8, 2022 Quote krispan-intel commented yesterday >SRIOV upstream task is on-going, ETA Q4'23 https://github.com/intel/linux-intel-lts/issues/33#issuecomment-1176996341 Wow so far away... kinda makes we wish I went for AMD instead now since IIRC I saw people passing through their 5600G iGPUs etc. 1 Quote Link to comment
SimonF Posted July 8, 2022 Share Posted July 8, 2022 On 7/2/2022 at 6:06 AM, jinlife said: Hello Acidcliff, I built a kernel with the IOV code in intel LTS Kernel 5.15. It looks fine to run but I have no 12th CPU to test the i915 SR-IOV ability. Would you please try it? It was built for unraid 6.10.3, please just overwrite these files to the flash drive. Please backup original 4 files, copying them back will revert the change. https://drive.google.com/drive/folders/1bAnedRHWaz7QGQAkO7xWohhnjDxSfbqS?usp=sharing It is just experiment, may not work. Maybe the code is not the only condition. Are you able to advise which kernel values you may have changed? Quote Link to comment
jinlife Posted July 9, 2022 Share Posted July 9, 2022 14 hours ago, SimonF said: Are you able to advise which kernel values you may have changed? I just built the whole kernel from intel LTS Kernel 5.15 which has the SR-IOV code. It may be not stable, but we could find out whether the SR-IOV code works fine or it is just earlier stage code. 1 Quote Link to comment
Da_iaji Posted July 10, 2022 Share Posted July 10, 2022 On 7/2/2022 at 1:06 PM, jinlife said: Hello Acidcliff, I built a kernel with the IOV code in intel LTS Kernel 5.15. It looks fine to run but I have no 12th CPU to test the i915 SR-IOV ability. Would you please try it? It was built for unraid 6.10.3, please just overwrite these files to the flash drive. Please backup original 4 files, copying them back will revert the change. https://drive.google.com/drive/folders/1bAnedRHWaz7QGQAkO7xWohhnjDxSfbqS?usp=sharing It is just experiment, may not work. Maybe the code is not the only condition. It seems that only branches with yocto fields in the tags have SR-IOV related code, but I checked and it seems that only linux510 tags have yocto fields, did you port the i915 SR-IOV patch for linux510 to linux515? Quote Link to comment
jinlife Posted July 26, 2022 Share Posted July 26, 2022 (edited) On 7/10/2022 at 6:13 PM, Da_iaji said: It seems that only branches with yocto fields in the tags have SR-IOV related code, but I checked and it seems that only linux510 tags have yocto fields, did you port the i915 SR-IOV patch for linux510 to linux515? There are 2 branches of 5.15 have the SR-IOV code. I guess you need to check again. Anyway, no one test it yet. I guess not many people need it. Edited July 26, 2022 by jinlife Quote Link to comment
eatoff Posted July 27, 2022 Share Posted July 27, 2022 I'm not that knowledgeable about Linux, but is the SR-IOV compatibility built into the new 6.11 RC1? Quote Link to comment
eatoff Posted July 30, 2022 Share Posted July 30, 2022 On 7/26/2022 at 1:00 PM, jinlife said: There are 2 branches of 5.15 have the SR-IOV code. I guess you need to check again. Anyway, no one test it yet. I guess not many people need it. I think people are scared to test, I know I am. I have 1 Unraid server and if it goes down then that's my 1 server. Still want SR-IOV to work though Quote Link to comment
domrockt Posted July 30, 2022 Share Posted July 30, 2022 Short question? When i try this could i use one of the vigpus for docker? And the rest for vm‘s? Quote Link to comment
smurofdiarnu Posted August 1, 2022 Share Posted August 1, 2022 (edited) On 7/10/2022 at 7:13 PM, Da_iaji said: It seems that only branches with yocto fields in the tags have SR-IOV related code, but I checked and it seems that only linux510 tags have yocto fields, did you port the i915 SR-IOV patch for linux510 to linux515? If you checkout the 5.15/linux branch then it does show some SRIOV commits related to i915: git log --author='intel.com' --all --grep='SRIOV' but then again it's hard to tell whether it includes the latest work or not... Edited August 1, 2022 by smurofdiarnu Quote Link to comment
jinlife Posted August 1, 2022 Share Posted August 1, 2022 (edited) On 7/28/2022 at 6:36 AM, eatoff said: I'm not that knowledgeable about Linux, but is the SR-IOV compatibility built into the new 6.11 RC1? Unraid 6.11 RC1 is based on Linux 5.18 branch but intel only has 5.15 as the latest one for now. There are many customized codes in intel's repository, I would say let's wait for intel to update it. Edited August 18, 2022 by jinlife Quote Link to comment
Sophon Posted September 26, 2022 Share Posted September 26, 2022 (edited) Hi, I managed to assign SR-IOV VFs to VM using Intel's WIP driver mentioned above, though I'm testing with Arch Linux. More about it here: https://www.reddit.com/r/VFIO/comments/xoeika/sriov_on_intel_uhd_graphics_770_xe_12th_gen/ https://github.com/strongtz/i915-sriov-dkms Edited September 26, 2022 by Sophon 1 Quote Link to comment
always67 Posted November 4, 2022 Share Posted November 4, 2022 Here is a guide for PVE Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.