Jorgen Posted July 20 Share Posted July 20 (edited) Since I got my i5-12400 CPU in a recent hardware refresh, I've struggled to pass through the integrated intel GPU (and audio) to a Windows 10 VM. I wanted the VM to use the iGPU with an attached monitor for HDMI output, but most guides I came across were either for older generation CPUs or were about virtual GPU usage via SR-IOV (which is also cool, but not what I wanted to achieve). I was able to isolate the iGPU and pass it through to a Win10 VM, installing the intel drivers, but it always ended with a code 43 error in the device manager, a black screen and this error in the VM logs: qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:00:02.0 Device option ROM contents are probably invalid (check dmesg). Skip option ROM probe with rombar=0, or load from file with romfile= Setting rombar=0 did remove the error from the log, but did not resolve code 43 or the black screen. However, it did lead me to this guide: https://www.cnx-software.com/2023/12/17/how-to-use-a-hdmi-monitor-usb-mouse-keyboard-in-promox-ve-on-an-intel-alder-lake-n-mini-pc/ And this github repo containing rom files for gen 12+ intel iGPUs! https://github.com/gangqizai/igd The instructions in the repo are all in Chinese, but a bit of google translate made it pretty clear how to use the files. After some trial and error, I've managed to get the iGPU passed through to a Win10 VM on unRAID and outputting to the monitor via HDMI. Happy days. Please read these notes from the repo maintainer before trying this! Quote Usage restrictions This ROM does not support commercial use and is only for technical research by DIY enthusiasts. This ROM only supports Intel core graphics and does not support AMD Only supports UEFI, normal boot. Secure boot is not supported yet. Only supports OVMF mode, seabios does not support it The memory must be at least 4G. If it is less than 4G, there may be problems. Pay attention to the BIOS settings: DVMT pre allocated, do not exceed 64M, 64M corresponds to x-igd-gms=0x2, if it exceeds 64M, x-igd-gms must be increased! Here are the steps/settings that worked for me on unRAID with my specific hardware. UnRAID settings: version: 6.12.11 intel_gpu_top plugin installed Host BIOS settings that MIGHT be relevant (ASRock Z690M-ITX/ax, i5-12400): VT-d: Enabled Primary graphics adapter: Onboard Above 4G Decoding: Enabled C.A.M. Clever access memory: Enabled SR-IOV support: Enabled Share Memory: 64M (DVMT pre-allocated) IGPU Multi-monitor: Enabled Steps Download the two ROM files from the GitHub repo and save to the domains share. I used /mnt/cache/domains/iGPU_ROM_files/ gen12_igd.rom (for the iGPU) gen12_gop.rom (for the sound card) Bind the iGPU and soundcard (and the rest of the devices in the same iommu group in my case) to VFIO in system devices Plug monitor into iGPU HDMI and/or DP ports Reboot unRAID Create a new Windows 10 VM with NoVNC graphics card, CPU host passthrough, i440fx-7.2 and OVMF Install Windows with NoVNC, install standard virtio drivers Shut down VM Add iGPU as a second graphics card for the VM in UI editor Add intel soundcard in UI editor Start VM Confirm iGPU is visible in device manager Download and install intel iGPU driver (I used https://www.intel.com/content/www/us/en/support/detect.html) Shut down VM Remove NoVNC from VM in UI editor and set iGPU as primary graphics card Browse for and set graphics card ROM file in UI editor to gen12_igd.rom Save changes Edit VM XML and add gen12_gop.rom to soundcard, following the syntax of the iGPU from step above I also added multifunction=on for the iGPU and soundcard, and changed the slot of the soundcard to be the same as the iGPU, but I don't know if either of these are actually required The relevant part of the VM XML looks like this now: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </source> <rom file='/mnt/cache/domains/iGPU_ROM_files/gen12_igd.rom'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x00' slot='0x1f' function='0x3'/> </source> <rom file='/mnt/cache/domains/iGPU_ROM_files/gen12_gop.rom'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1' multifunction='on'/> </hostdev> At the top of the XML, replace this line <domain type='kvm'> with <domain type='kvm' id='14' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> To allow us to add Qemu overrides commands to the XML Add this block of code at the end of the XML, just before the closing </domain> tag <qemu:override> <qemu:device alias='hostdev0'> <qemu:frontend> <qemu:property name='x-igd-opregion' type='bool' value='true'/> <qemu:property name='x-igd-gms' type='unsigned' value='2'/> </qemu:frontend> </qemu:device> </qemu:override> Start the VM and enjoy monitor output from the iGPU! Edited August 13 by Jorgen Spelling 2 Quote Link to comment
alayavijnana Posted July 20 Share Posted July 20 Does doing this make the iGPU inaccessible to docker containers, such as Jellyfin, for transcoding tasks? Quote Link to comment
Jorgen Posted July 20 Author Share Posted July 20 I don’t know for sure, but probably yes. Quote Link to comment
SimonF Posted July 20 Share Posted July 20 29 minutes ago, alayavijnana said: Does doing this make the iGPU inaccessible to docker containers, such as Jellyfin, for transcoding tasks? Yes any passthru makes the gpu not available in the host. 1 Quote Link to comment
alayavijnana Posted July 20 Share Posted July 20 Thanks guys! Disclaimer: I’m a newb. I have an Intel i5 12600k on a AsRock z690 Pro RS motherboard and am hoping to set up my Unraid server to handle docker containers (Jellyfin and Immich) whilst also running a Win11 VM for light gaming and a Linux VM for “daily driving”. When I get a discrete GPU (leaving the iGPU to handle the docker container tasks) can I leverage it for both VMs at the same time? Or is passthrough of a discrete GPU (or any PCIe controller for that matter) only limited to one VM at a time? Quote Link to comment
Jorgen Posted July 20 Author Share Posted July 20 Only one (running) VM at the time for GPU passthrough with display output. Quote Link to comment
Jorgen Posted July 26 Author Share Posted July 26 I've updated the BIOS settings after further reading, and to get rid of error messages related to resizable memory allocation in the unraid boot logs. New setting that works well so far: Above 4G Decoding: Enabled C.A.M. Clever access memory: Enabled Share Memory: 64M (DVMT pre-allocated) I'm now running dual monitor output in my Win10 VM from the iGPU, via HDMI and Displayport. No real-life problems so far, but I'm getting the below in the unraid logs when I start the VM: Tower kernel: vfio-pci 0000:00:02.0: vfio_ecap_init: hiding ecap 0x1b@0x100 Tower kernel: vfio-pci 0000:00:02.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0x7b09 Tower kernel: vfio-pci 0000:00:02.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0x7b09 Tower kernel: vfio-pci 0000:00:02.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0x7b09 Tower kernel: vfio-pci 0000:00:02.0: Invalid PCI ROM header signature: expecting 0xaa55, got 0x7b09 The first line is just reporting that some native extended capability of the iGPU is not being passed through to the VM. Not sure which one yet, but it doesn't seem to cause any issues. I will investigate ROM header signature warning and see if I should raise this on the ROM github page or if there's something else I'm missing. Again, doesn't seem to affect anything in practice, so priority is low. Quote Link to comment
Jorgen Posted July 29 Author Share Posted July 29 (edited) Was experiencing severe flickering in MS Teams app today, when watching a shared screen. It could be related to hardware transcoding, but no conclusive evidence. However, my syslog got spammed with large sections of this today, which I think corresponds with the Teams flickering episodes. Can't be sure though, will watch the log next time it happens. Jul 29 09:02:29 Tower kernel: dmar_fault: 3204 callbacks suppressed Jul 29 09:02:29 Tower kernel: DMAR: DRHD: handling fault status reg 3 Jul 29 09:02:29 Tower kernel: DMAR: [DMA Read NO_PASID] Request device [00:02.0] fault addr 0x70680000 [fault reason 0x06] PTE Read access is not set Jul 29 09:02:29 Tower kernel: DMAR: DRHD: handling fault status reg 3 Jul 29 09:02:29 Tower kernel: DMAR: [DMA Read NO_PASID] Request device [00:02.0] fault addr 0x70680000 [fault reason 0x06] PTE Read access is not set Jul 29 09:02:29 Tower kernel: DMAR: DRHD: handling fault status reg 3 Jul 29 09:02:29 Tower kernel: DMAR: [DMA Read NO_PASID] Request device [00:02.0] fault addr 0x70680000 [fault reason 0x06] PTE Read access is not set Jul 29 09:02:29 Tower kernel: DMAR: DRHD: handling fault status reg 3 Jul 29 09:03:43 Tower kernel: dmar_fault: 2705 callbacks suppressed EDIT: seems related to the amount of memory allocated to the iGPU: https://forum.proxmox.com/threads/dmar-dma-read-no_pasid-request-device-00-02-0-fault-addr-0xc8e8c000-fault-reason-0x06-pte-read-access-is-not-set-intel-610-integrated-graphic.128012/ I'll try increasing it in BIOS and work out what the ROM repo maintainer means with: Quote Pay attention to the BIOS settings: DVMT pre allocated, do not exceed 64M, 64M corresponds to x-igd-gms=0x2, if it exceeds 64M, x-igd-gms must be increased! Edited July 29 by Jorgen new info found Quote Link to comment
Jorgen Posted July 30 Author Share Posted July 30 On 7/29/2024 at 9:42 PM, Jorgen said: I'll try increasing it in BIOS and work out what the ROM repo maintainer means with: Quote Pay attention to the BIOS settings: DVMT pre allocated, do not exceed 64M, 64M corresponds to x-igd-gms=0x2, if it exceeds 64M, x-igd-gms must be increased! The flickering is definitely related to the log errors, they happen at the same time. No flickering, no error. Just increasing the shared memory (DVMT pre-allocated) in BIOS with no other changes did not improve things. I'm struggling to convert the ProxMox formatted arguments below to something unraid will accept in the XML. All three of these are needed according to the GitHub page, and I have none of them added. args: -set device.hostpci0.addr=02.0 -set device.hostpci0.x-igd-gms=0x2 -set device.hostpci0.x-igd-opregion=on Maybe something along the lines of https://docs.redhat.com/en/documentation/red_hat_enterprise_linux/6/html/virtualization_administration_guide/sub-sect-domain_commands-converting_qemu_arguments_to_domain_xml#sub-sect-Domain_Commands-Converting_QEMU_arguments_to_domain_XML Quote Link to comment
Jorgen Posted August 10 Author Share Posted August 10 On 7/30/2024 at 10:17 PM, Jorgen said: I'm struggling to convert the ProxMox formatted arguments below to something unraid will accept in the XML. All three of these are needed according to the GitHub page, and I have none of them added. args: -set device.hostpci0.addr=02.0 -set device.hostpci0.x-igd-gms=0x2 -set device.hostpci0.x-igd-opregion=on Ok, finally had some time to look into this. The need for both x-igd-gms=0x2 and x-igd-opregion=on is explained here https://github.com/qemu/qemu/blob/master/docs/igd-assign.txt. BUT that doc is 8 years old and written for older generation iGPUs. SR-IOV and UEFI seem to have changed things, but I haven't found any definitive sources to explain if and why these two lines are required for new generations. Nevertheless, the ROM github page calls them out as requirements, so who am I to argue. This is how to add them to an unraid VM config. First off, you need to edit the VM in XML mode. Replace <domain type='kvm'> with <domain type='kvm' id='14' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'> To allow us to add Qemu overrides commands to the XML Add this block of code at the end of the XML, just before the closing </domain> tag <qemu:override> <qemu:device alias='hostdev0'> <qemu:frontend> <qemu:property name='x-igd-opregion' type='bool' value='true'/> <qemu:property name='x-igd-gms' type='unsigned' value='2'/> </qemu:frontend> </qemu:device> </qemu:override> Unraid seems to automatically add hostdev alias names to the hostdev devices when the VM is running. For me, it correctly labelled the iGPU as hostdev0, but your milage may vary... This is what my iGPU and audio hostdevs look like in the XML while the VM is running: <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x00' slot='0x02' function='0x0'/> </source> <alias name='hostdev0'/> <rom file='/mnt/cache/domains/iGPU_ROM_files/gen12_igd.rom'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x00' slot='0x1f' function='0x3'/> </source> <alias name='hostdev1'/> <rom file='/mnt/cache/domains/iGPU_ROM_files/gen12_gop.rom'/> <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x1' multifunction='on'/> </hostdev> Since the flickering/log errors are intermittent, I'm not sure if this has resolved the issue. I've been running it like this for a few hours, and so far so good, even after stress testing by running 5 simultaneous 4K youtube streams at the same time. I also made two more changes to the system, which might have helped as well: Removed discrete GPU (GT710), the PCIe slot is now empty. Enabled HDMI sound for the iGPU in BIOS. Not sure why I had it disabled in the first place, or if it even mattered Will report back after a few more days of daily use Oh, I also spent a lot of time trying to work out how to support legacy mode, see instructions below, but it turns out unraid adds "-nodefaults" automatically so that was all taken care of already. Quote To make use of legacy mode, simply remove all other graphics options and use "-nographic" and either "-vga none" or "-nodefaults", along with adding the device using vfio-pci Quote Link to comment
Jorgen Posted August 13 Author Share Posted August 13 On 8/10/2024 at 9:18 PM, Jorgen said: Since the flickering/log errors are intermittent, I'm not sure if this has resolved the issue. I've been running it like this for a few hours, and so far so good, even after stress testing by running 5 simultaneous 4K youtube streams at the same time. I'm going to mark this as resolved, no more flickering or log errors for 2 days of daily use. It showed up many times a day before adding the qemu override lines. I'll update the first post with the required steps Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.