December 25, 20241 yr Community Expert 29 minutes ago, uniqf said: I followed all the guides I've manage to find (i think), and after battling for days I still can't get my RTX A4000 passover to my Ubuntu VM. The following are the extraction of the XML, would appreciate if someone would guide me in the right path, getting the famous 127 error message. <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/> </source> <rom file='/mnt/user/isos/rtxa4000.rom'/> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifunction='on'/> </hostdev> <hostdev mode='subsystem' type='pci' managed='yes'> <driver name='vfio'/> <source> <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/> </source> <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x1'/> </hostdev> <hostdev mode='subsystem' type='usb' managed='no'> <source> <vendor id='0x1547'/> <product id='0x1000'/> </source> <address type='usb' bus='0' port='1'/> </hostdev> <memballoon model='none'/> </devices> </domain> Any errors in the VM log?
December 25, 20241 yr hello @SimonF, i should've check more, but it present me with quite a fair bit of this logs which is likely associated to the GPU passthrough. I also attached a diagnostic log if that is helpful I've both Resizeable bar enabled/disabled, but i tried a clean restart of the Unraid server, and initiate a start of the VM configured with GPU passthrough, this is what I've observed. Dec 26 06:58:54 Tower kernel: br0: port 2(vnet0) entered blocking state Dec 26 06:58:54 Tower kernel: br0: port 2(vnet0) entered disabled state Dec 26 06:58:54 Tower kernel: device vnet0 entered promiscuous mode Dec 26 06:58:54 Tower kernel: br0: port 2(vnet0) entered blocking state Dec 26 06:58:54 Tower kernel: br0: port 2(vnet0) entered forwarding state Dec 26 06:58:57 Tower kernel: vfio-pci 0000:01:00.0: enabling device (0000 -> 0003) Dec 26 06:58:57 Tower kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x1e@0x258 Dec 26 06:58:57 Tower kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x19@0x900 Dec 26 06:58:57 Tower kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x26@0xc1c Dec 26 06:58:57 Tower kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x27@0xd00 Dec 26 06:58:57 Tower kernel: vfio-pci 0000:01:00.0: vfio_ecap_init: hiding ecap 0x25@0xe00 Dec 26 06:58:57 Tower kernel: vfio-pci 0000:01:00.1: enabling device (0000 -> 0002) Dec 26 06:58:57 Tower kernel: vfio-pci 0000:01:00.1: vfio_ecap_init: hiding ecap 0x25@0x160 Dec 26 06:58:58 Tower kernel: pcieport 0000:00:06.0: Data Link Layer Link Active not set in 1000 msec Dec 26 06:58:58 Tower kernel: vfio-pci 0000:01:00.1: vfio_bar_restore: reset recovery - restoring BARs Dec 26 06:58:58 Tower kernel: vfio-pci 0000:01:00.0: vfio_bar_restore: reset recovery - restoring BARs Dec 26 06:58:59 Tower kernel: vfio-pci 0000:01:00.0: timed out waiting for pending transaction; performing function level reset anyway Dec 26 06:59:00 Tower kernel: vfio-pci 0000:01:00.0: not ready 1023ms after FLR; waiting Dec 26 06:59:01 Tower kernel: vfio-pci 0000:01:00.0: not ready 2047ms after FLR; waiting Dec 26 06:59:03 Tower kernel: vfio-pci 0000:01:00.0: not ready 4095ms after FLR; waiting Dec 26 06:59:08 Tower kernel: vfio-pci 0000:01:00.0: not ready 8191ms after FLR; waiting Dec 26 06:59:16 Tower kernel: vfio-pci 0000:01:00.0: not ready 16383ms after FLR; waiting Dec 26 06:59:33 Tower kernel: vfio-pci 0000:01:00.0: not ready 32767ms after FLR; waiting Dec 26 07:00:08 Tower kernel: vfio-pci 0000:01:00.0: not ready 65535ms after FLR; giving up Dec 26 07:00:09 Tower kernel: vfio-pci 0000:01:00.0: vfio_bar_restore: reset recovery - restoring BARs Dec 26 07:00:09 Tower kernel: vfio-pci 0000:01:00.1: vfio_bar_restore: reset recovery - restoring BARs Dec 26 07:00:09 Tower kernel: vfio-pci 0000:01:00.0: vfio_bar_restore: reset recovery - restoring BARs Dec 26 07:00:09 Tower kernel: vfio-pci 0000:01:00.1: vfio_bar_restore: reset recovery - restoring BARs Dec 26 07:00:09 Tower kernel: vfio-pci 0000:01:00.0: vfio_bar_restore: reset recovery - restoring BARs Dec 26 07:00:09 Tower kernel: vfio-pci 0000:01:00.1: vfio_bar_restore: reset recovery - restoring BARs Dec 26 07:00:09 Tower kernel: vfio-pci 0000:01:00.0: vfio_bar_restore: reset recovery - restoring BARs Dec 26 07:00:09 Tower kernel: vfio-pci 0000:01:00.0: vfio_bar_restore: reset recovery - restoring BARs Dec 26 07:00:09 Tower kernel: vfio-pci 0000:01:00.0: vfio_bar_restore: reset recovery - restoring BARs Dec 26 07:00:09 Tower kernel: vfio-pci 0000:01:00.0: vfio_bar_restore: reset recovery - restoring BARs Dec 26 07:00:09 Tower kernel: vfio-pci 0000:01:00.0: vfio_bar_restore: reset recovery - restoring BARs Dec 26 07:00:09 Tower kernel: vfio-pci 0000:01:00.1: vfio_bar_restore: reset recovery - restoring BARs Dec 26 07:00:09 Tower kernel: vfio-pci 0000:01:00.1: vfio_bar_restore: reset recovery - restoring BARs Dec 26 07:00:09 Tower kernel: vfio-pci 0000:01:00.1: vfio_bar_restore: reset recovery - restoring BARs Dec 26 07:00:09 Tower kernel: vfio-pci 0000:01:00.0: vfio_bar_restore: reset recovery - restoring BARs Dec 26 07:00:09 Tower kernel: vfio-pci 0000:01:00.1: vfio_bar_restore: reset recovery - restoring BARs Dec 26 07:00:09 Tower kernel: vfio-pci 0000:01:00.0: vfio_bar_restore: reset recovery - restoring BARs Dec 26 07:00:09 Tower kernel: vfio-pci 0000:01:00.1: vfio_bar_restore: reset recovery - restoring BARs Dec 26 07:00:09 Tower kernel: vfio-pci 0000:01:00.0: vfio_bar_restore: reset recovery - restoring BARs Dec 26 07:00:09 Tower kernel: vfio-pci 0000:01:00.0: vfio_bar_restore: reset recovery - restoring BARs Dec 26 07:00:09 Tower kernel: vfio-pci 0000:01:00.1: vfio_bar_restore: reset recovery - restoring BARs Dec 26 07:00:09 Tower kernel: vfio-pci 0000:01:00.1: vfio_bar_restore: reset recovery - restoring BARs Dec 26 07:00:09 Tower kernel: vfio-pci 0000:01:00.0: vfio_bar_restore: reset recovery - restoring BARs Dec 26 07:00:09 Tower kernel: vfio-pci 0000:01:00.1: vfio_bar_restore: reset recovery - restoring BARs Dec 26 07:00:09 Tower kernel: vfio-pci 0000:01:00.0: vfio_bar_restore: reset recovery - restoring BARs Dec 26 07:00:09 Tower kernel: vfio-pci 0000:01:00.1: vfio_bar_restore: reset recovery - restoring BARs Dec 26 07:00:10 Tower kernel: vfio-pci 0000:01:00.0: vfio_bar_restore: reset recovery - restoring BARs Dec 26 07:00:10 Tower kernel: vfio-pci 0000:01:00.0: vfio_bar_restore: reset recovery - restoring BARs Dec 26 07:00:11 Tower kernel: usb 3-6: reset full-speed USB device number 3 using xhci_hcd tower-diagnostics-20241226-0703.zip Edited December 25, 20241 yr by uniqf
December 27, 20241 yr hello @SimonF, I've done the following too - Turning on/off Rsize bar, Above 4G encoding in bios - Append the syslinux with and without earlymodules=vfio-pci video=efifb:off,vesafb:off gfxpayload=text arguments - Swap GPU to various different slots of motherboard I just kept getting the same errors of the following: Dec 27 12:37:04 Tower kernel: vfio-pci 0000:01:00.0: not ready 1023ms after FLR; waiting Dec 27 12:37:05 Tower kernel: vfio-pci 0000:01:00.0: not ready 2047ms after FLR; waiting Dec 27 12:37:07 Tower kernel: vfio-pci 0000:01:00.0: not ready 4095ms after FLR; waiting Dec 27 12:37:12 Tower kernel: vfio-pci 0000:01:00.0: not ready 8191ms after FLR; waiting Dec 27 12:37:21 Tower kernel: vfio-pci 0000:01:00.0: not ready 16383ms after FLR; waiting Dec 27 12:37:37 Tower kernel: vfio-pci 0000:01:00.0: not ready 32767ms after FLR; waiting Dec 27 12:38:14 Tower kernel: vfio-pci 0000:01:00.0: not ready 65535ms after FLR; giving up The VM will eventually starts after ~1min (post the giving up of the vfio-pci), but the GPU isn't passthrough.
December 27, 20241 yr Community Expert 4 hours ago, uniqf said: hello @SimonF, I've done the following too - Turning on/off Rsize bar, Above 4G encoding in bios - Append the syslinux with and without earlymodules=vfio-pci video=efifb:off,vesafb:off gfxpayload=text arguments - Swap GPU to various different slots of motherboard I just kept getting the same errors of the following: Dec 27 12:37:04 Tower kernel: vfio-pci 0000:01:00.0: not ready 1023ms after FLR; waiting Dec 27 12:37:05 Tower kernel: vfio-pci 0000:01:00.0: not ready 2047ms after FLR; waiting Dec 27 12:37:07 Tower kernel: vfio-pci 0000:01:00.0: not ready 4095ms after FLR; waiting Dec 27 12:37:12 Tower kernel: vfio-pci 0000:01:00.0: not ready 8191ms after FLR; waiting Dec 27 12:37:21 Tower kernel: vfio-pci 0000:01:00.0: not ready 16383ms after FLR; waiting Dec 27 12:37:37 Tower kernel: vfio-pci 0000:01:00.0: not ready 32767ms after FLR; waiting Dec 27 12:38:14 Tower kernel: vfio-pci 0000:01:00.0: not ready 65535ms after FLR; giving up The VM will eventually starts after ~1min (post the giving up of the vfio-pci), but the GPU isn't passthrough. The GPU is not able to complete a hardware reset. You could try disabling FLR but may result in higher power usage. vfio-pci.disable_idle_d3=1 vfio-pci.disable_vga=1
December 27, 20241 yr 13 minutes ago, SimonF said: The GPU is not able to complete a hardware reset. You could try disabling FLR but may result in higher power usage. vfio-pci.disable_idle_d3=1 vfio-pci.disable_vga=1 Thanks @SimonF, just insert this line and rebooted the server. Starting the VM with the same FLR giving up result...was really hoping this would work Edited December 27, 20241 yr by uniqf
December 27, 20241 yr Community Expert 8 minutes ago, uniqf said: Thanks @SimonF, just insert this line and rebooted the server. Starting the VM with the same FLR giving up result...was really hoping this would work You may need to do a power off and then power on for the card to reset properly.
December 27, 20241 yr 35 minutes ago, SimonF said: You may need to do a power off and then power on for the card to reset properly. Still the same result even if I powered off the Unraid Server and powered it on back, starting the VM with the same "not ready --- after FLR" message then eventually the giving up log i too want to clarify the "vfio-pci.disable_idle_d3=1 vfio-pci.disable_vga=1" is something that I append to the syslinux argument so hopefullly that's the right way of doing it not sure if this helps, this card definitely works before, i just migrated the unraid from an AMD platform to Intel, and now I can't make this GPU passthru works... Edited December 27, 20241 yr by uniqf
December 27, 20241 yr Out of many searches, i also found this thread @ I follow through the guide as well making the necessary changes but I am still unsuccessful passing through the GPU
December 27, 20241 yr Author Do you have the graphics driver installed in unraid? If so remove it & reboot. May not make any difference but could help. We want the card to be only passed through and picked up in the VM rather than host OS.
December 27, 20241 yr 48 minutes ago, dopeytree said: Do you have the graphics driver installed in unraid? If so remove it & reboot. May not make any difference but could help. We want the card to be only passed through and picked up in the VM rather than host OS. No driver installed at Unraid level, this GPU is intented for passthru to a VM only
December 28, 20241 yr Community Expert 10 hours ago, uniqf said: No driver installed at Unraid level, this GPU is intented for passthru to a VM only How much memory does your card have? If more that 16G you may need to add to the VM following xml, you can do from the gui in 7. <qemu:commandline> <qemu:arg value='-fw_cfg'/> <qemu:arg value='opt/ovmf/X-PciMmio64Mb,string=65536'/> </qemu:commandline> Also have you tried without the rom?
December 28, 20241 yr 51 minutes ago, SimonF said: How much memory does you card have? If more that 16G you may need to add to the VM following xml, you can do from the gui in 7. <qemu:commandline> <qemu:arg value='-fw_cfg'/> <qemu:arg value='opt/ovmf/X-PciMmio64Mb,string=65536'/> </qemu:commandline> Also have you tried without the rom? my GPU has exactly 16G, I'll try very shortly and see if it works, and yes I did try with and without the rom bios. Update @SimonF just tried (first try the qemu config didn't get save, but eventually figured out), the same dreaded "vfio-pci not ready, and giving up" message...i started to think this could be the motherboard/bios implementation that cause it doesn't get the GPU to passthrough I reverted all the other changes on other thread, just kept yours, and also upgraded to 7rc2 version. If its of any help (as i also upgraded to 7rc2), i also attach a latest copy of the diagnostic logs. tower-diagnostics-20241228-1741.zip Edited December 28, 20241 yr by uniqf
December 29, 20241 yr Today i also spent some times swapping the GPU to different slots (those x4 slots) but unfortunately everytime i try boot up the VM with GPU passthru configured, the same error message greeted me. Is there anything else I could try, I need this to be working but it seems I did an upgrade of my platform and now ended with a non-working GPU passthru 😞
December 29, 20241 yr Community Expert 4 hours ago, uniqf said: Today i also spent some times swapping the GPU to different slots (those x4 slots) but unfortunately everytime i try boot up the VM with GPU passthru configured, the same error message greeted me. Is there anything else I could try, I need this to be working but it seems I did an upgrade of my platform and now ended with a non-working GPU passthru 😞 Sorry I am out of ideas, does sound like a hardware conflict somewhere, Was it just a OS upgrade or Hardware also. Maybe start a separate thread
December 29, 20241 yr no worries @SimonF, i sincerely appreciate your help. it is not just an OS upgrade, I upgrade from my AMD platform to brand new Intel Core Ultra platform....as it isn't working, i also upgraded the Unraid from 6 to 7rc2.
December 29, 20241 yr Community Expert 21 minutes ago, uniqf said: no worries @SimonF, i sincerely appreciate your help. it is not just an OS upgrade, I upgrade from my AMD platform to brand new Intel Core Ultra platform....as it isn't working, i also upgraded the Unraid from 6 to 7rc2. The kernels used in both 6 and 7 wont support the newer motherboards/CPUs as they ideally needs 6.12. Or may be a bios fix may be required for the new hardware.
December 29, 20241 yr 56 minutes ago, uniqf said: it is not just an OS upgrade, I upgrade from my AMD platform to brand new Intel Core Ultra platform.... Never recommended to buy a brand new platform for a linux-based server, it often takes 1-2 years for full support to trickle down to your favorite distro. Edited December 29, 20241 yr by Kilrah
December 29, 20241 yr Thanks @SimonF and @Kilrah, it seems I exhausted all the options, will see how it goes may be a temp solution of dual booting, if I need to uses the GPU I will reboot and boot from a NVMe to make uses of the GPU
February 26, 20251 yr On 1/11/2023 at 7:37 AM, dopeytree said: GPU passthough is much easier than made out. You don't need to do any of the stuff in videos.. Set up the VM with software VNC. Load the VM. Click to open VNC in web browser. Install windows etc. Set up windows user. Close VM. Then Go to top menu choose unraid -> tool system -> devices scroll down and tick the NVIDIA GPU or AMD GPU & its soundcard. Reboot unraid. Edit VM. Add 2nd graphics card choose your AMD or Nvidia device. Add sound card choose your AMD or Nvidia device. Apply. Start VM. It should automatically install basic drivers for the card. Now go to amd or nvidia websites and download the drivers for your card. Run the installer for your card. Reboot VM. Now you have a gaming VM. Enjoy. Hey Guys! I followed the instructions on this topic, and now I cannot find my Unraid server on my network. I am new to all of it. What can I do? Thanks for your help!
February 26, 20251 yr Author Hey sorry about that. If its not on the network, it sounds like you maybe accidentally passed though the network card instead of or swell as the gpu? You might be able to try booting safe mode? so when the unraid usb stick first loads you get a few seconds of options try safe mode? It removes the plugins and may enable a default start-up? Otherwise it might be a case of either plugging in a keyboard, mouse & screen or going through config files on the usb stick. If this doesn't help then try posting in the main support forum and a pro will pop along. Edited February 26, 20251 yr by dopeytree
March 4, 20251 yr On 2/26/2025 at 4:04 PM, dopeytree said: Hey sorry about that. If its not on the network, it sounds like you maybe accidentally passed though the network card instead of or swell as the gpu? You might be able to try booting safe mode? so when the unraid usb stick first loads you get a few seconds of options try safe mode? It removes the plugins and may enable a default start-up? Otherwise it might be a case of either plugging in a keyboard, mouse & screen or going through config files on the usb stick. If this doesn't help then try posting in the main support forum and a pro will pop along. Hey man! Thanks!!! It worked at the end. I removed the sytem files of the USB stick and kept the config files. It worked at the end. And now I learn to do a safety copy every time that I change something on my homelab. Anyway, thanks for your help! Edited March 4, 20251 yr by Leops typing error.
April 28, 20251 yr Appreciate the tutorial. Im using an Intel ARC B580 trying to passthrough, along with an SP330 (Huawei OEM of X710-DA2). I could see there's a tickbox available in the SysDevs, however the tickbox of the SP330(Recognized as X710) is grey and unavailable. I'm Using an EPYC 7F32 and AsrockRack RomeD8-2T I have enabled IOMMU, Resizebar, Above4GDecoding,SR-IOV, the BOOT CSM is set to custom with the SP330's pcie slot ENABLED with UEFI. (in BIOS)
April 30, 20251 yr hello guys, i have a qeustion. but let's begin with my setup in the past: win10 vm with nvidia shield worked perfect for game streams as i had setup and easy acces via teamviewer and steam/shield. now i have a different setup with AMD GPU (my blade server not supporting big or let's say: power intense GPUs (it is a dl380 gen9 with secondary riser up to 150W GPUs but only if you have installed 1500W redundant PSUs, i have 2x500W). so actually a RX560 is installed it need's no extra power, all it's 75W is only coming from PCI-E slot. the actual question: i want to get rid of windows on my VMs (and all other PCs). i installed Garuda Dragonzed on the planned gaming VM, works perfect with virtual adapter. the only problem is wayland. other then x11 i can't go in with teamviewer, xrdp or other stuff. so i have no control over the VM before i can setup e.g. Parsec (it is installed but i can't set it up as host in the VM when virual gpu is active, even if both virtual and the radeon is passed trough). so i can't connect in any way when pass trough only is on my GPU. how do i fix this? simply use another DE (or setup a fresh VM with a garude DE with x11)? going back to windows is no option.
August 27, 2025Aug 27 First off, @dopeytree , this is a great guide, very concise! Thanks for sharing it with us.I just want to clarify and update few things, since it's been a couple years, and I'm sure folks are still searching for this now and then (I was!).Here are the original instructions, with a few notes/edits, to make it even easier!GPU passthough is much easier than made out.You don't need to do any of the stuff in videos.. Set up the VM with software VNC.Load the VM.Click to open VNC in web browser.Install windows etc.Set up windows user.Close VM. Then Go to top menu choose unraid -> tool system -> devices (This might say - In top navigation bar: select Tools > System Devices. Under PCI Devices and IOMMU Groups) scroll down and tick the NVIDIA GPU or AMD GPU & its soundcard. (scroll to bottom of section and click [Bind Selected to VFIO at Boot] ONLY if you plan to use your GPU with both VMs and Docker containers. If you are only using it for VMs, this step is not required) Reboot unraid.***NOTE: If you don't bind to the VFIO, there's no point in checking the boxes, and you might at as well skip this step, and save yourself a reboot. ;) Edit VM.Add 2nd graphics card choose your AMD or Nvidia device. (It's the small [+] at the lower left of the Graphics Card section when you're in Edit mode for the VM. Note that the VM must be stopped to make edits. duh.)Add sound card choose your AMD or Nvidia device. (same as for the GPU)Apply. Start VM.It should automatically install basic drivers for the card.Now go to amd or nvidia websites and download the drivers for your card.Run the installer for your card.Reboot VM. Now you have a gaming VM.Enjoy.Thanks again to @dopeytree for the awesome write-up! It got me from zero to pass-thru with very little fuss.Now I just have to figure out how to get decent performance over an RDP connection... ;)
August 29, 2025Aug 29 Community Expert On 8/27/2025 at 10:36 AM, Elmojo said:First off, @dopeytree , this is a great guide, very concise! Thanks for sharing it with us.I just want to clarify and update few things, since it's been a couple years, and I'm sure folks are still searching for this now and then (I was!).Here are the original instructions, with a few notes/edits, to make it even easier!GPU passthough is much easier than made out.You don't need to do any of the stuff in videos.. Set up the VM with software VNC.Load the VM.Click to open VNC in web browser.Install windows etc.Set up windows user.Close VM. Then Go to top menu choose unraid -> tool system -> devices (This might say - In top navigation bar: select Tools > System Devices. Under PCI Devices and IOMMU Groups) scroll down and tick the NVIDIA GPU or AMD GPU & its soundcard. (scroll to bottom of section and click [Bind Selected to VFIO at Boot] ONLY if you plan to use your GPU with both VMs and Docker containers. If you are only using it for VMs, this step is not required)Reboot unraid.***NOTE: If you don't bind to the VFIO, there's no point in checking the boxes, and you might at as well skip this step, and save yourself a reboot. ;) Edit VM.Add 2nd graphics card choose your AMD or Nvidia device. (It's the small [+] at the lower left of the Graphics Card section when you're in Edit mode for the VM. Note that the VM must be stopped to make edits. duh.)Add sound card choose your AMD or Nvidia device. (same as for the GPU)Apply. Start VM.It should automatically install basic drivers for the card.Now go to amd or nvidia websites and download the drivers for your card.Run the installer for your card.Reboot VM. Now you have a gaming VM.Enjoy.Thanks again to @dopeytree for the awesome write-up! It got me from zero to pass-thru with very little fuss.Now I just have to figure out how to get decent performance over an RDP connection... ;)Some cards are quirky, or maybe it's the motherboard. My RTX 4060 and 5050 doesn't want my GPU to be binded with its sound card.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.