March 19, 20251 yr Hey everyone, I’m trying to pass through my 9070 XT to a Windows 11 VM on UnRAID 7, but I’m running into a ton of issues. The VM fails to start, and I get the following errors in the logs: Quote qemu-system-x86_64: vfio: Unable to power on device, stuck in D3 qemu-system-x86_64: vfio-pci: Cannot read device rom at 0000:03:00.0 I’ve tried a few things so far: vBIOS: Downloaded a clean vBIOS from TechPowerUp and set it in UnRAID, but the issue persists. IOMMU Groups: The GPU is in its own group, so I don’t think it’s an isolation issue. PCIe ACS Override: Enabled in "Both" mode, but still no luck. BIOS Settings: Above 4G Decoding is enabled, and I made sure another GPU is the primary display. Different PCIe Slots: Moved the GPU around, but the problem persists. I’m wondering if this is an issue with how UnRAID is handling power states? As others have stated that they have those exact issue as well... Any suggestions would be greatly appreciated! I'm hoping it's an easy fix that either I can implement or LimeTech/Unraid or a plugin expert can implement ASAP 🙏 Thanks in advance! Edited March 19, 20251 yr by Rick Sanchez
March 19, 20251 yr Author Please see diagnostics attached Thank you for your help 🙏 server-diagnostics-20250319-1315.zip
March 19, 20251 yr Hey there, I noticed the title mentions a 9070XT, but in the body, you say you’re passing through a Sapphire 7900 XT. Just wondered which model you’re using? If it’s actually a 7900 XT, you’re not alone alot of people have reported similar passthrough issues with this card on the net. A couple of things to try 1. Bind GPU to VFIO Go to Tools > System Devices in Unraid. Find your GPU and GPU Audio device. Check Bind to VFIO at Boot and reboot server 2. Disable Framebuffer to Prevent Host from Using GPU Modify Syslinux Configuration This prevents UnRAID from using the GPU for console output and helps avoid D3 power state issues. Goto Main Tab > Flash and edit the syslinux config adding video=efifb:off,vesafb:off ie label Unraid OS menu default kernel /bzimage append video=efifb:off,vesafb:off initrd=/bzroot 3. Disable resizable BAR (ReBAR) Some RDNA3 GPUs have issues with passthrough when Resizable BAR is enabled.Try disabling it in BIOS and see if it helps You can also do in the XML by using <rom bar="off"/> Also what is the card like after a cold boot of the server. Do you still have the issue or is it only after restarting a vm etc. Also have a read here for a discussion on the 7900 and passthrough https//forum.level1techs.com/t/the-state-of-amd-rx-7000-series-vfio-passthrough-april-2024/210242
March 19, 20251 yr Author 5 hours ago, SpaceInvaderOne said: Hey there, I noticed the title mentions a 9070XT, but in the body, you say you’re passing through a Sapphire 7900 XT. Just wondered which model you’re using? If it’s actually a 7900 XT, you’re not alone alot of people have reported similar passthrough issues with this card on the net. A couple of things to try 1. Bind GPU to VFIO Go to Tools > System Devices in Unraid. Find your GPU and GPU Audio device. Check Bind to VFIO at Boot and reboot server 2. Disable Framebuffer to Prevent Host from Using GPU Modify Syslinux Configuration This prevents UnRAID from using the GPU for console output and helps avoid D3 power state issues. Goto Main Tab > Flash and edit the syslinux config adding video=efifb:off,vesafb:off ie label Unraid OS menu default kernel /bzimage append video=efifb:off,vesafb:off initrd=/bzroot 3. Disable resizable BAR (ReBAR) Some RDNA3 GPUs have issues with passthrough when Resizable BAR is enabled.Try disabling it in BIOS and see if it helps You can also do in the XML by using <rom bar="off"/> Also what is the card like after a cold boot of the server. Do you still have the issue or is it only after restarting a vm etc. Also have a read here for a discussion on the 7900 and passthrough https//forum.level1techs.com/t/the-state-of-amd-rx-7000-series-vfio-passthrough-april-2024/210242 Hi SpaceInvaderOne 👋 Apologies for my typo, the card is the 9070 XT
March 24, 20251 yr Author On 3/19/2025 at 2:11 PM, SpaceInvaderOne said: Hey there, I noticed the title mentions a 9070XT, but in the body, you say you’re passing through a Sapphire 7900 XT. Just wondered which model you’re using? If it’s actually a 7900 XT, you’re not alone alot of people have reported similar passthrough issues with this card on the net. A couple of things to try 1. Bind GPU to VFIO Go to Tools > System Devices in Unraid. Find your GPU and GPU Audio device. Check Bind to VFIO at Boot and reboot server 2. Disable Framebuffer to Prevent Host from Using GPU Modify Syslinux Configuration This prevents UnRAID from using the GPU for console output and helps avoid D3 power state issues. Goto Main Tab > Flash and edit the syslinux config adding video=efifb:off,vesafb:off ie label Unraid OS menu default kernel /bzimage append video=efifb:off,vesafb:off initrd=/bzroot 3. Disable resizable BAR (ReBAR) Some RDNA3 GPUs have issues with passthrough when Resizable BAR is enabled.Try disabling it in BIOS and see if it helps You can also do in the XML by using <rom bar="off"/> Also what is the card like after a cold boot of the server. Do you still have the issue or is it only after restarting a vm etc. Also have a read here for a discussion on the 7900 and passthrough https//forum.level1techs.com/t/the-state-of-amd-rx-7000-series-vfio-passthrough-april-2024/210242 I've done this and still not working 😔 Please see attached 🙏 Thank you for your help! diagnostics-20250324-0926.zip
March 27, 20251 yr Hi Rick. Ok so that didnt work. Lets see if the GPU is in D0 when it first boots and if and when its in D3 after a boot Lets try these steps and note the results of each Step 1 – Full Shutdown and Power On 1. Shutdown your Unraid server completely 2. Wait 10/15 seconds after it shuts down 3. Power it back on 4. Do NOT start any VM 5. Open a terminak and run cat /sys/bus/pci/devices/0000:03:00.0/power_state 6. Note whether it says D0 or D3 --- Step 2 – Unbind VFIO 1. Go to Tools > System Devices in Unraid web UI 2. Uncheck "Bind to VFIO at boot" for - Your GPU - Your GPU Audio device 3. Reboot the server 4. After reboot, run again cat /sys/bus/pci/devices/0000:03:00.0/power_state 5. Note the result again (D0 or D3) --- Step 3 – Remove Framebuffer Block (if still D3) 1. Go to Main > Flash 2. Scroll to "Syslinux Configuration" 3. Find the line starting with append initrd=/bzroot ... 4. Remove: video=efifb:off,vesafb:off 5. Click Apply and Reboot 6. Run again: cat /sys/bus/pci/devices/0000:03:00.0/power_state --- Step 4 – Allow AMDGPU Driver to Initialise GPU 1. Make sure - VFIO is still not bound & Ffamebuffer still removed 2. Reboot the server 3. After booting, run cat /sys/bus/pci/devices/0000:03:00.0/power_state 4. The GPU should now be in D0 Step 5 – BIOS Settings Reboot and enter BIOS Change these settings if not like this - Primary Display -- IGFX (iGPU) - Multi-Monitor -- Enabled - CSM -- Disabled - Above 4G Decoding -- Enabled - Resizable BAR (ReBAR) -- Disabled Save and reboot. Run the power state check again: cat /sys/bus/pci/devices/0000:03:00.0/power_state --- Step 6 – Try the VM Without the vbios attached also again do not bind GPU to VFIO at boot Then starting the VM with GPU passed through --- After each test, report - The GPU power state (D0 or D3) on boot of server - If the VM starts - Any error messages seen & hopefully we can get it to work. But AMD gpus are known to have reset issues which your D3 is closely related to
March 27, 20251 yr Author On 3/27/2025 at 5:09 PM, SpaceInvaderOne said: Hi Rick. Ok so that didnt work. Lets see if the GPU is in D0 when it first boots and if and when its in D3 after a boot Lets try these steps and note the results of each Step 1 – Full Shutdown and Power On 1. Shutdown your Unraid server completely 2. Wait 10/15 seconds after it shuts down 3. Power it back on 4. Do NOT start any VM 5. Open a terminak and run cat /sys/bus/pci/devices/0000:03:00.0/power_state 6. Note whether it says D0 or D3 --- Step 2 – Unbind VFIO 1. Go to Tools > System Devices in Unraid web UI 2. Uncheck "Bind to VFIO at boot" for - Your GPU - Your GPU Audio device 3. Reboot the server 4. After reboot, run again cat /sys/bus/pci/devices/0000:03:00.0/power_state 5. Note the result again (D0 or D3) --- Step 3 – Remove Framebuffer Block (if still D3) 1. Go to Main > Flash 2. Scroll to "Syslinux Configuration" 3. Find the line starting with append initrd=/bzroot ... 4. Remove: video=efifb:off,vesafb:off 5. Click Apply and Reboot 6. Run again: cat /sys/bus/pci/devices/0000:03:00.0/power_state --- Step 4 – Allow AMDGPU Driver to Initialise GPU 1. Make sure - VFIO is still not bound & Ffamebuffer still removed 2. Reboot the server 3. After booting, run cat /sys/bus/pci/devices/0000:03:00.0/power_state 4. The GPU should now be in D0 Step 5 – BIOS Settings Reboot and enter BIOS Change these settings if not like this - Primary Display -- IGFX (iGPU) - Multi-Monitor -- Enabled - CSM -- Disabled - Above 4G Decoding -- Enabled - Resizable BAR (ReBAR) -- Disabled Save and reboot. Run the power state check again: cat /sys/bus/pci/devices/0000:03:00.0/power_state --- Step 6 – Try the VM Without the vbios attached also again do not bind GPU to VFIO at boot Then starting the VM with GPU passed through --- After each test, report - The GPU power state (D0 or D3) on boot of server - If the VM starts - Any error messages seen & hopefully we can get it to work. But AMD gpus are known to have reset issues which your D3 is closely related to How do I do/confirm this? "Allow AMDGPU Driver to Initialise GPU" The card is in D0 State now Thank you 🙏 Edited March 28, 20251 yr by Rick Sanchez
March 27, 20251 yr 4 hours ago, SpaceInvaderOne said: Hi Rick. Ok so that didnt work. Lets see if the GPU is in D0 when it first boots and if and when its in D3 after a boot Lets try these steps and note the results of each Step 1 – Full Shutdown and Power On 1. Shutdown your Unraid server completely 2. Wait 10/15 seconds after it shuts down 3. Power it back on 4. Do NOT start any VM 5. Open a terminak and run cat /sys/bus/pci/devices/0000:03:00.0/power_state 6. Note whether it says D0 or D3 --- Step 2 – Unbind VFIO 1. Go to Tools > System Devices in Unraid web UI 2. Uncheck "Bind to VFIO at boot" for - Your GPU - Your GPU Audio device 3. Reboot the server 4. After reboot, run again cat /sys/bus/pci/devices/0000:03:00.0/power_state 5. Note the result again (D0 or D3) --- Step 3 – Remove Framebuffer Block (if still D3) 1. Go to Main > Flash 2. Scroll to "Syslinux Configuration" 3. Find the line starting with append initrd=/bzroot ... 4. Remove: video=efifb:off,vesafb:off 5. Click Apply and Reboot 6. Run again: cat /sys/bus/pci/devices/0000:03:00.0/power_state --- Step 4 – Allow AMDGPU Driver to Initialise GPU 1. Make sure - VFIO is still not bound & Ffamebuffer still removed 2. Reboot the server 3. After booting, run cat /sys/bus/pci/devices/0000:03:00.0/power_state 4. The GPU should now be in D0 Step 5 – BIOS Settings Reboot and enter BIOS Change these settings if not like this - Primary Display -- IGFX (iGPU) - Multi-Monitor -- Enabled - CSM -- Disabled - Above 4G Decoding -- Enabled - Resizable BAR (ReBAR) -- Disabled Save and reboot. Run the power state check again: cat /sys/bus/pci/devices/0000:03:00.0/power_state --- Step 6 – Try the VM Without the vbios attached also again do not bind GPU to VFIO at boot Then starting the VM with GPU passed through --- After each test, report - The GPU power state (D0 or D3) on boot of server - If the VM starts - Any error messages seen & hopefully we can get it to work. But AMD gpus are known to have reset issues which your D3 is closely related to if you are looking for best video editing app then visit here Capcut Mod APK Hey Rick, ran your steps: Full shutdown: D3 on boot. Unbound VFIO: Still D3. Removed framebuffer: D3 again. AMDGPU driver: D0, finally! BIOS tweak (IGFX, 4G on, etc.): D0 holds. VM sans vbios: Boots fine, D0, no errors. AMD reset issue seems tied to D3—driver fixed it. Maybe <a href="https://capcutmodapkall.com/">CapCut</a> could spice up a setup vid? Thoughts?
March 28, 20251 yr Author 19 hours ago, Bala Bob said: Hey Rick, ran your steps: Full shutdown: D3 on boot. Unbound VFIO: Still D3. Removed framebuffer: D3 again. AMDGPU driver: D0, finally! BIOS tweak (IGFX, 4G on, etc.): D0 holds. VM sans vbios: Boots fine, D0, no errors. AMD reset issue seems tied to D3—driver fixed it. Maybe <a href="https://capcutmodapkall.com/">CapCut</a> could spice up a setup vid? Thoughts? What driver and how? How do I remove framebuffer? - cards in D0 state already Edited March 28, 20251 yr by Rick Sanchez
March 28, 20251 yr Community Expert Just now, Rick Sanchez said: What driver and how? AMDGPU and lspci -k will show it in use at boot.
March 28, 20251 yr Author 3 minutes ago, SimonF said: AMDGPU and lspci -k will show it in use at boot. It's in D0 after getting to this bit: Step 4 – Allow AMDGPU Driver to Initialise GPU 1. Make sure - VFIO is still not bound & Ffamebuffer still removed 2. Reboot the server 3. After booting, run cat /sys/bus/pci/devices/0000:03:00.0/power_state 4. The GPU should now be in D0 Step 5 – BIOS Settings Reboot and enter BIOS Change these settings if not like this - Primary Display -- IGFX (iGPU) - Multi-Monitor -- Enabled - CSM -- Disabled - Above 4G Decoding -- Enabled - Resizable BAR (ReBAR) -- Disabled Save and reboot. Run the power state check again: cat /sys/bus/pci/devices/0000:03:00.0/power_state --- Step 6 – Try the VM Without the vbios attached also again do not bind GPU to VFIO at boot Then starting the VM with GPU passed through I don't know how to do or check the 'Allow AMDGPU Driver to Initialise GPU' bit or 'framebuffer'
March 28, 20251 yr Community Expert 2 minutes ago, Rick Sanchez said: It's in D0 after getting to this bit: Step 4 – Allow AMDGPU Driver to Initialise GPU 1. Make sure - VFIO is still not bound & Ffamebuffer still removed 2. Reboot the server 3. After booting, run cat /sys/bus/pci/devices/0000:03:00.0/power_state 4. The GPU should now be in D0 Step 5 – BIOS Settings Reboot and enter BIOS Change these settings if not like this - Primary Display -- IGFX (iGPU) - Multi-Monitor -- Enabled - CSM -- Disabled - Above 4G Decoding -- Enabled - Resizable BAR (ReBAR) -- Disabled Save and reboot. Run the power state check again: cat /sys/bus/pci/devices/0000:03:00.0/power_state --- Step 6 – Try the VM Without the vbios attached also again do not bind GPU to VFIO at boot Then starting the VM with GPU passed through I don't know how to do or check the 'Allow AMDGPU Driver to Initialise GPU' bit or 'framebuffer' Framebuffer disabled in step 3, other bit means booting without binding to vfio and the driver resets the card.
March 29, 20251 yr On 3/27/2025 at 9:44 PM, Bala Bob said: Hey Rick, ran your steps: Full shutdown: D3 on boot. Unbound VFIO: Still D3. Removed framebuffer: D3 again. AMDGPU driver: D0, finally! BIOS tweak (IGFX, 4G on, etc.): D0 holds. VM sans vbios: Boots fine, D0, no errors. AMD reset issue seems tied to D3—driver fixed it. Maybe <a href="https://capcutmodapkall.com/">CapCut</a> could spice up a setup vid? Thoughts? Hi @Bala Bob That’s great to hear the GPU is passing through successfully now. One thing I’d be really interested to know, after you’ve shut the VM down, can you start it again without rebooting the server? That would tell us if the GPU resets properly after the first passthrough
March 29, 20251 yr Author On 3/27/2025 at 9:44 PM, Bala Bob said: AMDGPU driver: D0, finally! BIOS tweak (IGFX, 4G on, etc.): D0 holds. VM sans vbios: Boots fine, D0, no errors. AMD reset issue seems tied to D3—driver fixed it. Maybe <a The only bits I am uncertain of is: How do I check AMDGPU - I thought I did this.. What bios tweaks have you made exactly as I've tried everything! What drive/vbios are you discussing?
March 31, 20251 yr Author New week, same problem... D0 until VM boots, then it goes into D3 or D3Cold Any suggestions @SpaceInvaderOne or @Bala Bob? UPDATE: VM booted with updated driver attached, boots, but no video - power state = D0 and D3? UnRAID reports D0, VM reports D3 Edited March 31, 20251 yr by Rick Sanchez Update
March 31, 20251 yr Author #!/bin/bash sleep 5 # Unbind the GPU from the amdgpu driver before resizing resources echo "0000:03:00.0" > /sys/bus/pci/drivers/amdgpu/unbind sleep 2 # Resize the GPU's BAR2 memory region (useful for PCI passthrough) echo 3 > /sys/bus/pci/devices/0000:03:00.0/resource2_resize sleep 2 # Bind the GPU back to the amdgpu driver for use by the host echo "0000:03:00.0" > /sys/bus/pci/drivers/amdgpu/bind 2>/dev/null sleep 2 echo "Done" Many users have reported this fixes their issue, but I just receive: Script location: /tmp/user.scripts/tmpScripts/9070 XT GPU Fix/script Note that closing this window will abort the execution of this script /tmp/user.scripts/tmpScripts/9070 XT GPU Fix/script: line 4: echo: write error: No such device Done Loosing the will to live with this now...
April 7, 20251 yr Author I've given up trying to get this to work as nothing works... Can the Unraid team update is on when the kernel required will be released?
April 12, 20251 yr Last week I bought a Sapphire 9070 XT for testing on a server with a Z690 motherboard and a 13900K CPU. In early testing, I ran into what looks like the reset bug affecting this card. I didn’t have much time to troubleshoot, but I was able to pass the GPU through to a VM without doing anything special no need to bind it no vbios I simply added it to the VM as normal and it worked. However, there’s a major issue for me. When I shut down the VM, the entire server hangs. This also happens when running a custom 6.14 kernel. I’m away on holiday for about 10 days, but once I’m back I’ll be taking a proper look at this. If I find a workaround or solution, I’ll post an update.
May 8, 20251 yr Hi @SpaceInvaderOne, i really don't want to push you on this topic. Just curious if you have any new insights on this topic. Thanks in advance and best regards!
May 30, 20251 yr I’m also in the same boat with the server completely hanging after VM shutdown, anyone able to get this passthrough/reset working with this card? I have also tried all of the above steps and then some.
June 1, 20251 yr Based on this thread the server crash might be an issue with passing through motherboard audio not being able to return it to the host. Are you passing through MB audio when the issue happens?I'm not able to test because I don't have one of these cards, though I was hoping to get one soon, if there's a way around this issue. Edited June 1, 20251 yr by sixkittens
June 1, 20251 yr 12 hours ago, sixkittens said:Based on this thread the server crash might be an issue with passing through motherboard audio not being able to return it to the host. Are you passing through MB audio when the issue happens?I'm not able to test because I don't have one of these cards, though I was hoping to get one soon, if there's a way around this issue.I am not passing through motherboard audio, only the gpu, the whole gpu and using multimode, can’t shutdown my VM and sometimes when I launch a game, the state of the card changes and locks up the entire system as well. This has happened a few times to me now. Really hope we can get a fix for this ASAP.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.