sixkittens

Okay then, what version of Unraid are you running? These lockups seem to no longer occur on Unraid 7.3.

It sounds like you might be having a different issue to what is discussed here if you are unable to open the VM page in the Unraid UI. Have you tried rebooting your server?

Make sure that you do not have the GPU or sound device bound to VFIO at boot, this seems to invariably cause the GPU to lock up (I have no idea why). Then try to boot your VM and if you still get no video, check the VM log by clicking the VM's icon and selecting from the drop-down menu. The log should tell you if there are errors preventing video output.

Was this with or without CPU migration enabled? There's a known issue with Linux VM instability that occurs when CPU migration is turned off

Meanwhile I have been on 7.3.0 for a few days, shutting my VM down each night without the system hanging once so far. Hopefully the trend continues. I did have to enable CPU migration to prevent instability issues I was having but this seems to be exclusive to Linux VM's.

I recommend unbinding the GPU from VFIO, and testing again. Not sure how you got it to work before, but binding to VFIO (for me anyway) causes the GPU to get stuck in the wrong power state and there's no video output as a result. I intend to look further into what might be causing the lockup when bound, but I don't know enough about how these things work to have a good starting point. I have added a ROM that I dumped myself from GPU-Z, and updated to 7.3.0, which has a newer kernel, so it may yet prevent the lockups when restarting the vm.

I haven't had an opportunity to test this, but there is a known VM instability for hosts on 6.18 or newer where VM migration is disabled, so enabling migration may fix the issue. https://discussion.fedoraproject.org/t/kvm-guests-become-unstable-on-6-18-kernel/182870/11 Edit: to clarify, that's Linux kernel 6.18, not referring to an Unraid version Edit 2: I can confirm that enabling CPU migration has prevented the kernel panics for me, at least for the trigger that I was able to identify (KDE Discover checking for updates)

Similar issue here with 7.3 but uncertain if it's related. My Fedora VM boots fine, but after login I get a kernel panic. It appears to be related to the kernel version of Unraid, because I had the same issue a while back when I was testing a newer kernel to try and fix passthrough issues. Weirdly the kernel panic happens whenever KDE Discover tries to find updates. Does your VM try to call packagekit or something at boot time?

Hi Arie, Are you referring to a vbios? Any more information you can provide would be helpful as we have been struggling with this for months now (I had mostly given up and just never shut down my VM). I have a vbios file that I dumped directly from my card but it did not fix anything. So if you can confirm what the romfile is and where you got it, as well as any other steps you took to get things working that would be much appreciated.

Nothing definitive that I've been able to find. There's a few recommendations out there that work for some people, but I haven't found anything that makes an actual difference in my testing. I'm hopeful that the 7.3 update might have a positive effect, moving to the 6.18 kernel. In my case, there's about a 20% (guessing) chance of the GPU locking up after VM stop and crashing the server if the VM starts again, but none of the suggested fixes have made an improvement on that for me.

In your post you mention the AMD vendor-reset module/plugin which is only compatible with cards up to the Navi 14 architecture, i.e 5000 series cards, newer cards that experience the vendor reset bug when passing through are not supported by the module and, as far as I am aware, require fixes from the in-kernel drivers. Additionally, the issue being discussed in this thread isn't to do with being able to pass the card through as such, it's preventing the hard freeze that occurs once a VM using the card is shut down and a new VM session starts. If you are interested in reading up on what other people have written who have managed to prevent the reset bug on newer cards, I'll link some resources here: https://forum.level1techs.com/t/vfio-pass-through-working-on-9070xt/227194 https://forum.proxmox.com/threads/working-win11-vm-wont-boot-after-passthrough-of-gpu-vendor-reset-not-able-to-set-reset_method-solution-update-kernel-unbind-via-hook-script.166362/ https://forum.proxmox.com/threads/working-amd-rx-9070xt-support.163370/ https://www.reddit.com/r/unRAID/comments/1j52fkc/unable_to_pass_through_amd_rx_9070_xt/ So far I have not had much success following the steps in the threads linked above, but for whatever reason my server doesn't freeze every time, possibly because I am running a Linux VM rather than Windows. Since having a newer host kernel seems to play a role in other users' success, I am hoping for a stable custom kernel for Unraid 7.2 to be released (linked in the reddit thread) before I try anything further, as the unstable one caused my VM to repeatedly kernel panic. In all cases where people have it working, it appears to be necessary to manually unbind and rebind the GPU before starting a new VM session. Unfortunately these issues with 7000 and 9000 series cards don't seem to be well known outside of the hobby virtualization communities, and Limetech don't seem to have any ideas, so a definite fix might not be forthcoming any time soon.

Adding this line in the syslinux command line does not seem to prevent the GPU from binding to D3 when the GPU is bound to VFIO at boot (as mentioned before, it does not do it for me when it's unbound), nor do spaceinvaderone's suggestions of disabling the framebuffer and resizeable BAR. I decided to revisit this because despite not having the shutdown issue with my Linux VM, I am on rare occasion seeing the server freeze when I try to start the VM instead. I have not had much luck so far though. Edit: The weird thing is that the VM log says the GPU is stuck in D3 and won't restart if I shut it down, like I would expect if it was stuck in D3, but cat /sys/bus/pci/devices/0000:03:00.0/power_state always reports D0

In the qemu command line? Doesn't seem to help with the D3 power issue for me in my test just now when binding my GPU for the Linux VM. I can run the VM unbound but when the VM is off the displays don't power down so eventually I'd like to be able to bind the GPU again. Haven't tested windows yet Edit: nevermind, when unbound it does actually seem to fix the monitors staying on. How strange. I still get no video out when bound, which is indicative of some problem still existing, but that seems to do something at least. I'll try test windows tomorrow. Edit 2: Scratch that, it doesn't get saved if entered into the qemu args section. Guess the change I noticed was from binding and unbinding the GPU.

The issue does not occur with Linux-based VM's only Windows ones.

Unfortunately I haven't been able to set up a Windows VM to test the issue, for some reason the installation hangs for both Windows 10 and 11. That said I have had no issues with my Linux VM since updating my previous post. Edit: managed to install Win 11, but it's no go. "vfio: Unable to power on device, stuck in D3" showing up in the logs and a hard reset is still needed to stop the VM if the OS shuts down even after adding the line from my earlier message to the XML.

Update on my last comment. I'm getting issues sometimes when playing games where the VM locks up completely and has to be force stopped and restarted. It may be the power state change issue that has been mentioned earlier, but I am unsure how to check. Hoping that someone at Lime Tech is looking into this issue, I would hate to have to go back to my 1080ti after buying a new card. Edit: Adding the following to my XML under the cpu section fixed the crashes and the VFIO_MAP_DMA error: <domain...> <cpu...> ... <maxphysaddr mode='passthrough' limit='39'/> </cpu> ... </domain>I'm not sure what the 'limit' section does, since I saw people suggesting both with and without it, but it works with and I haven't tested without.

I ended up getting one of these cards anyway and I am pleased to find that I'm not having any issues with my Linux VM when shutting down or rebooting. I'm on Unraid 7.1.2 which is on the newer 6.12 kernel compared to 7.0.1 which I believe was still on 6.6 and I have the GPU and audio device unbound in VFIO Unfortunately I cannot confirm anything with Windows because I don't have a windows VM set up, but this thread on Reddit has a script that seems to prevent this issue if run before starting the VM. Note that they use the custom 6.14 kernel, which may or may not still be necessary for Windows to work. Running the Linux VM, I am still seeing the following errors repeated in the logs qemu-system-x86_64: vfio: Cannot reset device 0000:00:1f.5, no available reset mechanism. qemu-system-x86_64: vfio: Cannot reset device 0000:00:14.0, no available reset mechanism. qemu-system-x86_64: vfio: Cannot reset device 0000:00:1f.3, no available reset mechanism. and qemu-system-x86_64: VFIO_MAP_DMA failed: Invalid argument qemu-system-x86_64: vfio_container_dma_map(0x55c154e14fe0, 0x383000000000, 0x400000000, 0x144bc0000000) = -22 (Invalid argument) But since there haven't been any issues I can see I haven't looked into them too much yet

Based on this thread the server crash might be an issue with passing through motherboard audio not being able to return it to the host. Are you passing through MB audio when the issue happens? I'm not able to test because I don't have one of these cards, though I was hoping to get one soon, if there's a way around this issue.

I'm getting a weird issue in Windows 11 where executables will not run from a VirtioFS mounted drive when mounting them using the script from the recommended post. Instead, Windows will show a network error stating that it cannot access the file path. This only happens with some executables though, not all. I have tried to follow the directions on the gitlab page to see if I can get it working that way, but the commands always return "access denied". In one case, a file gave an error that windows cannot access the temporary version of the executable in the %appdata%\local\temp folder, and another gave a ShellExecuteEX failed; code 1203 error. But most cases the error just says that windows cannot access that file path. This does not happen if the executables are run from the vdisk. Does anyone have any theories as to what might be going on? Edit: just checked, error 1203 is "ERROR_NO_NET_OR_BAD_PATH: The network path was either typed incorrectly, does not exist, or the network provider is not currently available. Please try retyping the path or contact your network administrator." Which is basically the same as the other issues; windows can't access the file. When using network shares, I've had to use a registry edit that allows the system to see the shares, which is necessary to run executables. Could there be something similar going on here?

This does work if you also change the <vendor_id state='on' value='other'/> into <vendor_id state='on' value='randomid'/> See this page for reference.

Had my USB die on me today and I am also unable to download the backup. Same error message.

Same issue here. Problem is definitely not solved.