I'm having some issues. Had a RTX 3050 card I was able to passthrough just fine, but that card is no good. Just got a brand new RX 6600 and am trying to passthrough to Windows 11 VM. The VM has a SSD passed through. I can boot straight to Windows on bare metal and the RX 6600 works just fine, so I know the card is good. The issue I am having is I can give the VM only VNC and everything works fine. Initial setup I can passthrough the card for a few minutes, long enough to install drivers and stress test, sometimes not that much time. The screen will go blank and entire server becomes unresponsive. After a few minutes the server will reboot. At this point, when the VM is started, the first core gets to 100% and no display, including VNC. After about ten minutes the VM will finally boot with no display on GPU but VNC will work. Device Manager then shows error 43, and no drivers will install for it. If I remove the GPU from configs and leave VNC, it boots normal.

 

I have already done VFIO binding, disabled video for UnRAID, booted in UEFI and Legacy mode to no difference. I am attaching the Diagnostic file, but I am not seeing anything that stands out to me, but I am still fresh to UnRAID systems

tower-diagnostics-20240423-1948.zip

Here is the current log syslog file as well. Not truly seeing any issues. will attempt to get something in the logs with the crash in a few

syslog.txt

The following is the VM logs when attempting to boot with the GPU attached

 

 

 

-tpmdev emulator,id=tpm-tpm0,chardev=chrtpm \
-device '{"driver":"tpm-tis","tpmdev":"tpm-tpm0","id":"tpm0"}' \
-device '{"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-vnc 0.0.0.0:0,websocket=5700,audiodev=audio1 \
-k en-us \
-device '{"driver":"qxl-vga","id":"video0","max_outputs":1,"ram_size":67108864,"vram_size":67108864,"vram64_size_mb":0,"vgamem_mb":16,"bus":"pcie.0","addr":"0x1"}' \
-device '{"driver":"vfio-pci","host":"0000:0e:00.0","id":"hostdev0","bus":"pci.3","addr":"0x0","romfile":"/mnt/cache/isos/vfios/243157.rom"}' \
-device '{"driver":"vfio-pci","host":"0000:0e:00.1","id":"hostdev1","bus":"pci.4","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:05:00.0","id":"hostdev2","bus":"pci.5","addr":"0x0"}' \
-device '{"driver":"usb-host","hostdevice":"/dev/bus/usb/003/003","id":"hostdev3","bus":"usb.0","port":"2"}' \
-device '{"driver":"usb-host","hostdevice":"/dev/bus/usb/001/002","id":"hostdev4","bus":"usb.0","port":"3"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
char device redirected to /dev/pts/0 (label charserial0)
qxl_send_events: spice-server bug: guest stopped, ignoring
2024-04-23T23:46:56.566822Z qemu-system-x86_64: terminating on signal 15 from pid 11113 (/usr/sbin/libvirtd)
2024-04-23 23:46:58.567+0000: shutting down, reason=destroyed
2024-04-23 23:47:43.841+0000: Starting external device: TPM Emulator
/usr/bin/swtpm socket --ctrl 'type=unixio,path=/run/libvirt/qemu/swtpm/2-Windows 11 fresh-swtpm.sock,mode=0600' --tpmstate dir=/var/lib/libvirt/swtpm/b7221e7a-31f0-8de2-b0ed-20e67fd3f7b6/tpm2,mode=0600 --log 'file=/var/log/swtpm/libvirt/qemu/Windows 11 fresh-swtpm.log' --terminate --tpm2
2024-04-23 23:47:43.873+0000: starting up libvirt version: 8.7.0, qemu version: 7.2.0, kernel: 6.1.79-Unraid, hostname: Tower
LC_ALL=C \
PATH=/bin:/sbin:/usr/bin:/usr/sbin \
HOME='/var/lib/libvirt/qemu/domain-2-Windows 11 fresh' \
XDG_DATA_HOME='/var/lib/libvirt/qemu/domain-2-Windows 11 fresh/.local/share' \
XDG_CACHE_HOME='/var/lib/libvirt/qemu/domain-2-Windows 11 fresh/.cache' \
XDG_CONFIG_HOME='/var/lib/libvirt/qemu/domain-2-Windows 11 fresh/.config' \
/usr/local/sbin/qemu \
-name 'guest=Windows 11 fresh,debug-threads=on' \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-2-Windows 11 fresh/master-key.aes"}' \
-blockdev '{"driver":"file","filename":"/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/etc/libvirt/qemu/nvram/b7221e7a-31f0-8de2-b0ed-20e67fd3f7b6_VARS-pure-efi-tpm.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
-machine pc-q35-7.2,usb=off,dump-guest-core=off,mem-merge=off,memory-backend=pc.ram,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format \
-accel kvm \
-cpu host,migratable=on,topoext=on,hv-time=on,hv-relaxed=on,hv-vapic=on,hv-spinlocks=0x1fff,hv-vendor-id=none,kvm=off,host-cache-info=on,l3-cache=off \
-m 8192 \
-object '{"qom-type":"memory-backend-ram","id":"pc.ram","size":8589934592}' \
-overcommit mem-lock=off \
-smp 6,sockets=1,dies=1,cores=3,threads=2 \
-uuid b7221e7a-31f0-8de2-b0ed-20e67fd3f7b6 \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=35,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device '{"driver":"pcie-root-port","port":16,"chassis":1,"id":"pci.1","bus":"pcie.0","multifunction":true,"addr":"0x2"}' \
-device '{"driver":"pcie-root-port","port":17,"chassis":2,"id":"pci.2","bus":"pcie.0","addr":"0x2.0x1"}' \
-device '{"driver":"pcie-root-port","port":18,"chassis":3,"id":"pci.3","bus":"pcie.0","addr":"0x2.0x2"}' \
-device '{"driver":"pcie-root-port","port":19,"chassis":4,"id":"pci.4","bus":"pcie.0","addr":"0x2.0x3"}' \
-device '{"driver":"pcie-root-port","port":20,"chassis":5,"id":"pci.5","bus":"pcie.0","addr":"0x2.0x4"}' \
-device '{"driver":"qemu-xhci","p2":15,"p3":15,"id":"usb","bus":"pcie.0","addr":"0x7"}' \
-device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.2","addr":"0x0"}' \
-blockdev '{"driver":"file","filename":"/mnt/user/isos/Win11_23H2_English_x64v2.iso","node-name":"libvirt-3-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-3-format","read-only":true,"driver":"raw","file":"libvirt-3-storage"}' \
-device '{"driver":"ide-cd","bus":"ide.0","drive":"libvirt-3-format","id":"sata0-0-0","bootindex":2}' \
-blockdev '{"driver":"file","filename":"/mnt/user/isos/virtio-win-0.1.225-1.iso","node-name":"libvirt-2-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":true,"driver":"raw","file":"libvirt-2-storage"}' \
-device '{"driver":"ide-cd","bus":"ide.1","drive":"libvirt-2-format","id":"sata0-0-1"}' \
-blockdev '{"driver":"host_device","filename":"/dev/disk/by-id/ata-Samsung_SSD_870_EVO_1TB_S75BNL0WC18464H","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \
-device '{"driver":"ide-hd","bus":"ide.2","drive":"libvirt-1-format","id":"sata0-0-2","bootindex":1,"write-cache":"on","serial":"vdisk1"}' \
-netdev tap,fd=36,id=hostnet0 \
-device '{"driver":"virtio-net","netdev":"hostnet0","id":"net0","mac":"52:54:00:68:6a:5b","bus":"pci.1","addr":"0x0"}' \
-chardev pty,id=charserial0 \
-device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0}' \
-chardev socket,id=charchannel0,fd=34,server=on,wait=off \
-device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \
-chardev 'socket,id=chrtpm,path=/run/libvirt/qemu/swtpm/2-Windows 11 fresh-swtpm.sock' \
-tpmdev emulator,id=tpm-tpm0,chardev=chrtpm \
-device '{"driver":"tpm-tis","tpmdev":"tpm-tpm0","id":"tpm0"}' \
-device '{"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-vnc 0.0.0.0:0,websocket=5700,audiodev=audio1 \
-k en-us \
-device '{"driver":"qxl-vga","id":"video0","max_outputs":1,"ram_size":67108864,"vram_size":67108864,"vram64_size_mb":0,"vgamem_mb":16,"bus":"pcie.0","addr":"0x1"}' \
-device '{"driver":"vfio-pci","host":"0000:0e:00.0","id":"hostdev0","bus":"pci.3","addr":"0x0","romfile":"/mnt/cache/isos/vfios/243157.rom"}' \
-device '{"driver":"vfio-pci","host":"0000:0e:00.1","id":"hostdev1","bus":"pci.4","addr":"0x0"}' \
-device '{"driver":"vfio-pci","host":"0000:05:00.0","id":"hostdev2","bus":"pci.5","addr":"0x0"}' \
-device '{"driver":"usb-host","hostdevice":"/dev/bus/usb/003/003","id":"hostdev3","bus":"usb.0","port":"2"}' \
-device '{"driver":"usb-host","hostdevice":"/dev/bus/usb/001/002","id":"hostdev4","bus":"usb.0","port":"3"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
char device redirected to /dev/pts/0 (label charserial0)
qxl_send_events: spice-server bug: guest stopped, ignoring

This syslog is when attempting to boot the VM with the GPU attached still

syslog-2.txt

please review documentation in another form:

https://forums.unraid.net/topic/164064-cant-passthrough-nvidia-gpu-anymore-to-vm-used-to-in-previous-unraid-versions/#:~:text=Your hardware steps checklist%3A

the tower diagnositc is better than syslog as there is quite a bit i have to rummage thorugh and comapre.
IT is better to get a picture of the system drivers over the devices you are wanting to passthorugh as then we can verify corect stubiing 
including the output of lscpi -v to verify the vfio-pci kerneal driver in use.

then there xml edits needed inteh VM to corecly pass a g-card.

I'm currently reviewing your diagnostic zip.

After reviewing your diagnostic file. I Have found that your vfio-pci stubbing is bad.
Kernel driver in use: amdgpu
should be vfio-pci
....

I also recommend installing the amd reset plugin.


In your case if the system drivers show a green circle and a checkbox to the correct GPU... Then you will need a vfio modprobe edit

https://forums.unraid.net/topic/164064-cant-passthrough-nvidia-gpu-anymore-to-vm-used-to-in-previous-unraid-versions/#:~:text=access to modprobe...-,/Tools/SysDrivers,-Top right%3A

 

#Prepare amd g-Card Kerneal Driver:
For AMD
softdep radeon pre: vfio-pci
softdep amdgpu pre: vfio-pci

#GPU:  Navi 23 [Radeon RX 6600/6600 XT/6600M] [1849:5217] [AMD/ATI] Navi 21/23 HDMI/DP Audio Controller [1002:ab28]
options vfio-pci ids=1849:5217,1002:ab28 disable_idle_d3=1 disable_vga=1

#USB PCI Card
#options vfio-pci ids= disable_idle_d3=1 enable_sriov disable_denylist

also you appear to be making quite a few windows VM


Review to fix gpu xml data: