Windows VM with GPU Passthrough Eventually Locks Up

randing · October 15, 2023

Unraid version 6.12.4, Intel Xeon E-2288G CPU, ASUS WS C246M PRO/SE motherboard, 64gb of Samsung M391A2K43BB1-CTD ECC that passed 24 hours of memtest86, SilverStone FX600 PSU, Windows 11 Pro VM (also tested Server 2022). New to Unraid this year, so not a lot of experience with versions prior to 6.12.x. Having an issue where Windows VMs with GPU passthrough eventually lock up with all CPU cores pegged at 100%. Passing through an Nvidia GPU or Intel UHD causes the same behavior. Sometimes it locks up within an hour, sometimes after a day or two, Nvidia seems a bit more stable. A second Windows VM configured the same way but with VNC virtual graphics runs fine.

Edgard666 · October 20, 2023

Hi Randing,

I have the exact same problem with a Windows 11 pro VM with a GeForce GTX1070 passthrough on Unraid 6.12.4.

I use the Nvidia card as a graphic card and as a sound card also on the same VM. If I remove the Nvidia card from the VM completely, no more issue. If I leave either the sound card only or the GC only, some or all the CPU cores goes up to 100% after some time and the machine become unreachable and non-responsive to normal “stop". It’s not possible to PING it anymore, and I have to do a “force stop” and restart it to be able to access it again.

And there is nothing on the Windows Events Viewer which could help pinpoint the cause of the problem.

I tried to change the Nvidia drivers to an older version, the virtual machine version from 7.1 to 7.0 and to use an oldest VirtIO Drivers ISO version, but nothing helped.

I also tried to connect a screen on the GC itself, but no image is visible, so that doesn’t help.

For me the problem is quicker when the machine is on heavy load, but it can happen after 5 minutes or after a few hours, it really varies, but never take more than a few hours at best.

The Windows 11 installation is quite new and clean, and during the first few days after the installation, I didn’t have any issue. It just happened once, and doesn’t stop happening since then. The Windows CPU core isolation feature is disabled, and the Windows Defender is also disable, so it’s not related to them.

I also have two other Windows (one W11 and one W10) VMs running on the same Unraid server, but without a GPU passthrough, and they don't have any issues.

Here is a copy of my VM XML File, in case that could help :

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm' id='12'>
  <name>Ed-W11</name>
  <uuid>a2fb0a97-e071-37cf-688c-c4280557958f</uuid>
  <description>VM Win 11</description>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 11" icon="windows11.png" os="windowstpm"/>
  </metadata>
  <memory unit='KiB'>24641536</memory>
  <currentMemory unit='KiB'>24641536</currentMemory>
  <memoryBacking>
    <nosharepages/>
    <source type='memfd'/>
    <access mode='shared'/>
  </memoryBacking>
  <vcpu placement='static'>16</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='12'/>
    <vcpupin vcpu='2' cpuset='3'/>
    <vcpupin vcpu='3' cpuset='13'/>
    <vcpupin vcpu='4' cpuset='4'/>
    <vcpupin vcpu='5' cpuset='14'/>
    <vcpupin vcpu='6' cpuset='5'/>
    <vcpupin vcpu='7' cpuset='15'/>
    <vcpupin vcpu='8' cpuset='6'/>
    <vcpupin vcpu='9' cpuset='16'/>
    <vcpupin vcpu='10' cpuset='7'/>
    <vcpupin vcpu='11' cpuset='17'/>
    <vcpupin vcpu='12' cpuset='8'/>
    <vcpupin vcpu='13' cpuset='18'/>
    <vcpupin vcpu='14' cpuset='9'/>
    <vcpupin vcpu='15' cpuset='19'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-7.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/a2fb0a97-e071-37cf-688c-c4280557958f_VARS-pure-efi-tpm.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='8' threads='2'/>
    <cache mode='passthrough'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domainsfast/Ed-W11/vdisk1.img' index='2'/>
      <backingStore/>
      <target dev='hdc' bus='virtio'/>
      <serial>vdisk1</serial>
      <boot order='1'/>
      <alias name='virtio-disk2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Ed-W11/vdisk2.img' index='1'/>
      <backingStore/>
      <target dev='hdd' bus='virtio'/>
      <serial>vdisk2</serial>
      <alias name='virtio-disk3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <alias name='usb'/>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <alias name='usb'/>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <alias name='usb'/>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:10:12:31'/>
      <source bridge='br0'/>
      <target dev='vnet11'/>
      <model type='virtio-net'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-12-Ed-W11/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <alias name='input0'/>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'>
      <alias name='input1'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input2'/>
    </input>
    <tpm model='tpm-tis'>
      <backend type='emulator' version='2.0' persistent_state='yes'/>
      <alias name='tpm0'/>
    </tpm>
    <audio id='1' type='none'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+100</label>
    <imagelabel>+0:+100</imagelabel>
  </seclabel>
</domain>

The "Fix Common Problems" plugin doesn't find anyting, so I must say taht I can't think of anything new to try to make it work.

Nothing seams to help

So if you someone has an idea, feel free to share please 😞

The VM does have CPU PINNING. It has access to all cores except 0/10 and 1/11, which are reserved for Unraid.

And a complete VM Logs, from start to crash:


/usr/bin/swtpm socket --ctrl type=unixio,path=/run/libvirt/qemu/swtpm/13-Ed-W11-swtpm.sock,mode=0600 --tpmstate dir=/var/lib/libvirt/swtpm/a2fb0a97-e071-37cf-688c-c4280557958f/tpm2,mode=0600 --log file=/var/log/swtpm/libvirt/qemu/Ed-W11-swtpm.log --terminate --tpm2
2023-10-20 16:39:05.695+0000: starting up libvirt version: 8.7.0, qemu version: 7.1.0, kernel: 6.1.49-Unraid, hostname: NAS
LC_ALL=C \
PATH=/bin:/sbin:/usr/bin:/usr/sbin \
HOME=/var/lib/libvirt/qemu/domain-13-Ed-W11 \
XDG_DATA_HOME=/var/lib/libvirt/qemu/domain-13-Ed-W11/.local/share \
XDG_CACHE_HOME=/var/lib/libvirt/qemu/domain-13-Ed-W11/.cache \
XDG_CONFIG_HOME=/var/lib/libvirt/qemu/domain-13-Ed-W11/.config \
/usr/local/sbin/qemu \
-name guest=Ed-W11,debug-threads=on \
-S \
-object '{"qom-type":"secret","id":"masterKey0","format":"raw","file":"/var/lib/libvirt/qemu/domain-13-Ed-W11/master-key.aes"}' \
-blockdev '{"driver":"file","filename":"/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd","node-name":"libvirt-pflash0-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash0-format","read-only":true,"driver":"raw","file":"libvirt-pflash0-storage"}' \
-blockdev '{"driver":"file","filename":"/etc/libvirt/qemu/nvram/a2fb0a97-e071-37cf-688c-c4280557958f_VARS-pure-efi-tpm.fd","node-name":"libvirt-pflash1-storage","auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-pflash1-format","read-only":false,"driver":"raw","file":"libvirt-pflash1-storage"}' \
-machine pc-i440fx-7.1,usb=off,dump-guest-core=off,mem-merge=off,memory-backend=pc.ram,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format \
-accel kvm \
-cpu host,migratable=on,hv-time=on,host-cache-info=on,l3-cache=off \
-m 24064 \
-object '{"qom-type":"memory-backend-memfd","id":"pc.ram","share":true,"x-use-canonical-path-for-ramblock-id":false,"size":25232932864}' \
-overcommit mem-lock=off \
-smp 16,sockets=1,dies=1,cores=8,threads=2 \
-uuid a2fb0a97-e071-37cf-688c-c4280557958f \
-display none \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=36,server=on,wait=off \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device '{"driver":"ich9-usb-ehci1","id":"usb","bus":"pci.0","addr":"0x7.0x7"}' \
-device '{"driver":"ich9-usb-uhci1","masterbus":"usb.0","firstport":0,"bus":"pci.0","multifunction":true,"addr":"0x7"}' \
-device '{"driver":"ich9-usb-uhci2","masterbus":"usb.0","firstport":2,"bus":"pci.0","addr":"0x7.0x1"}' \
-device '{"driver":"ich9-usb-uhci3","masterbus":"usb.0","firstport":4,"bus":"pci.0","addr":"0x7.0x2"}' \
-device '{"driver":"virtio-serial-pci","id":"virtio-serial0","bus":"pci.0","addr":"0x3"}' \
-blockdev '{"driver":"file","filename":"/mnt/user/domainsfast/Ed-W11/vdisk1.img","node-name":"libvirt-2-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-2-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-2-storage"}' \
-device '{"driver":"virtio-blk-pci","bus":"pci.0","addr":"0x4","drive":"libvirt-2-format","id":"virtio-disk2","bootindex":1,"write-cache":"on","serial":"vdisk1"}' \
-blockdev '{"driver":"file","filename":"/mnt/user/domains/Ed-W11/vdisk2.img","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \
-device '{"driver":"virtio-blk-pci","bus":"pci.0","addr":"0x5","drive":"libvirt-1-format","id":"virtio-disk3","write-cache":"on","serial":"vdisk2"}' \
-netdev tap,fd=38,id=hostnet0 \
-device '{"driver":"virtio-net","netdev":"hostnet0","id":"net0","mac":"52:54:00:10:12:31","bus":"pci.0","addr":"0x2"}' \
-chardev pty,id=charserial0 \
-device '{"driver":"isa-serial","chardev":"charserial0","id":"serial0","index":0}' \
-chardev socket,id=charchannel0,fd=35,server=on,wait=off \
-device '{"driver":"virtserialport","bus":"virtio-serial0.0","nr":1,"chardev":"charchannel0","id":"channel0","name":"org.qemu.guest_agent.0"}' \
-chardev socket,id=chrtpm,path=/run/libvirt/qemu/swtpm/13-Ed-W11-swtpm.sock \
-tpmdev emulator,id=tpm-tpm0,chardev=chrtpm \
-device '{"driver":"tpm-tis","tpmdev":"tpm-tpm0","id":"tpm0"}' \
-device '{"driver":"usb-tablet","id":"input0","bus":"usb.0","port":"1"}' \
-audiodev '{"id":"audio1","driver":"none"}' \
-device '{"driver":"vfio-pci","host":"0000:01:00.0","id":"hostdev0","bus":"pci.0","addr":"0x6"}' \
-device '{"driver":"vfio-pci","host":"0000:01:00.1","id":"hostdev1","bus":"pci.0","addr":"0x8"}' \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
char device redirected to /dev/pts/0 (label charserial0)

Edited October 20, 2023 by Edgard666

randing · October 20, 2023

Edgard666, thanks for your thoughts, and good notes. I've tried every variation of VM configuration I could come up with and built VMs from scratch a few times while testing, all with the same result. I manually rolled back to 6.11.5 this week, but that didn't help. There seems to be a few people having the same issue:

Edited October 21, 2023 by randing

Edgard666 · October 21, 2023

Indeed, I searched the internet from top to bottom and did find a lot of people having the same issue. But there never was a real solution mentioned... Or at least I didn't find it. 😞

Sometimes, reinstalling the VM seams to work, but it's not a solution. And that same problem can occur with Windows VMs, but also Linux VMs on occasion. So it has to be a KVM/unRAID issue. Maybe the KVM drivers...

All I could isolate from those forum posts, is that it's a PCI-E passthrough bug...

Edited October 21, 2023 by Edgard666

SimonF · October 21, 2023

20 hours ago, Edgard666 said: