I set up a windows 11 gaming vm about two days ago; everything seemed to be working great in terms of performance and passthrough of gpu, nvme, usb controllers etc. I ended up switching windows sleep back on after originally turning it off to see if I could save my monitor from staying awake when the vm was not in use. The vm went to sleep and showed up as paused in the vm manager, trying to start it from that state resulted in an error so I force killed it and started it back up again which caused the entire unraid server to crash and reboot.

 

Since then anytime I try to start the windows 11 vm or create a new one and start that unraid completely crashes. I can run unraid fine otherwise and my ubuntu server vm (with the same gpu passed through) can start up just fine. I can also boot the machine directly into windows instead of unraid and everything seems to work perfectly there (except bluetooth for some reason). Would appreciate any suggestions on how to resolve this as having to reboot each time I want to use windows is a bit annoying:

 

VM setup in the pictures below the only editing I did to the xml was to follow step 1 of this page: 

 

 

Edit: I have tried to create a new windows vm with a fresh install by using a vdisk and not passing the vfio bound nvme but trying to start that vm and install windows also causes unraid to crash.

 

Edited August 18, 20232 yr by fadynakhla
additional info

Here is the vm xml; it seems that for some reason the few xml edits I made are no longer there:

 

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>Windows 11</name>
  <uuid>fba0813a-d1a0-5031-957b-d8cbfb6d1ab6</uuid>
  <description>Gaming Rig</description>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 11" icon="windows11.png" os="windowstpm"/>
  </metadata>
  <memory unit='KiB'>24641536</memory>
  <currentMemory unit='KiB'>24641536</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>14</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='1'/>
    <vcpupin vcpu='1' cpuset='9'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='10'/>
    <vcpupin vcpu='4' cpuset='3'/>
    <vcpupin vcpu='5' cpuset='11'/>
    <vcpupin vcpu='6' cpuset='4'/>
    <vcpupin vcpu='7' cpuset='12'/>
    <vcpupin vcpu='8' cpuset='5'/>
    <vcpupin vcpu='9' cpuset='13'/>
    <vcpupin vcpu='10' cpuset='6'/>
    <vcpupin vcpu='11' cpuset='14'/>
    <vcpupin vcpu='12' cpuset='7'/>
    <vcpupin vcpu='13' cpuset='15'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-7.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/fba0813a-d1a0-5031-957b-d8cbfb6d1ab6_VARS-pure-efi-tpm.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv mode='custom'>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='7' threads='2'/>
    <cache mode='passthrough'/>
    <feature policy='require' name='topoext'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:92:ef:42'/>
      <source bridge='br0'/>
      <model type='virtio-net'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <tpm model='tpm-tis'>
      <backend type='emulator' version='2.0' persistent_state='yes'/>
    </tpm>
    <audio id='1' type='none'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0e' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x11' slot='0x00' function='0x4'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x1b1c'/>
        <product id='0x0c32'/>
      </source>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>

 

50 minutes ago, fadynakhla said:

Here is the vm xml; it seems that for some reason the few xml edits I made are no longer there:

 

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>Windows 11</name>
  <uuid>fba0813a-d1a0-5031-957b-d8cbfb6d1ab6</uuid>
  <description>Gaming Rig</description>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 11" icon="windows11.png" os="windowstpm"/>
  </metadata>
  <memory unit='KiB'>24641536</memory>
  <currentMemory unit='KiB'>24641536</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>14</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='1'/>
    <vcpupin vcpu='1' cpuset='9'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='10'/>
    <vcpupin vcpu='4' cpuset='3'/>
    <vcpupin vcpu='5' cpuset='11'/>
    <vcpupin vcpu='6' cpuset='4'/>
    <vcpupin vcpu='7' cpuset='12'/>
    <vcpupin vcpu='8' cpuset='5'/>
    <vcpupin vcpu='9' cpuset='13'/>
    <vcpupin vcpu='10' cpuset='6'/>
    <vcpupin vcpu='11' cpuset='14'/>
    <vcpupin vcpu='12' cpuset='7'/>
    <vcpupin vcpu='13' cpuset='15'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-7.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/fba0813a-d1a0-5031-957b-d8cbfb6d1ab6_VARS-pure-efi-tpm.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv mode='custom'>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='7' threads='2'/>
    <cache mode='passthrough'/>
    <feature policy='require' name='topoext'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:92:ef:42'/>
      <source bridge='br0'/>
      <model type='virtio-net'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <tpm model='tpm-tis'>
      <backend type='emulator' version='2.0' persistent_state='yes'/>
    </tpm>
    <audio id='1' type='none'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x0e' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x11' slot='0x00' function='0x4'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x1b1c'/>
        <product id='0x0c32'/>
      </source>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>

 

making changes  via the gui will replace custom xml changes.

 

Do you have an external syslog server? maybe post diagnostics?

27 minutes ago, SimonF said:

making changes  ka the gui will replace custom xml changes.

 

Yeah just hadn't realized I had made any additional gui changes since but may have been  adding usb devices or something

 

27 minutes ago, SimonF said:

Do you have an external syslog server? maybe post diagnostics?

I dont have syslog but can set it up then try to start the vm, if i use flash or local (share) would the syslog likely contain useful info to look at what caused the crash or do i need to use remote? Not sure if local syslog would be properly written to for a whole system crash

Here is the diagnostics file:

 

tower-diagnostics-20230818-1258.zip

Edited August 18, 20232 yr by fadynakhla

5 minutes ago, fadynakhla said:

would the syslog likely contain useful info to look at what caused the crash or do i need to use remote?

Does not need to be remote. Syslog is lost during a reboot as in Ram and will only show from system start. May provide info i..e call traces. But does sound like hardware lock up.

Cool thank you! I'll set this up and then crash the machine and share syslog when I get the chance. Do you think it would be worth wiping the drive with the windows install? I was considering it but since the issue also occured when I tried to spin up a fresh one without that drive at all I'm assuming its something else.

7 minutes ago, fadynakhla said:

Cool thank you! I'll set this up and then crash the machine and share syslog when I get the chance. Do you think it would be worth wiping the drive with the windows install? I was considering it but since the issue also occured when I tried to spin up a fresh one without that drive at all I'm assuming its something else.

It would see likely not to be nvme. Have you tried the new vm just with virtual rather than nvidia gpu tl check it works ok.

15 minutes ago, SimonF said:

It would see likely not to be nvme. Have you tried the new vm just with virtual rather than nvidia gpu tl check it works ok.

Actually no I havent I figured it was fine as that was working properly with drivers installing automatically and gaming well etc. Also, I dont have the gpu bound to vfio at boot not sure if that has an impact but passes through fine to both ubuntu and windows at least until windows broke

 

Edit: I'll try out new vm with virtual instead

Edited August 18, 20232 yr by fadynakhla

Update: I was able to create and start a fresh windows vm with virtual graphics and then also a fresh vm with graphics card passed through (nothing else), so I tried to boot the existing vm and whole machine crashed again.

 

The share I created and set for syslogs is still empty though oddly

 

Edit: I set the graphics for the existing vm to virtual and removed all passed through devices (except the boot drive) and can start the vm with vnc

Edit 2: System wide crash happens again when I pass through the usb controllers and start...

Edited August 19, 20232 yr by fadynakhla

So I think I have figured out what was going, unfortunately looks like the issue is with the usb hub built into my monitor that I am plugging it in to one of the passed through ports. When I unplug it I can pass the usb controller start the vm then plug the monitor's usb hub in after the vm starts to use it. Kinda annoying but i suppose it works

FInal update on this, I'm not sure why but after some testing I isolated the issue to one of the usb controllers on my motherboard. I rearanged the usb devices that I dont want to pass through and tested out other usb controllers and am having no issues at all starting and stopping my VM with the controller passed through and any number of devices attached. To anyone experiencing this sort of issue I would recommend trying other usb controllers.