Unraid crash on Windows 10 reboot

[woodman]

Hello Everyone,

 

I am a new user to Unraid within the past 2 months.  I am very impressed with Unraid and grateful for its community which has been very helpful in getting everything I need setup.

 

Here is some background on the issue I am having:

 

My windows 10 VM will crash almost everytime it is rebooted.  The crash sometimes involves video artifacts on startup and the monitor going black in the final stages of Windows 10 boot.  After this, my Unraid server becomes unresponsive and I am forced to hard reboot.  I have found that if I uninstall all Nvidia drivers including those that come preinstalled with windows and therefore use "MS generic driver" the reboot works but GPU acceleration is effectively gone.  I have tried many different versions of nvidia drivers.  Shutting down the VM first and then starting it from the Unraid dashboard manually does not cause a crash and works every time, so this has been my workaround for now.

 

I have a Xeon quad core processor and I am passing my one and only video card Gigabyte GTX750ti.  To set up my VM I followed gridrunner's latest Windows 10 VM guide and his guide on passing a single Nvidia GPU to a VM.  I have double checked that the vbios version I have matches the vbios physically on the GPU.  When I first set everything up, I did not have this problem of crashing on reboot and it only developed later.  I thought the issue was related to me passing my fresco pcie usb controller by stubbing from unraid and passing it to the vm.  However, disabling that passthrough and all USB devices I still have a full system crash on reboot.  This leads me to believe this is related to my GPU passthrough.  The fact that by first shutting down and then starting again works everytime gives me hope that there may be a solution.

 

I am willing to provide any further details to those who can help me troubleshoot this.  Please let me know how I can best provide the crash logs generated when the VM crash causes unraid to crash.

 

Thanks for your help.

 

Below is my system devices:

 

Quote

IOMMU group 0:	[8086:0c08] 00:00.0 Host bridge: Intel Corporation Xeon E3-1200 v3 Processor DRAM Controller (rev 06)
IOMMU group 1:	[8086:0c01] 00:01.0 PCI bridge: Intel Corporation Xeon E3-1200 v3/4th Gen Core Processor PCI Express x16 Controller (rev 06)
IOMMU group 2:	[8086:8cb1] 00:14.0 USB controller: Intel Corporation 9 Series Chipset Family USB xHCI Controller
IOMMU group 3:	[8086:8cba] 00:16.0 Communication controller: Intel Corporation 9 Series Chipset Family ME Interface #1
IOMMU group 4:	[8086:15a1] 00:19.0 Ethernet controller: Intel Corporation Ethernet Connection (2) I218-V
IOMMU group 5:	[8086:8cad] 00:1a.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #2
IOMMU group 6:	[8086:8ca0] 00:1b.0 Audio device: Intel Corporation 9 Series Chipset Family HD Audio Controller
IOMMU group 7:	[8086:8c90] 00:1c.0 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 1 (rev d0)
IOMMU group 8:	[8086:244e] 00:1c.3 PCI bridge: Intel Corporation 82801 PCI Bridge (rev d0)
IOMMU group 9:	[8086:8c98] 00:1c.4 PCI bridge: Intel Corporation 9 Series Chipset Family PCI Express Root Port 5 (rev d0)
IOMMU group 10:	[8086:8ca6] 00:1d.0 USB controller: Intel Corporation 9 Series Chipset Family USB EHCI Controller #1
IOMMU group 11:	[8086:8cc6] 00:1f.0 ISA bridge: Intel Corporation 9 Series Chipset Family H97 Controller
	[8086:8c82] 00:1f.2 SATA controller: Intel Corporation 9 Series Chipset Family SATA Controller [AHCI Mode]
	[8086:8ca2] 00:1f.3 SMBus: Intel Corporation 9 Series Chipset Family SMBus Controller
IOMMU group 12:	[10de:1380] 01:00.0 VGA compatible controller: NVIDIA Corporation GM107 [GeForce GTX 750 Ti] (rev a2)
	[10de:0fbc] 01:00.1 Audio device: NVIDIA Corporation Device 0fbc (rev a1)
IOMMU group 13:	[1b21:1080] 03:00.0 PCI bridge: ASMedia Technology Inc. ASM1083/1085 PCIe to PCI Bridge (rev 03)
IOMMU group 14:	[1b73:1100] 05:00.0 USB controller: Fresco Logic FL1100 USB 3.0 Host Controller (rev 10)

 

 

Win 10 VM XML:

Quote

<domain type='kvm' id='1'>
  <name>Windows 10</name>
  <uuid>19005b3e-58f8-173b-0c64-523553389622</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>12582912</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='2'/>
    <vcpupin vcpu='1' cpuset='6'/>
    <vcpupin vcpu='2' cpuset='3'/>
    <vcpupin vcpu='3' cpuset='7'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.11'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/19005b3e-58f8-173b-0c64-523553389622_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none'>
    <topology sockets='1' cores='2' threads='2'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/disks/Samsung_120_ssd/Windows 10/vdisk1.img'/>
      <backingStore/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <alias name='virtio-disk2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/software/Microsoft/Windows10_2.24.2018.iso'/>
      <backingStore/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <alias name='ide0-0-0'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.141-1.iso'/>
      <backingStore/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <alias name='ide0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:6c:22:8f'/>
      <source bridge='br0'/>
      <target dev='vnet0'/>
      <model type='virtio'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-1-Windows 10/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <rom file='/mnt/user/domains/vbios/750ti.dump'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x1b' function='0x0'/>
      </source>
      <alias name='hostdev2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <memballoon model='virtio'>
      <alias name='balloon0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </memballoon>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+100</label>
    <imagelabel>+0:+100</imagelabel>
  </seclabel>
</domain>
 

 

 

 

VM form view attached

 

Edited April 7, 2018 by woodman

[woodman]

After some more testing,  if I set max memory to 8192 (same as initial memory), the VM has stopped crashing on reboot.  I am glad that it now works!  Any explanation as to why this solves the issue?

Edited April 7, 2018 by woodman

[PROJECTBLUE]

On 4/7/2018 at 11:08 PM, woodman said:

After some more testing,  if I set max memory to 8192 (same as initial memory), the VM has stopped crashing on reboot.  I am glad that it now works!  Any explanation as to why this solves the issue?

 

Just guessing here... but if KVM works anything like Hyper-V does it's required to use a static amount of memory for the virtual machine when passing through PCI-E devices.
Something to do with memory allocation for the hardware.