[Solved] Windows 10 GTX 960 Reboot Loop

[stratus]

I'm new to Unraid. I just installed Windows 10 in a VM and the install went well. Everything seems fine over VNC, but I want to pass through the video card. My GTX 960 shows up in Device Manager but without drivers. I've tried installing them both from Device Manager and by using Nvidia's install package, but after install, booting back into Windows is impossible. It gets stuck in a reboot loop (system_service_exception), then it offers repair options. Everything fails; Windows tells me the drive is locked, repair options are unavailable, etc. It can't seem to see any drives. I can't do anything except create a new VM and try again, which I've done 3 times.

 

Any suggestions about how I might proceed? Thanks!

[stratus]

I can't delete the original thread above, but the issue doesn't seem to be related to the video card after all. I'm unable to reboot successfully after Windows has been installed (even though it reboots fine during the installation). On this Windows install attempt (the 5th) I tried making the boot partition active before rebooting. Now it's stuck in a loop with the attached message: "File: \Boot\BCD Status: 0xc0000000f Info: The Boot Configuration Data for your PC is missing or contains errors." When I was still able to get into recovery tools in an early install, trying to list disks using diskpart from the command prompt didn't show my partitions.

[stratus]

I installed a Windows 8.1 VM and confirmed that this doesn't just happen with Windows 10. Windows 8.1 exhibits the same behaviour: fine until after installation and then rebooting throws the system_service_exception error. With Windows 8.1 I do see more recovery tools, but many tell me that an OS isn't installed, and when I go to a file manager I only see the install DVD and the VirtlIO drives. If I remove the GTX 960 from the VM template, Windows will start. Reenable it and the errors are back and the drive goes away. I enabled PCIe ACS Override, which didn't seem to help at all (except that I see a black screen with mouse cursor in VNC on troubleshooting screens now).

 

Any ideas about next steps or why the drive might be disappearing and throwing the Windows error?

[stratus]

I am still unfortunately not making progress. I've tried Windows VMs with both bios types and machine types. I tried specifying a GPU ROM. I was able to see the disk properly in recovery tools once by setting the bus to ide, per http://lime-technology.com/forum/index.php?topic=39493.0, and there I could confirm a 3(b) BSOD related to the graphics card. On reboot, however, I can't see the disk again (per my second post), even though my XML file hasn't changed. It seems as if everything is fine during the OS install, then after install Windows can't see the boot drive, and when it can, it hits GPU errors.

 

This was a new build using common hardware that seems very similar to what has worked for other people (Unraid 6.0.1, i7-4790k, eVGA GTX 960, ASRock Z97 Extreme6).

 

In case it's useful, I'm including my XML file (ide reverted to virtio).

 

<domain type='kvm' xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <name>Win8.1</name>
  <uuid>1570c638-1e34-21fa-df9a-dc157fb26f4e</uuid>
  <metadata>
    <vmtemplate name="Custom" icon="windows.png" os="windows"/>
  </metadata>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <nosharepages/>
    <locked/>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='0'/>
    <vcpupin vcpu='1' cpuset='1'/>
    <vcpupin vcpu='2' cpuset='2'/>
    <vcpupin vcpu='3' cpuset='3'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-2.3'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough'>
    <topology sockets='1' cores='4' threads='1'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/bin/qemu-system-x86_64</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/Downloads/iso/Win81.iso'/>
      <target dev='hda' bus='ide'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/Downloads/win-driver/virtio-win-0.1.109.iso'/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/Win81/Win8.1/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </disk>
    <controller type='usb' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='ide' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:9d:c3:04'/>
      <source bridge='virbr0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target port='0'/>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/Win8.1.org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'/>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='vmvga' vram='16384' heads='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <hostdev mode='subsystem' type='usb' managed='yes'>
      <source>
        <vendor id='0x045e'/>
        <product id='0x0768'/>
      </source>
    </hostdev>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </memballoon>
  </devices>
  <qemu:commandline>
    <qemu:arg value='-device'/>
    <qemu:arg value='ioh3420,bus=pci.0,addr=1c.0,multifunction=on,port=2,chassis=1,id=root.1'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=01:00.0,bus=root.1,addr=00.0,multifunction=on,x-vga=on,romfile=/mnt/user/windows10/EVGA.GTX960.2048.141229.rom'/>
    <qemu:arg value='-device'/>
    <qemu:arg value='vfio-pci,host=00:1b.0,bus=root.1,addr=01.0'/>
  </qemu:commandline>
</domain>

[jonp]

Hi stratus,

 

Sorry for the delay in getting to this one.  First and foremost, let's go over a few basics:

 

1 - Is your motherboard BIOS up to date?

2 - When you assign your GPU, did you remove the VNC graphics device or leave it there?

3 - Have you tried using a different physical storage device for your VM image?

4 - After having this error, please go to the Tools -> Diagnostics page and click the "Collect" button there.  Upload the zip file it downloads to the forum here so I can review it.

 

Thanks!

[stratus]

Thanks Jon, I really appreciate the reply.

 

1 - The motherboard BIOS wasn't updated when I first started, but it was after the most recent attempts.

2 - I didn't do an initial install without VNC. After install then yes, I tried disabling it and only had the GTX 960.

3 - No. All attempts were on the cache drive only (the only SSD in the system). I can try again in the array.

4 - Attached. It looks like previous attempts aren't included, since I'd rebooted, so I tried running some combinations again. I tried Windows 10 with the GTX 960 only (no VNC) and heard the fans spinning up every 30 seconds, so I assume it's in a reboot loop. The backlight on my keyboard (also passed through) was flickering a lot throughout, which was odd.

 

Let me know if there's anything else that might help.

tower-diagnostics-20150818-1040.zip

[jonp]

Thanks Jon, I really appreciate the reply.

 

1 - The motherboard BIOS wasn't updated when I first started, but it was after the most recent attempts.

2 - I didn't do an initial install without VNC. After install then yes, I tried disabling it and only had the GTX 960.

3 - No. All attempts were on the cache drive only (the only SSD in the system). I can try again in the array.

4 - Attached. It looks like previous attempts aren't included, since I'd rebooted, so I tried running some combinations again. I tried Windows 10 with the GTX 960 only (no VNC) and heard the fans spinning up every 30 seconds, so I assume it's in a reboot loop. The backlight on my keyboard (also passed through) was flickering a lot throughout, which was odd.

 

Let me know if there's anything else that might help.

 

Ok, reviewing your diagnostics now and already noticed a few things: 

 

You're using "latest" virtio drivers, not the "stable" branch.

• Please download the "stable" version of those drivers and try setting up a new VM with those. 
  
• Sometimes the "latest" version can be buggy with certain setups.
  

 

 

You are creating a share for each VM and COW is set to Auto

• Since your cache device is formatted with BTRFS, you need to disable COW for stable operations for virtual disks.
  
• You can only do this when creating a new share (you cannot modify this setting on an existing share).
  
• Create a new share for your VM and change the Enable Copy-on-write setting from Auto to No.
  
• Recreate your VM(s)
  

 

Please try these adjustments and report back.

 

EDIT:

 

I should also mention, you don't need to create a share on a per-VM basis.  Just create a share called vdisks and place each VM inside.  VM Manager will automatically create sub-folders for each vdisk per VM.

[stratus]

Thanks Jon!

 

I tried what was suggested - stable branch and COW to No. VNC off.

 

I had the same issues, unfortunately. Q35 does let me get further though. With i440fx I get the "system_service_exception" immediately after rebooting following the install. With Q35 and I can boot into Windows again. Then I go to Device Manager and it sees 2 display adapters (neither labeled as the GTX 960). When I try to update drivers, however, I get "system_thread_exception_not_handled (nvlddmkm.sys)". On reboot it's back to "system_service_exception" and it doesn't see the drive.

 

Same result with VNC disabled as an adapter (I confirmed the output was working from the card by restarting the system, and the output is fine right up until I start the VM).

 

 

[jonp]

Thanks Jon!

 

I tried what was suggested - stable branch and COW to No. VNC off.

 

I had the same issues, unfortunately. Q35 does let me get further though. With i440fx I get the "system_service_exception" immediately after rebooting following the install. With Q35 and I can boot into Windows again. Then I go to Device Manager and it sees 2 display adapters (neither labeled as the GTX 960). When I try to update drivers, however, I get "system_thread_exception_not_handled (nvlddmkm.sys)". On reboot it's back to "system_service_exception" and it doesn't see the drive.

 

Same result with VNC disabled as an adapter (I confirmed the output was working from the card by restarting the system, and the output is fine right up until I start the VM).

 

Ok, I want you to try creating a new VM from scratch.  Leave the machine type set to i440 and set the BIOS to OVMF.  Do NOT install the VM over VNC (select the NVIDIA GPU).  You'll also need to select a USB mouse/keyboard to assign so you can do the install.  What happens when you do this?

[stratus]

Thanks Jon. No luck, unfortunately. I installed a new VM with OVMF, no VNC (960 only), no sound card, and USB keyboard connected. I don't see anything on the screen when I start it.

 

I'll paste the log with the attempts (the initial boot, reboot, attempt with Q35 - which yields the error, and attempt with QEMU64).

 

2015-08-19 10:57:33.080+0000: starting up libvirt version: 1.2.15, qemu version: 2.3.0
Domain id=3 is tainted: high-privileges
Domain id=3 is tainted: host-cpu
char device redirected to /dev/pts/0 (label charserial0)
qemu: terminating on signal 15 from pid 2506
2015-08-19 10:58:28.672+0000: shutting down
2015-08-19 10:58:45.380+0000: starting up libvirt version: 1.2.15, qemu version: 2.3.0
LC_ALL=C PATH=/bin:/sbin:/usr/bin:/usr/sbin HOME=/ QEMU_AUDIO_DRV=none /usr/bin/qemu-system-x86_64 -name WinVM -S -machine pc-q35-2.3,accel=kvm,usb=off,mem-merge=off -cpu host -drive file=/usr/share/qemu/ovmf-x64/OVMF-pure-efi.fd,if=pflash,format=raw,unit=0 -m 4096 -realtime mlock=on -smp 1,sockets=1,cores=1,threads=1 -uuid 79e6d7a6-6927-7857-a636-3b051ca098df -nographic -no-user-config -nodefaults -chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/WinVM.monitor,server,nowait -mon chardev=charmonitor,id=monitor,mode=control -rtc base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard -no-hpet -no-shutdown -boot strict=on -device i82801b11-bridge,id=pci.1,bus=pcie.0,addr=0x1e -device pci-bridge,chassis_nr=2,id=pci.2,bus=pci.1,addr=0x1 -device ich9-usb-ehci1,id=usb,bus=pci.2,addr=0x2.0x7 -device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.2,multifunction=on,addr=0x2 -device virtio-serial-pci,id=virtio-serial0,bus=pci.2,addr=0x3 -drive file=/mnt/user/Win102,addr=0x5 -device usb-host,hostbus=1,hostaddr=4,id=hostdev1 -device virtio-balloon-pci,id=balloon0,bus=pci.2,addr=0x6 -msg timestamp=on
Domain id=4 is tainted: high-privileges
Domain id=4 is tainted: host-cpu
char device redirected to /dev/pts/0 (label charserial0)
2015-08-19T10:58:45.902045Z qemu-system-x86_64: -device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1: Can't create IDE unit 1, bus supports only 1 units
2015-08-19T10:58:45.902065Z qemu-system-x86_64: -device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1: Device initialization failed.
2015-08-19T10:58:45.902072Z qemu-system-x86_64: -device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1: Device 'ide-cd' could not be initialized
2015-08-19 10:58:46.136+0000: shutting down
2015-08-19 11:00:58.045+0000: starting up libvirt version: 1.2.15, qemu version: 2.3.0
Domain id=5 is tainted: high-privileges
Domain id=5 is tainted: host-cpu
char device redirected to /dev/pts/0 (label charserial0)
qemu: terminating on signal 15 from pid 2506
2015-08-19 11:01:32.454+0000: shutting down
2015-08-19 11:02:46.152+0000: starting up libvirt version: 1.2.15, qemu version: 2.3.0
Domain id=6 is tainted: high-privileges
char device redirected to /dev/pts/0 (label charserial0)
qemu: terminating on signal 15 from pid 2506
2015-08-19 11:03:15.482+0000: shutting down

[jonp]

Ok, a few more things to try.

 

1 - have you tried reseating the GPU in another PCI slot?

2 - check your bios settings and tell me if you have anything you can change related to integrated graphics (if you can "disable" it or something)

[stratus]

I tried with another slot. I confirmed it's functional (there's output from boot until VM start) but it doesn't help with the issue. I tried installing 2 new VMs (both with VNC disabled on install) and there's no display.

 

I checked the BIOS as well and the closest option is choosing the default graphics, which was set to PCIE and is supposed to disable onboard graphics when it's selected. Onboard graphics have never been an option in VM settings either.

[jonp]

I checked the BIOS as well and the closest option is choosing the default graphics, which was set to PCIE

 

Bingo...  That's your problem.  Change this option to use the on-board graphics.  This will then allow unRAID to output it's console to on-board and use the PCIE GPU for the VM.

 

I probably need to add this to the wiki...

[stratus]

Wow, that was it! My earlier VMs still had issues, but they did output immediately via HDMI after making that change. I installed a new Windows VM and everything seems fine. I've rebooted it multiple times, installed the official Nvidia drivers, and everything is good.

 

It does not work with VNC. After the install I tried pushing my luck and turning VNC back on, but it actually brought down Unraid completely. I'll just work around that.

 

Thanks again for your help!

[jonp]

Wow, that was it! My earlier VMs still had issues, but they did output immediately via HDMI after making that change. I installed a new Windows VM and everything seems fine. I've rebooted it multiple times, installed the official Nvidia drivers, and everything is good.

 

It does not work with VNC. After the install I tried pushing my luck and turning VNC back on, but it actually brought down Unraid completely. I'll just work around that.

 

Thanks again for your help!

You are most welcome!  Glad we got it sorted...