Can't boot Windows 11 more than once... strange problem unRAID 611.12 & 6.11.3


craigr

Recommended Posts

Originally I had a Windows 10 VM setup to run on my NVMe bare-metal using spaces_win_clover.img.  I then upgraded to Windows 11 and eliminated the need for spaces_win_clover.img (I think it was not compatible with Windows 11 or maybe I just didn't want to use it anymore because it was no longer needed in unRAID).

 

However, after I did that I could no longer get my Windows 11 VM to boot properly or consistently.  This is the thread where I sort of sorted it out or at least got my VM working again:

With the release of unRAID 6.11.2 and 6.11.3 I have encountered the problem all over again.  With this line in my xml file the VM will boot properly every time, the key line being <boot dev='hd'/>

 

  <os>
    <type arch='x86_64' machine='pc-i440fx-7.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/1a8fdacb-aad4-4bbb-71ea-732b0ea1051a_VARS-pure-efi-tpm.fd</nvram>
    <boot dev='hd'/>

 

I have completely removed the VM (not the VM and disk, just the VM) and started over.  I have assigned the NVMe to boot order 1 which adds the line for boot order in the xml file 

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
      </source>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0c' function='0x0'/>
    </hostdev>

 

When I do that, there is no line <boot dev='hd'/> generated by the template.

 

Here is where it gets really weird.  Only on the first boot after doing this will OVMF finish loading and boot Windows 11.  If I shut down or restart Windows, OVMF will freeze and I cannot boot again until I go back, edit the xml to remove the boot order line, and restore the <boot dev='hd'/> ?!?

 

When I boot with <boot dev='hd'/> this is what I get:

 

IMG_5055.jpg.1a469279ccd01335a9bdef4838e3fe63.jpg

 

I can also press esc and enter the OVMF BIOS.  I have tried removing the TWO windows boot managers that are in there and put the NVMe as the first boot device and saved (I've done this like 20 times).  However, every time I go back into the OVFM BIOS all the boot options are back as if I never deleted or reordered them 😖.

 

On my first boot with an xml containing <boot order='1'/> I get the exact same above screen and can enter the OVFM Bios with esc.  But as stated, once I shutdown Windows and reboot I will only get the first "Windows Boot Manager," OVFM will not respond to esc, and it just freezes there.  Finished.  Done.  I have to go back to the xml, remove boot order and restore boot dev=hd.

 

Why do I have two windows boot managers, why can't I delete them, why can I boot fine directly from my NVMe when I am in the OVFM BIOS and launch there, why can't I use boot order in my xml and boot more than once?  Here is my entire current xml that boots:

 

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm' id='2'>
  <name>Windows 11</name>
  <uuid>1a8fdacb-aad4-4bbb-71ea-732b0ea1051a</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 11" icon="windows11.png" os="windowstpm"/>
  </metadata>
  <memory unit='KiB'>38273024</memory>
  <currentMemory unit='KiB'>38273024</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='4'/>
    <vcpupin vcpu='1' cpuset='12'/>
    <vcpupin vcpu='2' cpuset='5'/>
    <vcpupin vcpu='3' cpuset='13'/>
    <vcpupin vcpu='4' cpuset='6'/>
    <vcpupin vcpu='5' cpuset='14'/>
    <vcpupin vcpu='6' cpuset='7'/>
    <vcpupin vcpu='7' cpuset='15'/>
  </cputune>
  <resource>
    <partition>/machine</partition>
  </resource>
  <os>
    <type arch='x86_64' machine='pc-i440fx-7.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/1a8fdacb-aad4-4bbb-71ea-732b0ea1051a_VARS-pure-efi-tpm.fd</nvram>
    <boot dev='hd'/>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv mode='custom'>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='4' threads='2'/>
    <cache mode='passthrough'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/pool/ISOs/virtio-win-0.1.225-2.iso' index='1'/>
      <backingStore/>
      <target dev='hdb' bus='ide'/>
      <readonly/>
      <alias name='ide0-0-1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='usb' index='0' model='qemu-xhci' ports='15'>
      <alias name='usb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'>
      <alias name='pci.0'/>
    </controller>
    <controller type='ide' index='0'>
      <alias name='ide'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <alias name='virtio-serial0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:29:50:c9'/>
      <source bridge='br0'/>
      <target dev='vnet1'/>
      <model type='virtio-net'/>
      <alias name='net0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </interface>
    <serial type='pty'>
      <source path='/dev/pts/0'/>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
      <alias name='serial0'/>
    </serial>
    <console type='pty' tty='/dev/pts/0'>
      <source path='/dev/pts/0'/>
      <target type='serial' port='0'/>
      <alias name='serial0'/>
    </console>
    <channel type='unix'>
      <source mode='bind' path='/var/lib/libvirt/qemu/channel/target/domain-2-Windows 11/org.qemu.guest_agent.0'/>
      <target type='virtio' name='org.qemu.guest_agent.0' state='connected'/>
      <alias name='channel0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'>
      <alias name='input0'/>
    </input>
    <input type='keyboard' bus='ps2'>
      <alias name='input1'/>
    </input>
    <tpm model='tpm-tis'>
      <backend type='emulator' version='2.0' persistent_state='yes'/>
      <alias name='tpm0'/>
    </tpm>
    <audio id='1' type='none'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev0'/>
      <rom file='/mnt/pool/vdisks/vbios/My_Inno3D.GTX1650.4096.(version_90.17.3D.00.95).rom'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <alias name='hostdev1'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x1f' function='0x3'/>
      </source>
      <alias name='hostdev2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x06' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x14' function='0x0'/>
      </source>
      <alias name='hostdev3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x00' slot='0x1f' function='0x5'/>
      </source>
      <alias name='hostdev4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
      </source>
      <alias name='hostdev5'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x0a' function='0x0'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
  <seclabel type='dynamic' model='dac' relabel='yes'>
    <label>+0:+100</label>
    <imagelabel>+0:+100</imagelabel>
  </seclabel>
</domain>

 

PLEASE SOMEBODY HELP ME.  I HAVE SPENT HOURS!!!

 

Edited by craigr
Link to comment
  • craigr changed the title to Can't boot Windows 11 more than once... strange problem unRAID 611.12 & 6.11.3

Can you please try the attached ovmf?

Just decompress the zip, backup your original files:

/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd

/etc/libvirt/qemu/nvram/1a8fdacb-aad4-4bbb-71ea-732b0ea1051a_VARS-pure-efi-tpm.fd

 

move these files in a secure place

rename the extracted files to OVMF_CODE-pure-efi-tpm.fd and 1a8fdacb-aad4-4bbb-71ea-732b0ea1051a_VARS-pure-efi-tpm.fd

move these renamed files to:
/usr/share/qemu/ovmf-x64/

/etc/libvirt/qemu/nvram/

 

Once done, modify the vm template, deleting <boot dev='hd'/> and adding <boot order='1'/> to the nvme block (let's try the traditional way first).

If it doesn't boot, shutdown the vm, compress 1a8fdacb-aad4-4bbb-71ea-732b0ea1051a_VARS-pure-efi-tpm.fd and share it here

Before shutting down the vm check if you have the double entry 'Windows boot manager'

 

OVMF-secboot.zip

Link to comment
On 11/10/2022 at 3:05 AM, ghost82 said:

Can you please try the attached ovmf?

Just decompress the zip, backup your original files:

/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd

/etc/libvirt/qemu/nvram/1a8fdacb-aad4-4bbb-71ea-732b0ea1051a_VARS-pure-efi-tpm.fd

 

move these files in a secure place

rename the extracted files to OVMF_CODE-pure-efi-tpm.fd and 1a8fdacb-aad4-4bbb-71ea-732b0ea1051a_VARS-pure-efi-tpm.fd

move these renamed files to:
/usr/share/qemu/ovmf-x64/

/etc/libvirt/qemu/nvram/

 

Once done, modify the vm template, deleting <boot dev='hd'/> and adding <boot order='1'/> to the nvme block (let's try the traditional way first).

If it doesn't boot, shutdown the vm, compress 1a8fdacb-aad4-4bbb-71ea-732b0ea1051a_VARS-pure-efi-tpm.fd and share it here

Before shutting down the vm check if you have the double entry 'Windows boot manager'

 

OVMF-secboot.zip 1.23 MB · 1 download

This has worked for me, the VM is rebooting without issues now. 

 

*edit: Getting inconsistent results, so I don't think this is actually working for me.

Edited by ChadTaljaardt
Link to comment
On 11/10/2022 at 8:05 AM, ghost82 said:

Can you please try the attached ovmf?

Just decompress the zip, backup your original files:

/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd

/etc/libvirt/qemu/nvram/1a8fdacb-aad4-4bbb-71ea-732b0ea1051a_VARS-pure-efi-tpm.fd

 

move these files in a secure place

rename the extracted files to OVMF_CODE-pure-efi-tpm.fd and 1a8fdacb-aad4-4bbb-71ea-732b0ea1051a_VARS-pure-efi-tpm.fd

move these renamed files to:
/usr/share/qemu/ovmf-x64/

/etc/libvirt/qemu/nvram/

 

Once done, modify the vm template, deleting <boot dev='hd'/> and adding <boot order='1'/> to the nvme block (let's try the traditional way first).

If it doesn't boot, shutdown the vm, compress 1a8fdacb-aad4-4bbb-71ea-732b0ea1051a_VARS-pure-efi-tpm.fd and share it here

Before shutting down the vm check if you have the double entry 'Windows boot manager'

 

OVMF-secboot.zip 1.23 MB · 1 download

Hi @ghost82 Where do you get the images from?

Link to comment
1 hour ago, SimonF said:

Where do you get the images from?

I compile them from official edk github source, I usually compile stable branches, the attached is the latest stable available (august 2022 if I remember well).

1 hour ago, ChadTaljaardt said:

Getting inconsistent results

Please provide additional info and provide the vars file after booting or trying to boot the vm.

Link to comment

So I am no longer getting the issue that was in the thread, for me the change you suggested fixed that. But everytime windows now shutdown or reboots, I need to manually go into Unraid and "force stop" the vm, then start it again to make it work. Rebooting or shutting down through windows alone doesn't work, It just returns a black screen when I connect through VNC. 

 

Here is that file you wanted: 

96bea338-251d-99df-ea99-d0ccb00b4fa2_VARS-pure-efi-tpm.fd.zip

Link to comment
On 11/14/2022 at 10:12 PM, ChadTaljaardt said:

So I am no longer getting the issue that was in the thread, for me the change you suggested fixed that. But everytime windows now shutdown or reboots, I need to manually go into Unraid and "force stop" the vm, then start it again to make it work. Rebooting or shutting down through windows alone doesn't work, It just returns a black screen when I connect through VNC. 

 

Here is that file you wanted: 

96bea338-251d-99df-ea99-d0ccb00b4fa2_VARS-pure-efi-tpm.fd.zip

 

i have exactly the same problem (windows 11), it's a bit annoying.
I already installed a completely new windows 11 vm - same issue. can't be restarted or get out of sleep (must be closed then with termination (force shut down)).

Link to comment
  • 2 weeks later...
  • 1 month later...

Hey guys, 

Just wanted to follow up on this thread. 
I was running into the same issues, and ended up finding this thread. 

I went into the VM > Edit and I removed the "OS Install ISO" Location and the "VirtIO Drivers ISO" Location fields. Updated the VM at the bottom of the page, and it booted right up. 
Let me know if you have the same success. 
 

  • Like 3
Link to comment
On 1/18/2023 at 12:52 AM, Sibobby1 said:

Hey guys, 

Just wanted to follow up on this thread. 
I was running into the same issues, and ended up finding this thread. 

I went into the VM > Edit and I removed the "OS Install ISO" Location and the "VirtIO Drivers ISO" Location fields. Updated the VM at the bottom of the page, and it booted right up. 
Let me know if you have the same success. 
 

Thank you. This worked for me.

  • Like 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.