VM boots to UEFI Interactive Shell instead of Windows


Recommended Posts

ENV:

  • Supermicro + Intel CPU
  • Windows 10 VM installed on a passthrough NVMe drive. 

 

PROBLEM:  VM boots to UEFI Interactive Shell instead of Windows

 

VM boots to UEFI Interactive Shell instead of the OS.  Exiting out of UEFI shell, I can manually boot into Windows via Boot Manager  selecting UEFI Samsung SSD drive.  In the BIOS I have tried deleting/changing boot device order so only the NVMe drive is in the boot sequence; however, these changes do not persist.  Please see attached snapshots.  

 

image.png.0ad146bc979a93a086d8ae91e3624bc6.png

image.png

image.png

Edited by Deep Insights
additional snapshot added
Link to comment
  • 3 months later...

I'm also having this problem.  In my case I'm passing through a PCIE usb controller with a separate unraid USB drive.  I can boot from this if I select manually.  I can change the boot order and save. "Continue" will boot correctly, but upon restarting the VM it boots back to EFI shell.

 

I can see that nvram file is being updated at /etc/libvirt/qemu/nvram so it seems to be saving the settings, but why not applying them?

 

Anyone else found a solution to this?

Link to comment

Not sure if this will solve your issue, but I had a similar problem after I moved from vdisk to sata disk passthrough: OVMF booted to UEFI shell instead of booting my mac os vm.

What I did was adding a boot entry from the UEFI shell with the bcfg command:

 

Once into the UEFI shell:

map fs*

 

This will list the hd, search for the uuid of the hd you want to boot from (let's say it's fs3)

fs3:

list files on this partition, my bootx64.efi was inside /EFI/BOOT, so in my case:

cd /EFI/BOOT

Inside this folder there was BOOTx64.efi

bcfg boot add 1 bootx64.efi "BigSur"

1 is priority, "BigSur" is the boot entry name.

 

Before applying the bcfg command check that you can boot the os, in my case, once at cd /EFI/BOOT I simply gave the command:

bootx64.efi

and magically my mac os booted.

If it boots, try the bcfg command.

 

In my case the change was persistent.

Edited by ghost82
  • Like 1
  • Upvote 1
Link to comment
51 minutes ago, jortan said:

Appreciate the feedback - but I had already tried configuring EFI boot using bcfg.  Still not persistent.  Seems like this is related to passthrough specifically - perhaps because qemu can't see the device when the machine is starting?

Can you share the xml, especially the disik section?maybe there's something useful in it to figure it out.

Link to comment
4 minutes ago, ghost82 said:

I don't think so, because if you boot into the ovmf bios you see that entry, boot is after the "forced bios menu", so qemu sees your drive.

 

29 minutes ago, ghost82 said:

Can you share the xml, especially the disik section?maybe there's something useful in it to figure it out.

 

What I mean is we're talking about devices passed through to the guest.  In my case it's an entire USB controller, so the boot device isn't listed at all in the XML file.  In that sense QEMU isn't aware of the boot device when it's starting the VM (even though the guest definitely sees it)  Maybe the OVMF settings are being ignored/reset for that reason?

 

I will try to post my XML soon but I've messed up my libvirt service by poking at things and can't start it currently - will try restarting unraid later when it's not being relied on.

Link to comment
1 minute ago, jortan said:

 

 

What I mean is we're talking about devices passed through to the guest.  In my case it's an entire USB controller, so the boot device isn't listed at all in the XML file.  In that sense QEMU isn't aware of the boot device when it's starting the VM (even though the guest definitely sees it)  Maybe the OVMF settings are being ignored/reset for that reason?

 

I will try to post my XML soon but I've messed up my libvirt service by poking at things and can't start it currently - will try restarting unraid later when it's not being relied on.

ok, understood, so no disk section in the xml, it's the same as my setup (only sata controller passthrough), not sure what's happening there..maybe you can try to delete ovmf_vars (and maybe ovmf_code) file and replace with fresh ones, sometimes the ovmf_vars caused me some issues that I solved by replacing with a fresh one, then if it still doesn't work, retry with the bcfg command.

Link to comment
15 hours ago, ghost82 said:

ok, understood, so no disk section in the xml, it's the same as my setup (only sata controller passthrough), not sure what's happening there..maybe you can try to delete ovmf_vars (and maybe ovmf_code) file and replace with fresh ones, sometimes the ovmf_vars caused me some issues that I solved by replacing with a fresh one, then if it still doesn't work, retry with the bcfg command.

 

I've recreated the VM multiple times, so multiple fresh vars files have shown the issue.  Perhaps an alternative OVMF version might help - though messing around with the OVMS paths in my VM XML is what stopped me from starting the unraid VM service yesterday.

Edited by jortan
Link to comment
2 hours ago, jortan said:

 

I've recreated the VM multiple times, so multiple fresh vars files have shown the issue.  Perhaps an alternative OVMF version might help - though messing around with the OVMS paths in my VM XML is what stopped me from starting the unraid VM service yesterday.

Not sure what version you are running, but If you want to try the latest stable release of edk2 ovmf here you can find the files:

 

I'm not too much confident it will work...

Edited by ghost82
Link to comment
8 hours ago, jortan said:

I was able to boot the VM with the alternative OVMF code/vars files, but the problem remains.

 

Very frustrating!

I suspect of a bug in edk2 ovmf then..if I have time I will try to replicate and maybe open a bug in their github.

Edited by ghost82
  • Like 1
Link to comment
11 hours ago, ghost82 said:

Did you try this (last block of code)?

I didn't know you can set the boot order in the block of code that you pass through

 

Nice, that will probably fix @Deep Insights issue (as he's passing through a block device in XML) but not mine (as I'm passing through a USB controller)

 

I'll do some playing around and see if I can get this to work though.

  • Like 1
Link to comment
5 hours ago, jortan said:

but not mine (as I'm passing through a USB controller)

mmm. not sure, I'm referring to the pci passthrough, not the block disk (step 2).

 

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <boot order='1'/>
      <alias name='hostdev3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>

 

He also was passing through the controller, not a usb one, but should not change.

Edited by ghost82
  • Thanks 1
Link to comment

Hello, im not trying to hijack the thread by i am experiencing a similar problem trying to passthrough a GTX 960 to a windows 10 VM.

I have dumped my video cards bios per spaceinvader videos, i have tried seabios, ovmf, i have edited the XML every which way i can think of and am using the latest windows 10 ISO from microsoft. when using VNC i am able to create a windows 10 VM following spaceinvader instructions and boot into windows successfully, when passingthrough the GPU i get thrown into the UEFI interactive shell and typing exit and trying to boot from the boot manager does not work. I have tried multiple different bios files IOMMU is enabled, HVM is enabled, the card is in its own IOMMU group with the audio part of the card. at this point i have created well over 70+ VMs trying to get this to work  can someone please offer some guidance as to what i am doing wrong. attached is the code for my current VM

 

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>Windows 10 Gaming</name>
  <uuid>af68722e-fe54-4855-f02b-9ffaea16b684</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows 10" icon="windows.png" os="windows10"/>
  </metadata>
  <memory unit='KiB'>8388608</memory>
  <currentMemory unit='KiB'>8388608</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>8</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='1'/>
    <vcpupin vcpu='1' cpuset='2'/>
    <vcpupin vcpu='2' cpuset='3'/>
    <vcpupin vcpu='3' cpuset='4'/>
    <vcpupin vcpu='4' cpuset='15'/>
    <vcpupin vcpu='5' cpuset='16'/>
    <vcpupin vcpu='6' cpuset='17'/>
    <vcpupin vcpu='7' cpuset='18'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-q35-5.1'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/af68722e-fe54-4855-f02b-9ffaea16b684_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='4' threads='2'/>
    <cache mode='passthrough'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Windows 10 Gaming/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/Operating Systems/Win10_21H1_English_x64.iso'/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.190-1.iso'/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0xa'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0xb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xc'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xd'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:5a:36:5d'/>
      <source bridge='virbr0'/>
      <model type='virtio-net'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x87' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/domains/vbios/gtx960org.dump'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x87' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x1'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x046d'/>
        <product id='0xc52e'/>
      </source>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>

 

20210531_153309.jpg

Edited by Sinister
Adding more images
Link to comment
On 5/26/2021 at 4:04 PM, ghost82 said:

mmm. not sure, I'm referring to the pci passthrough, not the block disk (step 2).

 


    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
      </source>
      <boot order='1'/>
      <alias name='hostdev3'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x09' function='0x0'/>
    </hostdev>

 

He also was passing through the controller, not a usb one, but should not change.

 

That did it - thank you!  For anyone else with this issue, these lines were added:

 

        <boot order='1'/>
        <alias name='usbboot'/>

 

You'll want to check any other instances of "boot order" setting in the XML and make everything else something other than "1"

 

  • Like 1
Link to comment
11 hours ago, Sinister said:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x87' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/domains/vbios/gtx960org.dump'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x87' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x1'/>
    </hostdev>

 

Please open a new discussion for this, as your issue has nothing to do with boot order.

First, your topology for gpu+audio is wrong, you have video on one bus and audio on another bus, hdmi audio and gpu video should be in the same bus, same slot, but different function (multifunction device).

It should be like this:

    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x87' slot='0x00' function='0x0'/>
      </source>
      <rom file='/mnt/user/domains/vbios/gtx960org.dump'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0' multifunction='on'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x87' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x1'/>
    </hostdev>

Second, dump your own vbios, do not download one from techpowerup.

After fixing this, if it still boots to the uefi shell, press esc multiple times when you boot the vm, see if in the boot options there is your hard drive and if it's not in first position, change its order to the top one.

Link to comment
  • 1 year later...
On 6/1/2021 at 12:50 PM, jortan said:

 

That did it - thank you!  For anyone else with this issue, these lines were added:

 

        <boot order='1'/>
        <alias name='usbboot'/>

 

You'll want to check any other instances of "boot order" setting in the XML and make everything else something other than "1"

 

 

Hey @jortan and @Deep Insights I think I may have the same issue as you guys:

 

 

I installed Windows 11 VM on a passthrough NVME drive and it keeps booting into the UEFI Shell. It seems like @jortan fixed this but I'm not 100% sure I understand the solution. I am pretty new to all of this. Am I having the same issue as you guys and if so what can I do to fix it? Any help would be appreciated!

 

Link to comment
12 hours ago, venicenerd said:

 

I installed Windows 11 VM on a passthrough NVME drive and it keeps booting into the UEFI Shell. It seems like @jortan fixed this but I'm not 100% sure I understand the solution. I am pretty new to all of this. Am I having the same issue as you guys and if so what can I do to fix it? Any help would be appreciated!

 

 

Here's another post with information more specifically about setting boot order for passthrough nvme:

 

  • Like 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.