Working VM suddenly booting to UEFI Shell

[chonnymon]

Hi, I have had a windows server 2016 VM running for last 18+ Months no problems.

This morning I have come to use it and it has booted to UEFI interactive shell, I have gone into VM bios and tried to manually select the disk but nothing happens. I haven't changed any settings.

I have searched the forums but have not been able to find a solution, any help would be appreciated

 

Below is log and xml

 

-machine pc-i440fx-4.2,accel=kvm,usb=off,dump-guest-core=off,mem-merge=off,pflash0=libvirt-pflash0-format,pflash1=libvirt-pflash1-format \
-cpu host,migratable=on,topoext=on,host-cache-info=on,l3-cache=off \
-m 16384 \
-overcommit mem-lock=off \
-smp 4,sockets=1,dies=1,cores=2,threads=2 \
-uuid 3487ed8f-b4e7-c936-3a1e-8c77cacdb409 \
-no-user-config \
-nodefaults \
-chardev socket,id=charmonitor,fd=32,server,nowait \
-mon chardev=charmonitor,id=monitor,mode=control \
-rtc base=localtime,driftfix=slew \
-global kvm-pit.lost_tick_policy=delay \
-no-hpet \
-no-shutdown \
-boot strict=on \
-device ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x7.0x7 \
-device ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x7 \
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x7.0x1 \
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x7.0x2 \
-device ahci,id=sata0,bus=pci.0,addr=0x5 \
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x4 \
-blockdev '{"driver":"file","filename":"/mnt/user/domains/Sharepoint/vdisk1.img","node-name":"libvirt-1-storage","cache":{"direct":false,"no-flush":false},"auto-read-only":true,"discard":"unmap"}' \
-blockdev '{"node-name":"libvirt-1-format","read-only":false,"cache":{"direct":false,"no-flush":false},"driver":"raw","file":"libvirt-1-storage"}' \
-device ide-hd,bus=sata0.2,drive=libvirt-1-format,id=sata0-0-2,bootindex=1,write-cache=on \
-netdev tap,fd=37,id=hostnet0,vhost=on,vhostfd=38 \
-device virtio-net-pci,netdev=hostnet0,id=net0,mac=52:54:00:76:bc:b7,bus=pci.0,addr=0x3 \
-chardev pty,id=charserial0 \
-device isa-serial,chardev=charserial0,id=serial0 \
-chardev socket,id=charchannel0,fd=39,server,nowait \
-device virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=org.qemu.guest_agent.0 \
-device usb-tablet,id=input0,bus=usb.0,port=1 \
-vnc 0.0.0.0:0,websocket=5700 \
-k en-us \
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vram64_size_mb=0,vgamem_mb=16,max_outputs=1,bus=pci.0,addr=0x2 \
-device virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x8 \
-sandbox on,obsolete=deny,elevateprivileges=deny,spawn=deny,resourcecontrol=deny \
-msg timestamp=on
2022-12-15 10:21:58.030+0000: Domain id=21 is tainted: high-privileges
2022-12-15 10:21:58.030+0000: Domain id=21 is tainted: host-cpu
char device redirected to /dev/pts/0 (label charserial0)

<?xml version='1.0' encoding='UTF-8'?>
<domain type='kvm'>
  <name>Sharepoint</name>
  <uuid>3487ed8f-b4e7-c936-3a1e-8c77cacdb409</uuid>
  <metadata>
    <vmtemplate xmlns="unraid" name="Windows Server 2016" icon="windows.png" os="windows2016"/>
  </metadata>
  <memory unit='KiB'>16777216</memory>
  <currentMemory unit='KiB'>16777216</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>4</vcpu>
  <cputune>
    <vcpupin vcpu='0' cpuset='6'/>
    <vcpupin vcpu='1' cpuset='18'/>
    <vcpupin vcpu='2' cpuset='7'/>
    <vcpupin vcpu='3' cpuset='19'/>
  </cputune>
  <os>
    <type arch='x86_64' machine='pc-i440fx-4.2'>hvm</type>
    <loader readonly='yes' type='pflash'>/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd</loader>
    <nvram>/etc/libvirt/qemu/nvram/3487ed8f-b4e7-c936-3a1e-8c77cacdb409_VARS-pure-efi.fd</nvram>
  </os>
  <features>
    <acpi/>
    <apic/>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='2' threads='2'/>
    <cache mode='passthrough'/>
    <feature policy='require' name='topoext'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='rtc' tickpolicy='catchup'/>
    <timer name='pit' tickpolicy='delay'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Sharepoint/vdisk1.img'/>
      <target dev='hdc' bus='sata'/>
      <boot order='1'/>
      <address type='drive' controller='0' bus='0' target='0' unit='2'/>
    </disk>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x05' function='0x0'/>
    </controller>
    <controller type='pci' index='0' model='pci-root'/>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x04' function='0x0'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:76:bc:b7'/>
      <source bridge='br0'/>
      <model type='virtio'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='1'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <graphics type='vnc' port='-1' autoport='yes' websocket='-1' listen='0.0.0.0' keymap='en-us'>
      <listen type='address' address='0.0.0.0'/>
    </graphics>
    <video>
      <model type='qxl' ram='65536' vram='65536' vgamem='16384' heads='1' primary='yes'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x02' function='0x0'/>
    </video>
    <memballoon model='virtio'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x08' function='0x0'/>
    </memballoon>
  </devices>
</domain>

 

[ghost82]

Paste the output of these two commands:

fdisk -l /mnt/user/domains/Sharepoint/vdisk1.img

 

qemu-img info /mnt/user/domains/Sharepoint/vdisk1.img

 

[chonnymon]

2 minutes ago, ghost82 said:

Paste the output of these two commands:

fdisk -l /mnt/user/domains/Sharepoint/vdisk1.img

 

qemu-img info /mnt/user/domains/Sharepoint/vdisk1.img

 

Thanks for your quick response

 

root@Meshify:~# fdisk -l /mnt/user/domains/Sharepoint/vdisk1.img
Disk /mnt/user/domains/Sharepoint/vdisk1.img: 200 GiB, 214748364800 bytes, 419430400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

 

root@Meshify:~# qemu-img info /mnt/user/domains/Sharepoint/vdisk1.img
image: /mnt/user/domains/Sharepoint/vdisk1.img
file format: raw
virtual size: 200 GiB (214748364800 bytes)
disk size: 90.2 GiB

 

[ghost82]

13 minutes ago, chonnymon said:

root@Meshify:~# fdisk -l /mnt/user/domains/Sharepoint/vdisk1.img
Disk /mnt/user/domains/Sharepoint/vdisk1.img: 200 GiB, 214748364800 bytes, 419430400 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes

 

If you are sure the output is only this your vdisk got corrupted as it doesn't list any partition on it.

The output should list something like:

[root@tower kali]# fdisk -l /media/6TB/kali/Kali.img
Disk /media/6TB/kali/Kali.img: 150 GiB, 161061273600 bytes, 314572800 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: DF66D708-DD93-4359-A4C5-C03FBBE746CE

Dispositivo                 Start      Fine   Settori   Size Tipo
/media/6TB/kali/Kali.img1 1026048 314570751 313544704 149,5G Linux filesystem
/media/6TB/kali/Kali.img3    2048   1026047   1024000   500M EFI System

Partition table entries are not in disk order.

 

You can try to reapir the disk with fdisk or any other command line tool to repair filesystems, make a backup of the disk before attempting to repair it.

Edited December 15, 2022 by ghost82

[chonnymon]

12 minutes ago, ghost82 said:

 

If you are sure the output is only this your vdisk got corrupted as it doesn't list any partition on it.

The output should list something like:

[root@tower kali]# fdisk -l /media/6TB/kali/Kali.img
Disk /media/6TB/kali/Kali.img: 150 GiB, 161061273600 bytes, 314572800 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: gpt
Disk identifier: DF66D708-DD93-4359-A4C5-C03FBBE746CE

Dispositivo                 Start      Fine   Settori   Size Tipo
/media/6TB/kali/Kali.img1 1026048 314570751 313544704 149,5G Linux filesystem
/media/6TB/kali/Kali.img3    2048   1026047   1024000   500M EFI System

Partition table entries are not in disk order.

 

You can try to reapir the disk with fsck or any other command line tool to repair filesystems, make a backup of the disk before attempting to repair it.

Thank you again for your reply, yes that's all that outputted

 

I am making a copy of vdisk1.img now

Sorry if this is obvious question, to attempt repair with fsck would the command just go in unraid terminal
 

Quote

fsck /mnt/user/domains/Sharepoint/vdisk1.img

 

[ghost82]

12 minutes ago, chonnymon said:

Sorry if this is obvious question, to attempt repair with fsck would the command just go in unraid terminal

I did it in the past but I can't remember, sorry; in real I'm not sure even if fdisk or fsck can do it..

If partitions are still on the vdisk you may be able to repair/re-create the partition table: maybe also gpart can do it, see here (gpart/parted should be available in unrid, and note that the link is for real hds, /dev/hda, etc., but you can apply that commands to vdisk images too):

https://ubuntuforums.org/showthread.php?t=370121

 

Just experiment with tools and see if you go somewhere, that's why it's important to have a backup of the original corrupted vdisk, so you can copy it again and start over.

Edited December 15, 2022 by ghost82

[chonnymon]

Thank you for your help, it has taken me all day but just got VM booted again.

You pointed me in the right direction, the solution that worked for me was getting a GParted LiveCD booting that with the vdisk.img and the using the testdisk application to search for and recreate partitions. I also did a chkdsk from the windows boot cd which found 1 error but not sure if that helped or not.

 

do you have any advice for how I can prevent this happening again? or is there anything that could periodically back up the VM?

 

[ghost82]

Nice that you fixed it!

testdisk is very powerful, I remember I used it on a physical damaged hd to recover data.

 

14 hours ago, chonnymon said:

do you have any advice for how I can prevent this happening again? or is there anything that could periodically back up the VM?

I think this happened because of one of these 2 reasons:

1. improper shutdown of vm (like a forced shutdown) or any other thing caused inside the guest, filesystem got corrupted;

2. damaged physical hd where the vdisk is saved

 

I don't know of any plugin to backup vms (I don't use any), but you could simply shutdown the vm and copy/backup the vdisk image, better on another physical hd.

All the data are in the vdisk, this can be used when creating a new vm or you can simply mount partitions contained in the vdisk to access data.

You could automate the backup with a script + cron job, logic should be:

1. check if the vm is running

2. if it's running, shutdown the vm

3. copy/backup the vdisk image

Edited December 16, 2022 by ghost82