VM Restore - qemu/nvram folder missing

[sjameson]

All, I had a windows 11 VM crash and it restarted with the windows recovery message on the screen. Not sure what caused it to crash at all, just saw that it was paused on the VM's tab, and my monitor showed that recovery screen. 

 

I have a backup that I can use to restore it, but I'm getting stuck. I have the .xml file, the vdisk1.img, and the .fd file. I put the vdisk where it should go in my domains share that I use for VM vdisks, used the .xml file to create a VM in form mode, but the .fd file part is where I'm getting stuck. I had deleted the libvirt file to rebuild and see if that would kick my VM into gear before doing a full restore (excuse my naivete if that just made things harder), but obviously it didn't work. When I went to copy the .fd file (minus my backup timestamp on the file name) into the nvram folder, I noticed that there is no etc/libvirt/qemu/nvram folder on my server (thinking it has something to do with me deleting the libvirt image). Can I just make a new folder there and put the .fd file in, or are there different steps I should take to restore that folder?

[bmartino1]

unriad diag logs please, what version of unraid? stable v6 or prerelease 7?

if you have touched and removed the libvirt img any nvram and other data is now gone for a restore.
v7 vm is gone,
v6 no snapshots vm can potentialy recovered.

Also how did you make backups?
Appdata backup plugin or file copy?

Edited August 26 by bmartino1

[sjameson]

direwolf-diagnostics-20240826-1246.zip

Apologies, working off my phone today with my VM being down and didn't grab those. See attached. 

 

Stable release of Unraid. 6.12.10

[bmartino1]

Diag to report not a problem yet, just sharing...

ok, so your nvram file is gone. (Expected since loss of libvirt image.

2024-08-26 19:14:04.578+0000: 21667: error : qemuPrepareNVRAM:4559 : operation failed: unable to find any master var store for loader: /usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi-tpm.fd
2024-08-26 19:14:04.597+0000: 21667: warning : virHostdevReAttachUSBDevices:1832 : Unable to find device 000.000 in list of active USB devices
2024-08-26 19:14:04.597+0000: 21667: warning : virHostdevReAttachUSBDevices:1832 : Unable to find device 000.000 in list of active USB devices
2024-08-26 19:14:04.597+0000: 21667: warning : virHostdevReAttachUSBDevices:1832 : Unable to find device 000.000 in list of active USB devices

per diag qemu win 11 vm txt file, i'm not seeing a problem yet. we will need to make some edit to the xml to recover.

reviewing xml file now:

[sjameson]

Backups done via VM Backup plugin (not as regularyl as I should've been doing, but most important files are on server shares - just software that will need to be re-downloaded).

 

I have a libvirt file from months prior to the vdisk backup (for some reason it didn't backup the libvirt at that time). 

 

[bmartino1]

Ok, using the plugin there should be a restore feauture. witch can bring you back. if you want that old of a restore...

If you have not touched the vdisk.img file it should contain your lattest data. We can attempt to recreate while using that vdisk for a windows 11 vm

please make sure your windows vdisk.img is located at:
/mnt/user/domains/Windows 11/vdisk1.img

and go to the unraid VM tab and use this code to try and boot.

go to unraid web ui > Vm add VM custom and past the xml code click create only, then start the vm if no errors apear on screen and confirm Windows boot.
 

<domain type='kvm'>
  <name>Windows 11</name>
  <uuid>7195ada1-5c3c-f4fd-c4f5-4c664df5a8f2</uuid>
  <metadata>
    <vmtemplate xmlns="http://unraid.net/xmlns" name="Windows 11" icon="windows11.png" os="windowstpm"/>
  </metadata>
  <memory unit='KiB'>46137344</memory>
  <currentMemory unit='KiB'>4194304</currentMemory>
  <memoryBacking>
    <nosharepages/>
  </memoryBacking>
  <vcpu placement='static'>20</vcpu>
  <os>
    <type arch='x86_64' machine='pc-q35-7.2'>hvm</type>
  </os>
  <features>
    <acpi/>
    <apic/>
    <hyperv mode='custom'>
      <relaxed state='on'/>
      <vapic state='on'/>
      <spinlocks state='on' retries='8191'/>
      <vendor_id state='on' value='none'/>
    </hyperv>
  </features>
  <cpu mode='host-passthrough' check='none' migratable='on'>
    <topology sockets='1' dies='1' cores='10' threads='2'/>
    <cache mode='passthrough'/>
    <feature policy='require' name='topoext'/>
  </cpu>
  <clock offset='localtime'>
    <timer name='hypervclock' present='yes'/>
    <timer name='hpet' present='no'/>
  </clock>
  <on_poweroff>destroy</on_poweroff>
  <on_reboot>restart</on_reboot>
  <on_crash>restart</on_crash>
  <devices>
    <emulator>/usr/local/sbin/qemu</emulator>
    <disk type='file' device='disk'>
      <driver name='qemu' type='raw' cache='writeback'/>
      <source file='/mnt/user/domains/Windows 11/vdisk1.img'/>
      <target dev='hdc' bus='virtio'/>
      <serial>vdisk1</serial>
      <boot order='1'/>
      <address type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/Windows/Win11_23H2_English_x64v2.iso'/>
      <target dev='hda' bus='sata'/>
      <readonly/>
      <boot order='2'/>
      <address type='drive' controller='0' bus='0' target='0' unit='0'/>
    </disk>
    <disk type='file' device='cdrom'>
      <driver name='qemu' type='raw'/>
      <source file='/mnt/user/isos/virtio-win-0.1.248-1.iso'/>
      <target dev='hdb' bus='sata'/>
      <readonly/>
      <address type='drive' controller='0' bus='0' target='0' unit='1'/>
    </disk>
    <controller type='pci' index='0' model='pcie-root'/>
    <controller type='pci' index='1' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='1' port='0x8'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x0' multifunction='on'/>
    </controller>
    <controller type='pci' index='2' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='2' port='0x9'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x1'/>
    </controller>
    <controller type='pci' index='3' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='3' port='0xa'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x2'/>
    </controller>
    <controller type='pci' index='4' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='4' port='0xb'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x3'/>
    </controller>
    <controller type='pci' index='5' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='5' port='0xc'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x4'/>
    </controller>
    <controller type='pci' index='6' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='6' port='0xd'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x5'/>
    </controller>
    <controller type='pci' index='7' model='pcie-root-port'>
      <model name='pcie-root-port'/>
      <target chassis='7' port='0xe'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x01' function='0x6'/>
    </controller>
    <controller type='pci' index='8' model='pcie-to-pci-bridge'>
      <model name='pcie-pci-bridge'/>
      <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
    </controller>
    <controller type='virtio-serial' index='0'>
      <address type='pci' domain='0x0000' bus='0x02' slot='0x00' function='0x0'/>
    </controller>
    <controller type='sata' index='0'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x1f' function='0x2'/>
    </controller>
    <controller type='usb' index='0' model='ich9-ehci1'>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x7'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci1'>
      <master startport='0'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x0' multifunction='on'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci2'>
      <master startport='2'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x1'/>
    </controller>
    <controller type='usb' index='0' model='ich9-uhci3'>
      <master startport='4'/>
      <address type='pci' domain='0x0000' bus='0x00' slot='0x07' function='0x2'/>
    </controller>
    <interface type='bridge'>
      <mac address='52:54:00:c4:d2:18'/>
      <source bridge='br0'/>
      <model type='virtio-net'/>
      <address type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
    </interface>
    <serial type='pty'>
      <target type='isa-serial' port='0'>
        <model name='isa-serial'/>
      </target>
    </serial>
    <console type='pty'>
      <target type='serial' port='0'/>
    </console>
    <channel type='unix'>
      <target type='virtio' name='org.qemu.guest_agent.0'/>
      <address type='virtio-serial' controller='0' bus='0' port='1'/>
    </channel>
    <input type='tablet' bus='usb'>
      <address type='usb' bus='0' port='4'/>
    </input>
    <input type='mouse' bus='ps2'/>
    <input type='keyboard' bus='ps2'/>
    <tpm model='tpm-tis'>
      <backend type='emulator' version='2.0' persistent_state='yes'/>
    </tpm>
    <audio id='1' type='none'/>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x05' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='pci' managed='yes'>
      <driver name='vfio'/>
      <source>
        <address domain='0x0000' bus='0x01' slot='0x00' function='0x1'/>
      </source>
      <address type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x0b05'/>
        <product id='0x1984'/>
      </source>
      <address type='usb' bus='0' port='1'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x413c'/>
        <product id='0x3012'/>
      </source>
      <address type='usb' bus='0' port='2'/>
    </hostdev>
    <hostdev mode='subsystem' type='usb' managed='no'>
      <source>
        <vendor id='0x8087'/>
        <product id='0x0032'/>
      </source>
      <address type='usb' bus='0' port='3'/>
    </hostdev>
    <memballoon model='none'/>
  </devices>
</domain>

some things were removed for unriad vm recreation.

Edited August 26 by bmartino1

[sjameson]

I can't do anything remotely with that VM right now unfortunately, so I can't see what's going on in that machine right now. But, it looks like it booted. Just need to check on my monitor what's going on once I get home. 

 

There is still no nvram folder. Do I not need that .fd file for everything to work correctly?

 

I see the OVMF lines are removed from the code you provided, does that not matter?

[bmartino1]

nvram file you have may not be capable of being restores, I would rather have unraid regenerate the file.

Removing the ovmf line is what allows the current libvirt to generate a nvram file.

once the system boots once a new ovmf line should be created. you may need to use the webUI (non xml) to set some setting to reapply after a successful boot.

Your pinned CPU are also removed form the xml code.

This is what i have seen when testing potential failures and how to restore form them.

in unraid we can then go to libvirt log and the vm log to see any potential problems. But otherwise the new vm should work moving forward.

[sjameson]

Just for anyone else who finds this thread down the line:

 

SeaBIOS will not work with my setup due to the GPU requiring UEFI for passthrough. I was able to restore to a previous backup with the following steps (the custom XML VM provided above did not work as it wouldn't boot with my GPU due to the BIOS being changed to SeaBIOS):

 

Created new nvram folder in etc/libvirt/qemu for my OVMF VARS files

Copied backup .fd into the newly created nvram folder (make sure to remove the timestamp from the beginning of any backup files)

Copied backup vdisk1 into my domains folder associated with this VM

Copied old backup libvirt.img (xml and vdisk 1 were from late July, and the libvirt image was from back in January) and replaced the current libvirt image in /mnt/user/system/libvirt/libvirt.img

 

Used the xml to create new windows 11 VM via xml mode

 

Everything booted up at that point and I only lost a month's worth of software that I had setup (no important documents, just the software packages themselves). 

 

Probably going to rethink my backup strategy for VM's now and make things more automatic. I stopped doing automatic backups because I had to shut down the VM in order to do so. Not sure if I want to redo everything and try to start using snapshots instead, but I'm not sure how straightforward that conversion would be. Likely going to just do more frequent manual backups after major changes to software/settings that I want to persist.

 

@bmartino1 I appreciate the help! Your solution will likely help a lot of other people, but didn't quite help me due to requiring UEFI boot for my GPU passthrough.

 

Long story short: if you have a .fd VARS file for a backup restore, but your nvram folder is gone, just create a new nvram folder where it should go (/etc/libvirt/qemu/nvram/) and copy your .fd file in there and start up that VM!

Edited August 27 by sjameson