VM Backup Plugin


Recommended Posts

Hi,

 

I'm trying to set up this plugin to backup a VM that is running. I.e. it needs to be shut off before the backup.

 

The plugin logs that it tries to shut it off, but it's not working. I've tested and the VM can ble cleanly shut down with "Stop" through the WebUI, so theres nothing preventing the VM from being shut off. 

 

Theres also nothing in the syslog indicating that the VM has gotten a shutdown-command, so I'm pretty sure this is not working from the plugin. 

 

Any pointers? I obviously don't want to enable force shutdown/kill.

 

Thanks!

 

 image.thumb.png.5579a1cdff9c69aa310c2fc6ac223171.pngimage.thumb.png.686361859800fc4f5fb3af3f326074b3.png 

Link to comment

Hi community, recently my VM backup script needs hours for a backup. One VM is backed up within 30 minutes, the other VM takes tens of hours, with the script seemingly pausing. Here's a snippet from the log file. Anyone have any idea what happens between 10pm and almost 6am?

2021-07-26 21:40:24 information: copy of backup of /mnt/user/Backup-VM/Server 2019/20210724_1234_server2019.img vdisk to /mnt/user/Backup-VM/Server 2019/20210726_2140_server2019.img starting.
2021-07-26 22:49:36 information: copy of /mnt/user/Backup-VM/Server 2019/20210724_1234_server2019.img to /mnt/user/Backup-VM/Server 2019/20210726_2140_server2019.img complete.
2021-07-26 22:49:36 information: extension for /mnt/user/isos/windows_2019_Server.iso on Server 2019 was found in vdisks_extensions_to_skip. skipping disk.
2021-07-26 22:49:36 information: extension for /mnt/user/isos/virtio-win-0.1.190-1.iso on Server 2019 was found in vdisks_extensions_to_skip. skipping disk.
2021-07-26 22:49:36 information: skip_vm_shutdown is false. beginning vm shutdown procedure.
2021-07-26 22:49:36 infomration: Server 2019 is running. vm desired state is shut off.
2021-07-26 22:49:43 information: performing 20 30 second cycles waiting for Server 2019 to shutdown cleanly.
2021-07-26 22:49:43 information: cycle 1 of 20: waiting 30 seconds before checking if the vm has entered the desired state.
2021-07-26 22:50:13 information: Server 2019 is shut off. vm desired state is shut off. can_backup_vm set to y.
2021-07-26 22:50:13 information: actually_copy_files is 1.
2021-07-26 22:50:13 information: can_backup_vm flag is y. starting backup of Server 2019 configuration, nvram, and vdisk(s).
2021-07-26 22:50:13 information: copy of Server 2019.xml to /mnt/user/Backup-VM/Server 2019/20210726_2140_Server 2019.xml complete.
2021-07-26 22:50:13 information: copy of /etc/libvirt/qemu/nvram/dec2ff71-7e5e-7cd2-e5b3-b5e22cdc9a06_VARS-pure-efi.fd to /mnt/user/Backup-VM/Server 2019/20210726_2140_dec2ff71-7e5e-7cd2-e5b3-b5e22cdc9a06_VARS-pure-efi.fd complete.
2021-07-27 05:57:02 information: copy of /mnt/disk1/domains/Server 2019/server2019.img to /mnt/user/Backup-VM/Server 2019/20210726_2140_server2019.img complete.
2021-07-27 05:57:06 information: backup of /mnt/disk1/domains/Server 2019/server2019.img vdisk to /mnt/user/Backup-VM/Server 2019/20210726_2140_server2019.img complete.
2021-07-27 05:57:06 information: extension for /mnt/user/isos/windows_2019_Server.iso on Server 2019 was found in vdisks_extensions_to_skip. skipping disk.
2021-07-27 05:57:06 information: extension for /mnt/user/isos/virtio-win-0.1.190-1.iso on Server 2019 was found in vdisks_extensions_to_skip. skipping disk.
2021-07-27 05:57:06 information: the extensions of the vdisks that were backed up are img.
2021-07-27 05:57:06 information: vm_state is running. vm_original_state is running. not starting Server 2019.
2021-07-27 05:57:06 information: backup of Server 2019 to /mnt/user/Backup-VM/Server 2019 completed.
2021-07-27 05:57:06 information: number of days to keep backups set to indefinitely.
2021-07-27 05:57:06 information: cleaning out backups over 2 in location /mnt/user/Backup-VM/Server 2019/
2021-07-27 05:57:06 information: did not find any config files to remove.
2021-07-27 05:57:06 information: did not find any nvram files to remove.
2021-07-27 05:57:06 information: did not find any vdisk image files to remove.
2021-07-27 05:57:06 information: did not find any vm log files to remove.

 

Link to comment

Can we please please get a newbee friendly tutorial how to restore a vm? 

A few others also asked and noone really seams to care. 

I figured out that i have datestamped files. Always 3:

FD-File like ....VARS-pure-efi

QCOW2 File... 

xml file...

 

I guess i know what to do with the xml and the qcow2, but where do i put the fd file?

  • Like 1
Link to comment
57 minutes ago, charley said:

Can we please please get a newbee friendly tutorial how to restore a vm? 

A few others also asked and noone really seams to care. 

I figured out that i have datestamped files. Always 3:

FD-File like ....VARS-pure-efi

QCOW2 File... 

xml file...

 

I guess i know what to do with the xml and the qcow2, but where do i put the fd file?

 

From the XML view of my VM it appears the FD file goes in:

/etc/libvirt/qemu/nvram/

 

Which is one level down from the XML file in:

/etc/libvirt/qemu/

 

Link to comment

Hi everyone,

 

I am use this script/plugin since a couple of days but have now the following issue:
 

The Windows 10 VM is running 24/7 and backup is planned every day at 21:00, but the state of the VM is paused. Is it possbile to resume the VM automatically after the backup was finished or that the vm is not paused while a VM backup is performed?

Also I have set it to to snapshot backups but my files are always the same size (22GB each vdisk)

 

Update: System is going to sleep because somehow the energy settings have been changed (not by me). So this issue is solved but the one with creating snapshots instead of every day full backups is still open. Anyone has an idea?

Edited by PsYcRo
Link to comment

I have a problem with the vm backup tool it was working fine then i upgraded to unraid V6.10.0-rc1 and the snap shot part of the program stopped working it is giving me this error any idea what is causing it

 

failure: snapshot command failed on vdisk1.snap for Windows 10 Pro.

 

 

Thank you

20210811_1603_unraid-vmbackup.log

Edited by JP s
Adding Test File
  • Like 1
Link to comment

i keep getting an error complaining about file storage when doing snapshots. I'm not sure what it's referencing, i have set the backup location to /mnt/user/backups/vmbackup/ and this location works when stopping and backing up.

 

vm_state is running. vdisk_type is raw
2021-08-14 04:43:14 information: qemu agent found. enabling quiesce on snapshot.
error: internal error: missing storage backend for 'file' storage

 

Link to comment
  • 3 weeks later...
On 8/12/2021 at 1:15 AM, JP s said:

I have a problem with the vm backup tool it was working fine then i upgraded to unraid V6.10.0-rc1 and the snap shot part of the program stopped working it is giving me this error any idea what is causing it

 

failure: snapshot command failed on vdisk1.snap for Windows 10 Pro

 

 

Similar issue here after the 6.10.0-rc1 update. Both automatic and manual Backups don't create a disk snapshot, only the .fd and .xml file for the VM. 

 

Log attached. 

20210830_1416_unraid-vmbackup.txt

  • Like 1
Link to comment
  • 5 weeks later...
On 12/18/2019 at 11:39 PM, JTok said:

Mod note: this plugin has multiple known issues, use with care.

 

This is a beta version of a plugin I am working on to replace the unraid-vmbackup script I have been maintaining here and here.

It is based off the latest version of the script (v1.3.1), and I will be maintaining both for now as there are some use-cases for the script that the plugin cannot currently replace. This is provided as-is with no guarantees.

 

That said, and without further ado, the current version of the plugin can be found here:

https://github.com/JTok/unraid.vmbackup/tree/v0.2.2

 

-JTok

 

Feel free to buy me a beer (or two 😋)

 

What are the known issues? What can be done safely?

 

Reading the most recent comments, this seems to have compatibility issues with the 6.10.0-rc1 version?

 

Any way to view the progress as it's backing up?

Edited by Econaut
  • Haha 1
Link to comment
  • 2 weeks later...

Mark me down as another who can't backup on 6.10.0-rc1...same errors as above.  Working until rc1.

 

Please update :) 

 

And thanks for the work on this plugin!

2021-10-11 10:43:42 Start logging to log file.
2021-10-11 10:43:42 information: send_notifications is 1. notifications will be sent.
2021-10-11 10:43:42 information: only_send_error_notifications is 0. normal notifications will be sent if send_notifications is enabled.
2021-10-11 10:43:42 information: keep_log_file is 1. log files will be kept.
2021-10-11 10:43:42 information: number_of_log_files_to_keep is 1. this is probably a sufficient number of log files to keep.
2021-10-11 10:43:42 information: enable_vm_log_file is 0. vm specific logs will not be created.
2021-10-11 10:43:42 information: backup_all_vms is 1. vms_to_backup will be ignored. all vms will be backed up.
2021-10-11 10:43:42 information: use_snapshots is 1. vms will be backed up using snapshots if possible.
2021-10-11 10:43:42 information: kill_vm_if_cant_shutdown is 0. vms will not be forced to shutdown if a clean shutdown can not be detected.
2021-10-11 10:43:42 information: set_vm_to_original_state is 1. vms will be set to their original state after backup.
2021-10-11 10:43:42 information: number_of_days_to_keep_backups is 0. backups will be kept indefinitely. be sure to set number_of_backups_to_keep to keep backups storage usage down.
2021-10-11 10:43:42 information: number_of_backups_to_keep is 3. this is probably a sufficient number of backups to keep.
2021-10-11 10:43:42 information: inline_zstd_compress is 1. vdisk images will be inline compressed but will not be compared afterwards or post compressed.
2021-10-11 10:43:42 information: zstd_level is 3.
2021-10-11 10:43:42 information: zstd_threads is 2.
2021-10-11 10:43:42 information: snapshot_extension is snap. continuing.
2021-10-11 10:43:42 information: snaphot extension not found in vdisk_extensions_to_skip. extension was added.
2021-10-11 10:43:42 information: snapshot_fallback is 0. snapshots will fallback to standard backups.
2021-10-11 10:43:42 information: pause_vms is 0. vms will be shutdown for standard backups.
2021-10-11 10:43:42 information: enable_reconstruct_write is 0. reconstruct write will not be enabled by this script.
2021-10-11 10:43:42 information: backup_xml is 1. vms will have their xml configurations backed up.
2021-10-11 10:43:42 information: backup_nvram is 1. vms will have their nvram backed up.
2021-10-11 10:43:42 information: backup_vdisks is 1. vms will have their vdisks backed up.
2021-10-11 10:43:42 information: start_vm_after_backup is 0. vms will not be started following successful backup.
2021-10-11 10:43:42 information: start_vm_after_failure is 0. vms will not be started following an unsuccessful backup.
2021-10-11 10:43:42 information: disable_delta_sync is 0. rsync will be used to perform delta sync backups.
2021-10-11 10:43:42 information: rsync_only is 0. cp will be used when applicable.
2021-10-11 10:43:42 information: actually_copy_files is 1. files will be copied.
2021-10-11 10:43:42 information: clean_shutdown_checks is 20. this is probably a sufficient number of shutdown checks.
2021-10-11 10:43:42 information: seconds_to_wait is 30. this is probably a sufficient number of seconds to wait between shutdown checks.
2021-10-11 10:43:42 information: keep_error_log_file is 1. error log files will be kept.
2021-10-11 10:43:42 information: number_of_error_log_files_to_keep is 10. this is probably a sufficient error number of log files to keep.
2021-10-11 10:43:42 information: started attempt to backup Windows 10 to /mnt/user/UNRAID/YALE-FORTY-FIVE/YALE-FORTY-FIVE/BACKUP/VM_Backups
2021-10-11 10:43:42 information: Windows 10 can be found on the system. attempting backup.
2021-10-11 10:43:42 information: removing old local Windows 10.xml.
2021-10-11 10:43:42 information: creating local Windows 10.xml to work with during backup.
2021-10-11 10:43:42 information: /mnt/user/UNRAID/YALE-FORTY-FIVE/YALE-FORTY-FIVE/BACKUP/VM_Backups/Windows 10 exists. continuing.
2021-10-11 10:43:42 information: skip_vm_shutdown is false and use_snapshots is 1. skipping vm shutdown procedure. Windows 10 is running. can_backup_vm set to y.
2021-10-11 10:43:42 information: actually_copy_files is 1.
2021-10-11 10:43:42 information: can_backup_vm flag is y. starting backup of Windows 10 configuration, nvram, and vdisk(s).
2021-10-11 10:43:42 information: copy of Windows 10.xml to /mnt/user/UNRAID/YALE-FORTY-FIVE/YALE-FORTY-FIVE/BACKUP/VM_Backups/Windows 10/20211011_1043_Windows 10.xml complete.
2021-10-11 10:43:42 information: copy of /etc/libvirt/qemu/nvram/4a2b120f-0ea9-846a-6e11-f097002e442d_VARS-pure-efi.fd to /mnt/user/UNRAID/YALE-FORTY-FIVE/YALE-FORTY-FIVE/BACKUP/VM_Backups/Windows 10/20211011_1043_4a2b120f-0ea9-846a-6e11-f097002e442d_VARS-pure-efi.fd complete.
2021-10-11 10:43:42 information: able to perform snapshot for disk /mnt/cache/domains/UNRAID_VM1/vdisk1.img on Windows 10. use_snapshots is 1. vm_state is running. vdisk_type is raw
2021-10-11 10:43:42 information: qemu agent found. enabling quiesce on snapshot.
2021-10-11 10:43:42 failure: snapshot command failed on vdisk1.snap for Windows 10.
2021-10-11 10:43:42 failure: snapshot_fallback is 0. skipping backup for Windows 10 to prevent data loss. no cleanup will be performed for this vm.
2021-10-11 10:43:42 information: finished attempt to backup Windows 10 to /mnt/user/UNRAID/YALE-FORTY-FIVE/YALE-FORTY-FIVE/BACKUP/VM_Backups.
2021-10-11 10:43:42 information: cleaning out logs over 1.
2021-10-11 10:43:42 information: removed '/mnt/user/UNRAID/YALE-FORTY-FIVE/YALE-FORTY-FIVE/BACKUP/VM_Backups/logs/20211011_1042_unraid-vmbackup.log'.
2021-10-11 10:43:42 information: cleaning out error logs over 10.
2021-10-11 10:43:42 information: did not find any error log files to remove.
2021-10-11 10:43:42 warning: errors found. creating error log file.
2021-10-11 10:43:42 Stop logging to error log file.

 

  • Thanks 1
Link to comment
  • 2 weeks later...

Hi all,

First of all BIG Thanks for the plugin , real good thing , using it on 5 systems for months , no issues whatsoever

- except VM states , i did think its plugin issues , but then after some playing and trial and error ALL the issues are related to VMs(mostly Windows being windows and settings after updates etc)

 

Now i do have a question , how is it so fast? if i use anything using same disks (from NVME to NVME) be that duplicati or anything really i tried a lot , it takes around 20 minutes to backup up to 200GB (be that 1,2 or 4 VMs)

But the plugin does it in 2 minutes,

 

where can i read , or can some1 explain why the plugin is so fast and everything else is not?

mainly asking because id like a restore option, in the meanwhile its not an issue, especially since i didn't even need it yet, but ill do it manually if needed.

 

Also i assume that (attached screenshot) this is a mistyped value.image.png.6c45251014ed2d9be47c096a1ca860ef.png

Edited by Xcage
Link to comment
  • 3 weeks later...

HOW TO RESTORE!!!!
For anyone this might help- if you try to restore the img file that is in the zst compressed file: you may receive an error of - 
 Decoding error (36) : Restored data doesn't match checksum

 

To get around this run the following (replace with your file name) 

zstd -d -C --no-check 20211114_0300_vdisk1.img.zst

Where 20211114_0300_vdisk1.img.zst is your file name to replace. If this doesn't work in terminal, try Cygwin (Windows) and run it. https://www.cygwin.com/

Place your .zst file in the c:\Cygwin folder. Much easier! 

For some reason, every VM backed up with zst selected is created like this and throws an error (for me). They ALL restore without issue (for me) doing this.  

PS, you'll need to create a custom VM once your .img file is restored. Paste the contents of the associated backed up XML file in. LOOK AT YOUR XML FILE  closely!! Ensure the img file is in the same folder as was/is spec'd in the XML and also ensure the following are present: 

(File names may vary, look at backed up files) 

 

/usr/share/qemu/ovmf-x64/OVMF_CODE-pure-efi.fd
/etc/libvirt/qemu/nvram/4a2b120f-0ea9-846a-6e11-f097002e442d_VARS-pure-efi.fd
 

For what its worth- I have disabled zst compression and just have it back up the img file!! No need to take a chance.  Hope this helps! 

 

Edited by dja
  • Like 1
Link to comment
  • 3 weeks later...

Great plugin, but i have a few oddities 

 

when the plugin runs at it schedule time, i get an email that says...

 

Description: cannot run /boot/config/plugins/vmbackup/configs/Weekly_Backups/, and yet the plug does seem to run. 

 

When it running, it seems to take a copy of an existing backup first, 

 

2021-12-05 15:31:47 information: copy of backup of /mnt/user/sysbackup/vms/autobackup/LibreNMS/20211205_0440_librenms-ubuntu-20.04-amd64.img vdisk to /mnt/user/sysbackup/vms/autobackup/LibreNMS/20211205_1527_librenms-ubuntu-20.04-amd64.img starting.
'/mnt/user/sysbackup/vms/autobackup/LibreNMS/20211205_0440_librenms-ubuntu-20.04-amd64.img' -> '/mnt/user/sysbackup/vms/autobackup/LibreNMS/20211205_1527_librenms-ubuntu-20.04-amd64.img'
2021-12-05 15:36:48 information: copy of /mnt/user/sysbackup/vms/autobackup/LibreNMS/20211205_0440_librenms-ubuntu-20.04-amd64.img to /mnt/user/sysbackup/vms/autobackup/LibreNMS/20211205_1527_librenms-ubuntu-20.04-amd64.img complete.
 

Then the backup runs, but it cant see any other files to remove in the backup directory, so im gradually building up backup files even though i have the number of files to keep set to 2...

 

 

2021-12-05 15:40:57 information: copy of /mnt/user/vms/librenms/librenms-ubuntu-20.04-amd64.img to /mnt/user/sysbackup/vms/autobackup/LibreNMS/20211205_1527_librenms-ubuntu-20.04-amd64.img complete.
2021-12-05 15:40:57 information: backup of /mnt/user/vms/librenms/librenms-ubuntu-20.04-amd64.img vdisk to /mnt/user/sysbackup/vms/autobackup/LibreNMS/20211205_1527_librenms-ubuntu-20.04-amd64.img complete.
2021-12-05 15:40:57 information: the extensions of the vdisks that were backed up are img.
2021-12-05 15:40:57 information: vm_state is shut off. vm_original_state is running. starting LibreNMS.
Domain LibreNMS started

2021-12-05 15:40:59 information: backup of LibreNMS to /mnt/user/sysbackup/vms/autobackup/LibreNMS completed.
2021-12-05 15:40:59 information: number of days to keep backups set to indefinitely.
2021-12-05 15:40:59 information: cleaning out backups over 2 in location /mnt/user/sysbackup/vms/autobackup/LibreNMS/
2021-12-05 15:40:59 information: removed '/mnt/user/sysbackup/vms/autobackup/LibreNMS/20211128_1302_LibreNMS.xml' config file.
2021-12-05 15:40:59 information: did not find any nvram files to remove.
2021-12-05 15:40:59 information: did not find any vdisk image files to remove.
2021-12-05 15:40:59 information: removed '/mnt/user/sysbackup/vms/autobackup/LibreNMS/20211128_1302_unraid-vmbackup.log' vm log file.
2021-12-05 15:40:59 information: removing local LibreNMS.xml.
removed 'LibreNMS.xml'

Link to comment

I've set up to allow snapshots on my Win10 VM that's running my cameras.

I think I've config'd per the instructions, but when the backup runs, the script fails, and the log shows the following error:

"error: internal error: missing storage backend for 'file' storage"

Any idea what I need to change?

It works fine if I allow it to shut down the VM first, but I'd prefer to have it take a snapshot instead, if that will work.

Link to comment
6 minutes ago, Elmojo said:

I've set up to allow snapshots on my Win10 VM that's running my cameras.

I think I've config'd per the instructions, but when the backup runs, the script fails, and the log shows the following error:

"error: internal error: missing storage backend for 'file' storage"

Any idea what I need to change?

It works fine if I allow it to shut down the VM first, but I'd prefer to have it take a snapshot instead, if that will work.

It sounds like the target for your backup is not quite right. Where are you backing up to? 

Link to comment
Just now, Elmojo said:

Thanks for the quick reply!

I have a share on my array called "xVM Backups"

It's at /mnt/user/Backup/xVM Backups/

Is that wrong?

That sounds right if the path exist. Can you navigate to the target in terminal?

Also, can you post the log file? (forgot to ask!)  You may need to enable that. (3rd tab, 'other settings')  If you don't have it enabled now, you may need to run the backup again and then attach that file here.  I've got a Win10 VM running with Virtio drivers installed and it works well, you should be able to backup.  Do you have Virtio drivers installed for the VM? 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.