JTok

@andreidelait Unfortunately, the snapshot mechanism is just for allowing backups to be performed relatively safely while the virtual machine is running. I'm over-simplifying this a little bit, but essentially, when a VM is running, it is making changes to the disk image. In order to safely make a backup of a disk image and reduce the chance of data loss, we must stop the VM from making changes to the disk image. We can do that two ways currently. The first is by turning off the VM and making a copy of the disk image while the VM is not in use. The second is to use snapshots. Snapshots work by creating a second disk image for the VM to start writing to instead of the original disk image. Which is why it is just the changes and not a full copy, just like you were thinking. However, since the snapshots are live disk images, we can't safely back them up for the same reason we couldn't back up the original disk image before. But, after making a snapshot, we can safely make copy of the (now static) original disk image. So, you are right that the snapshot itself is the changes to the original disk image, but sadly, we can't safely make backups from them. What you are describing is a delta backup. It is possible I might look into how to perform more complicated backup schemes that use delta backups like incremental or differential backups, but that will be a little ways down the road. -JTok P.S. A note about the QEMU Guest Agent. This is entirely unnecessary information, but since we are on the topic of snapshots, and in case anyone is wondering why it is a good idea to use the guest agent, here is an explanation. Again, I am simplifying a bit. We can actually create snapshots without the guest agent installed. One of the things the guest agent does is allow the guest OS to be informed of things that we are doing at the host level, such as making a snapshot. When a snapshot is performed *without* the guest agent installed the guest just keeps writing to the "hard drive" without being aware that the disk file it is writing to has been changed to a snapshot image file. This can leave the data on the original disk image in an inconsistent state, meaning that if you try to restore to it there may be issues with some of the information resulting in data loss or corruption. Certain applications are particularly sensitive to this, such as those with databases like an SQL server or Active Directory. When a snapshot is performed *with* the guest agent installed, the guest OS is asked to quiesce. Meaning that it will flush any write caches to disk, and let all current writes to the "hard drive" finish while preventing new writes from starting. This prepares the original disk image for a backup by ensuring that its data is in a consistent state. Once the original disk image is in a consistent state, the guest is informed it can start writing again, only this time the writes are going to the snapshot.

This version fixes some major issues and adds some advanced functionality. If you were running a weekly schedule, it will actually be running daily using the previous version of the plugin. To fix this, just disable and re-enable the schedule. v0.1.8 - 2019/12/24 Penny Wise and Pound Stupid - fixed weekly cronjob. - fixed validation to allow spaces in paths. - fixed cronjob not being removed during uninstall. - fixed extension matching case sensitivity. - added fix for stuck snapshots. - added ability to abort running scripts. - changed default folder start location based on restrictive validation setting. - updated help. - updated lists to be sorted alphabetically. https://github.com/JTok/unraid.vmbackup/tree/v0.1.8 -JTok

bug-fix release. v1.2.3 - 2019/12/24 Leftovers - fixed extension matching case sensitivity. Script here: https://github.com/JTok/unraid-vmbackup/tree/v1.2.3 -JTok

It looks like the validation isn't allowing paths with spaces. I will add fixing that to my to-do list. Thanks.

unRAID should still detect updates in the plugin screen without it being part of CA (assuming I didn’t mess something up). And the CA update plugin should also still do automatic updates from what I can tell. That said, I've been in contact with Squid about adding it, so that should get done soon.

New release with some minor functional changes and a few bug fixes. This was mostly background stuff with the exception of adding a "Backup Now" button. v0.1.7 - 2019/12/22 Ms. Frizzle - fixed minor issues. - fixed paths requiring trailing slashes. - fixed backup location drop-down color when using black theme. - added additional logging. - added donate button. - added backup now button. - changed backup location drop-down start folder to /mnt/. - updated help. - updated readme. https://github.com/JTok/unraid.vmbackup/tree/v0.1.7 -JTok

I hadn’t really thought about it, but I probably should lol. I’ll put that on my to-do list, but that’s probably a post-beta feature right now. Sent from my iPhone using Tapatalk

[emoji23] yeah! That’s on my radar! When testing, it was a little frustrating at times without one. Sent from my iPhone using Tapatalk

Weird. I’ll check it out. Thanks [emoji106] Sent from my iPhone using Tapatalk

Make sure the path you’re typing contains a trailing slash. I’m going to change that requirement in a future release, but I didn’t change it yet for the beta. Sent from my iPhone using Tapatalk

You should be able to manually type any path that starts with /mnt/ even though the drop down list only shows shares. If you need a path that starts with something other than /mnt/: Under the Danger Zone tab, at the very bottom. there is a setting called “Disable restrictive validation”. If you set it to ‘Yes’, it should allow you to chose any path. Sent from my iPhone using Tapatalk

It is official! I finally have the first beta version of the plugin available here: -JTok

⚠️ NOTICE: I currently do not have the time to do more than maintain this project for myself at the moment. I will eventually do more work on it as my life permits, but for now development will be sporadic at best. I am more happy to add someone else to the project to maintain it, or even accept some PRs as that is much easier for me to find time for

realizelol strikes again! v0.2.9 - 2024/05/02 Widdershins add a more detailed VM description to the empty vdisk_path notification (@realizelol) add a flexible variable to the sleep-timer before ACPI VM shutdown so VM can recognize the "power-button-shutdown" (@realizelol) https://github.com/JTok/unraid.vmbackup/tree/v0.2.9 -JTok

⚠️ NOTICE: I currently do not have the time to do more than maintain this project for myself at the moment. I will eventually do more work on it as my life permits, but for now development will be sporadic at best. I am more happy to add someone else to the project to maintain it, or even accept some PRs as that is much easier for me to find time for Mod note: this plugin has multiple known issues, use with care. This is a beta version of a plugin I am working on to replace the unraid-vmbackup script I have been maintaining here and here. It is based off the latest version of the script (v1.3.1), and I will be maintaining both for now as there are some use-cases for the script that the plugin cannot currently replace. This is provided as-is with no guarantees. That said, and without further ado, the current version of the plugin can be found here: https://github.com/JTok/unraid.vmbackup/tree/v0.2.9 -JTok Feel free to buy me a beer (or two 😋)

The biggest change in this release is that when backup_all_vms is enabled, vms_to_backup becomes an exclusion list. v1.2.2 - 2019/12/18 Sam Vimes - added logging to local vm config dump actions. - added ability to use vms_to_backup as an exclusion list when backup_all_vms is enabled. - fixed rm commands missing -fv options. Script here: https://github.com/JTok/unraid-vmbackup/tree/v1.2.2 -JTok

I had hoped my next post would be about the plugin beta, but I'm still working on that. Though it should be ready for testing very soon. This is just a bug-fix for the regular script (details below). There is some bad news about the bug. If you were running v1.2.0 with snapshots and experienced the bug explained below, you will need to fix your VMs before the backup script will work with them again. The good news is that running the backup script will not cause any further issues, but backups will fail for affected VMs. There are two ways to fix it. The first is to manually fix it yourself, which is safest if you understand what you are doing. I would highly recommend doing that if you are comfortable. The only two commands you have to run against *each* vdisk that has a broken snapshot are: virsh blockcommit "$vm" "${vdisk_specs[$disk]}" --active --wait --verbose --pivot rm -fv "$disk_directory/$snap_name" where "$vm" is the name of the vm and "${vdisk_specs[$disk]}" is the value for dev for each vdisk in your VM's xml. Keep the double quotes. The xml line will look like: <target dev='hdc' bus='scsi'/> the second command above is VERY dangerous, so be certain you know what you are doing. You will change the quoted part to the path of the SNAPSHOT vdisk, keeping the quotes this time as well. Be very careful not to accidentally use the path of your actual vdisk. It will then delete the no longer needed snapshot. A second option is to use the attached "fix-stuck-snapshots" script. The script will only work on VMs that are running, so be sure that any VM that needs fixed is turned on. I have tested it on my system, and I tried to make it as safe as possible... but I cannot, and will not, guarantee that something couldn't make it delete the wrong vdisk and cause you to lose data. Use at your own risk. As long as you are using the default snapshot extension from the previous script, you should be able to just run it once and then remove it. It will push some event notifications and log to the syslog. If you have any questions or need any help feel free to reply here, and I'll do my best. v1.2.1 - 2019/12/11 Moth-man is Real - fixed bug that could cause snapshots not to be removed in specific situations. Script here: https://github.com/JTok/unraid-vmbackup/tree/v1.2.1 -JTok Extra details for those that care: The script would not remove a snapshot if a VM that was shut off got backed up before one that was running. If no VMs were off, then this would not happen. If the VMs that were off got backed up after all the ones that were on, this would not happen. fix-stuck-snapshots.sh

Funny you should mention that 😁. I'm getting very close to having a beta version of the script as a plugin available. I'm expecting to have that done in the next few days (maybe even as early as tomorrow). A good idea. I think I have a relatively elegant way to implement that in mind. Expect it to show up in a future version. 👍

It hasn't been very long since the last update, but I had some other changes I wanted to get added before the weekend. I've tested this version as thoroughly as I am able, but it has a lot of under-the-hood changes this time, so I apologize now if I missed something. release highlights (full notes below): 1) the ability to back up *all* VMs without explicitly listing them. 2) option to choose your own snapshot extension. 3) the option to fall back to a standard backup if a snapshot fails (this means shutting down the VM like normal). 4) the option to pause the VMs instead of shutting them down during a standard backup. (fair warning, I do not know how safe this actually is, so use at your own risk.) v1.2.0 - 2019/11/27 Starring Tom Hanks - major rewrites to make script smaller, slightly more efficient, and more modular. - added option to choose different snapshot extension. - added option to allow all vms to be backed up without having to explicitly list them. - added option to revert back to standard backup if snapshot fails. - added option to pause vms during standard backups instead of shutting down. - added support for pulling nvram extension from the config file. - fixed 'integer expression expected' error when performing regular expression integer comparisons. - fixed dry run option not working. - workaround for 'namespace warning' error when working with config file. - improved readability of log file by removing unnecessary line breaks. Script here: https://github.com/JTok/unraid-vmbackup/tree/v1.2.0 -JTok

Yeah, both of those are not easily avoided currently. However, neither of them should cause an issue. The "namespace warning" is because xmlns is just "unraid", and the parser is expecting an absolute path (like http://unraid.net/foo). Unless the way UnRAID does xml creation is changed to use an absolute URI, that error will continue to appear. The "integer expression expected" is because of a regular expression I'm using in the script. In the next update I'm intending to catch the error and handle it more gracefully. Essentially though it's just letting you know that of the two variables it is comparing, at least one of them is not actually a number.

@bastl I think I fixed it. Please grab the updated v1.1.5 script from the GitHub link I used earlier and let me know. Thanks, JTok

Yes, that is exactly how it works. I was able to replicate your issue, and I'm working on it now. I'm hoping to have it fixed in the next few hours.

I know it's been a while, but I made some big-ish changes to the script that some of you may find useful. I haven't really read back through the thread yet, so I'm not sure if someone else has made any additions not included here. The biggest changes are 1. vdisk extensions are pulled directly from the config file, so you can name them anything you want (except .snap). This should prevent the issue @rorton was having. 2. SNAPSHOTS!! Snapshots are now able to be used to allow live backups, but there are some caveats so be sure to pay attention to the notes in the script when enabling it. BIG thanks to @Dikkekop for that one. 3. any extension can be blacklisted now, not just iso. v1.1.5 - 2019/11/21 In Loving Memory of Net Neutrality - fixed some errors not generating an error log file. - updated comments. - updated documentation to include important information. - reverse changelog order. - added support for more vdisk extensions by parsing them directly from the config. - added user variable (case-insensitive) for blacklisting extensions. iso included by default. - fixed issue with where files would not be removed after compressing when config files were not set to be backed up. - updated comments to use more consistent terms. - added option to allow specific vms to be backed up without shutting down. - added option to allow vms to be backed up using snapshots. disabled by default. - fixed script trying to remove leftover files that weren't part of the backup. This thing has gotten rather complex, and I only use some of the functions in my backups, so as always: I've tested it as best as I am able to, but cannot guarantee it will work for you. Please verify your backups are running correctly when using the script. Script here: https://github.com/JTok/unraid-vmbackup/tree/v1.1.5 -JTok

So... I honestly I haven't looked at this for more than a few months since it has been working for my needs and my personal life has been rather busy. However, I did start working on (and am planning to eventually finish) some updates to this script as well as a version that works with btrfs snapshots. Things have slowed down a little lately, so I've started getting back into these kind of projects. I will take another look at the vdisk skipping, plus I'm working on adding the ability to parse the vdisk names directly from the xml file as well as a few other things. Unfortunately I don't have a firm timeline right now, but I expect to be able to devote a decent amount of time over the next month or so. -JTok

@jonathanm Are you booting directly to the vdi image, or did you convert it to another format first? I only ask because since I based this on the original script, it only looks for img and qcow2 files when backing up virtual disks. In hindsight, this is rather fragile, so I think I'll try to switch over to parsing the VM configuration to get the attached disks and their extensions. (Note to self: Also include a blacklist variable so users can exclude extensions they don't want backed up like iso.) Thanks, JTok