Jump to content
danioj

unraid-autovmbackup: automate backup of virtual machines in unRAID - v0.4

167 posts in this topic Last Reply

Recommended Posts

Thanks for the great script danioj, nice work :)

 

For those wanting to reduce the size of the .img I modified the second rsync command, adding a zip and rm so that it's like this:
 

rsync -av$rsync_dry_run_option "$disk" "$backup_location/$vm/$timestamp$new_disk" && zip -j "$backup_location/$vm/$timestamp".zip "$backup_location/$vm/$timestamp"* && rm "$backup_location/$vm/$timestamp"*.img "$backup_location/$vm/$timestamp"*.xml

 

Not elegant but it seems to work ok.

Share this post


Link to post

This is a great script. Is there any way to only restart the VM if it as actually running before backup. 

Share this post


Link to post

After successfully backing up my VM for months, suddenly the script failed. See attached log. I updated to the latest version.  Has anything changed in the update that I need to modify in my script? Script attached.

log.txt

script.txt

Edited by Woodpusherghd

Share this post


Link to post
4 hours ago, Woodpusherghd said:

After successfully backing up my VM for months, suddenly the script failed. See attached log. I updated to the latest version.  Has anything changed in the update that I need to modify in my script? Script attached.

log.txt

script.txt

SOLVED  I had left the date stamp on my VM image as I had restored t from a previous backup.  Apparently the script had a problem with the time stamp.  I renamed the image without the time stamp and the script ran properly.

Share this post


Link to post
On 10/03/2017 at 0:10 AM, giantkingsquid said:

Thanks for the great script danioj, nice work :)

 

For those wanting to reduce the size of the .img I modified the second rsync command, adding a zip and rm so that it's like this:
 


rsync -av$rsync_dry_run_option "$disk" "$backup_location/$vm/$timestamp$new_disk" && zip -j "$backup_location/$vm/$timestamp".zip "$backup_location/$vm/$timestamp"* && rm "$backup_location/$vm/$timestamp"*.img "$backup_location/$vm/$timestamp"*.xml

 

Not elegant but it seems to work ok.

 

 

Hi there - this works well but have tweaked it slightly to include the name of the virtual machine in the filename.

 

rsync -av$rsync_dry_run_option "$disk" "$backup_location/$vm/$timestamp$new_disk" && zip -j "$backup_location/$vm/$timestamp$vm".zip "$backup_location/$vm/$timestamp"* && rm "$backup_location/$vm/$timestamp"*.img "$backup_location/$vm/$timestamp"*.xml

 

Share this post


Link to post

Also tweaked further to clean out old zip files in order to to fill up too much diskspace.

 

Look for this piece of code (around line 1075):

echo "information: finished attempt to backup "$vms_to_backup" to $backup_location."

Then add this extra code directly beneath:

# Send location where it wants to clean out the backed up files
echo "information: cleaning out backups older than 2 days in location	" $backup_location/$vm/

# Delete all files older than two 24hr periods
find $backup_location/$vm/ -mtime +2 -exec rm -f {} \;

I tested it with the -mmin command in stead of  -mtime in order speed up testing.

 

Share this post


Link to post

More fine tuning in the works ...

  • Tidy up of old backups was somewhat too rigid as it cleans out all backups older than a predefined number of days. The new script should leave at least one backup in place.
  • Fix a small personal niggle : changing the date/timestamp to european dateformat (yyyymmdd) since I keep mixing up months and days with US notation.

First tests with small sample files went well. Will let the script run with production setting for a while and post back results in this thread.

Share this post


Link to post

My tests with only one VM went fine. Be careful to check the results if you want to use the script with multiple VM's. 

 

I have set the script to run every night, even though the server will not be running 24x7 in practice. It takes about 1,5 hours to create a zipped backup of my VM plus xml file and always leaves at least one backupfile in the folder.

 

I have added the the script for those interested. Feel free to comment and improve.

script

Share this post


Link to post

Has anyone modified this to work for backing up passed through physical drives into an img? 

I have been wishing to setup a script for shutting down a windows VM, DDing the drive into a vdisk, and restarting the VM. 

I am curious if you have considered having this check whether the VM is on a vdisk or a physical disk and change the backup approach depending(rsync of the img vs dd of the physical disk)?

Share this post


Link to post

has anyone cracked deleting old backups automatically?  I'm tempted to add at the start of the script:

 

#!/bin/bash

rm -rf /mnt/user/backup/vmbackup/

####################################################################
#								   #
# WARNING - PLEASE CONSIDER THIS A WORK IN PROGRESS. I HAVE TESTED #
# IT ON MY SERVER AND THERE WAS NO ISSUE  BUT THAT DOESNT MEAN IN  #

to delete my backup folder before new backups are created, but I'm wondering if there's a better solution?

Share this post


Link to post

Is this still in the works or has it been made in to a plug in?. Would love this for my set up. might even buy some one a case of beer!!

  

Share this post


Link to post
On 9/13/2017 at 2:41 PM, deeks said:

My tests with only one VM went fine. Be careful to check the results if you want to use the script with multiple VM's. 

 

I have set the script to run every night, even though the server will not be running 24x7 in practice. It takes about 1,5 hours to create a zipped backup of my VM plus xml file and always leaves at least one backupfile in the folder.

 

I have added the the script for those interested. Feel free to comment and improve.

script

I wanted to say thanks to the OP, deeks and the others that helped make this script! It just what I was looking for. Works great!

Share this post


Link to post

Thanks for your work, this script works perfectly.

I changed it a little bit so I can disable backups for certain disks of a VM. In my case I didn't want to backup a disk where I store games on, as it would take forever and use a lot of space, but backup the rest of the VM. So with the following entry, a vdisk will not be backed up if it's named nobak.img. This part can be added after the .img / .qcow code around line 971.

 

					# -----------------------------------------------------------
					# check if filename is "nobak.img" and if so don't create a backup of this vdisk
					# -----------------------------------------------------------

					
					if [ "$new_disk" == "nobak.img" ]; then

					
						echo "warning: $disk of $vm is not backed up because it is named nobak.img"

						
						continue

						
					fi

 

Edited by Symon

Share this post


Link to post

I like this script, but I had to modify my copy because I have used Virt-Manager to switch my VM to using the VirtIO-SCSI driver instead, this allows for the 

discard="unmap"

option to be set on the storage, making it self-shrinking...  This makes the files sparse, but makes the VM backup take way longer than needed with this script...  I used the information found here: https://serverfault.com/questions/665335/what-is-fastest-way-to-copy-a-sparse-file-what-method-results-in-the-smallest-f to modify my copy of the script to run significantly faster...

 

Perhaps this script could be modified to either detect the "unmap" option, or allow for a customizable switch setting to change the sparse file copy procedure..?

Also an option to use some minimal inline compression on the backup? gZip or PigZip?

Edited by Warrentheo

Share this post


Link to post
On 2/1/2018 at 4:05 AM, Symon said:

Thanks for your work, this script works perfectly.

I changed it a little bit so I can disable backups for certain disks of a VM. In my case I didn't want to backup a disk where I store games on, as it would take forever and use a lot of space, but backup the rest of the VM. So with the following entry, a vdisk will not be backed up if it's named nobak.img. This part can be added after the .img / .qcow code around line 971.

 


					# -----------------------------------------------------------
					# check if filename is "nobak.img" and if so don't create a backup of this vdisk
					# -----------------------------------------------------------

					
					if [ "$new_disk" == "nobak.img" ]; then

					
						echo "warning: $disk of $vm is not backed up because it is named nobak.img"

						
						continue

						
					fi

 

 

This is already part of the script, which I discovered by accident... Just name the image file with anything other then .img at the end...  (It already skips the backup of the .iso files this way)

 

On 2/12/2018 at 9:41 AM, IamSpartacus said:

How does one install this script?  Trying via the WebGUI plugin install just throws errors.

 

This is not a plugin, just a shell script...  Either run it manually from the command line, or watch these videos:

Backing up and restoring VMs in unRAID manually

Backing up VMs in unRAID automatically

Share this post


Link to post
48 minutes ago, Warrentheo said:

I like this script, but I had to modify my copy because I have used Virt-Manager to switch my VM to using the VirtIO-SCSI driver instead, this allows for the 


discard="unmap"

option to be set on the storage, making it self-shrinking...  This makes the files sparse, but makes the VM backup take way longer than needed with this script...  I used the information found here: https://serverfault.com/questions/665335/what-is-fastest-way-to-copy-a-sparse-file-what-method-results-in-the-smallest-f to modify my copy of the script to run significantly faster...

 

Perhaps this script could be modified to either detect the "unmap" option, or allow for a customizable switch setting to change the sparse file copy procedure..?

Also an option to use some minimal inline compression on the backup? gZip or PigZip?

Could you share your modified script please as my 3 of my 4 VMs use the VirtIO-SCSI driver as well.

Share this post


Link to post
29 minutes ago, DZMM said:

Could you share your modified script please as my 3 of my 4 VMs use the VirtIO-SCSI driver as well.

Check out the options in that link, but since I have not had a chance yet to test my backups by making sure the reversing of the backup process remains functional I don't want to tell people what commands to use just yet (I am at the "Know just enough to be dangerous" portion of my self training)...  But that link mentions almost every option you could hope for, and how effective it is...  It also mentions that the rsync option this script defaults to is just about the worst option for sparse files though :$  That said, a slow but guaranteed to work option might be the better choice for a mission critical backup file here...  Your own situation will guide you...

Share this post


Link to post
On 19.2.2018 at 6:18 PM, Warrentheo said:

Check out the options in that link, but since I have not had a chance yet to test my backups by making sure the reversing of the backup process remains functional I don't want to tell people what commands to use just yet (I am at the "Know just enough to be dangerous" portion of my self training)...  But that link mentions almost every option you could hope for, and how effective it is...  It also mentions that the rsync option this script defaults to is just about the worst option for sparse files though :$  That said, a slow but guaranteed to work option might be the better choice for a mission critical backup file here...  Your own situation will guide you...

 

Do you have an update on this? Does reversing a backup works for you?

I have no idea what exactly to change in the script to backup my scsi Win10 vm faster with the options you linked.

A short guide would be appreciated :)

Share this post


Link to post

I have created a variation on the original script that uses cp for the initial copy and then uses rsync for subsequent copies.

 

I did a whole lot of testing on my own system and I found that while dd was slightly faster than cp, it did not create sparse files very well. Rsync was so much slower that it was nearly pointless to run. It did sparse files about as well as dd though. All of my tests were from my SSD cache array to my storage array.

I also found that using rsync to delta sync was about as fast as re-copying the whole file using cp/dd when using --inplace. However, since the inplace argument creates some risk, and I did want to keep one rolling backup, I have the script make a copy first. Effectively the script does this:

 

First run:

uses cp to create a sparse copy of your vdisk. It also makes a copy of your xml.

 

Second run:

BEFORE shutting down the VM it makes a copy of the backup from step one using cp.

Then it shuts down the VM and uses rsync to perform a delta sync with the inplace argument.

 

Subsequent runs:

BEFORE shutting down the VM it uses rsync to perform a delta sync of the latest backup to the second copy with the inplace argument.

Then it shuts down the VM and uses rsync to perform a delta sync with the inplace argument.

 

 

I also changed the script to make a note of the state of the VM when it starts the backup so that it can restore the VM to that state. I have some VMs that I don't keep running, so I don't want them started after a backup. Paused VMs will just be left in the stopped state.

 

I *think* the timestamps will still work with this, but it will just use the cp option for everything. I'm also pretty sure it will keep making them forever without clearing them out. I am working on fixing that.

 

You can find the script here: https://github.com/JTok/unraid-vmbackup


Please be sure to read the Change Log and To-Do List as they contain important notes about the state of the script currently.

 

-JTok

 

Edit 1:

forgot to mention: the original script does not make sparse copies.

Additional explanation:

When you create a VM you choose a disk size. Let's say we pick 40GB. Then we install an OS. The OS is taking up 10GB of that 40GB, so the vdisk will say it is 40GB, but only take up 10GB of actual space on the storage disk. This is highly desirable with lots of VMs. However, the original script didn't use the --sparse argument with rsync so it would create a 40GB vdisk that took up 40GB of actual space on the storage disk even though 30GB of that was actually just zeros. So sparse files are basically files that the file system knows are smaller than they appear to be, because they don't have all the unnecessary zeros in them.

 

Edit 2:

I have tested restoring the files using cp from the shell to put them back and they do work. However, I have not tested restoring them via the shares directly. You have been warned.

Edited by JTok
grammar fixes

Share this post


Link to post

I have updated the script on github to include the following changes:

 

  • Moved variables close to the top to make them easier to access.
  • Added the ability to skip specific vdisks.
  • Restoring the VMs to their previous state can be overridden.
  • Backing up xml and/or vdisks can be skipped.
  • Number of days to keep backups can be set.
  • Number of backups to keep can be set.
  • Always checks for img and qcow2 files by default.
  • Timestamps are always used except when the number of backups being kept is set to 1.
  • Toggle for compressing backups.
  • Option to compare backups to original file and retry once.
  • A few other minor things.

 

Currently I've tested everything as well as I can, but I can't guarantee everything will work, so please feel free to let me know if you find any issues. I've tried to make it easy and customizable as possible.

 

 

You can find the script here: https://github.com/JTok/unraid-vmbackup

 

-JTok

 

Edited by JTok
formatting error

Share this post


Link to post
6 hours ago, JTok said:

I have updated the script on github to include the following changes:

 

  • Moved variables close to the top to make them easier to access.
  • Added the ability to skip specific vdisks.
  • Restoring the VMs to their previous state can be overridden.
  • Backing up xml and/or vdisks can be skipped.
  • Number of days to keep backups can be set.
  • Number of backups to keep can be set.
  • Always checks for img and qcow2 files by default.
  • Timestamps are always used except when the number of backups being kept is set to 1.
  • Toggle for compressing backups.
  • Option to compare backups to original file and retry once.
  • A few other minor things.

 

Currently I've tested everything as well as I can, but I can't guarantee everything will work, so please feel free to let me know if you find any issues. I've tried to make it easy and customizable as possible.

 

 

You can find the script here: https://github.com/JTok/unraid-vmbackup

 

-JTok

 

Thanks for this - going to try now.  Having sparse backups plus control over the number of versions is excellent, as I have to remember every week or so to delete my old weekly backups.

Share this post


Link to post

I have made a few updates to the script again.

 

  • Added the option to disable delta syncs.
  • Added option to only use rsync.
    • Dry-runs can be used with the rsync_only option.
  • Dates and times are now included in log messages.
  • Log files can be generated (enabled by default)
    • Error logs can be managed separately from regular logs.
    • Option to choose the number of logs to keep.
  • Log messages can be sent through unRAID notification system.

 

As with the last update, I've tested as best I can, but I can't guarantee everything. 

 

The script can still be found here: https://github.com/JTok/unraid-vmbackup

 

-JTok

 

Share this post


Link to post

Hey, I'm not very good at writing scripts or anything. But as an avid and active user of your script could I request added support for backing up the nvram file(s) as well?

 

Thanks for all your work!

Share this post


Link to post
13 hours ago, PieQuest said:

Hey, I'm not very good at writing scripts or anything. But as an avid and active user of your script could I request added support for backing up the nvram file(s) as well?

 

Thanks for all your work!

 

That should be easy enough to add. I'll add it to my to-do list. I'm planning on having the next version out sometime next week, and I think I should be able to get this into that release.

 

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.