giantkingsquid Posted March 9, 2017 Share Posted March 9, 2017 Thanks for the great script danioj, nice work For those wanting to reduce the size of the .img I modified the second rsync command, adding a zip and rm so that it's like this: rsync -av$rsync_dry_run_option "$disk" "$backup_location/$vm/$timestamp$new_disk" && zip -j "$backup_location/$vm/$timestamp".zip "$backup_location/$vm/$timestamp"* && rm "$backup_location/$vm/$timestamp"*.img "$backup_location/$vm/$timestamp"*.xml Not elegant but it seems to work ok. Quote Link to comment
jas0r Posted April 16, 2017 Share Posted April 16, 2017 This is a great script. Is there any way to only restart the VM if it as actually running before backup. Quote Link to comment
Woodpusherghd Posted June 4, 2017 Share Posted June 4, 2017 (edited) After successfully backing up my VM for months, suddenly the script failed. See attached log. I updated to the latest version. Has anything changed in the update that I need to modify in my script? Script attached. log.txt script.txt Edited June 4, 2017 by Woodpusherghd Quote Link to comment
Woodpusherghd Posted June 5, 2017 Share Posted June 5, 2017 4 hours ago, Woodpusherghd said: After successfully backing up my VM for months, suddenly the script failed. See attached log. I updated to the latest version. Has anything changed in the update that I need to modify in my script? Script attached. log.txt script.txt SOLVED I had left the date stamp on my VM image as I had restored t from a previous backup. Apparently the script had a problem with the time stamp. I renamed the image without the time stamp and the script ran properly. Quote Link to comment
deeks Posted July 15, 2017 Share Posted July 15, 2017 On 10/03/2017 at 0:10 AM, giantkingsquid said: Thanks for the great script danioj, nice work For those wanting to reduce the size of the .img I modified the second rsync command, adding a zip and rm so that it's like this: rsync -av$rsync_dry_run_option "$disk" "$backup_location/$vm/$timestamp$new_disk" && zip -j "$backup_location/$vm/$timestamp".zip "$backup_location/$vm/$timestamp"* && rm "$backup_location/$vm/$timestamp"*.img "$backup_location/$vm/$timestamp"*.xml Not elegant but it seems to work ok. Hi there - this works well but have tweaked it slightly to include the name of the virtual machine in the filename. rsync -av$rsync_dry_run_option "$disk" "$backup_location/$vm/$timestamp$new_disk" && zip -j "$backup_location/$vm/$timestamp$vm".zip "$backup_location/$vm/$timestamp"* && rm "$backup_location/$vm/$timestamp"*.img "$backup_location/$vm/$timestamp"*.xml 1 Quote Link to comment
deeks Posted July 15, 2017 Share Posted July 15, 2017 Also tweaked further to clean out old zip files in order to to fill up too much diskspace. Look for this piece of code (around line 1075): echo "information: finished attempt to backup "$vms_to_backup" to $backup_location." Then add this extra code directly beneath: # Send location where it wants to clean out the backed up files echo "information: cleaning out backups older than 2 days in location " $backup_location/$vm/ # Delete all files older than two 24hr periods find $backup_location/$vm/ -mtime +2 -exec rm -f {} \; I tested it with the -mmin command in stead of -mtime in order speed up testing. Quote Link to comment
deeks Posted September 4, 2017 Share Posted September 4, 2017 More fine tuning in the works ... Tidy up of old backups was somewhat too rigid as it cleans out all backups older than a predefined number of days. The new script should leave at least one backup in place. Fix a small personal niggle : changing the date/timestamp to european dateformat (yyyymmdd) since I keep mixing up months and days with US notation. First tests with small sample files went well. Will let the script run with production setting for a while and post back results in this thread. Quote Link to comment
deeks Posted September 13, 2017 Share Posted September 13, 2017 My tests with only one VM went fine. Be careful to check the results if you want to use the script with multiple VM's. I have set the script to run every night, even though the server will not be running 24x7 in practice. It takes about 1,5 hours to create a zipped backup of my VM plus xml file and always leaves at least one backupfile in the folder. I have added the the script for those interested. Feel free to comment and improve. script 2 Quote Link to comment
CJICantLie Posted October 20, 2017 Share Posted October 20, 2017 Has anyone modified this to work for backing up passed through physical drives into an img? I have been wishing to setup a script for shutting down a windows VM, DDing the drive into a vdisk, and restarting the VM. I am curious if you have considered having this check whether the VM is on a vdisk or a physical disk and change the backup approach depending(rsync of the img vs dd of the physical disk)? Quote Link to comment
DZMM Posted November 1, 2017 Share Posted November 1, 2017 has anyone cracked deleting old backups automatically? I'm tempted to add at the start of the script: #!/bin/bash rm -rf /mnt/user/backup/vmbackup/ #################################################################### # # # WARNING - PLEASE CONSIDER THIS A WORK IN PROGRESS. I HAVE TESTED # # IT ON MY SERVER AND THERE WAS NO ISSUE BUT THAT DOESNT MEAN IN # to delete my backup folder before new backups are created, but I'm wondering if there's a better solution? Quote Link to comment
Gary489 Posted January 7, 2018 Share Posted January 7, 2018 Is this still in the works or has it been made in to a plug in?. Would love this for my set up. might even buy some one a case of beer!! Quote Link to comment
Tbigs2011 Posted January 16, 2018 Share Posted January 16, 2018 On 9/13/2017 at 2:41 PM, deeks said: My tests with only one VM went fine. Be careful to check the results if you want to use the script with multiple VM's. I have set the script to run every night, even though the server will not be running 24x7 in practice. It takes about 1,5 hours to create a zipped backup of my VM plus xml file and always leaves at least one backupfile in the folder. I have added the the script for those interested. Feel free to comment and improve. script I wanted to say thanks to the OP, deeks and the others that helped make this script! It just what I was looking for. Works great! Quote Link to comment
Symon Posted February 1, 2018 Share Posted February 1, 2018 (edited) Thanks for your work, this script works perfectly. I changed it a little bit so I can disable backups for certain disks of a VM. In my case I didn't want to backup a disk where I store games on, as it would take forever and use a lot of space, but backup the rest of the VM. So with the following entry, a vdisk will not be backed up if it's named nobak.img. This part can be added after the .img / .qcow code around line 971. # ----------------------------------------------------------- # check if filename is "nobak.img" and if so don't create a backup of this vdisk # ----------------------------------------------------------- if [ "$new_disk" == "nobak.img" ]; then echo "warning: $disk of $vm is not backed up because it is named nobak.img" continue fi Edited February 1, 2018 by Symon Quote Link to comment
IamSpartacus Posted February 12, 2018 Share Posted February 12, 2018 How does one install this script? Trying via the WebGUI plugin install just throws errors. Quote Link to comment
Warrentheo Posted February 19, 2018 Share Posted February 19, 2018 (edited) I like this script, but I had to modify my copy because I have used Virt-Manager to switch my VM to using the VirtIO-SCSI driver instead, this allows for the discard="unmap" option to be set on the storage, making it self-shrinking... This makes the files sparse, but makes the VM backup take way longer than needed with this script... I used the information found here: https://serverfault.com/questions/665335/what-is-fastest-way-to-copy-a-sparse-file-what-method-results-in-the-smallest-f to modify my copy of the script to run significantly faster... Perhaps this script could be modified to either detect the "unmap" option, or allow for a customizable switch setting to change the sparse file copy procedure..? Also an option to use some minimal inline compression on the backup? gZip or PigZip? Edited February 19, 2018 by Warrentheo Quote Link to comment
Warrentheo Posted February 19, 2018 Share Posted February 19, 2018 On 2/1/2018 at 4:05 AM, Symon said: Thanks for your work, this script works perfectly. I changed it a little bit so I can disable backups for certain disks of a VM. In my case I didn't want to backup a disk where I store games on, as it would take forever and use a lot of space, but backup the rest of the VM. So with the following entry, a vdisk will not be backed up if it's named nobak.img. This part can be added after the .img / .qcow code around line 971. # ----------------------------------------------------------- # check if filename is "nobak.img" and if so don't create a backup of this vdisk # ----------------------------------------------------------- if [ "$new_disk" == "nobak.img" ]; then echo "warning: $disk of $vm is not backed up because it is named nobak.img" continue fi This is already part of the script, which I discovered by accident... Just name the image file with anything other then .img at the end... (It already skips the backup of the .iso files this way) On 2/12/2018 at 9:41 AM, IamSpartacus said: How does one install this script? Trying via the WebGUI plugin install just throws errors. This is not a plugin, just a shell script... Either run it manually from the command line, or watch these videos: Backing up and restoring VMs in unRAID manually Backing up VMs in unRAID automatically Quote Link to comment
DZMM Posted February 19, 2018 Share Posted February 19, 2018 48 minutes ago, Warrentheo said: I like this script, but I had to modify my copy because I have used Virt-Manager to switch my VM to using the VirtIO-SCSI driver instead, this allows for the discard="unmap" option to be set on the storage, making it self-shrinking... This makes the files sparse, but makes the VM backup take way longer than needed with this script... I used the information found here: https://serverfault.com/questions/665335/what-is-fastest-way-to-copy-a-sparse-file-what-method-results-in-the-smallest-f to modify my copy of the script to run significantly faster... Perhaps this script could be modified to either detect the "unmap" option, or allow for a customizable switch setting to change the sparse file copy procedure..? Also an option to use some minimal inline compression on the backup? gZip or PigZip? Could you share your modified script please as my 3 of my 4 VMs use the VirtIO-SCSI driver as well. Quote Link to comment
Warrentheo Posted February 19, 2018 Share Posted February 19, 2018 29 minutes ago, DZMM said: Could you share your modified script please as my 3 of my 4 VMs use the VirtIO-SCSI driver as well. Check out the options in that link, but since I have not had a chance yet to test my backups by making sure the reversing of the backup process remains functional I don't want to tell people what commands to use just yet (I am at the "Know just enough to be dangerous" portion of my self training)... But that link mentions almost every option you could hope for, and how effective it is... It also mentions that the rsync option this script defaults to is just about the worst option for sparse files though That said, a slow but guaranteed to work option might be the better choice for a mission critical backup file here... Your own situation will guide you... Quote Link to comment
Marv Posted March 13, 2018 Share Posted March 13, 2018 On 19.2.2018 at 6:18 PM, Warrentheo said: Check out the options in that link, but since I have not had a chance yet to test my backups by making sure the reversing of the backup process remains functional I don't want to tell people what commands to use just yet (I am at the "Know just enough to be dangerous" portion of my self training)... But that link mentions almost every option you could hope for, and how effective it is... It also mentions that the rsync option this script defaults to is just about the worst option for sparse files though That said, a slow but guaranteed to work option might be the better choice for a mission critical backup file here... Your own situation will guide you... Do you have an update on this? Does reversing a backup works for you? I have no idea what exactly to change in the script to backup my scsi Win10 vm faster with the options you linked. A short guide would be appreciated Quote Link to comment
JTok Posted March 13, 2018 Share Posted March 13, 2018 (edited) I have created a variation on the original script that uses cp for the initial copy and then uses rsync for subsequent copies. I did a whole lot of testing on my own system and I found that while dd was slightly faster than cp, it did not create sparse files very well. Rsync was so much slower that it was nearly pointless to run. It did sparse files about as well as dd though. All of my tests were from my SSD cache array to my storage array. I also found that using rsync to delta sync was about as fast as re-copying the whole file using cp/dd when using --inplace. However, since the inplace argument creates some risk, and I did want to keep one rolling backup, I have the script make a copy first. Effectively the script does this: First run: uses cp to create a sparse copy of your vdisk. It also makes a copy of your xml. Second run: BEFORE shutting down the VM it makes a copy of the backup from step one using cp. Then it shuts down the VM and uses rsync to perform a delta sync with the inplace argument. Subsequent runs: BEFORE shutting down the VM it uses rsync to perform a delta sync of the latest backup to the second copy with the inplace argument. Then it shuts down the VM and uses rsync to perform a delta sync with the inplace argument. I also changed the script to make a note of the state of the VM when it starts the backup so that it can restore the VM to that state. I have some VMs that I don't keep running, so I don't want them started after a backup. Paused VMs will just be left in the stopped state. I *think* the timestamps will still work with this, but it will just use the cp option for everything. I'm also pretty sure it will keep making them forever without clearing them out. I am working on fixing that. You can find the script here: https://github.com/JTok/unraid-vmbackup Please be sure to read the Change Log and To-Do List as they contain important notes about the state of the script currently. -JTok Edit 1: forgot to mention: the original script does not make sparse copies. Additional explanation: When you create a VM you choose a disk size. Let's say we pick 40GB. Then we install an OS. The OS is taking up 10GB of that 40GB, so the vdisk will say it is 40GB, but only take up 10GB of actual space on the storage disk. This is highly desirable with lots of VMs. However, the original script didn't use the --sparse argument with rsync so it would create a 40GB vdisk that took up 40GB of actual space on the storage disk even though 30GB of that was actually just zeros. So sparse files are basically files that the file system knows are smaller than they appear to be, because they don't have all the unnecessary zeros in them. Edit 2: I have tested restoring the files using cp from the shell to put them back and they do work. However, I have not tested restoring them via the shares directly. You have been warned. Edited March 13, 2018 by JTok grammar fixes Quote Link to comment
JTok Posted March 15, 2018 Share Posted March 15, 2018 (edited) I have updated the script on github to include the following changes: Moved variables close to the top to make them easier to access. Added the ability to skip specific vdisks. Restoring the VMs to their previous state can be overridden. Backing up xml and/or vdisks can be skipped. Number of days to keep backups can be set. Changed to use mmin instead of mtime because of issues with accuracy. see: https://unix.stackexchange.com/a/92351 Number of backups to keep can be set. Always checks for img and qcow2 files by default. Timestamps are always used except when the number of backups being kept is set to 1. Toggle for compressing backups. Option to compare backups to original file and retry once. A few other minor things. Currently I've tested everything as well as I can, but I can't guarantee everything will work, so please feel free to let me know if you find any issues. I've tried to make it easy and customizable as possible. You can find the script here: https://github.com/JTok/unraid-vmbackup -JTok Edited March 15, 2018 by JTok formatting error 1 Quote Link to comment
DZMM Posted March 15, 2018 Share Posted March 15, 2018 6 hours ago, JTok said: I have updated the script on github to include the following changes: Moved variables close to the top to make them easier to access. Added the ability to skip specific vdisks. Restoring the VMs to their previous state can be overridden. Backing up xml and/or vdisks can be skipped. Number of days to keep backups can be set. Changed to use mmin instead of mtime because of issues with accuracy. see: https://unix.stackexchange.com/a/92351 Number of backups to keep can be set. Always checks for img and qcow2 files by default. Timestamps are always used except when the number of backups being kept is set to 1. Toggle for compressing backups. Option to compare backups to original file and retry once. A few other minor things. Currently I've tested everything as well as I can, but I can't guarantee everything will work, so please feel free to let me know if you find any issues. I've tried to make it easy and customizable as possible. You can find the script here: https://github.com/JTok/unraid-vmbackup -JTok Thanks for this - going to try now. Having sparse backups plus control over the number of versions is excellent, as I have to remember every week or so to delete my old weekly backups. Quote Link to comment
JTok Posted March 18, 2018 Share Posted March 18, 2018 I have made a few updates to the script again. Added the option to disable delta syncs. Added option to only use rsync. Dry-runs can be used with the rsync_only option. Dates and times are now included in log messages. Log files can be generated (enabled by default) Error logs can be managed separately from regular logs. Option to choose the number of logs to keep. Log messages can be sent through unRAID notification system. As with the last update, I've tested as best I can, but I can't guarantee everything. The script can still be found here: https://github.com/JTok/unraid-vmbackup -JTok Quote Link to comment
PieQuest Posted March 23, 2018 Share Posted March 23, 2018 Hey, I'm not very good at writing scripts or anything. But as an avid and active user of your script could I request added support for backing up the nvram file(s) as well? Thanks for all your work! Quote Link to comment
JTok Posted March 23, 2018 Share Posted March 23, 2018 13 hours ago, PieQuest said: Hey, I'm not very good at writing scripts or anything. But as an avid and active user of your script could I request added support for backing up the nvram file(s) as well? Thanks for all your work! That should be easy enough to add. I'll add it to my to-do list. I'm planning on having the next version out sometime next week, and I think I should be able to get this into that release. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.