danioj Posted March 30, 2016 Share Posted March 30, 2016 WARNING - PLEASE CONSIDER THIS A WORK IN PROGRESS. I HAVE TESTED IT ON MY SERVER AND THERE WAS NO ISSUE BUT THAT DOESNT MEAN IN ANY WAY IT IS FREE FROM BUGS/ISSUS SO PLEASE USE AT YOUR OWN RISK UNTIL IT HAS BEEN TESTED FURTHER - WARNING Well, this is what happens when I am in hospital with nothing to do but wait for tests. For those who don't like what I have done, then I am sorry! Anyway, I was discussing the need for backup of Virtual Machine vdisks. I certainly need a method that is not manual and I thought I would write a "Proof of Concept" script to do this for us to see if it might be something that would be needed by others too. Like I said above, this is a proof of concept initially and it is missing most of the refinements that go into good code and a released product. For that reason I would suggest if you don't know what you're doing then don't use this. If there are people who want to test this in their test environment and or give feedback please do the following: - download script from github (link at bottom of the post). - copy the script to your flash drive. - read the notes in the file. - fill in the mandatory inputs (e.g. virtual machine list and backup location). - change optional switches if you wish to. - from the cli execute it like below (for now). sh backup_unraid_vms.sh - write a post and give feedback # this script has been created to automate the backup of unRAID vm's vdisks(s) to a specified location. # this script does not yet run using variables passed from the cli as yet but there is intention to do so if there is interest. # for now the variables below are what are needed to be ammended to have the script run. # Change Log # v0.1 20160330. # initial release. # # notes: initial realease is more concept and to see if there is a need for it. it lacks basic error handling and recovery. # which could be developed later. # v0.2 20160330. # bug fixes. # # changed how the bash script is recognising the failed shutdown flag as after testing even a failed shutdown would start a backup. # changed default mins to wait for shutdown to 6. # v0.3 20160415. # bug fixes and enhancements. # # applied strickter naming conventions to variables. # cleaned up the code, added comments, removed unecessary 'do' loops. # added input validation and verification. # added addition status messages. applied a bit more consistency. # added ability to deal with vm's with names that have spaces in them. # vms now seperated by "new line" not space. # added rsync copy over standard nix cp command. # added backup of vm xml configuration. # added option to add timestamps to backup files. # added script name check for version control. # added option to enable / disable the script. # added option to start vm after failure. # added option to "dry-run" copy operation - note all other functionality is enabled. # added ability to ignore vm's which are not installed on the system. # set defaults for all options to 0. # added guidence for options and inputs. # added constraint to only backup .img vdisks so to skip installation media etc. # changed method of obtaining vdisks list. # fixed issue which had script fail if no vdisks were present. # v0.4 20160416. # administration. # # script name changed to facilitate code being added to github. # bug tracker (order by severity) # <reference> <description> <link to post> <severity> <date added> <by whome> <accepted/refected> <comment> # to do list # core # error capturing and handling -- started (v0.3). # apply stricter naming conventions -- done (v0.3). # code clean up -- ongoing (v0.3) # input validation and verification -- done (v0.3). # logging. # clean up status messages -- done (v0.3). # deal with vm's named with spaces -- done (v0.3). # possible improvements # use rsync to copy the vdisks instead of copy -- done (v0.3). # possible future features # backup vm xml file -- done (v0.3). # have the script run from variables passed in from the cli. # add timestamps to files -- done (v0.3). # add iterations of backups and number of backups to maintain at any given time. # plugin -- started (v0.3). # plugin to-do's # basic structure and requirements -- done (v0.3) # git-hub account # ui validation via menus # .... much much more .... script now available via github: https://github.com/danioj/unraid-autovmbackup.git 1 Quote Link to comment
stchas Posted March 31, 2016 Share Posted March 31, 2016 Thanks, danioj. I'll test over the weekend. Quote Link to comment
danioj Posted March 31, 2016 Author Share Posted March 31, 2016 Thanks, danioj. I'll test over the weekend. Thanks. Let me know what you think. I tested it over and over again when I had telnet access in the hospital and if the inputs are similar to as I have put in the example it seems to work just fine. If there is enough interest in it I shall tighten it up. Quote Link to comment
stchas Posted April 1, 2016 Share Posted April 1, 2016 I was not able to get the script to run successfully because of a couple of things specific to my setup, one being the name of my VM has spaces in it, and the other that I had my VM shutdown prior to running the script. Here's the play-by-play: root@Tower:/boot/custom/bin# backup_unraid_vms.sh starting backup_unraid_vms.sh to backup Win7 Virtual Desktop to /mnt/user/System/Win7-VM starting backup of Win7 checking /mnt/user/System/Win7-VM/Win7 exists so i have somewhere to put the backup files /mnt/user/System/Win7-VM/Win7 doesnt exist so i am going to create it /mnt/user/System/Win7-VM/Win7 has been created backup of Win7 failed completed backup_unraid_vms.sh to backup Win7 Virtual Desktop to /mnt/user/System/Win7-VM starting backup of Virtual checking /mnt/user/System/Win7-VM/Virtual exists so i have somewhere to put the backup files /mnt/user/System/Win7-VM/Virtual doesnt exist so i am going to create it /mnt/user/System/Win7-VM/Virtual has been created backup of Virtual failed completed backup_unraid_vms.sh to backup Win7 Virtual Desktop to /mnt/user/System/Win7-VM starting backup of Desktop checking /mnt/user/System/Win7-VM/Desktop exists so i have somewhere to put the backup files /mnt/user/System/Win7-VM/Desktop doesnt exist so i am going to create it /mnt/user/System/Win7-VM/Desktop has been created backup of Desktop failed completed backup_unraid_vms.sh to backup Win7 Virtual Desktop to /mnt/user/System/Win7-VM Things to fix: 1. It needs to handle VM names with embedded spaces. My VM's name is "Win7 Virtual Desktop." When the script parses vms_to_backup="Win7 Virtual Desktop," it identifies VMs with names "Win7," "Virtual," and "Desktop" and creates subfolders for those backup vdisks. 2. It needs to trap invalid VM names (already on your list), such as what occurred in item 1 above (e.g., "Win7" is not a valid VM name). 3. I was not able to confirm, but I don't think it will back up vdisks for VMs that are already shutdown when the script starts. When you define the vm_state variable, use "virsh list --all" instead of "virsh list." Then add a state check for VM already shutdown. For this condition, you'll have to decide whether to restart the VM (my vote would be "no"). if [ "$vm_state" == "$vm shut off" ]; then echo "$vm is already in shutdown state" can_backup_vm="y" fi I really appreciate the work you've done putting this script together. I expect it will get a lot of use as people discover how easy it is to use and how effectively it fills the need for backing up VMs. Quote Link to comment
Bugman1400 Posted April 2, 2016 Share Posted April 2, 2016 Why do the VMs have to be shut down to copy the vdisk1.img? What happens if you try to copy it without shutting it down......will it affect the VM startup when you try to restore with copied vdisk1.img? Quote Link to comment
JonathanM Posted April 2, 2016 Share Posted April 2, 2016 What happens if you try to copy it without shutting it down......will it affect the VM startup when you try to restore with copied vdisk1.img? Short answer? It depends. May work, may not work. In general, it's not worth the risk of a backup not working. Why do you want to back it up while it's running? Quote Link to comment
stchas Posted April 2, 2016 Share Posted April 2, 2016 Why do the VMs have to be shut down to copy the vdisk1.img? For a Windows VM, for example, I want to make sure Windows shuts down to force it to flush any volatile memory and disk buffers to the vdisk prior to making the backup. Quote Link to comment
Bugman1400 Posted April 2, 2016 Share Posted April 2, 2016 Is it easy to turn this into a plugin so we don't have mess with the flash? Seems like any improvements that are made or features that added would be handled well through the update process with the plugins. Or, would setting up a plugin be too much effort? Quote Link to comment
danioj Posted April 2, 2016 Author Share Posted April 2, 2016 Why do the VMs have to be shut down to copy the vdisk1.img? What happens if you try to copy it without shutting it down......will it affect the VM startup when you try to restore with copied vdisk1.img? What happens if you try to copy it without shutting it down......will it affect the VM startup when you try to restore with copied vdisk1.img? Short answer? It depends. May work, may not work. In general, it's not worth the risk of a backup not working. Why do you want to back it up while it's running? To add to what jonathanm said: Copying the image files is ALWAYS ok if the guest virtual machines are stopped. But if they are running its possible that as the copy is taking place changes are made within the guest, leading to an inconsistent copy. Quote Link to comment
danioj Posted April 2, 2016 Author Share Posted April 2, 2016 I was not able to get the script to run successfully because of a couple of things specific to my setup, one being the name of my VM has spaces in it, and the other that I had my VM shutdown prior to running the script. Here's the play-by-play: Thanks for trying it out. It's good to captures as many individual setups as possible. Things to fix: 1. It needs to handle VM names with embedded spaces. My VM's name is "Win7 Virtual Desktop." When the script parses vms_to_backup="Win7 Virtual Desktop," it identifies VMs with names "Win7," "Virtual," and "Desktop" and creates subfolders for those backup vdisks. Agreed. 2. It needs to trap invalid VM names (already on your list), such as what occurred in item 1 above (e.g., "Win7" is not a valid VM name). Agreed. 3. I was not able to confirm, but I don't think it will back up vdisks for VMs that are already shutdown when the script starts. When you define the vm_state variable, use "virsh list --all" instead of "virsh list." Then add a state check for VM already shutdown. For this condition, you'll have to decide whether to restart the VM (my vote would be "no"). if [ "$vm_state" == "$vm shut off" ]; then echo "$vm is already in shutdown state" can_backup_vm="y" fi I will have to check this again because for me, when the VM's were shutdown it worked perfectly. Is it easy to turn this into a plugin so we don't have mess with the flash? Seems like any improvements that are made or features that added would be handled well through the update process with the plugins. Or, would setting up a plugin be too much effort? I am already looking at a plugin for another script I wrote. I have been in hospital this week but am working on migrating this into a plugin already. Even if others don't want it, I want it for myself Quote Link to comment
Bugman1400 Posted April 3, 2016 Share Posted April 3, 2016 Sounds like a plan......looking forward to the plugin. Quote Link to comment
Bugman1400 Posted April 8, 2016 Share Posted April 8, 2016 Dan, Hope you're out of the hospital by now and things went well. Feeling back in action yet? Cheers. Quote Link to comment
Bugman1400 Posted April 15, 2016 Share Posted April 15, 2016 Uhhh....you okay Dan? Quote Link to comment
danioj Posted April 15, 2016 Author Share Posted April 15, 2016 Thanks for the kind words. I'm recovering. Made some in-roads into this while bed ridden though!! Quote Link to comment
danioj Posted April 15, 2016 Author Share Posted April 15, 2016 Dan, Hope you're out of the hospital by now and things went well. Feeling back in action yet? Cheers. Well, this is the first day I have actually had an hour without any pain. Anyway, as a result I took the opportunity to test the updates I made to the script while I was in hospital living off ice-cream! This is a VERY significant update. Having all that time made me realise just how much was wrong with the damn thing before. I have made many changes. I had to remove the code from the forum now as there is too much for a post, but the file is of course attached. Have a look at the change log. Any comments are welcome. After testing myself I feel it is getting to a point where it might be getting good enough for a plugin. As I mentioned earlier, I am pursuing that now! Quote Link to comment
CHBMB Posted April 15, 2016 Share Posted April 15, 2016 Dan, you using github? Probably the best place for the code.... Sent from my LG-H815 using Tapatalk Quote Link to comment
danioj Posted April 15, 2016 Author Share Posted April 15, 2016 Dan, you using github? Probably the best place for the code.... Sent from my LG-H815 using Tapatalk Good pointer. Done. Much better. Quote Link to comment
danioj Posted April 16, 2016 Author Share Posted April 16, 2016 How about a link? It's on the bottom of the first post. Quote Link to comment
Bugman1400 Posted April 22, 2016 Share Posted April 22, 2016 Any status on the plug-in? I'm interested in learning the process of making a plugin anyhow. Is there a noobs tutorial for this? Quote Link to comment
Bugman1400 Posted April 29, 2016 Share Posted April 29, 2016 Dan, is your script supposed to run automatically every x days? How do you set this up? Quote Link to comment
AcidRaZor Posted May 2, 2016 Share Posted May 2, 2016 Thanks for this! Where do I store the script on my flash? Anywhere? Quote Link to comment
JM2005 Posted August 4, 2016 Share Posted August 4, 2016 Any news on the Plug-In? Thanks for your hard work! Quote Link to comment
Squid Posted August 14, 2016 Share Posted August 14, 2016 Any news on the Plug-In? Thanks for your hard work! Right now, you could easily schedule this to run whenever you want by using the user scripts plugin, or alternatively set this script to run in conjunction with appdata backup with CA. Quote Link to comment
bungee91 Posted August 14, 2016 Share Posted August 14, 2016 run in conjunction with appdata backup with CA. Is there any chance of adding VM backup natively into CA's backup function? I think this would be the most streamlined way while we await (at some point) getting snapshot's added into the VM manager. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.