Jump to content
danioj

unraid-autovmbackup: automate backup of virtual machines in unRAID - v0.4

167 posts in this topic Last Reply

Recommended Posts

 
2) Run the script with the vdisks_to_skip field filled out for one of the vdisks so that only one gets backed up. Then rename the one vdisk you have a backup of and change the VM configuration.


Cheers JTok, went with option two and it seems to be working as intended now.



Sent from my iPhone using Tapatalk

Share this post


Link to post

A bug in your script just saved my bacon!  My pfsense VM crashed and got corrupted and I'd lost all my config backups as I think they were in the same folder where the xml backup is done.

 

Anyway, I luckily spotted that the script had backed up the pfsense VM image file even though I had set it not to!

 

# list of specific vdisks to be skipped. use the full path.
vdisks_to_skip="
/mnt/disks/sm961/domains/Woody/vdisk1.img
"

I didn't want to backup the image as I didn't want my whole network to go down and 'normally' restoring via the config is easy.  Anyway, thanks for the bug as it saved my life.

 

I've now made multiple copies of my config so please feel free to fix the bug - logs attached ;-)

 

20180528_0200_unraid-vmbackup.log

Edited by DZMM

Share this post


Link to post

Hello folks, 

 

I have been using this to back up a VM, and a recent power outage corrupted the VM.

 

I copied over the img, but that was insufficient to restore the VM.

I assume this means the configuration is damaged, and while I believe that is part of the backup (the xml file?) I am not sure the proper way to utilize that in the restore process.

 

Can someone guide me in properly restoring from the backup?

Share this post


Link to post

Is there a way to shutdown OSX VMs by the script (or remotely in general)? 

as far as I know I cant right now.  i need to cleanly shutdown manually from within or schedule a shutdown just before the script runs.

 

Share this post


Link to post
Is there a way to shutdown OSX VMs by the script (or remotely in general)? 
as far as I know I cant right now.  i need to cleanly shutdown manually from within or schedule a shutdown just before the script runs.
 
Not sure what you're talking about. When I run this script (with the proper settings)....each of my VMs automatically shut down before they backup in sequence.

Recheck your settings in the script....there are MANY settings

Share this post


Link to post

What I mean is that the Windows based VMs do shutdown.  the Apple OS HighSierra and even before , can not. its not working if I try to stop the VM. the only way is to manually shutdown from within the VM (inside the OSX). 

Share this post


Link to post
2 minutes ago, dadarara said:

What I mean is that the Windows based VMs do shutdown.  the Apple OS HighSierra and even before , can not. its not working if I try to stop the VM. the only way is to manually shutdown from within the VM (inside the OSX). 

No idea. I don't use Mac OS. You can try using the force shutdown option in the settings. Other than that, I dunno. Good luck.

Share this post


Link to post

Force shutdown is exactly what he is complaining about. It will be forced after some time if the clean shutdown isn't working. Except from a sheduled shutdown inside the VM i have no idea. I quickly tested it and it doesn't work for me either. Sorry :(

Share this post


Link to post

First thank you for a good script!

 

I hope it is OK to suggest improvements. :) I noticed that with time stamp my backup location quickly ran into space issues. So my suggestion is to add some sort of clean-up before backing up. I have done some google search myself and found this command:

find ./{backup location} -mtime +{days} -type f -delete

Now I am far from being able to edit this script. But it has not held me back from trying to analyse it a bit. But to me it seems like backup location can re-use the backup location already needed to be specified in the script. The amount of days a backup should be kept safe (not deleted) looks like it could be set as a variable too. I don't know if it is possible to use a derivate of this script to only check files backed-up by the script, or maybe just xml and vdi files. Again I am not an expert here. But maybe one out here with the skills can and want to take a look at it? :)

 

/Alphahelix

Share this post


Link to post

Ok I think I have found a way to have the script delete old files. Please correct me if I have pasted the addition a wrong place, or if there is a better way to do this. But here it is:

 

 

autkvmbackup-with deletion.sh

Edited by Alphahelix

Share this post


Link to post
On 10/6/2018 at 8:46 PM, Alphahelix said:

Ok I think I have found a way to have the script delete old files. Please correct me if I have pasted the addition a wrong place, or if there is a better way to do this. But here it is:

 

 

autkvmbackup-with deletion.sh

There's already a variable to cover this

 

Edit: actually 2

 

# default is 0. set this to the number of days backups should be kept. 0 means indefinitely.
number_of_days_to_keep_backups="0"
# default is 0. set this to the number of backups that should be kept. 0 means infinitely.
# WARNING: If VM has multiple vdisks, then they must end in sequential numbers in order to be correctly backed up (i.e. vdisk1.img, vdisk2.img, etc.).
number_of_backups_to_keep="2"

 

Edited by DZMM

Share this post


Link to post
On 11/24/2018 at 6:03 PM, DZMM said:

There's already a variable to cover this

 

Edit: actually 2

 


# default is 0. set this to the number of days backups should be kept. 0 means indefinitely.
number_of_days_to_keep_backups="0"

# default is 0. set this to the number of backups that should be kept. 0 means infinitely.
# WARNING: If VM has multiple vdisks, then they must end in sequential numbers in order to be correctly backed up (i.e. vdisk1.img, vdisk2.img, etc.).
number_of_backups_to_keep="2"

 

I don't seem to be able to find them... I even tried to download the script file again. from https://github.com/danioj/unraid-autovmbackup/blob/master/unraid-autovmbackup-0-4.sh But I could have overseen er newer revision than the 3 year old one from the link.

Share this post


Link to post
On 5/31/2018 at 11:45 AM, DZMM said:

A bug in your script just saved my bacon!  My pfsense VM crashed and got corrupted and I'd lost all my config backups as I think they were in the same folder where the xml backup is done.

 

Anyway, I luckily spotted that the script had backed up the pfsense VM image file even though I had set it not to!

 


# list of specific vdisks to be skipped. use the full path.
vdisks_to_skip="
/mnt/disks/sm961/domains/Woody/vdisk1.img
"

I didn't want to backup the image as I didn't want my whole network to go down and 'normally' restoring via the config is easy.  Anyway, thanks for the bug as it saved my life.

 

I've now made multiple copies of my config so please feel free to fix the bug - logs attached ;-)

 

20180528_0200_unraid-vmbackup.log

 

This is still broken from what i can tell, is this still being maintained?

 

 

Share this post


Link to post
20 minutes ago, Lebowski said:

 

This is still broken from what i can tell, is this still being maintained?

 

 

Nah. I did this and got really sick. My priorities have changed and tbh I have abandoned this. It was a good idea when i had time but even I back up manually now.

Share this post


Link to post

So... I honestly I haven't looked at this for more than a few months since it has been working for my needs and my personal life has been rather busy.
However, I did start working on (and am planning to eventually finish) some updates to this script as well as a version that works with btrfs snapshots.
Things have slowed down a little lately, so I've started getting back into these kind of projects. I will take another look at the vdisk skipping, plus I'm working on adding the ability to parse the vdisk names directly from the xml file as well as a few other things.
Unfortunately I don't have a firm timeline right now, but I expect to be able to devote a decent amount of time over the next month or so.

 

-JTok

Share this post


Link to post

Hi JTok

 

First of all big thnx for the script. Im using your script and want to add a feature which allows to backup running vm's. From just a few tests it seems to work but i have to adjust your script for it to be compatible with your bultin functions like stopping the vm before backup and starting it after. For now i outcommented and added some lines of code to make it work.

 

Backup running vm's: https://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit

 

Share this post


Link to post
2 hours ago, Dikkekop said:

Hi JTok

 

First of all big thnx for the script. Im using your script and want to add a feature which allows to backup running vm's. From just a few tests it seems to work but i have to adjust your script for it to be compatible with your bultin functions like stopping the vm before backup and starting it after. For now i outcommented and added some lines of code to make it work.

 

Backup running vm's: https://wiki.libvirt.org/page/Live-disk-backup-with-active-blockcommit

 

Wow....I always thought it was only possible to safely/correctly backup VMs if they are shutdown. Your method is safe!?

Share this post


Link to post

Don't know yet its stil a w.i.p. and needs testing. From what i understand its safe if guest additions are installed in a vm: NOTE-1: Above, if you have QEMU guest agent installed in your virtual machine, try '--quiesce' option with virsh snapshot-create-as[. . .] to ensure you have a consistent disk state.

 

Share this post


Link to post
9 hours ago, Stupifier said:

Wow....I always thought it was only possible to safely/correctly backup VMs if they are shutdown. Your method is safe!?

It should correctly backup the DISK properly and safely, but the running OS will not be given the chance to commit any pending changes.

 

Picture this scenario. You have a document open for editing in the VM. The OS has the file open for writing, but keeps your edits in RAM for speed. The VM is backed up, your open file is backed up in the open pending state. You use this backup to restore, and your document is corrupted because it is restored as being open but no application is going to close it properly.

 

Now, what happens if that same thing happens to critical system files? It may work well 95% of the time, but backing up a running OS without the OS itself being informed or involved in the backup is not safe. If the OS is aware and can prepare properly, it will work fine.

 

4 hours ago, Dikkekop said:

From what i understand its safe if guest additions are installed in a vm:

 

Do you have a list of supported OS versions?

Share this post


Link to post

This is an w.i.p. i dont have an answer for that.

 

About your "document open for editing".

 

Correct me if i'm wrong

Word creates a temp file for editing leaving the original file as it is (maby its marked or something so no other user can edit the file while its open..but i dont think this "mark" is saved in the file itself). When you hit the save button its writes the changes to file. So if you backup while file is open it saves the original file as it is before editing and before saving. If you save the file after backup is done, the next backup wil contain changes made while file was open. I wil test this.....

 

the --quiesce option can be used if guest additions are installed inside the vm and will flush the memory buffers to disk. But i have to test this... again W.I.P.

This is al theory.. i want to test things first..

 

 

Share this post


Link to post

Oke i have done some testing with libvirt commandline tool virsh... and made a basic script.... seems to work. Almost done merging this with: https://github.com/JTok/unraid-vmbackup Version v1.1.4 - 2018/05/19. So the orginal script is still functional with all its parameters/functionality and i will add one to enable "Live backup". Just to be clear this wil not include current RAM state into backup. This will backup only the disk at the current state, it will flush latest i/o to disk before snapshot/backup to get a consistent disk state (requires guest agent inside vm, script will check if it is installed..). If VM is restored it will act like its been forced shutdown.

 

Tested:

- backup of windows server 2016 is restorable, also tested an open word file wich does after a restore is still accesable.

- backup of an debian vm is restorable while running dd if=/dev/zero of=/root/testfile bs=1G count=1 oflag=direct wich creates a 1 gb file. libvirt is waiting for it (if guest agent is installed inside vm) to finish before creating snapshot/backup so this is really nice.

 

I will post updated script i a couple of days and would like to ask you guys to test. Maby do not test this on your critical vm's/unraid box but thats up to you

 

I will add marks in scipt to mark what i added or edited.

 

 

 

 

 

 

 

 

 

 

 

 

 

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.