Is there a way to suspend the VM's and make a backup to the array weekly? 

 

I know I can do something if I create a cron job to shutdown and copy it weekly,  but I would like to have the VMs suspend instead of shutdown.

 

Thanks

I think that would be a good idea too. 

Is there a way to suspend the VM's and make a backup to the array weekly? 

 

I know I can do something if I create a cron job to shutdown and copy it weekly,  but I would like to have the VMs suspend instead of shutdown.

 

Thanks

I found this site.  Not sure if it would work for us.  It just looks like instead of using

xm pause myVMOS

we would use

xl pause myVMOS

then

xl resume myVMOS

Backupscript for xen machines.

 

#!/usr/bin/bash

vmHome="/mnt/cache/VM"
backupHome="/mnt/user/VMBackup"
logHome="/mnt/user/VMBackup"

xl list | tail -n+3 | cut -d ' ' -f 1 | while read line; do
        xl pause $line
        rsync -av --progress --"log-file=$logHome/$(date +%Y%m%d)_backup.log" $vmHome/$line $backup$
        xl unpause $line

done

 

Its not perfect, more of a example of how you can do it.

How long the VM becomes "Unavailable" during the backup ?  I guess it depends on the size of the VM.  Is there something like Snapshots in Xen VM world ?  In VMware, I think you can do a Snapshot and backup that snapshot while the VM is still running.

How long the VM becomes "Unavailable" during the backup ?  I guess it depends on the size of the VM.  Is there something like Snapshots in Xen VM world ?  In VMware, I think you can do a Snapshot and backup that snapshot while the VM is still running.

Snapshots in ESXi work fine as long as you do not do pass through of PCI devices.  But the minute you pass through a PCI/PCIe device snapshotting is turned OFF for that VM.

The best thing you can do to speed up your backup is not not back up to the same drive. If your VM's are hosted outside of your RAID and not on the cache drive, you can copy your VM's to the cache drive for minimal downtime and then let the mover put the backed up files into the array protected drives later.

 

It would be best to shut down the VM prior to backing it up so that the backup is in a clean state.

With a combination of rsync and using nfs to mount your home directorys (at least in linux) the actual diffs between backups should be minimal and so backups should be fast.

This is cutoff at right side - can you repost?

 

Thanks

Myk

 

 

Backupscript for xen machines.

 

#!/usr/bin/bash

vmHome="/mnt/cache/VM"
backupHome="/mnt/user/VMBackup"
logHome="/mnt/user/VMBackup"

xl list | tail -n+3 | cut -d ' ' -f 1 | while read line; do
        xl pause $line
        rsync -av --progress --"log-file=$logHome/$(date +%Y%m%d)_backup.log" $vmHome/$line $backup$
        xl unpause $line

done

 

Its not perfect, more of a example of how you can do it.

Ah man... Sorry bout that. I think I posted a copy on my github. GitHub.com/lonix/BUUX

 

 

Sent from my iPad using Tapatalk

I Rewrote the backup-script.

Includes Offline Machines, But skip the ones marked for deletion.

 

you find it here: https://raw.githubusercontent.com/lonix/BUUX/master/Backup.sh

 

Code:

#!/usr/bin/bash

vmHome="/mnt/cache/VM"
backupHome="/mnt/user/VMBackup"
logHome="/mnt/user/VMBackup"

#xl list | tail -n+3 | cut -d ' ' -f 1 | while read line; do
#
#done
function log(){

stamp=$(date "+%Y/%m/%d %H:%M:%S [$$]")
echo $stamp $1 >> "$logHome/$(date +%Y%m%d)_backup.log"
}



size=$(du -hs $vmHome | cut -f 1)
log "Initalizing backup"
log "$size of data for Potential Backup" 
mapfile -s2 -t running < <(xl list|cut -d ' ' -f 1)
#running=( $(xl list | tail -n+3 | cut -d ' ' -f 1) )
cd $vmHome
for dir in */ ; do
dir=${dir%/}
backup=$(rsync -rpsazPt --log-file="$logHome/$(date +%Y%m%d)_backup.log" "$vmHome/$dir" "$backupHome")
if [[ $dir == *DELETE* ]]; then
log "Skipping $dir reason: deleted"
	continue
fi
for i in "${running[@]}"
do
	if  [[ "$i" == "$dir" ]];
	then
#			echo "$dir: Running"
		log "$dir has been paused"
		xl pause "$dir"
		$backup
		xl unpause "$dir"
		log "$dir has been resumed"
		continue 2
	fi
		$backup
done
done

log "Finnished backing up"

I successfully backup using your script a Tretflix VM, but the "resume" didn't work, I had to do a "xl destroy" because it was not responding.  But at least I have a backup now

I successfully backup using your script a Tretflix VM, but the "resume" didn't work, I had to do a "xl destroy" because it was not responding.  But at least I have a backup now

 

Can you send me the log file ? or any other information that might help me track down the reason that happend ?

i'll try another backup tonight, i'll send it later

Awesome lonix, the script worked great.

2014/05/09 23:33:13 [18705] Initalizing backup
2014/05/09 23:33:13 [18705] 51G of data for Potential Backup
2014/05/09 23:33:13 [18717] building file list
2014/05/09 23:33:13 [18717] cd+++++++++ lost+found/
2014/05/09 23:33:13 [18717] sent 63 bytes  received 25 bytes  total size 0
2014/05/09 23:33:13 [18705] ubuntu has been paused
2014/05/09 23:33:13 [18724] building file list
2014/05/09 23:33:13 [18724] cd+++++++++ ubuntu/
2014/05/09 23:33:13 [18724] >f+++++++++ ubuntu/README.md-
2014/05/09 23:33:14 [18724] >f+++++++++ ubuntu/initrd.gz
2014/05/09 23:33:14 [18724] >f+++++++++ ubuntu/ubuntu.cfg
2014/05/09 23:33:14 [18724] >f+++++++++ ubuntu/ubuntu.cfg-
2014/05/09 23:59:49 [18724] >f+++++++++ ubuntu/ubuntu.img
2014/05/09 23:59:49 [18724] >f+++++++++ ubuntu/ubuntu.png
2014/05/09 23:59:49 [18724] >f+++++++++ ubuntu/vmlinuz
2014/05/09 23:59:49 [18724] sent 43,339,466,771 bytes  received 158 bytes  total size 53,723,447,093
2014/05/09 23:59:49 [18705] ubuntu has been resumed
2014/05/09 23:59:49 [18705] Finnished backing up

Glad to hear it.

 

I use my script wed and sun. And backup content Of vm's with another script dayli

 

Sent from my GT-I9505 using Tapatalk

 

 

I did a backup last night and it worked, but my VM was already powered Off.  It's because I changed the cache drive from a WD Green 2TB to a realllly nice Samsung EVO 840 500GB   It's pretty fast !

 

After, I did copy back the VM and they worked great, so backup script work well.

Awesome script!

 

I just had one question,  I actually forgot I posted this and was looking around my self for a bit.

 

Why pause instead of suspend?

I'm curious ... any thoughts on this to speed things up a bit?

 

http://lime-technology.com/forum/index.php?topic=10503.0

 

I'm seeing a heavy CPU load due to rsync, but to be fair right now dom0 has 1 vcpu assigned and pinned.

Ok, so with this script, backup is realy easy. Thanks.

 

How about restore? Did this work properbly?

 

Regards

Owel

I'm curious ... any thoughts on this to speed things up a bit?

 

http://lime-technology.com/forum/index.php?topic=10503.0

 

I'm seeing a heavy CPU load due to rsync, but to be fair right now dom0 has 1 vcpu assigned and pinned.

 

Hi Alex, no i did not look at it, but a quick look determine it should not a apply in this case as we don't use either rsa or ssh, (Local copy you know).

(Latest version on BUUX Git, as this seems to have been adopted into the Buux Familiy.)

 

Awesome script!

 

I just had one question,  I actually forgot I posted this and was looking around my self for a bit.

 

Why pause instead of suspend?

I newer saw any suspend option? as a matter of fact still cant find it.

 

 

Ok, so with this script, backup is realy easy. Thanks.

 

How about restore? Did this work properbly?

 

Regards

Owel

 

Done 2 Restores so far works like a charm.

 

 

 

General Note: This script uses far to much downtime and is not suited for data-backup. Just a OS level backup. In my enviroment i use this script each week (sun 6AM) And Webotechs backupscript for all my linux boxes, Dayli.https://code.google.com/p/unraid-weebotech/downloads/list

 

My 2 NAS's Unraid and ReadyNAS runs a full copy to crashplan (all my backups are stored on unraid). In other words Everything is up there somewhere.

hmmm I was thinking the same thing about it not being over SSH.  Does rsync default to not using encryption when it knows it isn't running between local folders?  Either way the rsync process sure was hammering my CPU.  I mean like 93% solid and it was really slowing down the xfer.  I could tell it was working much harder on the part of the .img files that actually had data.  xfer speeds were below 20MB/s for those sections and then jumped up to 70MB/s in the empty sections.  It was most obvious in my data.img which is 40GB but only filled with 15GB.  But those 15GB are very incompressible Plex jpg index thumbnails.  They are also unlikely to ever change again and thus be skipped on the diff ... but still, I like puzzles

 

Maybe we should be using the --inplace option like Tom uses for the mover?  The warnings for that make me nervous but so long as the VM is paused I can't see there being a problem of access during the process.  For that matter (off-topic a bit) I've always worried about accessing stuff while the mover is running and the presence of the --inplace option makes me even more so now

 

Well in any case, I'll play around a bit unless you have some thoughts. 

I'm curious ... any thoughts on this to speed things up a bit?

 

http://lime-technology.com/forum/index.php?topic=10503.0

 

I'm seeing a heavy CPU load due to rsync, but to be fair right now dom0 has 1 vcpu assigned and pinned.

 

Hi Alex, no i did not look at it, but a quick look determine it should not a apply in this case as we don't use either rsa or ssh, (Local copy you know).

(Latest version on BUUX Git, as this seems to have been adopted into the Buux Familiy.)

 

Awesome script!

 

I just had one question,  I actually forgot I posted this and was looking around my self for a bit.

 

Why pause instead of suspend?

I newer saw any suspend option? as a matter of fact still cant find it.

 

 

Ok, so with this script, backup is realy easy. Thanks.

 

How about restore? Did this work properbly?

 

Regards

Owel

 

Done 2 Restores so far works like a charm.

 

 

 

General Note: This script uses far to much downtime and is not suited for data-backup. Just a OS level backup. In my enviroment i use this script each week (sun 6AM) And Webotechs backupscript for all my linux boxes, Dayli.https://code.google.com/p/unraid-weebotech/downloads/list

 

My 2 NAS's Unraid and ReadyNAS runs a full copy to crashplan (all my backups are stored on unraid). In other words Everything is up there somewhere.

 

Is there a way to run that script with Cron or something ??  I want it to be automated... 

 

Let me know if you can provide a Cron command line let's say for a backup during the night on monday at 3am or something similar. 

 

Thanks !

So I started digging around the man rsync pages.  The first thing I noticed is that -a already implies -rlptgoD so we can remove -rpt just for elegance. That leaves us with -asP.

 

Then I saw this

-z, --compress

    With this option, rsync compresses the file data as it is sent to the destination machine, which reduces the amount of data being transmitted -- something that is useful over a slow connection.

I tried running without -z and it REALLY sped things up given my single vcpu given to dom0.  I don't have the numbers from last night with -z but I recall it was on the order of 19MB/s for the parts of my images that had data.  Without -z I had an average of about 77MB/s for arch.img and 71MB/s

 

Then I read this:

--inplace

    This option changes how rsync transfers a file when the file's data needs to be updated: instead of the default method of creating a new copy of the file and moving it into place when it is complete, rsync instead writes the updated data directly to the destination file.

This has several effects:

(1) in-use binaries cannot be updated (either the OS will prevent this from happening, or binaries that attempt to swap-in their data will misbehave or crash),

(2) the file's data will be in an inconsistent state during the transfer,

(3) a file's data may be left in an inconsistent state after the transfer if the transfer is interrupted or if an update fails,

(4) a file that does not have write permissions can not be updated, and

(5) the efficiency of rsync's delta-transfer algorithm may be reduced if some data in the destination file is overwritten before it can be copied to a position later in the file (one exception to this is if you combine this option with --backup, since rsync is smart enough to use the backup file as the basis file for the transfer).

 

WARNING: you should not use this option to update files that are being accessed by others, so be careful when choosing to use this for a copy. This option is useful for transfer of large files with block-based changes or appended data, and also on systems that are disk bound, not network bound.  The option implies --partial (since an interrupted transfer does not delete the file)

 

Generally I think it is safe to say the destination file isn't being touched, and the source has been paused, so this should be safe.  Tom runs it with the MOVER script fwiw.  Which does make me feel justified in my worries about messing with files while the mover is running, but I trust Tom knows what he is doing.  In any case I ran a test without -z and with --inplace:

rsync -asP --inplace

sending incremental file list

arch/

arch/arch.img

15,000,000,000 100%  84.97MB/s    0:02:48 (xfr#1, to-chk=2/7)

arch/data.img

42,949,672,960 100%  73.91MB/s    0:09:14 (xfr#2, to-chk=0/7)

 

My very sparse arch.img is clearly faster, and my half full data.img is a touch faster.  So my suggestion is to change the script to use "-asP --inplace" to get a good boost in xfer speeds especially with a dom0 served by only a single cpu.

Update the script on github and I'll check it out

 

Sent from my GT-I9505 using Tapatalk