May 20, 201412 yr So I started digging around the man rsync pages. The first thing I noticed is that -a already implies -rlptgoD so we can remove -rpt just for elegance. That leaves us with -asP. Then I saw this -z, --compress With this option, rsync compresses the file data as it is sent to the destination machine, which reduces the amount of data being transmitted -- something that is useful over a slow connection. I tried running without -z and it REALLY sped things up given my single vcpu given to dom0. I don't have the numbers from last night with -z but I recall it was on the order of 19MB/s for the parts of my images that had data. Without -z I had an average of about 77MB/s for arch.img and 71MB/s Then I read this: --inplace This option changes how rsync transfers a file when the file's data needs to be updated: instead of the default method of creating a new copy of the file and moving it into place when it is complete, rsync instead writes the updated data directly to the destination file. This has several effects: (1) in-use binaries cannot be updated (either the OS will prevent this from happening, or binaries that attempt to swap-in their data will misbehave or crash), (2) the file's data will be in an inconsistent state during the transfer, (3) a file's data may be left in an inconsistent state after the transfer if the transfer is interrupted or if an update fails, (4) a file that does not have write permissions can not be updated, and (5) the efficiency of rsync's delta-transfer algorithm may be reduced if some data in the destination file is overwritten before it can be copied to a position later in the file (one exception to this is if you combine this option with --backup, since rsync is smart enough to use the backup file as the basis file for the transfer). WARNING: you should not use this option to update files that are being accessed by others, so be careful when choosing to use this for a copy. This option is useful for transfer of large files with block-based changes or appended data, and also on systems that are disk bound, not network bound. The option implies --partial (since an interrupted transfer does not delete the file) Generally I think it is safe to say the destination file isn't being touched, and the source has been paused, so this should be safe. Tom runs it with the MOVER script fwiw. Which does make me feel justified in my worries about messing with files while the mover is running, but I trust Tom knows what he is doing. In any case I ran a test without -z and with --inplace: rsync -asP --inplace sending incremental file list arch/ arch/arch.img 15,000,000,000 100% 84.97MB/s 0:02:48 (xfr#1, to-chk=2/7) arch/data.img 42,949,672,960 100% 73.91MB/s 0:09:14 (xfr#2, to-chk=0/7) My very sparse arch.img is clearly faster, and my half full data.img is a touch faster. So my suggestion is to change the script to use "-asP --inplace" to get a good boost in xfer speeds especially with a dom0 served by only a single cpu. -asP --inplace --no-check is the new running on github thanks to alex
May 20, 201412 yr I'm curious ... any thoughts on this to speed things up a bit? http://lime-technology.com/forum/index.php?topic=10503.0 I'm seeing a heavy CPU load due to rsync, but to be fair right now dom0 has 1 vcpu assigned and pinned. Hi Alex, no i did not look at it, but a quick look determine it should not a apply in this case as we don't use either rsa or ssh, (Local copy you know). (Latest version on BUUX Git, as this seems to have been adopted into the Buux Familiy.) Awesome script! I just had one question, I actually forgot I posted this and was looking around my self for a bit. Why pause instead of suspend? I newer saw any suspend option? as a matter of fact still cant find it. Ok, so with this script, backup is realy easy. Thanks. How about restore? Did this work properbly? Regards Owel Done 2 Restores so far works like a charm. General Note: This script uses far to much downtime and is not suited for data-backup. Just a OS level backup. In my enviroment i use this script each week (sun 6AM) And Webotechs backupscript for all my linux boxes, Dayli.https://code.google.com/p/unraid-weebotech/downloads/list My 2 NAS's Unraid and ReadyNAS runs a full copy to crashplan (all my backups are stored on unraid). In other words Everything is up there somewhere. Is there a way to run that script with Cron or something ?? I want it to be automated... Let me know if you can provide a Cron command line let's say for a backup during the night on monday at 3am or something similar. Thanks ! this is my crontab part of my go file (/boot/config/go), now see if you can estrapulate how to do this yourself now: #Crontab crontab -l >/tmp/crontab echo "#Backup all VMs at 6 AM Sundays" >>/tmp/crontab echo "0 6 * * 0 /boot&/scripts/VMBackup.sh 1> /dev/null" >>/tmp/crontab echo "#Backup of various linux-Machines" >>/tmp/crontab echo "0 0 * * * /boot/scripts/backup.sh /boot/scripts/flash.conf 1> /dev/null" >> /tmp/crontab echo "0 1 * * * /boot/scripts/backup.sh /boot/scripts/pms.conf 1> /dev/null" >>/tmp/crontab echo "0 2 * * * /boot/scripts/backup.sh /boot/scripts/test14.conf 1> /dev/null" >>/tmp/crontab echo "0 3 * * * /boot/scripts/backup.sh /boot/scripts/tretflix.conf 1> /dev/null" >>/tmp/crontab crontab /tmp/crontab
May 20, 201412 yr Hey thanks for the cron info. I was just searching that last night. Did you see even more performance enhancement with --no-check? I had tried it once last night when I still had -z in use but when it didn't make a difference then (drop in a larger compression pond perhaps) I forgot to include that in my later testing. Mostly because I'm concerned about not hash checking. I know it isn't going over a real network but I just wasn't sure.
May 20, 201412 yr Hey thanks for the cron info. I was just searching that last night. Did you see even more performance enhancement with --no-check? I had tried it once last night when I still had -z in use but when it didn't make a difference then (drop in a larger compression pond perhaps) I forgot to include that in my later testing. Mostly because I'm concerned about not hash checking. I know it isn't going over a real network but I just wasn't sure. In my case the no check worked better than the compression. but i guess that can be a bit idividual. dependent on storage media etc. The think is checksums are not compared when using rsync in local mode (non ssh/rsh). But still calculated, and can cause some slowdown at end of large files.
May 20, 201412 yr ha! ... weird. I'll certainly add the change to mine and see what happens. Can't hurt. I also played around with --sparse but it looked like it was hammering the CPU too during the populated section of my images. But then it FLEW LIKE THE WIND on the sparse parts. I mean like 140MB/s. The end result was decent but left me worried it would just degrade as the images filled up. So I googled around looking for more info on --sparse and I found this: http://www.barricane.com/rsync-vm-sparse-inplace-kvm-vmware. But that looks like it will require a bit more logic to your scripts. If exists on target --inplace else --sparse. The thing that makes me wonder is I keep calling my image files "sparse" but I'm not sure if that is technically true since they actually present as their full size, though internally I know they must be filled out with zeroes. So I'm not sure if this will really help us. I'll test tonight if I get the chance. Oh one other thing I had been thinking about last as I read about turning off compression, is the possibility of adding .BIF to the compression exclusion list. For Plex users at least that is the biggest culprit in filling up our Library with incompressible files. They are actually just jpegs (which are already excluded) but with an extension rsync doesn't know to ignore. So, for Plex users that need to use compression for non-local rsyncs, they might really want to consider adding a .BIF exclusion
May 20, 201412 yr can the script be adjusted to look for a .conf file for the config vars at the top? than way when new versions are avail - we just have to pull down the script and not modify it? Thanks Myk
May 20, 201412 yr +1 good idea. I'll check the Crontab part you shared also. Just a strange behavior. Yesterday, I ran it manually before going to bed for the night. This morning I checked and only 1 or my 2 VM was Unpaused... I just did a xl unpause VMname and it unpaused it... Looks like it's not 100% good all the time ? I did not check the Log yet, i'll do tonight.
May 20, 201412 yr I'm curious ... any thoughts on this to speed things up a bit? http://lime-technology.com/forum/index.php?topic=10503.0 I'm seeing a heavy CPU load due to rsync, but to be fair right now dom0 has 1 vcpu assigned and pinned. Hi Alex, no i did not look at it, but a quick look determine it should not a apply in this case as we don't use either rsa or ssh, (Local copy you know). (Latest version on BUUX Git, as this seems to have been adopted into the Buux Familiy.) Awesome script! I just had one question, I actually forgot I posted this and was looking around my self for a bit. Why pause instead of suspend? I newer saw any suspend option? as a matter of fact still cant find it. Ok, so with this script, backup is realy easy. Thanks. How about restore? Did this work properbly? Regards Owel Done 2 Restores so far works like a charm. General Note: This script uses far to much downtime and is not suited for data-backup. Just a OS level backup. In my enviroment i use this script each week (sun 6AM) And Webotechs backupscript for all my linux boxes, Dayli.https://code.google.com/p/unraid-weebotech/downloads/list My 2 NAS's Unraid and ReadyNAS runs a full copy to crashplan (all my backups are stored on unraid). In other words Everything is up there somewhere. Is there a way to run that script with Cron or something ?? I want it to be automated... Let me know if you can provide a Cron command line let's say for a backup during the night on monday at 3am or something similar. Thanks ! this is my crontab part of my go file (/boot/config/go), now see if you can estrapulate how to do this yourself now: #Crontab crontab -l >/tmp/crontab echo "#Backup all VMs at 6 AM Sundays" >>/tmp/crontab echo "0 6 * * 0 /boot&/scripts/VMBackup.sh 1> /dev/null" >>/tmp/crontab echo "#Backup of various linux-Machines" >>/tmp/crontab echo "0 0 * * * /boot/scripts/backup.sh /boot/scripts/flash.conf 1> /dev/null" >> /tmp/crontab echo "0 1 * * * /boot/scripts/backup.sh /boot/scripts/pms.conf 1> /dev/null" >>/tmp/crontab echo "0 2 * * * /boot/scripts/backup.sh /boot/scripts/test14.conf 1> /dev/null" >>/tmp/crontab echo "0 3 * * * /boot/scripts/backup.sh /boot/scripts/tretflix.conf 1> /dev/null" >>/tmp/crontab crontab /tmp/crontab What is the other backup.sh and all those .conf ? I also have a Tretflix and a PMS (but on the same VM, in Tretflix) running here... care to share the conf and the script ?
May 20, 201412 yr +1 good idea. I'll check the Crontab part you shared also. Just a strange behavior. Yesterday, I ran it manually before going to bed for the night. This morning I checked and only 1 or my 2 VM was Unpaused... I just did a xl unpause VMname and it unpaused it... Looks like it's not 100% good all the time ? I did not check the Log yet, i'll do tonight. I wonder if there needs to be a minor delay after rsync completes before unpausing the VM. Or maybe a delay between unpausing VMs? cron jobs: looks like he is using a single backup script (the how) with a bunch of config files (the what and where). Neat idea lonix.
May 20, 201412 yr There is the BackupVM.sh and another Backup.sh with config files (for the content of the VM), that is what i'm asking. The BackupVM.sh is straitfoward.
May 20, 201412 yr There is the BackupVM.sh and another Backup.sh with config files (for the content of the VM), that is what i'm asking. The BackupVM.sh is straitfoward. im sure i mentioned it several times before, but the backup vim is my script ( it's only for backing up the os itself not for the data. ) for data backups (I.e. Pms library ) I use another script with several configs as you see, that is a bit more complex and requires some tinkering a link further up should point you to the original author of that script and it's download. Sent from my iPad using Tapatalk
May 21, 201412 yr Thanks for the script I used it to back up my ubuntu vm and it worked great the first time. However, when I tried running it again a couple of hours later, I expected it to be done very quickly as rsync is supposed to onlytransfer the blocks that changed. But it seems that in my case it copies the entire 70GB img file every time and it takes about an hour at about 22MB/s. Any ideas why that might be? Another question is regarding restoring from a backup. Since the backup is made at a paused state, how does one go about restoring it properly when no vm is running? So let's say I restart the unraid machine and no vms are running. Do I just copy the backup img into the respective location and do xl create? Is it ok to fresh start a vm image that was previously suspended and not shut down or is there a way to start it in a paused state to be resumed? (I guess it would be the same as doing an xl destroy and xl create) Thanks
May 22, 201412 yr Over a network to a remote destination you are right. However apparently rsync does not use delta-transfer with local syncs. The logic seems to be it takes longer to do the diff calcs than it does to just move the entire file when you aren't network limited. Given that I'm getting average xfer speeds of 75MB/s I suppose that holds water. Can you pin down why you're only seeing 22MB/s? Are you writing to the array or to a cache drive? That looks like array speeds. Or it also looks like cpu bottlenecked speed with compression and/or check-summing turned on. Now that said, I just struck gold with this google find https://lists.samba.org/archive/rsync/2008-July/021256.html ... --no-whole-file ... first time i've run across it in my crash course the last few days. I don't know i've I'll get around to trying it tonight. but if anyone else is game ... and yup ... start here and then click through to the next series of posts in the thread https://lists.samba.org/archive/rsync/2005-October/013915.html ... if you aren't network limited then doing anything that involves extra disk i/o is just going to slow you down. Even at 22MB/s, is that is your max disk i/o, then doing anything that demands even more will slow you down because your disk i/o limited. Figureing out the delta, compression, check-sums, all increase disk i/o over a raw read and send.
May 22, 201412 yr So lonix ... up for a challenge? Well it would be a challenge for me. I have a feeling for you it might not be. Right now this creates one copy and overwrites it each time. So how about a version that will keep timestamped copies [vmname]-[date]. And to be sure we don't waste too much space, it only keeps the most recent N user defined versions? ;D
May 22, 201412 yr Over a network to a remote destination you are right. However apparently rsync does not use delta-transfer with local syncs. The logic seems to be it takes longer to do the diff calcs than it does to just move the entire file when you aren't network limited. Given that I'm getting average xfer speeds of 75MB/s I suppose that holds water. Can you pin down why you're only seeing 22MB/s? Are you writing to the array or to a cache drive? That looks like array speeds. Or it also looks like cpu bottlenecked speed with compression and/or check-summing turned on. Now that said, I just struck gold with this google find https://lists.samba.org/archive/rsync/2008-July/021256.html ... --no-whole-file ... first time i've run across it in my crash course the last few days. I don't know i've I'll get around to trying it tonight. but if anyone else is game ... and yup ... start here and then click through to the next series of posts in the thread https://lists.samba.org/archive/rsync/2005-October/013915.html ... if you aren't network limited then doing anything that involves extra disk i/o is just going to slow you down. Even at 22MB/s, is that is your max disk i/o, then doing anything that demands even more will slow you down because your disk i/o limited. Figureing out the delta, compression, check-sums, all increase disk i/o over a raw read and send. ah, that makes sense. Yes I am doing a local copy and writing to the array directly (I don't mess around with the cache drive for crucial data like backups. Media on the other hand goes to cache first) so the speeds are what I normally observe. So if I were to force rsync to do delta transfer, it would probably still take an hour (or longer), the only difference would be the amount of data transferred. I get that. In terms of cpu load, xl top shows around 100% during transfer but I have a quad core, so that shouldn't be the bottleneck. Thanks
May 25, 201412 yr So I tried to setup a crontab in my GO file. #Crontab crontab -l >/tmp/crontab echo "#Backup all VMs at 4 AM Sundays" >>/tmp/crontab echo "0 4 * * 0 /boot&/scripts/VMBackup.sh 1> /dev/null" >>/tmp/crontab crontab /tmp/crontab This is the result I got in the syslog. May 25 04:00:01 Tower crond[1397]: exit status 127 from user root /boot&/scripts/VMBackup.sh 1> /dev/null May 25 04:00:01 Tower crond[7668]: unable to exec /usr/sbin/sendmail: cron output for user root /boot&/scripts/VMBackup.sh 1> /dev/null to /dev/null Not sure how to correct. From what I understand "exit status 127" means the command is not found, but the VMBackup.sh is located on my flash drive in a folder named "scripts. Any ideas/clues on how to correct? *edit* Tried running VMBackup.sh via telnet, got this error.... du: cannot access ‘/mnt/xendisk\r’: No such file or directory : No such file or directory /mnt/xendisk sending incremental file list rsync: change_dir "/mnt/xendisk\#015" failed: No such file or directory (2) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.0] /mnt/xendisk is the disk I mounted outside the array to host my VM's. I'm not sure why it can not see it.
May 25, 201412 yr Should '/boot&/scripts/VMBackup.sh' not be '/boot/scripts/VMBackup.sh' ? Can you post the content of VMBackup.sh, as the second error is likely a typo in the script itself I'm guessing Sent from my Nexus 5 using Tapatalk
May 25, 201412 yr Should '/boot&/scripts/VMBackup.sh' not be '/boot/scripts/VMBackup.sh' ? Can you post the content of VMBackup.sh, as the second error is likely a typo in the script itself I'm guessing Sent from my Nexus 5 using Tapatalk I also thought about that but the script did not run when I ran it from telnet myself. VMBackup.sh #!/bin/bash #IF YOU ARE LOOKING FOR CONFIG FILE PLEASE USE VMBackup.conf ##Configuration Part ##Defaults vmHome="/mnt/cache/VM" backupHome="/mnt/user/VMBackup" logHome="/mnt/user/VMBackup/logs" backupCount="0" if [ -f "VMBackup.conf" ]; then echo "Config Loaded" source "VMBackup.conf"; fi ##Functions function log(){ stamp=$(date "+%Y/%m/%d %H:%M:%S [$$]") echo "$stamp" "$1" >> "$logHome/$(date +%Y%m%d)_backup.log" } function sync(){ rsync -asP --inplace --no-checksum --log-file="$logHome/$(date +%Y%m%d)_backup.log" "$vmHome/$dir" "$backupHome" } function cleanup(){ #echo "Cleanup started" if [[ -d "$backupHome"/backup"$backupCount" ]]; then log "Deleteing last backup: backup$backupCount" rm -r $backupHome/backup"$backupCount" fi for (( i=backupCount; i>=2 ; i-- )); do if [[ -d "$backupHome"/backup$((i-1)) ]]; then log "Moving: backup$((i-1)) to backup$i" mv $backupHome/backup$((i-1)) $backupHome/backup$i fi done log "Creating new backup folder: $backupHome/backup1" mkdir $backupHome/backup1 } ##VERSION Version="1.3" latest=$(curl -s https://raw.githubusercontent.com/lonix/BUUX/master/VMBackup-version) clear if [ "$Version" != "$latest" ]; then log "You are running version $Version, but $latest is latest." echo "New version is avalible, newest version is $latest" echo "you are running version $Version" echo "" echo "New this version: " echo "-----------------------------------------" curl "https://raw.githubusercontent.com/lonix/BUUX/master/VMBackup-changes" echo "-----------------------------------------" echo "To upgrade simple copy-paste this into your console:" echo "-----------------------------------------" echo "cd /boot && wget https://raw.githubusercontent.com/lonix/BUUX/master/VMBackup.sh -O VMBackup.sh && chmod +x VMBackup.sh" echo "-----------------------------------------" # sleep 5 fi if (( $backupCount != 0 )); then log "Cleanup is started." cleanup backupHome="$backupHome"/backup1 log "Cleanup is done" fi ##INIT size=$(du -hs $vmHome | cut -f 1) log "Initialized backup" log "$size of data for Potential Backup" mapfile -s2 -t running < <(xl list|cut -d ' ' -f 1) cd $vmHome for dir in */ ; do dir=${dir%/} if [[ $dir == *DELETE* ]]; then log "Skipping $dir reason: deleted" continue fi for i in "${running[@]}" do if [[ "$i" == "$dir" ]]; then # echo "$dir: Running" xl pause "$dir" log "$dir has been paused" sync xl unpause "$dir" log "$dir has been resumed" continue 2 fi done sync done log "Finished backing up" My VMBackup.conf vmHome="/mnt/xendisk" backupHome="/mnt/user/MyBackups/VM" logHome="/mnt/user/MyBackups"
May 25, 201412 yr Should '/boot&/scripts/VMBackup.sh' not be '/boot/scripts/VMBackup.sh' ? Can you post the content of VMBackup.sh, as the second error is likely a typo in the script itself I'm guessing Sent from my Nexus 5 using Tapatalk it can be where ever you want it, so long as you reference it right. I personally like boot /scripts for old reasons. Sent from my Nexus 7 using Tapatalk
May 26, 201412 yr [hide] So I tried to setup a crontab in my GO file. #Crontab crontab -l >/tmp/crontab echo "#Backup all VMs at 4 AM Sundays" >>/tmp/crontab echo "0 4 * * 0 /boot&/scripts/VMBackup.sh 1> /dev/null" >>/tmp/crontab crontab /tmp/crontab This is the result I got in the syslog. May 25 04:00:01 Tower crond[1397]: exit status 127 from user root /boot&/scripts/VMBackup.sh 1> /dev/null May 25 04:00:01 Tower crond[7668]: unable to exec /usr/sbin/sendmail: cron output for user root /boot&/scripts/VMBackup.sh 1> /dev/null to /dev/null Not sure how to correct. From what I understand "exit status 127" means the command is not found, but the VMBackup.sh is located on my flash drive in a folder named "scripts. Any ideas/clues on how to correct? *edit* Tried running VMBackup.sh via telnet, got this error.... du: cannot access ‘/mnt/xendisk\r’: No such file or directory : No such file or directory /mnt/xendisk sending incremental file list rsync: change_dir "/mnt/xendisk\#015" failed: No such file or directory (2) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1165) [sender=3.1.0] /mnt/xendisk is the disk I mounted outside the array to host my VM's. I'm not sure why it can not see it. [/hide] Finally figured it out yesterday. I had the Ubuntu VM to autostart and somehow while the VM was active and I was able to vnc and access it and the share on my Windows machine, the share did not exist according to unraid. So I disabled the the autostart of the VM and now it all works ok.
Archived
This topic is now archived and is closed to further replies.