May 19, 201313 yr I am running an unRAID VM in ESXi along with a FreeNAS VM and others. I use the FreeNAS to backup a few of my important unRAID shares via rsync. Today I noticed I somehow lost a file (deleted?) and realized my backup plan is severely flawed. Because I use the -delete option for rsync, the file that was somehow deleted from my unRAID share was also deleted from my backup when the rsync script ran last night (how stupid am I, right). So now I am looking for a method to do a proper backup with versioning. I'm not very knowledgeable in Linux or rsync, but is there a way to do a backup with versioning? I know I could just remove the -delete option and the file still would have been there today, but I would like to do it the proper way if possible. Plus there are some files where it would be really handy to have a couple versions backed up. How do you guys do it?
May 19, 201313 yr Well... i had a similar issue and did not found a real solid way but what i am doimg now suffices for my needs; I do a daily rsync to a share called user_yesterday I do a weekly rsync to a share called user_lastweek I do a monthly rsync to a share called user_lasmonth All without the -delete option. It is not versioning but its a little step in that direction. Sent from my iPad using Tapatalk HD
May 19, 201313 yr ...are you running ZFS on that FreeNAS VM? If yes, just do a snap(shot) plan on FreeNAS...the snaps will preserve files even when deleted by rsync...use windows "previous version" to re-fetch it from your FreeNAS VM. Snaps on ZFS are unlimited in numbers and basically free (in terms of space consumed).
May 19, 201313 yr I do essentially what Helmonder described, although with a bit more paranoia [i have Mon_Backups, Wed_Backups, Fri_Backups, 1st_Backups, and 15th_Backups folders that are all automated to be done as indicated (On Mon, Wed, Fri, and the 1st and 15th of each month) ... and you should NEVER use a "Delete" option for backups => that defeats the whole idea of having copies of files you might need !! For static data (movies, etc. - by far the biggest consumer of space), I simply have one complete set of backups on an extra set of drives.
May 20, 201313 yr Author ... and you should NEVER use a "Delete" option for backups => that defeats the whole idea of having copies of files you might need !! Did you miss the part where I said I "realized my backup plan is severely flawed."?
May 20, 201313 yr Not sure this would work on unraid. But I use rsnapshot. also could look into freenas' snapshot feature. Sent from my SGH-I727R using Tapatalk 2
May 20, 201313 yr The rsync_linked_backup script I have on my google code page can do dated backups. It's like rsnapshot, but it's only in shell and does not require python. It does not delete the archives automatically yet. I'm still writing the script which requires some external tools to keep a specific amount of dailys, weeklys and monthlys. In the meantime, it uses linked-dest on the backup machine. It's more of a pull backup tool, so it requires execution on the backup machine with remote rsync access to the machine to be backed up. It can function with rsync over ssh or rsync to an rsync server. i.e. rsync://username:password@servername:/rsyncblock/files. If you change the dated output directory you can do weeklys using variables from date/strftime for output directory name. ie. current default is %Y%m%d which is daily. weekly could be %Y-%V monthly could be %Y-%m hourly could be %Y%m%d-%H The only requirement is that the dated backup directories sort in the correct order so the prior backup folder is linked to the current folder. This requires at least 1x the current backup source, plus the space required for each revision. Here's an example of the distribution for one developer box backup. root@rgclws:/local/backups/npgvm3 # du -hs . 7.8G root@rgclws:/local/backups/npgvm3 # du -hs * 6.9G 20130507 108M 20130508 67M 20130509 70M 20130510 70M 20130511 95M 20130512 66M 20130513 101M 20130514 75M 20130515 68M 20130516 69M 20130517 69M 20130518 63M 20130519 63M 20130520 8.0K etc 15M log If I were to remove the first directory, all the files would still exist and 20130508 would then be the 6.9G directory Filecount distribution. find . -type d -maxdepth 1 | while read dir;do echo -e "$dir: \c" ; find $dir -type f | wc -l ; done .: 1639339 ./log: 15 ./20130509: 116853 ./etc: 1 ./20130507: 118299 ./20130508: 116816 ./20130510: 116865 ./20130511: 116885 ./20130512: 116886 ./20130513: 116888 ./20130514: 116895 ./20130515: 117071 ./20130516: 117081 ./20130517: 117130 ./20130518: 117220 ./20130519: 117218 ./20130520: 117216 Edit: Let me add. You can do this to another disk on unRAID as long as it's local or to a disk share. I doubt very much this would work correctly using user shares. the rsync uses a feature called link-dest I.E. RSYNCOPTS="${RSYNCOPTS} --link-dest=${BACKUPDIR}/${LAST_BACKUP_DATE}" The last backup is determined by sorting. root@rgclws:/local/backups/npgvm3 # ls -l total 68 drwxr-xr-x 5 root root 4096 May 7 05:52 20130507 drwxr-xr-x 4 root root 4096 May 8 03:02 20130508 drwxr-xr-x 4 root root 4096 May 9 03:00 20130509 drwxr-xr-x 4 root root 4096 May 10 03:00 20130510 drwxr-xr-x 4 root root 4096 May 11 03:02 20130511 drwxr-xr-x 4 root root 4096 May 12 03:00 20130512 drwxr-xr-x 4 root root 4096 May 13 03:00 20130513 drwxr-xr-x 4 root root 4096 May 14 03:01 20130514 drwxr-xr-x 4 root root 4096 May 15 03:00 20130515 drwxr-xr-x 4 root root 4096 May 16 03:03 20130516 drwxr-xr-x 4 root root 4096 May 17 03:00 20130517 drwxr-xr-x 4 root root 4096 May 18 03:00 20130518 drwxr-xr-x 4 root root 4096 May 19 03:00 20130519 drwxr-xr-x 4 root root 4096 May 20 03:00 20130520 drwxr-xr-x 2 root root 4096 May 6 13:49 etc -rwxr--r-- 1 root root 102 May 20 10:28 filecount drwxr-xr-x 2 root root 4096 May 20 03:02 log root@rgclws:/local/backups/npgvm3 # ls -1 | sort | grep '[0-9]' | tail -2 20130519 20130520 Here's the scriplet in the script. ls -1 ${BACKUPDIR} | sort | grep '[0-9]' | tail -2 > ${TMPFILE} while read LAST_BACKUP do if [ "${LAST_BACKUP}" = "${BACKUPDATEMASK}" ] then continue fi export LAST_BACKUP_DATE=${LAST_BACKUP} done < ${TMPFILE} rm -f ${TMPFILE}
May 20, 201313 yr Why not use backup software that does archiving? I have a folder in each of my disk backups that contains all files that no longer exist on my master disks. I have lost files in a similar way and now I archive everything for as long as I have space.
May 20, 201313 yr Why not use backup software that does archiving? I have a folder in each of my disk backups that contains all files that no longer exist on my master disks. I have lost files in a similar way and now I archive everything for as long as I have space. Can you recommend a backuptool to run on unraid ? The thing is that there are enough windows based tools out there, but to make the homeserver thing be as transparent as possible this would need to run ON the unraid server..
May 20, 201313 yr s'more reading based on rsync. http://www.mikerubel.org/computers/rsync_snapshots/ rsync also has the following options. -b, --backup make backups (see --suffix & --backup-dir) --backup-dir=DIR make backups into hierarchy based in DIR --suffix=SUFFIX backup suffix (default ~ w/o --backup-dir) If you were to choose a suffix, it could be according to the date and or time. my only beef with that is it would according to the current date/time. I suppose you could do an rsync to list what is different, then use a script to copy the current file to a file dated according to the files modification time. However with all of that. It's just as easy to use dated directories as I do.
May 20, 201313 yr ... and you should NEVER use a "Delete" option for backups => that defeats the whole idea of having copies of files you might need !! Did you miss the part where I said I "realized my backup plan is severely flawed."? Mea Culpa No, I didn't miss it ... I was just writing a generic answer and included that for the benefit of anyone else who was reading that bit of advice -- obviously you already realize it as a result of what happened that resulted in this question. As for whether the backup features you want are available on UnRAID [i.e. "... The thing is that there are enough windows based tools out there, but to make the homeserver thing be as transparent as possible this would need to run ON the unraid server..."] ==> you may want to reconsider whether this needs to run on UnRAID. As you've noted, there are a lot of good choices for Windows ... and since the Windows box has to be running to back it up, there's no particular disadvantage to having the backup software running on the Windows box. Your goal of the backup being as "... transparent as possible " would seem to imply that you want backups to be done from ONE place, so all your systems don't have to be running the backup software. That is, however, achievable without the "one place" being your UnRAID box. My backups are all run from my primary Windows machine -- I have a set of "profiles" in SyncBack SE that do all of my backups from our various systems to the UnRAID backup server at 4:00am every day. The scheduled backup starts with a few WOL commands to turn on all of the PCs (in case they're off) ... then the backup itself runs 3 minutes later. I could (but don't) even leave the UnRAID server off and turn it on the same way.
May 20, 201313 yr Author Wow! Tons of great information guys. Thanks! Sounds like plenty of options out there and I have a lot to look into. Since I am backing up to FreeNAS with ZFS and it does have the option do snapshots, I've read a little about that today (as much as work would allow) and it seems too good to be true, but maybe I'm missing something. From what I gather, the snapshots take up almost no space, but allow for file recovery. So if I took a snapshot and then deleted every file in the volume, it all could be recovered from the snapshot? And if you take daily snapshots then you essentially havea type of versioning?
May 20, 201313 yr Yes, Snapshots let you recover to any of the snapshots. They're relatively small, since they store the difference info since the last snapshot.
May 20, 201313 yr Yes, Snapshots let you recover to any of the snapshots. They're relatively small, since they store the difference info since the last snapshot. I have rsnapshot working fine on my server. A important documents share on disk2 is being backed up every 2 hours onto disk23 like this: Hourly.0 Hourly.1 Hourly.2 Hourly.3 Hourly.4 Hourly.5 Daily.0 Daily.1 Daily.2 Daily.3 Daily.4 Daily.5 Daily.6 Daily.7 Weekly.0 Weekly.1 Weekly.2 Weekly.3 Monthly.0 Monthly.1 Monthly.2 Monthly.3 And this is configurable to allow more or less versions to be saved. Total space consumed is almost 0 for the additional versions since they are hard links. If you have a lot of changes to your backed up files, you will require additional storage space. I was surprised how easy it was to get going. See here: http://lime-technology.com/forum/index.php?topic=26888.0
May 20, 201313 yr Note rsnapshot/rsync will let you backup to a remote server via SSH. Which is what I do with really important files. God forbid I have a fire in the house. No amount of local backupa may be safe. Sent from my SGH-I727R using Tapatalk 2
May 21, 201313 yr Note rsnapshot/rsync will let you backup to a remote server via SSH. Which is what I do with really important files. God forbid I have a fire in the house. No amount of local backupa may be safe. Sent from my SGH-I727R using Tapatalk 2 I have a 2 complete "servers in a suitcase" that are switched between offsite locations as our business could not cope without our data. One is always offsite. With up to 100 GB being created every day on the server, SSH backups over the internet won't cut it...
May 21, 201313 yr Wow! Tons of great information guys. Thanks! Sounds like plenty of options out there and I have a lot to look into. Since I am backing up to FreeNAS with ZFS and it does have the option do snapshots, I've read a little about that today (as much as work would allow) and it seems too good to be true, but maybe I'm missing something. From what I gather, the snapshots take up almost no space, but allow for file recovery. So if I took a snapshot and then deleted every file in the volume, it all could be recovered from the snapshot? And if you take daily snapshots then you essentially havea type of versioning? Yes, as I said...make a nice plan for doing ZFS snaps regularly...like after every rsync run. You can access the snap's files in your explorer when sharing the folder containing the snap vs CIFS/SMB. See here: http://breden.org.uk/2008/05/12/home-fileserver-backups-from-zfs-snapshots/ , here http://blog.night-shade.org.uk/2013/03/backing-up-and-archiving-with-zfs-snapshots/ ..and here: https://blogs.oracle.com/amw/entry/using_windows_previous_versions_to1 for a start.
Archived
This topic is now archived and is closed to further replies.