How to Backup Unraid

[landS]

My current Main Server is a Win 7 Machine (2500K processor) with 8 HD's (3 TB),  each pair in a mirrored array.  This is what I initially stored all my legally owned movie collection on.

 

Now my entire collection is copied to my always on unraid server (in my signature) with 1 parity drive. 

 

Is there an option to have an unraid machine that I turn on once a month to backup the main unraid machine so that I can do away with the power hungry Win 7 Machine? 

 

I can sell the Adaptec SAS controllers and nearly fund an entire 2nd Unraid box.  I already have enough disks to re-purpose from the win 7 machine.

 

Thanks!

[garycase]

As long as you have another UnRAID license for it, just build a 2nd server with enough storage capacity to backup your server;  give it a DIFFERENT ID  [Can't have two servers named "Tower"  ];  and then just switch it on as desired and run a synchronization utility to do the backup.

 

For example ...

 

If the main server is named "Media", and the backup server is named "MyBackups", and the share you wanted to backup is named "Movies" ... then you could initially do the complete backup with the following command:

 

Copy \\Media\Movies  \\MyBackups\Movies

 

If you wanted this to be automatic, you could do the following:

 

(a)  Configure the backup server for Wake-on-LAN

(b)  Set up a scheduled task on either a Windows PC or the main UnRAID server to do the following:

 

    ->  Send a magic packet to turn on the backup server

    ->  Wait 2 minutes (or whatever time it takes to boot)

    ->  Run a synchronization utility to update your backups

    ->  Send a "Shutdown" command to the backup server

 

I know how to do this in Windows ... you'd need a "Linux guy" to provide details on the script and Cron setup needed to do it in Linux.

 

[ZeroK]

There are lots of ways to back up unraid.  If you have two unraid servers then rsync is the best but if you only have an unraid server and a windows computer then garycase's way is correct or this way.

 

This is how I back up my two unraid servers to my windows home server.  I use robocopy and scheduler.  Robocopy using this command will only copy things that have changed and ignores the rest.  It's always done as robocopy <source> <destination> <options> (/MIR makes a mirror image of the source and if it's already there it skips it so backups take less time if you only added one movie)  /R:0 (will retry an open file a specific amount of time in case someone has it open)  and  /W:1 (will wait 1 second before trying again) (without these options if it comes to a file that is open it will wait for user input)

 

robocopy \\TOWER\HD_MOVIES \\BACKUP\HD_MOVIES /MIR /R:0 /W:1

 

Create a text document, make sure that your file extensions are unhidden and name it whatever you want but put a cmd file extension on it, for instance, backup.cmd  Put the above command into it, save it, then test it on some data you don't care about first to make sure it works as you want it, then set it up in the scheduler to run whenever you want.

 

You can have multiple lines in the same file so if you want to backup multiple shares it will start the next share once it finishes with the first command.  For instance...

 

robocopy \\TOWER\HD_MOVIES \\BACKUP\HD_MOVIES /MIR /R:0 /W:1

robocopy \\TOWER\TV_SHOWS \\BACKUP\TV_SHOWS /MIR /R:0 /W:1

robocopy \\TOWER\MP3s \\BACKUP\MP3s /MIR /R:0 /W:1

 

Great thing is, Robocopy comes free with windows.  There is an option for multithreaded copying, meaning it will copy multiple files at the same time, but this option usually ends up taking just as much time and saturates the bandwidth, plus it doesn't show a progress bar.

[landS]

Thanks!

 

What I am really thinking about doing is having the following:

 

Windows Work Machine - For office use, gaming, encoding when new purchases are made.  This machine changes frequently and tends to be VERY power hungry.  No need to keep hard drives full of storage on this rig.  This machine should only have network access for read/write to the file storage location

 

Tower - My main storage repository.  Holds ALL my windows backups, work files, music files, movie files

 

Tower2 - a backup of all the storage repository.  Holds a copy of all the data should anything happen to the first Tower.

 

****

 

Originally my main file repository was on my Windows machine on Mirrored disks.

 

Now my main file repository is on my Unraid machine with a single disk fault tolerance.

 

I would like to do away with my Windows machine storage and follow Gary's advice:

 

(a)  Configure the backup server for Wake-on-LAN

(b)  Set up a scheduled task on either a Windows PC or the main UnRAID server to do the following:

 

    ->  Send a magic packet to turn on the backup server

    ->  Wait 2 minutes (or whatever time it takes to boot)

    ->  Run a synchronization utility to update your backups

    ->  Send a "Shutdown" command to the backup server

 

So now the search is on in how to use rsync, the pluses and the minuses

[RockDawg]

Lately I've been considering setting up a second backup unRAID server myself.  Would there be any negatives to running both as VM's on the same host? 

[garycase]

Lately I've been considering setting up a second backup unRAID server myself.  Would there be any negatives to running both as VM's on the same host?

 

:)

 

Think about what you're proposing -- remember this is a BACKUP.    As in designed to protect you if something happens to your main machine !!

 

Clearly your backup should NOT be on the same machine you're backing up.  While this is indeed better than NOT backing up ... many people have a backup disk mounted in the same machine -- it only protects you against a failed disk => not against a major problem with the machine due to a power spike, catastrophic failure of the power supply (which can "take out" a lot of components with it;  flood or fire damage; etc.    My backups are at the far end of our house ... and serious corporate backups are not only off-site but in a different city.      An example of some GOOD backups => there was NO data lost by any of the major financial companies in the World Trade Centers on 9/11 !!!

 

In other words, it's MUCH better to build a low-power backup system and put it somewhere else in your home.    Turn it on via WOL, do your backups, and then shut it down.

 

[JonathanM]

In other words, it's MUCH better to build a low-power backup system and put it somewhere else in your home.    Turn it on via WOL, do your backups, and then shut it down.

If you are only using it for short periods, power consumption isn't a concern. Just throw together the cheapest box you can, use your castoff smaller hard drives, whatever gets you a current backup. Do it sooner rather than later. I deal with people on a regular basis that all say, I was planning on backing it up, but I never got around to it. Data recovery is VERY expensive in some cases, you could put together a very nice 20TB unraid server for the average price of an expedited clean room recovery.

[RockDawg]

I didn't consider a power surge or something similar.  I run through a UPS, which helps a lot, but I know it won't prevent everything.  Off site backup is just not feasible for me so that's out of the question and I don't see a lot of value to putting the backup server on another side of my house except for maybe a fire that is isolated to one area of my house.  I know I can't/won't protect against every possible case.  Just looking for something to cover most cases and especially data corruption and accidental deletion.

[garycase]

I didn't consider a power surge or something similar.  I run through a UPS, which helps a lot, but I know it won't prevent everything.  Off site backup is just not feasible for me so that's out of the question and I don't see a lot of value to putting the backup server on another side of my house except for maybe a fire that is isolated to one area of my house.  I know I can't/won't protect against every possible case.  Just looking for something to cover most cases and especially data corruption and accidental deletion.

 

Understand ... my "other side of the house" comment was just to emphasize the "not on the same machine" aspect of it      But if you have Cat-6 wired throughout the home, it's not a bad idea to have it in a different room ... most fires don't totally destroy a home;  burglaries don't usually involve taking EVERYTHING (so if they take one, they'll likely miss the other);  etc.    And a major electrical "hit" (e.g. lightning strike) that "takes out" one of your UPS's and damages connected equipment would be unlikely to "take out" a second one as well.   

 

But even a backup array on the same physical machine (in a 2nd VM) would indeed protect against data corrupted and accidental deletion ==> it's the same concept as having an extra drive in a system with an automated backup script that backs up to the 2nd drive.

 

The key thing is, as jonathanm noted, to DO IT !!    A backup that you "meant to do" provides ZERO protection 

 

I agree, of course, that if it's just a backup server and only turned on for the backups, that the power consumption isn't a major concern ... my "low power" comment wasn't mean to mean spend a lot of $$ to keep power consumption low; but to note that you clearly don't need a high-power CPU, 7200rpm drives, etc. that you might want in a primary server.

[Kryspy]

What is the easiest way to backup to a external hard drive from unRAID?  What file format to use?  I also have an iMac.

 

Kryspy

 

*Nevermind....... bit torrent sync seems like a good fit for me at the moment

 

[dchamb]

Well I am also interested in a good backup plan that allows me to take the backup disks offsite easily. The only way I know how to do that is to get some drives that can at least match the capacity of what I am using on the unRaid system (8TB right now), and connect them either through esata or USB 3.0. Now unless I use backup software that can span volumes, I would need to break up the files so that they can fit on the external drives, since I don't know of anyone producing drives greater than 4TB these days.

[garycase]

If you want to physically move the backup media off-site, then the simplest approach is to backup to a set of bare drives (using an eSATA or USB dock) and simply store them in WiebeTech DriveBoxes.

http://www.overstock.com/Electronics/WiebeTech-DriveBox-3851-0000-11-Hard-Disk-Case/5792263/product.html

 

I do this with one set of backups, and store them in a fireproof, waterproof, data-rated safe.

 

There are several ways to keep them up-to-date:

 

(1)  Copy everything you write to your array to a "current backups" disk at the same time; and simply replace that drive (and store it away) when it gets full.    This is what I do, since I'm the only one who every writes anything to our array.

 

(2)  Backup the entire array to a backup disk, copying only files that have been modified since a specified time.    As long as you keep track of the time you do each backup; and then use that as the starting time for the next one, you'll capture all new/changed data with each new backup.

 

(3)  Backup the individual UnRAID disks one-on-one with a matching size backup disk.  Note that this potentially requires cycling through as many backup disks as you have array disks for each backup cycle ... although depending on your usage it can be far less of a hassle than that.  For example, if you use a "fill-up" allocation strategy, then UnRAID only writes to one disk until it's full ... so backups are significantly simplified; although you'd still need to use the technique in #2 to ensure you captured any changed files from other disks.

 

Depending on the amount of data you have, it may be far easier to just use a portable backup server; and carry the server off-site.  There are a couple of nice portable mini-ITX cases that can hold 6 drives.    With 4TB drives, this would give you 20TB of protected storage ... or 25TB if you wait for the forthcoming 5TB units that both WD and Seagate are expected to ship "any day now" (at least sometime this year).

 

A couple of nice cases for this ...

 

http://www.newegg.com/Product/Product.aspx?Item=N82E16811345027

http://www.newegg.com/Product/Product.aspx?Item=N82E16811112348

[tr0910]

Depending on the amount of data you have, it may be far easier to just use a portable backup server; and carry the server off-site.  There are a couple of nice portable mini-ITX cases that can hold 6 drives.    With 4TB drives, this would give you 20TB of protected storage ... or 25TB if you wait for the forthcoming 5TB units that both WD and Seagate are expected to ship "any day now" (at least sometime this year).

 

A couple of nice cases for this ...

 

http://www.newegg.com/Product/Product.aspx?Item=N82E16811345027

http://www.newegg.com/Product/Product.aspx?Item=N82E16811112348

 

This is what I do.  I had an old computer with a handle that was my first unRaid server in this:

 

http://www.newegg.ca/Product/Product.aspx?Item=N82E16811154094

 

I can get 9 disks into this, and my 24 bay server at present only has 8 disks that need backed up off site.  So each week I hook this up and let it run the following script for a few hours.  (It is surprisingly how efficient the following script is.  About 10 min per 3tb disk if there is no changes)

 

echo "Starting Sync to Tower3"

# Set up mounts to Tower3
mkdir /mnt/t3disk1
mkdir /mnt/t3disk2
mkdir /mnt/t3disk3
mkdir /mnt/t3disk4
mkdir /mnt/t3disk5
mkdir /mnt/t3disk6
mkdir /mnt/t3disk7
mkdir /mnt/t3disk8
mkdir /mnt/t3disk9
mount -t nfs tower3:/mnt/disk1/ /mnt/t3disk1
mount -t nfs tower3:/mnt/disk2/ /mnt/t3disk2
mount -t nfs tower3:/mnt/disk3/ /mnt/t3disk3
mount -t nfs tower3:/mnt/disk4/ /mnt/t3disk4
mount -t nfs tower3:/mnt/disk5/ /mnt/t3disk5
mount -t nfs tower3:/mnt/disk6/ /mnt/t3disk6
mount -t nfs tower3:/mnt/disk7/ /mnt/t3disk7
mount -t nfs tower3:/mnt/disk8/ /mnt/t3disk8
mount -t nfs tower3:/mnt/disk9/ /mnt/t3disk9
echo mounts to t3 set up

# Backup from production server to 2nd level backup via cron and log results
# version .12

# Set up email header

echo To: [email protected] >> /boot/logs/cronlogs/t3summary.log
echo From: [email protected] >> /boot/logs/cronlogs/t3summary.log
echo Subject: Tower3 rsync summary >> /boot/logs/cronlogs/t3summary.log
echo   >> /boot/logs/cronlogs/t3summary.log

# Backup Disk 1 - 9

echo ====== moving to Disk1 =========== >> /boot/logs/cronlogs/t3summary.log
date >> /boot/logs/cronlogs/t3summary.log
rsync -av --stats --progress /mnt/disk1/ /mnt/t3disk1/  >> /boot/logs/cronlogs/t3disk1.log

echo ====== moving to Disk2 =========== >> /boot/logs/cronlogs/t3summary.log
date >> /boot/logs/cronlogs/t3summary.log
rsync -av --stats --progress /mnt/disk2/ /mnt/t3disk2/  >> /boot/logs/cronlogs/t3disk2.log

echo ====== moving to Disk3 =========== >> /boot/logs/cronlogs/t3summary.log
date >> /boot/logs/cronlogs/t3summary.log
rsync -av --stats --progress /mnt/disk3/ /mnt/t3disk3/  >> /boot/logs/cronlogs/t3disk3.log

echo ====== moving to Disk4 =========== >> /boot/logs/cronlogs/t3summary.log
date >> /boot/logs/cronlogs/t3summary.log
rsync -av --stats --progress /mnt/disk4/ /mnt/t3disk4/  >> /boot/logs/cronlogs/t3disk4.log

echo ====== moving to Disk5 =========== >> /boot/logs/cronlogs/t3summary.log
date >> /boot/logs/cronlogs/t3summary.log
rsync -av --stats --progress /mnt/disk5/ /mnt/t3disk5/  >> /boot/logs/cronlogs/t3disk5.log

echo ====== moving to Disk6 =========== >> /boot/logs/cronlogs/t3summary.log
date >> /boot/logs/cronlogs/t3summary.log
rsync -av --stats --progress /mnt/disk6/ /mnt/t3disk6/  >> /boot/logs/cronlogs/t3disk6.log

echo ====== moving to Disk7 =========== >> /boot/logs/cronlogs/t3summary.log
date >> /boot/logs/cronlogs/t3summary.log
rsync -av --stats --progress /mnt/disk7/ /mnt/t3disk7/  >> /boot/logs/cronlogs/t3disk7.log

echo ====== moving to Disk8 =========== >> /boot/logs/cronlogs/t3summary.log
date >> /boot/logs/cronlogs/t3summary.log
rsync -av --stats --progress /mnt/disk8/ /mnt/t3disk8/  >> /boot/logs/cronlogs/t3disk8.log

echo ====== moving to Disk9 =========== >> /boot/logs/cronlogs/t3summary.log
date >> /boot/logs/cronlogs/t3summary.log
rsync -av --stats --progress /mnt/disk9/ /mnt/t3disk9/  >> /boot/logs/cronlogs/t3disk9.log

echo ====== moving to end =========== >> /boot/logs/cronlogs/t3summary.log
date >> /boot/logs/cronlogs/t3summary.log

# Wait for 5 minutes
sleep 300

# Release mounts to Tower3
umount /mnt/t3disk1
umount /mnt/t3disk2
umount /mnt/t3disk3
umount /mnt/t3disk4
umount /mnt/t3disk5
umount /mnt/t3disk6
umount /mnt/t3disk7
umount /mnt/t3disk8
umount /mnt/t3disk9

sleep 60
rmdir /mnt/t3disk1
rmdir /mnt/t3disk2
rmdir /mnt/t3disk3
rmdir /mnt/t3disk4
rmdir /mnt/t3disk5
rmdir /mnt/t3disk6
rmdir /mnt/t3disk7
rmdir /mnt/t3disk8
rmdir /mnt/t3disk9
echo mounts to t3 taken down

  # Send email of summary of results
  ssmtp [email protected] < /boot/logs/cronlogs/t3summary.log
  cd /boot/logs/cronlogs  
  mv t3summary.log "`date +%Y%m%d_%H%M`_t3summary.logarchive"

 

When this is done, the servers are mirrored up and the computer with a handle gets carried back offsite.  With 9 disks, the backup server weighs about 30 lb.  Note: you need to match disk capacities up exactly if you have different sized disks in your system.  Disk1 must be the same size on both computers, and Disk2 etc....

[garycase]

Note: you need to match disk capacities up exactly if you have different sized disks in your system.  Disk1 must be the same size on both computers, and Disk2 etc....

 

That's not a requirement unless your goal is to mirror the specific disks on the array (as you're doing).

 

If you simply backup the shares, it doesn't matter whether the disk sizes are the same, or even if the number of disks is the same.    I do this with a backup server that has a significantly different disk complement than my main media server.  I simply run a Syncback profile (from a Windows machine) that backs up \\Media\DVDs  to  \\XtraBakups\DVDs

 

[it also does a few other shares, but that's the major share, and you get the idea.]

 

 

[tucansam]

My $0.02.

 

I decided that, while a major PITA, I can theoretically re-encode all of my DVDs and BluRays should my server blow up.  Plus they take up, by far, the most room on the server (and in boxes in the garage, ha!).  So my backups are only of the family's critical files, homework, legal papers, taxes, family photos, etc.  Stuff that is literally not replaceable.

 

I have two unraid servers.  One is full of 4TB drives, a major investment for me.  My second server is full of older disks, and when I upgrade in my main server, the second server gets the old drive (ie, I started out with 120 and 320GB drives and now have a few 3TB drives to add when it comes time).

 

I use rsync to copy the critical stuff over to the second server.  It has just one share, simply, "backup."  I can bring it online in a pinch if the main server is dead and I need data *right now*, or I can use it to copy stuff back.

 

rsync was a major PITA for me, and I have been unable to automate it; it requires near constant babysitting from me, and likes to give me errors and trouble.  I don't have a lot of free time anymore, so maybe some day I'll be able to get it all working.  For now, once a week (or sooner if something major is added to the main server, like a gig of wedding photos or something), I run a bunch of rsync commands to get the data moved.

 

I like the idea of moving data with Windows, except that, in my experience, Windows copies everything to itself first, and then the second machine, acting like a conduit for the data.  So, if I copy/paste from server1 to server2 using Explorer, the process slows because my Windows system is currently on a powerline ethernet adapter.  rsync lets the two servers, sitting side-by-side, copy at gig speeds.  Once I get my house wired, this won't be so much of an issue, but for now...

 

As for off-site, I would love to be able to do that.  I have toyed with the idea of setting up a server at my folks' house, so they can have the benefits of unraid, and each household can backup to the other's server.  But even with an initial rsync copy, continuous weekly copying of even just differential data is going to eat up the "allowed bandwidth" our retarded ISPs allow.  I'd be getting over quota emails constantly, by the time the kids have watched a few Netflix streams and I've downloaded a bunch of new ISOs.  So off-site remains elusive for me, but my eventual goal.

 

My final idea, as has been mentioned, is to simply carry the server to another location (my locker at work for example).  This was the idea when I built the second server using an ammo can.  But I've yet to actually do it.  Yep, my fault.

[DaleWilliams]

This was the idea when I built the second server using an ammo can. 

[tucansam]

This was the idea when I built the second server using an ammo can. 

 

 

Yep.  Works fantastic.

[garycase]

I looked at your pictures -- the ammo can server is indeed unique 

[Falc410]

I'm planning to build 2 servers for my dad, although I don't have much experience with unRaid yet as I have only been reading forums so far.

 

I'm planning to build him a small 6-8TB machine with needs to be backed up occasionally. The Backup server will only be switched on during backup and otherwise stays off. I was also thinking of using unRaid for this server but can I use rsync to backup a complete Share (which might be more than 4TB) to another share on another server (that is automatically distributed over 2 disks)? Or do I need to backup the harddrives individually (like in the script on the 1st page shown in this thread). Than I couldn't quite mix and match disks anymore.

 

Also, do I need a parity disk in the backup machine? Since I don't want too spend too much money, I was hoping to to not have one. Or maybe I should just use some other OS which offers me the ability to configure the HDDs as JBOD.

[garycase]

... do I need a parity disk in the backup machine? Since I don't want too spend too much money, I was hoping to to not have one.

 

There's no requirement that you assign a parity disk.  Of course if you don't use one, then the backup system won't be fault-tolerant.  On the other hand, writes to it will be much faster, since maintaining the integrity of parity is why writes are relatively slow on UnRAID (since each write requires 4 disk operations).

 

 

... Or maybe I should just use some other OS which offers me the ability to configure the HDDs as JBOD.

 

UnRAID without a parity disk is essentially that.  You can create a single share (e.g. "MyBigDisk") that uses all of the disks for its storage space.

[superloopy]

Why wouldn't you 'parity' a backup server?

 

Surely if the need ever arose for you to recreate those backups you might not be able to do so, hardly a worthy backup in my opinion.

 

We have all been there ..... spend the dosh and don't cut corners. UnRaid must be about the cheapest way to maintain a viable backup.

 

 

[Falc410]

Thanks for the answer. I might include a parity disk at a later time, but since the backup system should only run like once in a week (the data is not changed that often, so maybe even bi-weekly) there should be little usage of the harddrives. It's highly unlikely that they will fail right away when I need to do a restore. It's always a cost vs risk tradeoff. If I had more money I would maybe even build a 3rd unRaid server...but I just calculated the total cost and I'm already way over 1000 Euros....

[superloopy]

It's highly unlikely that they will fail right away when I need to do a restore .....

 

But they WILL fail and you cannot predict in what circumstance .... every drive does, and when you need it, it won't be there. That's when a parity drive will be a godsend. Think on my friend.

[garycase]

I certainly agree a parity drive is a good idea.

 

On the other hand, it's also true that the likelihood of 3 simultaneous drive failures -- which is what would be needed to lose data [2 failures on the main UnRAID server and one failure on the backup server]  is relatively low        Assuming, of course, that you "train" your dad to pay attention to the status of the main server (or install an e-mail notification add-on so you will KNOW when a drive has failed) AND that you pay attention to the status of the backup server as well (again, an e-mail notification would be best).

 

 

 

[superloopy1]

I know this an old thread but i've read it all and think i understand around 75% of it. What i'm after is the exact rsync command which will allow me to sync a SHARE, not a disk, between two servers. Once armed with this i'm sure i can take the script outlined here and give it a whirl. I have two unRaid servers, the backup being around 50TB of unused space after i've upgraded the primary. Seems sense to copy across on a regular basis. Does rsync just copy many changes, allowing the time taken to sync to decrease after the initial sync up? Maybe i'll drop my understanding level to 60% after that question eh!