Backup HDD Sructures/Strategies

[Joseph]

Hey unRAIDers,

 

Long story short, I recently lost contents on a disk that is non recoverable (yes, its really gone.) The plan is to build a backup server soon, but there's a couple of ways to go about it, so I wanted to hear your experiences. When backing up a large multi disk array, is it better to have the same number of disks and have the contents mirrored to the backup like this:

 

unRAID    Backup

Disk1  =>  Disk 1

Disk2  =>  Disk 2

Disk3  =>  Disk 3

and so forth...

 

Vs this:

 

unRAID                   Backup

40TBs on                40 TBs on

10 4TB HDDs  =>   5 8TB HDDs

 

Now that my unRAID is up and running, sifting thru the remaining file structure of the array, trying to figure out what needs to be replaced, has been hit and miss. So it seems in my case the first example backup would be better should another failure like this occur, but it will require duplicate hardware. Any thoughts are welcomed and appreciated.

[DoeBoye]

You could do user share backups, so it really wouldn't matter what the backup server look like, so long as it has the same or more capacity. Would give you more flexibility for the drive selection (So the second scenario would work well).

 

That said, have you considered a cloud backup strategy rather than a whole new backup server? There's some pretty decent dockers (crashplan etc) that would automate the whole process. Unless you already have the parts lying around, it might be more cost effective as well, and you would be getting an off-site backup with versioning...

Edited February 1, 2018 by DoeBoye

[Joseph]

59 minutes ago, DoeBoye said:

You could do user share backups, so it really wouldn't matter what the backup server look like, so long as it has the same or more capacity. Would give you more flexibility for the drive selection (So the second scenario would work well).

Thanks for your ideas...so how would you recover just the contents of one disk and not the entire user share in this case?

 

1 hour ago, DoeBoye said:

That said, have you considered a cloud backup strategy rather than a whole new backup server? There's some pretty decent dockers (crashplan etc) that would automate the whole process. Unless you already have the parts lying around, it might be more cost effective as well, and you would be getting an off-site backup with versioning...

I like the idea of offsite backup... how many years does it take to backup 40+TB across the interwebs?

[trurl]

I use some of my old server disks with Unassigned Devices to make backups of the most important user shares, and rotate these disks so there are "grandfathers". I store the latest 2 of these backups offsite.

 

I also have a backup server (currently out-of-service waiting for time to repair) where I backup the less important user shares. And I don't bother to backup some of my user shares at all. So the backup server doesn't need the capacity. In fact, it is populated with disks that I replaced from my main server when I upsized. Not only does the backup server not have the same capacity, but its disks are mostly smaller than the disks in my main server.

 

So, there are lots of ways to do this. Just decide what is important and make a plan.

 

Different people have different priorities. I am not accumulating as much media as I did at first, so my backup server, which still has good disks, already has most of my stuff backed up. Except for music. My music is part of my offsite backups, along with documents, etc.

[Joseph]

4 minutes ago, trurl said:

I use some of my old server disks with Unassigned Devices to make backups of the most important user shares, and rotate these disks so there are "grandfathers". I store the latest 2 of these backups offsite.

That's a god idea too. I think what DoeBoye said about user shares is making me reconsider how I store my media project files. Instead one lump media share available to all disks in the array, break them down by shares based on my client's last name. Like a share for A-E, F-J, K-O, P-T, U-Z and assign which drives are used for each share. The only problem is, some drives will fill up faster because some clients have larger projects than others.

[trurl]

34 minutes ago, Joseph said:

Like a share for A-E, F-J, K-O, P-T, U-Z and assign which drives are used for each share. The only problem is, some drives will fill up faster because some clients have larger projects than others.

You could divide the alphabet differently. Some letters will need more room than others, S-T could take a whole drive, for example. Maybe a phone book (are there still?) could be a guide, how many pages does each letter take.

[Joseph]

1 minute ago, trurl said:

You could divide the alphabet differently. Some letters will need more room than others, S-T could take a whole drive, for example. Maybe a phone book (are there still?) could be a guide, how many pages does each letter take.

Exactly, but drives filling up unevenly would be based upon project size; so its not uncommon for someone whose last name starts with an unpopular letter to have bigger demands or repeat business more frequently than someone with a more common one.

[trurl]

1 minute ago, Joseph said:

Exactly, but drives filling up unevenly would be based upon project size; so its not uncommon for someone whose last name starts with an unpopular letter to have bigger demands or repeat business more frequently than someone with a more common one.

You would have to adapt the idea to your specific details of course.

[Joseph]

1 minute ago, trurl said:

You would have to adapt the idea to your specific details of course.

*Nod*

[tdallen]

I’ve contemplated running a tree command on each disk prior to doing a backup so you have an inventory of what was on the disk.  For the time being I’m matching shares to disks, it’s less flexible but easier to implement.

[Joseph]

Has anyone had any experiences of running backup software via a VM on unRAID shares? Obviously if the backup server was another unRAID box the data would have to go through the network to get from unRAID to Backup but at least you could do incremental backups and restore files that are missing since last backup. Is that a viable option?

[Vr2Io]

Have a backup unRAID which sync from main, make a script at backup side under plugin "user script".

 

Main (NFS disk share) <--- Backup ( execute rsync script )

 

Usually I will shift out the old disk from main to backup, main side always the new/faster/largest drive. ( backup no parity )

 

i.e.

6TB -> 3TB x2

6Tb -> 6TB

etc

 

I think your backup unRAID haven't enough drive slot, so you arrange those largest disk in backup side.

Edited February 2, 2018 by Benson

[Perforator]

After a recent cable problem I setup a cron job that runs the first of the month listing each disks contents to a separate file. At least this way if something happens in the future i have a somewhat recent list of what was there so i can figure out what is missing.

[DoeBoye]

18 hours ago, Joseph said:

I like the idea of offsite backup... how many years does it take to backup 40+TB across the interwebs?

Ugh. That's a lot of data to backup. Many people will only back up their personal stuff to the cloud and not bother with media (movies, tv, music), so even though their array may be 40TB, usually the cloud backup will be 1 or 2.

 

If you have 40TB of data that needs to be backed up, and if you are considering the Cloud route, some of the services offer the option of mailing them a drive (drives in your case) to get the initial backup done. Might be a faster option for you...

Edited February 2, 2018 by DoeBoye

[Joseph]

33 minutes ago, Perforator said:

After a recent cable problem I setup a cron job that runs the first of the month listing each disks contents to a separate file. At least this way if something happens in the future i have a somewhat recent list of what was there so i can figure out what is missing.

Wish I would have done that! Is that easy to setup?

[Joseph]

21 minutes ago, DoeBoye said:

Ugh. That's a lot of data to backup. Many people will only back up their personal stuff to the cloud and not bother with media (movies, tv, music), so even though their array may be 40TB, usually the cloud backup will be 1 or 2.

That makes sense.

 

22 minutes ago, DoeBoye said:

If you have 40TB of data that needs to be backed up, and if you are considering the Cloud route, some of the services offer the option of mailing them a drive (drives in your case) to get the initial backup done. Might be a faster option for you...

mailing in is another good option if I go the cloud route. Thanks!!

[Joseph]

Since an unRAID backup box would not be a production box...

* Is a cache pool even necessary? It will fill up faster than the amount of data being copied.

* Is it correct to assume 5900rpm drives would be ok?

[trurl]

44 minutes ago, Joseph said:

Since an unRAID backup box would not be a production box...

* Is a cache pool even necessary? It will fill up faster than the amount of data being copied.

* Is it correct to assume 5900rpm drives would be ok?

I put a single SSD cache for apps on mine but haven't needed it so far. I don't cache user shares on my main server either. And since the backup server is only booted up long enough for the incremental backup it wouldn't make any sense to cache since mover would just mean it had to stay powered up longer.

[Joseph]

2 minutes ago, trurl said:

I put a single SSD cache for apps on mine but haven't needed it so far ... it wouldn't make any sense to cache since mover would just mean it had to stay powered up longer.

Good point!

[Perforator]

1 hour ago, Joseph said:

Wish I would have done that! Is that easy to setup?

The hardest part was figuring out the command.

[Joseph]

2 hours ago, Perforator said:

The hardest part was figuring out the command.

If you're up to sharing your chron job, it would be greatly appreciated. But no worries if its something you don't want to do.