Questions about backups that I hope aren't dumb

[TSM]

Hello Folks, 

 

On Christmas Day last year I narrowly averted a minor tragedy.  2 data drives came up with the dreaded Red X at the same time!!!!!  After extensive troubleshooting, and looking through the forums, I think I've resolved the problem and my server is now back stable.  

 

I'd been considering setting up a backup server for years and never pulled the trigger, but the experience of having 2 data drives fail at once, pretty much nailed it for me that I need to do something.  

 

I'm trying to architect this in my mind now. 

 

I've pretty much decided that the duplicacy docker is probably the right way to go with software.  Compression, deduplication, encryption look good only saves deltas on changed files, and everything I've read so far, people claim it's easy to use.  But I'm wondering how powerful the server it's running on reasonably needs to be.  

 

Currently I have roughly 130TB of data on my array.  Does all that need to be backed up?  Probably not, but I'm going to do it anyway!!!  The cpu on my current unraid server is a 2.9ghz dual core celeron.  I don't want it to take months to complete the initial backup.  I've had difficulty finding hardware requirements for duplicacy.  Should I upgrade that cpu?  Or should I put the new more powerful processer in the backup server?  Should I put a better processor in both?  And when considering what processor to put in which server, does it matter which server the actual duplicacy docker is running on?  Should it run on the primary or the backup?  Or does it make a difference?   

 

Thanks in advance for any advice,

[ConnerVT]

Backup strategy is very personal.  Everyone has a different opinion of what is valuable, and what is irreplaceable.

 

So the first thing to do is assess what of your 130TB is irreplaceable.  Is the data business or personal?  For me, it is personal data.  Photos, personal movies, important documents and the like.  Backups of my personal daily driver computer.  I have backup up (with Macrium) that system where I can easily re-image over 1TB of data.  While I would miss my media library, built up over a number of years, the cost to back it up is more than I am willing to invest a backup system to do.  I would rebuild if all was lost.

 

A bigger concern is where, physically, you store the backup.  It does no good to have it in a system sitting next to the current one if some major tragedy happens at your site.  I back up my backups, on drives in USB enclosures.  Roughly each month, I swapped drives and keep one in my desk at work.

 

Backup to a system at your location is certain to be faster than cloud backups.  While the initial backup will take some time (especially if you chose to back up all 130TB) at least you don't need to sit and watch while it takes place.  Afterward, a periodic incremental backup can be scheduled to keep the backup in sync.  My computers back up to my server daily.  Weekly, Unraid backs up my appdata and VMs, followed by a backup to the aforementioned USB external drives.

[TimTheSettler]

I used to do what ConnorVT does today and it's an easy, effective, and relatively cheap way to keep backups.  But, as he points out, you'll need to decide what to back up.  If you do the external storage thing then you'll be limited by the size of that external storage.

 

You're concerned about the CPU but the bottleneck you should be worried about is the hard drive speeds and and the network (if that's involved).  It took me a couple days to create the initial backup of my movies and TV shows (about 16TB).

[ConnerVT]

Obviously one isn't backing up 130TB to USB external HDDs.  😆

 

But TimTheSettler is correct - Drive and network throughput is likely the limiting factor.  Especially painful for non-fiber Internet users and cloud backup solutions.

[TSM]

3 hours ago, ConnerVT said:

Obviously one isn't backing up 130TB to USB external HDDs.  😆

 

But TimTheSettler is correct - Drive and network throughput is likely the limiting factor.  Especially painful for non-fiber Internet users and cloud backup solutions.

Thanks folks.  Yeah, I know that just with straight file copies you don't need much processing power.  Thus why I'm currently using a celeron.👍 But I guess I was thinking that if using a proper backup application, that's performing intelligent operations on the files like deduplication, compression, encryption,, determining changes for an incremental, that it might be more cpu dependent.  

[TimTheSettler]

I use Vorta (Borg) so I can't really give you a proper comparison of Duplicacy.  Borg was recommended to me by a Linux friend who uses it at work to back up their servers.  It's free which is nice but I also like paid apps because it guarantees that the app will be supported and updated regularly.

 

Anyway, I installed Vorta on my test server which is an old machine with an old CPU and old hard drives that I used to use as offline backups (similar to what ConnorVT does today).

 

The screenshots below were taken during the backup process (back up 212GB from a folder on that machine to another folder on that same machine).  They show that the backup software doesn't really tax the system.  It's been running for about 30 minutes now and it's about 25% done.

 

If your CPU is about as powerful as mine on this server (my test server - which is using an AMD A10-5800K CPU) then you should be ok.  However, one of the things that I like about unRAID is that you can change the hardware.  For example, I had an old system lying around and I tried to use it but it was too slow (Core2Duo).  The CPU was always at 50% even though nothing was running.  I went out and bought a new mobo, CPU, and memory and installed it all into the old box and hooked up the hard drives.  Turned on the machine and unRAID picked up where it left off (Upgrading hardware in an existing set-up).

 

So my recommendation here is to build a system with what you already have and then upgrade the different parts as needed once you see it in action.

 

 

Ignore the errors you see for the Parity 2 drive which is slowly starting to fail but still works ok.

 

 

Edited January 14, 2023 by TimTheSettler

[TimTheSettler]

Some results following the completion of the archive process.  Originally 212GB.  Compressed and deduplicated down to 211.4GB