[SOLVED] Backup / Sync Options - Unraid1 to Unraid2

[DivideBy0]

What good reliable options we have to backup one unraid to another over the network, not everything but some specific folders.  Kind of like install it, set it and leave it.  Command line options, gui, apps, plugins, can you please share from experience.  I want something reliable and let it run. I don't have time or patience to thinker with stuff so I need something that really works   Files check integrity during the backup is a must have  

 

Thanks bunch

Edited June 28, 2020 by johnwhicker

[DivideBy0]

I see a script option.  Anything else?

 

 

[1812]

cloudberry and nextcloud  duplicati are worth a look at. I use cloudberry.

Edited June 27, 2020 by 1812

[danioj]

Are you after backup or file sync? Each have their own use cases and are different (while they might help serve similar goals in some cases).

If it’s the former then I use Duplicati (Docker available)
If it’s the latter then I use Syncthing (Docker Av/liable)

Both are stable and sold. Duplicati is true backup in that it has versioning, archiving etc. Syncthing is best used to keep locations in Sync and can be wowed for backup as it’s bi directional.

Not sure what feature in Nextcloud you could use to serve as your backup solution unless there is a app you can install within it that does something similar to the above.

[DivideBy0]

4 minutes ago, danioj said:

Are you after backup or file sync? Each have their own use cases and are different (while they might help serve similar goals in some cases).

If it’s the former then I use Duplicati (Docker available)
If it’s the latter then I use Syncthing (Docker Av/liable)

Both are stable and sold. Duplicati is true backup in that it has versioning, archiving etc. Syncthing is best used to keep locations in Sync and can be wowed for backup as it’s bi directional.

Not sure what feature in Nextcloud you could use to serve as your backup solution unless there is a app you can install within it that does something similar to the above.

 

Thanks Dani. I am just looking to keep one directory in sync across 2 Unraid boxes, no need for a sophisticated backup for sure.  My only requirement is a file integrity check during the sync.    Looks like  Syncthing may just do the job.  Let me pock around this one  

[JonathanM]

18 hours ago, johnwhicker said:

I am just looking to keep one directory in sync across 2 Unraid boxes, no need for a sophisticated backup for sure.

So you have backup elsewhere? File sync makes it really easy to mess up, because you expect the second copy to magically understand when to stop syncing. Example, overwrite a document, file sync overwrites the same document. No recovery possible. Ransomware encrypts one end, the other end is encrypted as well. Entire folder deleted? Gone. You say OK, I'll only sync periodically, I can catch any errors before the sync happens and recover. No. No you can't.

 

There are really good reasons to set up file sync, mainly to provide fast local access in 2 distant locations for people who need to actively work on the same set of files.

 

My apologies if you already know all this, but many people think they want sync and haven't really thought through what they are really after.

[DivideBy0]

21 minutes ago, jonathanm said:

So you have backup elsewhere? File sync makes it really easy to mess up, because you expect the second copy to magically understand when to stop syncing. Example, overwrite a document, file sync overwrites the same document. No recovery possible. Ransomware encrypts one end, the other end is encrypted as well. Entire folder deleted? Gone. You say OK, I'll only sync periodically, I can catch any errors before the sync happens and recover. No. No you can't.

 

There are really good reasons to set up file sync, mainly to provide fast local access in 2 distant locations for people who need to actively work on the same set of files.

 

My apologies if you already know all this, but many people think they want sync and haven't really thought through what they are really after.

 

Thanks for the feedback, guidance and possible scenarios.  I always welcome good ideas, scenarios and challenges as it will help design the right approach.

 

Yes I do have a full backup with CrashPlan so that covers me for online backups.

 

This is a media folder (pictures/videos) that I want to replicate on a second UnRaid as an immediate copy & local backup, so that's the reason I asked for possible methods. Call me paranoid but having multiple copies /  backups won't hurt, right?

 

It is my intention to do this only manually and that's the reason I mentioned that the files integrity check is very important.  Here is my high level scenario:

 

Unraid 1 (media folder) ----> Unraid 2 (media folder exact copy) -----> Initial manual sync job with file integrity check during sync process.

Unraid 1 (original media folder + updates) -----> Unraid 2 (original media folder exact copy + deltas/updates) -----> Still manual sync job with file integrity check during sync process but only the deltas (whatever was added to original Unraid 1 media folder. This will be done with manual interaction to understand the deltas.

 

I plan to do this manually as part of my process flow when I upload pictures / videos to Unraid 1 media folder.  The file difference will also be done with manual interaction and nothing automated. 

 

I am not to worried about ransomware as my entire environment is in Mac OSX but of course never say never, right.  Having a manual process with manual interaction in place will help prevent me copying encrypted data across or making any changes unless I understand the deltas and all that. 

 

Most likely I will end up with a customized script or configure Syncthing to do all this for me.  I will come back and post the solution I decided on.

 

[JonathanM]

16 minutes ago, johnwhicker said:

Having a manual process with manual interaction in place will help prevent me copying encrypted data across or making any changes unless I understand the deltas and all that. 

Seems like you have a handle on it. Most people that say they want to sync their media want it done automatically, because they think that's what they want.

 

You have laid out a method that uses sync to manually create a backup, instead of using a backup utility which should accomplish the same thing. Which is cool, but not what is commonly done.

 

By making the process manual, it should be as safe as a backup, with the benefits of keeping only the copies you really need vs. versioning and storing every change. That benefit is offset by the time it takes you to manually manage it.

 

As long as you are happy with the end result, sounds good to me.🙂

[DivideBy0]

3 minutes ago, jonathanm said:

Seems like you have a handle on it. Most people that say they want to sync their media want it done automatically, because they think that's what they want.

 

You have laid out a method that uses sync to manually create a backup, instead of using a backup utility which should accomplish the same thing. Which is cool, but not what is commonly done.

 

By making the process manual, it should be as safe as a backup, with the benefits of keeping only the copies you really need vs. versioning and storing every change. That benefit is offset by the time it takes you to manually manage it.

 

As long as you are happy with the end result, sounds good to me.🙂

Thanks for checking my thinking Sir.  The more eyes on it the better for sure  

[Greygoose]

Borg is another option to look into.

[DivideBy0]

22 hours ago, danioj said:

Are you after backup or file sync? Each have their own use cases and are different (while they might help serve similar goals in some cases).

If it’s the former then I use Duplicati (Docker available)
If it’s the latter then I use Syncthing (Docker Av/liable)

Both are stable and sold. Duplicati is true backup in that it has versioning, archiving etc. Syncthing is best used to keep locations in Sync and can be wowed for backup as it’s bi directional.

Not sure what feature in Nextcloud you could use to serve as your backup solution unless there is a app you can install within it that does something similar to the above.

 

I've got Syncthing installed and configured / talking in both boxes but man, setting up a local directory to be synced between the 2 is not as intuitive as one would think   I am kind of stuck and asked for help on the Syncthing forum.  It shouldn't be that hard to set it up right?

 

 

Edited June 28, 2020 by johnwhicker

[DivideBy0]

Well found the solution to my own problem   You'd could just add extra Host Paths as required, in my case / ROOT and be done with it    Work perfect now.