Syncing Large Amounts of Data

[Big Mike]

So I have a very large media library, which I sync to a TrueNAS box as a backup. At the moment I use RSYNC but have tried Syncthing, but It never seemed to work reliably due to the huge amount of data. Does anyone do something similar? and what do you use?

 

Thanks

[ConnerVT]

For me, rsync and homegrown User Scripts have been the best solution.  There are two different backup strategies going on with my server:

 

-- Back up photos, backups from systems/devices on my network (which had previously been backed up to the server), and Unraid backups (such as appdata and flash drive).  These go onto one of two external USB drives, which I swap occasionally and keep the idle one in my desk at work.

 

-- Backup of my media files (except 4K movies, just not enough storage space at this time) get backed up to another Unraid server.  This backup server sleeps most of the time, only to be woken up every twice a month to mirror sync the media library.  There is just over 18TB of data.  The initial sync takes some time (at 1 Gb/s network speeds) but the subsequent sync is more in the 15-30 minute range, depending how much was added in the previous 2 weeks.

 

Below is a thread I created when I started the backup server build.  The whole thread contains useful conversation, but I linked to the post that has my backup script.  Hope it gives you some ideas.

 

 

[Big Mike]

Thanks, that's certainly "food for thought".

[TimTheSettler]

13 hours ago, Big Mike said:

huge amount of data

 

Define "huge amount of data".  Large number of files/folders or the size of the files?

 

I use syncthing to sync data across four of my servers at different locations.  No problems in general but I have noticed that large files (8GB+) tend to take a while to sync.  For example, 1 8GB file will take much longer than 100 files totalling 8GB.

 

 

[Big Mike]

I'm backing up around 27TB of data, mostly .mp4 and .mp3. The issue I found with Syncthing was it never really finished cataloguing the data, even when given weeks, when it was already copied at the destination. Maybe if I had let it do the initial copy I wouldn’t have had this problem.

 

I do use Syncthing for syncing data to mobile devices, and found it to be great, just not with huge amounts of data.

[Rysz]

Never had any major problems with Syncthing, but I noticed when I skipped updating it for too long performance got a bit impaired at times. 

[TimTheSettler]

18 hours ago, Big Mike said:

Maybe if I had let it do the initial copy I wouldn’t have had this problem.

 

I see what you're saying.  I generally copy the data so that my two folders are identical and then I point syncthing to those folders and from that point on let it manage the synchronization.

 

You might want to give it a try again because it's constantly updated and maybe there was a flaw when you first used it.

 

One last note is that I've come across a couple odd-ball cases where the folder/device gets confused.  It's usually because I tried to do something I shouldn't have (as Rysz points out) like pausing the folder for too long.  If something looks weird then you can recreate the folder (delete it and create it again) or reset the folder using the following curl api call.

 

https://docs.syncthing.net/rest/system-reset-post.html

[Big Mike]

Don’t get me wrong, I’m not dissing Syncthing, I think it’s excellent. I just couldn’t get it to work for me in this specific case. I would have loved to use the recycle folder function in particular.

[LelaHoward]

I also have the same problem, what should I do to solve this problem?