Help me find the most reliable way to sync to servers across the world

[tr0910]

One behind the Great Firewall of China, and one in the USA.  (250gb / 30,000 files mostly static Word, Excel and Image files)

 

I have tried rsync over ssh, rsync inside VPN.  Both require mounting the disk from the other side of the world, and it works, but then it stumbles and requires the VPN to be reset, or some other maintenance required.  And it is terribly slow.  Since the files are mostly static, the speed isn't a big deal, but I need something that won't stop working no matter what.  I can't babysit this.

 

rsync -avPX /mnt/disks/192.168.13.142_Backups/USAFiles/ /mnt/USAFiles/

 

Things I haven't tried

- rsync daemon

- bit torrent

 

[Lebowski]

I use BT Sync, I use it to backup / sync my parents data who are in Holland and I'm in Australia. You can use a relay/tracker server that may get around the firewall issues you might encounter?

 

 

[tr0910]

I use BT Sync, I use it to backup / sync my parents data who are in Holland and I'm in Australia. You can use a relay/tracker server that may get around the firewall issues you might encounter?

 

The firewall issues in China are varied, and frustrating.  Mainly they just slow things down, unless you are one of the sites like Facebook, but every day is different behind the Great FireWall.  You can create a VPN between the servers (I use the openVPN plugin for unRaid now, but this adds another layer of complexity to the mix.  The client will drop the vpn connection).  I am hoping for simple and reliable.

 

Is BT Sync as good at handling thousands of small files?  When I think torrents, I think larger zips or such like files.  How much data and how many files are you syncing?

 

What about files that get edited, like a Word document that changes.  Any other pros and cons between BT Sync and rsync??

[Lebowski]

Small files work well, it's a smaller amount of data but heaps of files. 56,654 Files, 22.8GB (They save websites to the HDD so it creates a heap of files)

 

No need for port forwarding or anything like that, relay / tracker servers solve this I guess. But we don't have anything trying to block it.

 

For files like Word, the temp files that Word creates sync while the file is open and just update at each end once done/saved. I don't think it does complete file transfer not deduplication for file updates.

 

As for being a Linux noob I cant compare it to rsync.

[tr0910]

As for being a Linux noob I cant compare it to rsync.

 

Well from one linux noob to another we'll figure it out.  Where did you install it from? 

 

Squid's Community Applications is blocked here in China.  I will have to install it manually and give it a whirl.  Sounds like your use is enough like mine that it should be comparable.  I have preseeded both sides with full copy of files.  It will only be the differences that I am dealing with.

 

And I am not needing the sync to be a 2-way sync.  Oneway is fine.

 

USAfiles -> China

Chinafiles -> USA

[Lebowski]

I use BTsync docker, using Limetechs template built into unRAID. So you should be able to get it?

 

If not the template is here unless github is blocked?

 

https://github.com/limetech/docker-templates

 

I use one way sync, as I don't need to edit my parents files.

 

I was misleading earlier about port forwarding, you do need to port forward. It was working via UPnP and I didn't notice. I turned off UPnP on my router just now and manually forwarded 5555 UPD/TCP.

 

You might be able to configure to another port not blocked? possibly port 80 externally to 5555 even? You can use a proxy also but I don't know how to set that up. Others here would have better advice to get around that.

 

EDIT: my parents machine uses a windows install of BTSync.

 

 

 

 

[tr0910]

I was misleading earlier about port forwarding, you do need to port forward. It was working via UPnP and I didn't notice. I turned off UPnP on my router just now and manually forwarded 5555 UPD/TCP.

 

You might be able to configure to another port not blocked? possibly port 80 externally to 5555 even? You can use a proxy also but I don't know how to set that up. Others here would have better advice to get around that.

 

Just got my dd-wrt router installed to replace the Chinese one.  I have port 5555 forwarded but nothing else.  Testing inside China for now until I get it working.  (so there is no Chinese firewall issues to worry about)

 

When I try and sync between a W10 laptop and unRaid, I get a never ending

 

Connecting...

 

dialog.  Might you have other ports forwarded as well??  I notice a 3838 port in the config as well as the 5555. 

 

Just to validate, I do have 2 Win10 laptops running sync successfully.  So I think it is something in the unRaid Docker setup.

 

In unRaid I have only have added a new mapping of container volume to host path of. 

 

/data          /mnt/disk1

 

All else is set up as per default.

[Lebowski]

I just have 5555 forwarded. (Not sure what 3838 is for)

 

My path setting is /mnt/user/

 

should at least work on your internal lan?

 

[tr0910]

I just have 5555 forwarded. (Not sure what 3838 is for)

 

My path setting is /mnt/user/

 

should at least work on your internal lan?

 

Yes, it should work on the internal lan.  So in frustration, a full reboot was applied to all computers involved.

 

That fixed it over the local lan.  Now we try more global tests.

[Lebowski]

I have had issue like that, normally restating the docker fixes that. (only on first setup or change to share, seems rock solid after that)

[tr0910]

I have had issue like that, normally restating the docker fixes that. (only on first setup or change to share, seems rock solid after that)

 

I have completed some initial global testing between the USA and China , and it is definitely faster to move files through the Great Firewall this way.  I am getting from 3x to 8x the speed I was getting with rsync.  Of course rsync was tested a few days ago so it is not a fair comparo. 

 

Further testing is required to see if it can handle syncing 50,000 files and hundreds of GB.

[tr0910]

We need to figure out why it won't work with pre-populated data...  It has been syncing for 3 days now my WAN ip address has changed every day.  So it seems resilient...

 

http://lime-technology.com/forum/index.php?topic=40654.msg475496#msg475496