SYNC one UNRAID Folder to Multiple Cloud Services

[Rishi.Gupta]

Hello All!

 

I have been using UNRAID for about 3 months now. I love it!

 

One of the things which I wanted to do, and is very important for me, is the ability to two-way synchronize (not only backup) an existing folder in UNRAID with multiple cloud services (Onedrive, google drive and dropbox in m,y case).

 

I have explored many options. RClone seems to come closest, but I am not sure whether it uses the same folder for different services, or makes a new folder for each. Also, from what I have gleaned, it only does ONE-WAY sync, which is not useful for me.

 

Another software which I explored was duplicati, but that seems to just do backups, not syncs. Also, it doesn't backup the files individually, but images, which again is of no use for me in this scenario.

 

Syncthing replicates folders between 2 different REMOTE computers, and local folder synchronization is not in their immediate agenda.

 

Moreover, there seem to be no dockers for google drive/onedrive. 

 

One final option which everyone seems to settle on is a dedicated VM. I am more than happy to do that, as long as it works. I will love if someone could walk me through the broad steps on how to achieve this. Dropbox, for example, does not work with network folders anymore, and breaks after some time with symlinks.

 

So my Question is: Can anyone please suggest a solution (ANYTHING THAT WORKS) so that I can get ONE folder on UNRAID to synchronize IN REAL TIME with MULTIPLE cloud services in a BIDIRECTIONAL manner.

 

I will really appreciate the help.

 

Regards

 

Thank you.

[testdasi]

3 hours ago, Rishi.Gupta said:

Syncthing replicates folders between 2 different REMOTE computers, and local folder synchronization is not in their immediate agenda.

Moreover, there seem to be no dockers for google drive/onedrive.

Syncthing acts as a remote but it still saves data locally. You can set up 2 different Syncthing dockers with different names and folder mappings and unique IP (may not even need unique IP as long as unique ports) and you will be able to sync between 2 local folders (acting as 2 separate remotes).

e.g.: /mnt/user/folder1 -> Syncthing docker A <--- handshake ---> Syncthing docker B <--- /mnt/user/folder2

 

Rclone sync is one-way but rclone mount is a real-time-ish representation if the cloud storage. That means you can integrate rclone to the equation by mapping a rclone mount to the syncthing docker.

e.g.: mount gdrive and onedrive using rclone to /mnt/user/mount_rclone/gdrive and /mnt/user/mount_rclone/onedrive then:

/mnt/user/mount_rclone/ -> Syncthing docker A <--- handshake ---> Syncthing docker B <--- /mnt/user/sync/

 

You can also use VM too but it is extremely convoluted as you mentioned network drive for whatever the f reason is heavily frowned upon by cloud storage providers. You therefore need a vdisk that matches the cloud storage size dedicated for the saving of such data + syncthing to create the remote which then handshake to the "docker B" in the example above.

[Rishi.Gupta]

Thank you @testdasi for this detailed explanation! I was kind of hoping against the 2 syncthing solution but that seems to be the only thing there! Your answer did make it clearer as to how to implement it.

 

A Further queries then:

 

rclone can get data FROM the cloud, so I can use that for that direction of data movement. Are you aware of any solution for pushing data TO gdrive and onedrive? I have managed 2-way sync for dropbox by using an ext4 formatted unassigned drive and the dropbox docker. Syncs perfectly. I can probably set up a sync between my primary UNRAID folder and this drive using the 2 syncthing docker solution. That will work for dropbox.

Any suggestions for gdrive and onedrive?

 

Thanks

[testdasi]

35 minutes ago, Rishi.Gupta said:

rclone can get data FROM the cloud, so I can use that for that direction of data movement. Are you aware of any solution for pushing data TO gdrive and onedrive? I have managed 2-way sync for dropbox by using an ext4 formatted unassigned drive and the dropbox docker. Syncs perfectly. I can probably set up a sync between my primary UNRAID folder and this drive using the 2 syncthing docker solution. That will work for dropbox.

Any suggestions for gdrive and onedrive?

 

Thanks

Read up on rclone mount.

I think you are confused about rclone sync (which is one way, it runs until the sync is complete) and rclone mount (which is a mount point, it runs until the process is killed or the server is shutdown)

 

Essentially rclone mount acts like a local folder, except what you read from the mount point is directly from the cloud and what you write to there is directly to the cloud. You don't use rclone mount to sync. You use syncthing to sync WITH a rclone mount (with = to + from <-- which direction it goes depends on which side syncthing determines to be newer).

 

Obviously, this requires a good Internet connection and enough memory that rclone doesn't get killed due to out-of-memory errors.

 

May be read this topic below on how to integrate rclone to Unraid with DZMM's excellent script. That may make things clearer.

 

Edited May 13, 2020 by testdasi

[Rishi.Gupta]

Hey @testdasi

Thanks so much for your inputs. I was able to get dropbox sync up and running using 2 syncthing installations. Now whatever I write to my xfs UNRAID array first gets synched to my ext4 drive via the 2 syncthing installations, and then to dropbox via the dropbox docker monitoring the ext4 drive. This also  happens in the reverse direction. So thats done and dusted!

Now Ill look at the rclone mount thing and try and get that to work for the other two.

 

Regards