(SOLVED) Advanced container mapping for Resilio Sync


Congles

Recommended Posts

Hi everyone,

 

I have an obscure question for you. As you can see in the image below, I have a few shares in a waterfall structure. They house about 15 TB of photography and the "Archive" folder has all my offline photography (another ~15TB)

 

image.png.ac694124bbf6bff4555edbdc6b918fc2.png

 

I have 2 other locations I'm constantly syncing the files to using Resilio Sync. (pic below)

 

image.png.c3c4b261cc7d82a34652419b729c6560.png

 

My issue is the files in stage 1-3 get moved through the stages and resilio isn't able to see outside of its individual share, so in effect, if I move 500GB of photos from stage 1_New to stage 2_Import, this results in 500GB being deleted from the first sync and then re-sync'd via the sync sync. Obviously that's a waste of bandwidth.

 

Unfortunately, using a single share is not an option. I need different cache settings for each share and since there's no option to elect a single folder inside a share to use cache (confirmed in this thread) I'm forced to have 3 shares.

 

---

 

So, with my use case out of the way, I'm thinking of creating another share (called "resilio") and then symlinking the 3 folders from above to that location so that the path would look like this:

 

  • /mnt/user/Resilio
    • /1_New
    • /2_Import
    • /3_Processed

 

Then I could create just one Resilio sync job, point it at /mnt/user/Resilio and it'll be able to track the movements of files across the 3 folders.

Does anyone know 1, if this is possible and 2, how to go about it?

 

 

Thanks!

 

Edited by Congles
added solved to topic and changed topic to reflect conversation better
Link to comment

Bump*

 

I have tried a few things since posting, Unfortunately Resilio has got back to me on a support ticket and has informed me that they don't support syncing of symlinks from linux machines.

 

So I'm left now wondering if anyone out there knows a way I could achieve what I need - a single share for resilio to reference, while still having separate unraid shares?

 

Link to comment

I assume Resilio sync is in a docker correct?

 

If so you could add multiple bind mounts to your container as follows.

/mnt/user/1_New -> /somepath/1_New

/mnt/user/2_Import -> /somepath/2_Import 

/mnt/user/3_Processed-> /somepath/3_Processed

 

then tell the resililo sync application to sync the directory /somepath/

If the app also stores state info in a hidden subfolder of the sync directory (such as /somepath/.resillio_sync)

then you would also want to mount it into the container as well. example: /mnt/user/appdata/resillioSync/somepath/resillio_sync -> /somepath/.resillio_sync

 

I have never used this app myself but i imagine this should work.

Edited by primeval_god
Link to comment

Thanks for the suggestion!

 

I didn't realize you could map multiple folders to a "sub" container path in the way you've suggested.

 

After a bit of fiddling around I discovered resilio won't show any container path unless it's a child of the primary hostpath /sync. So in the below screenshot, I had to nest the extra host paths under /sync/photos/<folder> in order to have them appear. (Actually I first tried manipulating the sync.conf file to do it another way and that worked but is messy and don't recommend it to anyone stumbling upon this)

 

image.thumb.png.77d82f4644aecf0b0fa82a194c7dcdbe.png

 

 

 

Before I'm confident, I still have to test whether it will sync properly with a small test folder locally. Resilio's own business support told me it wasn't possible so I'll report back if it's all running smoothly. (They also said syncing files from difference logical disks isn't possible but here we are (Unraid), so I take what they say with some salt)

 

 

 

 

 

 

Link to comment

Just a quick note on your configuration above. Unless necessary its usually not recommended to map the entire array into a docker container (Host Path 2: /mnt/user -> /sync). Usually its preferable to give a container access to only the paths that it really needs, this improves security and lows the risk of an application accidentally touching files it shouldnt. With sync programs like RisilioSync it can be kind of a hassle to first map the individual directories you want to sync in docker and then select them in the applications, which is why some choose to just map everything. 

Link to comment

Thanks for the heads up, I am definitely aware of that. Unfortunately with the way Resilio works and my needs to have one of the shares loaded into cache, I don't have another option.

 

It appears there may be a small visual bug with the way I've set things up currently (as per my previous post) because Resilio now says there's ~200k files needing to be sync'd, but isn't syncing them (because they're already there)... Gotta leave it for a few more days to see if it clears up.

 

If this idea fails I'll probably just have to create a single share "off cache" for the resilio sync and then do a local sync to another on-cache share of just the folders/files I want to be working on

Link to comment
11 hours ago, Congles said:

Thanks for the heads up, I am definitely aware of that. Unfortunately with the way Resilio works and my needs to have one of the shares loaded into cache, I don't have another option.

I am not quite sure i understand this. Assuming that what i first suggested works then it should be possible to only map the individual shares you need to sync (i.e. /mnt/user/3_Processed-> /sync/3_Processed) rather than map the entire array (i.e. /mnt/user -> /sync). Again its not really critical, I just wanted to make clear that i was suggesting mapping all shares individually and removing the blanket mapping.

Link to comment

Hmm, yeah I think I know what you're getting at... it would be a pain to setup because I would effectively have to add every mapped folder I want to sync to, but that's at least a one time setup. Currently I'm waiting to see if the UI file-transfer bug I mentioned earlier goes away before trying anything else.

 

If I understand you correctly, I should set the main container path, /sync, to something like mnt/user/Photo Sync and then set the extra container paths to:

mnt/user/Photo Sync/1_New

mnt/user/Photo Sync/2_Import

...

 

I'll give that a go at some point...

Link to comment
17 hours ago, Congles said:

If I understand you correctly, I should set the main container path, /sync, to something like mnt/user/Photo Sync and then set the extra container paths to:

mnt/user/Photo Sync/1_New

mnt/user/Photo Sync/2_Import

Sort of but not exactly. My suggestion would be to just remove the direct binding to /sync altogether, and leave the bindings to the subfolders of sync the way you have them now. The only potential gotcha i can think of would be what if the application stores anything in the sync folder outside of the individual sync directories you map in.

Link to comment

Oh yep, I did try that but unfortunately it's baked into the container. I also noticed someone discuss it on the linuxserver resilio thread. Basically you can bind /sync to anywhere you like, but all the subfolders for sharing need to be child folders of that /sync location.

Link to comment

Hey @primeval_god, thanks again for all the help on this. I've reached a resolution, albeit not what I set out to do, but a better result:

 

We're now using Syncthing instead of Resilio - which has the bonus of being free but also some recent improvements (since we last tested it about 2 years ago) has meant it's easily twice as fast as Resilio, probably 3 times faster.

 

As for the container mappings, I ended up moving everything to a single share and am going to now look into how to have a single folder within that share, available on the cache for my own working. I suspect this will require another local-sync program to create and maintain a mirror of the data in said folder, on the cache drive. Any thoughts you have there to steer me in the right direction would be appreciated, but I'll do my research and open another thread if I need.

 

Thank you a tonne again!

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.