Congles Posted September 30, 2020 Share Posted September 30, 2020 (edited) Hi everyone, I have an obscure question for you. As you can see in the image below, I have a few shares in a waterfall structure. They house about 15 TB of photography and the "Archive" folder has all my offline photography (another ~15TB) I have 2 other locations I'm constantly syncing the files to using Resilio Sync. (pic below) My issue is the files in stage 1-3 get moved through the stages and resilio isn't able to see outside of its individual share, so in effect, if I move 500GB of photos from stage 1_New to stage 2_Import, this results in 500GB being deleted from the first sync and then re-sync'd via the sync sync. Obviously that's a waste of bandwidth. Unfortunately, using a single share is not an option. I need different cache settings for each share and since there's no option to elect a single folder inside a share to use cache (confirmed in this thread) I'm forced to have 3 shares. --- So, with my use case out of the way, I'm thinking of creating another share (called "resilio") and then symlinking the 3 folders from above to that location so that the path would look like this: /mnt/user/Resilio /1_New /2_Import /3_Processed Then I could create just one Resilio sync job, point it at /mnt/user/Resilio and it'll be able to track the movements of files across the 3 folders. Does anyone know 1, if this is possible and 2, how to go about it? Thanks! Edited October 15, 2020 by Congles added solved to topic and changed topic to reflect conversation better Quote Link to comment
Congles Posted October 1, 2020 Author Share Posted October 1, 2020 Bump* I have tried a few things since posting, Unfortunately Resilio has got back to me on a support ticket and has informed me that they don't support syncing of symlinks from linux machines. So I'm left now wondering if anyone out there knows a way I could achieve what I need - a single share for resilio to reference, while still having separate unraid shares? Quote Link to comment
primeval_god Posted October 2, 2020 Share Posted October 2, 2020 (edited) I assume Resilio sync is in a docker correct? If so you could add multiple bind mounts to your container as follows. /mnt/user/1_New -> /somepath/1_New /mnt/user/2_Import -> /somepath/2_Import /mnt/user/3_Processed-> /somepath/3_Processed then tell the resililo sync application to sync the directory /somepath/ If the app also stores state info in a hidden subfolder of the sync directory (such as /somepath/.resillio_sync) then you would also want to mount it into the container as well. example: /mnt/user/appdata/resillioSync/somepath/resillio_sync -> /somepath/.resillio_sync I have never used this app myself but i imagine this should work. Edited October 2, 2020 by primeval_god Quote Link to comment
Congles Posted October 2, 2020 Author Share Posted October 2, 2020 Thanks for the suggestion! I didn't realize you could map multiple folders to a "sub" container path in the way you've suggested. After a bit of fiddling around I discovered resilio won't show any container path unless it's a child of the primary hostpath /sync. So in the below screenshot, I had to nest the extra host paths under /sync/photos/<folder> in order to have them appear. (Actually I first tried manipulating the sync.conf file to do it another way and that worked but is messy and don't recommend it to anyone stumbling upon this) Before I'm confident, I still have to test whether it will sync properly with a small test folder locally. Resilio's own business support told me it wasn't possible so I'll report back if it's all running smoothly. (They also said syncing files from difference logical disks isn't possible but here we are (Unraid), so I take what they say with some salt) Quote Link to comment
primeval_god Posted October 3, 2020 Share Posted October 3, 2020 Just a quick note on your configuration above. Unless necessary its usually not recommended to map the entire array into a docker container (Host Path 2: /mnt/user -> /sync). Usually its preferable to give a container access to only the paths that it really needs, this improves security and lows the risk of an application accidentally touching files it shouldnt. With sync programs like RisilioSync it can be kind of a hassle to first map the individual directories you want to sync in docker and then select them in the applications, which is why some choose to just map everything. Quote Link to comment
Congles Posted October 5, 2020 Author Share Posted October 5, 2020 Thanks for the heads up, I am definitely aware of that. Unfortunately with the way Resilio works and my needs to have one of the shares loaded into cache, I don't have another option. It appears there may be a small visual bug with the way I've set things up currently (as per my previous post) because Resilio now says there's ~200k files needing to be sync'd, but isn't syncing them (because they're already there)... Gotta leave it for a few more days to see if it clears up. If this idea fails I'll probably just have to create a single share "off cache" for the resilio sync and then do a local sync to another on-cache share of just the folders/files I want to be working on Quote Link to comment
primeval_god Posted October 5, 2020 Share Posted October 5, 2020 11 hours ago, Congles said: Thanks for the heads up, I am definitely aware of that. Unfortunately with the way Resilio works and my needs to have one of the shares loaded into cache, I don't have another option. I am not quite sure i understand this. Assuming that what i first suggested works then it should be possible to only map the individual shares you need to sync (i.e. /mnt/user/3_Processed-> /sync/3_Processed) rather than map the entire array (i.e. /mnt/user -> /sync). Again its not really critical, I just wanted to make clear that i was suggesting mapping all shares individually and removing the blanket mapping. Quote Link to comment
Congles Posted October 5, 2020 Author Share Posted October 5, 2020 Hmm, yeah I think I know what you're getting at... it would be a pain to setup because I would effectively have to add every mapped folder I want to sync to, but that's at least a one time setup. Currently I'm waiting to see if the UI file-transfer bug I mentioned earlier goes away before trying anything else. If I understand you correctly, I should set the main container path, /sync, to something like mnt/user/Photo Sync and then set the extra container paths to: mnt/user/Photo Sync/1_New mnt/user/Photo Sync/2_Import ... I'll give that a go at some point... Quote Link to comment
primeval_god Posted October 6, 2020 Share Posted October 6, 2020 17 hours ago, Congles said: If I understand you correctly, I should set the main container path, /sync, to something like mnt/user/Photo Sync and then set the extra container paths to: mnt/user/Photo Sync/1_New mnt/user/Photo Sync/2_Import Sort of but not exactly. My suggestion would be to just remove the direct binding to /sync altogether, and leave the bindings to the subfolders of sync the way you have them now. The only potential gotcha i can think of would be what if the application stores anything in the sync folder outside of the individual sync directories you map in. Quote Link to comment
Congles Posted October 6, 2020 Author Share Posted October 6, 2020 Oh yep, I did try that but unfortunately it's baked into the container. I also noticed someone discuss it on the linuxserver resilio thread. Basically you can bind /sync to anywhere you like, but all the subfolders for sharing need to be child folders of that /sync location. Quote Link to comment
primeval_god Posted October 7, 2020 Share Posted October 7, 2020 Right but docker should still allow you to do bind mounts to children of /sync such as /mnt/user/1_New -> /sync/1_New even if you dont have a /mnt/user -> /sync binding. Quote Link to comment
Congles Posted October 9, 2020 Author Share Posted October 9, 2020 Oh I see what you're saying. I'll give that a try at some point in the next week or so. For now it'll have to wait as I have about 10 sync jobs I would need to redirect before removing the /sync binding. Quote Link to comment
Congles Posted October 15, 2020 Author Share Posted October 15, 2020 Hey @primeval_god, thanks again for all the help on this. I've reached a resolution, albeit not what I set out to do, but a better result: We're now using Syncthing instead of Resilio - which has the bonus of being free but also some recent improvements (since we last tested it about 2 years ago) has meant it's easily twice as fast as Resilio, probably 3 times faster. As for the container mappings, I ended up moving everything to a single share and am going to now look into how to have a single folder within that share, available on the cache for my own working. I suspect this will require another local-sync program to create and maintain a mirror of the data in said folder, on the cache drive. Any thoughts you have there to steer me in the right direction would be appreciated, but I'll do my research and open another thread if I need. Thank you a tonne again! Quote Link to comment
primeval_god Posted October 16, 2020 Share Posted October 16, 2020 Glad to help and glad to hear your found something that works for you. I have used Syncthing myself for quite some time, though for a much different application. I have been really happy with Syncthing's performance. 1 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.