Guide: How To Use Rclone To Mount Cloud Drives And Play Files


DZMM

Recommended Posts

20 minutes ago, DZMM said:

Yes it does if done correctly.  If you're making changes to the mount outside of rclone that will be the source of your problem.  Or, your settings aren't right - best advice I can give is use my scripts.

Sorry I think I may not have explained my situation properly. I've not been using mergerfs. And without mergerfs, the partial scan doesn't work correct? And now I'm intending to use mergerfs to solve this problem. However, I don't have write access to the drive. It's a shared drive and I only have read access. If the owner makes changes directly to the gdrive, like adding folders etc, will the changes be reflected in the mergerfs folder? So basically I'm not going to be having a local folder that uploads to the drive. I just want to use mergerfs for the updates if that makes sense. I wish to use your scripts except for the local folder part where it uploads to the drive. Will it still work or have I understood this wrongly?

Edited by blizz
Link to comment

On my latest reboot I get this from the second mountscript:

And the mergerfs directory is empty.

 

Script Starting May 14, 2020 11:30.01

Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_mount_sh/log.txt

14.05.2020 11:30:01 INFO: Creating local folders.
14.05.2020 11:30:01 INFO: *** Starting mount of remote googleSh_crypt
14.05.2020 11:30:01 INFO: Checking if this script is already running.
14.05.2020 11:30:01 INFO: Script not running - proceeding.
14.05.2020 11:30:01 INFO: Mount not running. Will now mount googleSh_crypt remote.
14.05.2020 11:30:01 INFO: Recreating mountcheck file for googleSh_crypt remote.
2020/05/14 11:30:01 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "mountcheck" "googleSh_crypt:" "-vv" "--no-traverse"]
2020/05/14 11:30:01 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf"
2020/05/14 11:30:03 DEBUG : mountcheck: Modification times differ by -15m0.133405247s: 2020-05-14 11:30:01.832405247 +0200 CEST, 2020-05-14 09:15:01.699 +0000 UTC
2020/05/14 11:30:05 INFO : mountcheck: Copied (replaced existing)
2020/05/14 11:30:05 INFO :
Transferred: 32 / 32 Bytes, 100%, 19 Bytes/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 1.6s

2020/05/14 11:30:05 DEBUG : 11 go routines active
2020/05/14 11:30:05 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "mountcheck" "googleSh_crypt:" "-vv" "--no-traverse"]
14.05.2020 11:30:05 INFO: *** Creating mount for remote googleSh_crypt
14.05.2020 11:30:05 INFO: sleeping for 5 seconds
14.05.2020 11:30:10 INFO: continuing...
14.05.2020 11:30:10 INFO: Successful mount of googleSh_crypt mount.
14.05.2020 11:30:10 INFO: Mergerfs already installed, proceeding to create mergerfs mount
14.05.2020 11:30:10 INFO: Creating googleSh_crypt mergerfs mount.
14.05.2020 11:30:10 INFO: Checking if googleSh_crypt mergerfs mount created.
14.05.2020 11:30:10 INFO: Check successful, googleSh_crypt mergerfs mount created.
14.05.2020 11:30:10 INFO: Starting dockers.
sonarr
14.05.2020 11:30:10 INFO: Script complete
Script Finished May 14, 2020 11:30.10

Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_mount_sh/log.txt

 

14.05.2020 11:34:57 INFO: Creating local folders.
14.05.2020 11:34:57 INFO: *** Starting mount of remote googleSh_crypt
14.05.2020 11:34:57 INFO: Checking if this script is already running.
14.05.2020 11:34:57 INFO: Script not running - proceeding.
14.05.2020 11:34:57 INFO: Mount not running. Will now mount googleSh_crypt remote.
14.05.2020 11:34:57 INFO: Recreating mountcheck file for googleSh_crypt remote.
2020/05/14 11:34:57 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "mountcheck" "googleSh_crypt:" "-vv" "--no-traverse"]
2020/05/14 11:34:57 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf"
2020/05/14 11:34:58 DEBUG : mountcheck: Modification times differ by -4m55.337592306s: 2020-05-14 11:34:57.169592306 +0200 CEST, 2020-05-14 09:30:01.832 +0000 UTC
2020/05/14 11:35:00 INFO : mountcheck: Copied (replaced existing)
2020/05/14 11:35:00 INFO :
Transferred: 32 / 32 Bytes, 100%, 20 Bytes/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 1.5s

2020/05/14 11:35:00 DEBUG : 11 go routines active
2020/05/14 11:35:00 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "mountcheck" "googleSh_crypt:" "-vv" "--no-traverse"]
14.05.2020 11:35:00 INFO: *** Creating mount for remote googleSh_crypt
14.05.2020 11:35:00 INFO: sleeping for 5 seconds
2020/05/14 11:35:01 Fatal error: Directory is not empty: /mnt/user/mount_rclone/googleSh_crypt If you want to mount it anyway use: --allow-non-empty option
14.05.2020 11:35:05 INFO: continuing...
14.05.2020 11:35:05 INFO: Successful mount of googleSh_crypt mount.
14.05.2020 11:35:05 INFO: Check successful, googleSh_crypt mergerfs mount in place.
14.05.2020 11:35:05 INFO: dockers already started.
14.05.2020 11:35:05 INFO: Script complete
Script Finished May 14, 2020 11:35.05

Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_mount_sh/log.txt

Edited by Bjur
Link to comment
4 hours ago, blizz said:

Sorry I think I may not have explained my situation properly. I've not been using mergerfs. And without mergerfs, the partial scan doesn't work correct? And now I'm intending to use mergerfs to solve this problem. However, I don't have write access to the drive. It's a shared drive and I only have read access. If the owner makes changes directly to the gdrive, like adding folders etc, will the changes be reflected in the mergerfs folder? So basically I'm not going to be having a local folder that uploads to the drive. I just want to use mergerfs for the updates if that makes sense. I wish to use your scripts except for the local folder part where it uploads to the drive. Will it still work or have I understood this wrongly?

Mergerfs just shows what rclone knows about.  If changes are made direct to gdrive, rclone won't know about these until it polls the remote to see if there are new files - that's why your scans are 'failing'. 

 

If you're mounting a remote where changes are made outside of the rclone i.e. your mate is loading files and not rclone, then you need your mate to install plex autoscan or something similar, so that it tells plex/rclone to re-scan the changed directory.  

Link to comment
3 hours ago, Bjur said:

2020/05/14 11:35:01 Fatal error: Directory is not empty: /mnt/user/mount_rclone/googleSh_crypt If you want to mount it anyway use: --allow-non-empty option

there's something in /mnt/user/mount_rclone/googleSh_crypt prior to the mount that you need to check and delete if appropriate.

Link to comment
6 minutes ago, DZMM said:

Mergerfs just shows what rclone knows about.  If changes are made direct to gdrive, rclone won't know about these until it polls the remote to see if there are new files - that's why your scans are 'failing'. 

 

If you're mounting a remote where changes are made outside of the rclone i.e. your mate is loading files and not rclone, then you need your mate to install plex autoscan or something similar, so that it tells plex/rclone to re-scan the changed directory.  

Hey mate! Thanks for the reply again. Rclone does work properly ; my poll interval is set to 5m so the changes show up in the rclone folder. It was just that Plex was unable to see the changes unless I did a manual scan.

 

And I just set up mergerfs and it works! The changes get reflected in the mergerfs folder, even when it is added directly to the remote by the owner. The problem now is still Plex is not seeing it with partial scan, even though it is in mergerfs. I gather from your last reply that I might need to install the plex autoscan plugin to solve this. My only doubt is why even though the file appears in mergerfs, which is "local", it is not appearing in Plex when it should. By right, for local files you don't need plex autoscan, correct?

Link to comment
16 minutes ago, DZMM said:

there's something in /mnt/user/mount_rclone/googleSh_crypt prior to the mount that you need to check and delete if appropriate.

Thanks. If using Krusader nothing is there even when enabling show hidden files. When browsing in shell it showed mountcheck file. Even when deleting it wouldn't mount correctly and show folders/mountcheck in mergerfs. I think I got it working now, but will report back if not.

Thanks.

Link to comment

@watchmeexplode5 Would it make more sense to me to make a Download folder to /local/download instead of /local/REMOTE/downloads. Limit the local folder share to only 1 disk and the completed folders as:

/local/googleFi_crypt/movies

or

/local/googleSh_crypt/series

 

It should then be easy to move the completed files quickly and still have a separate download folder before the remotes.

Does that makes sense?

Link to comment

@Bjur

Shouldn't be any issue to have a single download and that location is fine. In your case with the two mounts, I think it makes sense to have a single download like you listed. 
The scripts (if you leave it default) will auto create /local/REMOTE/downloads folder but feel free to ignore it or edit it to something like this:

 

googleFi_crypt:
MountFolders=\{"movies"\} # comma separated list of folders to create within the mount
googleSh_crypt:
MountFolders=\{"series"\} # comma separated list of folders to create within the mount

 

Edited by watchmeexplode5
Link to comment
4 hours ago, blizz said:

My only doubt is why even though the file appears in mergerfs, which is "local", it is not appearing in Plex when it should. By right, for local files you don't need plex autoscan, correct?

Not sure how Plex scans for changes in a directory.  My setup has sonarr/radarr etc telling plex something has changed which works perfectly.

Link to comment
3 hours ago, Bjur said:

Thanks. If using Krusader nothing is there even when enabling show hidden files. When browsing in shell it showed mountcheck file. Even when deleting it wouldn't mount correctly and show folders/mountcheck in mergerfs. I think I got it working now, but will report back if not.

Thanks.

Krusader has caused problems for other people - it needs restarting after a new mount.

 

I get confused with your filepaths, but if the mount isn't working then the mountcheck file shouldn't be there and needs deleting...the mountcheck file should be on the remote and it's presence is used to check the mount has completed successfully.

Link to comment

hey @DZMM finally got around to testing out your script, but I seem to be hitting a wall. I've gotten the script to run successfully, and it seems at first glance to be working. Problem is, as soon as I try to touch any of the data - aka mount anything, move anything, copy anything, etc - the mouths disconnect and are going away. Any tips on where I can start to look to debug? I know not a lot of detail, hoping to start with debug so I can see what I can find first before bogging you down with lots of info to troubleshoot. 

 

I'll add an edit - I thought it might be relative to the whole mapping scenario you highlighted, but another issue I've noticed before I could even get to that - after a short while the mounts actually disappear. So whether I do anything or not, they seem to disappear after a short time as well. I'm really scratching my head. 

Edited by 1activegeek
Link to comment

@watchmeexplode5: Thanks. So I think I will create a manual download folder in

/mnt/local/Downloads

/mnt/local/Downloads/temp

/mnt/local/Downloads/finished

 

and then let Filebot move them to:

/mnt/local/googleFi_crypt

/mnt/local/googleSh_crypt

 

What will happen when you move files to local folder while an upload is running. Will that be fine or will it mess the upload up?

Link to comment

Okay have you tried it. Just read earlier today on Rclone it was not a good idea, so I just want to make sure before I try it out. I don't won't to mess with the upload I'm running since it will take a long time to finish. Thanks for the help. 

Edited by Bjur
Link to comment
15 minutes ago, Bjur said:

Okay have you tried it. Just read earlier today on Rclone it was not a good idea, so I just want to make sure before I try it out. I don't won't to mess with the upload I'm running since it will take a long time to finish. Thanks for the help. 

I think you're confusing uploading files directly via the mount (which we don't) which is bad if you don't know what you are doing , and via rclone upload (which we do) which is 100% safe.

 

If it didn't work, this whole 69 page thread would be a waste of time!

Edited by DZMM
Link to comment
16 hours ago, DZMM said:

@1activegeek post your rclone config and your mount script please.

 

When you say the mount goes missing, where do you mean?  In dockers, file explorer, shell etc?

[1a_mediaz]
type = drive
scope = drive
team_drive = ~~REDACTED~~
service_account_file = /boot/config/plugins/rclone-beta/sa/1.json
server_side_across_configs = TRUE

Sorry for the lack of details. So I mean as in after I mount it, I can LS and see the contents. A few moments will pass, then if I try and LS on the directory - the folders are gone and hence the "disappear". 

rclone_mount.sh

Link to comment
40 minutes ago, 1activegeek said:

[1a_mediaz]
type = drive
scope = drive
team_drive = ~~REDACTED~~
service_account_file = /boot/config/plugins/rclone-beta/sa/1.json
server_side_across_configs = TRUE

Sorry for the lack of details. So I mean as in after I mount it, I can LS and see the contents. A few moments will pass, then if I try and LS on the directory - the folders are gone and hence the "disappear". 

rclone_mount.sh 9.52 kB · 1 download

1a_mediaz is where your files are, unencrypted?

Link to comment
1 hour ago, DZMM said:

1a_mediaz is where your files are, unencrypted?

Yes, correct. That's just one mount, I have some others I'll work on and/or work on adjusting to point at crypt volume - but for now yes that's the example I'm working with to make sure its working as expected first. Did I miss something about this being reliant on crypt mount? I figured it should work either way as long as the destination is my intended destination. 

Link to comment

Ok, so after some consolidated and focused testing - I think I'm ok. I think I had a mixup of signals, way too much chaos at work, and extreme late nights trying to play with tech and not remembering what I did. 

 

The short of my issue I believe - and to help in case others do similar - is that I was switching between rclone variants, and testing the new shortcuts function, and shifting between running the script in the UR UI vs running in background. The initial tests I think had an issue because of the rclone vs rclone-beta because I was testing the new shortcuts. Then I tried shifting to the beta, and tried simply running the script - a few times I left the window open, but after time I eventually closed the window. This is what I believe led to the "miraculous disappearance" scenario I was left with. I had originally been doing it with the background, but when I hit an issue initially, it was annoying finding the process to kill so I just ran it via the GUI And left it there. That switch is where I believe I messed it up. Right now, I've got it up and running using rclone-beta with shortcuts enabled (that 1a_mediaz is a pool from multiple TD locations under one mount). 

 

Next I'll venture into the territory with the merger_fs stuff, but at the moment that's not a requirement for me. I just want to be sure I've got a solid mount/unmount for my core rclone location. I'll adjust the script for my need to add other mounts beyond the one. This is awesome, thank you for the work and effort. Sorry for the firedrill over nothing!! 

Link to comment

Question: Every time the scripts is run I get the following error on multiple Movies/TV

 

Quote

Duplicate directory found in destination - ignoring

 

I'm not even trying to upload the one listed but somehow they show on the log. Why is it doing this?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.