Guide: How To Use Rclone To Mount Cloud Drives And Play Files


DZMM

Recommended Posts

@DZMM

 

I'm so close, yet so far... the upload script runs fine, without any errors, but  for some reason "nothing" is happening... 

I see folders and files in this folder:

/appdata/other/rclone/remotes/gdrive_media_vfs/gdrive/a24321dsfasdfafadsf132/

 

but those are physical files on my HD, taking up about 200GB of space. 

Link to comment
4 hours ago, axeman said:

My unmount wasn't working because of the checker file name... i added a * to the end of the find command. committed in my fork and did a pull request for you. my first ever pull request woo. 

That's weird as the version 0.9.2 on my machine had it but github had a weird 0.9.1 version.  Anyway thanks for spotting and I've fixed - other people with this problem please update unmount script to 0.9.2

 

#!/bin/bash

#######################
### Cleanup Script ####
#######################
#### Version 0.9.2 ####
#######################

echo "$(date "+%d.%m.%Y %T") INFO: *** Starting rclone_cleanup script ***"

####### Cleanup Tracking Files #######

echo "$(date "+%d.%m.%Y %T") INFO: *** Removing Tracking Files ***"

find /mnt/user/appdata/other/rclone/remotes -name dockers_started* -delete
find /mnt/user/appdata/other/rclone/remotes -name mount_running* -delete
find /mnt/user/appdata/other/rclone/remotes -name upload_running* -delete
echo "$(date "+%d.%m.%Y %T") INFO: ***Finished Cleanup! ***"

exit

 

Link to comment
41 minutes ago, axeman said:

@DZMM

 

I'm so close, yet so far... the upload script runs fine, without any errors, but  for some reason "nothing" is happening... 

I see folders and files in this folder:

/appdata/other/rclone/remotes/gdrive_media_vfs/gdrive/a24321dsfasdfafadsf132/

 

but those are physical files on my HD, taking up about 200GB of space. 

what settings do you have for your mount and upload scripts?

Link to comment
13 minutes ago, DZMM said:

what settings do you have for your mount and upload scripts?

Mount:
# REQUIRED SETTINGS
RcloneRemoteName="gdrive_media_vfs" # Name of rclone remote mount WITHOUT ':'. NOTE: Choose your encrypted remote for sensitive data
RcloneMountShare="/mnt/user/mount_rclone" # where your rclone remote will be located without trailing slash  e.g. /mnt/user/mount_rclone
MergerfsMountShare="/mnt/user/mount_mergerfs" # location without trailing slash  e.g. /mnt/user/mount_mergerfs. Enter 'ignore' to disable
DockerStart="" # list of dockers, separated by space, to start once mergerfs mount verified. Remember to disable AUTOSTART for dockers added in docker settings page
LocalFilesShare="/mnt/user/Videos" # location of the local files and MountFolders you want to upload without trailing slash to rclone e.g. /mnt/user/local. Enter 'ignore' to disable
MountFolders=\{"downloads/complete,downloads/intermediate,downloads/seeds,movies,tv,Videos/Animation_shorts,Trailers"\} # comma separated list of folders to create within the mount


Upload:
# REQUIRED SETTINGS
RcloneCommand="sync" # choose your rclone command e.g. move, copy, sync
RcloneRemoteName="gdrive_media_vfs" # Name of rclone remote mount WITHOUT ':'.
RcloneUploadRemoteName="gdrive_media_vfs" # If you have a second remote created for uploads put it here.  Otherwise use the same remote as RcloneRemoteName.
LocalFilesShare="/mnt/user/Videos" # location of the local files without trailing slash you want to rclone to use
RcloneMountShare="/mnt/user/mount_rclone" # where your rclone mount is located without trailing slash  e.g. /mnt/user/mount_rclone
MinimumAge="15m" # sync files suffix ms|s|m|h|d|w|M|y
ModSort="ascending" # "ascending" oldest files first, "descending" newest files first

#kept all the rest default, except this stuff:

# Is this a backup job
BackupJob="Y" # Y/N. Syncs or Copies files from LocalFilesLocation to BackupRemoteLocation, rather than moving from LocalFilesLocation/RcloneRemoteName
BackupRemoteLocation="backup" # choose location on mount for deleted sync files
BackupRemoteDeletedLocation="backup_deleted" # choose location on mount for deleted sync files
BackupRetention="2M" # How long to keep deleted sync files suffix ms|s|m|h|d|w|M|y

 

Link to comment
23 minutes ago, DZMM said:

I'm so close, yet so far... the upload script runs fine, without any errors, but  for some reason "nothing" is happening... 

 

and the files you are upload aren't appearing in the rclone mount?

 

23 minutes ago, DZMM said:

/appdata/other/rclone/remotes/gdrive_media_vfs/gdrive/a24321dsfasdfafadsf132

 

no remote files should be going there - it's only used for the script checker files, so I've no idea how you managed that.  Maybe when you were trying to setup you did something odd??

 

My advice is to unmount everything, delete /appdata/other/rclone, delete any residual files and folders out of /mnt/user/mount_rclone and /mnt/user/mount_mergerfs and start with a fresh run.  Lots of people have problems initially when they don't set it up right.  I think you know what you are doing now, so a clean start is probably best.

Link to comment

 

38 minutes ago, DZMM said:

and the files you are upload aren't appearing in the rclone mount?

 

no remote files should be going there - it's only used for the script checker files, so I've no idea how you managed that.  Maybe when you were trying to setup you did something odd??

 

My advice is to unmount everything, delete /appdata/other/rclone, delete any residual files and folders out of /mnt/user/mount_rclone and /mnt/user/mount_mergerfs and start with a fresh run.  Lots of people have problems initially when they don't set it up right.  I think you know what you are doing now, so a clean start is probably best.

/appdata/other/rclone has the service_accounts folder. I kept that and deleted everything else. The rest of my settings look OK right? 

 

I have my appdata, mount_mergerfs and mount_rclone set to "cache only" that can't cause this, can it? 

 

After the deletes, it's not detecting rclone:

"26.05.2020 16:26:05 INFO: Checking if rclone installed successfully.
26.05.2020 16:26:05 INFO: rclone not installed - will try again later."

 

I looked and it seems it's just checking for the existence of a file right? or is there more to it? 

Edited by axeman
Link to comment
37 minutes ago, axeman said:

/appdata/other/rclone has the service_accounts folder. I kept that and deleted everything else. The rest of my settings look OK right? 

yes

 

38 minutes ago, axeman said:

After the deletes, it's not detecting rclone:

"26.05.2020 16:26:05 INFO: Checking if rclone installed successfully.
26.05.2020 16:26:05 INFO: rclone not installed - will try again later."

 

I looked and it seems it's just checking for the existence of a file right? or is there more to it? 

 
 
 

the upload script doesn't run if the mount is not running.

Link to comment
17 minutes ago, DZMM said:

yes

 

the upload script doesn't run if the mount is not running.

hmm it is running:

 

26.05.2020 17:23:24 INFO: Creating local folders.
26.05.2020 17:23:24 INFO: *** Starting mount of remote gdrive_media_vfs
26.05.2020 17:23:24 INFO: Checking if this script is already running.
26.05.2020 17:23:24 INFO: Script not running - proceeding.
26.05.2020 17:23:24 INFO: Mount not running. Will now mount gdrive_media_vfs remote.
26.05.2020 17:23:24 INFO: Recreating mountcheck file for gdrive_media_vfs remote.
2020/05/26 17:23:24 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "mountcheck" "gdrive_media_vfs:" "-vv" "--no-traverse"]
2020/05/26 17:23:24 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf"
2020/05/26 17:23:24 DEBUG : mountcheck: Modification times differ by -57m50.100450373s: 2020-05-26 17:23:24.211100396 -0400 EDT, 2020-05-26 16:25:34.110650023 -0400 EDT
2020/05/26 17:23:24 INFO : mountcheck: Copied (replaced existing)
2020/05/26 17:23:24 INFO :
Transferred: 32 / 32 Bytes, 100%, 138.821 kBytes/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 0.0s

2020/05/26 17:23:24 DEBUG : 4 go routines active
2020/05/26 17:23:24 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "mountcheck" "gdrive_media_vfs:" "-vv" "--no-traverse"]
26.05.2020 17:23:24 INFO: *** Creating mount for remote gdrive_media_vfs
26.05.2020 17:23:24 INFO: sleeping for 5 seconds
2020/05/26 17:23:24 INFO : Encrypted drive 'gdrive_media_vfs:': poll-interval is not supported by this remote
26.05.2020 17:23:29 INFO: continuing...
26.05.2020 17:23:29 INFO: Successful mount of gdrive_media_vfs mount.
26.05.2020 17:23:29 INFO: Check successful, gdrive_media_vfs mergerfs mount in place.
26.05.2020 17:23:29 INFO: dockers already started.
26.05.2020 17:23:29 INFO: Script complete

How do I check that mergerfs is installed? Is it supposed to be a docker? When I check in dockers, I don't see anything (before this, I haven't been using Docker). 

 

We're almost there - i can almost taste it!

Link to comment
41 minutes ago, axeman said:

How do I check that mergerfs is installed?

 

 

41 minutes ago, axeman said:

26.05.2020 17:23:29 INFO: Check successful, gdrive_media_vfs mergerfs mount in place.

 

Or look in your mergerfs location.  Or look in /bin

Link to comment

@DZMM

 

I deleted my rclone remotes and recreated - and now all seems to be fine. Every now and then the upload script seems to crash. Not sure why it's happening. 

 

two more questions.

 

1. my rclone mode is set to "sync" and i'm using the backup = 'Y' option... does "sync' imply that if i go to google drive and manually delete something, it'll delete from my array? So i'm guessing I want "copy". 

 

2. when I look at my mergerfs_mount\videos ... i just see the "MountFolders" that I specified. I thought when I go to the mergerfs_mount, I would see all of my shares/files from local array and google drive? 

 

Thanks - I was really close to just giving up on this - and am almost seeing the light at the end of the tunnel. 

 

Link to comment

@axeman

 

#1 if you delete locally a file that rclone has synced,then rather than delete it on the remote it is moved to the backupdir for your chosen number of days

 

#2 it should show your files.  Are you using krusader? If so, restart it as it has problems. Or try ssh or Windows explorer

Link to comment

@watchmeexplode5 @Kaizac @testdasi and everyone else - I need help testing rclone union please which landed yesterday as part of rclone 1.5.2.

 

https://forum.rclone.org/t/rclone-1-52-release/16718

 

I've created a test union ok and playback seems good - better than mergerfs, although only tried a few files. 

 

It'd be great if we can get this running as I think it'll be easier to support than mergerfs which has been brilliant but must be installed via a docker. We'll also be just using one app.

 

I've encountered one problem so far in that -dir-cache-time applies to local folder changes as well, so a small number is needed to spot any changes made to /mnt/user/local.  I've asked if there's a way to have a long cache for just the remotes.

 

 I've asked for advice here: https://forum.rclone.org/t/rclone-union-cache-time/16728/1

 

My settings so far:

 

[tdrive_union]
type = union
upstreams = /mnt/user/local/tdrive_vfs tdrive_vfs: tdrive_uhd_vfs: tdrive_t_adults_vfs: gdrive_media_vfs:
action_policy = all
create_policy = ff
search_policy = ff
rclone mount --allow-other --buffer-size 256M --dir-cache-time 1m --drive-chunk-size 512M --log-level INFO --vfs-read-chunk-size 128M --vfs-read-chunk-size-limit off --vfs-cache-mode writes tdrive_union: /mnt/user/mount_mergerfs/union_test

 

Edited by DZMM
  • Thanks 1
Link to comment
25 minutes ago, Kaizac said:

@DZMM did you try --vfs-cache-poll-interval duration? Or the normal poll-interval?

 

https://forum.rclone.org/t/cant-get-poll-interval-working-with-union-remote/13353

nope but it defaults to 1m so it wouldn't help - I think rclone looks for new changes based on --dir-cache-time - when I had this set to 720h as usual changes weren't getting picked up - I waited about 5 mins.  With --dir-cache-time 1m they got picked up pretty quickly.

Link to comment

This new rclone union backend feature looks awsome!

 

I just gave it a try with your mount settings, playback starts almost instantly!

 

We'll see how it behave with a few concurrent stream tonight...

 

I saw your posts on rclone's forum regarding hardlinks, fingers crossed!

Link to comment
7 hours ago, DZMM said:

@axeman

 

#1 if you delete locally a file that rclone has synced,then rather than delete it on the remote it is moved to the backupdir for your chosen number of days

 

#2 it should show your files.  Are you using krusader? If so, restart it as it has problems. Or try ssh or Windows explorer

Thanks - backup will be awesome for woopsies.

 

I don't see anything at all. I'm starting to think something is wrong with mergerFS ... I don't see it in my Dockers. Should it be there as a docker? Also whenever I reboot the server, on the first run of the mount script, it says mergerFS not installed and proceeds to "install" it I never see anything in Docker. 

 

Maybe I can be test dummy for the union backend you guys are talking about above? I'm ready, willing, and (somewhat) able to test for you, with some guidance. 

Link to comment
4 hours ago, DZMM said:

nope but it defaults to 1m so it wouldn't help - I think rclone looks for new changes based on --dir-cache-time - when I had this set to 720h as usual changes weren't getting picked up - I waited about 5 mins.  With --dir-cache-time 1m they got picked up pretty quickly.

 

I've just read the full post you linked to and saw Nick's comment - I'm going to try after work removing the -dir-cache-time and having poll set to maybe 1s.  Need to read up a bit first/have a refresher on what both are doing 

Edited by DZMM
Link to comment
5 minutes ago, axeman said:

I don't see anything at all. I'm starting to think something is wrong with mergerFS ... I don't see it in my Dockers. Should it be there as a docker? Also whenever I reboot the server, on the first run of the mount script, it says mergerFS not installed and proceeds to "install" it I never see anything in Docker. 

It doesn't appear in your /dockers page - it's a bit of an odd case.  It has to be re-installed everytime unRAID starts as part of the script.

 

It's why I want to remove it if possible - because unRAID can't support it natively e.g. via a plugin or a 'normal' docker, it's a bit confusing for unRAID users.  Not mergerfs' problem, but just makes it a bit clunky.  Having everything in rclone will be a lot cleaner.

Link to comment

Well after a night of tryout :

 

Read from the union mount is solid, up to 5 concurrent streams without any issue whatsoever, the overnight plex library scan went fine but took slighly more time than when it was reading from Mergerfs. (we're tallking about 30 more seconds which I think is completly ok)

Deep scan of a manually added mkv went faster than Mergerfs though.

I gave a look at the google dev console to check the api key metrics, looks exactly the same than when using mergerfs so I'd say we're safe from rate limits bans (I made new key to be able to tell the difference).

 

Write to the union mount on the other hand is sketchy at least with my current mount settings (same as yours). Couldn't figure a way to let radarr import a movie to the mount. (I need further investigating this, may be a persmission issue) And when trying to mess with mount settings (playing with dir-cache-time mostly) I even managed to prevent myself from writing to the mount via SMB at some point (from my hackintosh AND from a regular Win laptop)

 

Did you have any more luck?

 

Disclaimer : I not a dev, I'm just geek enough to understand your scripts and tweak them to my liking but couldn't write them from scratch. Just so you know I might not be of a big help here...

 

EDIT :

Further testing and wondering...

 

Sonarr seams happier with the union backend, it imports files fine to the union mounted remote but files are being written straight to google drive 🤔

Edited by Tecneo
Avoid double post
  • Like 1
Link to comment
On 5/28/2020 at 9:58 AM, DZMM said:

It doesn't appear in your /dockers page - it's a bit of an odd case.  It has to be re-installed everytime unRAID starts as part of the script.

 

It's why I want to remove it if possible - because unRAID can't support it natively e.g. via a plugin or a 'normal' docker, it's a bit confusing for unRAID users.  Not mergerfs' problem, but just makes it a bit clunky.  Having everything in rclone will be a lot cleaner.

okay .. is that why I can't seem to shutdown the array safely? for some reason that mergerfs never stops, and without a manual kill command, my array refuses to shutdown. 

 

there must be something wrong with my mergerFS setup. it's just not mounting anything at all. mind boggling. 

Link to comment

@DZMM

 

So been trying some tinkering... and noticed something

 

IF I specify LocalFilesShare="/mnt/user/Videos" the script won't upload anything. 

 

However, if i create a new folder at: "/mnt/user/local"  and do this: LocalFilesShare="/mnt/user/local"  and start putting files in there, those files get happily uploaded. Will I need to reorganize my existing videos in to a new folder for the script to upload? Could this be a permission thing? 

 

Thanks!

Link to comment
On 5/19/2020 at 11:24 PM, DZMM said:

Not sure of the context of @watchmeexplode5 advice, but it's best to forget /mnt/user/local exists and focus all file management activity on /mnt/user/mount_mergerfs as there's less chance of something going wrong

Hi, sorry I have a couple of extra questions:

 

1. If I set my download folder to /mnt/user/mount_mergerfs/downloads I will only get around 10 MBps, if I set my download folder to /mnt/local/downloads I will get around 70 MBps. So when you write there's less chance something goes wrong I want to make sure I get more benefits when using mergerfs folder, since my DL speed is sincerely limited if I use that method. Is other experiencing the same and are there any major disadvantages to use local instead?

 

2. How often do you scan library in Plex, and will a "Run a partial scan when changes are detected" result in quickly API ban?

Edited by Bjur
Link to comment
On 5/7/2020 at 2:03 AM, remedy said:

also having this issue. i don't restart my array often so its not a huge problems, but fusermount -uz /path/to/mount alone isnt getting it done, I have to do the ps -ef | grep /mnt/user and kill the rclone pids.

 

I've also been having this issue since moving to mergerfs. I stop all my VM/Dockers first and this is still happening.

 

Once I kill /usr/local/sbin/shfs /mnt/user -disks 63 2048000000 -o noatime,allow_other -o remember=330 then the array stops

Link to comment

Super exiting to see that you're looking at going to an even quicker option with union over merge. Regarding that will you be doing a write-up once you get the kinks ironed out? Truly appreciate your github now and your help in this forum. 

I have two questions that I am hoping someone can help with. 
I have my mergefs and rclone setup thanks to the previously mentioned github by DZMM. Thank you. I wanted to add a second container for 4k Movies and 4K TV.  I created two folders one named 4kmovies the other 4ktv in /mnt/user/mount_mergerfs/name_of_remote

 

I first added 4kmovies and 4ktv to the mount script for folders.
 

I then set up a second sonarr and radarr from different repos (is this the easiest way to do this?) When I set everything up like I did with my original sonarr and radarr the only thing I changed was the port number in the container and then pointed the folder path within the GUI of the container. My problem is when a file is completed and moved they are going into the basic movies or tv folder I created previously. Did I miss a step should I have restarted my mount script? 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.