Guide: How To Use Rclone To Mount Cloud Drives And Play Files


DZMM

Recommended Posts

16 minutes ago, axeman said:

This could be a killer setup for a camera DVR, if we can set the upload time separately based on share. Like for a DVR setup, push up to cloud as quickly as possible, in case of a robbery, data has already been pushed to cloud. 

yes, you could create multiple instances of the script to do backups, moves or syncs.  Or, just setup a plain old rclone move on a cron job.

 

17 minutes ago, axeman said:

Also wondering if it's OK to point the mount to a cache drive. So that when we're streaming a movie, it doesn't write to an array share? 

All the streaming is done in RAM.

Link to comment
28 minutes ago, DZMM said:

yes, you could create multiple instances of the script to do backups, moves or syncs.  Or, just setup a plain old rclone move on a cron job.

 

All the streaming is done in RAM.

Thanks - will think about this after getting the videos going. 

 

Now - for the exclude directive, can I put a bunch of --exclude in one comannd? 

Command1 = "--exclude animation/* --exclude tv_series --exclude my_other_folder --exclude this_folder_too" ?

 

I have many folders to exclude (more than I can with the 8 commands). 

Link to comment
1 hour ago, axeman said:

Now - for the exclude directive, can I put a bunch of --exclude in one comannd? 

Command1 = "--exclude animation/* --exclude tv_series --exclude my_other_folder --exclude this_folder_too" ?

Yes you can also mix and match commands

Edited by DZMM
Link to comment
4 hours ago, axeman said:
  1 hour ago, DZMM said:

yes, you could create multiple instances of the script to do backups, moves or syncs.  Or, just setup a plain old rclone move on a cron job.

You could also add your CCTV share as an extra path to your main mergerfs mount. Probably a better idea to have only one upload script running as your upload speed isn't great I.e. easier to control his much upload bandwidth is used

Link to comment
7 minutes ago, DZMM said:

You could also add your CCTV share as an extra path to your main mergerfs mount. Probably a better idea to have only one upload script running as your upload speed isn't great I.e. easier to control his much upload bandwidth is used

awesome... hold that thought though. That's a future problem lol. I don't have my cameras fully writing to a network share yet. in a Post Covid world, I'm looking at obtaining an IP camera system. 

 

OT question - when the hell do you sleep? you seem to be online 24/7

Link to comment

I still think I'm doing something wrong. When I try to list the directories using rclone gives me an error that the team drive doesn't exist. What's funny is that when I tried it earlier I was uploading somewhere I just don't know where those files went. 

 

Hope you are at least enjoying your time while awake. 

Edited by axeman
Link to comment
2 hours ago, DZMM said:

You could also add your CCTV share as an extra path to your main mergerfs mount. Probably a better idea to have only one upload script running as your upload speed isn't great I.e. easier to control his much upload bandwidth is used

So it can't seem to shutdown properly. And even though I have the schedule completely turned off, even after a reboot, I can't seem to start the uploader script. It says the script is already running. 

 

starting to feel like I might have bit off more than I can chew with this setup. 🥵

Link to comment

So I find that the path rclone\remotes\gdrive_media_vfs\gdrive\ has some files in it. Probably the scrap stuff that RClone was working on encrypting before upload?

 

Also see an empty file called "upload_running_daily_upload" is that what's used to see if something is in progress? 

 

 

I was able to figure out the team drive thing... it was because I created the client id outside of the project that has the service accounts. thought I read somewhere that it didn't matter. It does. 

Edited by axeman
Link to comment
5 hours ago, axeman said:

Also see an empty file called "upload_running_daily_upload" is that what's used to see if something is in progress

Yes.  You haven't setup the unmount script properly to remove all the verification files.  Best way to run imo is at array start.  Delete that file manually and the upload will start.

Link to comment

I'm just at a complete loss and I'm completely new at this. I thought I had things figured out as files uploaded and they are seen by plex but since that first batch of a little over 100G of files nothing else has uploaded. These files did that 2 days ago and since then nothing has been uploading. When running the upload script I get the following: 
26.05.2020 01:32:35 INFO: *** Rclone move selected. Files will be moved from /mnt/user/local/gdrive_vfs for gdrive_vfs ***
26.05.2020 01:32:35 INFO: *** Starting rclone_upload script for gdrive_vfs ***
26.05.2020 01:32:35 INFO: Exiting as script already running.
Script Finished May 26, 2020 01:32.35

Full logs for this script are available at /tmp/user.scripts/tmpScripts/Upload Script/log.txt

The thing is nothing is actually uploading from what I can tell.

I have done the following to try and remedy the situation. I moved the upload, mount, and unmount scripts to another folder so that they would not be seen or run when the array started. I then shut down the server and restarted it. After I started the array I went and recreated the mount, upload and unmount scripts and started the mount script with what I think is a success. But when it comes to starting the "New" upload script it shows that its already running. 

 

Trying to figure out what the heck I did wrong originally and where I can start or restart at now. Thanks.

Link to comment
1 hour ago, Hypner said:

26.05.2020 01:32:35 INFO: Exiting as script already running.

You haven't setup correctly the cleanup script properly to delete this or something else went wrong while you were trying to fix everything.

 

Look in /mnt/user/appdata/other/rclone/remotes/whatever_your_upload_remote_is_called and delete whatever file is in there. 


After you've deleted, run the upload script again and it should recreate the file once you've started and delete it when it's finished.  Double-check to make sure.

 

If you end up rebooting or something while an upload is in progress, setting the cleanup/unmount script at array start will make sure all the control files are deleted.

Link to comment
3 hours ago, DZMM said:

Yes.  You haven't setup the unmount script properly to remove all the verification files.  Best way to run imo is at array start.  Delete that file manually and the upload will start.

 

Weird I have it run on startup. It doesn't seem to run at all. I wonder if something got messed up in the copy/pasta. 

 

Any idea why the array doesn't stop? It seems to get stuck on

 Array Stopping•Retry unmounting user share(s)...

 

MergerFS was the culprit. I followed this: 

 

Edited by axeman
Link to comment
4 hours ago, DZMM said:

You haven't setup correctly the cleanup script properly to delete this or something else went wrong while you were trying to fix everything.

 

Look in /mnt/user/appdata/other/rclone/remotes/whatever_your_upload_remote_is_called and delete whatever file is in there. 


After you've deleted, run the upload script again and it should recreate the file once you've started and delete it when it's finished.  Double-check to make sure.

 

If you end up rebooting or something while an upload is in progress, setting the cleanup/unmount script at array start will make sure all the control files are deleted.

I checked in /mnt/user/appdata/other/rclone/remotes/whatever_your_upload_remote_is_called and had no files. Tried to run the upload script again and got the same results telling me the script was already running.  Tried running the cleanup script again and it says it runs fine but afterwards the upload script is telling me its already running. Any other suggestions? reboot? 

 

Correction** There was the upload script still running deleted it and was able to get the uploader script going. I failed to have the cleanup at the start of the array prior so thank you for instruction. Much appreciated. 

Edited by Hypner
Link to comment
3 minutes ago, Hypner said:

I checked in /mnt/user/appdata/other/rclone/remotes/whatever_your_upload_remote_is_called and had no files. Tried to run the upload script again and got the same results telling me the script was already running.  Tried running the cleanup script again and it says it runs fine but afterwards the upload script is telling me its already running. Any other suggestions? reboot? 

 

do you have anything in /mnt/user/appdata/other/rclone/remotes/ ? any other folder? 

Link to comment

@DZMM

 

I'm so close, yet so far... the upload script runs fine, without any errors, but  for some reason "nothing" is happening... 

I see folders and files in this folder:

/appdata/other/rclone/remotes/gdrive_media_vfs/gdrive/a24321dsfasdfafadsf132/

 

but those are physical files on my HD, taking up about 200GB of space. 

Link to comment
4 hours ago, axeman said:

My unmount wasn't working because of the checker file name... i added a * to the end of the find command. committed in my fork and did a pull request for you. my first ever pull request woo. 

That's weird as the version 0.9.2 on my machine had it but github had a weird 0.9.1 version.  Anyway thanks for spotting and I've fixed - other people with this problem please update unmount script to 0.9.2

 

#!/bin/bash

#######################
### Cleanup Script ####
#######################
#### Version 0.9.2 ####
#######################

echo "$(date "+%d.%m.%Y %T") INFO: *** Starting rclone_cleanup script ***"

####### Cleanup Tracking Files #######

echo "$(date "+%d.%m.%Y %T") INFO: *** Removing Tracking Files ***"

find /mnt/user/appdata/other/rclone/remotes -name dockers_started* -delete
find /mnt/user/appdata/other/rclone/remotes -name mount_running* -delete
find /mnt/user/appdata/other/rclone/remotes -name upload_running* -delete
echo "$(date "+%d.%m.%Y %T") INFO: ***Finished Cleanup! ***"

exit

 

Link to comment
41 minutes ago, axeman said:

@DZMM

 

I'm so close, yet so far... the upload script runs fine, without any errors, but  for some reason "nothing" is happening... 

I see folders and files in this folder:

/appdata/other/rclone/remotes/gdrive_media_vfs/gdrive/a24321dsfasdfafadsf132/

 

but those are physical files on my HD, taking up about 200GB of space. 

what settings do you have for your mount and upload scripts?

Link to comment
13 minutes ago, DZMM said:

what settings do you have for your mount and upload scripts?

Mount:
# REQUIRED SETTINGS
RcloneRemoteName="gdrive_media_vfs" # Name of rclone remote mount WITHOUT ':'. NOTE: Choose your encrypted remote for sensitive data
RcloneMountShare="/mnt/user/mount_rclone" # where your rclone remote will be located without trailing slash  e.g. /mnt/user/mount_rclone
MergerfsMountShare="/mnt/user/mount_mergerfs" # location without trailing slash  e.g. /mnt/user/mount_mergerfs. Enter 'ignore' to disable
DockerStart="" # list of dockers, separated by space, to start once mergerfs mount verified. Remember to disable AUTOSTART for dockers added in docker settings page
LocalFilesShare="/mnt/user/Videos" # location of the local files and MountFolders you want to upload without trailing slash to rclone e.g. /mnt/user/local. Enter 'ignore' to disable
MountFolders=\{"downloads/complete,downloads/intermediate,downloads/seeds,movies,tv,Videos/Animation_shorts,Trailers"\} # comma separated list of folders to create within the mount


Upload:
# REQUIRED SETTINGS
RcloneCommand="sync" # choose your rclone command e.g. move, copy, sync
RcloneRemoteName="gdrive_media_vfs" # Name of rclone remote mount WITHOUT ':'.
RcloneUploadRemoteName="gdrive_media_vfs" # If you have a second remote created for uploads put it here.  Otherwise use the same remote as RcloneRemoteName.
LocalFilesShare="/mnt/user/Videos" # location of the local files without trailing slash you want to rclone to use
RcloneMountShare="/mnt/user/mount_rclone" # where your rclone mount is located without trailing slash  e.g. /mnt/user/mount_rclone
MinimumAge="15m" # sync files suffix ms|s|m|h|d|w|M|y
ModSort="ascending" # "ascending" oldest files first, "descending" newest files first

#kept all the rest default, except this stuff:

# Is this a backup job
BackupJob="Y" # Y/N. Syncs or Copies files from LocalFilesLocation to BackupRemoteLocation, rather than moving from LocalFilesLocation/RcloneRemoteName
BackupRemoteLocation="backup" # choose location on mount for deleted sync files
BackupRemoteDeletedLocation="backup_deleted" # choose location on mount for deleted sync files
BackupRetention="2M" # How long to keep deleted sync files suffix ms|s|m|h|d|w|M|y

 

Link to comment
23 minutes ago, DZMM said:

I'm so close, yet so far... the upload script runs fine, without any errors, but  for some reason "nothing" is happening... 

 

and the files you are upload aren't appearing in the rclone mount?

 

23 minutes ago, DZMM said:

/appdata/other/rclone/remotes/gdrive_media_vfs/gdrive/a24321dsfasdfafadsf132

 

no remote files should be going there - it's only used for the script checker files, so I've no idea how you managed that.  Maybe when you were trying to setup you did something odd??

 

My advice is to unmount everything, delete /appdata/other/rclone, delete any residual files and folders out of /mnt/user/mount_rclone and /mnt/user/mount_mergerfs and start with a fresh run.  Lots of people have problems initially when they don't set it up right.  I think you know what you are doing now, so a clean start is probably best.

Link to comment

 

38 minutes ago, DZMM said:

and the files you are upload aren't appearing in the rclone mount?

 

no remote files should be going there - it's only used for the script checker files, so I've no idea how you managed that.  Maybe when you were trying to setup you did something odd??

 

My advice is to unmount everything, delete /appdata/other/rclone, delete any residual files and folders out of /mnt/user/mount_rclone and /mnt/user/mount_mergerfs and start with a fresh run.  Lots of people have problems initially when they don't set it up right.  I think you know what you are doing now, so a clean start is probably best.

/appdata/other/rclone has the service_accounts folder. I kept that and deleted everything else. The rest of my settings look OK right? 

 

I have my appdata, mount_mergerfs and mount_rclone set to "cache only" that can't cause this, can it? 

 

After the deletes, it's not detecting rclone:

"26.05.2020 16:26:05 INFO: Checking if rclone installed successfully.
26.05.2020 16:26:05 INFO: rclone not installed - will try again later."

 

I looked and it seems it's just checking for the existence of a file right? or is there more to it? 

Edited by axeman
Link to comment
37 minutes ago, axeman said:

/appdata/other/rclone has the service_accounts folder. I kept that and deleted everything else. The rest of my settings look OK right? 

yes

 

38 minutes ago, axeman said:

After the deletes, it's not detecting rclone:

"26.05.2020 16:26:05 INFO: Checking if rclone installed successfully.
26.05.2020 16:26:05 INFO: rclone not installed - will try again later."

 

I looked and it seems it's just checking for the existence of a file right? or is there more to it? 

 
 
 

the upload script doesn't run if the mount is not running.

Link to comment
17 minutes ago, DZMM said:

yes

 

the upload script doesn't run if the mount is not running.

hmm it is running:

 

26.05.2020 17:23:24 INFO: Creating local folders.
26.05.2020 17:23:24 INFO: *** Starting mount of remote gdrive_media_vfs
26.05.2020 17:23:24 INFO: Checking if this script is already running.
26.05.2020 17:23:24 INFO: Script not running - proceeding.
26.05.2020 17:23:24 INFO: Mount not running. Will now mount gdrive_media_vfs remote.
26.05.2020 17:23:24 INFO: Recreating mountcheck file for gdrive_media_vfs remote.
2020/05/26 17:23:24 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "mountcheck" "gdrive_media_vfs:" "-vv" "--no-traverse"]
2020/05/26 17:23:24 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf"
2020/05/26 17:23:24 DEBUG : mountcheck: Modification times differ by -57m50.100450373s: 2020-05-26 17:23:24.211100396 -0400 EDT, 2020-05-26 16:25:34.110650023 -0400 EDT
2020/05/26 17:23:24 INFO : mountcheck: Copied (replaced existing)
2020/05/26 17:23:24 INFO :
Transferred: 32 / 32 Bytes, 100%, 138.821 kBytes/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 0.0s

2020/05/26 17:23:24 DEBUG : 4 go routines active
2020/05/26 17:23:24 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "mountcheck" "gdrive_media_vfs:" "-vv" "--no-traverse"]
26.05.2020 17:23:24 INFO: *** Creating mount for remote gdrive_media_vfs
26.05.2020 17:23:24 INFO: sleeping for 5 seconds
2020/05/26 17:23:24 INFO : Encrypted drive 'gdrive_media_vfs:': poll-interval is not supported by this remote
26.05.2020 17:23:29 INFO: continuing...
26.05.2020 17:23:29 INFO: Successful mount of gdrive_media_vfs mount.
26.05.2020 17:23:29 INFO: Check successful, gdrive_media_vfs mergerfs mount in place.
26.05.2020 17:23:29 INFO: dockers already started.
26.05.2020 17:23:29 INFO: Script complete

How do I check that mergerfs is installed? Is it supposed to be a docker? When I check in dockers, I don't see anything (before this, I haven't been using Docker). 

 

We're almost there - i can almost taste it!

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.