Guide: How To Use Rclone To Mount Cloud Drives And Play Files


DZMM

Recommended Posts

On 8/3/2020 at 6:37 AM, Emilio5639 said:

I think I followed the guide pretty well, and I am receiving the following error in the Upload Script:

 

rclone not installed - will try again later.

 

Here is portion of the script that I am using:

 

#!/bin/bash

######################
### Upload Script ####
######################
### Version 0.95.5 ###
######################

####### EDIT ONLY THESE SETTINGS #######

# INSTRUCTIONS
# 1. Edit the settings below to match your setup
# 2. NOTE: enter RcloneRemoteName WITHOUT ':'
# 3. Optional: Add additional commands or filters
# 4. Optional: Use bind mount settings for potential traffic shaping/monitoring
# 5. Optional: Use service accounts in your upload remote
# 6. Optional: Use backup directory for rclone sync jobs

# REQUIRED SETTINGS
RcloneCommand="move" # choose your rclone command e.g. move, copy, sync
RcloneRemoteName="gdrive_media_vfs" # Name of rclone remote mount WITHOUT ':'.
RcloneUploadRemoteName="gdrive_upload_vfs" # If you have a second remote created for uploads put it here.  Otherwise use the same remote as RcloneRemoteName.
LocalFilesShare="/mnt/user/Plex-Media" # location of the local files without trailing slash you want to rclone to use
RcloneMountShare="/mnt/user/mount_Plex-Media" # where your rclone mount is located without trailing slash  e.g. /mnt/user/mount_rclone
MinimumAge="15m" # sync files suffix ms|s|m|h|d|w|M|y
ModSort="ascending" # "ascending" oldest files first, "descending" newest files first

# Note: Again - remember to NOT use ':' in your remote name above

# Bandwidth limits: specify the desired bandwidth in kBytes/s, or use a suffix b|k|M|G. Or 'off' or '0' for unlimited.  The script uses --drive-stop-on-upload-limit which stops the script if the 750GB/day limit is achieved, so you no longer have to slow 'trickle' your files all day if you don't want to e.g. could just do an unlimited job overnight.
BWLimit1Time="01:00"
BWLimit1="off"
BWLimit2Time="08:00"
BWLimit2="15M"
BWLimit3Time="16:00"
BWLimit3="12M"

# OPTIONAL SETTINGS

# Add name to upload job
JobName="_daily_upload" # Adds custom string to end of checker file.  Useful if you're running multiple jobs against the same remote.

# Add extra commands or filters
Command1="--exclude downloads/**"
Command2=""
Command3=""
Command4=""
Command5=""
Command6=""
Command7=""
Command8=""

# Bind the mount to an IP address
CreateBindMount="N" # Y/N. Choose whether or not to bind traffic to a network adapter.
RCloneMountIP="10.0.0.62" # Choose IP to bind upload to.
NetworkAdapter="eth0" # choose your network adapter. eth0 recommended.
VirtualIPNumber="1" # creates eth0:x e.g. eth0:1.

# Use Service Accounts.  Instructions: https://github.com/xyou365/AutoRclone
UseServiceAccountUpload="N" # Y/N. Choose whether to use Service Accounts.
ServiceAccountDirectory="/mnt/user/appdata/other/rclone/service_accounts" # Path to your Service Account's .json files.
ServiceAccountFile="sa_gdrive_upload" # Enter characters before counter in your json files e.g. for sa_gdrive_upload1.json -->sa_gdrive_upload100.json, enter "sa_gdrive_upload".
CountServiceAccounts="15" # Integer number of service accounts to use.

# Is this a backup job
BackupJob="N" # Y/N. Syncs or Copies files from LocalFilesLocation to BackupRemoteLocation, rather than moving from LocalFilesLocation/RcloneRemoteName
BackupRemoteLocation="backup" # choose location on mount for deleted sync files
BackupRemoteDeletedLocation="backup_deleted" # choose location on mount for deleted sync files
BackupRetention="90d" # How long to keep deleted sync files suffix ms|s|m|h|d|w|M|y

####### END SETTINGS #######
 

This is my first time setting something like this up, any help would be appreciated.

Have you installed the rclone plugin?

Link to comment
On 8/4/2020 at 11:05 PM, DZMM said:

All looks good.  The reason I do a tdrive-->tdrive transfer as an extra step, is because I only have one mergerfs mount that combines all my tdrives and local files.  The upload script for the associated local folder uploads all the files to a single tdrive, so behind the scenes I move the files to their correct tdrive. 

 

You could run an upload script for each tdrive, but for my setup that be a pain to traffic manage and the tdrive-->tdrive transfers are mega fast as they are done server side.

That is making more sense to me now. Although i am still slightly confused with the setup still. If you are moving all files to one tdrive, to then move into their correct tdrive, does that mean there is a point where plex would not see the file? Because you point a plex library to the MergerFSMountLocation, and lets say TV shows are in tdrive2. You initially upload to tdrive1.


Also in my first Upload Script would i just change "RcloneUploadRemoteName" to my tdrive i am initially sending my files to before being moved server side.

 

Edited by KeyBoardDabbler
Link to comment
23 hours ago, KeyBoardDabbler said:

Although i am still slightly confused with the setup still. If you are moving all files to one tdrive, to then move into their correct tdrive, does that mean there is a point where plex would not see the file?

 

No, because rclone is moving the files - server side from tdrive_x-->tdrive_y, so it is aware of the change of location.  Because both tdrive_x and tdrive_y are included in my expanded mergerfs mount, plex, sonarr etc aren't aware that the real location has moved - it's the same approach as for when files move from local to gdrive.

 

23 hours ago, KeyBoardDabbler said:

Also in my first Upload Script would i just change "RcloneUploadRemoteName" to my tdrive i am initially sending my files to before being moved server side.

 
 
 

Yes, and then use my extra script or similar to 'sort' the files server side into the right tdrives.

Edited by DZMM
  • Thanks 1
Link to comment
37 minutes ago, KeyBoardDabbler said:

Because you point a plex library to the MergerFSMountLocation, and lets say TV shows are in tdrive2. You initially upload to tdrive1.

 
 

For clarity, here's my flow:

 

1. Mount Tdrive_1, tdrive_2, tdrive_3 etc - do not create a mergerfs mount for tdrive_2 and upwards

2. Create a single mergerfs mount with local folder and the Tdrive_1 mount, tdrive_2 mount, tdrive_3 mount etc

3. Point plex, sonarr etc at the mergerfs mount

4. new files get added to the mergerfs mount's local folder

5. upload script moves ALL files from the local folder to tdrive_1

6. I looked at tdrive_1 and worked what the encrypted name was e.g. for TV_Shows adasfghertdfghthgdf

7. TV_shows should be in tdrive_2, so extra script runs overnight to move tdrive_1/crypt/adasfghertdfghthgdf to tdrive_2/crypt/adasfghertdfghthgdf

8. The move is done by rclone so it adjusts the tdrive_1 and tdrive_2 mounts in real-time, so in mergerfs mount files haven't 'moved' from /TV_Shows and everyone is happy

Edited by DZMM
  • Thanks 2
Link to comment

Hi all. The mount script is not starting the dockers. I see this on the log

 

Quote

mergerfs version: 2.30.0
FUSE library version: 2.9.7-mergerfs_2.30.0
using FUSE kernel interface version 7.31
'build/mergerfs' -> '/build/mergerfs'
05.08.2020 09:30:54 INFO: *sleeping for 5 seconds
05.08.2020 09:30:59 INFO: Mergerfs installed successfully, proceeding to create mergerfs mount.
05.08.2020 09:30:59 INFO: Creating google_vfs mergerfs mount.
05.08.2020 09:30:59 INFO: Checking if google_vfs mergerfs mount created.
05.08.2020 09:30:59 INFO: Check successful, google_vfs mergerfs mount created.
05.08.2020 09:30:59 INFO: dockers already started.
05.08.2020 09:30:59 INFO: Script complete
Script Finished Aug 05, 2020 09:30.59

But the dockers are not started,

 

Quote

DockerStart="plex radarr sonarr-sma radarr-sma" # list of dockers, separated by space, to start once mergerfs mount verified. Remember to disable AUTOSTART for dockers added in docker settings page

 

Link to comment
18 hours ago, bugster said:

Hi all. The mount script is not starting the dockers. I see this on the log

 

But the dockers are not started,

 

 

You need to run the unmount script or manually delete the /mnt/cache/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started file - somehow you've got the scripts running slightly out of sync

Link to comment
23 hours ago, DZMM said:

For clarity, here's my flow:

 

1. Mount Tdrive_1, tdrive_2, tdrive_3 etc - do not create a mergerfs mount for tdrive_2 and upwards

2. Create a single mergerfs mount with local folder and the Tdrive_1 mount, tdrive_2 mount, tdrive_3 mount etc

3. Point plex, sonarr etc at the mergerfs mount

4. new files get added to the mergerfs mount's local folder

5. upload script moves ALL files from the local folder to tdrive_1

6. I looked at tdrive_1 and worked what the encrypted name was e.g. for TV_Shows adasfghertdfghthgdf

7. TV_shows should be in tdrive_2, so extra script runs overnight to move tdrive_1/crypt/adasfghertdfghthgdf to tdrive_2/crypt/adasfghertdfghthgdf

8. The move is done by rclone so it adjusts the tdrive_1 and tdrive_2 mounts in real-time, so in mergerfs mount files haven't 'moved' from /TV_Shows and everyone is happy

Thankyou for explaining the steps, i am going to take another attempt.

 

First i need to get at least 1 mount working using the mergerFS way. Just attempting to mount my original gdrive. Everything seem ok but no files are showing in /mount_unionfs/gdrive_media_vfs.

Revert back to my unionfs mount script and everything works as expected.

Quote

06.08.2020 20:22:56 INFO: Creating local folders.
06.08.2020 20:22:56 INFO: *** Starting mount of remote gdrive_media_vfs
06.08.2020 20:22:56 INFO: Checking if this script is already running.
06.08.2020 20:22:56 INFO: Script not running - proceeding.
06.08.2020 20:22:56 INFO: *** Checking if online
06.08.2020 20:22:57 PASSED: *** Internet online
06.08.2020 20:22:57 INFO: Mount not running. Will now mount gdrive_media_vfs remote.
06.08.2020 20:22:57 INFO: Recreating mountcheck file for gdrive_media_vfs remote.
2020/08/06 20:22:57 DEBUG : rclone: Version "v1.52.2-250-g4f9a80e2-beta" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone-beta/.rclone.conf" "copy" "mountcheck" "gdrive_media_vfs:" "-vv" "--no-traverse"]
2020/08/06 20:22:57 DEBUG : Using config file from "/boot/config/plugins/rclone-beta/.rclone.conf"
2020/08/06 20:22:57 DEBUG : fs cache: adding new entry for parent of "mountcheck", "/usr/local/emhttp"
2020/08/06 20:22:59 DEBUG : mountcheck: Modification times differ by -8m31.432593065s: 2020-08-06 20:22:57.396593065 +1200 NZST, 2020-08-06 08:14:25.964 +0000 UTC
2020/08/06 20:23:01 INFO : mountcheck: Copied (replaced existing)
2020/08/06 20:23:01 INFO :
Transferred: 32 / 32 Bytes, 100%, 17 Bytes/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 1.8s

2020/08/06 20:23:01 DEBUG : 6 go routines active
06.08.2020 20:23:01 INFO: *** Creating mount for remote gdrive_media_vfs
06.08.2020 20:23:01 INFO: sleeping for 5 seconds
2020/08/06 20:23:03 INFO : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
06.08.2020 20:23:06 INFO: continuing...
06.08.2020 20:23:07 INFO: Successful mount of gdrive_media_vfs mount.
06.08.2020 20:23:07 INFO: Mergerfs already installed, proceeding to create mergerfs mount
06.08.2020 20:23:07 INFO: Creating gdrive_media_vfs mergerfs mount.
06.08.2020 20:23:07 INFO: Checking if gdrive_media_vfs mergerfs mount created.
06.08.2020 20:23:07 INFO: Check successful, gdrive_media_vfs mergerfs mount created.
06.08.2020 20:23:07 INFO: Starting dockers.
ombi
06.08.2020 20:23:07 INFO: Script complete
Script Finished Aug 06, 2020 20:23.07

Full logs for this script are available at /tmp/user.scripts/tmpScripts/rClone_Mount/log.txt

2020/08/06 20:23:31 INFO : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)

 

Edited by KeyBoardDabbler
Link to comment
7 hours ago, KeyBoardDabbler said:

Thankyou for explaining the steps, i am going to take another attempt.

 

First i need to get at least 1 mount working using the mergerFS way. Just attempting to mount my original gdrive. Everything seem ok but no files are showing in /mount_unionfs/gdrive_media_vfs.

Revert back to my unionfs mount script and everything works as expected.

 

all looks fine - it even says it's overwritten the existing mountcheck file which is a good sign. 

 

When you say 'no files' - what are you using to look?

 

post your mount scripts for both unionfs and mergerfs as there's something different in there.

Link to comment
3 hours ago, DZMM said:

When you say 'no files' - what are you using to look?

 

post your mount scripts for both unionfs and mergerfs as there's something different in there.

I am just using file explorer on windows and ubuntu and navigating to the network share like normal. Nothing shows when using krusader though either.

 

Quote

#!/bin/bash

#######  Check if script is already running  ##########

if [[ -f "/mnt/user/appdata/other/rclone/rclone_mount_running" ]]; then

echo "$(date "+%d.%m.%Y %T") INFO: Exiting script already running."

exit

else

touch /mnt/user/appdata/other/rclone/rclone_mount_running

fi

#######  End Check if script already running  ##########

#######  Start rclone gdrive mount  ##########

# check if gdrive mount already created

if [[ -f "/mnt/user/mount_rclone/google_vfs/mountcheck" ]]; then

echo "$(date "+%d.%m.%Y %T") INFO: Check rclone vfs already mounted."

else

echo "$(date "+%d.%m.%Y %T") INFO: mounting rclone vfs."

# create directories for rclone mount and unionfs mount

mkdir -p /mnt/user/appdata/other/rclone
mkdir -p /mnt/user/mount_rclone/google_vfs
mkdir -p /mnt/user/mount_unionfs/google_vfs
mkdir -p /mnt/user/rclone_upload/google_vfs

rclone mount --allow-other --buffer-size 256M --dir-cache-time 72h --drive-chunk-size 512M --fast-list --log-level INFO --vfs-read-chunk-size 128M --vfs-read-chunk-size-limit off gdrive_media_vfs: /mnt/user/mount_rclone/google_vfs &

# check if mount successful

# slight pause to give mount time to finalise

sleep 5

if [[ -f "/mnt/user/mount_rclone/google_vfs/mountcheck" ]]; then

echo "$(date "+%d.%m.%Y %T") INFO: Check rclone gdrive vfs mount success."

else

echo "$(date "+%d.%m.%Y %T") CRITICAL: rclone gdrive vfs mount failed - please check for problems."

rm /mnt/user/appdata/other/rclone/rclone_mount_running

exit

fi

fi

#######  End rclone gdrive mount  ##########

#######  Start unionfs mount   ##########

if [[ -f "/mnt/user/mount_unionfs/google_vfs/mountcheck" ]]; then

echo "$(date "+%d.%m.%Y %T") INFO: Check successful, unionfs already mounted."

else

unionfs -o cow,allow_other,direct_io,auto_cache,sync_read /mnt/user/rclone_upload/google_vfs=RW:/mnt/user/mount_rclone/google_vfs=RO /mnt/user/mount_unionfs/google_vfs

if [[ -f "/mnt/user/mount_unionfs/google_vfs/mountcheck" ]]; then

echo "$(date "+%d.%m.%Y %T") INFO: Check successful, unionfs mounted."

else

echo "$(date "+%d.%m.%Y %T") CRITICAL: unionfs Remount failed."

rm /mnt/user/appdata/other/rclone/rclone_mount_running

exit

fi

fi

#######  End Mount unionfs   ##########

############### starting dockers that need unionfs mount ######################

# only start dockers once

if [[ -f "/mnt/user/appdata/other/rclone/dockers_started" ]]; then

echo "$(date "+%d.%m.%Y %T") INFO: dockers already started"

else

touch /mnt/user/appdata/other/rclone/dockers_started

echo "$(date "+%d.%m.%Y %T") INFO: Starting dockers."

docker start plex
docker start tautulli
docker start radarr
docker start sonarr

fi

############### end dockers that need unionfs mount ######################

exit

Quote

#!/bin/bash

######################
#### Mount Script ####
######################
### Version 0.96.7 ###
######################

####### EDIT ONLY THESE SETTINGS #######

# INSTRUCTIONS
# 1. Change the name of the rclone remote and shares to match your setup
# 2. NOTE: enter RcloneRemoteName WITHOUT ':'
# 3. Optional: include custom command and bind mount settings
# 4. Optional: include extra folders in mergerfs mount

# REQUIRED SETTINGS
RcloneRemoteName="gdrive_media_vfs" # Name of rclone remote mount WITHOUT ':'. NOTE: Choose your encrypted remote for sensitive data
RcloneMountShare="/mnt/user/mount_rclone" # where your rclone remote will be located without trailing slash  e.g. /mnt/user/mount_rclone
LocalFilesShare="ignore" # location of the local files and MountFolders you want to upload without trailing slash to rclone e.g. /mnt/user/local. Enter 'ignore' to disable
MergerfsMountShare="/mnt/user/mount_unionfs" # location without trailing slash  e.g. /mnt/user/mount_mergerfs. Enter 'ignore' to disable
DockerStart="ombi" # list of dockers, separated by space, to start once mergerfs mount verified. Remember to disable AUTOSTART for dockers added in docker settings page
MountFolders=\{"downloads/complete,downloads/intermediate,downloads/seeds,movies,tv"\} # comma separated list of folders to create within the mount

# Note: Again - remember to NOT use ':' in your remote name above

# OPTIONAL SETTINGS

# Add extra paths to mergerfs mount in addition to LocalFilesShare
LocalFilesShare2="ignore" # without trailing slash e.g. /mnt/user/other__remote_mount/or_other_local_folder.  Enter 'ignore' to disable
LocalFilesShare3="ignore"
LocalFilesShare4="ignore"

# Add extra commands or filters
Command1="--rc"
Command2=""
Command3=""
Command4=""
Command5=""
Command6=""
Command7=""
Command8=""

CreateBindMount="N" # Y/N. Choose whether to bind traffic to a particular network adapter
RCloneMountIP="192.168.1.252" # My unraid IP is 172.30.12.2 so I create another similar IP address
NetworkAdapter="eth0" # choose your network adapter. eth0 recommended
VirtualIPNumber="2" # creates eth0:x e.g. eth0:1.  I create a unique virtual IP addresses for each mount & upload so I can monitor and traffic shape for each of them

####### END SETTINGS #######

###############################################################################
#####   DO NOT EDIT ANYTHING BELOW UNLESS YOU KNOW WHAT YOU ARE DOING   #######
###############################################################################

####### Preparing mount location variables #######
RcloneMountLocation="$RcloneMountShare/$RcloneRemoteName" # Location for rclone mount
LocalFilesLocation="$LocalFilesShare/$RcloneRemoteName" # Location for local files to be merged with rclone mount
MergerFSMountLocation="$MergerfsMountShare/$RcloneRemoteName" # Rclone data folder location

####### create directories for rclone mount and mergerfs mounts #######
mkdir -p /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName #for script files
if [[  $LocalFileShare == 'ignore' ]]; then
    echo "$(date "+%d.%m.%Y %T") INFO: Not creating local folders as requested."
else
    echo "$(date "+%d.%m.%Y %T") INFO: Creating local folders."
    eval mkdir -p $LocalFilesLocation/"$MountFolders"
fi
mkdir -p $RcloneMountLocation
mkdir -p $MergerFSMountLocation

#######  Check if script is already running  #######
echo "$(date "+%d.%m.%Y %T") INFO: *** Starting mount of remote ${RcloneRemoteName}"
echo "$(date "+%d.%m.%Y %T") INFO: Checking if this script is already running."
if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running" ]]; then
    echo "$(date "+%d.%m.%Y %T") INFO: Exiting script as already running."
    exit
else
    echo "$(date "+%d.%m.%Y %T") INFO: Script not running - proceeding."
    touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
fi

####### Checking have connectivity #######

echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if online"
ping -q -c2 google.com > /dev/null # -q quiet, -c number of pings to perform
if [ $? -eq 0 ]; then # ping returns exit status 0 if successful
    echo "$(date "+%d.%m.%Y %T") PASSED: *** Internet online"
else
    echo "$(date "+%d.%m.%Y %T") FAIL: *** No connectivity.  Will try again on next run"
    rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
    exit
fi

#######  Create Rclone Mount  #######

# Check If Rclone Mount Already Created
if [[ -f "$RcloneMountLocation/mountcheck" ]]; then
    echo "$(date "+%d.%m.%Y %T") INFO: Success ${RcloneRemoteName} remote is already mounted."
else
    echo "$(date "+%d.%m.%Y %T") INFO: Mount not running. Will now mount ${RcloneRemoteName} remote."
# Creating mountcheck file in case it doesn't already exist
    echo "$(date "+%d.%m.%Y %T") INFO: Recreating mountcheck file for ${RcloneRemoteName} remote."
    touch mountcheck
    rclone copy mountcheck $RcloneRemoteName: -vv --no-traverse
# Check bind option
    if [[  $CreateBindMount == 'Y' ]]; then
        echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}"
        ping -q -c2 $RCloneMountIP > /dev/null # -q quiet, -c number of pings to perform
        if [ $? -eq 0 ]; then # ping returns exit status 0 if successful
            echo "$(date "+%d.%m.%Y %T") INFO: *** IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}"
        else
            echo "$(date "+%d.%m.%Y %T") INFO: *** Creating IP address ${RCloneMountIP} for remote ${RcloneRemoteName}"
            ip addr add $RCloneMountIP/24 dev $NetworkAdapter label $NetworkAdapter:$VirtualIPNumber
        fi
        echo "$(date "+%d.%m.%Y %T") INFO: *** Created bind mount ${RCloneMountIP} for remote ${RcloneRemoteName}"
    else
        RCloneMountIP=""
        echo "$(date "+%d.%m.%Y %T") INFO: *** Creating mount for remote ${RcloneRemoteName}"
    fi
# create rclone mount
    rclone mount \
    --allow-other \
    --buffer-size 256M \
    --dir-cache-time 720h \
    --drive-chunk-size 512M \
    --log-level INFO \
    --vfs-read-chunk-size 128M \
    --vfs-read-chunk-size-limit off \
    --vfs-cache-mode writes \
    --bind=$RCloneMountIP \
    $RcloneRemoteName: $RcloneMountLocation &

# Check if Mount Successful
    echo "$(date "+%d.%m.%Y %T") INFO: sleeping for 5 seconds"
# slight pause to give mount time to finalise
    sleep 5
    echo "$(date "+%d.%m.%Y %T") INFO: continuing..."
    if [[ -f "$RcloneMountLocation/mountcheck" ]]; then
        echo "$(date "+%d.%m.%Y %T") INFO: Successful mount of ${RcloneRemoteName} mount."
    else
        echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mount failed - please check for problems.  Stopping dockers"
        docker stop $DockerStart
        rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
        exit
    fi
fi

####### Start MergerFS Mount #######

if [[  $MergerfsMountShare == 'ignore' ]]; then
    echo "$(date "+%d.%m.%Y %T") INFO: Not creating mergerfs mount as requested."
else
    if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then
        echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount in place."
    else
# check if mergerfs already installed
        if [[ -f "/bin/mergerfs" ]]; then
            echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs already installed, proceeding to create mergerfs mount"
        else
# Build mergerfs binary
            echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs not installed - installing now."
            mkdir -p /mnt/user/appdata/other/rclone/mergerfs
            docker run -v /mnt/user/appdata/other/rclone/mergerfs:/build --rm trapexit/mergerfs-static-build
            mv /mnt/user/appdata/other/rclone/mergerfs/mergerfs /bin
# check if mergerfs install successful
            echo "$(date "+%d.%m.%Y %T") INFO: *sleeping for 5 seconds"
            sleep 5
            if [[ -f "/bin/mergerfs" ]]; then
                echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs installed successfully, proceeding to create mergerfs mount."
            else
                echo "$(date "+%d.%m.%Y %T") ERROR: Mergerfs not installed successfully.  Please check for errors.  Exiting."
                rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
                exit
            fi
        fi
# Create mergerfs mount
        echo "$(date "+%d.%m.%Y %T") INFO: Creating ${RcloneRemoteName} mergerfs mount."
# Extra Mergerfs folders
        if [[  $LocalFilesShare2 != 'ignore' ]]; then
            echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare2} to ${RcloneRemoteName} mergerfs mount."
            LocalFilesShare2=":$LocalFilesShare2"
        else
            LocalFilesShare2=""
        fi
        if [[  $LocalFilesShare3 != 'ignore' ]]; then
            echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare3} to ${RcloneRemoteName} mergerfs mount."
            LocalFilesShare3=":$LocalFilesShare3"
        else
            LocalFilesShare3=""
        fi
        if [[  $LocalFilesShare4 != 'ignore' ]]; then
            echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare4} to ${RcloneRemoteName} mergerfs mount."
            LocalFilesShare4=":$LocalFilesShare4"
        else
            LocalFilesShare4=""
        fi
# make sure mergerfs mount point is empty
        mv $MergerFSMountLocation $LocalFilesLocation
        mkdir -p $MergerFSMountLocation
# mergerfs mount command
        mergerfs $LocalFilesLocation:$RcloneMountLocation$LocalFilesShare2$LocalFilesShare3$LocalFilesShare4 $MergerFSMountLocation -o rw,async_read=false,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=partial,dropcacheonclose=true
# check if mergerfs mount successful
        echo "$(date "+%d.%m.%Y %T") INFO: Checking if ${RcloneRemoteName} mergerfs mount created."
        if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then
            echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount created."
        else
            echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mergerfs mount failed.  Stopping dockers."
            docker stop $DockerStart
            rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
            exit
        fi
    fi
fi

####### Starting Dockers That Need Mergerfs Mount To Work Properly #######

# only start dockers once
if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started" ]]; then
    echo "$(date "+%d.%m.%Y %T") INFO: dockers already started."
else
    touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started
    echo "$(date "+%d.%m.%Y %T") INFO: Starting dockers."
    docker start $DockerStart
fi

rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
echo "$(date "+%d.%m.%Y %T") INFO: Script complete"

exit

 

Link to comment
33 minutes ago, KeyBoardDabbler said:

I am just using file explorer on windows and ubuntu and navigating to the network share like normal. Nothing shows when using krusader though either.

 

 

LocalFilesShare="ignore" - looking at your unionfs script, shouldn't this be LocalFilesShare="/mnt/user/rclone_upload/google_vfs"??

 

Otherwise, looks fine and your files from gdrive_media_vfs: should be showing in the mergerfs mount.  If still not working, post the logs

  • Like 1
Link to comment
4 hours ago, DZMM said:

post your script logs

Here is my mount Folder log:

 

Script location: /tmp/user.scripts/tmpScripts/rclone mount/script
Note that closing this window will abort the execution of this script
06.08.2020 13:26:53 INFO: Creating local folders.
06.08.2020 13:26:53 INFO: *** Starting mount of remote gdrive_media_vfs
06.08.2020 13:26:53 INFO: Checking if this script is already running.
06.08.2020 13:26:53 INFO: Script not running - proceeding.
06.08.2020 13:26:53 INFO: *** Checking if online
06.08.2020 13:26:54 PASSED: *** Internet online
06.08.2020 13:26:54 INFO: Mount not running. Will now mount gdrive_media_vfs remote.
06.08.2020 13:26:54 INFO: Recreating mountcheck file for gdrive_media_vfs remote.
2020/08/06 13:26:54 DEBUG : rclone: Version "v1.52.2" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "mountcheck" "gdrive_media_vfs:" "-vv" "--no-traverse"]
2020/08/06 13:26:54 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf"
2020/08/06 13:26:54 DEBUG : fs cache: renaming cache item "mountcheck" to be canonical "/"
2020/08/06 13:26:55 DEBUG : mountcheck: Modification times differ by -52.019083302s: 2020-08-06 13:26:54.305083302 -0700 PDT, 2020-08-06 20:26:02.286 +0000 UTC
2020/08/06 13:26:56 INFO : mountcheck: Copied (replaced existing)
2020/08/06 13:26:56 INFO :
Transferred: 32 / 32 Bytes, 100%, 25 Bytes/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 1.2s

2020/08/06 13:26:56 DEBUG : 7 go routines active
06.08.2020 13:26:56 INFO: *** Creating mount for remote gdrive_media_vfs
06.08.2020 13:26:56 INFO: sleeping for 5 seconds
06.08.2020 13:27:01 INFO: continuing...
06.08.2020 13:27:01 INFO: Successful mount of gdrive_media_vfs mount.
06.08.2020 13:27:01 INFO: Check successful, gdrive_media_vfs mergerfs mount in place.
06.08.2020 13:27:01 INFO: dockers already started.
06.08.2020 13:27:01 INFO: Script complete

 

Upload Script:

 

Script location: /tmp/user.scripts/tmpScripts/rclone upload/script
Note that closing this window will abort the execution of this script
06.08.2020 13:28:05 INFO: *** Rclone move selected. Files will be moved from /mnt/user/Plex-Media/gdrive_media_vfs for gdrive_upload_vfs ***
06.08.2020 13:28:05 INFO: *** Starting rclone_upload script for gdrive_upload_vfs ***
06.08.2020 13:28:05 INFO: Script not running - proceeding.
06.08.2020 13:28:05 INFO: Checking if rclone installed successfully.
06.08.2020 13:28:05 INFO: rclone not installed - will try again later.

Link to comment
8 hours ago, DZMM said:

LocalFilesShare="ignore" - looking at your unionfs script, shouldn't this be LocalFilesShare="/mnt/user/rclone_upload/google_vfs"??

 

Otherwise, looks fine and your files from gdrive_media_vfs: should be showing in the mergerfs mount.  If still not working, post the logs

I have since tried with the correct path but still not showing any of the files in my drive.

 

the mount script has created the subfolders but when i navigate into /gdrive_media_vfs it is empty.

 

UPDATE

after a reboot of my server everything is now normal and operating?? 

 

Edited by KeyBoardDabbler
Link to comment
2 hours ago, KeyBoardDabbler said:

I have since tried with the correct path but still not showing any of the files in my drive.

 

the mount script has created the subfolders but when i navigate into /gdrive_media_vfs it is empty.

 

UPDATE

after a reboot of my server everything is now normal and operating?? 

 

Dunno - I was going to say after reading your logs that everything was 'working', so try a reboot.  Sometimes when people are repeatedly mounting to debug the system seems to go a bit loopy.

Link to comment
10 hours ago, Emilio5639 said:

Here is my mount Folder log:

 

Script location: /tmp/user.scripts/tmpScripts/rclone mount/script
Note that closing this window will abort the execution of this script
06.08.2020 13:26:53 INFO: Creating local folders.
06.08.2020 13:26:53 INFO: *** Starting mount of remote gdrive_media_vfs
06.08.2020 13:26:53 INFO: Checking if this script is already running.
06.08.2020 13:26:53 INFO: Script not running - proceeding.
06.08.2020 13:26:53 INFO: *** Checking if online
06.08.2020 13:26:54 PASSED: *** Internet online
06.08.2020 13:26:54 INFO: Mount not running. Will now mount gdrive_media_vfs remote.
06.08.2020 13:26:54 INFO: Recreating mountcheck file for gdrive_media_vfs remote.
2020/08/06 13:26:54 DEBUG : rclone: Version "v1.52.2" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "mountcheck" "gdrive_media_vfs:" "-vv" "--no-traverse"]
2020/08/06 13:26:54 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf"
2020/08/06 13:26:54 DEBUG : fs cache: renaming cache item "mountcheck" to be canonical "/"
2020/08/06 13:26:55 DEBUG : mountcheck: Modification times differ by -52.019083302s: 2020-08-06 13:26:54.305083302 -0700 PDT, 2020-08-06 20:26:02.286 +0000 UTC
2020/08/06 13:26:56 INFO : mountcheck: Copied (replaced existing)
2020/08/06 13:26:56 INFO :
Transferred: 32 / 32 Bytes, 100%, 25 Bytes/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 1.2s

2020/08/06 13:26:56 DEBUG : 7 go routines active
06.08.2020 13:26:56 INFO: *** Creating mount for remote gdrive_media_vfs
06.08.2020 13:26:56 INFO: sleeping for 5 seconds
06.08.2020 13:27:01 INFO: continuing...
06.08.2020 13:27:01 INFO: Successful mount of gdrive_media_vfs mount.
06.08.2020 13:27:01 INFO: Check successful, gdrive_media_vfs mergerfs mount in place.
06.08.2020 13:27:01 INFO: dockers already started.
06.08.2020 13:27:01 INFO: Script complete

 

Upload Script:

 

Script location: /tmp/user.scripts/tmpScripts/rclone upload/script
Note that closing this window will abort the execution of this script
06.08.2020 13:28:05 INFO: *** Rclone move selected. Files will be moved from /mnt/user/Plex-Media/gdrive_media_vfs for gdrive_upload_vfs ***
06.08.2020 13:28:05 INFO: *** Starting rclone_upload script for gdrive_upload_vfs ***
06.08.2020 13:28:05 INFO: Script not running - proceeding.
06.08.2020 13:28:05 INFO: Checking if rclone installed successfully.
06.08.2020 13:28:05 INFO: rclone not installed - will try again later.

Can you post your mount script (haven't seen yet) and your upload script.  Somewhere there's a path difference, or it doesn't like the dash in Plex-Media and you need to use an underscore.

Link to comment

I am back again 🙃

 

can you please explain how have you have managed to get the following folder structure?

Quote

mount_rclone/tdrive_vfs/music

mount_rclone/tdrive_vfs/web_shows

mount_rclone/tdrive_vfs/tv_anime

 

I get the following error whenever i try to mount more than one tdrive in the same $RcloneMountLocation. Once the script finishes only the first mounted tdrive shows. 

Quote

08.08.2020 15:56:27 INFO: Creating local folders.
08.08.2020 15:56:27 INFO: *** Starting mount of remote tdrive_music_vfs
08.08.2020 15:56:27 INFO: Checking if this script is already running.
08.08.2020 15:56:27 INFO: Script not running - proceeding.
08.08.2020 15:56:27 INFO: *** Checking if online
08.08.2020 15:56:28 PASSED: *** Internet online
08.08.2020 15:56:28 INFO: Mount not running. Will now mount tdrive_music_vfs remote.
08.08.2020 15:56:28 INFO: Recreating mountcheck file for tdrive_music_vfs remote.
2020/08/08 15:56:28 DEBUG : rclone: Version "v1.52.2-285-g4d7f9130-beta" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone-beta/.rclone.conf" "copy" "mountcheck" "tdrive_music_vfs:" "-vv" "--no-traverse"]
2020/08/08 15:56:28 DEBUG : Using config file from "/boot/config/plugins/rclone-beta/.rclone.conf"
2020/08/08 15:56:28 DEBUG : fs cache: adding new entry for parent of "mountcheck", "/usr/local/emhttp"
2020/08/08 15:56:31 DEBUG : mountcheck: Modification times differ by -3m7.938405814s: 2020-08-08 15:56:28.730405814 +1200 NZST, 2020-08-08 03:53:20.792 +0000 UTC
2020/08/08 15:56:32 INFO : mountcheck: Copied (replaced existing)
2020/08/08 15:56:32 INFO :
Transferred: 32 / 32 Bytes, 100%, 17 Bytes/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 4.2s

2020/08/08 15:56:32 DEBUG : 10 go routines active
08.08.2020 15:56:32 INFO: *** Creating mount for remote tdrive_music_vfs
08.08.2020 15:56:33 INFO: sleeping for 5 seconds
2020/08/08 15:56:35 INFO : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0)
08.08.2020 15:56:38 INFO: continuing...
08.08.2020 15:56:38 INFO: Successful mount of tdrive_music_vfs mount.
08.08.2020 15:56:38 INFO: *** Starting mount of remote tdrive_web_shows_vfs
08.08.2020 15:56:38 INFO: Checking if this script is already running.
08.08.2020 15:56:38 INFO: Script not running - proceeding.
08.08.2020 15:56:38 INFO: *** Checking if online
08.08.2020 15:56:39 PASSED: *** Internet online
08.08.2020 15:56:39 INFO: Mount not running. Will now mount tdrive_web_shows_vfs remote.
08.08.2020 15:56:39 INFO: Recreating mountcheck file for tdrive_web_shows_vfs remote.
2020/08/08 15:56:39 DEBUG : rclone: Version "v1.52.2-285-g4d7f9130-beta" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone-beta/.rclone.conf" "copy" "mountcheck" "tdrive_web_shows_vfs:" "-vv" "--no-traverse"]
2020/08/08 15:56:39 DEBUG : Using config file from "/boot/config/plugins/rclone-beta/.rclone.conf"
2020/08/08 15:56:39 DEBUG : fs cache: adding new entry for parent of "mountcheck", "/usr/local/emhttp"
2020/08/08 15:56:41 DEBUG : mountcheck: Modification times differ by -26m50.494549599s: 2020-08-08 15:56:39.383549599 +1200 NZST, 2020-08-08 03:29:48.889 +0000 UTC
2020/08/08 15:56:43 INFO : mountcheck: Copied (replaced existing)
2020/08/08 15:56:43 INFO :
Transferred: 32 / 32 Bytes, 100%, 19 Bytes/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 3.9s

2020/08/08 15:56:43 DEBUG : 10 go routines active
08.08.2020 15:56:43 INFO: *** Creating mount for remote tdrive_web_shows_vfs
08.08.2020 15:56:43 INFO: sleeping for 5 seconds
2020/08/08 15:56:45 Fatal error: Directory is not empty: /mnt/user/mount_rclone/tdrive_vfs If you want to mount it anyway use: --allow-non-empty option
08.08.2020 15:56:48 INFO: continuing...
08.08.2020 15:56:48 INFO: Successful mount of tdrive_web_shows_vfs mount.
08.08.2020 15:56:48 INFO: Not creating mergerfs mount as requested.
08.08.2020 15:56:48 INFO: Starting dockers.
ombi
08.08.2020 15:56:48 INFO: Script complete
Script Finished Aug 08, 2020 15:56.48

 

I have no trouble mounting them like the below but it is not very clean to navigate

Quote

mount_rclone/tdrive_music_vfs/music (first folder in tdrive share)

mount_rclone/tdrive_web_shows_vfs/web_shows (first folder in tdrive share)

mount_rclone/tdrive_tv_anime/tv_anime (first folder in tdrive share)

 

 

Edited by KeyBoardDabbler
Link to comment
1 hour ago, KeyBoardDabbler said:

I am back again 🙃

 

can you please explain how have you have managed to get the following folder structure?

 

I get the following error whenever i try to mount more than one tdrive in the same $RcloneMountLocation. Once the script finishes only the first mounted tdrive shows. 

 

I have no trouble mounting them like the below but it is not very clean to navigate

 

I mount them all in separate folders - sorry if I put them all in the same folder in my write-up - that's a mistake.

 

Re navigation, it shouldn't matter as you should never ever need to use or visit the rclone mounts unless you are troubleshooting - the mergerfs folder is the only one you use day-2-day.

 

Sounds like you got it working then?

Link to comment
6 hours ago, DZMM said:

I mount them all in separate folders - sorry if I put them all in the same folder in my write-up - that's a mistake.

 

Re navigation, it shouldn't matter as you should never ever need to use or visit the rclone mounts unless you are troubleshooting - the mergerfs folder is the only one you use day-2-day.

 

Sounds like you got it working then?

Ahhh that explains it, it did throw me off a little bit.. Now that i have everything setup the /mount_merger folder layout is perfectly fine.

 

I have everything setup and ran through the process, almost there but...

I am now struggling with the last part, server side moving. I have tried various folders to move but they all error. I am most certain i have identified the correct encrypted folder names.

Quote

2020/08/09 00:03:42 DEBUG : rclone: Version "v1.52.3-290-g82975109-beta" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone-beta/.rclone.conf" "move" "gdrive_vfs:crypt/1ej1l**********************hcm1v7g9957vnidqij60" "tdrive_movies_documentary_vfs:crypt/f*******************9qpq7ab71c3fc860" "--user-agent=transfer" "-vv" "--buffer-size" "512M" "--drive-chunk-size" "512M" "--tpslimit" "8" "--checkers" "8" "--transfers" "4" "--order-by" "modtime,ascending" "--exclude" "*fuse_hidden*" "--exclude" "*_HIDDEN" "--exclude" ".recycle**" "--exclude" ".Recycle.Bin/**" "--exclude" "*.backup~*" "--exclude" "*.partial~*" "--drive-stop-on-upload-limit" "--delete-empty-src-dirs"]
2020/08/09 00:03:42 DEBUG : Using config file from "/boot/config/plugins/rclone-beta/.rclone.conf"
2020/08/09 00:03:42 INFO : Starting HTTP transaction limiter: max 8 transactions/s with burst 1
2020/08/09 00:03:49 ERROR : : error reading source directory: directory not found
2020/08/09 00:03:49 DEBUG : Encrypted drive 'tdrive_movies_documentary_vfs:crypt/f*******************9qpq7ab71c3fc860': Waiting for checks to finish
2020/08/09 00:03:49 DEBUG : Encrypted drive 'tdrive_movies_documentary_vfs:crypt/f*******************9qpq7ab71c3fc860': Waiting for transfers to finish
2020/08/09 00:03:49 INFO : There was nothing to transfer
2020/08/09 00:03:49 ERROR : Attempt 1/3 failed with 1 errors and: directory not found
2020/08/09 00:03:50 ERROR : : error reading source directory: directory not found
2020/08/09 00:03:50 DEBUG : Encrypted drive 'tdrive_movies_documentary_vfs:crypt/f*******************9qpq7ab71c3fc860': Waiting for checks to finish
2020/08/09 00:03:50 DEBUG : Encrypted drive 'tdrive_movies_documentary_vfs:crypt/f*******************9qpq7ab71c3fc860': Waiting for transfers to finish
2020/08/09 00:03:50 INFO : There was nothing to transfer
2020/08/09 00:03:50 ERROR : Attempt 2/3 failed with 1 errors and: directory not found
2020/08/09 00:03:50 ERROR : : error reading source directory: directory not found
2020/08/09 00:03:50 DEBUG : Encrypted drive 'tdrive_movies_documentary_vfs:crypt/f*******************9qpq7ab71c3fc860': Waiting for checks to finish
2020/08/09 00:03:50 DEBUG : Encrypted drive 'tdrive_movies_documentary_vfs:crypt/f*******************9qpq7ab71c3fc860': Waiting for transfers to finish
2020/08/09 00:03:50 INFO : There was nothing to transfer
2020/08/09 00:03:50 ERROR : Attempt 3/3 failed with 1 errors and: directory not found
2020/08/09 00:03:50 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 8.4s

2020/08/09 00:03:50 DEBUG : 14 go routines active
2020/08/09 00:03:50 Failed to move: directory not found
Script Finished Aug 09, 2020 00:03.50

Full logs for this script are available at /tmp/user.scripts/tmpScripts/rClone_Move/log.txt

Script Starting Aug 09, 2020 00:08.49

Full logs for this script are available at /tmp/user.scripts/tmpScripts/rClone_Move/log.txt

2020/08/09 00:08:49 DEBUG : rclone: Version "v1.52.3-290-g82975109-beta" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone-beta/.rclone.conf" "move" "tdrive_upload_vfs:crypt/**************************fk40rft2t0t7neqvb9hi7psvm8cg" "tdrive_movies_bio_vfs:crypt/*********************7neqvb9hi7psvm8cg" "--user-agent=transfer" "-vv" "--buffer-size" "512M" "--drive-chunk-size" "512M" "--tpslimit" "8" "--checkers" "8" "--transfers" "4" "--order-by" "modtime,ascending" "--exclude" "*fuse_hidden*" "--exclude" "*_HIDDEN" "--exclude" ".recycle**" "--exclude" ".Recycle.Bin/**" "--exclude" "*.backup~*" "--exclude" "*.partial~*" "--drive-stop-on-upload-limit" "--delete-empty-src-dirs"]
2020/08/09 00:08:49 DEBUG : Using config file from "/boot/config/plugins/rclone-beta/.rclone.conf"
2020/08/09 00:08:49 INFO : Starting HTTP transaction limiter: max 8 transactions/s with burst 1
2020/08/09 00:08:57 ERROR : : error reading source directory: directory not found
2020/08/09 00:08:57 DEBUG : Encrypted drive 'tdrive_movies_bio_vfs:crypt/*********************7neqvb9hi7psvm8cg': Waiting for checks to finish
2020/08/09 00:08:57 DEBUG : Encrypted drive 'tdrive_movies_bio_vfs:crypt/*********************7neqvb9hi7psvm8cg': Waiting for transfers to finish
2020/08/09 00:08:57 INFO : There was nothing to transfer
2020/08/09 00:08:57 ERROR : Attempt 1/3 failed with 1 errors and: directory not found
2020/08/09 00:08:58 ERROR : : error reading source directory: directory not found
2020/08/09 00:08:58 DEBUG : Encrypted drive 'tdrive_movies_bio_vfs:crypt/*********************7neqvb9hi7psvm8cg': Waiting for checks to finish
2020/08/09 00:08:58 DEBUG : Encrypted drive 'tdrive_movies_bio_vfs:crypt/*********************7neqvb9hi7psvm8cg': Waiting for transfers to finish
2020/08/09 00:08:58 INFO : There was nothing to transfer
2020/08/09 00:08:58 ERROR : Attempt 2/3 failed with 1 errors and: directory not found
2020/08/09 00:08:58 ERROR : : error reading source directory: directory not found
2020/08/09 00:08:58 DEBUG : Encrypted drive 'tdrive_bio_vfs:crypt/*********************7neqvb9hi7psvm8cg': Waiting for checks to finish
2020/08/09 00:08:58 DEBUG : Encrypted drive 'tdrive_bio_vfs:crypt/*********************7neqvb9hi7psvm8cg': Waiting for transfers to finish
2020/08/09 00:08:58 INFO : There was nothing to transfer
2020/08/09 00:08:58 ERROR : Attempt 3/3 failed with 1 errors and: directory not found
2020/08/09 00:08:58 INFO :
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 1 (retrying may help)
Elapsed time: 9.7s

2020/08/09 00:08:58 DEBUG : 19 go routines active
2020/08/09 00:08:58 Failed to move: directory not found
Script Finished Aug 09, 2020 00:08.58

 

Edited by KeyBoardDabbler
Link to comment
1 hour ago, KeyBoardDabbler said:

I am now struggling with the last part, server side moving. I have tried various folders to move but they all error. I am most certain i have identified the correct encrypted folder names.

I think it's because you are using the remotes that decrypt the files - you need to use the 'raw' remotes that process the encrypted files i.e.

rclone move tdrive_upload:crypt/**************************fk40rft2t0t7neqvb9hi7psvm8cg" "tdrive_movies_bio:crypt/*********************7neqvb9hi7psvm8cg"

because you are moving the files server-side i.e. encrypted.

  • Thanks 1
Link to comment
15 hours ago, DZMM said:

I think it's because you are using the remotes that decrypt the files - you need to use the 'raw' remotes that process the encrypted files i.e.


rclone move tdrive_upload:crypt/**************************fk40rft2t0t7neqvb9hi7psvm8cg" "tdrive_movies_bio:crypt/*********************7neqvb9hi7psvm8cg"

because you are moving the files server-side i.e. encrypted.

I think you was correct, this change has got me to the finish line. Things are running really good.

I have a lot to rearrange now.

 

very grateful for the help and all the work you have put into this project. 

 

I do not have any issue moving tdrive > tdrive share but i am running into permission error when trying to move files out of gdrive > tdrive.

sa group has been added to the members on all drives. Status has been set to "manage" and "enabled" migration in settings.

 

Quote

Couldn't move: googleapi: Error 403: The domain administrator has not allowed writers to move items into a shared drive.

 

The main difference is gdrive uses the client_id, client_secret & token whereas tdrive uses service_account_file because it can not be used with the gdrive. Either way i have both users added as members ect.

Edited by KeyBoardDabbler
Link to comment
10 hours ago, KeyBoardDabbler said:

I do not have any issue moving tdrive > tdrive share but i am running into permission error when trying to move files out of gdrive > tdrive.

sa group has been added to the members on all drives. Status has been set to "manage" and "enabled" migration in settings.

I haven't come across this problem before - I think because I was moving my music mtdrive-->gdrive.

 

10 hours ago, KeyBoardDabbler said:

very grateful for the help and all the work you have put into this project. 

 

You're welcome.  I'm happy to share as I'm sure it's saving people $$$ in drives etc.

Link to comment

Hi all,

 

Thought i'd ask here as there appears to be quite a few advanced users...

 

I've decided to swap over to encryption and am trying to work out how to copy from the my current un-encrypted drive to an encrypted drive. My understanding is I can only use server side copy if both source and destination are same (both encrypted or not). I think AutoRclone is what I need to setup in order to do what I want, but I am not entirely sure. I believe I need to download and then upload again, so for speed i'll fire up a cloud instance with Hetzner.

 

Can anyone confirm or point me in the right direction please?

 

Thanks.

Link to comment
On 8/6/2020 at 11:29 PM, DZMM said:

Can you post your mount script (haven't seen yet) and your upload script.  Somewhere there's a path difference, or it doesn't like the dash in Plex-Media and you need to use an underscore.

Here are all of the Scripts that I have set up.  I went ahead and changed my folder to:

Plex_Media

 

RClone Mount Script:

 

#!/bin/bash

######################
#### Mount Script ####
######################
### Version 0.96.7 ###
######################

####### EDIT ONLY THESE SETTINGS #######

# INSTRUCTIONS
# 1. Change the name of the rclone remote and shares to match your setup
# 2. NOTE: enter RcloneRemoteName WITHOUT ':'
# 3. Optional: include custom command and bind mount settings
# 4. Optional: include extra folders in mergerfs mount

# REQUIRED SETTINGS
RcloneRemoteName="gdrive_media_vfs" # Name of rclone remote mount WITHOUT ':'. NOTE: Choose your encrypted remote for sensitive data
RcloneMountShare="/mnt/user/mount_Plex_Media" # where your rclone remote will be located without trailing slash  e.g. /mnt/user/mount_rclone
LocalFilesShare="/mnt/user/Plex_Media" # location of the local files and MountFolders you want to upload without trailing slash to rclone e.g. /mnt/user/local. Enter 'ignore' to disable
MergerfsMountShare="/mnt/user/mount_mergerfs" # location without trailing slash  e.g. /mnt/user/mount_mergerfs. Enter 'ignore' to disable
DockerStart="plex binhex-sonarr radarr" # list of dockers, separated by space, to start once mergerfs mount verified. Remember to disable AUTOSTART for dockers added in docker settings page
MountFolders=\{"downloads/complete,downloads/intermediate,downloads/seeds,movies,tv"\} # comma separated list of folders to create within the mount

# Note: Again - remember to NOT use ':' in your remote name above

# OPTIONAL SETTINGS

# Add extra paths to mergerfs mount in addition to LocalFilesShare
LocalFilesShare2="ignore" # without trailing slash e.g. /mnt/user/other__remote_mount/or_other_local_folder.  Enter 'ignore' to disable
LocalFilesShare3="ignore"
LocalFilesShare4="ignore"

# Add extra commands or filters
Command1="--rc"
Command2=""
Command3=""
Command4=""
Command5=""
Command6=""
Command7=""
Command8=""

CreateBindMount="N" # Y/N. Choose whether to bind traffic to a particular network adapter
RCloneMountIP="10.0.0.162" # My unraid IP is 172.30.12.2 so I create another similar IP address
NetworkAdapter="eth0" # choose your network adapter. eth0 recommended
VirtualIPNumber="2" # creates eth0:x e.g. eth0:1.  I create a unique virtual IP addresses for each mount & upload so I can monitor and traffic shape for each of them

####### END SETTINGS #######

###############################################################################
#####   DO NOT EDIT ANYTHING BELOW UNLESS YOU KNOW WHAT YOU ARE DOING   #######
###############################################################################

####### Preparing mount location variables #######
RcloneMountLocation="$RcloneMountShare/$RcloneRemoteName" # Location for rclone mount
LocalFilesLocation="$LocalFilesShare/$RcloneRemoteName" # Location for local files to be merged with rclone mount
MergerFSMountLocation="$MergerfsMountShare/$RcloneRemoteName" # Rclone data folder location

####### create directories for rclone mount and mergerfs mounts #######
mkdir -p /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName #for script files
if [[  $LocalFileShare == 'ignore' ]]; then
    echo "$(date "+%d.%m.%Y %T") INFO: Not creating local folders as requested."
else
    echo "$(date "+%d.%m.%Y %T") INFO: Creating local folders."
    eval mkdir -p $LocalFilesLocation/"$MountFolders"
fi
mkdir -p $RcloneMountLocation
mkdir -p $MergerFSMountLocation

#######  Check if script is already running  #######
echo "$(date "+%d.%m.%Y %T") INFO: *** Starting mount of remote ${RcloneRemoteName}"
echo "$(date "+%d.%m.%Y %T") INFO: Checking if this script is already running."
if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running" ]]; then
    echo "$(date "+%d.%m.%Y %T") INFO: Exiting script as already running."
    exit
else
    echo "$(date "+%d.%m.%Y %T") INFO: Script not running - proceeding."
    touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
fi

####### Checking have connectivity #######

echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if online"
ping -q -c2 google.com > /dev/null # -q quiet, -c number of pings to perform
if [ $? -eq 0 ]; then # ping returns exit status 0 if successful
    echo "$(date "+%d.%m.%Y %T") PASSED: *** Internet online"
else
    echo "$(date "+%d.%m.%Y %T") FAIL: *** No connectivity.  Will try again on next run"
    rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
    exit
fi

#######  Create Rclone Mount  #######

# Check If Rclone Mount Already Created
if [[ -f "$RcloneMountLocation/mountcheck" ]]; then
    echo "$(date "+%d.%m.%Y %T") INFO: Success ${RcloneRemoteName} remote is already mounted."
else
    echo "$(date "+%d.%m.%Y %T") INFO: Mount not running. Will now mount ${RcloneRemoteName} remote."
# Creating mountcheck file in case it doesn't already exist
    echo "$(date "+%d.%m.%Y %T") INFO: Recreating mountcheck file for ${RcloneRemoteName} remote."
    touch mountcheck
    rclone copy mountcheck $RcloneRemoteName: -vv --no-traverse
# Check bind option
    if [[  $CreateBindMount == 'Y' ]]; then
        echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}"
        ping -q -c2 $RCloneMountIP > /dev/null # -q quiet, -c number of pings to perform
        if [ $? -eq 0 ]; then # ping returns exit status 0 if successful
            echo "$(date "+%d.%m.%Y %T") INFO: *** IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}"
        else
            echo "$(date "+%d.%m.%Y %T") INFO: *** Creating IP address ${RCloneMountIP} for remote ${RcloneRemoteName}"
            ip addr add $RCloneMountIP/24 dev $NetworkAdapter label $NetworkAdapter:$VirtualIPNumber
        fi
        echo "$(date "+%d.%m.%Y %T") INFO: *** Created bind mount ${RCloneMountIP} for remote ${RcloneRemoteName}"
    else
        RCloneMountIP=""
        echo "$(date "+%d.%m.%Y %T") INFO: *** Creating mount for remote ${RcloneRemoteName}"
    fi
# create rclone mount
    rclone mount \
    --allow-other \
    --buffer-size 256M \
    --dir-cache-time 720h \
    --drive-chunk-size 512M \
    --log-level INFO \
    --vfs-read-chunk-size 128M \
    --vfs-read-chunk-size-limit off \
    --vfs-cache-mode writes \
    --bind=$RCloneMountIP \
    $RcloneRemoteName: $RcloneMountLocation &

# Check if Mount Successful
    echo "$(date "+%d.%m.%Y %T") INFO: sleeping for 5 seconds"
# slight pause to give mount time to finalise
    sleep 5
    echo "$(date "+%d.%m.%Y %T") INFO: continuing..."
    if [[ -f "$RcloneMountLocation/mountcheck" ]]; then
        echo "$(date "+%d.%m.%Y %T") INFO: Successful mount of ${RcloneRemoteName} mount."
    else
        echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mount failed - please check for problems.  Stopping dockers"
        docker stop $DockerStart
        rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
        exit
    fi
fi

####### Start MergerFS Mount #######

if [[  $MergerfsMountShare == 'ignore' ]]; then
    echo "$(date "+%d.%m.%Y %T") INFO: Not creating mergerfs mount as requested."
else
    if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then
        echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount in place."
    else
# check if mergerfs already installed
        if [[ -f "/bin/mergerfs" ]]; then
            echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs already installed, proceeding to create mergerfs mount"
        else
# Build mergerfs binary
            echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs not installed - installing now."
            mkdir -p /mnt/user/appdata/other/rclone/mergerfs
            docker run -v /mnt/user/appdata/other/rclone/mergerfs:/build --rm trapexit/mergerfs-static-build
            mv /mnt/user/appdata/other/rclone/mergerfs/mergerfs /bin
# check if mergerfs install successful
            echo "$(date "+%d.%m.%Y %T") INFO: *sleeping for 5 seconds"
            sleep 5
            if [[ -f "/bin/mergerfs" ]]; then
                echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs installed successfully, proceeding to create mergerfs mount."
            else
                echo "$(date "+%d.%m.%Y %T") ERROR: Mergerfs not installed successfully.  Please check for errors.  Exiting."
                rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
                exit
            fi
        fi
# Create mergerfs mount
        echo "$(date "+%d.%m.%Y %T") INFO: Creating ${RcloneRemoteName} mergerfs mount."
# Extra Mergerfs folders
        if [[  $LocalFilesShare2 != 'ignore' ]]; then
            echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare2} to ${RcloneRemoteName} mergerfs mount."
            LocalFilesShare2=":$LocalFilesShare2"
        else
            LocalFilesShare2=""
        fi
        if [[  $LocalFilesShare3 != 'ignore' ]]; then
            echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare3} to ${RcloneRemoteName} mergerfs mount."
            LocalFilesShare3=":$LocalFilesShare3"
        else
            LocalFilesShare3=""
        fi
        if [[  $LocalFilesShare4 != 'ignore' ]]; then
            echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare4} to ${RcloneRemoteName} mergerfs mount."
            LocalFilesShare4=":$LocalFilesShare4"
        else
            LocalFilesShare4=""
        fi
# make sure mergerfs mount point is empty
        mv $MergerFSMountLocation $LocalFilesLocation
        mkdir -p $MergerFSMountLocation
# mergerfs mount command
        mergerfs $LocalFilesLocation:$RcloneMountLocation$LocalFilesShare2$LocalFilesShare3$LocalFilesShare4 $MergerFSMountLocation -o rw,async_read=false,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=partial,dropcacheonclose=true
# check if mergerfs mount successful
        echo "$(date "+%d.%m.%Y %T") INFO: Checking if ${RcloneRemoteName} mergerfs mount created."
        if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then
            echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount created."
        else
            echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mergerfs mount failed.  Stopping dockers."
            docker stop $DockerStart
            rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
            exit
        fi
    fi
fi

####### Starting Dockers That Need Mergerfs Mount To Work Properly #######

# only start dockers once
if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started" ]]; then
    echo "$(date "+%d.%m.%Y %T") INFO: dockers already started."
else
    touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started
    echo "$(date "+%d.%m.%Y %T") INFO: Starting dockers."
    docker start $DockerStart
fi

rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
echo "$(date "+%d.%m.%Y %T") INFO: Script complete"

exit

 

RClone Upload Script:

 

#!/bin/bash

######################
### Upload Script ####
######################
### Version 0.95.5 ###
######################

####### EDIT ONLY THESE SETTINGS #######

# INSTRUCTIONS
# 1. Edit the settings below to match your setup
# 2. NOTE: enter RcloneRemoteName WITHOUT ':'
# 3. Optional: Add additional commands or filters
# 4. Optional: Use bind mount settings for potential traffic shaping/monitoring
# 5. Optional: Use service accounts in your upload remote
# 6. Optional: Use backup directory for rclone sync jobs

# REQUIRED SETTINGS
RcloneCommand="move" # choose your rclone command e.g. move, copy, sync
RcloneRemoteName="gdrive_media_vfs" # Name of rclone remote mount WITHOUT ':'.
RcloneUploadRemoteName="gdrive_upload_vfs" # If you have a second remote created for uploads put it here.  Otherwise use the same remote as RcloneRemoteName.
LocalFilesShare="/mnt/user/Plex_Media" # location of the local files without trailing slash you want to rclone to use
RcloneMountShare="/mnt/user/mount_Plex_Media" # where your rclone mount is located without trailing slash  e.g. /mnt/user/mount_rclone
MinimumAge="15m" # sync files suffix ms|s|m|h|d|w|M|y
ModSort="ascending" # "ascending" oldest files first, "descending" newest files first

# Note: Again - remember to NOT use ':' in your remote name above

# Bandwidth limits: specify the desired bandwidth in kBytes/s, or use a suffix b|k|M|G. Or 'off' or '0' for unlimited.  The script uses --drive-stop-on-upload-limit which stops the script if the 750GB/day limit is achieved, so you no longer have to slow 'trickle' your files all day if you don't want to e.g. could just do an unlimited job overnight.
BWLimit1Time="01:00"
BWLimit1="off"
BWLimit2Time="08:00"
BWLimit2="15M"
BWLimit3Time="16:00"
BWLimit3="12M"

# OPTIONAL SETTINGS

# Add name to upload job
JobName="_daily_upload" # Adds custom string to end of checker file.  Useful if you're running multiple jobs against the same remote.

# Add extra commands or filters
Command1="--exclude downloads/**"
Command2=""
Command3=""
Command4=""
Command5=""
Command6=""
Command7=""
Command8=""

# Bind the mount to an IP address
CreateBindMount="N" # Y/N. Choose whether or not to bind traffic to a network adapter.
RCloneMountIP="10.0.0.62" # Choose IP to bind upload to.
NetworkAdapter="eth0" # choose your network adapter. eth0 recommended.
VirtualIPNumber="1" # creates eth0:x e.g. eth0:1.

# Use Service Accounts.  Instructions: https://github.com/xyou365/AutoRclone
UseServiceAccountUpload="N" # Y/N. Choose whether to use Service Accounts.
ServiceAccountDirectory="/mnt/user/appdata/other/rclone/service_accounts" # Path to your Service Account's .json files.
ServiceAccountFile="sa_gdrive_upload" # Enter characters before counter in your json files e.g. for sa_gdrive_upload1.json -->sa_gdrive_upload100.json, enter "sa_gdrive_upload".
CountServiceAccounts="15" # Integer number of service accounts to use.

# Is this a backup job
BackupJob="N" # Y/N. Syncs or Copies files from LocalFilesLocation to BackupRemoteLocation, rather than moving from LocalFilesLocation/RcloneRemoteName
BackupRemoteLocation="backup" # choose location on mount for deleted sync files
BackupRemoteDeletedLocation="backup_deleted" # choose location on mount for deleted sync files
BackupRetention="90d" # How long to keep deleted sync files suffix ms|s|m|h|d|w|M|y

####### END SETTINGS #######

###############################################################################
#####    DO NOT EDIT BELOW THIS LINE UNLESS YOU KNOW WHAT YOU ARE DOING   #####
###############################################################################

####### Preparing mount location variables #######
if [[  $BackupJob == 'Y' ]]; then
    LocalFilesLocation="$LocalFilesShare"
    echo "$(date "+%d.%m.%Y %T") INFO: *** Backup selected.  Files will be copied or synced from ${LocalFilesLocation} for ${RcloneUploadRemoteName} ***"
else
    LocalFilesLocation="$LocalFilesShare/$RcloneRemoteName"
    echo "$(date "+%d.%m.%Y %T") INFO: *** Rclone move selected.  Files will be moved from ${LocalFilesLocation} for ${RcloneUploadRemoteName} ***"
fi

RcloneMountLocation="$RcloneMountShare/$RcloneRemoteName" # Location of rclone mount

####### create directory for script files #######
mkdir -p /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName #for script files

#######  Check if script already running  ##########
echo "$(date "+%d.%m.%Y %T") INFO: *** Starting rclone_upload script for ${RcloneUploadRemoteName} ***"
if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/upload_running$JobName" ]]; then
    echo "$(date "+%d.%m.%Y %T") INFO: Exiting as script already running."
    exit
else
    echo "$(date "+%d.%m.%Y %T") INFO: Script not running - proceeding."
    touch /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/upload_running$JobName
fi

#######  check if rclone installed  ##########
echo "$(date "+%d.%m.%Y %T") INFO: Checking if rclone installed successfully."
if [[ -f "$RcloneMountLocation/mountcheck" ]]; then
    echo "$(date "+%d.%m.%Y %T") INFO: rclone installed successfully - proceeding with upload."
else
    echo "$(date "+%d.%m.%Y %T") INFO: rclone not installed - will try again later."
    rm /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/upload_running$JobName
    exit
fi

####### Rotating serviceaccount.json file if using Service Accounts #######
if [[ $UseServiceAccountUpload == 'Y' ]]; then
    cd /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/
    CounterNumber=$(find -name 'counter*' | cut -c 11,12)
    CounterCheck="1"
    if [[ "$CounterNumber" -ge "$CounterCheck" ]];then
        echo "$(date "+%d.%m.%Y %T") INFO: Counter file found for ${RcloneUploadRemoteName}."
    else
        echo "$(date "+%d.%m.%Y %T") INFO: No counter file found for ${RcloneUploadRemoteName}. Creating counter_1."
        touch /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/counter_1
        CounterNumber="1"
    fi
    ServiceAccount="--drive-service-account-file=$ServiceAccountDirectory/$ServiceAccountFile$CounterNumber.json"
    echo "$(date "+%d.%m.%Y %T") INFO: Adjusted service_account_file for upload remote ${RcloneUploadRemoteName} to ${ServiceAccountFile}${CounterNumber}.json based on counter ${CounterNumber}."
else
    echo "$(date "+%d.%m.%Y %T") INFO: Uploading using upload remote ${RcloneUploadRemoteName}"
    ServiceAccount=""
fi

#######  Upload files  ##########

# Check bind option
if [[  $CreateBindMount == 'Y' ]]; then
    echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if IP address ${RCloneMountIP} already created for upload to remote ${RcloneUploadRemoteName}"
    ping -q -c2 $RCloneMountIP > /dev/null # -q quiet, -c number of pings to perform
    if [ $? -eq 0 ]; then # ping returns exit status 0 if successful
        echo "$(date "+%d.%m.%Y %T") INFO: *** IP address ${RCloneMountIP} already created for upload to remote ${RcloneUploadRemoteName}"
    else
        echo "$(date "+%d.%m.%Y %T") INFO: *** Creating IP address ${RCloneMountIP} for upload to remote ${RcloneUploadRemoteName}"
        ip addr add $RCloneMountIP/24 dev $NetworkAdapter label $NetworkAdapter:$VirtualIPNumber
    fi
else
    RCloneMountIP=""
fi

#  Remove --delete-empty-src-dirs if rclone sync or copy
if [[  $RcloneCommand == 'move' ]]; then
    echo "$(date "+%d.%m.%Y %T") INFO: *** Using rclone move - will add --delete-empty-src-dirs to upload."
    DeleteEmpty="--delete-empty-src-dirs "
else
    echo "$(date "+%d.%m.%Y %T") INFO: *** Not using rclone move - will remove --delete-empty-src-dirs to upload."
    DeleteEmpty=""
fi

#  Check --backup-directory
if [[  $BackupJob == 'Y' ]]; then
    echo "$(date "+%d.%m.%Y %T") INFO: *** Will backup to ${BackupRemoteLocation} and use  ${BackupRemoteDeletedLocation} as --backup-directory with ${BackupRetention} retention for ${RcloneUploadRemoteName}."
    LocalFilesLocation="$LocalFilesShare"
    BackupDir="--backup-dir $RcloneUploadRemoteName:$BackupRemoteDeletedLocation"
else
    BackupRemoteLocation=""
    BackupRemoteDeletedLocation=""
    BackupRetention=""
    BackupDir=""
fi

# process files
    rclone $RcloneCommand $LocalFilesLocation $RcloneUploadRemoteName:$BackupRemoteLocation $ServiceAccount $BackupDir \
    --user-agent="$RcloneUploadRemoteName" \
    -vv \
    --buffer-size 512M \
    --drive-chunk-size 512M \
    --tpslimit 8 \
    --checkers 8 \
    --transfers 4 \
    --order-by modtime,$ModSort \
    --min-age $MinimumAge \
    $Command1 $Command2 $Command3 $Command4 $Command5 $Command6 $Command7 $Command8 \
    --exclude *fuse_hidden* \
    --exclude *_HIDDEN \
    --exclude .recycle** \
    --exclude .Recycle.Bin/** \
    --exclude *.backup~* \
    --exclude *.partial~* \
    --drive-stop-on-upload-limit \
    --bwlimit "${BWLimit1Time},${BWLimit1} ${BWLimit2Time},${BWLimit2} ${BWLimit3Time},${BWLimit3}" \
    --bind=$RCloneMountIP $DeleteEmpty

# Delete old files from mount
if [[  $BackupJob == 'Y' ]]; then
    echo "$(date "+%d.%m.%Y %T") INFO: *** Removing files older than ${BackupRetention} from $BackupRemoteLocation for ${RcloneUploadRemoteName}."
    rclone delete --min-age $BackupRetention $RcloneUploadRemoteName:$BackupRemoteDeletedLocation
fi

#######  Remove Control Files  ##########

# update counter and remove other control files
if [[  $UseServiceAccountUpload == 'Y' ]]; then
    if [[ "$CounterNumber" == "$CountServiceAccounts" ]];then
        rm /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/counter_*
        touch /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/counter_1
        echo "$(date "+%d.%m.%Y %T") INFO: Final counter used - resetting loop and created counter_1."
    else
        rm /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/counter_*
        CounterNumber=$((CounterNumber+1))
        touch /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/counter_$CounterNumber
        echo "$(date "+%d.%m.%Y %T") INFO: Created counter_${CounterNumber} for next upload run."
    fi
else
    echo "$(date "+%d.%m.%Y %T") INFO: Not utilising service accounts."
fi

# remove dummy file
rm /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/upload_running$JobName
echo "$(date "+%d.%m.%Y %T") INFO: Script complete"

exit

 

RClone Unmount:

 

#!/bin/bash

#######################
### Cleanup Script ####
#######################
#### Version 0.9.2 ####
#######################

echo "$(date "+%d.%m.%Y %T") INFO: *** Starting rclone_cleanup script ***"

####### Cleanup Tracking Files #######

echo "$(date "+%d.%m.%Y %T") INFO: *** Removing Tracking Files ***"

find /mnt/user/appdata/other/rclone/remotes -name dockers_started* -delete
find /mnt/user/appdata/other/rclone/remotes -name mount_running* -delete
find /mnt/user/appdata/other/rclone/remotes -name upload_running* -delete
echo "$(date "+%d.%m.%Y %T") INFO: ***Finished Cleanup! ***"

exit

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.