Guide: How To Use Rclone To Mount Cloud Drives And Play Files


DZMM

Recommended Posts

16 hours ago, privateer said:

Any clue what's going on here? I pulled my server back up and get this error - no changes to my scripts from when it was running...

 

13.10.2020 10:53:56 INFO: Creating local folders.
13.10.2020 10:53:56 INFO: *** Starting mount of remote gdrive_media_vfs
13.10.2020 10:53:56 INFO: Checking if this script is already running.
13.10.2020 10:53:56 INFO: Script not running - proceeding.
13.10.2020 10:53:56 INFO: *** Checking if online
13.10.2020 10:53:57 PASSED: *** Internet online
13.10.2020 10:53:57 INFO: Success gdrive_media_vfs remote is already mounted.
13.10.2020 10:53:57 INFO: Mergerfs already installed, proceeding to create mergerfs mount
13.10.2020 10:53:57 INFO: Creating gdrive_media_vfs mergerfs mount.
* ERROR: unable to parse 'branches' - ignore/gdrive_media_vfs:/mnt/user/mount_rclone/gdrive_media_vfs
* ERROR: mountpoint not set
13.10.2020 10:53:57 INFO: Checking if gdrive_media_vfs mergerfs mount created.
13.10.2020 10:53:57 CRITICAL: gdrive_media_vfs mergerfs mount failed. Stopping dockers.

think you've got a formatting error in your script options - post your script please

Link to comment
3 hours ago, jonodman said:

Hey guys pretty new to unraid ive just setup my server and with a complete fresh setup and rclone as the guide says i cant stop my array 

any ideas

 

Oct 13 13:45:49 Tower emhttpd: shcmd (109): sync
Oct 13 13:45:49 Tower emhttpd: shcmd (110): umount /mnt/user0
Oct 13 13:45:49 Tower emhttpd: shcmd (111): rmdir /mnt/user0
Oct 13 13:45:49 Tower emhttpd: shcmd (112): umount /mnt/user
Oct 13 13:45:49 Tower root: umount: /mnt/user: target is busy.
Oct 13 13:45:49 Tower emhttpd: shcmd (112): exit status: 32
Oct 13 13:45:49 Tower emhttpd: shcmd (113): rmdir /mnt/user
Oct 13 13:45:49 Tower root: rmdir: failed to remove '/mnt/user': Device or resource busy
Oct 13 13:45:49 Tower emhttpd: shcmd (113): exit status: 1
Oct 13 13:45:49 Tower emhttpd: shcmd (115): /usr/local/sbin/update_cron
Oct 13 13:45:49 Tower emhttpd: Retry unmounting user share(s)...
Oct 13 13:45:54 Tower emhttpd: shcmd (116): umount /mnt/user
Oct 13 13:45:54 Tower root: umount: /mnt/user: target is busy.

 

 
 

We've never been able to pinpoint the problem as it seems intermittent.  I haven't had problems for a few months now.

Link to comment
4 hours ago, DZMM said:

think you've got a formatting error in your script options - post your script please

 

#!/bin/bash

######################
#### Mount Script ####
######################
### Version 0.96.7 ###
######################

####### EDIT ONLY THESE SETTINGS #######

# INSTRUCTIONS
# 1. Change the name of the rclone remote and shares to match your setup
# 2. NOTE: enter RcloneRemoteName WITHOUT ':'
# 3. Optional: include custom command and bind mount settings
# 4. Optional: include extra folders in mergerfs mount

# REQUIRED SETTINGS
RcloneRemoteName="gdrive_media_vfs" # Name of rclone remote mount WITHOUT ':'. NOTE: Choose your encrypted remote for sensitive data
RcloneMountShare="/mnt/user/mount_rclone" # where your rclone remote will be located without trailing slash  e.g. /mnt/user/mount_rclone
LocalFilesShare="ignore" # location of the local files and MountFolders you want to upload without trailing slash to rclone e.g. /mnt/user/local. Enter 'ignore' to disable
MergerfsMountShare="/mnt/user/mount_unionfs" # location without trailing slash  e.g. /mnt/user/mount_mergerfs. Enter 'ignore' to disable
DockerStart="binhex-rtorrentvpn plex ombi tautulli radarr radarr4k sonarr sonarr4k varken" # list of dockers, separated by space, to start once mergerfs mount verified. Remember to disable AUTOSTART for dockers added in docker settings page
MountFolders=\{"downloads/complete,downloads/intermediate,downloads/seeds,movies,tv_shows,4kmovies"\} # comma separated list of folders to create within the mount

# Note: Again - remember to NOT use ':' in your remote name above

# OPTIONAL SETTINGS

# Add extra paths to mergerfs mount in addition to LocalFilesShare
LocalFilesShare2="ignore" # without trailing slash e.g. /mnt/user/other__remote_mount/or_other_local_folder.  Enter 'ignore' to disable
LocalFilesShare3="ignore"
LocalFilesShare4="ignore"

# Add extra commands or filters
Command1="--rc"
Command2=""
Command3=""
Command4=""
Command5=""
Command6=""
Command7=""
Command8=""

CreateBindMount="N" # Y/N. Choose whether to bind traffic to a particular network adapter
RCloneMountIP="192.168.1.123" # My unraid IP is 172.30.12.2 so I create another similar IP address
NetworkAdapter="eth0" # choose your network adapter. eth0 recommended
VirtualIPNumber="2" # creates eth0:x e.g. eth0:1.  I create a unique virtual IP addresses for each mount & upload so I can monitor and traffic shape for each of them

####### END SETTINGS #######

###############################################################################
#####   DO NOT EDIT ANYTHING BELOW UNLESS YOU KNOW WHAT YOU ARE DOING   #######
###############################################################################

####### Preparing mount location variables #######
RcloneMountLocation="$RcloneMountShare/$RcloneRemoteName" # Location for rclone mount
LocalFilesLocation="$LocalFilesShare/$RcloneRemoteName" # Location for local files to be merged with rclone mount
MergerFSMountLocation="$MergerfsMountShare/$RcloneRemoteName" # Rclone data folder location

####### create directories for rclone mount and mergerfs mounts #######
mkdir -p /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName #for script files
if [[  $LocalFileShare == 'ignore' ]]; then
    echo "$(date "+%d.%m.%Y %T") INFO: Not creating local folders as requested."
else
    echo "$(date "+%d.%m.%Y %T") INFO: Creating local folders."
    eval mkdir -p $LocalFilesLocation/"$MountFolders"
fi
mkdir -p $RcloneMountLocation
mkdir -p $MergerFSMountLocation

#######  Check if script is already running  #######
echo "$(date "+%d.%m.%Y %T") INFO: *** Starting mount of remote ${RcloneRemoteName}"
echo "$(date "+%d.%m.%Y %T") INFO: Checking if this script is already running."
if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running" ]]; then
    echo "$(date "+%d.%m.%Y %T") INFO: Exiting script as already running."
    exit
else
    echo "$(date "+%d.%m.%Y %T") INFO: Script not running - proceeding."
    touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
fi

####### Checking have connectivity #######

echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if online"
ping -q -c2 google.com > /dev/null # -q quiet, -c number of pings to perform
if [ $? -eq 0 ]; then # ping returns exit status 0 if successful
    echo "$(date "+%d.%m.%Y %T") PASSED: *** Internet online"
else
    echo "$(date "+%d.%m.%Y %T") FAIL: *** No connectivity.  Will try again on next run"
    rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
    exit
fi

#######  Create Rclone Mount  #######

# Check If Rclone Mount Already Created
if [[ -f "$RcloneMountLocation/mountcheck" ]]; then
    echo "$(date "+%d.%m.%Y %T") INFO: Success ${RcloneRemoteName} remote is already mounted."
else
    echo "$(date "+%d.%m.%Y %T") INFO: Mount not running. Will now mount ${RcloneRemoteName} remote."
# Creating mountcheck file in case it doesn't already exist
    echo "$(date "+%d.%m.%Y %T") INFO: Recreating mountcheck file for ${RcloneRemoteName} remote."
    touch mountcheck
    rclone copy mountcheck $RcloneRemoteName: -vv --no-traverse
# Check bind option
    if [[  $CreateBindMount == 'Y' ]]; then
        echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}"
        ping -q -c2 $RCloneMountIP > /dev/null # -q quiet, -c number of pings to perform
        if [ $? -eq 0 ]; then # ping returns exit status 0 if successful
            echo "$(date "+%d.%m.%Y %T") INFO: *** IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}"
        else
            echo "$(date "+%d.%m.%Y %T") INFO: *** Creating IP address ${RCloneMountIP} for remote ${RcloneRemoteName}"
            ip addr add $RCloneMountIP/24 dev $NetworkAdapter label $NetworkAdapter:$VirtualIPNumber
        fi
        echo "$(date "+%d.%m.%Y %T") INFO: *** Created bind mount ${RCloneMountIP} for remote ${RcloneRemoteName}"
    else
        RCloneMountIP=""
        echo "$(date "+%d.%m.%Y %T") INFO: *** Creating mount for remote ${RcloneRemoteName}"
    fi
# create rclone mount
    rclone mount \
    --allow-other \
    --buffer-size 256M \
    --dir-cache-time 720h \
    --drive-chunk-size 512M \
    --log-level INFO \
    --vfs-read-chunk-size 128M \
    --vfs-read-chunk-size-limit off \
    --vfs-cache-mode writes \
    --bind=$RCloneMountIP \
    $RcloneRemoteName: $RcloneMountLocation &

# Check if Mount Successful
    echo "$(date "+%d.%m.%Y %T") INFO: sleeping for 5 seconds"
# slight pause to give mount time to finalise
    sleep 5
    echo "$(date "+%d.%m.%Y %T") INFO: continuing..."
    if [[ -f "$RcloneMountLocation/mountcheck" ]]; then
        echo "$(date "+%d.%m.%Y %T") INFO: Successful mount of ${RcloneRemoteName} mount."
    else
        echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mount failed - please check for problems.  Stopping dockers"
        docker stop $DockerStart
        rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
        exit
    fi
fi

####### Start MergerFS Mount #######

if [[  $MergerfsMountShare == 'ignore' ]]; then
    echo "$(date "+%d.%m.%Y %T") INFO: Not creating mergerfs mount as requested."
else
    if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then
        echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount in place."
    else
# check if mergerfs already installed
        if [[ -f "/bin/mergerfs" ]]; then
            echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs already installed, proceeding to create mergerfs mount"
        else
# Build mergerfs binary
            echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs not installed - installing now."
            mkdir -p /mnt/user/appdata/other/rclone/mergerfs
            docker run -v /mnt/user/appdata/other/rclone/mergerfs:/build --rm trapexit/mergerfs-static-build
            mv /mnt/user/appdata/other/rclone/mergerfs/mergerfs /bin
# check if mergerfs install successful
            echo "$(date "+%d.%m.%Y %T") INFO: *sleeping for 5 seconds"
            sleep 5
            if [[ -f "/bin/mergerfs" ]]; then
                echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs installed successfully, proceeding to create mergerfs mount."
            else
                echo "$(date "+%d.%m.%Y %T") ERROR: Mergerfs not installed successfully.  Please check for errors.  Exiting."
                rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
                exit
            fi
        fi
# Create mergerfs mount
        echo "$(date "+%d.%m.%Y %T") INFO: Creating ${RcloneRemoteName} mergerfs mount."
# Extra Mergerfs folders
        if [[  $LocalFilesShare2 != 'ignore' ]]; then
            echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare2} to ${RcloneRemoteName} mergerfs mount."
            LocalFilesShare2=":$LocalFilesShare2"
        else
            LocalFilesShare2=""
        fi
        if [[  $LocalFilesShare3 != 'ignore' ]]; then
            echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare3} to ${RcloneRemoteName} mergerfs mount."
            LocalFilesShare3=":$LocalFilesShare3"
        else
            LocalFilesShare3=""
        fi
        if [[  $LocalFilesShare4 != 'ignore' ]]; then
            echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare4} to ${RcloneRemoteName} mergerfs mount."
            LocalFilesShare4=":$LocalFilesShare4"
        else
            LocalFilesShare4=""
        fi
# make sure mergerfs mount point is empty
        mv $MergerFSMountLocation $LocalFilesLocation
        mkdir -p $MergerFSMountLocation
# mergerfs mount command
        mergerfs $LocalFilesLocation:$RcloneMountLocation$LocalFilesShare2$LocalFilesShare3$LocalFilesShare4 $MergerFSMountLocation -o rw,async_read=false,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=partial,dropcacheonclose=true
# check if mergerfs mount successful
        echo "$(date "+%d.%m.%Y %T") INFO: Checking if ${RcloneRemoteName} mergerfs mount created."
        if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then
            echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount created."
        else
            echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mergerfs mount failed.  Stopping dockers."
            docker stop $DockerStart
            rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
            exit
        fi
    fi
fi

####### Starting Dockers That Need Mergerfs Mount To Work Properly #######

# only start dockers once
if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started" ]]; then
    echo "$(date "+%d.%m.%Y %T") INFO: dockers already started."
else
    touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started
    echo "$(date "+%d.%m.%Y %T") INFO: Starting dockers."
    docker start $DockerStart
fi

rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
echo "$(date "+%d.%m.%Y %T") INFO: Script complete"

exit
 

Link to comment
1 hour ago, privateer said:

LocalFilesShare="ignore" # location of the local files and MountFolders you want to upload without trailing slash to rclone e.g. /mnt/user/local. Enter 'ignore' to disable
MergerfsMountShare="/mnt/user/mount_unionfs" # location without trailing slash  e.g. /mnt/user/mount_mergerfs. Enter 'ignore' to disable

ok, logic fail in my script.  If you don't want to create a mergerfs mount you need to set:

 

MergerfsMountShare="ignore"

You haven't, so it's trying to create a to create a mergerfs mount - but it can't because LocalFileShare is set to ignore.

 

To fix either set MergerfsMountShare to ignore, or add a location for LocalFileShare

Link to comment
15 hours ago, DZMM said:

can you post a copy of your email please if you can as mine didn't have that detail

Sorry just checked and pricing wasn't actually in the email, but following the instructions on upgrading it looks like Enterprise Standard has unlimited storage with only one user needed for 20/month. Bit more than I pay now but I guess it's still cheaper than if they enforced the 5 user thing on business.

https://support.google.com/a/answer/10085477?authuser=2

I haven't upgraded yet, I might hold off until they make me!

 

Edited by tsmebro
  • Thanks 1
Link to comment

I want to make sure I'm understanding this correctly.

 

Steps I intend to take for shares:

  1. Create User Shares with names of gDriveStaging(LocalFileShare), gDrive(RcloneMountShare), mergerFS(MergerfsMountShare) and set to "Prefer Cache"
  2. Set the variables in the scripts to match

 

Rclone:

  • With encryption, RcloneRemoteName should be the crypt remote
  • When the array is stopped, Unraid will handle the unmount as it's a User Share
  • When the array is started, the mount script will just need to be ran again

 

Random questions:

  • Is there a particular reason why vfs-cache-mode is writes, instead of full? Any gotchas when using full(I assume that a cache-dir would  need to be set to be sure it doesn't try to full up the root fs)?
  • How do the service accounts work? It look like the counter is updated, but the next service account will only be used at the next upload interval? Do the service accounts help with the 750GB upload limit?
  • It is mentioned that MergerFS works better when everything is mapped like /user --Maps to--> /mnt/user
    • Would there be any issues with the following setup?
      • Current Setup:
        • Google Drive path: /mnt/disks/gdrive_secure/
        • Radarr: /ShareName ---> /mnt/disks/gdrive_secure/ShareName
        • Sonarr: /ShareName/TV ---> /mnt/disks/gdrive_secure/ShareName/TV
      • Proposed Setup
        • MergerFS Path: /mnt/user/gDrive
        • Radarr: /ShareName ---> /mnt/user/gDrive/ShareName
        • Sonarr: /ShareName/TV ---> /mnt/user/gDrive/ShareName/TV
    • The goal is to avoid having to remap all of the directories and rescan everything. But if mergerFS is going to make it a pain, rescanning/remapping isn't the end of the world
Edited by M1kep_
Link to comment
10 hours ago, tsmebro said:

Sorry just checked and pricing wasn't actually in the email, but following the instructions on upgrading it looks like Enterprise Standard has unlimited storage with only one user needed for 20/month. Bit more than I pay now but I guess it's still cheaper than if they enforced the 5 user thing on business.

https://support.google.com/a/answer/10085477?authuser=2

I haven't upgraded yet, I might hold off until they make me!

 

Thanks.  I was too scared to click on the upgrade link before.... I went for it this time and it's spitting out £15.30/mth in the UK - strangely no discount for annual.  I'm paying £7.82/mth at the moment - double the price, but still great value for over 600TB of storage!!  

 

I'm not upgrading either yet - unless I can find a coupon for Enterprise Standard.

 

It's a good sign that Google aren't shutting down the facility - there are bigger things behind this change, rather than trying to address users like us.  I still believe we are a drop in the ocean Vs some institutions e.g. research universities.

 

827745545_FireShotCapture393-Switchyoursubscription-AdminConsole-admin.google_com.thumb.png.5892a268d817b466c602fc627e974026.png

 

  • Thanks 1
Link to comment
11 hours ago, M1kep_ said:

Create User Shares with names of gDriveStaging(LocalFileShare), gDrive(RcloneMountShare), mergerFS(MergerfsMountShare) and set to "Prefer Cache"

 
 
 
 
 

prefer cache is a preference - go with wherever you want files storing

 

11 hours ago, M1kep_ said:

When the array is started, the mount script will just need to be ran again

Correct.  Advice is to run using user scripts on a schedule e.g. every 2-5 mins, so if for whatever reason the mount fails, it will stop the dockers to try and avoid any meltdowns

 

11 hours ago, M1kep_ said:

Is there a particular reason why vfs-cache-mode is writes, instead of full?

 
 
 
 
 

since rclone 1.53 (I think) full works better - before it was a bad idea.  I've updated my local scripts, but I had a few problems updating github so that's a bit behind (it also takes advantage of the new cache feature, which makes a BIG difference).  I'll try again to update github again this weekend.

 

11 hours ago, M1kep_ said:

How do the service accounts work? It look like the counter is updated, but the next service account will only be used at the next upload interval? Do the service accounts help with the 750GB upload limit?

 
 
 
 
 

Correct - rclone added a nice feature a few releases back that stopped a transfer job when the 750GB limit is hit.  SAs allow a different user or quota to be used on the next run, so if you set a cron job, you can do multiple amounts of 750GB/day e.g. 16 SAs allows you to max out a 1Gbps upload 24/7.

 

11 hours ago, M1kep_ said:

Proposed Setup

  • MergerFS Path: /mnt/user/gDrive
  • Radarr: /ShareName ---> /mnt/user/gDrive/ShareName
  • Sonarr: /ShareName/TV ---> /mnt/user/gDrive/ShareName/TV

 

 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

yeah, you won't get the full benefit of mergerfs.  I would honestly just do this for all your dockers:

 

/user --> /mnt/user 

 

and then within radarr add as your media locations /user/gDrive/ShareName/Movies

 

etc etc

 

I just find it easier to do this as all dockers will always match up, and I think the convention of mapping /downloads --> xxxxxx /media --> xxxxx etc, just makes live more complicated.  Moving is easy:

 

1. add the new /user --> /mnt/user mapping

2. add it /user/whatever as a new media location in radarr

3. go to Movie Editor, create a custom filter e.g. show movies with /ShareName/Movies etc in path

4. select all and move them to /user/whatever within Radarr

 

 

Edited by DZMM
Link to comment
9 hours ago, DZMM said:

since rclone 1.53 (I think) full works better - before it was a bad idea.  I've updated my local scripts, but I had a few problems updating github so that's a bit behind (it also takes advantage of the new cache feature, which makes a BIG difference).  I'll try again to update github again this weekend.

Awesome! Yeah, the vfs local caching is really awesome. Really merges the gap between hosting on cloud and locally. Especially for files that are being accessed recently/often. I'll keep an eye out on the GitHub to see if any changes show up. If you don't get a chance to update, if you could possibly provide your mount flags at least that'd be very useful.

 

 

I'll probably work on getting this migrated over this weekend. Thanks again for the help

Link to comment
34 minutes ago, M1kep_ said:

Awesome! Yeah, the vfs local caching is really awesome. Really merges the gap between hosting on cloud and locally. Especially for files that are being accessed recently/often. I'll keep an eye out on the GitHub to see if any changes show up. If you don't get a chance to update, if you could possibly provide your mount flags at least that'd be very useful.

 

 

I'll probably work on getting this migrated over this weekend. Thanks again for the help

 
 

Here's my updated mount script.  I had problems adding @watchmeexplode5's pull that creates a single config file.   Adds support for vfs local caching and also changed the mount defaults, which for me, has significantly improved playback and skipping:

 

Edit: I just spotted I've hardcoded the cache size at 400GB - I'll make this configurable when I have time to sort out github.

#!/bin/bash

######################
#### Mount Script ####
######################
### Version 0.96.8 ###
######################

####### EDIT ONLY THESE SETTINGS #######

# INSTRUCTIONS
# 1. Change the name of the rclone remote and shares to match your setup
# 2. NOTE: enter RcloneRemoteName WITHOUT ':'
# 3. Optional: include custom command and bind mount settings
# 4. Optional: include extra folders in mergerfs mount

# REQUIRED SETTINGS
RcloneRemoteName="tdrive_vfs"
RcloneMountShare="/mnt/user/mount_rclone"
LocalFilesShare="/mnt/user/local"
MergerfsMountShare="/mnt/user/mount_mergerfs"
DockerStart="duplicati lazylibrarian LDAPforPlex letsencrypt nzbget ombi organizrv2 plex qbittorrentvpn radarr radarr-uhd radarr-collections sonarr sonarr-uhd tautulli"
MountFolders=\{"downloads/complete,downloads/seeds,documentaries/kids,documentaries/adults,movies_adults_gd,movies_kids_gd,tv_adults_gd,tv_kids_gd,uhd/tv_adults_gd,uhd/tv_kids_gd,uhd/documentaries/kids,uhd/documentaries/adults"\}

# OPTIONAL SETTINGS

# Add extra paths to mergerfs mount in addition to LocalFilesShare
LocalFilesShare2="/mnt/user/mount_rclone/gdrive_media_vfs" # without trailing slash e.g. /mnt/user/other__remote_mount/or_other_local_folder.  Enter 'ignore' to disable
LocalFilesShare3=""
LocalFilesShare4=""
LocalFilesShare5=""
LocalFilesShare6=""

# Add extra commands or filters
Command1="--rc"
Command2=""
Command3=""
Command4=""
Command5=""
Command6=""
Command7=""
Command8=""

CreateBindMount="N" # Y/N. Choose whether to bind traffic to a particular network adapter
RCloneMountIP="192.168.1.252" # My unraid IP is 172.30.12.2 so I create another similar IP address
NetworkAdapter="eth0" # choose your network adapter. eth0 recommended
VirtualIPNumber="2" # creates eth0:x e.g. eth0:1.  I create a unique virtual IP addresses for each mount & upload so I can monitor and traffic shape for each of them

####### END SETTINGS #######

###############################################################################
#####   DO NOT EDIT ANYTHING BELOW UNLESS YOU KNOW WHAT YOU ARE DOING   #######
###############################################################################

####### Preparing mount location variables #######
LocalFilesLocation="$LocalFilesShare/$RcloneRemoteName"
RcloneMountLocation="$RcloneMountShare/$RcloneRemoteName"
MergerFSMountLocation="$MergerfsMountShare/$RcloneRemoteName"

####### create directories for rclone mount and mergerfs mounts #######
mkdir -p /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName #for script files
mkdir -p /mnt/user0/mount_rclone/cache/$RcloneRemoteName #for cache files
if [[  $LocalFileShare == 'ignore' ]]; then
	echo "$(date "+%d.%m.%Y %T") INFO: Not creating local folders as requested."
else
	echo "$(date "+%d.%m.%Y %T") INFO: Creating local folders."
	eval mkdir -p $LocalFilesLocation/"$MountFolders"
fi
mkdir -p $RcloneMountLocation
mkdir -p $MergerFSMountLocation

#######  Check if script is already running  #######
echo "$(date "+%d.%m.%Y %T") INFO: *** Starting mount of remote ${RcloneRemoteName}"
echo "$(date "+%d.%m.%Y %T") INFO: Checking if this script is already running."
if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running" ]]; then
	echo "$(date "+%d.%m.%Y %T") INFO: Exiting script as already running."
	exit
else
	echo "$(date "+%d.%m.%Y %T") INFO: Script not running - proceeding."
	touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
fi

#######  Create Rclone Mount  #######

# Check If Rclone Mount Already Created
if [[ -f "$RcloneMountLocation/mountcheck" ]]; then
	echo "$(date "+%d.%m.%Y %T") INFO: Success ${RcloneRemoteName} remote is already mounted."
else
	echo "$(date "+%d.%m.%Y %T") INFO: Mount not running. Will now mount ${RcloneRemoteName} remote."
# Creating mountcheck file in case it doesn't already exist
	echo "$(date "+%d.%m.%Y %T") INFO: Recreating mountcheck file for ${RcloneRemoteName} remote."
	touch mountcheck
	rclone copy mountcheck $RcloneRemoteName: -vv --no-traverse
# Check bind option
	if [[  $CreateBindMount == 'Y' ]]; then
		echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}"
		ping -q -c2 $RCloneMountIP > /dev/null # -q quiet, -c number of pings to perform
		if [ $? -eq 0 ]; then # ping returns exit status 0 if successful
			echo "$(date "+%d.%m.%Y %T") INFO: *** IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}"
		else
			echo "$(date "+%d.%m.%Y %T") INFO: *** Creating IP address ${RCloneMountIP} for remote ${RcloneRemoteName}"
			ip addr add $RCloneMountIP/24 dev $NetworkAdapter label $NetworkAdapter:$VirtualIPNumber
		fi
		echo "$(date "+%d.%m.%Y %T") INFO: *** Created bind mount ${RCloneMountIP} for remote ${RcloneRemoteName}"
	else
		RCloneMountIP=""
		echo "$(date "+%d.%m.%Y %T") INFO: *** Creating mount for remote ${RcloneRemoteName}"
	fi
# create rclone mount
	rclone mount \
	--allow-other \
	--dir-cache-time 720h \
	--log-level INFO \
	--poll-interval 15s \
	--cache-dir=/mnt/user0/mount_rclone/cache/$RcloneRemoteName \
	--vfs-cache-mode full \
	--vfs-cache-max-size 400G \
	--vfs-cache-max-age 336h \
	--bind=$RCloneMountIP \
	$RcloneRemoteName: $RcloneMountLocation &

# Check if Mount Successful
	echo "$(date "+%d.%m.%Y %T") INFO: sleeping for 5 seconds"
# slight pause to give mount time to finalise
	sleep 10
	echo "$(date "+%d.%m.%Y %T") INFO: continuing..."
	if [[ -f "$RcloneMountLocation/mountcheck" ]]; then
		echo "$(date "+%d.%m.%Y %T") INFO: Successful mount of ${RcloneRemoteName} mount."
	else
		echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mount failed - please check for problems."
		docker stop $DockerStart
		find /mnt/user/appdata/other/rclone/remotes -name mount_running* -delete
		rm /mnt/user/appdata/other/scripts/running/fast_check
		exit
	fi
fi

####### Start MergerFS Mount #######

if [[  $MergerfsMountShare == 'ignore' ]]; then
	echo "$(date "+%d.%m.%Y %T") INFO: Not creating mergerfs mount as requested."
else
	if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then
		echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount in place."
	else
# check if mergerfs already installed
		if [[ -f "/bin/mergerfs" ]]; then
			echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs already installed, proceeding to create mergerfs mount"
		else
# Build mergerfs binary
			echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs not installed - installing now."
			mkdir -p /mnt/user/appdata/other/rclone/mergerfs
			docker run -v /mnt/user/appdata/other/rclone/mergerfs:/build --rm trapexit/mergerfs-static-build
			mv /mnt/user/appdata/other/rclone/mergerfs/mergerfs /bin
# check if mergerfs install successful
			echo "$(date "+%d.%m.%Y %T") INFO: *sleeping for 5 seconds"
			sleep 10
			if [[ -f "/bin/mergerfs" ]]; then
				echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs installed successfully, proceeding to create mergerfs mount."
			else
				echo "$(date "+%d.%m.%Y %T") ERROR: Mergerfs not installed successfully.  Please check for errors.  Exiting."
				docker stop $DockerStart
				find /mnt/user/appdata/other/rclone/remotes -name mount_running* -delete
				rm /mnt/user/appdata/other/scripts/running/fast_check
				exit
			fi
		fi
# Create mergerfs mount
		echo "$(date "+%d.%m.%Y %T") INFO: Creating ${RcloneRemoteName} mergerfs mount."
# Extra Mergerfs folders
		if [[  $LocalFilesShare2 != 'ignore' ]]; then
			echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare2} to ${RcloneRemoteName} mergerfs mount."
			LocalFilesShare2=":$LocalFilesShare2"
		else
			LocalFilesShare2=""
		fi
		if [[  $LocalFilesShare3 != 'ignore' ]]; then
			echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare3} to ${RcloneRemoteName} mergerfs mount."
			LocalFilesShare3=":$LocalFilesShare3"
		else
			LocalFilesShare3=""
		fi
		if [[  $LocalFilesShare4 != 'ignore' ]]; then
			echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare4} to ${RcloneRemoteName} mergerfs mount."
			LocalFilesShare4=":$LocalFilesShare4"
		else
			LocalFilesShare4=""
		fi
		if [[  $LocalFilesShare5 != 'ignore' ]]; then
			echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare5} to ${RcloneRemoteName} mergerfs mount."
			LocalFilesShare5=":$LocalFilesShare5"
		else
			LocalFilesShare5=""
		fi
		if [[  $LocalFilesShare6 != 'ignore' ]]; then
			echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare6} to ${RcloneRemoteName} mergerfs mount."
			LocalFilesShare6=":$LocalFilesShare6"
		else
			LocalFilesShare6=""
		fi
# mergerfs mount command
		mergerfs $LocalFilesLocation:$RcloneMountLocation$LocalFilesShare2$LocalFilesShare3$LocalFilesShare4$LocalFilesShare5$LocalFilesShare6 $MergerFSMountLocation -o rw,async_read=false,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=partial,dropcacheonclose=true
# check if mergerfs mount successful
		echo "$(date "+%d.%m.%Y %T") INFO: Checking if ${RcloneRemoteName} mergerfs mount created."
		if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then
			echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount created."
		else
			echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mergerfs mount failed."
			docker stop $DockerStart
			find /mnt/user/appdata/other/rclone/remotes -name mount_running* -delete
			rm /mnt/user/appdata/other/scripts/running/fast_check
			exit
		fi
	fi
fi

####### Starting Dockers That Need Mergerfs Mount To Work Properly #######

echo "$(date "+%d.%m.%Y %T") INFO: Starting dockers."
docker start $DockerStart

rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
echo "$(date "+%d.%m.%Y %T") INFO: ${RcloneRemoteName} Script complete"

exit

 

Edited by DZMM
Link to comment

Okay, pretty much finished going through the whole thing and building out my setup plan. Two more questions:

 

1 hour ago, DZMM said:

--cache-dir=/mnt/user0/mount_rclone/cache/$RcloneRemoteName \


I've found that the /user0/ mount writes to the disks instead of the cache. With that being said, what is your mount_rclone share set as for the cache settings in Unraid? With Prefer, I feel like the mover would constantly try to move the data off of the array and throw it on cache.

Also, does having the cache in the mount_rclone folder cause it to be uploaded to google as well as that's a mounted folder?

 

1 hour ago, DZMM said:

# Add extra paths to mergerfs mount in addition to LocalFilesShare

LocalFilesShare2="/mnt/user/mount_rclone/gdrive_media_vfs" # without trailing slash e.g. /mnt/user/other__remote_mount/or_other_local_folder. Enter 'ignore' to disable

Is this needed, or an artifact of your specific setup and can be replaced with "ignore"?

Edited by M1kep_
Link to comment
52 minutes ago, M1kep_ said:

Okay, pretty much finished going through the whole thing and building out my setup plan. Two more questions:

 


I've found that the /user0/ mount writes to the disks instead of the cache. With that being said, what is your mount_rclone share set as for the cache settings in Unraid? With Prefer, I feel like the mover would constantly try to move the data off of the array and throw it on cache.

Also, does having the cache in the mount_rclone folder cause it to be uploaded to google as well as that's a mounted folder?

 

Is this needed, or an artifact of your specific setup and can be replaced with "ignore"?

my cache isn't big enough for a permanent vfs cache.  choose your own preferred location

 

53 minutes ago, M1kep_ said:

Is this needed, or an artifact of your specific setup and can be replaced with "ignore"?

put to ignore - I left it in as an example - in my real script all 6 are populated with various tdrives.

Link to comment

I just spotted this in my upload logs:

 

2020/10/16 08:37:03 INFO :
Transferred: 1.344T / 9.679 TBytes, 14%, 13.292 MBytes/s, ETA 1w14h39m29s
Checks: 286 / 290, 99%
Deleted: 143
Renamed: 143
Transferred: 143 / 2190, 7%
Elapsed time: 29h27m1.0s

I wonder if Google have removed/increased the 750GB/day transfer limit?

 

Edit: I think it's still 750GB/day - my elapsed time is high so I must have fluked transferring less than 750GB over 24 hours 

Edited by DZMM
Link to comment

I have recently started using PiHole to block ads and route my DNS. The issue i've run into is that now the mount script fails because it tries to run before the internet connection is reestablished since PiHole is starting up at the same time. Is there a way to delay the script running a little bit to allow PiHole it get up and running?

 

Generally it isn't a problem because I just re-run the startup mount script, but if the server resets when i'm away from home everything stays down until I re-run the script.

 

I 100% know this is my fault for the setup I have running and I really appreciate all the work and help you guys provide, just wanted to see if there is an idea on how to deal with this.

Link to comment
1 hour ago, veritas2884 said:

I have recently started using PiHole to block ads and route my DNS. The issue i've run into is that now the mount script fails because it tries to run before the internet connection is reestablished since PiHole is starting up at the same time. Is there a way to delay the script running a little bit to allow PiHole it get up and running?

 

Generally it isn't a problem because I just re-run the startup mount script, but if the server resets when i'm away from home everything stays down until I re-run the script.

 

I 100% know this is my fault for the setup I have running and I really appreciate all the work and help you guys provide, just wanted to see if there is an idea on how to deal with this.

The mount script checks if there's internet connectivity.  Add a cron job so the script runs if the test fails.

 

####### Checking have connectivity #######

echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if online"
ping -q -c2 google.com > /dev/null # -q quiet, -c number of pings to perform
if [ $? -eq 0 ]; then # ping returns exit status 0 if successful
	echo "$(date "+%d.%m.%Y %T") PASSED: *** Internet online"
else
	echo "$(date "+%d.%m.%Y %T") FAIL: *** No connectivity.  Will try again on next run"
	rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
	exit
fi

 

Link to comment
2 hours ago, animeking said:

need some assistance, i keep getting the gdrive api ban. im really not a rclone expert with writing scripts or commands to stop the bans. so if someone is willing to guide me i would be happy to listen so i can stop getting the api bans. so far all i have accomplished is mounted gdrive and made a script. i followed the spaceinvader... 

image.png.45b17c375693fc5c69620aff6cf6ad3b.png

 

Have you looked at page 1, all the scripts are linked there. There is already a turn key solution in place.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.