Guide: How To Use Rclone To Mount Cloud Drives And Play Files


DZMM

Recommended Posts

You just set up mergerfs - you don't really have to do anything :) I use a bunch of different sonarrs and radarrs to manage my splits. Year based is the easiest because you can allow for that in the series level folder. And movie too. If you need some help with moving stuff between team drives I can help you there.

 

The up loaders I guess can be switched but movies will not ever likely exceed the limit of a single drive.

 

TV - I currently use four different team drives. five if you count anime.

  • Like 1
  • Thanks 1
Link to comment
12 minutes ago, Spladge said:

You just set up mergerfs - you don't really have to do anything :) I use a bunch of different sonarrs and radarrs to manage my splits. Year based is the easiest because you can allow for that in the series level folder. And movie too. If you need some help with moving stuff between team drives I can help you there.

 

The up loaders I guess can be switched but movies will not ever likely exceed the limit of a single drive.

 

TV - I currently use four different team drives. five if you count anime.

Interresting way of doing it by date actually, if you share a bit more details on how you do it, that would be nice :)

isn't there a limitation on how much you can move between drives?

Link to comment

Well its very simple - I have a sonarr that looks after everything from 1980-1989 another for  1990-1999, 2000-20010 etc. Same for movies.

 

Those are then structured the same way in teamdrives. `teamdrive_name:tv/1980s` and then it all comes together in a merger under `/mergerdir/tv`.

  • Thanks 2
Link to comment
4 hours ago, DZMM said:

@Thel1988 I'm going to ditch the cut command tonight as it's not working 100%.  Will post an update later

Ok, Updates finished https://github.com/BinsonBuzz/unraid_rclone_mount.  Tidy ups to mount and cleanup script. 

 

Upload script has some good changes:

  1. configurable --min-age as part of the config section
  2. configurable --exclusion as part of the config section.  I've added 8 which should be plenty
  3. Service account counters work 100% now (I think!)
  4. Added ability to do backup jobs

For 99% of users this should mean the main script doesn't need touching. 

 

I added #4 to reduce the amount of edits I have to do to support my own jobs, including my backup job. Now that the main body supports my backup job, future updates will be much faster and they'll be fewer errors.

 

Edited by DZMM
  • Thanks 2
Link to comment

Hi just updated my scripts the mounting seems to work OK. But the upload errors out. I have just changed the remotename and remoteuploadname to the same remote just like the original scripts. here is the error:

 

Quote

Script Starting Wed, 12 Feb 2020 20:54:24 +1100

Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_upload/log.txt

12.02.2020 20:54:24 INFO: *** Rclone move selected. Files will be moved from /mnt/user/local/gdrive_media_vfs for gdrive_media_vfs ***
12.02.2020 20:54:24 INFO: *** Starting rclone_upload script for gdrive_media_vfs ***
12.02.2020 20:54:24 INFO: Script not running - proceeding.
12.02.2020 20:54:24 INFO: Checking if rclone installed successfully.
12.02.2020 20:54:24 INFO: rclone installed successfully - proceeding with upload.
12.02.2020 20:54:24 INFO: Uploading using upload remote gdrive_media_vfs
12.02.2020 20:54:24 INFO: *** Using rclone move - will add --delete-empty-src-dirs to upload.
Error: unknown flag: --order-by
Usage:
rclone move source:path dest:path [flags]

Flags:
--create-empty-src-dirs Create empty source dirs on destination after move
--delete-empty-src-dirs Delete empty source dirs after move
-h, --help help for move

Use "rclone [command] --help" for more information about a command.
Use "rclone help flags" for to see the global flags.
Use "rclone help backends" for a list of supported services.

2020/02/12 20:54:24 Fatal error: unknown flag: --order-by
12.02.2020 20:54:24 INFO: Not utilising service accounts.
12.02.2020 20:54:24 INFO: Script complete
Script Finished Wed, 12 Feb 2020 20:54:24 +1100

Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_upload/log.txt
 

I'm sure its something i have over looked. any help would be great.

 

here is my changes for the upload script

 

Quote

# REQUIRED SETTINGS
RcloneCommand="move" # choose your rclone command e.g. move, copy, sync
RcloneRemoteName="gdrive_media_vfs" # Name of rclone remote mount WITHOUT ':'.
RcloneUploadRemoteName="gdrive_media_vfs" # If you have a second remote created for uploads put it here.  Otherwise use the same remote as RcloneRemoteName.
LocalFilesShare="/mnt/user/local" # location of the local files without trailing slash you want to rclone to use
RcloneMountShare="/mnt/user/mount_rclone" # where your rclone mount is located without trailing slash  e.g. /mnt/user/mount_rclone
MinimumAge="10m" # sync files suffix ms|s|m|h|d|w|M|y

 

Link to comment
37 minutes ago, Nomad007 said:

Hi just updated my scripts the mounting seems to work OK. But the upload errors out. I have just changed the remotename and remoteuploadname to the same remote just like the original scripts.

 If all the custom exclusions weren't filled in and used, it errored out.  

 

I've just fixed this so they can be empty and now you can enter custom commands, rather than just exclusions - see v0.95.1 https://github.com/BinsonBuzz/unraid_rclone_mount/blob/latest---mergerfs-support/rclone_upload

 

# Add extra commands or filters
Command1="--exclude downloads/**"
Command2=""
Command3=""
Command4=""
Command5=""
Command6=""
Command7=""
Command8=""
# process files
	rclone $RcloneCommand $LocalFilesLocation $RcloneUploadRemoteName:$BackupRemoteLocation $BackupDir \
	--user-agent="$RcloneUploadRemoteName" \
	-vv \
	--buffer-size 512M \
	--drive-chunk-size 512M \
	--tpslimit 8 \
	--checkers 8 \
	--transfers 4 \
	--order-by modtime,ascending \
	--min-age $MinimumAge \
	$Command1 $Command2 $Command3 $Command4 $Command5 $Command6 $Command7 $Command8 \
	--exclude *fuse_hidden* \
	--exclude *_HIDDEN \
	--exclude .recycle** \
	--exclude .Recycle.Bin/** \
	--exclude *.backup~* \
	--exclude *.partial~* \
	--drive-stop-on-upload-limit \
	--bwlimit "${BWLimit1Time},${BWLimit1} ${BWLimit2Time},${BWLimit2} ${BWLimit3Time},${BWLimit3}" \
	--bind=$RCloneMountIP $DeleteEmpty

 

  • Thanks 1
Link to comment
1 hour ago, Spladge said:

Note that I have not checked yet, but is there an option /flag / fork for strictly unencypted remotes? If not I might convert once you have settled everything down and perhaps it could sit in another branch?

You can use any remote with the scripts

  • Like 1
  • Haha 1
Link to comment
On 2/11/2020 at 8:25 PM, Spladge said:

You just set up mergerfs - you don't really have to do anything :) 

@Spladge Thanks.  I wish I'd thought about adding more folders to my main mergerfs mount, before I created a new one just last month when my music folders & files pushed me over the 400K limit - I spent a fair while having to move, rescan and fix tag errors 😞

 

Anyway, planning for the future as I will probably need to create another remote at some point and because I know you're not the only person with more than two folders in their unions, I made some updates last night to the mount script https://github.com/BinsonBuzz/unraid_rclone_mount/blob/latest---mergerfs-support/rclone_mount

 

Quote

0.96.0 Changes:
- New: Added option to not create mergerfs mount (e.g. if just creating rclone mount to add to another mergerfs mount)
- New: Added option to add extra folders to mergerfs mount
- Fix: Only create mountcheck file if not already mounted
- Fix: Simplified bindmount

Scripts should be finished now, except for any bug fixes as the ability to add custom commands and extra mergerfs folders should cover any future ideas.

On 2/11/2020 at 8:25 PM, Spladge said:

I use a bunch of different sonarrs and radarrs to manage my splits. Year based is the easiest because you can allow for that in the series level folder. And movie too.

What's your folder structure?  At the moment for TV I've got the following folders added to Plex:

  • /mount_mergerfs/tv/kids/tv show (year)
  • /mount_mergerfs/tv/adults/tv show (year)
  • /mount_mergerfs/uhd/tv/kids/tv show (year)
  • /mount_mergerfs/uhd/tv/adults/tv show (year)

With 2x Sonarr dockers - one for non-UHD and one for UHD. 

 

If I add another sonarr instance in a new remote (assuming just HD), I can't see a way around having to add two more folders to plex:

  • /mount_mergerfs/new_sonarr/tv/kids
  • /mount_mergerfs/new_sonarr/tv/adults

Or, are you saying you do this with your dockers:

  • Docker1: /mount_mergerfs/tv/2000/tv show (2001)/
  • Docker2: /mount_mergerfs/tv/2010/tv show (2012)/
  • Docker3: /mount_mergerfs/tv/2020/tv show (2020)/

and Plex is ok if you only add /mount_mergerfs/tv to the library, and it can work out the rest?

Link to comment

I had a close look at the new scripts (mainly for the --drive-stop-on-upload-limit option) on Github and I have to say it makes the old ones look like Neanderthal stone tools. 😅

 

Am I right to say these are the new features compared to the old scripts?

  1. CreateBindMount - so the rclone download / upload goes through a different IP address from the server IP?
  2. mergerfs instead of unionfs - for hardlinks + no COW
  3. Service Accounts - automatic switching of upload accounts
  4. Backup / Upload switch - rclone sync vs rclone move
  5. Variable-ize various things instead of hard-coding
  6. BW Limit by time - looks like it's a rclone functionality and there's no logic in the script, unlike (1) -> (4)?

Just want to double check before I selectively adapt some to my current (old) scripts.

 

A small proposal: why don't you also variable-ize the script location (/mnt/user/appdata/other/rclone/remotes)?

Seems a bit out of place for all the other paths to be variables and this one is hard-coded.

Edited by testdasi
Link to comment
1 hour ago, testdasi said:

I had a close look at the new scripts (mainly for the --drive-stop-on-upload-limit option) on Github and I have to say it makes the old ones look like Neanderthal stone tools. 😅

I agree!  Although I'm sure someone who knows how to code will wonder why on earth I'm doing certain things, I've got no real script or coding experience and I feel like I've progressed a lot in the last two weeks.

 

1 hour ago, testdasi said:

Am I right to say these are the new features compared to the old scripts?

  1. CreateBindMount - so the rclone download / upload goes through a different IP address from the server IP?
  2. mergerfs instead of unionfs - for hardlinks + no COW
  3. Service Accounts - automatic switching of upload accounts
  4. Backup / Upload switch - rclone sync vs rclone move
  5. Variable-ize various things instead of hard-coding
  6. BW Limit by time - looks like it's a rclone functionality and there's no logic in the script, unlike (1) -> (4)?

Yes, I've basically added in things that I use that I hadn't included before as I was worried they'd make support harder e.g. bindmounts, bwlimits, multiple remotes etc as well as making it easier to setup with fewer potential issues = less support questions + encourage more people to try.  It's also setup so migration from unionfs or any other setup should only take a few mins now.

 

Rotating Service accounts is a big find for me - I think you were using my mount rotation version which although worked, looks a bit dumb now.

 

One feature you've missed is the latest one allowing extra folders to be added easily to mergerfs - again, I think this will reduce the number of issues on this thread.

 

Re bindmounts - yes.  I give each remote and upload_remote a unique IP to stop all the traffic appearing on my unRAID IP in pfsense.  This way I can traffic shape to give my streaming mounts top priority, and various priority levels for the other remotes.

 

Backup /upload switch - again added selfishly so the script can support my backup job.

 

Bwlimit - there's no logic as it's hardcoded into the rclone move job.  It should still work if anyone (wanted to) deleted the config entries - I haven't tried to see what happens, but I doubt anyone would do that.

 

1 hour ago, testdasi said:

A small proposal: why don't you also variable-ize the script location (/mnt/user/appdata/other/rclone/remotes)?

Seems a bit out of place for all the other paths to be variables and this one is hard-coded.

I went hard-coded as it makes the cleanup script a lot easier to run via user scripts as the files to be deleted are always in the same place.  If it was variable, users would have to enter the same location (or all the locations if they are running multiple scripts for multiple remotes)  they've chosen in the cleanup script.

Edited by DZMM
Link to comment

stupid question, but if i want to use this script i need modify :

/boot/config/plugins/rclone/scripts/rclone_mount_plugin/script ?

 

seems like i can't modify it via the gui (toolong?)

 

EDIT: busy installing:User Scripts

Edited by bar1
Link to comment

So I got the idea from someone here to use a VPS... and I managed to sucessfully set this up and it is working great! Using the scripts to upload to my teamdrive, freeing up my bandwidth!

 

So the VPS I use is a 1core, 1g ram, 50gb storage, 1g connection, 15tb/month for $35 a year. The setup is to just install docker and try to keep it minimal with letsencrypt, nzbget, and sonarr. 

 

Pros

  • scripts work great with a few tweaks to get best performance
  • free up your bandwidth or help upload even more

Cons

  • basic knowlege with linux, terminal, docker
  • your providers restrcitions
  • storage space for initial download (can't download anything bigger than your drive supports)
  • extra cost to use this setup
Link to comment
44 minutes ago, bar1 said:

ok i followed the instructions properly this time (i think...).

but the drive doesn't map.

i am getting this:

/tmp/user.scripts/tmpScripts/Rclone/script: line 206: syntax error: unexpected end of file

 

 

Posting the script you used would help!

Link to comment
2 hours ago, bar1 said:

#!/bin/bash

######################
#### Mount Script ####
######################
### Version 0.96.0 ###
######################

####### EDIT ONLY THESE SETTINGS #######

# INSTRUCTIONS
# 1. Edit the change the name of the rclone remote and shares to match your setup
# 2. NOTE: enter RcloneRemoteName WITHOUT ':'
# 3. Optional: include custom command and bind mount settings
# 4. Optional: include extra folders in mergerfs mount

# REQUIRED SETTINGS
RcloneRemoteName="Bar1-xxx2" # Name of rclone remote mount WITHOUT ':'. NOTE: Choose your encrypted remote for sensitive data
#RcloneRemoteName="Bar1-xxx" # Name of rclone remote mount WITHOUT ':'. NOTE: Choose your encrypted remote for sensitive data
LocalFilesShare="/mnt/user/Media" # location of the local files you want to upload without trailing slash to rclone e.g. /mnt/user/local
RcloneMountShare="/mnt/user/Bar1-xxx" # where your rclone remote will be located without trailing slash  e.g. /mnt/user/mount_rclone
MergerfsMountShare="ignore" # location without trailing slash  e.g. /mnt/user/mount_mergerfs of the share you will use for Plex, Sonarr etc that combines files on your server's 'LocalFilesShare' with files on gdrive 'RcloneMountShare'
DockerStart="Bar1-xxx" # list of dockers, separated by space, to start once mergerfs mount verified. Remember to disable AUTOSTART in docker settings page
MountFolders=\{"downloads/complete,movies,tv,comics"\} # comma separated list of folders to create within the mount

# Note: Again - remember to NOT use ':' in your remote name above

# OPTIONAL SETTINGS

# Add extra paths to mergerfs mount in addition to LocalFilesShare
LocalFilesShare2="ignore" # without trailing slash e.g. //mnt/user/Bar1-xxx.  Enter 'ignore' to disable
LocalFilesShare3="ignore"
LocalFilesShare4="ignore"

# Add extra commands or filters
Command1="--rc"
Command2=""
Command3=""
Command4=""
Command5=""
Command6=""
Command7=""
Command8=""

CreateBindMount="N" # Y/N. Choose whether to bind traffic to a particular network adapter
RCloneMountIP="192.168.1.201" # My unraid IP is 172.30.12.2 so I create another similar IP address
NetworkAdapter="eth0" # choose your network adapter. eth0 recommended
VirtualIPNumber="2" # creates eth0:x e.g. eth0:1.  I create a unique virtual IP addresses for each mount & upload so I can monitor and traffic shape for each of them

####### END SETTINGS #######

###############################################################################
########   DO NOT EDIT ANYTHING UNLESS YOU KNOW WHAT YOU ARE DOING   ##########
###############################################################################

####### Preparing mount location variables #######
LocalFilesLocation="$LocalFilesShare/$RcloneRemoteName" # Location for local files to be merged with rclone mount
RcloneMountLocation="$RcloneMountShare/$RcloneRemoteName" # Location for rclone mount
MergerFSMountLocation="$MergerfsMountShare/$RcloneRemoteName" # Rclone data folder location

####### create directories for rclone mount and mergerfs mounts #######
mkdir -p /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName # for script files
eval mkdir -p $LocalFilesLocation/"$MountFolders"
mkdir -p $RcloneMountLocation
mkdir -p $MergerFSMountLocation

#######  Check if script is already running  #######
echo "$(date "+%d.%m.%Y %T") INFO: *** Starting mount of remote ${RcloneRemoteName}"
echo "$(date "+%d.%m.%Y %T") INFO: Checking if this script is already running."
if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running" ]]; then
    echo "$(date "+%d.%m.%Y %T") INFO: Exiting script as already running."
    exit
else
    echo "$(date "+%d.%m.%Y %T") INFO: Script not running - proceeding."
    touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
fi

#######  Check If Rclone Mount Already Created  #######

if [[ -f "$RcloneMountLocation/mountcheck" ]]; then
    echo "$(date "+%d.%m.%Y %T") INFO: Success ${RcloneRemoteName} remote is already mounted."
else
    echo "$(date "+%d.%m.%Y %T") INFO: Mount not running. Will now mount ${RcloneRemoteName} remote."

#######  Create RClone Mount  #######

# Check If Rclone Mount Already Created
if [[ -f "$RcloneMountLocation/mountcheck" ]]; then
    echo "$(date "+%d.%m.%Y %T") INFO: Success ${RcloneRemoteName} remote is already mounted."
else
    echo "$(date "+%d.%m.%Y %T") INFO: Mount not running. Will now mount ${RcloneRemoteName} remote."
# Creating mountcheck file in case it doesn't already exist
    echo "$(date "+%d.%m.%Y %T") INFO: Recreating mountcheck file for ${RcloneRemoteName} remote."
    touch mountcheck
    rclone copy mountcheck $RcloneRemoteName: -vv --no-traverse
    echo "$(date "+%d.%m.%Y %T") INFO: Completed creation of mountcheck file for ${RcloneRemoteName} remote."
# Check bind option
    if [[  $CreateBindMount == 'Y' ]]; then
        echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}"
        ping -q -c2 $RCloneMountIP > /dev/null # -q quiet, -c number of pings to perform
        if [ $? -eq 0 ]; then # ping returns exit status 0 if successful
            echo "$(date "+%d.%m.%Y %T") INFO: *** IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}"
        else
            echo "$(date "+%d.%m.%Y %T") INFO: *** Creating IP address ${RCloneMountIP} for remote ${RcloneRemoteName}"
            ip addr add $RCloneMountIP/24 dev $NetworkAdapter label $NetworkAdapter:$VirtualIPNumber
        fi
    else
        RCloneMountIP=""
    fi
# create rclone mount
    echo "$(date "+%d.%m.%Y %T") INFO: *** Creating bind mount ${RCloneMountIP} for remote ${RcloneRemoteName}"
    rclone mount \
    --allow-other \
    --buffer-size 256M \
    --dir-cache-time 720h \
    --drive-chunk-size 512M \
    --log-level INFO \
    --vfs-read-chunk-size 128M \
    --vfs-read-chunk-size-limit off \
    --vfs-cache-mode writes \
    --bind=$RCloneMountIP \
    $RcloneRemoteName: $RcloneMountLocation &

# Check if Mount Successful
    echo "$(date "+%d.%m.%Y %T") INFO: sleeping for 5 seconds"
# slight pause to give mount time to finalise
    sleep 5
    echo "$(date "+%d.%m.%Y %T") INFO: continuing..."
    if [[ -f "$RcloneMountLocation/mountcheck" ]]; then
        echo "$(date "+%d.%m.%Y %T") INFO: Successful mount of ${RcloneRemoteName} mount."
    else
        echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mount failed - please check for problems."
        rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
        exit
    fi
fi

####### Start MergerFS Mount #######

if [[  $MergerfsMountShare == 'ignore' ]]; then
    echo "$(date "+%d.%m.%Y %T") INFO: Not creating mergerfs mount as requested."
else
    if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then
        echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount in place."
    else
# check if mergerfs already installed
        if [[ -f "/bin/mergerfs" ]]; then
            echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs already installed, proceeding to create mergerfs mount"
        else
# Build mergerfs binary
            echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs not installed - installing now."
            mkdir -p /mnt/user/appdata/other/rclone/mergerfs
            docker run -v /mnt/user/appdata/other/rclone/mergerfs:/build --rm trapexit/mergerfs-static-build
            mv /mnt/user/appdata/other/rclone/mergerfs/mergerfs /bin
        fi
# Create mergerfs mount
        echo "$(date "+%d.%m.%Y %T") INFO: Creating ${RcloneRemoteName} mergerfs mount."
# Extra Mergerfs folders
        if [[  $LocalFilesShare2 != 'ignore' ]]; then
            echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare2} to ${RcloneRemoteName} mergerfs mount."
            LocalFilesShare2=":$LocalFilesShare2"
        else
            LocalFilesShare2=""
        fi
        if [[  $LocalFilesShare3 != 'ignore' ]]; then
            echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare3} to ${RcloneRemoteName} mergerfs mount."
            LocalFilesShare3=":$LocalFilesShare3"
        else
            LocalFilesShare3=""
        fi

        if [[  $LocalFilesShare4 != 'ignore' ]]; then
            echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare4} to ${RcloneRemoteName} mergerfs mount."
            LocalFilesShare4=":$LocalFilesShare4"
        else
            LocalFilesShare4=""
        fi
# mergerfs mount command
        mergerfs $LocalFilesLocation:$RcloneMountLocation$LocalFilesShare2$LocalFilesShare3$LocalFilesShare4 $MergerFSMountLocation -o rw,async_read=false,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=partial,dropcacheonclose=true
# check if mergerfs mount successful
        echo "$(date "+%d.%m.%Y %T") INFO: Checking if ${RcloneRemoteName} mergerfs mount created."
        if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then
            echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount created."
        else
            echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mergerfs mount failed."
            rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
        exit
        fi
    fi
fi

####### Starting Dockers That Need Mergerfs Mount To Work Properly #######

# only start dockers once
if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started" ]]; then
    echo "$(date "+%d.%m.%Y %T") INFO: dockers already started."
else
    touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started
    echo "$(date "+%d.%m.%Y %T") INFO: Starting dockers."
    docker start $DockerStart
fi

rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
echo "$(date "+%d.%m.%Y %T") INFO: Script complete"

exit

can you post the full logs as can see where it's failing from when you run the script and your rclone config (remote passwords) please - can you use the code formatting as well when you post please.

Link to comment
3 hours ago, senpaibox said:

So I got the idea from someone here to use a VPS... and I managed to sucessfully set this up and it is working great! Using the scripts to upload to my teamdrive, freeing up my bandwidth!

 

So the VPS I use is a 1core, 1g ram, 50gb storage, 1g connection, 15tb/month for $35 a year. The setup is to just install docker and try to keep it minimal with letsencrypt, nzbget, and sonarr. 

 

Pros

  • scripts work great with a few tweaks to get best performance
  • free up your bandwidth or help upload even more

Cons

  • basic knowlege with linux, terminal, docker
  • your providers restrcitions
  • storage space for initial download (can't download anything bigger than your drive supports)
  • extra cost to use this setup

Nice way to get a 1G connection.  Which provider did you go with?

Link to comment
#!/bin/bash

######################
#### Mount Script ####
######################
### Version 0.96.0 ###
######################

####### EDIT ONLY THESE SETTINGS #######

# INSTRUCTIONS
# 1. Edit the change the name of the rclone remote and shares to match your setup
# 2. NOTE: enter RcloneRemoteName WITHOUT ':'
# 3. Optional: include custom command and bind mount settings
# 4. Optional: include extra folders in mergerfs mount

# REQUIRED SETTINGS
RcloneRemoteName="Bar1-xxx" # Name of rclone remote mount WITHOUT ':'. NOTE: Choose your encrypted remote for sensitive data
#RcloneRemoteName="Bar1-xxx" # Name of rclone remote mount WITHOUT ':'. NOTE: Choose your encrypted remote for sensitive data
LocalFilesShare="/mnt/user/Media" # location of the local files you want to upload without trailing slash to rclone e.g. /mnt/user/local
RcloneMountShare="/mnt/user/Bar1-Rxxx" # where your rclone remote will be located without trailing slash  e.g. /mnt/user/mount_rclone
MergerfsMountShare="ignore" # location without trailing slash  e.g. /mnt/user/mount_mergerfs of the share you will use for Plex, Sonarr etc that combines files on your server's 'LocalFilesShare' with files on gdrive 'RcloneMountShare'
DockerStart="Bar1-xxx" # list of dockers, separated by space, to start once mergerfs mount verified. Remember to disable AUTOSTART in docker settings page
MountFolders=\{"downloads/complete,downloads/intermediate,downloads/seeds,movies,tv"\} # comma separated list of folders to create within the mount
# Note: Again - remember to NOT use ':' in your remote name above

# OPTIONAL SETTINGS

# Add extra paths to mergerfs mount in addition to LocalFilesShare
LocalFilesShare2="ignore" # without trailing slash e.g. //mnt/user/.  Enter 'ignore' to disable
LocalFilesShare3="ignore"
LocalFilesShare4="ignore"

# Add extra commands or filters
Command1="--rc"
Command2=""
Command3=""
Command4=""
Command5=""
Command6=""
Command7=""
Command8=""

CreateBindMount="N" # Y/N. Choose whether to bind traffic to a particular network adapter
RCloneMountIP="192.168.1.201" # My unraid IP is 172.30.12.2 so I create another similar IP address
NetworkAdapter="eth0" # choose your network adapter. eth0 recommended
VirtualIPNumber="2" # creates eth0:x e.g. eth0:1.  I create a unique virtual IP addresses for each mount & upload so I can monitor and traffic shape for each of them

####### END SETTINGS #######

###############################################################################
########   DO NOT EDIT ANYTHING UNLESS YOU KNOW WHAT YOU ARE DOING   ##########
###############################################################################

####### Preparing mount location variables #######
LocalFilesLocation="$LocalFilesShare/$RcloneRemoteName" # Location for local files to be merged with rclone mount
RcloneMountLocation="$RcloneMountShare/$RcloneRemoteName" # Location for rclone mount
MergerFSMountLocation="$MergerfsMountShare/$RcloneRemoteName" # Rclone data folder location

####### create directories for rclone mount and mergerfs mounts #######
mkdir -p /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName # for script files
eval mkdir -p $LocalFilesLocation/"$MountFolders"
mkdir -p $RcloneMountLocation
mkdir -p $MergerFSMountLocation

#######  Check if script is already running  #######
echo "$(date "+%d.%m.%Y %T") INFO: *** Starting mount of remote ${RcloneRemoteName}"
echo "$(date "+%d.%m.%Y %T") INFO: Checking if this script is already running."
if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running" ]]; then
	echo "$(date "+%d.%m.%Y %T") INFO: Exiting script as already running."
	exit
else
	echo "$(date "+%d.%m.%Y %T") INFO: Script not running - proceeding."
	touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
fi

#######  Check If Rclone Mount Already Created  #######

if [[ -f "$RcloneMountLocation/mountcheck" ]]; then
	echo "$(date "+%d.%m.%Y %T") INFO: Success ${RcloneRemoteName} remote is already mounted."
else
	echo "$(date "+%d.%m.%Y %T") INFO: Mount not running. Will now mount ${RcloneRemoteName} remote."

#######  Create RClone Mount  #######

# Check If Rclone Mount Already Created
if [[ -f "$RcloneMountLocation/mountcheck" ]]; then
	echo "$(date "+%d.%m.%Y %T") INFO: Success ${RcloneRemoteName} remote is already mounted."
else
	echo "$(date "+%d.%m.%Y %T") INFO: Mount not running. Will now mount ${RcloneRemoteName} remote."
# Creating mountcheck file in case it doesn't already exist
	echo "$(date "+%d.%m.%Y %T") INFO: Recreating mountcheck file for ${RcloneRemoteName} remote."
	touch mountcheck
	rclone copy mountcheck $RcloneRemoteName: -vv --no-traverse
	echo "$(date "+%d.%m.%Y %T") INFO: Completed creation of mountcheck file for ${RcloneRemoteName} remote."
# Check bind option
	if [[  $CreateBindMount == 'Y' ]]; then
		echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}"
		ping -q -c2 $RCloneMountIP > /dev/null # -q quiet, -c number of pings to perform
		if [ $? -eq 0 ]; then # ping returns exit status 0 if successful
			echo "$(date "+%d.%m.%Y %T") INFO: *** IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}"
		else
			echo "$(date "+%d.%m.%Y %T") INFO: *** Creating IP address ${RCloneMountIP} for remote ${RcloneRemoteName}"
			ip addr add $RCloneMountIP/24 dev $NetworkAdapter label $NetworkAdapter:$VirtualIPNumber
		fi
	else
		RCloneMountIP=""
	fi
# create rclone mount
	echo "$(date "+%d.%m.%Y %T") INFO: *** Creating bind mount ${RCloneMountIP} for remote ${RcloneRemoteName}"
	rclone mount \
	--allow-other \
	--buffer-size 256M \
	--dir-cache-time 720h \
	--drive-chunk-size 512M \
	--log-level INFO \
	--vfs-read-chunk-size 128M \
	--vfs-read-chunk-size-limit off \
	--vfs-cache-mode writes \
	--bind=$RCloneMountIP \
	$RcloneRemoteName: $RcloneMountLocation &

# Check if Mount Successful
	echo "$(date "+%d.%m.%Y %T") INFO: sleeping for 5 seconds"
# slight pause to give mount time to finalise
	sleep 5
	echo "$(date "+%d.%m.%Y %T") INFO: continuing..."
	if [[ -f "$RcloneMountLocation/mountcheck" ]]; then
		echo "$(date "+%d.%m.%Y %T") INFO: Successful mount of ${RcloneRemoteName} mount."
	else
		echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mount failed - please check for problems."
		rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
		exit
	fi
fi

####### Start MergerFS Mount #######

if [[  $MergerfsMountShare == 'ignore' ]]; then
	echo "$(date "+%d.%m.%Y %T") INFO: Not creating mergerfs mount as requested."
else
	if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then
		echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount in place."
	else
# check if mergerfs already installed
		if [[ -f "/bin/mergerfs" ]]; then
			echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs already installed, proceeding to create mergerfs mount"
		else
# Build mergerfs binary
			echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs not installed - installing now."
			mkdir -p /mnt/user/appdata/other/rclone/mergerfs
			docker run -v /mnt/user/appdata/other/rclone/mergerfs:/build --rm trapexit/mergerfs-static-build
			mv /mnt/user/appdata/other/rclone/mergerfs/mergerfs /bin
		fi
# Create mergerfs mount
		echo "$(date "+%d.%m.%Y %T") INFO: Creating ${RcloneRemoteName} mergerfs mount."
# Extra Mergerfs folders
		if [[  $LocalFilesShare2 != 'ignore' ]]; then
			echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare2} to ${RcloneRemoteName} mergerfs mount."
			LocalFilesShare2=":$LocalFilesShare2"
		else
			LocalFilesShare2=""
		fi
		if [[  $LocalFilesShare3 != 'ignore' ]]; then
			echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare3} to ${RcloneRemoteName} mergerfs mount."
			LocalFilesShare3=":$LocalFilesShare3"
		else
			LocalFilesShare3=""
		fi

		if [[  $LocalFilesShare4 != 'ignore' ]]; then
			echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare4} to ${RcloneRemoteName} mergerfs mount."
			LocalFilesShare4=":$LocalFilesShare4"
		else
			LocalFilesShare4=""
		fi
# mergerfs mount command
		mergerfs $LocalFilesLocation:$RcloneMountLocation$LocalFilesShare2$LocalFilesShare3$LocalFilesShare4 $MergerFSMountLocation -o rw,async_read=false,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=partial,dropcacheonclose=true
# check if mergerfs mount successful
		echo "$(date "+%d.%m.%Y %T") INFO: Checking if ${RcloneRemoteName} mergerfs mount created."
		if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then
			echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount created."
		else
			echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mergerfs mount failed."
			rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
		exit
		fi
	fi
fi

####### Starting Dockers That Need Mergerfs Mount To Work Properly #######

# only start dockers once
if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started" ]]; then
	echo "$(date "+%d.%m.%Y %T") INFO: dockers already started."
else
	touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started
	echo "$(date "+%d.%m.%Y %T") INFO: Starting dockers."
	docker start $DockerStart
fi

rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running
echo "$(date "+%d.%m.%Y %T") INFO: Script complete"

exit

14.02.2020 13:44:20 INFO: *** Starting mount of remote Bar1-xxxx
14.02.2020 13:44:20 INFO: Checking if this script is already running.
14.02.2020 13:44:20 INFO: Script not running - proceeding.
/tmp/user.scripts/tmpScripts/Rclone/script: line 206: syntax error: unexpected end of file
Script Finished Fri, 14 Feb 2020 13:44:20 +0200

Full logs for this script are available at /tmp/user.scripts/tmpScripts/Rclone/log.txt
 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.