Spladge Posted February 11, 2020 Share Posted February 11, 2020 You just set up mergerfs - you don't really have to do anything I use a bunch of different sonarrs and radarrs to manage my splits. Year based is the easiest because you can allow for that in the series level folder. And movie too. If you need some help with moving stuff between team drives I can help you there. The up loaders I guess can be switched but movies will not ever likely exceed the limit of a single drive. TV - I currently use four different team drives. five if you count anime. 1 1 Quote Link to comment
Thel1988 Posted February 11, 2020 Share Posted February 11, 2020 12 minutes ago, Spladge said: You just set up mergerfs - you don't really have to do anything I use a bunch of different sonarrs and radarrs to manage my splits. Year based is the easiest because you can allow for that in the series level folder. And movie too. If you need some help with moving stuff between team drives I can help you there. The up loaders I guess can be switched but movies will not ever likely exceed the limit of a single drive. TV - I currently use four different team drives. five if you count anime. Interresting way of doing it by date actually, if you share a bit more details on how you do it, that would be nice isn't there a limitation on how much you can move between drives? Quote Link to comment
Spladge Posted February 11, 2020 Share Posted February 11, 2020 Well its very simple - I have a sonarr that looks after everything from 1980-1989 another for 1990-1999, 2000-20010 etc. Same for movies. Those are then structured the same way in teamdrives. `teamdrive_name:tv/1980s` and then it all comes together in a merger under `/mergerdir/tv`. 2 Quote Link to comment
DZMM Posted February 12, 2020 Author Share Posted February 12, 2020 (edited) 4 hours ago, DZMM said: @Thel1988 I'm going to ditch the cut command tonight as it's not working 100%. Will post an update later Ok, Updates finished https://github.com/BinsonBuzz/unraid_rclone_mount. Tidy ups to mount and cleanup script. Upload script has some good changes: configurable --min-age as part of the config section configurable --exclusion as part of the config section. I've added 8 which should be plenty Service account counters work 100% now (I think!) Added ability to do backup jobs For 99% of users this should mean the main script doesn't need touching. I added #4 to reduce the amount of edits I have to do to support my own jobs, including my backup job. Now that the main body supports my backup job, future updates will be much faster and they'll be fewer errors. Edited February 12, 2020 by DZMM 2 Quote Link to comment
Nomad007 Posted February 12, 2020 Share Posted February 12, 2020 Hi just updated my scripts the mounting seems to work OK. But the upload errors out. I have just changed the remotename and remoteuploadname to the same remote just like the original scripts. here is the error: Quote Script Starting Wed, 12 Feb 2020 20:54:24 +1100 Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_upload/log.txt 12.02.2020 20:54:24 INFO: *** Rclone move selected. Files will be moved from /mnt/user/local/gdrive_media_vfs for gdrive_media_vfs *** 12.02.2020 20:54:24 INFO: *** Starting rclone_upload script for gdrive_media_vfs *** 12.02.2020 20:54:24 INFO: Script not running - proceeding. 12.02.2020 20:54:24 INFO: Checking if rclone installed successfully. 12.02.2020 20:54:24 INFO: rclone installed successfully - proceeding with upload. 12.02.2020 20:54:24 INFO: Uploading using upload remote gdrive_media_vfs 12.02.2020 20:54:24 INFO: *** Using rclone move - will add --delete-empty-src-dirs to upload. Error: unknown flag: --order-by Usage: rclone move source:path dest:path [flags] Flags: --create-empty-src-dirs Create empty source dirs on destination after move --delete-empty-src-dirs Delete empty source dirs after move -h, --help help for move Use "rclone [command] --help" for more information about a command. Use "rclone help flags" for to see the global flags. Use "rclone help backends" for a list of supported services. 2020/02/12 20:54:24 Fatal error: unknown flag: --order-by 12.02.2020 20:54:24 INFO: Not utilising service accounts. 12.02.2020 20:54:24 INFO: Script complete Script Finished Wed, 12 Feb 2020 20:54:24 +1100 Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_upload/log.txt I'm sure its something i have over looked. any help would be great. here is my changes for the upload script Quote # REQUIRED SETTINGS RcloneCommand="move" # choose your rclone command e.g. move, copy, sync RcloneRemoteName="gdrive_media_vfs" # Name of rclone remote mount WITHOUT ':'. RcloneUploadRemoteName="gdrive_media_vfs" # If you have a second remote created for uploads put it here. Otherwise use the same remote as RcloneRemoteName. LocalFilesShare="/mnt/user/local" # location of the local files without trailing slash you want to rclone to use RcloneMountShare="/mnt/user/mount_rclone" # where your rclone mount is located without trailing slash e.g. /mnt/user/mount_rclone MinimumAge="10m" # sync files suffix ms|s|m|h|d|w|M|y Quote Link to comment
Thel1988 Posted February 12, 2020 Share Posted February 12, 2020 Have you checked which version of rclone you have installed? You need at least 1.51 to support the order by function. Quote Link to comment
DZMM Posted February 12, 2020 Author Share Posted February 12, 2020 37 minutes ago, Nomad007 said: Hi just updated my scripts the mounting seems to work OK. But the upload errors out. I have just changed the remotename and remoteuploadname to the same remote just like the original scripts. If all the custom exclusions weren't filled in and used, it errored out. I've just fixed this so they can be empty and now you can enter custom commands, rather than just exclusions - see v0.95.1 https://github.com/BinsonBuzz/unraid_rclone_mount/blob/latest---mergerfs-support/rclone_upload # Add extra commands or filters Command1="--exclude downloads/**" Command2="" Command3="" Command4="" Command5="" Command6="" Command7="" Command8="" # process files rclone $RcloneCommand $LocalFilesLocation $RcloneUploadRemoteName:$BackupRemoteLocation $BackupDir \ --user-agent="$RcloneUploadRemoteName" \ -vv \ --buffer-size 512M \ --drive-chunk-size 512M \ --tpslimit 8 \ --checkers 8 \ --transfers 4 \ --order-by modtime,ascending \ --min-age $MinimumAge \ $Command1 $Command2 $Command3 $Command4 $Command5 $Command6 $Command7 $Command8 \ --exclude *fuse_hidden* \ --exclude *_HIDDEN \ --exclude .recycle** \ --exclude .Recycle.Bin/** \ --exclude *.backup~* \ --exclude *.partial~* \ --drive-stop-on-upload-limit \ --bwlimit "${BWLimit1Time},${BWLimit1} ${BWLimit2Time},${BWLimit2} ${BWLimit3Time},${BWLimit3}" \ --bind=$RCloneMountIP $DeleteEmpty 1 Quote Link to comment
Nomad007 Posted February 12, 2020 Share Posted February 12, 2020 🙏 Thank you it was my silliness. I thought Rclone was up to date. It was not as soon as i updated it, the upload script worked. Thanks for your help guys Quote Link to comment
Spladge Posted February 12, 2020 Share Posted February 12, 2020 Note that I have not checked yet, but is there an option /flag / fork for strictly unencypted remotes? If not I might convert once you have settled everything down and perhaps it could sit in another branch? Quote Link to comment
DZMM Posted February 12, 2020 Author Share Posted February 12, 2020 1 hour ago, Spladge said: Note that I have not checked yet, but is there an option /flag / fork for strictly unencypted remotes? If not I might convert once you have settled everything down and perhaps it could sit in another branch? You can use any remote with the scripts 1 1 Quote Link to comment
DZMM Posted February 13, 2020 Author Share Posted February 13, 2020 On 2/11/2020 at 8:25 PM, Spladge said: You just set up mergerfs - you don't really have to do anything @Spladge Thanks. I wish I'd thought about adding more folders to my main mergerfs mount, before I created a new one just last month when my music folders & files pushed me over the 400K limit - I spent a fair while having to move, rescan and fix tag errors 😞 Anyway, planning for the future as I will probably need to create another remote at some point and because I know you're not the only person with more than two folders in their unions, I made some updates last night to the mount script https://github.com/BinsonBuzz/unraid_rclone_mount/blob/latest---mergerfs-support/rclone_mount Quote 0.96.0 Changes: - New: Added option to not create mergerfs mount (e.g. if just creating rclone mount to add to another mergerfs mount) - New: Added option to add extra folders to mergerfs mount - Fix: Only create mountcheck file if not already mounted - Fix: Simplified bindmount Scripts should be finished now, except for any bug fixes as the ability to add custom commands and extra mergerfs folders should cover any future ideas. On 2/11/2020 at 8:25 PM, Spladge said: I use a bunch of different sonarrs and radarrs to manage my splits. Year based is the easiest because you can allow for that in the series level folder. And movie too. What's your folder structure? At the moment for TV I've got the following folders added to Plex: /mount_mergerfs/tv/kids/tv show (year) /mount_mergerfs/tv/adults/tv show (year) /mount_mergerfs/uhd/tv/kids/tv show (year) /mount_mergerfs/uhd/tv/adults/tv show (year) With 2x Sonarr dockers - one for non-UHD and one for UHD. If I add another sonarr instance in a new remote (assuming just HD), I can't see a way around having to add two more folders to plex: /mount_mergerfs/new_sonarr/tv/kids /mount_mergerfs/new_sonarr/tv/adults Or, are you saying you do this with your dockers: Docker1: /mount_mergerfs/tv/2000/tv show (2001)/ Docker2: /mount_mergerfs/tv/2010/tv show (2012)/ Docker3: /mount_mergerfs/tv/2020/tv show (2020)/ and Plex is ok if you only add /mount_mergerfs/tv to the library, and it can work out the rest? Quote Link to comment
testdasi Posted February 13, 2020 Share Posted February 13, 2020 (edited) I had a close look at the new scripts (mainly for the --drive-stop-on-upload-limit option) on Github and I have to say it makes the old ones look like Neanderthal stone tools. 😅 Am I right to say these are the new features compared to the old scripts? CreateBindMount - so the rclone download / upload goes through a different IP address from the server IP? mergerfs instead of unionfs - for hardlinks + no COW Service Accounts - automatic switching of upload accounts Backup / Upload switch - rclone sync vs rclone move Variable-ize various things instead of hard-coding BW Limit by time - looks like it's a rclone functionality and there's no logic in the script, unlike (1) -> (4)? Just want to double check before I selectively adapt some to my current (old) scripts. A small proposal: why don't you also variable-ize the script location (/mnt/user/appdata/other/rclone/remotes)? Seems a bit out of place for all the other paths to be variables and this one is hard-coded. Edited February 13, 2020 by testdasi Quote Link to comment
DZMM Posted February 13, 2020 Author Share Posted February 13, 2020 (edited) 1 hour ago, testdasi said: I had a close look at the new scripts (mainly for the --drive-stop-on-upload-limit option) on Github and I have to say it makes the old ones look like Neanderthal stone tools. 😅 I agree! Although I'm sure someone who knows how to code will wonder why on earth I'm doing certain things, I've got no real script or coding experience and I feel like I've progressed a lot in the last two weeks. 1 hour ago, testdasi said: Am I right to say these are the new features compared to the old scripts? CreateBindMount - so the rclone download / upload goes through a different IP address from the server IP? mergerfs instead of unionfs - for hardlinks + no COW Service Accounts - automatic switching of upload accounts Backup / Upload switch - rclone sync vs rclone move Variable-ize various things instead of hard-coding BW Limit by time - looks like it's a rclone functionality and there's no logic in the script, unlike (1) -> (4)? Yes, I've basically added in things that I use that I hadn't included before as I was worried they'd make support harder e.g. bindmounts, bwlimits, multiple remotes etc as well as making it easier to setup with fewer potential issues = less support questions + encourage more people to try. It's also setup so migration from unionfs or any other setup should only take a few mins now. Rotating Service accounts is a big find for me - I think you were using my mount rotation version which although worked, looks a bit dumb now. One feature you've missed is the latest one allowing extra folders to be added easily to mergerfs - again, I think this will reduce the number of issues on this thread. Re bindmounts - yes. I give each remote and upload_remote a unique IP to stop all the traffic appearing on my unRAID IP in pfsense. This way I can traffic shape to give my streaming mounts top priority, and various priority levels for the other remotes. Backup /upload switch - again added selfishly so the script can support my backup job. Bwlimit - there's no logic as it's hardcoded into the rclone move job. It should still work if anyone (wanted to) deleted the config entries - I haven't tried to see what happens, but I doubt anyone would do that. 1 hour ago, testdasi said: A small proposal: why don't you also variable-ize the script location (/mnt/user/appdata/other/rclone/remotes)? Seems a bit out of place for all the other paths to be variables and this one is hard-coded. I went hard-coded as it makes the cleanup script a lot easier to run via user scripts as the files to be deleted are always in the same place. If it was variable, users would have to enter the same location (or all the locations if they are running multiple scripts for multiple remotes) they've chosen in the cleanup script. Edited February 13, 2020 by DZMM Quote Link to comment
bar1 Posted February 13, 2020 Share Posted February 13, 2020 (edited) stupid question, but if i want to use this script i need modify : /boot/config/plugins/rclone/scripts/rclone_mount_plugin/script ? seems like i can't modify it via the gui (toolong?) EDIT: busy installing:User Scripts Edited February 13, 2020 by bar1 Quote Link to comment
DZMM Posted February 14, 2020 Author Share Posted February 14, 2020 Thanks @Spladge I was hoping there was another way. Not a problem though Quote Link to comment
Spladge Posted February 14, 2020 Share Posted February 14, 2020 I have not tried but it would be easy enough to set up a test library and just put a few things in it. Quote Link to comment
bar1 Posted February 14, 2020 Share Posted February 14, 2020 (edited) ok i followed the instructions properly this time (i think...). but the drive doesn't map. i am getting this: /tmp/user.scripts/tmpScripts/Rclone/script: line 206: syntax error: unexpected end of file Edited February 14, 2020 by bar1 Quote Link to comment
takeover Posted February 14, 2020 Share Posted February 14, 2020 So I got the idea from someone here to use a VPS... and I managed to sucessfully set this up and it is working great! Using the scripts to upload to my teamdrive, freeing up my bandwidth! So the VPS I use is a 1core, 1g ram, 50gb storage, 1g connection, 15tb/month for $35 a year. The setup is to just install docker and try to keep it minimal with letsencrypt, nzbget, and sonarr. Pros scripts work great with a few tweaks to get best performance free up your bandwidth or help upload even more Cons basic knowlege with linux, terminal, docker your providers restrcitions storage space for initial download (can't download anything bigger than your drive supports) extra cost to use this setup Quote Link to comment
DZMM Posted February 14, 2020 Author Share Posted February 14, 2020 44 minutes ago, bar1 said: ok i followed the instructions properly this time (i think...). but the drive doesn't map. i am getting this: /tmp/user.scripts/tmpScripts/Rclone/script: line 206: syntax error: unexpected end of file Posting the script you used would help! Quote Link to comment
bar1 Posted February 14, 2020 Share Posted February 14, 2020 can i make the script mount 2 drives? (like i tried to do). Quote Link to comment
DZMM Posted February 14, 2020 Author Share Posted February 14, 2020 2 hours ago, bar1 said: #!/bin/bash ###################### #### Mount Script #### ###################### ### Version 0.96.0 ### ###################### ####### EDIT ONLY THESE SETTINGS ####### # INSTRUCTIONS # 1. Edit the change the name of the rclone remote and shares to match your setup # 2. NOTE: enter RcloneRemoteName WITHOUT ':' # 3. Optional: include custom command and bind mount settings # 4. Optional: include extra folders in mergerfs mount # REQUIRED SETTINGS RcloneRemoteName="Bar1-xxx2" # Name of rclone remote mount WITHOUT ':'. NOTE: Choose your encrypted remote for sensitive data #RcloneRemoteName="Bar1-xxx" # Name of rclone remote mount WITHOUT ':'. NOTE: Choose your encrypted remote for sensitive data LocalFilesShare="/mnt/user/Media" # location of the local files you want to upload without trailing slash to rclone e.g. /mnt/user/local RcloneMountShare="/mnt/user/Bar1-xxx" # where your rclone remote will be located without trailing slash e.g. /mnt/user/mount_rclone MergerfsMountShare="ignore" # location without trailing slash e.g. /mnt/user/mount_mergerfs of the share you will use for Plex, Sonarr etc that combines files on your server's 'LocalFilesShare' with files on gdrive 'RcloneMountShare' DockerStart="Bar1-xxx" # list of dockers, separated by space, to start once mergerfs mount verified. Remember to disable AUTOSTART in docker settings page MountFolders=\{"downloads/complete,movies,tv,comics"\} # comma separated list of folders to create within the mount # Note: Again - remember to NOT use ':' in your remote name above # OPTIONAL SETTINGS # Add extra paths to mergerfs mount in addition to LocalFilesShare LocalFilesShare2="ignore" # without trailing slash e.g. //mnt/user/Bar1-xxx. Enter 'ignore' to disable LocalFilesShare3="ignore" LocalFilesShare4="ignore" # Add extra commands or filters Command1="--rc" Command2="" Command3="" Command4="" Command5="" Command6="" Command7="" Command8="" CreateBindMount="N" # Y/N. Choose whether to bind traffic to a particular network adapter RCloneMountIP="192.168.1.201" # My unraid IP is 172.30.12.2 so I create another similar IP address NetworkAdapter="eth0" # choose your network adapter. eth0 recommended VirtualIPNumber="2" # creates eth0:x e.g. eth0:1. I create a unique virtual IP addresses for each mount & upload so I can monitor and traffic shape for each of them ####### END SETTINGS ####### ############################################################################### ######## DO NOT EDIT ANYTHING UNLESS YOU KNOW WHAT YOU ARE DOING ########## ############################################################################### ####### Preparing mount location variables ####### LocalFilesLocation="$LocalFilesShare/$RcloneRemoteName" # Location for local files to be merged with rclone mount RcloneMountLocation="$RcloneMountShare/$RcloneRemoteName" # Location for rclone mount MergerFSMountLocation="$MergerfsMountShare/$RcloneRemoteName" # Rclone data folder location ####### create directories for rclone mount and mergerfs mounts ####### mkdir -p /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName # for script files eval mkdir -p $LocalFilesLocation/"$MountFolders" mkdir -p $RcloneMountLocation mkdir -p $MergerFSMountLocation ####### Check if script is already running ####### echo "$(date "+%d.%m.%Y %T") INFO: *** Starting mount of remote ${RcloneRemoteName}" echo "$(date "+%d.%m.%Y %T") INFO: Checking if this script is already running." if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Exiting script as already running." exit else echo "$(date "+%d.%m.%Y %T") INFO: Script not running - proceeding." touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running fi ####### Check If Rclone Mount Already Created ####### if [[ -f "$RcloneMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Success ${RcloneRemoteName} remote is already mounted." else echo "$(date "+%d.%m.%Y %T") INFO: Mount not running. Will now mount ${RcloneRemoteName} remote." ####### Create RClone Mount ####### # Check If Rclone Mount Already Created if [[ -f "$RcloneMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Success ${RcloneRemoteName} remote is already mounted." else echo "$(date "+%d.%m.%Y %T") INFO: Mount not running. Will now mount ${RcloneRemoteName} remote." # Creating mountcheck file in case it doesn't already exist echo "$(date "+%d.%m.%Y %T") INFO: Recreating mountcheck file for ${RcloneRemoteName} remote." touch mountcheck rclone copy mountcheck $RcloneRemoteName: -vv --no-traverse echo "$(date "+%d.%m.%Y %T") INFO: Completed creation of mountcheck file for ${RcloneRemoteName} remote." # Check bind option if [[ $CreateBindMount == 'Y' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}" ping -q -c2 $RCloneMountIP > /dev/null # -q quiet, -c number of pings to perform if [ $? -eq 0 ]; then # ping returns exit status 0 if successful echo "$(date "+%d.%m.%Y %T") INFO: *** IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}" else echo "$(date "+%d.%m.%Y %T") INFO: *** Creating IP address ${RCloneMountIP} for remote ${RcloneRemoteName}" ip addr add $RCloneMountIP/24 dev $NetworkAdapter label $NetworkAdapter:$VirtualIPNumber fi else RCloneMountIP="" fi # create rclone mount echo "$(date "+%d.%m.%Y %T") INFO: *** Creating bind mount ${RCloneMountIP} for remote ${RcloneRemoteName}" rclone mount \ --allow-other \ --buffer-size 256M \ --dir-cache-time 720h \ --drive-chunk-size 512M \ --log-level INFO \ --vfs-read-chunk-size 128M \ --vfs-read-chunk-size-limit off \ --vfs-cache-mode writes \ --bind=$RCloneMountIP \ $RcloneRemoteName: $RcloneMountLocation & # Check if Mount Successful echo "$(date "+%d.%m.%Y %T") INFO: sleeping for 5 seconds" # slight pause to give mount time to finalise sleep 5 echo "$(date "+%d.%m.%Y %T") INFO: continuing..." if [[ -f "$RcloneMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Successful mount of ${RcloneRemoteName} mount." else echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mount failed - please check for problems." rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running exit fi fi ####### Start MergerFS Mount ####### if [[ $MergerfsMountShare == 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Not creating mergerfs mount as requested." else if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount in place." else # check if mergerfs already installed if [[ -f "/bin/mergerfs" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs already installed, proceeding to create mergerfs mount" else # Build mergerfs binary echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs not installed - installing now." mkdir -p /mnt/user/appdata/other/rclone/mergerfs docker run -v /mnt/user/appdata/other/rclone/mergerfs:/build --rm trapexit/mergerfs-static-build mv /mnt/user/appdata/other/rclone/mergerfs/mergerfs /bin fi # Create mergerfs mount echo "$(date "+%d.%m.%Y %T") INFO: Creating ${RcloneRemoteName} mergerfs mount." # Extra Mergerfs folders if [[ $LocalFilesShare2 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare2} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare2=":$LocalFilesShare2" else LocalFilesShare2="" fi if [[ $LocalFilesShare3 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare3} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare3=":$LocalFilesShare3" else LocalFilesShare3="" fi if [[ $LocalFilesShare4 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare4} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare4=":$LocalFilesShare4" else LocalFilesShare4="" fi # mergerfs mount command mergerfs $LocalFilesLocation:$RcloneMountLocation$LocalFilesShare2$LocalFilesShare3$LocalFilesShare4 $MergerFSMountLocation -o rw,async_read=false,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=partial,dropcacheonclose=true # check if mergerfs mount successful echo "$(date "+%d.%m.%Y %T") INFO: Checking if ${RcloneRemoteName} mergerfs mount created." if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount created." else echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mergerfs mount failed." rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running exit fi fi fi ####### Starting Dockers That Need Mergerfs Mount To Work Properly ####### # only start dockers once if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: dockers already started." else touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started echo "$(date "+%d.%m.%Y %T") INFO: Starting dockers." docker start $DockerStart fi rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running echo "$(date "+%d.%m.%Y %T") INFO: Script complete" exit can you post the full logs as can see where it's failing from when you run the script and your rclone config (remote passwords) please - can you use the code formatting as well when you post please. Quote Link to comment
DZMM Posted February 14, 2020 Author Share Posted February 14, 2020 3 hours ago, senpaibox said: So I got the idea from someone here to use a VPS... and I managed to sucessfully set this up and it is working great! Using the scripts to upload to my teamdrive, freeing up my bandwidth! So the VPS I use is a 1core, 1g ram, 50gb storage, 1g connection, 15tb/month for $35 a year. The setup is to just install docker and try to keep it minimal with letsencrypt, nzbget, and sonarr. Pros scripts work great with a few tweaks to get best performance free up your bandwidth or help upload even more Cons basic knowlege with linux, terminal, docker your providers restrcitions storage space for initial download (can't download anything bigger than your drive supports) extra cost to use this setup Nice way to get a 1G connection. Which provider did you go with? Quote Link to comment
takeover Posted February 14, 2020 Share Posted February 14, 2020 (edited) 42 minutes ago, DZMM said: Nice way to get a 1G connection. Which provider did you go with? Got the deal on black friday with ChicagoVPS, also I am using a Google Cloud VPS which is free for the first year with the $300 credit trial Edited February 14, 2020 by senpaibox Quote Link to comment
bar1 Posted February 14, 2020 Share Posted February 14, 2020 #!/bin/bash ###################### #### Mount Script #### ###################### ### Version 0.96.0 ### ###################### ####### EDIT ONLY THESE SETTINGS ####### # INSTRUCTIONS # 1. Edit the change the name of the rclone remote and shares to match your setup # 2. NOTE: enter RcloneRemoteName WITHOUT ':' # 3. Optional: include custom command and bind mount settings # 4. Optional: include extra folders in mergerfs mount # REQUIRED SETTINGS RcloneRemoteName="Bar1-xxx" # Name of rclone remote mount WITHOUT ':'. NOTE: Choose your encrypted remote for sensitive data #RcloneRemoteName="Bar1-xxx" # Name of rclone remote mount WITHOUT ':'. NOTE: Choose your encrypted remote for sensitive data LocalFilesShare="/mnt/user/Media" # location of the local files you want to upload without trailing slash to rclone e.g. /mnt/user/local RcloneMountShare="/mnt/user/Bar1-Rxxx" # where your rclone remote will be located without trailing slash e.g. /mnt/user/mount_rclone MergerfsMountShare="ignore" # location without trailing slash e.g. /mnt/user/mount_mergerfs of the share you will use for Plex, Sonarr etc that combines files on your server's 'LocalFilesShare' with files on gdrive 'RcloneMountShare' DockerStart="Bar1-xxx" # list of dockers, separated by space, to start once mergerfs mount verified. Remember to disable AUTOSTART in docker settings page MountFolders=\{"downloads/complete,downloads/intermediate,downloads/seeds,movies,tv"\} # comma separated list of folders to create within the mount # Note: Again - remember to NOT use ':' in your remote name above # OPTIONAL SETTINGS # Add extra paths to mergerfs mount in addition to LocalFilesShare LocalFilesShare2="ignore" # without trailing slash e.g. //mnt/user/. Enter 'ignore' to disable LocalFilesShare3="ignore" LocalFilesShare4="ignore" # Add extra commands or filters Command1="--rc" Command2="" Command3="" Command4="" Command5="" Command6="" Command7="" Command8="" CreateBindMount="N" # Y/N. Choose whether to bind traffic to a particular network adapter RCloneMountIP="192.168.1.201" # My unraid IP is 172.30.12.2 so I create another similar IP address NetworkAdapter="eth0" # choose your network adapter. eth0 recommended VirtualIPNumber="2" # creates eth0:x e.g. eth0:1. I create a unique virtual IP addresses for each mount & upload so I can monitor and traffic shape for each of them ####### END SETTINGS ####### ############################################################################### ######## DO NOT EDIT ANYTHING UNLESS YOU KNOW WHAT YOU ARE DOING ########## ############################################################################### ####### Preparing mount location variables ####### LocalFilesLocation="$LocalFilesShare/$RcloneRemoteName" # Location for local files to be merged with rclone mount RcloneMountLocation="$RcloneMountShare/$RcloneRemoteName" # Location for rclone mount MergerFSMountLocation="$MergerfsMountShare/$RcloneRemoteName" # Rclone data folder location ####### create directories for rclone mount and mergerfs mounts ####### mkdir -p /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName # for script files eval mkdir -p $LocalFilesLocation/"$MountFolders" mkdir -p $RcloneMountLocation mkdir -p $MergerFSMountLocation ####### Check if script is already running ####### echo "$(date "+%d.%m.%Y %T") INFO: *** Starting mount of remote ${RcloneRemoteName}" echo "$(date "+%d.%m.%Y %T") INFO: Checking if this script is already running." if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Exiting script as already running." exit else echo "$(date "+%d.%m.%Y %T") INFO: Script not running - proceeding." touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running fi ####### Check If Rclone Mount Already Created ####### if [[ -f "$RcloneMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Success ${RcloneRemoteName} remote is already mounted." else echo "$(date "+%d.%m.%Y %T") INFO: Mount not running. Will now mount ${RcloneRemoteName} remote." ####### Create RClone Mount ####### # Check If Rclone Mount Already Created if [[ -f "$RcloneMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Success ${RcloneRemoteName} remote is already mounted." else echo "$(date "+%d.%m.%Y %T") INFO: Mount not running. Will now mount ${RcloneRemoteName} remote." # Creating mountcheck file in case it doesn't already exist echo "$(date "+%d.%m.%Y %T") INFO: Recreating mountcheck file for ${RcloneRemoteName} remote." touch mountcheck rclone copy mountcheck $RcloneRemoteName: -vv --no-traverse echo "$(date "+%d.%m.%Y %T") INFO: Completed creation of mountcheck file for ${RcloneRemoteName} remote." # Check bind option if [[ $CreateBindMount == 'Y' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}" ping -q -c2 $RCloneMountIP > /dev/null # -q quiet, -c number of pings to perform if [ $? -eq 0 ]; then # ping returns exit status 0 if successful echo "$(date "+%d.%m.%Y %T") INFO: *** IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}" else echo "$(date "+%d.%m.%Y %T") INFO: *** Creating IP address ${RCloneMountIP} for remote ${RcloneRemoteName}" ip addr add $RCloneMountIP/24 dev $NetworkAdapter label $NetworkAdapter:$VirtualIPNumber fi else RCloneMountIP="" fi # create rclone mount echo "$(date "+%d.%m.%Y %T") INFO: *** Creating bind mount ${RCloneMountIP} for remote ${RcloneRemoteName}" rclone mount \ --allow-other \ --buffer-size 256M \ --dir-cache-time 720h \ --drive-chunk-size 512M \ --log-level INFO \ --vfs-read-chunk-size 128M \ --vfs-read-chunk-size-limit off \ --vfs-cache-mode writes \ --bind=$RCloneMountIP \ $RcloneRemoteName: $RcloneMountLocation & # Check if Mount Successful echo "$(date "+%d.%m.%Y %T") INFO: sleeping for 5 seconds" # slight pause to give mount time to finalise sleep 5 echo "$(date "+%d.%m.%Y %T") INFO: continuing..." if [[ -f "$RcloneMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Successful mount of ${RcloneRemoteName} mount." else echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mount failed - please check for problems." rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running exit fi fi ####### Start MergerFS Mount ####### if [[ $MergerfsMountShare == 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Not creating mergerfs mount as requested." else if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount in place." else # check if mergerfs already installed if [[ -f "/bin/mergerfs" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs already installed, proceeding to create mergerfs mount" else # Build mergerfs binary echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs not installed - installing now." mkdir -p /mnt/user/appdata/other/rclone/mergerfs docker run -v /mnt/user/appdata/other/rclone/mergerfs:/build --rm trapexit/mergerfs-static-build mv /mnt/user/appdata/other/rclone/mergerfs/mergerfs /bin fi # Create mergerfs mount echo "$(date "+%d.%m.%Y %T") INFO: Creating ${RcloneRemoteName} mergerfs mount." # Extra Mergerfs folders if [[ $LocalFilesShare2 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare2} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare2=":$LocalFilesShare2" else LocalFilesShare2="" fi if [[ $LocalFilesShare3 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare3} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare3=":$LocalFilesShare3" else LocalFilesShare3="" fi if [[ $LocalFilesShare4 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare4} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare4=":$LocalFilesShare4" else LocalFilesShare4="" fi # mergerfs mount command mergerfs $LocalFilesLocation:$RcloneMountLocation$LocalFilesShare2$LocalFilesShare3$LocalFilesShare4 $MergerFSMountLocation -o rw,async_read=false,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=partial,dropcacheonclose=true # check if mergerfs mount successful echo "$(date "+%d.%m.%Y %T") INFO: Checking if ${RcloneRemoteName} mergerfs mount created." if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount created." else echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mergerfs mount failed." rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running exit fi fi fi ####### Starting Dockers That Need Mergerfs Mount To Work Properly ####### # only start dockers once if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: dockers already started." else touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started echo "$(date "+%d.%m.%Y %T") INFO: Starting dockers." docker start $DockerStart fi rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running echo "$(date "+%d.%m.%Y %T") INFO: Script complete" exit 14.02.2020 13:44:20 INFO: *** Starting mount of remote Bar1-xxxx 14.02.2020 13:44:20 INFO: Checking if this script is already running. 14.02.2020 13:44:20 INFO: Script not running - proceeding. /tmp/user.scripts/tmpScripts/Rclone/script: line 206: syntax error: unexpected end of file Script Finished Fri, 14 Feb 2020 13:44:20 +0200 Full logs for this script are available at /tmp/user.scripts/tmpScripts/Rclone/log.txt Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.