DZMM Posted August 4, 2020 Author Share Posted August 4, 2020 On 8/3/2020 at 6:37 AM, Emilio5639 said: I think I followed the guide pretty well, and I am receiving the following error in the Upload Script: rclone not installed - will try again later. Here is portion of the script that I am using: #!/bin/bash ###################### ### Upload Script #### ###################### ### Version 0.95.5 ### ###################### ####### EDIT ONLY THESE SETTINGS ####### # INSTRUCTIONS # 1. Edit the settings below to match your setup # 2. NOTE: enter RcloneRemoteName WITHOUT ':' # 3. Optional: Add additional commands or filters # 4. Optional: Use bind mount settings for potential traffic shaping/monitoring # 5. Optional: Use service accounts in your upload remote # 6. Optional: Use backup directory for rclone sync jobs # REQUIRED SETTINGS RcloneCommand="move" # choose your rclone command e.g. move, copy, sync RcloneRemoteName="gdrive_media_vfs" # Name of rclone remote mount WITHOUT ':'. RcloneUploadRemoteName="gdrive_upload_vfs" # If you have a second remote created for uploads put it here. Otherwise use the same remote as RcloneRemoteName. LocalFilesShare="/mnt/user/Plex-Media" # location of the local files without trailing slash you want to rclone to use RcloneMountShare="/mnt/user/mount_Plex-Media" # where your rclone mount is located without trailing slash e.g. /mnt/user/mount_rclone MinimumAge="15m" # sync files suffix ms|s|m|h|d|w|M|y ModSort="ascending" # "ascending" oldest files first, "descending" newest files first # Note: Again - remember to NOT use ':' in your remote name above # Bandwidth limits: specify the desired bandwidth in kBytes/s, or use a suffix b|k|M|G. Or 'off' or '0' for unlimited. The script uses --drive-stop-on-upload-limit which stops the script if the 750GB/day limit is achieved, so you no longer have to slow 'trickle' your files all day if you don't want to e.g. could just do an unlimited job overnight. BWLimit1Time="01:00" BWLimit1="off" BWLimit2Time="08:00" BWLimit2="15M" BWLimit3Time="16:00" BWLimit3="12M" # OPTIONAL SETTINGS # Add name to upload job JobName="_daily_upload" # Adds custom string to end of checker file. Useful if you're running multiple jobs against the same remote. # Add extra commands or filters Command1="--exclude downloads/**" Command2="" Command3="" Command4="" Command5="" Command6="" Command7="" Command8="" # Bind the mount to an IP address CreateBindMount="N" # Y/N. Choose whether or not to bind traffic to a network adapter. RCloneMountIP="10.0.0.62" # Choose IP to bind upload to. NetworkAdapter="eth0" # choose your network adapter. eth0 recommended. VirtualIPNumber="1" # creates eth0:x e.g. eth0:1. # Use Service Accounts. Instructions: https://github.com/xyou365/AutoRclone UseServiceAccountUpload="N" # Y/N. Choose whether to use Service Accounts. ServiceAccountDirectory="/mnt/user/appdata/other/rclone/service_accounts" # Path to your Service Account's .json files. ServiceAccountFile="sa_gdrive_upload" # Enter characters before counter in your json files e.g. for sa_gdrive_upload1.json -->sa_gdrive_upload100.json, enter "sa_gdrive_upload". CountServiceAccounts="15" # Integer number of service accounts to use. # Is this a backup job BackupJob="N" # Y/N. Syncs or Copies files from LocalFilesLocation to BackupRemoteLocation, rather than moving from LocalFilesLocation/RcloneRemoteName BackupRemoteLocation="backup" # choose location on mount for deleted sync files BackupRemoteDeletedLocation="backup_deleted" # choose location on mount for deleted sync files BackupRetention="90d" # How long to keep deleted sync files suffix ms|s|m|h|d|w|M|y ####### END SETTINGS ####### This is my first time setting something like this up, any help would be appreciated. Have you installed the rclone plugin? Quote Link to comment
KeyBoardDabbler Posted August 5, 2020 Share Posted August 5, 2020 (edited) On 8/4/2020 at 11:05 PM, DZMM said: All looks good. The reason I do a tdrive-->tdrive transfer as an extra step, is because I only have one mergerfs mount that combines all my tdrives and local files. The upload script for the associated local folder uploads all the files to a single tdrive, so behind the scenes I move the files to their correct tdrive. You could run an upload script for each tdrive, but for my setup that be a pain to traffic manage and the tdrive-->tdrive transfers are mega fast as they are done server side. That is making more sense to me now. Although i am still slightly confused with the setup still. If you are moving all files to one tdrive, to then move into their correct tdrive, does that mean there is a point where plex would not see the file? Because you point a plex library to the MergerFSMountLocation, and lets say TV shows are in tdrive2. You initially upload to tdrive1. Also in my first Upload Script would i just change "RcloneUploadRemoteName" to my tdrive i am initially sending my files to before being moved server side. Edited August 6, 2020 by KeyBoardDabbler Quote Link to comment
DZMM Posted August 5, 2020 Author Share Posted August 5, 2020 (edited) 23 hours ago, KeyBoardDabbler said: Although i am still slightly confused with the setup still. If you are moving all files to one tdrive, to then move into their correct tdrive, does that mean there is a point where plex would not see the file? No, because rclone is moving the files - server side from tdrive_x-->tdrive_y, so it is aware of the change of location. Because both tdrive_x and tdrive_y are included in my expanded mergerfs mount, plex, sonarr etc aren't aware that the real location has moved - it's the same approach as for when files move from local to gdrive. 23 hours ago, KeyBoardDabbler said: Also in my first Upload Script would i just change "RcloneUploadRemoteName" to my tdrive i am initially sending my files to before being moved server side. Yes, and then use my extra script or similar to 'sort' the files server side into the right tdrives. Edited August 6, 2020 by DZMM 1 Quote Link to comment
DZMM Posted August 5, 2020 Author Share Posted August 5, 2020 (edited) 37 minutes ago, KeyBoardDabbler said: Because you point a plex library to the MergerFSMountLocation, and lets say TV shows are in tdrive2. You initially upload to tdrive1. For clarity, here's my flow: 1. Mount Tdrive_1, tdrive_2, tdrive_3 etc - do not create a mergerfs mount for tdrive_2 and upwards 2. Create a single mergerfs mount with local folder and the Tdrive_1 mount, tdrive_2 mount, tdrive_3 mount etc 3. Point plex, sonarr etc at the mergerfs mount 4. new files get added to the mergerfs mount's local folder 5. upload script moves ALL files from the local folder to tdrive_1 6. I looked at tdrive_1 and worked what the encrypted name was e.g. for TV_Shows adasfghertdfghthgdf 7. TV_shows should be in tdrive_2, so extra script runs overnight to move tdrive_1/crypt/adasfghertdfghthgdf to tdrive_2/crypt/adasfghertdfghthgdf 8. The move is done by rclone so it adjusts the tdrive_1 and tdrive_2 mounts in real-time, so in mergerfs mount files haven't 'moved' from /TV_Shows and everyone is happy Edited August 5, 2020 by DZMM 2 Quote Link to comment
bugster Posted August 5, 2020 Share Posted August 5, 2020 Hi all. The mount script is not starting the dockers. I see this on the log Quote mergerfs version: 2.30.0 FUSE library version: 2.9.7-mergerfs_2.30.0 using FUSE kernel interface version 7.31 'build/mergerfs' -> '/build/mergerfs' 05.08.2020 09:30:54 INFO: *sleeping for 5 seconds 05.08.2020 09:30:59 INFO: Mergerfs installed successfully, proceeding to create mergerfs mount. 05.08.2020 09:30:59 INFO: Creating google_vfs mergerfs mount. 05.08.2020 09:30:59 INFO: Checking if google_vfs mergerfs mount created. 05.08.2020 09:30:59 INFO: Check successful, google_vfs mergerfs mount created. 05.08.2020 09:30:59 INFO: dockers already started. 05.08.2020 09:30:59 INFO: Script complete Script Finished Aug 05, 2020 09:30.59 But the dockers are not started, Quote DockerStart="plex radarr sonarr-sma radarr-sma" # list of dockers, separated by space, to start once mergerfs mount verified. Remember to disable AUTOSTART for dockers added in docker settings page Quote Link to comment
DZMM Posted August 6, 2020 Author Share Posted August 6, 2020 18 hours ago, bugster said: Hi all. The mount script is not starting the dockers. I see this on the log But the dockers are not started, You need to run the unmount script or manually delete the /mnt/cache/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started file - somehow you've got the scripts running slightly out of sync Quote Link to comment
KeyBoardDabbler Posted August 6, 2020 Share Posted August 6, 2020 (edited) 23 hours ago, DZMM said: For clarity, here's my flow: 1. Mount Tdrive_1, tdrive_2, tdrive_3 etc - do not create a mergerfs mount for tdrive_2 and upwards 2. Create a single mergerfs mount with local folder and the Tdrive_1 mount, tdrive_2 mount, tdrive_3 mount etc 3. Point plex, sonarr etc at the mergerfs mount 4. new files get added to the mergerfs mount's local folder 5. upload script moves ALL files from the local folder to tdrive_1 6. I looked at tdrive_1 and worked what the encrypted name was e.g. for TV_Shows adasfghertdfghthgdf 7. TV_shows should be in tdrive_2, so extra script runs overnight to move tdrive_1/crypt/adasfghertdfghthgdf to tdrive_2/crypt/adasfghertdfghthgdf 8. The move is done by rclone so it adjusts the tdrive_1 and tdrive_2 mounts in real-time, so in mergerfs mount files haven't 'moved' from /TV_Shows and everyone is happy Thankyou for explaining the steps, i am going to take another attempt. First i need to get at least 1 mount working using the mergerFS way. Just attempting to mount my original gdrive. Everything seem ok but no files are showing in /mount_unionfs/gdrive_media_vfs. Revert back to my unionfs mount script and everything works as expected. Quote 06.08.2020 20:22:56 INFO: Creating local folders. 06.08.2020 20:22:56 INFO: *** Starting mount of remote gdrive_media_vfs 06.08.2020 20:22:56 INFO: Checking if this script is already running. 06.08.2020 20:22:56 INFO: Script not running - proceeding. 06.08.2020 20:22:56 INFO: *** Checking if online 06.08.2020 20:22:57 PASSED: *** Internet online 06.08.2020 20:22:57 INFO: Mount not running. Will now mount gdrive_media_vfs remote. 06.08.2020 20:22:57 INFO: Recreating mountcheck file for gdrive_media_vfs remote. 2020/08/06 20:22:57 DEBUG : rclone: Version "v1.52.2-250-g4f9a80e2-beta" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone-beta/.rclone.conf" "copy" "mountcheck" "gdrive_media_vfs:" "-vv" "--no-traverse"] 2020/08/06 20:22:57 DEBUG : Using config file from "/boot/config/plugins/rclone-beta/.rclone.conf" 2020/08/06 20:22:57 DEBUG : fs cache: adding new entry for parent of "mountcheck", "/usr/local/emhttp" 2020/08/06 20:22:59 DEBUG : mountcheck: Modification times differ by -8m31.432593065s: 2020-08-06 20:22:57.396593065 +1200 NZST, 2020-08-06 08:14:25.964 +0000 UTC 2020/08/06 20:23:01 INFO : mountcheck: Copied (replaced existing) 2020/08/06 20:23:01 INFO : Transferred: 32 / 32 Bytes, 100%, 17 Bytes/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 1.8s 2020/08/06 20:23:01 DEBUG : 6 go routines active 06.08.2020 20:23:01 INFO: *** Creating mount for remote gdrive_media_vfs 06.08.2020 20:23:01 INFO: sleeping for 5 seconds 2020/08/06 20:23:03 INFO : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0) 06.08.2020 20:23:06 INFO: continuing... 06.08.2020 20:23:07 INFO: Successful mount of gdrive_media_vfs mount. 06.08.2020 20:23:07 INFO: Mergerfs already installed, proceeding to create mergerfs mount 06.08.2020 20:23:07 INFO: Creating gdrive_media_vfs mergerfs mount. 06.08.2020 20:23:07 INFO: Checking if gdrive_media_vfs mergerfs mount created. 06.08.2020 20:23:07 INFO: Check successful, gdrive_media_vfs mergerfs mount created. 06.08.2020 20:23:07 INFO: Starting dockers. ombi 06.08.2020 20:23:07 INFO: Script complete Script Finished Aug 06, 2020 20:23.07 Full logs for this script are available at /tmp/user.scripts/tmpScripts/rClone_Mount/log.txt 2020/08/06 20:23:31 INFO : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0) Edited August 6, 2020 by KeyBoardDabbler Quote Link to comment
Emilio5639 Posted August 6, 2020 Share Posted August 6, 2020 On 8/4/2020 at 4:06 AM, DZMM said: Have you installed the rclone plugin? The one I downloaded was from the community apps and it was: rclone Waseh Backup, Cloud, Tools:Utilities, Plugins Quote Link to comment
DZMM Posted August 6, 2020 Author Share Posted August 6, 2020 7 hours ago, KeyBoardDabbler said: Thankyou for explaining the steps, i am going to take another attempt. First i need to get at least 1 mount working using the mergerFS way. Just attempting to mount my original gdrive. Everything seem ok but no files are showing in /mount_unionfs/gdrive_media_vfs. Revert back to my unionfs mount script and everything works as expected. all looks fine - it even says it's overwritten the existing mountcheck file which is a good sign. When you say 'no files' - what are you using to look? post your mount scripts for both unionfs and mergerfs as there's something different in there. Quote Link to comment
DZMM Posted August 6, 2020 Author Share Posted August 6, 2020 6 hours ago, Emilio5639 said: The one I downloaded was from the community apps and it was: rclone Waseh Backup, Cloud, Tools:Utilities, Plugins post your script logs Quote Link to comment
KeyBoardDabbler Posted August 6, 2020 Share Posted August 6, 2020 3 hours ago, DZMM said: When you say 'no files' - what are you using to look? post your mount scripts for both unionfs and mergerfs as there's something different in there. I am just using file explorer on windows and ubuntu and navigating to the network share like normal. Nothing shows when using krusader though either. Quote #!/bin/bash ####### Check if script is already running ########## if [[ -f "/mnt/user/appdata/other/rclone/rclone_mount_running" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Exiting script already running." exit else touch /mnt/user/appdata/other/rclone/rclone_mount_running fi ####### End Check if script already running ########## ####### Start rclone gdrive mount ########## # check if gdrive mount already created if [[ -f "/mnt/user/mount_rclone/google_vfs/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check rclone vfs already mounted." else echo "$(date "+%d.%m.%Y %T") INFO: mounting rclone vfs." # create directories for rclone mount and unionfs mount mkdir -p /mnt/user/appdata/other/rclone mkdir -p /mnt/user/mount_rclone/google_vfs mkdir -p /mnt/user/mount_unionfs/google_vfs mkdir -p /mnt/user/rclone_upload/google_vfs rclone mount --allow-other --buffer-size 256M --dir-cache-time 72h --drive-chunk-size 512M --fast-list --log-level INFO --vfs-read-chunk-size 128M --vfs-read-chunk-size-limit off gdrive_media_vfs: /mnt/user/mount_rclone/google_vfs & # check if mount successful # slight pause to give mount time to finalise sleep 5 if [[ -f "/mnt/user/mount_rclone/google_vfs/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check rclone gdrive vfs mount success." else echo "$(date "+%d.%m.%Y %T") CRITICAL: rclone gdrive vfs mount failed - please check for problems." rm /mnt/user/appdata/other/rclone/rclone_mount_running exit fi fi ####### End rclone gdrive mount ########## ####### Start unionfs mount ########## if [[ -f "/mnt/user/mount_unionfs/google_vfs/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check successful, unionfs already mounted." else unionfs -o cow,allow_other,direct_io,auto_cache,sync_read /mnt/user/rclone_upload/google_vfs=RW:/mnt/user/mount_rclone/google_vfs=RO /mnt/user/mount_unionfs/google_vfs if [[ -f "/mnt/user/mount_unionfs/google_vfs/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check successful, unionfs mounted." else echo "$(date "+%d.%m.%Y %T") CRITICAL: unionfs Remount failed." rm /mnt/user/appdata/other/rclone/rclone_mount_running exit fi fi ####### End Mount unionfs ########## ############### starting dockers that need unionfs mount ###################### # only start dockers once if [[ -f "/mnt/user/appdata/other/rclone/dockers_started" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: dockers already started" else touch /mnt/user/appdata/other/rclone/dockers_started echo "$(date "+%d.%m.%Y %T") INFO: Starting dockers." docker start plex docker start tautulli docker start radarr docker start sonarr fi ############### end dockers that need unionfs mount ###################### exit Quote #!/bin/bash ###################### #### Mount Script #### ###################### ### Version 0.96.7 ### ###################### ####### EDIT ONLY THESE SETTINGS ####### # INSTRUCTIONS # 1. Change the name of the rclone remote and shares to match your setup # 2. NOTE: enter RcloneRemoteName WITHOUT ':' # 3. Optional: include custom command and bind mount settings # 4. Optional: include extra folders in mergerfs mount # REQUIRED SETTINGS RcloneRemoteName="gdrive_media_vfs" # Name of rclone remote mount WITHOUT ':'. NOTE: Choose your encrypted remote for sensitive data RcloneMountShare="/mnt/user/mount_rclone" # where your rclone remote will be located without trailing slash e.g. /mnt/user/mount_rclone LocalFilesShare="ignore" # location of the local files and MountFolders you want to upload without trailing slash to rclone e.g. /mnt/user/local. Enter 'ignore' to disable MergerfsMountShare="/mnt/user/mount_unionfs" # location without trailing slash e.g. /mnt/user/mount_mergerfs. Enter 'ignore' to disable DockerStart="ombi" # list of dockers, separated by space, to start once mergerfs mount verified. Remember to disable AUTOSTART for dockers added in docker settings page MountFolders=\{"downloads/complete,downloads/intermediate,downloads/seeds,movies,tv"\} # comma separated list of folders to create within the mount # Note: Again - remember to NOT use ':' in your remote name above # OPTIONAL SETTINGS # Add extra paths to mergerfs mount in addition to LocalFilesShare LocalFilesShare2="ignore" # without trailing slash e.g. /mnt/user/other__remote_mount/or_other_local_folder. Enter 'ignore' to disable LocalFilesShare3="ignore" LocalFilesShare4="ignore" # Add extra commands or filters Command1="--rc" Command2="" Command3="" Command4="" Command5="" Command6="" Command7="" Command8="" CreateBindMount="N" # Y/N. Choose whether to bind traffic to a particular network adapter RCloneMountIP="192.168.1.252" # My unraid IP is 172.30.12.2 so I create another similar IP address NetworkAdapter="eth0" # choose your network adapter. eth0 recommended VirtualIPNumber="2" # creates eth0:x e.g. eth0:1. I create a unique virtual IP addresses for each mount & upload so I can monitor and traffic shape for each of them ####### END SETTINGS ####### ############################################################################### ##### DO NOT EDIT ANYTHING BELOW UNLESS YOU KNOW WHAT YOU ARE DOING ####### ############################################################################### ####### Preparing mount location variables ####### RcloneMountLocation="$RcloneMountShare/$RcloneRemoteName" # Location for rclone mount LocalFilesLocation="$LocalFilesShare/$RcloneRemoteName" # Location for local files to be merged with rclone mount MergerFSMountLocation="$MergerfsMountShare/$RcloneRemoteName" # Rclone data folder location ####### create directories for rclone mount and mergerfs mounts ####### mkdir -p /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName #for script files if [[ $LocalFileShare == 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Not creating local folders as requested." else echo "$(date "+%d.%m.%Y %T") INFO: Creating local folders." eval mkdir -p $LocalFilesLocation/"$MountFolders" fi mkdir -p $RcloneMountLocation mkdir -p $MergerFSMountLocation ####### Check if script is already running ####### echo "$(date "+%d.%m.%Y %T") INFO: *** Starting mount of remote ${RcloneRemoteName}" echo "$(date "+%d.%m.%Y %T") INFO: Checking if this script is already running." if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Exiting script as already running." exit else echo "$(date "+%d.%m.%Y %T") INFO: Script not running - proceeding." touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running fi ####### Checking have connectivity ####### echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if online" ping -q -c2 google.com > /dev/null # -q quiet, -c number of pings to perform if [ $? -eq 0 ]; then # ping returns exit status 0 if successful echo "$(date "+%d.%m.%Y %T") PASSED: *** Internet online" else echo "$(date "+%d.%m.%Y %T") FAIL: *** No connectivity. Will try again on next run" rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running exit fi ####### Create Rclone Mount ####### # Check If Rclone Mount Already Created if [[ -f "$RcloneMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Success ${RcloneRemoteName} remote is already mounted." else echo "$(date "+%d.%m.%Y %T") INFO: Mount not running. Will now mount ${RcloneRemoteName} remote." # Creating mountcheck file in case it doesn't already exist echo "$(date "+%d.%m.%Y %T") INFO: Recreating mountcheck file for ${RcloneRemoteName} remote." touch mountcheck rclone copy mountcheck $RcloneRemoteName: -vv --no-traverse # Check bind option if [[ $CreateBindMount == 'Y' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}" ping -q -c2 $RCloneMountIP > /dev/null # -q quiet, -c number of pings to perform if [ $? -eq 0 ]; then # ping returns exit status 0 if successful echo "$(date "+%d.%m.%Y %T") INFO: *** IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}" else echo "$(date "+%d.%m.%Y %T") INFO: *** Creating IP address ${RCloneMountIP} for remote ${RcloneRemoteName}" ip addr add $RCloneMountIP/24 dev $NetworkAdapter label $NetworkAdapter:$VirtualIPNumber fi echo "$(date "+%d.%m.%Y %T") INFO: *** Created bind mount ${RCloneMountIP} for remote ${RcloneRemoteName}" else RCloneMountIP="" echo "$(date "+%d.%m.%Y %T") INFO: *** Creating mount for remote ${RcloneRemoteName}" fi # create rclone mount rclone mount \ --allow-other \ --buffer-size 256M \ --dir-cache-time 720h \ --drive-chunk-size 512M \ --log-level INFO \ --vfs-read-chunk-size 128M \ --vfs-read-chunk-size-limit off \ --vfs-cache-mode writes \ --bind=$RCloneMountIP \ $RcloneRemoteName: $RcloneMountLocation & # Check if Mount Successful echo "$(date "+%d.%m.%Y %T") INFO: sleeping for 5 seconds" # slight pause to give mount time to finalise sleep 5 echo "$(date "+%d.%m.%Y %T") INFO: continuing..." if [[ -f "$RcloneMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Successful mount of ${RcloneRemoteName} mount." else echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mount failed - please check for problems. Stopping dockers" docker stop $DockerStart rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running exit fi fi ####### Start MergerFS Mount ####### if [[ $MergerfsMountShare == 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Not creating mergerfs mount as requested." else if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount in place." else # check if mergerfs already installed if [[ -f "/bin/mergerfs" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs already installed, proceeding to create mergerfs mount" else # Build mergerfs binary echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs not installed - installing now." mkdir -p /mnt/user/appdata/other/rclone/mergerfs docker run -v /mnt/user/appdata/other/rclone/mergerfs:/build --rm trapexit/mergerfs-static-build mv /mnt/user/appdata/other/rclone/mergerfs/mergerfs /bin # check if mergerfs install successful echo "$(date "+%d.%m.%Y %T") INFO: *sleeping for 5 seconds" sleep 5 if [[ -f "/bin/mergerfs" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs installed successfully, proceeding to create mergerfs mount." else echo "$(date "+%d.%m.%Y %T") ERROR: Mergerfs not installed successfully. Please check for errors. Exiting." rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running exit fi fi # Create mergerfs mount echo "$(date "+%d.%m.%Y %T") INFO: Creating ${RcloneRemoteName} mergerfs mount." # Extra Mergerfs folders if [[ $LocalFilesShare2 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare2} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare2=":$LocalFilesShare2" else LocalFilesShare2="" fi if [[ $LocalFilesShare3 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare3} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare3=":$LocalFilesShare3" else LocalFilesShare3="" fi if [[ $LocalFilesShare4 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare4} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare4=":$LocalFilesShare4" else LocalFilesShare4="" fi # make sure mergerfs mount point is empty mv $MergerFSMountLocation $LocalFilesLocation mkdir -p $MergerFSMountLocation # mergerfs mount command mergerfs $LocalFilesLocation:$RcloneMountLocation$LocalFilesShare2$LocalFilesShare3$LocalFilesShare4 $MergerFSMountLocation -o rw,async_read=false,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=partial,dropcacheonclose=true # check if mergerfs mount successful echo "$(date "+%d.%m.%Y %T") INFO: Checking if ${RcloneRemoteName} mergerfs mount created." if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount created." else echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mergerfs mount failed. Stopping dockers." docker stop $DockerStart rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running exit fi fi fi ####### Starting Dockers That Need Mergerfs Mount To Work Properly ####### # only start dockers once if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: dockers already started." else touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started echo "$(date "+%d.%m.%Y %T") INFO: Starting dockers." docker start $DockerStart fi rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running echo "$(date "+%d.%m.%Y %T") INFO: Script complete" exit Quote Link to comment
DZMM Posted August 6, 2020 Author Share Posted August 6, 2020 33 minutes ago, KeyBoardDabbler said: I am just using file explorer on windows and ubuntu and navigating to the network share like normal. Nothing shows when using krusader though either. LocalFilesShare="ignore" - looking at your unionfs script, shouldn't this be LocalFilesShare="/mnt/user/rclone_upload/google_vfs"?? Otherwise, looks fine and your files from gdrive_media_vfs: should be showing in the mergerfs mount. If still not working, post the logs 1 Quote Link to comment
Emilio5639 Posted August 6, 2020 Share Posted August 6, 2020 4 hours ago, DZMM said: post your script logs Here is my mount Folder log: Script location: /tmp/user.scripts/tmpScripts/rclone mount/script Note that closing this window will abort the execution of this script 06.08.2020 13:26:53 INFO: Creating local folders. 06.08.2020 13:26:53 INFO: *** Starting mount of remote gdrive_media_vfs 06.08.2020 13:26:53 INFO: Checking if this script is already running. 06.08.2020 13:26:53 INFO: Script not running - proceeding. 06.08.2020 13:26:53 INFO: *** Checking if online 06.08.2020 13:26:54 PASSED: *** Internet online 06.08.2020 13:26:54 INFO: Mount not running. Will now mount gdrive_media_vfs remote. 06.08.2020 13:26:54 INFO: Recreating mountcheck file for gdrive_media_vfs remote. 2020/08/06 13:26:54 DEBUG : rclone: Version "v1.52.2" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "mountcheck" "gdrive_media_vfs:" "-vv" "--no-traverse"] 2020/08/06 13:26:54 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf" 2020/08/06 13:26:54 DEBUG : fs cache: renaming cache item "mountcheck" to be canonical "/" 2020/08/06 13:26:55 DEBUG : mountcheck: Modification times differ by -52.019083302s: 2020-08-06 13:26:54.305083302 -0700 PDT, 2020-08-06 20:26:02.286 +0000 UTC 2020/08/06 13:26:56 INFO : mountcheck: Copied (replaced existing) 2020/08/06 13:26:56 INFO : Transferred: 32 / 32 Bytes, 100%, 25 Bytes/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 1.2s 2020/08/06 13:26:56 DEBUG : 7 go routines active 06.08.2020 13:26:56 INFO: *** Creating mount for remote gdrive_media_vfs 06.08.2020 13:26:56 INFO: sleeping for 5 seconds 06.08.2020 13:27:01 INFO: continuing... 06.08.2020 13:27:01 INFO: Successful mount of gdrive_media_vfs mount. 06.08.2020 13:27:01 INFO: Check successful, gdrive_media_vfs mergerfs mount in place. 06.08.2020 13:27:01 INFO: dockers already started. 06.08.2020 13:27:01 INFO: Script complete Upload Script: Script location: /tmp/user.scripts/tmpScripts/rclone upload/script Note that closing this window will abort the execution of this script 06.08.2020 13:28:05 INFO: *** Rclone move selected. Files will be moved from /mnt/user/Plex-Media/gdrive_media_vfs for gdrive_upload_vfs *** 06.08.2020 13:28:05 INFO: *** Starting rclone_upload script for gdrive_upload_vfs *** 06.08.2020 13:28:05 INFO: Script not running - proceeding. 06.08.2020 13:28:05 INFO: Checking if rclone installed successfully. 06.08.2020 13:28:05 INFO: rclone not installed - will try again later. Quote Link to comment
KeyBoardDabbler Posted August 7, 2020 Share Posted August 7, 2020 (edited) 8 hours ago, DZMM said: LocalFilesShare="ignore" - looking at your unionfs script, shouldn't this be LocalFilesShare="/mnt/user/rclone_upload/google_vfs"?? Otherwise, looks fine and your files from gdrive_media_vfs: should be showing in the mergerfs mount. If still not working, post the logs I have since tried with the correct path but still not showing any of the files in my drive. the mount script has created the subfolders but when i navigate into /gdrive_media_vfs it is empty. UPDATE after a reboot of my server everything is now normal and operating?? Edited August 7, 2020 by KeyBoardDabbler Quote Link to comment
DZMM Posted August 7, 2020 Author Share Posted August 7, 2020 2 hours ago, KeyBoardDabbler said: I have since tried with the correct path but still not showing any of the files in my drive. the mount script has created the subfolders but when i navigate into /gdrive_media_vfs it is empty. UPDATE after a reboot of my server everything is now normal and operating?? Dunno - I was going to say after reading your logs that everything was 'working', so try a reboot. Sometimes when people are repeatedly mounting to debug the system seems to go a bit loopy. Quote Link to comment
DZMM Posted August 7, 2020 Author Share Posted August 7, 2020 10 hours ago, Emilio5639 said: Here is my mount Folder log: Script location: /tmp/user.scripts/tmpScripts/rclone mount/script Note that closing this window will abort the execution of this script 06.08.2020 13:26:53 INFO: Creating local folders. 06.08.2020 13:26:53 INFO: *** Starting mount of remote gdrive_media_vfs 06.08.2020 13:26:53 INFO: Checking if this script is already running. 06.08.2020 13:26:53 INFO: Script not running - proceeding. 06.08.2020 13:26:53 INFO: *** Checking if online 06.08.2020 13:26:54 PASSED: *** Internet online 06.08.2020 13:26:54 INFO: Mount not running. Will now mount gdrive_media_vfs remote. 06.08.2020 13:26:54 INFO: Recreating mountcheck file for gdrive_media_vfs remote. 2020/08/06 13:26:54 DEBUG : rclone: Version "v1.52.2" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "mountcheck" "gdrive_media_vfs:" "-vv" "--no-traverse"] 2020/08/06 13:26:54 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf" 2020/08/06 13:26:54 DEBUG : fs cache: renaming cache item "mountcheck" to be canonical "/" 2020/08/06 13:26:55 DEBUG : mountcheck: Modification times differ by -52.019083302s: 2020-08-06 13:26:54.305083302 -0700 PDT, 2020-08-06 20:26:02.286 +0000 UTC 2020/08/06 13:26:56 INFO : mountcheck: Copied (replaced existing) 2020/08/06 13:26:56 INFO : Transferred: 32 / 32 Bytes, 100%, 25 Bytes/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 1.2s 2020/08/06 13:26:56 DEBUG : 7 go routines active 06.08.2020 13:26:56 INFO: *** Creating mount for remote gdrive_media_vfs 06.08.2020 13:26:56 INFO: sleeping for 5 seconds 06.08.2020 13:27:01 INFO: continuing... 06.08.2020 13:27:01 INFO: Successful mount of gdrive_media_vfs mount. 06.08.2020 13:27:01 INFO: Check successful, gdrive_media_vfs mergerfs mount in place. 06.08.2020 13:27:01 INFO: dockers already started. 06.08.2020 13:27:01 INFO: Script complete Upload Script: Script location: /tmp/user.scripts/tmpScripts/rclone upload/script Note that closing this window will abort the execution of this script 06.08.2020 13:28:05 INFO: *** Rclone move selected. Files will be moved from /mnt/user/Plex-Media/gdrive_media_vfs for gdrive_upload_vfs *** 06.08.2020 13:28:05 INFO: *** Starting rclone_upload script for gdrive_upload_vfs *** 06.08.2020 13:28:05 INFO: Script not running - proceeding. 06.08.2020 13:28:05 INFO: Checking if rclone installed successfully. 06.08.2020 13:28:05 INFO: rclone not installed - will try again later. Can you post your mount script (haven't seen yet) and your upload script. Somewhere there's a path difference, or it doesn't like the dash in Plex-Media and you need to use an underscore. Quote Link to comment
KeyBoardDabbler Posted August 8, 2020 Share Posted August 8, 2020 (edited) I am back again 🙃 can you please explain how have you have managed to get the following folder structure? Quote mount_rclone/tdrive_vfs/music mount_rclone/tdrive_vfs/web_shows mount_rclone/tdrive_vfs/tv_anime I get the following error whenever i try to mount more than one tdrive in the same $RcloneMountLocation. Once the script finishes only the first mounted tdrive shows. Quote 08.08.2020 15:56:27 INFO: Creating local folders. 08.08.2020 15:56:27 INFO: *** Starting mount of remote tdrive_music_vfs 08.08.2020 15:56:27 INFO: Checking if this script is already running. 08.08.2020 15:56:27 INFO: Script not running - proceeding. 08.08.2020 15:56:27 INFO: *** Checking if online 08.08.2020 15:56:28 PASSED: *** Internet online 08.08.2020 15:56:28 INFO: Mount not running. Will now mount tdrive_music_vfs remote. 08.08.2020 15:56:28 INFO: Recreating mountcheck file for tdrive_music_vfs remote. 2020/08/08 15:56:28 DEBUG : rclone: Version "v1.52.2-285-g4d7f9130-beta" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone-beta/.rclone.conf" "copy" "mountcheck" "tdrive_music_vfs:" "-vv" "--no-traverse"] 2020/08/08 15:56:28 DEBUG : Using config file from "/boot/config/plugins/rclone-beta/.rclone.conf" 2020/08/08 15:56:28 DEBUG : fs cache: adding new entry for parent of "mountcheck", "/usr/local/emhttp" 2020/08/08 15:56:31 DEBUG : mountcheck: Modification times differ by -3m7.938405814s: 2020-08-08 15:56:28.730405814 +1200 NZST, 2020-08-08 03:53:20.792 +0000 UTC 2020/08/08 15:56:32 INFO : mountcheck: Copied (replaced existing) 2020/08/08 15:56:32 INFO : Transferred: 32 / 32 Bytes, 100%, 17 Bytes/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 4.2s 2020/08/08 15:56:32 DEBUG : 10 go routines active 08.08.2020 15:56:32 INFO: *** Creating mount for remote tdrive_music_vfs 08.08.2020 15:56:33 INFO: sleeping for 5 seconds 2020/08/08 15:56:35 INFO : vfs cache: cleaned: objects 0 (was 0) in use 0, to upload 0, uploading 0, total size 0 (was 0) 08.08.2020 15:56:38 INFO: continuing... 08.08.2020 15:56:38 INFO: Successful mount of tdrive_music_vfs mount. 08.08.2020 15:56:38 INFO: *** Starting mount of remote tdrive_web_shows_vfs 08.08.2020 15:56:38 INFO: Checking if this script is already running. 08.08.2020 15:56:38 INFO: Script not running - proceeding. 08.08.2020 15:56:38 INFO: *** Checking if online 08.08.2020 15:56:39 PASSED: *** Internet online 08.08.2020 15:56:39 INFO: Mount not running. Will now mount tdrive_web_shows_vfs remote. 08.08.2020 15:56:39 INFO: Recreating mountcheck file for tdrive_web_shows_vfs remote. 2020/08/08 15:56:39 DEBUG : rclone: Version "v1.52.2-285-g4d7f9130-beta" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone-beta/.rclone.conf" "copy" "mountcheck" "tdrive_web_shows_vfs:" "-vv" "--no-traverse"] 2020/08/08 15:56:39 DEBUG : Using config file from "/boot/config/plugins/rclone-beta/.rclone.conf" 2020/08/08 15:56:39 DEBUG : fs cache: adding new entry for parent of "mountcheck", "/usr/local/emhttp" 2020/08/08 15:56:41 DEBUG : mountcheck: Modification times differ by -26m50.494549599s: 2020-08-08 15:56:39.383549599 +1200 NZST, 2020-08-08 03:29:48.889 +0000 UTC 2020/08/08 15:56:43 INFO : mountcheck: Copied (replaced existing) 2020/08/08 15:56:43 INFO : Transferred: 32 / 32 Bytes, 100%, 19 Bytes/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 3.9s 2020/08/08 15:56:43 DEBUG : 10 go routines active 08.08.2020 15:56:43 INFO: *** Creating mount for remote tdrive_web_shows_vfs 08.08.2020 15:56:43 INFO: sleeping for 5 seconds 2020/08/08 15:56:45 Fatal error: Directory is not empty: /mnt/user/mount_rclone/tdrive_vfs If you want to mount it anyway use: --allow-non-empty option 08.08.2020 15:56:48 INFO: continuing... 08.08.2020 15:56:48 INFO: Successful mount of tdrive_web_shows_vfs mount. 08.08.2020 15:56:48 INFO: Not creating mergerfs mount as requested. 08.08.2020 15:56:48 INFO: Starting dockers. ombi 08.08.2020 15:56:48 INFO: Script complete Script Finished Aug 08, 2020 15:56.48 I have no trouble mounting them like the below but it is not very clean to navigate Quote mount_rclone/tdrive_music_vfs/music (first folder in tdrive share) mount_rclone/tdrive_web_shows_vfs/web_shows (first folder in tdrive share) mount_rclone/tdrive_tv_anime/tv_anime (first folder in tdrive share) Edited August 8, 2020 by KeyBoardDabbler Quote Link to comment
DZMM Posted August 8, 2020 Author Share Posted August 8, 2020 1 hour ago, KeyBoardDabbler said: I am back again 🙃 can you please explain how have you have managed to get the following folder structure? I get the following error whenever i try to mount more than one tdrive in the same $RcloneMountLocation. Once the script finishes only the first mounted tdrive shows. I have no trouble mounting them like the below but it is not very clean to navigate I mount them all in separate folders - sorry if I put them all in the same folder in my write-up - that's a mistake. Re navigation, it shouldn't matter as you should never ever need to use or visit the rclone mounts unless you are troubleshooting - the mergerfs folder is the only one you use day-2-day. Sounds like you got it working then? Quote Link to comment
KeyBoardDabbler Posted August 8, 2020 Share Posted August 8, 2020 (edited) 6 hours ago, DZMM said: I mount them all in separate folders - sorry if I put them all in the same folder in my write-up - that's a mistake. Re navigation, it shouldn't matter as you should never ever need to use or visit the rclone mounts unless you are troubleshooting - the mergerfs folder is the only one you use day-2-day. Sounds like you got it working then? Ahhh that explains it, it did throw me off a little bit.. Now that i have everything setup the /mount_merger folder layout is perfectly fine. I have everything setup and ran through the process, almost there but... I am now struggling with the last part, server side moving. I have tried various folders to move but they all error. I am most certain i have identified the correct encrypted folder names. Quote 2020/08/09 00:03:42 DEBUG : rclone: Version "v1.52.3-290-g82975109-beta" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone-beta/.rclone.conf" "move" "gdrive_vfs:crypt/1ej1l**********************hcm1v7g9957vnidqij60" "tdrive_movies_documentary_vfs:crypt/f*******************9qpq7ab71c3fc860" "--user-agent=transfer" "-vv" "--buffer-size" "512M" "--drive-chunk-size" "512M" "--tpslimit" "8" "--checkers" "8" "--transfers" "4" "--order-by" "modtime,ascending" "--exclude" "*fuse_hidden*" "--exclude" "*_HIDDEN" "--exclude" ".recycle**" "--exclude" ".Recycle.Bin/**" "--exclude" "*.backup~*" "--exclude" "*.partial~*" "--drive-stop-on-upload-limit" "--delete-empty-src-dirs"] 2020/08/09 00:03:42 DEBUG : Using config file from "/boot/config/plugins/rclone-beta/.rclone.conf" 2020/08/09 00:03:42 INFO : Starting HTTP transaction limiter: max 8 transactions/s with burst 1 2020/08/09 00:03:49 ERROR : : error reading source directory: directory not found 2020/08/09 00:03:49 DEBUG : Encrypted drive 'tdrive_movies_documentary_vfs:crypt/f*******************9qpq7ab71c3fc860': Waiting for checks to finish 2020/08/09 00:03:49 DEBUG : Encrypted drive 'tdrive_movies_documentary_vfs:crypt/f*******************9qpq7ab71c3fc860': Waiting for transfers to finish 2020/08/09 00:03:49 INFO : There was nothing to transfer 2020/08/09 00:03:49 ERROR : Attempt 1/3 failed with 1 errors and: directory not found 2020/08/09 00:03:50 ERROR : : error reading source directory: directory not found 2020/08/09 00:03:50 DEBUG : Encrypted drive 'tdrive_movies_documentary_vfs:crypt/f*******************9qpq7ab71c3fc860': Waiting for checks to finish 2020/08/09 00:03:50 DEBUG : Encrypted drive 'tdrive_movies_documentary_vfs:crypt/f*******************9qpq7ab71c3fc860': Waiting for transfers to finish 2020/08/09 00:03:50 INFO : There was nothing to transfer 2020/08/09 00:03:50 ERROR : Attempt 2/3 failed with 1 errors and: directory not found 2020/08/09 00:03:50 ERROR : : error reading source directory: directory not found 2020/08/09 00:03:50 DEBUG : Encrypted drive 'tdrive_movies_documentary_vfs:crypt/f*******************9qpq7ab71c3fc860': Waiting for checks to finish 2020/08/09 00:03:50 DEBUG : Encrypted drive 'tdrive_movies_documentary_vfs:crypt/f*******************9qpq7ab71c3fc860': Waiting for transfers to finish 2020/08/09 00:03:50 INFO : There was nothing to transfer 2020/08/09 00:03:50 ERROR : Attempt 3/3 failed with 1 errors and: directory not found 2020/08/09 00:03:50 INFO : Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA - Errors: 1 (retrying may help) Elapsed time: 8.4s 2020/08/09 00:03:50 DEBUG : 14 go routines active 2020/08/09 00:03:50 Failed to move: directory not found Script Finished Aug 09, 2020 00:03.50 Full logs for this script are available at /tmp/user.scripts/tmpScripts/rClone_Move/log.txt Script Starting Aug 09, 2020 00:08.49 Full logs for this script are available at /tmp/user.scripts/tmpScripts/rClone_Move/log.txt 2020/08/09 00:08:49 DEBUG : rclone: Version "v1.52.3-290-g82975109-beta" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone-beta/.rclone.conf" "move" "tdrive_upload_vfs:crypt/**************************fk40rft2t0t7neqvb9hi7psvm8cg" "tdrive_movies_bio_vfs:crypt/*********************7neqvb9hi7psvm8cg" "--user-agent=transfer" "-vv" "--buffer-size" "512M" "--drive-chunk-size" "512M" "--tpslimit" "8" "--checkers" "8" "--transfers" "4" "--order-by" "modtime,ascending" "--exclude" "*fuse_hidden*" "--exclude" "*_HIDDEN" "--exclude" ".recycle**" "--exclude" ".Recycle.Bin/**" "--exclude" "*.backup~*" "--exclude" "*.partial~*" "--drive-stop-on-upload-limit" "--delete-empty-src-dirs"] 2020/08/09 00:08:49 DEBUG : Using config file from "/boot/config/plugins/rclone-beta/.rclone.conf" 2020/08/09 00:08:49 INFO : Starting HTTP transaction limiter: max 8 transactions/s with burst 1 2020/08/09 00:08:57 ERROR : : error reading source directory: directory not found 2020/08/09 00:08:57 DEBUG : Encrypted drive 'tdrive_movies_bio_vfs:crypt/*********************7neqvb9hi7psvm8cg': Waiting for checks to finish 2020/08/09 00:08:57 DEBUG : Encrypted drive 'tdrive_movies_bio_vfs:crypt/*********************7neqvb9hi7psvm8cg': Waiting for transfers to finish 2020/08/09 00:08:57 INFO : There was nothing to transfer 2020/08/09 00:08:57 ERROR : Attempt 1/3 failed with 1 errors and: directory not found 2020/08/09 00:08:58 ERROR : : error reading source directory: directory not found 2020/08/09 00:08:58 DEBUG : Encrypted drive 'tdrive_movies_bio_vfs:crypt/*********************7neqvb9hi7psvm8cg': Waiting for checks to finish 2020/08/09 00:08:58 DEBUG : Encrypted drive 'tdrive_movies_bio_vfs:crypt/*********************7neqvb9hi7psvm8cg': Waiting for transfers to finish 2020/08/09 00:08:58 INFO : There was nothing to transfer 2020/08/09 00:08:58 ERROR : Attempt 2/3 failed with 1 errors and: directory not found 2020/08/09 00:08:58 ERROR : : error reading source directory: directory not found 2020/08/09 00:08:58 DEBUG : Encrypted drive 'tdrive_bio_vfs:crypt/*********************7neqvb9hi7psvm8cg': Waiting for checks to finish 2020/08/09 00:08:58 DEBUG : Encrypted drive 'tdrive_bio_vfs:crypt/*********************7neqvb9hi7psvm8cg': Waiting for transfers to finish 2020/08/09 00:08:58 INFO : There was nothing to transfer 2020/08/09 00:08:58 ERROR : Attempt 3/3 failed with 1 errors and: directory not found 2020/08/09 00:08:58 INFO : Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA - Errors: 1 (retrying may help) Elapsed time: 9.7s 2020/08/09 00:08:58 DEBUG : 19 go routines active 2020/08/09 00:08:58 Failed to move: directory not found Script Finished Aug 09, 2020 00:08.58 Edited August 8, 2020 by KeyBoardDabbler Quote Link to comment
DZMM Posted August 8, 2020 Author Share Posted August 8, 2020 1 hour ago, KeyBoardDabbler said: I am now struggling with the last part, server side moving. I have tried various folders to move but they all error. I am most certain i have identified the correct encrypted folder names. I think it's because you are using the remotes that decrypt the files - you need to use the 'raw' remotes that process the encrypted files i.e. rclone move tdrive_upload:crypt/**************************fk40rft2t0t7neqvb9hi7psvm8cg" "tdrive_movies_bio:crypt/*********************7neqvb9hi7psvm8cg" because you are moving the files server-side i.e. encrypted. 1 Quote Link to comment
KeyBoardDabbler Posted August 8, 2020 Share Posted August 8, 2020 (edited) 15 hours ago, DZMM said: I think it's because you are using the remotes that decrypt the files - you need to use the 'raw' remotes that process the encrypted files i.e. rclone move tdrive_upload:crypt/**************************fk40rft2t0t7neqvb9hi7psvm8cg" "tdrive_movies_bio:crypt/*********************7neqvb9hi7psvm8cg" because you are moving the files server-side i.e. encrypted. I think you was correct, this change has got me to the finish line. Things are running really good. I have a lot to rearrange now. very grateful for the help and all the work you have put into this project. I do not have any issue moving tdrive > tdrive share but i am running into permission error when trying to move files out of gdrive > tdrive. sa group has been added to the members on all drives. Status has been set to "manage" and "enabled" migration in settings. Quote Couldn't move: googleapi: Error 403: The domain administrator has not allowed writers to move items into a shared drive. The main difference is gdrive uses the client_id, client_secret & token whereas tdrive uses service_account_file because it can not be used with the gdrive. Either way i have both users added as members ect. Edited August 9, 2020 by KeyBoardDabbler Quote Link to comment
DZMM Posted August 9, 2020 Author Share Posted August 9, 2020 10 hours ago, KeyBoardDabbler said: I do not have any issue moving tdrive > tdrive share but i am running into permission error when trying to move files out of gdrive > tdrive. sa group has been added to the members on all drives. Status has been set to "manage" and "enabled" migration in settings. I haven't come across this problem before - I think because I was moving my music mtdrive-->gdrive. 10 hours ago, KeyBoardDabbler said: very grateful for the help and all the work you have put into this project. You're welcome. I'm happy to share as I'm sure it's saving people $$$ in drives etc. Quote Link to comment
_sb Posted August 9, 2020 Share Posted August 9, 2020 Hi all, Thought i'd ask here as there appears to be quite a few advanced users... I've decided to swap over to encryption and am trying to work out how to copy from the my current un-encrypted drive to an encrypted drive. My understanding is I can only use server side copy if both source and destination are same (both encrypted or not). I think AutoRclone is what I need to setup in order to do what I want, but I am not entirely sure. I believe I need to download and then upload again, so for speed i'll fire up a cloud instance with Hetzner. Can anyone confirm or point me in the right direction please? Thanks. Quote Link to comment
Emilio5639 Posted August 11, 2020 Share Posted August 11, 2020 On 8/6/2020 at 11:29 PM, DZMM said: Can you post your mount script (haven't seen yet) and your upload script. Somewhere there's a path difference, or it doesn't like the dash in Plex-Media and you need to use an underscore. Here are all of the Scripts that I have set up. I went ahead and changed my folder to: Plex_Media RClone Mount Script: #!/bin/bash ###################### #### Mount Script #### ###################### ### Version 0.96.7 ### ###################### ####### EDIT ONLY THESE SETTINGS ####### # INSTRUCTIONS # 1. Change the name of the rclone remote and shares to match your setup # 2. NOTE: enter RcloneRemoteName WITHOUT ':' # 3. Optional: include custom command and bind mount settings # 4. Optional: include extra folders in mergerfs mount # REQUIRED SETTINGS RcloneRemoteName="gdrive_media_vfs" # Name of rclone remote mount WITHOUT ':'. NOTE: Choose your encrypted remote for sensitive data RcloneMountShare="/mnt/user/mount_Plex_Media" # where your rclone remote will be located without trailing slash e.g. /mnt/user/mount_rclone LocalFilesShare="/mnt/user/Plex_Media" # location of the local files and MountFolders you want to upload without trailing slash to rclone e.g. /mnt/user/local. Enter 'ignore' to disable MergerfsMountShare="/mnt/user/mount_mergerfs" # location without trailing slash e.g. /mnt/user/mount_mergerfs. Enter 'ignore' to disable DockerStart="plex binhex-sonarr radarr" # list of dockers, separated by space, to start once mergerfs mount verified. Remember to disable AUTOSTART for dockers added in docker settings page MountFolders=\{"downloads/complete,downloads/intermediate,downloads/seeds,movies,tv"\} # comma separated list of folders to create within the mount # Note: Again - remember to NOT use ':' in your remote name above # OPTIONAL SETTINGS # Add extra paths to mergerfs mount in addition to LocalFilesShare LocalFilesShare2="ignore" # without trailing slash e.g. /mnt/user/other__remote_mount/or_other_local_folder. Enter 'ignore' to disable LocalFilesShare3="ignore" LocalFilesShare4="ignore" # Add extra commands or filters Command1="--rc" Command2="" Command3="" Command4="" Command5="" Command6="" Command7="" Command8="" CreateBindMount="N" # Y/N. Choose whether to bind traffic to a particular network adapter RCloneMountIP="10.0.0.162" # My unraid IP is 172.30.12.2 so I create another similar IP address NetworkAdapter="eth0" # choose your network adapter. eth0 recommended VirtualIPNumber="2" # creates eth0:x e.g. eth0:1. I create a unique virtual IP addresses for each mount & upload so I can monitor and traffic shape for each of them ####### END SETTINGS ####### ############################################################################### ##### DO NOT EDIT ANYTHING BELOW UNLESS YOU KNOW WHAT YOU ARE DOING ####### ############################################################################### ####### Preparing mount location variables ####### RcloneMountLocation="$RcloneMountShare/$RcloneRemoteName" # Location for rclone mount LocalFilesLocation="$LocalFilesShare/$RcloneRemoteName" # Location for local files to be merged with rclone mount MergerFSMountLocation="$MergerfsMountShare/$RcloneRemoteName" # Rclone data folder location ####### create directories for rclone mount and mergerfs mounts ####### mkdir -p /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName #for script files if [[ $LocalFileShare == 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Not creating local folders as requested." else echo "$(date "+%d.%m.%Y %T") INFO: Creating local folders." eval mkdir -p $LocalFilesLocation/"$MountFolders" fi mkdir -p $RcloneMountLocation mkdir -p $MergerFSMountLocation ####### Check if script is already running ####### echo "$(date "+%d.%m.%Y %T") INFO: *** Starting mount of remote ${RcloneRemoteName}" echo "$(date "+%d.%m.%Y %T") INFO: Checking if this script is already running." if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Exiting script as already running." exit else echo "$(date "+%d.%m.%Y %T") INFO: Script not running - proceeding." touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running fi ####### Checking have connectivity ####### echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if online" ping -q -c2 google.com > /dev/null # -q quiet, -c number of pings to perform if [ $? -eq 0 ]; then # ping returns exit status 0 if successful echo "$(date "+%d.%m.%Y %T") PASSED: *** Internet online" else echo "$(date "+%d.%m.%Y %T") FAIL: *** No connectivity. Will try again on next run" rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running exit fi ####### Create Rclone Mount ####### # Check If Rclone Mount Already Created if [[ -f "$RcloneMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Success ${RcloneRemoteName} remote is already mounted." else echo "$(date "+%d.%m.%Y %T") INFO: Mount not running. Will now mount ${RcloneRemoteName} remote." # Creating mountcheck file in case it doesn't already exist echo "$(date "+%d.%m.%Y %T") INFO: Recreating mountcheck file for ${RcloneRemoteName} remote." touch mountcheck rclone copy mountcheck $RcloneRemoteName: -vv --no-traverse # Check bind option if [[ $CreateBindMount == 'Y' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}" ping -q -c2 $RCloneMountIP > /dev/null # -q quiet, -c number of pings to perform if [ $? -eq 0 ]; then # ping returns exit status 0 if successful echo "$(date "+%d.%m.%Y %T") INFO: *** IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}" else echo "$(date "+%d.%m.%Y %T") INFO: *** Creating IP address ${RCloneMountIP} for remote ${RcloneRemoteName}" ip addr add $RCloneMountIP/24 dev $NetworkAdapter label $NetworkAdapter:$VirtualIPNumber fi echo "$(date "+%d.%m.%Y %T") INFO: *** Created bind mount ${RCloneMountIP} for remote ${RcloneRemoteName}" else RCloneMountIP="" echo "$(date "+%d.%m.%Y %T") INFO: *** Creating mount for remote ${RcloneRemoteName}" fi # create rclone mount rclone mount \ --allow-other \ --buffer-size 256M \ --dir-cache-time 720h \ --drive-chunk-size 512M \ --log-level INFO \ --vfs-read-chunk-size 128M \ --vfs-read-chunk-size-limit off \ --vfs-cache-mode writes \ --bind=$RCloneMountIP \ $RcloneRemoteName: $RcloneMountLocation & # Check if Mount Successful echo "$(date "+%d.%m.%Y %T") INFO: sleeping for 5 seconds" # slight pause to give mount time to finalise sleep 5 echo "$(date "+%d.%m.%Y %T") INFO: continuing..." if [[ -f "$RcloneMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Successful mount of ${RcloneRemoteName} mount." else echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mount failed - please check for problems. Stopping dockers" docker stop $DockerStart rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running exit fi fi ####### Start MergerFS Mount ####### if [[ $MergerfsMountShare == 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Not creating mergerfs mount as requested." else if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount in place." else # check if mergerfs already installed if [[ -f "/bin/mergerfs" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs already installed, proceeding to create mergerfs mount" else # Build mergerfs binary echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs not installed - installing now." mkdir -p /mnt/user/appdata/other/rclone/mergerfs docker run -v /mnt/user/appdata/other/rclone/mergerfs:/build --rm trapexit/mergerfs-static-build mv /mnt/user/appdata/other/rclone/mergerfs/mergerfs /bin # check if mergerfs install successful echo "$(date "+%d.%m.%Y %T") INFO: *sleeping for 5 seconds" sleep 5 if [[ -f "/bin/mergerfs" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs installed successfully, proceeding to create mergerfs mount." else echo "$(date "+%d.%m.%Y %T") ERROR: Mergerfs not installed successfully. Please check for errors. Exiting." rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running exit fi fi # Create mergerfs mount echo "$(date "+%d.%m.%Y %T") INFO: Creating ${RcloneRemoteName} mergerfs mount." # Extra Mergerfs folders if [[ $LocalFilesShare2 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare2} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare2=":$LocalFilesShare2" else LocalFilesShare2="" fi if [[ $LocalFilesShare3 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare3} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare3=":$LocalFilesShare3" else LocalFilesShare3="" fi if [[ $LocalFilesShare4 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare4} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare4=":$LocalFilesShare4" else LocalFilesShare4="" fi # make sure mergerfs mount point is empty mv $MergerFSMountLocation $LocalFilesLocation mkdir -p $MergerFSMountLocation # mergerfs mount command mergerfs $LocalFilesLocation:$RcloneMountLocation$LocalFilesShare2$LocalFilesShare3$LocalFilesShare4 $MergerFSMountLocation -o rw,async_read=false,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=partial,dropcacheonclose=true # check if mergerfs mount successful echo "$(date "+%d.%m.%Y %T") INFO: Checking if ${RcloneRemoteName} mergerfs mount created." if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount created." else echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mergerfs mount failed. Stopping dockers." docker stop $DockerStart rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running exit fi fi fi ####### Starting Dockers That Need Mergerfs Mount To Work Properly ####### # only start dockers once if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: dockers already started." else touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started echo "$(date "+%d.%m.%Y %T") INFO: Starting dockers." docker start $DockerStart fi rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running echo "$(date "+%d.%m.%Y %T") INFO: Script complete" exit RClone Upload Script: #!/bin/bash ###################### ### Upload Script #### ###################### ### Version 0.95.5 ### ###################### ####### EDIT ONLY THESE SETTINGS ####### # INSTRUCTIONS # 1. Edit the settings below to match your setup # 2. NOTE: enter RcloneRemoteName WITHOUT ':' # 3. Optional: Add additional commands or filters # 4. Optional: Use bind mount settings for potential traffic shaping/monitoring # 5. Optional: Use service accounts in your upload remote # 6. Optional: Use backup directory for rclone sync jobs # REQUIRED SETTINGS RcloneCommand="move" # choose your rclone command e.g. move, copy, sync RcloneRemoteName="gdrive_media_vfs" # Name of rclone remote mount WITHOUT ':'. RcloneUploadRemoteName="gdrive_upload_vfs" # If you have a second remote created for uploads put it here. Otherwise use the same remote as RcloneRemoteName. LocalFilesShare="/mnt/user/Plex_Media" # location of the local files without trailing slash you want to rclone to use RcloneMountShare="/mnt/user/mount_Plex_Media" # where your rclone mount is located without trailing slash e.g. /mnt/user/mount_rclone MinimumAge="15m" # sync files suffix ms|s|m|h|d|w|M|y ModSort="ascending" # "ascending" oldest files first, "descending" newest files first # Note: Again - remember to NOT use ':' in your remote name above # Bandwidth limits: specify the desired bandwidth in kBytes/s, or use a suffix b|k|M|G. Or 'off' or '0' for unlimited. The script uses --drive-stop-on-upload-limit which stops the script if the 750GB/day limit is achieved, so you no longer have to slow 'trickle' your files all day if you don't want to e.g. could just do an unlimited job overnight. BWLimit1Time="01:00" BWLimit1="off" BWLimit2Time="08:00" BWLimit2="15M" BWLimit3Time="16:00" BWLimit3="12M" # OPTIONAL SETTINGS # Add name to upload job JobName="_daily_upload" # Adds custom string to end of checker file. Useful if you're running multiple jobs against the same remote. # Add extra commands or filters Command1="--exclude downloads/**" Command2="" Command3="" Command4="" Command5="" Command6="" Command7="" Command8="" # Bind the mount to an IP address CreateBindMount="N" # Y/N. Choose whether or not to bind traffic to a network adapter. RCloneMountIP="10.0.0.62" # Choose IP to bind upload to. NetworkAdapter="eth0" # choose your network adapter. eth0 recommended. VirtualIPNumber="1" # creates eth0:x e.g. eth0:1. # Use Service Accounts. Instructions: https://github.com/xyou365/AutoRclone UseServiceAccountUpload="N" # Y/N. Choose whether to use Service Accounts. ServiceAccountDirectory="/mnt/user/appdata/other/rclone/service_accounts" # Path to your Service Account's .json files. ServiceAccountFile="sa_gdrive_upload" # Enter characters before counter in your json files e.g. for sa_gdrive_upload1.json -->sa_gdrive_upload100.json, enter "sa_gdrive_upload". CountServiceAccounts="15" # Integer number of service accounts to use. # Is this a backup job BackupJob="N" # Y/N. Syncs or Copies files from LocalFilesLocation to BackupRemoteLocation, rather than moving from LocalFilesLocation/RcloneRemoteName BackupRemoteLocation="backup" # choose location on mount for deleted sync files BackupRemoteDeletedLocation="backup_deleted" # choose location on mount for deleted sync files BackupRetention="90d" # How long to keep deleted sync files suffix ms|s|m|h|d|w|M|y ####### END SETTINGS ####### ############################################################################### ##### DO NOT EDIT BELOW THIS LINE UNLESS YOU KNOW WHAT YOU ARE DOING ##### ############################################################################### ####### Preparing mount location variables ####### if [[ $BackupJob == 'Y' ]]; then LocalFilesLocation="$LocalFilesShare" echo "$(date "+%d.%m.%Y %T") INFO: *** Backup selected. Files will be copied or synced from ${LocalFilesLocation} for ${RcloneUploadRemoteName} ***" else LocalFilesLocation="$LocalFilesShare/$RcloneRemoteName" echo "$(date "+%d.%m.%Y %T") INFO: *** Rclone move selected. Files will be moved from ${LocalFilesLocation} for ${RcloneUploadRemoteName} ***" fi RcloneMountLocation="$RcloneMountShare/$RcloneRemoteName" # Location of rclone mount ####### create directory for script files ####### mkdir -p /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName #for script files ####### Check if script already running ########## echo "$(date "+%d.%m.%Y %T") INFO: *** Starting rclone_upload script for ${RcloneUploadRemoteName} ***" if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/upload_running$JobName" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Exiting as script already running." exit else echo "$(date "+%d.%m.%Y %T") INFO: Script not running - proceeding." touch /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/upload_running$JobName fi ####### check if rclone installed ########## echo "$(date "+%d.%m.%Y %T") INFO: Checking if rclone installed successfully." if [[ -f "$RcloneMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: rclone installed successfully - proceeding with upload." else echo "$(date "+%d.%m.%Y %T") INFO: rclone not installed - will try again later." rm /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/upload_running$JobName exit fi ####### Rotating serviceaccount.json file if using Service Accounts ####### if [[ $UseServiceAccountUpload == 'Y' ]]; then cd /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/ CounterNumber=$(find -name 'counter*' | cut -c 11,12) CounterCheck="1" if [[ "$CounterNumber" -ge "$CounterCheck" ]];then echo "$(date "+%d.%m.%Y %T") INFO: Counter file found for ${RcloneUploadRemoteName}." else echo "$(date "+%d.%m.%Y %T") INFO: No counter file found for ${RcloneUploadRemoteName}. Creating counter_1." touch /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/counter_1 CounterNumber="1" fi ServiceAccount="--drive-service-account-file=$ServiceAccountDirectory/$ServiceAccountFile$CounterNumber.json" echo "$(date "+%d.%m.%Y %T") INFO: Adjusted service_account_file for upload remote ${RcloneUploadRemoteName} to ${ServiceAccountFile}${CounterNumber}.json based on counter ${CounterNumber}." else echo "$(date "+%d.%m.%Y %T") INFO: Uploading using upload remote ${RcloneUploadRemoteName}" ServiceAccount="" fi ####### Upload files ########## # Check bind option if [[ $CreateBindMount == 'Y' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if IP address ${RCloneMountIP} already created for upload to remote ${RcloneUploadRemoteName}" ping -q -c2 $RCloneMountIP > /dev/null # -q quiet, -c number of pings to perform if [ $? -eq 0 ]; then # ping returns exit status 0 if successful echo "$(date "+%d.%m.%Y %T") INFO: *** IP address ${RCloneMountIP} already created for upload to remote ${RcloneUploadRemoteName}" else echo "$(date "+%d.%m.%Y %T") INFO: *** Creating IP address ${RCloneMountIP} for upload to remote ${RcloneUploadRemoteName}" ip addr add $RCloneMountIP/24 dev $NetworkAdapter label $NetworkAdapter:$VirtualIPNumber fi else RCloneMountIP="" fi # Remove --delete-empty-src-dirs if rclone sync or copy if [[ $RcloneCommand == 'move' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: *** Using rclone move - will add --delete-empty-src-dirs to upload." DeleteEmpty="--delete-empty-src-dirs " else echo "$(date "+%d.%m.%Y %T") INFO: *** Not using rclone move - will remove --delete-empty-src-dirs to upload." DeleteEmpty="" fi # Check --backup-directory if [[ $BackupJob == 'Y' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: *** Will backup to ${BackupRemoteLocation} and use ${BackupRemoteDeletedLocation} as --backup-directory with ${BackupRetention} retention for ${RcloneUploadRemoteName}." LocalFilesLocation="$LocalFilesShare" BackupDir="--backup-dir $RcloneUploadRemoteName:$BackupRemoteDeletedLocation" else BackupRemoteLocation="" BackupRemoteDeletedLocation="" BackupRetention="" BackupDir="" fi # process files rclone $RcloneCommand $LocalFilesLocation $RcloneUploadRemoteName:$BackupRemoteLocation $ServiceAccount $BackupDir \ --user-agent="$RcloneUploadRemoteName" \ -vv \ --buffer-size 512M \ --drive-chunk-size 512M \ --tpslimit 8 \ --checkers 8 \ --transfers 4 \ --order-by modtime,$ModSort \ --min-age $MinimumAge \ $Command1 $Command2 $Command3 $Command4 $Command5 $Command6 $Command7 $Command8 \ --exclude *fuse_hidden* \ --exclude *_HIDDEN \ --exclude .recycle** \ --exclude .Recycle.Bin/** \ --exclude *.backup~* \ --exclude *.partial~* \ --drive-stop-on-upload-limit \ --bwlimit "${BWLimit1Time},${BWLimit1} ${BWLimit2Time},${BWLimit2} ${BWLimit3Time},${BWLimit3}" \ --bind=$RCloneMountIP $DeleteEmpty # Delete old files from mount if [[ $BackupJob == 'Y' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: *** Removing files older than ${BackupRetention} from $BackupRemoteLocation for ${RcloneUploadRemoteName}." rclone delete --min-age $BackupRetention $RcloneUploadRemoteName:$BackupRemoteDeletedLocation fi ####### Remove Control Files ########## # update counter and remove other control files if [[ $UseServiceAccountUpload == 'Y' ]]; then if [[ "$CounterNumber" == "$CountServiceAccounts" ]];then rm /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/counter_* touch /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/counter_1 echo "$(date "+%d.%m.%Y %T") INFO: Final counter used - resetting loop and created counter_1." else rm /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/counter_* CounterNumber=$((CounterNumber+1)) touch /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/counter_$CounterNumber echo "$(date "+%d.%m.%Y %T") INFO: Created counter_${CounterNumber} for next upload run." fi else echo "$(date "+%d.%m.%Y %T") INFO: Not utilising service accounts." fi # remove dummy file rm /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/upload_running$JobName echo "$(date "+%d.%m.%Y %T") INFO: Script complete" exit RClone Unmount: #!/bin/bash ####################### ### Cleanup Script #### ####################### #### Version 0.9.2 #### ####################### echo "$(date "+%d.%m.%Y %T") INFO: *** Starting rclone_cleanup script ***" ####### Cleanup Tracking Files ####### echo "$(date "+%d.%m.%Y %T") INFO: *** Removing Tracking Files ***" find /mnt/user/appdata/other/rclone/remotes -name dockers_started* -delete find /mnt/user/appdata/other/rclone/remotes -name mount_running* -delete find /mnt/user/appdata/other/rclone/remotes -name upload_running* -delete echo "$(date "+%d.%m.%Y %T") INFO: ***Finished Cleanup! ***" exit Quote Link to comment
DZMM Posted August 12, 2020 Author Share Posted August 12, 2020 On 8/11/2020 at 4:32 AM, Emilio5639 said: Here are all of the Scripts that I have set up. I went ahead and changed my folder to: Plex_Media Nothing jumps out to me as looking wrong. @Kaizac @watchmeexplode5 ???? Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.