DZMM Posted October 14, 2020 Author Share Posted October 14, 2020 16 hours ago, privateer said: Any clue what's going on here? I pulled my server back up and get this error - no changes to my scripts from when it was running... 13.10.2020 10:53:56 INFO: Creating local folders. 13.10.2020 10:53:56 INFO: *** Starting mount of remote gdrive_media_vfs 13.10.2020 10:53:56 INFO: Checking if this script is already running. 13.10.2020 10:53:56 INFO: Script not running - proceeding. 13.10.2020 10:53:56 INFO: *** Checking if online 13.10.2020 10:53:57 PASSED: *** Internet online 13.10.2020 10:53:57 INFO: Success gdrive_media_vfs remote is already mounted. 13.10.2020 10:53:57 INFO: Mergerfs already installed, proceeding to create mergerfs mount 13.10.2020 10:53:57 INFO: Creating gdrive_media_vfs mergerfs mount. * ERROR: unable to parse 'branches' - ignore/gdrive_media_vfs:/mnt/user/mount_rclone/gdrive_media_vfs * ERROR: mountpoint not set 13.10.2020 10:53:57 INFO: Checking if gdrive_media_vfs mergerfs mount created. 13.10.2020 10:53:57 CRITICAL: gdrive_media_vfs mergerfs mount failed. Stopping dockers. think you've got a formatting error in your script options - post your script please Quote Link to comment
DZMM Posted October 14, 2020 Author Share Posted October 14, 2020 3 hours ago, jonodman said: Hey guys pretty new to unraid ive just setup my server and with a complete fresh setup and rclone as the guide says i cant stop my array any ideas Oct 13 13:45:49 Tower emhttpd: shcmd (109): sync Oct 13 13:45:49 Tower emhttpd: shcmd (110): umount /mnt/user0 Oct 13 13:45:49 Tower emhttpd: shcmd (111): rmdir /mnt/user0 Oct 13 13:45:49 Tower emhttpd: shcmd (112): umount /mnt/user Oct 13 13:45:49 Tower root: umount: /mnt/user: target is busy. Oct 13 13:45:49 Tower emhttpd: shcmd (112): exit status: 32 Oct 13 13:45:49 Tower emhttpd: shcmd (113): rmdir /mnt/user Oct 13 13:45:49 Tower root: rmdir: failed to remove '/mnt/user': Device or resource busy Oct 13 13:45:49 Tower emhttpd: shcmd (113): exit status: 1 Oct 13 13:45:49 Tower emhttpd: shcmd (115): /usr/local/sbin/update_cron Oct 13 13:45:49 Tower emhttpd: Retry unmounting user share(s)... Oct 13 13:45:54 Tower emhttpd: shcmd (116): umount /mnt/user Oct 13 13:45:54 Tower root: umount: /mnt/user: target is busy. We've never been able to pinpoint the problem as it seems intermittent. I haven't had problems for a few months now. Quote Link to comment
privateer Posted October 14, 2020 Share Posted October 14, 2020 4 hours ago, DZMM said: think you've got a formatting error in your script options - post your script please #!/bin/bash ###################### #### Mount Script #### ###################### ### Version 0.96.7 ### ###################### ####### EDIT ONLY THESE SETTINGS ####### # INSTRUCTIONS # 1. Change the name of the rclone remote and shares to match your setup # 2. NOTE: enter RcloneRemoteName WITHOUT ':' # 3. Optional: include custom command and bind mount settings # 4. Optional: include extra folders in mergerfs mount # REQUIRED SETTINGS RcloneRemoteName="gdrive_media_vfs" # Name of rclone remote mount WITHOUT ':'. NOTE: Choose your encrypted remote for sensitive data RcloneMountShare="/mnt/user/mount_rclone" # where your rclone remote will be located without trailing slash e.g. /mnt/user/mount_rclone LocalFilesShare="ignore" # location of the local files and MountFolders you want to upload without trailing slash to rclone e.g. /mnt/user/local. Enter 'ignore' to disable MergerfsMountShare="/mnt/user/mount_unionfs" # location without trailing slash e.g. /mnt/user/mount_mergerfs. Enter 'ignore' to disable DockerStart="binhex-rtorrentvpn plex ombi tautulli radarr radarr4k sonarr sonarr4k varken" # list of dockers, separated by space, to start once mergerfs mount verified. Remember to disable AUTOSTART for dockers added in docker settings page MountFolders=\{"downloads/complete,downloads/intermediate,downloads/seeds,movies,tv_shows,4kmovies"\} # comma separated list of folders to create within the mount # Note: Again - remember to NOT use ':' in your remote name above # OPTIONAL SETTINGS # Add extra paths to mergerfs mount in addition to LocalFilesShare LocalFilesShare2="ignore" # without trailing slash e.g. /mnt/user/other__remote_mount/or_other_local_folder. Enter 'ignore' to disable LocalFilesShare3="ignore" LocalFilesShare4="ignore" # Add extra commands or filters Command1="--rc" Command2="" Command3="" Command4="" Command5="" Command6="" Command7="" Command8="" CreateBindMount="N" # Y/N. Choose whether to bind traffic to a particular network adapter RCloneMountIP="192.168.1.123" # My unraid IP is 172.30.12.2 so I create another similar IP address NetworkAdapter="eth0" # choose your network adapter. eth0 recommended VirtualIPNumber="2" # creates eth0:x e.g. eth0:1. I create a unique virtual IP addresses for each mount & upload so I can monitor and traffic shape for each of them ####### END SETTINGS ####### ############################################################################### ##### DO NOT EDIT ANYTHING BELOW UNLESS YOU KNOW WHAT YOU ARE DOING ####### ############################################################################### ####### Preparing mount location variables ####### RcloneMountLocation="$RcloneMountShare/$RcloneRemoteName" # Location for rclone mount LocalFilesLocation="$LocalFilesShare/$RcloneRemoteName" # Location for local files to be merged with rclone mount MergerFSMountLocation="$MergerfsMountShare/$RcloneRemoteName" # Rclone data folder location ####### create directories for rclone mount and mergerfs mounts ####### mkdir -p /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName #for script files if [[ $LocalFileShare == 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Not creating local folders as requested." else echo "$(date "+%d.%m.%Y %T") INFO: Creating local folders." eval mkdir -p $LocalFilesLocation/"$MountFolders" fi mkdir -p $RcloneMountLocation mkdir -p $MergerFSMountLocation ####### Check if script is already running ####### echo "$(date "+%d.%m.%Y %T") INFO: *** Starting mount of remote ${RcloneRemoteName}" echo "$(date "+%d.%m.%Y %T") INFO: Checking if this script is already running." if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Exiting script as already running." exit else echo "$(date "+%d.%m.%Y %T") INFO: Script not running - proceeding." touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running fi ####### Checking have connectivity ####### echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if online" ping -q -c2 google.com > /dev/null # -q quiet, -c number of pings to perform if [ $? -eq 0 ]; then # ping returns exit status 0 if successful echo "$(date "+%d.%m.%Y %T") PASSED: *** Internet online" else echo "$(date "+%d.%m.%Y %T") FAIL: *** No connectivity. Will try again on next run" rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running exit fi ####### Create Rclone Mount ####### # Check If Rclone Mount Already Created if [[ -f "$RcloneMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Success ${RcloneRemoteName} remote is already mounted." else echo "$(date "+%d.%m.%Y %T") INFO: Mount not running. Will now mount ${RcloneRemoteName} remote." # Creating mountcheck file in case it doesn't already exist echo "$(date "+%d.%m.%Y %T") INFO: Recreating mountcheck file for ${RcloneRemoteName} remote." touch mountcheck rclone copy mountcheck $RcloneRemoteName: -vv --no-traverse # Check bind option if [[ $CreateBindMount == 'Y' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}" ping -q -c2 $RCloneMountIP > /dev/null # -q quiet, -c number of pings to perform if [ $? -eq 0 ]; then # ping returns exit status 0 if successful echo "$(date "+%d.%m.%Y %T") INFO: *** IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}" else echo "$(date "+%d.%m.%Y %T") INFO: *** Creating IP address ${RCloneMountIP} for remote ${RcloneRemoteName}" ip addr add $RCloneMountIP/24 dev $NetworkAdapter label $NetworkAdapter:$VirtualIPNumber fi echo "$(date "+%d.%m.%Y %T") INFO: *** Created bind mount ${RCloneMountIP} for remote ${RcloneRemoteName}" else RCloneMountIP="" echo "$(date "+%d.%m.%Y %T") INFO: *** Creating mount for remote ${RcloneRemoteName}" fi # create rclone mount rclone mount \ --allow-other \ --buffer-size 256M \ --dir-cache-time 720h \ --drive-chunk-size 512M \ --log-level INFO \ --vfs-read-chunk-size 128M \ --vfs-read-chunk-size-limit off \ --vfs-cache-mode writes \ --bind=$RCloneMountIP \ $RcloneRemoteName: $RcloneMountLocation & # Check if Mount Successful echo "$(date "+%d.%m.%Y %T") INFO: sleeping for 5 seconds" # slight pause to give mount time to finalise sleep 5 echo "$(date "+%d.%m.%Y %T") INFO: continuing..." if [[ -f "$RcloneMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Successful mount of ${RcloneRemoteName} mount." else echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mount failed - please check for problems. Stopping dockers" docker stop $DockerStart rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running exit fi fi ####### Start MergerFS Mount ####### if [[ $MergerfsMountShare == 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Not creating mergerfs mount as requested." else if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount in place." else # check if mergerfs already installed if [[ -f "/bin/mergerfs" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs already installed, proceeding to create mergerfs mount" else # Build mergerfs binary echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs not installed - installing now." mkdir -p /mnt/user/appdata/other/rclone/mergerfs docker run -v /mnt/user/appdata/other/rclone/mergerfs:/build --rm trapexit/mergerfs-static-build mv /mnt/user/appdata/other/rclone/mergerfs/mergerfs /bin # check if mergerfs install successful echo "$(date "+%d.%m.%Y %T") INFO: *sleeping for 5 seconds" sleep 5 if [[ -f "/bin/mergerfs" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs installed successfully, proceeding to create mergerfs mount." else echo "$(date "+%d.%m.%Y %T") ERROR: Mergerfs not installed successfully. Please check for errors. Exiting." rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running exit fi fi # Create mergerfs mount echo "$(date "+%d.%m.%Y %T") INFO: Creating ${RcloneRemoteName} mergerfs mount." # Extra Mergerfs folders if [[ $LocalFilesShare2 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare2} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare2=":$LocalFilesShare2" else LocalFilesShare2="" fi if [[ $LocalFilesShare3 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare3} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare3=":$LocalFilesShare3" else LocalFilesShare3="" fi if [[ $LocalFilesShare4 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare4} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare4=":$LocalFilesShare4" else LocalFilesShare4="" fi # make sure mergerfs mount point is empty mv $MergerFSMountLocation $LocalFilesLocation mkdir -p $MergerFSMountLocation # mergerfs mount command mergerfs $LocalFilesLocation:$RcloneMountLocation$LocalFilesShare2$LocalFilesShare3$LocalFilesShare4 $MergerFSMountLocation -o rw,async_read=false,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=partial,dropcacheonclose=true # check if mergerfs mount successful echo "$(date "+%d.%m.%Y %T") INFO: Checking if ${RcloneRemoteName} mergerfs mount created." if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount created." else echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mergerfs mount failed. Stopping dockers." docker stop $DockerStart rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running exit fi fi fi ####### Starting Dockers That Need Mergerfs Mount To Work Properly ####### # only start dockers once if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: dockers already started." else touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started echo "$(date "+%d.%m.%Y %T") INFO: Starting dockers." docker start $DockerStart fi rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running echo "$(date "+%d.%m.%Y %T") INFO: Script complete" exit Quote Link to comment
DZMM Posted October 14, 2020 Author Share Posted October 14, 2020 1 hour ago, privateer said: LocalFilesShare="ignore" # location of the local files and MountFolders you want to upload without trailing slash to rclone e.g. /mnt/user/local. Enter 'ignore' to disable MergerfsMountShare="/mnt/user/mount_unionfs" # location without trailing slash e.g. /mnt/user/mount_mergerfs. Enter 'ignore' to disable ok, logic fail in my script. If you don't want to create a mergerfs mount you need to set: MergerfsMountShare="ignore" You haven't, so it's trying to create a to create a mergerfs mount - but it can't because LocalFileShare is set to ignore. To fix either set MergerfsMountShare to ignore, or add a location for LocalFileShare Quote Link to comment
privateer Posted October 14, 2020 Share Posted October 14, 2020 The setup I'm using right now doesn't involve auto-uploading, but instead manually moving files to be uploaded. Is the ignore setting on both what I'm looking for? Using the OG scripts I just manually dumped things into the upload folder. Quote Link to comment
tsmebro Posted October 15, 2020 Share Posted October 15, 2020 (edited) 15 hours ago, DZMM said: can you post a copy of your email please if you can as mine didn't have that detail Sorry just checked and pricing wasn't actually in the email, but following the instructions on upgrading it looks like Enterprise Standard has unlimited storage with only one user needed for 20/month. Bit more than I pay now but I guess it's still cheaper than if they enforced the 5 user thing on business. https://support.google.com/a/answer/10085477?authuser=2 I haven't upgraded yet, I might hold off until they make me! Edited October 15, 2020 by tsmebro 1 Quote Link to comment
M1kep_ Posted October 15, 2020 Share Posted October 15, 2020 (edited) I want to make sure I'm understanding this correctly. Steps I intend to take for shares: Create User Shares with names of gDriveStaging(LocalFileShare), gDrive(RcloneMountShare), mergerFS(MergerfsMountShare) and set to "Prefer Cache" Set the variables in the scripts to match Rclone: With encryption, RcloneRemoteName should be the crypt remote When the array is stopped, Unraid will handle the unmount as it's a User Share When the array is started, the mount script will just need to be ran again Random questions: Is there a particular reason why vfs-cache-mode is writes, instead of full? Any gotchas when using full(I assume that a cache-dir would need to be set to be sure it doesn't try to full up the root fs)? How do the service accounts work? It look like the counter is updated, but the next service account will only be used at the next upload interval? Do the service accounts help with the 750GB upload limit? It is mentioned that MergerFS works better when everything is mapped like /user --Maps to--> /mnt/user Would there be any issues with the following setup? Current Setup: Google Drive path: /mnt/disks/gdrive_secure/ Radarr: /ShareName ---> /mnt/disks/gdrive_secure/ShareName Sonarr: /ShareName/TV ---> /mnt/disks/gdrive_secure/ShareName/TV Proposed Setup MergerFS Path: /mnt/user/gDrive Radarr: /ShareName ---> /mnt/user/gDrive/ShareName Sonarr: /ShareName/TV ---> /mnt/user/gDrive/ShareName/TV The goal is to avoid having to remap all of the directories and rescan everything. But if mergerFS is going to make it a pain, rescanning/remapping isn't the end of the world Edited October 15, 2020 by M1kep_ Quote Link to comment
DZMM Posted October 15, 2020 Author Share Posted October 15, 2020 10 hours ago, tsmebro said: Sorry just checked and pricing wasn't actually in the email, but following the instructions on upgrading it looks like Enterprise Standard has unlimited storage with only one user needed for 20/month. Bit more than I pay now but I guess it's still cheaper than if they enforced the 5 user thing on business. https://support.google.com/a/answer/10085477?authuser=2 I haven't upgraded yet, I might hold off until they make me! Thanks. I was too scared to click on the upgrade link before.... I went for it this time and it's spitting out £15.30/mth in the UK - strangely no discount for annual. I'm paying £7.82/mth at the moment - double the price, but still great value for over 600TB of storage!! I'm not upgrading either yet - unless I can find a coupon for Enterprise Standard. It's a good sign that Google aren't shutting down the facility - there are bigger things behind this change, rather than trying to address users like us. I still believe we are a drop in the ocean Vs some institutions e.g. research universities. 1 Quote Link to comment
DZMM Posted October 15, 2020 Author Share Posted October 15, 2020 (edited) 11 hours ago, M1kep_ said: Create User Shares with names of gDriveStaging(LocalFileShare), gDrive(RcloneMountShare), mergerFS(MergerfsMountShare) and set to "Prefer Cache" prefer cache is a preference - go with wherever you want files storing 11 hours ago, M1kep_ said: When the array is started, the mount script will just need to be ran again Correct. Advice is to run using user scripts on a schedule e.g. every 2-5 mins, so if for whatever reason the mount fails, it will stop the dockers to try and avoid any meltdowns 11 hours ago, M1kep_ said: Is there a particular reason why vfs-cache-mode is writes, instead of full? since rclone 1.53 (I think) full works better - before it was a bad idea. I've updated my local scripts, but I had a few problems updating github so that's a bit behind (it also takes advantage of the new cache feature, which makes a BIG difference). I'll try again to update github again this weekend. 11 hours ago, M1kep_ said: How do the service accounts work? It look like the counter is updated, but the next service account will only be used at the next upload interval? Do the service accounts help with the 750GB upload limit? Correct - rclone added a nice feature a few releases back that stopped a transfer job when the 750GB limit is hit. SAs allow a different user or quota to be used on the next run, so if you set a cron job, you can do multiple amounts of 750GB/day e.g. 16 SAs allows you to max out a 1Gbps upload 24/7. 11 hours ago, M1kep_ said: Proposed Setup MergerFS Path: /mnt/user/gDrive Radarr: /ShareName ---> /mnt/user/gDrive/ShareName Sonarr: /ShareName/TV ---> /mnt/user/gDrive/ShareName/TV yeah, you won't get the full benefit of mergerfs. I would honestly just do this for all your dockers: /user --> /mnt/user and then within radarr add as your media locations /user/gDrive/ShareName/Movies etc etc I just find it easier to do this as all dockers will always match up, and I think the convention of mapping /downloads --> xxxxxx /media --> xxxxx etc, just makes live more complicated. Moving is easy: 1. add the new /user --> /mnt/user mapping 2. add it /user/whatever as a new media location in radarr 3. go to Movie Editor, create a custom filter e.g. show movies with /ShareName/Movies etc in path 4. select all and move them to /user/whatever within Radarr Edited October 15, 2020 by DZMM Quote Link to comment
M1kep_ Posted October 15, 2020 Share Posted October 15, 2020 9 hours ago, DZMM said: since rclone 1.53 (I think) full works better - before it was a bad idea. I've updated my local scripts, but I had a few problems updating github so that's a bit behind (it also takes advantage of the new cache feature, which makes a BIG difference). I'll try again to update github again this weekend. Awesome! Yeah, the vfs local caching is really awesome. Really merges the gap between hosting on cloud and locally. Especially for files that are being accessed recently/often. I'll keep an eye out on the GitHub to see if any changes show up. If you don't get a chance to update, if you could possibly provide your mount flags at least that'd be very useful. I'll probably work on getting this migrated over this weekend. Thanks again for the help Quote Link to comment
DZMM Posted October 15, 2020 Author Share Posted October 15, 2020 (edited) 34 minutes ago, M1kep_ said: Awesome! Yeah, the vfs local caching is really awesome. Really merges the gap between hosting on cloud and locally. Especially for files that are being accessed recently/often. I'll keep an eye out on the GitHub to see if any changes show up. If you don't get a chance to update, if you could possibly provide your mount flags at least that'd be very useful. I'll probably work on getting this migrated over this weekend. Thanks again for the help Here's my updated mount script. I had problems adding @watchmeexplode5's pull that creates a single config file. Adds support for vfs local caching and also changed the mount defaults, which for me, has significantly improved playback and skipping: Edit: I just spotted I've hardcoded the cache size at 400GB - I'll make this configurable when I have time to sort out github. #!/bin/bash ###################### #### Mount Script #### ###################### ### Version 0.96.8 ### ###################### ####### EDIT ONLY THESE SETTINGS ####### # INSTRUCTIONS # 1. Change the name of the rclone remote and shares to match your setup # 2. NOTE: enter RcloneRemoteName WITHOUT ':' # 3. Optional: include custom command and bind mount settings # 4. Optional: include extra folders in mergerfs mount # REQUIRED SETTINGS RcloneRemoteName="tdrive_vfs" RcloneMountShare="/mnt/user/mount_rclone" LocalFilesShare="/mnt/user/local" MergerfsMountShare="/mnt/user/mount_mergerfs" DockerStart="duplicati lazylibrarian LDAPforPlex letsencrypt nzbget ombi organizrv2 plex qbittorrentvpn radarr radarr-uhd radarr-collections sonarr sonarr-uhd tautulli" MountFolders=\{"downloads/complete,downloads/seeds,documentaries/kids,documentaries/adults,movies_adults_gd,movies_kids_gd,tv_adults_gd,tv_kids_gd,uhd/tv_adults_gd,uhd/tv_kids_gd,uhd/documentaries/kids,uhd/documentaries/adults"\} # OPTIONAL SETTINGS # Add extra paths to mergerfs mount in addition to LocalFilesShare LocalFilesShare2="/mnt/user/mount_rclone/gdrive_media_vfs" # without trailing slash e.g. /mnt/user/other__remote_mount/or_other_local_folder. Enter 'ignore' to disable LocalFilesShare3="" LocalFilesShare4="" LocalFilesShare5="" LocalFilesShare6="" # Add extra commands or filters Command1="--rc" Command2="" Command3="" Command4="" Command5="" Command6="" Command7="" Command8="" CreateBindMount="N" # Y/N. Choose whether to bind traffic to a particular network adapter RCloneMountIP="192.168.1.252" # My unraid IP is 172.30.12.2 so I create another similar IP address NetworkAdapter="eth0" # choose your network adapter. eth0 recommended VirtualIPNumber="2" # creates eth0:x e.g. eth0:1. I create a unique virtual IP addresses for each mount & upload so I can monitor and traffic shape for each of them ####### END SETTINGS ####### ############################################################################### ##### DO NOT EDIT ANYTHING BELOW UNLESS YOU KNOW WHAT YOU ARE DOING ####### ############################################################################### ####### Preparing mount location variables ####### LocalFilesLocation="$LocalFilesShare/$RcloneRemoteName" RcloneMountLocation="$RcloneMountShare/$RcloneRemoteName" MergerFSMountLocation="$MergerfsMountShare/$RcloneRemoteName" ####### create directories for rclone mount and mergerfs mounts ####### mkdir -p /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName #for script files mkdir -p /mnt/user0/mount_rclone/cache/$RcloneRemoteName #for cache files if [[ $LocalFileShare == 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Not creating local folders as requested." else echo "$(date "+%d.%m.%Y %T") INFO: Creating local folders." eval mkdir -p $LocalFilesLocation/"$MountFolders" fi mkdir -p $RcloneMountLocation mkdir -p $MergerFSMountLocation ####### Check if script is already running ####### echo "$(date "+%d.%m.%Y %T") INFO: *** Starting mount of remote ${RcloneRemoteName}" echo "$(date "+%d.%m.%Y %T") INFO: Checking if this script is already running." if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Exiting script as already running." exit else echo "$(date "+%d.%m.%Y %T") INFO: Script not running - proceeding." touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running fi ####### Create Rclone Mount ####### # Check If Rclone Mount Already Created if [[ -f "$RcloneMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Success ${RcloneRemoteName} remote is already mounted." else echo "$(date "+%d.%m.%Y %T") INFO: Mount not running. Will now mount ${RcloneRemoteName} remote." # Creating mountcheck file in case it doesn't already exist echo "$(date "+%d.%m.%Y %T") INFO: Recreating mountcheck file for ${RcloneRemoteName} remote." touch mountcheck rclone copy mountcheck $RcloneRemoteName: -vv --no-traverse # Check bind option if [[ $CreateBindMount == 'Y' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}" ping -q -c2 $RCloneMountIP > /dev/null # -q quiet, -c number of pings to perform if [ $? -eq 0 ]; then # ping returns exit status 0 if successful echo "$(date "+%d.%m.%Y %T") INFO: *** IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}" else echo "$(date "+%d.%m.%Y %T") INFO: *** Creating IP address ${RCloneMountIP} for remote ${RcloneRemoteName}" ip addr add $RCloneMountIP/24 dev $NetworkAdapter label $NetworkAdapter:$VirtualIPNumber fi echo "$(date "+%d.%m.%Y %T") INFO: *** Created bind mount ${RCloneMountIP} for remote ${RcloneRemoteName}" else RCloneMountIP="" echo "$(date "+%d.%m.%Y %T") INFO: *** Creating mount for remote ${RcloneRemoteName}" fi # create rclone mount rclone mount \ --allow-other \ --dir-cache-time 720h \ --log-level INFO \ --poll-interval 15s \ --cache-dir=/mnt/user0/mount_rclone/cache/$RcloneRemoteName \ --vfs-cache-mode full \ --vfs-cache-max-size 400G \ --vfs-cache-max-age 336h \ --bind=$RCloneMountIP \ $RcloneRemoteName: $RcloneMountLocation & # Check if Mount Successful echo "$(date "+%d.%m.%Y %T") INFO: sleeping for 5 seconds" # slight pause to give mount time to finalise sleep 10 echo "$(date "+%d.%m.%Y %T") INFO: continuing..." if [[ -f "$RcloneMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Successful mount of ${RcloneRemoteName} mount." else echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mount failed - please check for problems." docker stop $DockerStart find /mnt/user/appdata/other/rclone/remotes -name mount_running* -delete rm /mnt/user/appdata/other/scripts/running/fast_check exit fi fi ####### Start MergerFS Mount ####### if [[ $MergerfsMountShare == 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Not creating mergerfs mount as requested." else if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount in place." else # check if mergerfs already installed if [[ -f "/bin/mergerfs" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs already installed, proceeding to create mergerfs mount" else # Build mergerfs binary echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs not installed - installing now." mkdir -p /mnt/user/appdata/other/rclone/mergerfs docker run -v /mnt/user/appdata/other/rclone/mergerfs:/build --rm trapexit/mergerfs-static-build mv /mnt/user/appdata/other/rclone/mergerfs/mergerfs /bin # check if mergerfs install successful echo "$(date "+%d.%m.%Y %T") INFO: *sleeping for 5 seconds" sleep 10 if [[ -f "/bin/mergerfs" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs installed successfully, proceeding to create mergerfs mount." else echo "$(date "+%d.%m.%Y %T") ERROR: Mergerfs not installed successfully. Please check for errors. Exiting." docker stop $DockerStart find /mnt/user/appdata/other/rclone/remotes -name mount_running* -delete rm /mnt/user/appdata/other/scripts/running/fast_check exit fi fi # Create mergerfs mount echo "$(date "+%d.%m.%Y %T") INFO: Creating ${RcloneRemoteName} mergerfs mount." # Extra Mergerfs folders if [[ $LocalFilesShare2 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare2} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare2=":$LocalFilesShare2" else LocalFilesShare2="" fi if [[ $LocalFilesShare3 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare3} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare3=":$LocalFilesShare3" else LocalFilesShare3="" fi if [[ $LocalFilesShare4 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare4} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare4=":$LocalFilesShare4" else LocalFilesShare4="" fi if [[ $LocalFilesShare5 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare5} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare5=":$LocalFilesShare5" else LocalFilesShare5="" fi if [[ $LocalFilesShare6 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare6} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare6=":$LocalFilesShare6" else LocalFilesShare6="" fi # mergerfs mount command mergerfs $LocalFilesLocation:$RcloneMountLocation$LocalFilesShare2$LocalFilesShare3$LocalFilesShare4$LocalFilesShare5$LocalFilesShare6 $MergerFSMountLocation -o rw,async_read=false,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=partial,dropcacheonclose=true # check if mergerfs mount successful echo "$(date "+%d.%m.%Y %T") INFO: Checking if ${RcloneRemoteName} mergerfs mount created." if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount created." else echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mergerfs mount failed." docker stop $DockerStart find /mnt/user/appdata/other/rclone/remotes -name mount_running* -delete rm /mnt/user/appdata/other/scripts/running/fast_check exit fi fi fi ####### Starting Dockers That Need Mergerfs Mount To Work Properly ####### echo "$(date "+%d.%m.%Y %T") INFO: Starting dockers." docker start $DockerStart rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running echo "$(date "+%d.%m.%Y %T") INFO: ${RcloneRemoteName} Script complete" exit Edited October 15, 2020 by DZMM Quote Link to comment
M1kep_ Posted October 15, 2020 Share Posted October 15, 2020 (edited) Okay, pretty much finished going through the whole thing and building out my setup plan. Two more questions: 1 hour ago, DZMM said: --cache-dir=/mnt/user0/mount_rclone/cache/$RcloneRemoteName \ I've found that the /user0/ mount writes to the disks instead of the cache. With that being said, what is your mount_rclone share set as for the cache settings in Unraid? With Prefer, I feel like the mover would constantly try to move the data off of the array and throw it on cache. Also, does having the cache in the mount_rclone folder cause it to be uploaded to google as well as that's a mounted folder? 1 hour ago, DZMM said: # Add extra paths to mergerfs mount in addition to LocalFilesShare LocalFilesShare2="/mnt/user/mount_rclone/gdrive_media_vfs" # without trailing slash e.g. /mnt/user/other__remote_mount/or_other_local_folder. Enter 'ignore' to disable Is this needed, or an artifact of your specific setup and can be replaced with "ignore"? Edited October 15, 2020 by M1kep_ Quote Link to comment
DZMM Posted October 15, 2020 Author Share Posted October 15, 2020 52 minutes ago, M1kep_ said: Okay, pretty much finished going through the whole thing and building out my setup plan. Two more questions: I've found that the /user0/ mount writes to the disks instead of the cache. With that being said, what is your mount_rclone share set as for the cache settings in Unraid? With Prefer, I feel like the mover would constantly try to move the data off of the array and throw it on cache. Also, does having the cache in the mount_rclone folder cause it to be uploaded to google as well as that's a mounted folder? Is this needed, or an artifact of your specific setup and can be replaced with "ignore"? my cache isn't big enough for a permanent vfs cache. choose your own preferred location 53 minutes ago, M1kep_ said: Is this needed, or an artifact of your specific setup and can be replaced with "ignore"? put to ignore - I left it in as an example - in my real script all 6 are populated with various tdrives. Quote Link to comment
DZMM Posted October 16, 2020 Author Share Posted October 16, 2020 (edited) I just spotted this in my upload logs: 2020/10/16 08:37:03 INFO : Transferred: 1.344T / 9.679 TBytes, 14%, 13.292 MBytes/s, ETA 1w14h39m29s Checks: 286 / 290, 99% Deleted: 143 Renamed: 143 Transferred: 143 / 2190, 7% Elapsed time: 29h27m1.0s I wonder if Google have removed/increased the 750GB/day transfer limit? Edit: I think it's still 750GB/day - my elapsed time is high so I must have fluked transferring less than 750GB over 24 hours Edited October 16, 2020 by DZMM Quote Link to comment
veritas2884 Posted October 16, 2020 Share Posted October 16, 2020 I have recently started using PiHole to block ads and route my DNS. The issue i've run into is that now the mount script fails because it tries to run before the internet connection is reestablished since PiHole is starting up at the same time. Is there a way to delay the script running a little bit to allow PiHole it get up and running? Generally it isn't a problem because I just re-run the startup mount script, but if the server resets when i'm away from home everything stays down until I re-run the script. I 100% know this is my fault for the setup I have running and I really appreciate all the work and help you guys provide, just wanted to see if there is an idea on how to deal with this. Quote Link to comment
DZMM Posted October 16, 2020 Author Share Posted October 16, 2020 1 hour ago, veritas2884 said: I have recently started using PiHole to block ads and route my DNS. The issue i've run into is that now the mount script fails because it tries to run before the internet connection is reestablished since PiHole is starting up at the same time. Is there a way to delay the script running a little bit to allow PiHole it get up and running? Generally it isn't a problem because I just re-run the startup mount script, but if the server resets when i'm away from home everything stays down until I re-run the script. I 100% know this is my fault for the setup I have running and I really appreciate all the work and help you guys provide, just wanted to see if there is an idea on how to deal with this. The mount script checks if there's internet connectivity. Add a cron job so the script runs if the test fails. ####### Checking have connectivity ####### echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if online" ping -q -c2 google.com > /dev/null # -q quiet, -c number of pings to perform if [ $? -eq 0 ]; then # ping returns exit status 0 if successful echo "$(date "+%d.%m.%Y %T") PASSED: *** Internet online" else echo "$(date "+%d.%m.%Y %T") FAIL: *** No connectivity. Will try again on next run" rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running exit fi Quote Link to comment
animeking Posted October 19, 2020 Share Posted October 19, 2020 need some assistance, i keep getting the gdrive api ban. im really not a rclone expert with writing scripts or commands to stop the bans. so if someone is willing to guide me i would be happy to listen so i can stop getting the api bans. so far all i have accomplished is mounted gdrive and made a script. i followed the spaceinvader... Quote Link to comment
Bolagnaise Posted October 19, 2020 Share Posted October 19, 2020 2 hours ago, animeking said: need some assistance, i keep getting the gdrive api ban. im really not a rclone expert with writing scripts or commands to stop the bans. so if someone is willing to guide me i would be happy to listen so i can stop getting the api bans. so far all i have accomplished is mounted gdrive and made a script. i followed the spaceinvader... Have you looked at page 1, all the scripts are linked there. There is already a turn key solution in place. Quote Link to comment
animeking Posted October 19, 2020 Share Posted October 19, 2020 Turn key solution??? i did not see that @Bolagnaise Quote Link to comment
animeking Posted October 19, 2020 Share Posted October 19, 2020 I have enabled the api and made creditenals. But the script writing im not really understanding Sent from my SM-N986U using Tapatalk Quote Link to comment
Bolagnaise Posted October 20, 2020 Share Posted October 20, 2020 1 hour ago, animeking said: I have enabled the api and made creditenals. But the script writing im not really understanding Sent from my SM-N986U using Tapatalk You do not need to write any scripts, as per DZMM post on page 1, they are all listed on github https://github.com/BinsonBuzz/unraid_rclone_mount Quote Link to comment
animeking Posted October 20, 2020 Share Posted October 20, 2020 Does it matter if I mount at mnt/ or mnt/disksSent from my SM-N986U using Tapatalk Quote Link to comment
animeking Posted October 20, 2020 Share Posted October 20, 2020 i get this when i run the mount script Quote Link to comment
animeking Posted October 20, 2020 Share Posted October 20, 2020 I think i got it working. just need confirmation is this how it suppose to look. i need those api bans to stop!! upload.txt mount.txt Quote Link to comment
animeking Posted October 20, 2020 Share Posted October 20, 2020 I must have not set something up right i have already got a errorhttp://cloud.tapatalk.com/s/5f8ecb5b40329/log.txtSent from my SM-N986U using Tapatalk Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.