axeman

okay .. is that why I can't seem to shutdown the array safely? for some reason that mergerfs never stops, and without a manual kill command, my array refuses to shutdown. there must be something wrong with my mergerFS setup. it's just not mounting anything at all. mind boggling.

Thanks - backup will be awesome for woopsies. I don't see anything at all. I'm starting to think something is wrong with mergerFS ... I don't see it in my Dockers. Should it be there as a docker? Also whenever I reboot the server, on the first run of the mount script, it says mergerFS not installed and proceeds to "install" it I never see anything in Docker. Maybe I can be test dummy for the union backend you guys are talking about above? I'm ready, willing, and (somewhat) able to test for you, with some guidance.

@DZMM I deleted my rclone remotes and recreated - and now all seems to be fine. Every now and then the upload script seems to crash. Not sure why it's happening. two more questions. 1. my rclone mode is set to "sync" and i'm using the backup = 'Y' option... does "sync' imply that if i go to google drive and manually delete something, it'll delete from my array? So i'm guessing I want "copy". 2. when I look at my mergerfs_mount\videos ... i just see the "MountFolders" that I specified. I thought when I go to the mergerfs_mount, I would see all of my shares/files from local array and google drive? Thanks - I was really close to just giving up on this - and am almost seeing the light at the end of the tunnel.

hmm it is running: 26.05.2020 17:23:24 INFO: Creating local folders. 26.05.2020 17:23:24 INFO: *** Starting mount of remote gdrive_media_vfs 26.05.2020 17:23:24 INFO: Checking if this script is already running. 26.05.2020 17:23:24 INFO: Script not running - proceeding. 26.05.2020 17:23:24 INFO: Mount not running. Will now mount gdrive_media_vfs remote. 26.05.2020 17:23:24 INFO: Recreating mountcheck file for gdrive_media_vfs remote. 2020/05/26 17:23:24 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "mountcheck" "gdrive_media_vfs:" "-vv" "--no-traverse"] 2020/05/26 17:23:24 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf" 2020/05/26 17:23:24 DEBUG : mountcheck: Modification times differ by -57m50.100450373s: 2020-05-26 17:23:24.211100396 -0400 EDT, 2020-05-26 16:25:34.110650023 -0400 EDT 2020/05/26 17:23:24 INFO : mountcheck: Copied (replaced existing) 2020/05/26 17:23:24 INFO : Transferred: 32 / 32 Bytes, 100%, 138.821 kBytes/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 0.0s 2020/05/26 17:23:24 DEBUG : 4 go routines active 2020/05/26 17:23:24 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "mountcheck" "gdrive_media_vfs:" "-vv" "--no-traverse"] 26.05.2020 17:23:24 INFO: *** Creating mount for remote gdrive_media_vfs 26.05.2020 17:23:24 INFO: sleeping for 5 seconds 2020/05/26 17:23:24 INFO : Encrypted drive 'gdrive_media_vfs:': poll-interval is not supported by this remote 26.05.2020 17:23:29 INFO: continuing... 26.05.2020 17:23:29 INFO: Successful mount of gdrive_media_vfs mount. 26.05.2020 17:23:29 INFO: Check successful, gdrive_media_vfs mergerfs mount in place. 26.05.2020 17:23:29 INFO: dockers already started. 26.05.2020 17:23:29 INFO: Script complete How do I check that mergerfs is installed? Is it supposed to be a docker? When I check in dockers, I don't see anything (before this, I haven't been using Docker). We're almost there - i can almost taste it!

/appdata/other/rclone has the service_accounts folder. I kept that and deleted everything else. The rest of my settings look OK right? I have my appdata, mount_mergerfs and mount_rclone set to "cache only" that can't cause this, can it? After the deletes, it's not detecting rclone: "26.05.2020 16:26:05 INFO: Checking if rclone installed successfully. 26.05.2020 16:26:05 INFO: rclone not installed - will try again later." I looked and it seems it's just checking for the existence of a file right? or is there more to it?

Mount: # REQUIRED SETTINGS RcloneRemoteName="gdrive_media_vfs" # Name of rclone remote mount WITHOUT ':'. NOTE: Choose your encrypted remote for sensitive data RcloneMountShare="/mnt/user/mount_rclone" # where your rclone remote will be located without trailing slash e.g. /mnt/user/mount_rclone MergerfsMountShare="/mnt/user/mount_mergerfs" # location without trailing slash e.g. /mnt/user/mount_mergerfs. Enter 'ignore' to disable DockerStart="" # list of dockers, separated by space, to start once mergerfs mount verified. Remember to disable AUTOSTART for dockers added in docker settings page LocalFilesShare="/mnt/user/Videos" # location of the local files and MountFolders you want to upload without trailing slash to rclone e.g. /mnt/user/local. Enter 'ignore' to disable MountFolders=\{"downloads/complete,downloads/intermediate,downloads/seeds,movies,tv,Videos/Animation_shorts,Trailers"\} # comma separated list of folders to create within the mount Upload: # REQUIRED SETTINGS RcloneCommand="sync" # choose your rclone command e.g. move, copy, sync RcloneRemoteName="gdrive_media_vfs" # Name of rclone remote mount WITHOUT ':'. RcloneUploadRemoteName="gdrive_media_vfs" # If you have a second remote created for uploads put it here. Otherwise use the same remote as RcloneRemoteName. LocalFilesShare="/mnt/user/Videos" # location of the local files without trailing slash you want to rclone to use RcloneMountShare="/mnt/user/mount_rclone" # where your rclone mount is located without trailing slash e.g. /mnt/user/mount_rclone MinimumAge="15m" # sync files suffix ms|s|m|h|d|w|M|y ModSort="ascending" # "ascending" oldest files first, "descending" newest files first #kept all the rest default, except this stuff: # Is this a backup job BackupJob="Y" # Y/N. Syncs or Copies files from LocalFilesLocation to BackupRemoteLocation, rather than moving from LocalFilesLocation/RcloneRemoteName BackupRemoteLocation="backup" # choose location on mount for deleted sync files BackupRemoteDeletedLocation="backup_deleted" # choose location on mount for deleted sync files BackupRetention="2M" # How long to keep deleted sync files suffix ms|s|m|h|d|w|M|y

@DZMM I'm so close, yet so far... the upload script runs fine, without any errors, but for some reason "nothing" is happening... I see folders and files in this folder: /appdata/other/rclone/remotes/gdrive_media_vfs/gdrive/a24321dsfasdfafadsf132/ but those are physical files on my HD, taking up about 200GB of space.

do you have anything in /mnt/user/appdata/other/rclone/remotes/ ? any other folder?

My unmount wasn't working because of the checker file name... i added a * to the end of the find command. committed in my fork and did a pull request for you. my first ever pull request woo.

Weird I have it run on startup. It doesn't seem to run at all. I wonder if something got messed up in the copy/pasta. Any idea why the array doesn't stop? It seems to get stuck on : Array Stopping•Retry unmounting user share(s)... MergerFS was the culprit. I followed this:

So I find that the path rclone\remotes\gdrive_media_vfs\gdrive\ has some files in it. Probably the scrap stuff that RClone was working on encrypting before upload? Also see an empty file called "upload_running_daily_upload" is that what's used to see if something is in progress? I was able to figure out the team drive thing... it was because I created the client id outside of the project that has the service accounts. thought I read somewhere that it didn't matter. It does.

So it can't seem to shutdown properly. And even though I have the schedule completely turned off, even after a reboot, I can't seem to start the uploader script. It says the script is already running. starting to feel like I might have bit off more than I can chew with this setup. 🥵

I still think I'm doing something wrong. When I try to list the directories using rclone gives me an error that the team drive doesn't exist. What's funny is that when I tried it earlier I was uploading somewhere I just don't know where those files went. Hope you are at least enjoying your time while awake.

awesome... hold that thought though. That's a future problem lol. I don't have my cameras fully writing to a network share yet. in a Post Covid world, I'm looking at obtaining an IP camera system. OT question - when the hell do you sleep? you seem to be online 24/7

Thanks - will think about this after getting the videos going. Now - for the exclude directive, can I put a bunch of --exclude in one comannd? Command1 = "--exclude animation/* --exclude tv_series --exclude my_other_folder --exclude this_folder_too" ? I have many folders to exclude (more than I can with the 8 commands).

heh, okay - I was afraid of rebooting. Will try that. Can I get stupid(er) for a minute? This could be a killer setup for a camera DVR, if we can set the upload time separately based on share. Like for a DVR setup, push up to cloud as quickly as possible, in case of a robbery, data has already been pushed to cloud. Also wondering if it's OK to point the mount to a cache drive. So that when we're streaming a movie, it doesn't write to an array share?

Thank you for your continued patience with this... I agree - the approach was wrong. I'm going to restart this. What's the best way to stop the upload that's currently running? should I run the clean up script, or is there another way?

So my UnRaid Array looks like this: \\server\Videos\3D_Movies \\server\Videos\Animation \\server\Videos\TV_Shows For now, I just started with /videos/3D_Movies to see how this all works. Say a month from now, I say, this is all great, I want to do all of my array. Can I just change the path on rclone_mount to the /videos ? So I ran the script through UnRaid gui... i see it reporting 14.874 MBytes/s ... i'm guessing that's mbits? My upload speed is only 40mbit max, which translates to really 5MBytes/sec

Thanks - I was afraid of that... but figured, hey it's in quotes so it should be fine. But just so I understand, the individual movie folders underneath that are fine to have spaces, right? Also, assuming I do go full tilt a year from now... and want to do the full "videos" share... will that end up creating duplicates of the 3D Movies folder?

Thanks, that's what I thought too, but double checked, and looked ok. Note, I'm only trying one subfolder in my share, perhaps that's the issue? I have a videos share, and then folders like 3D Movies, Animation, HD Movies, TV Series. I started out with the smallest of those. rclone_mount: #!/bin/bash ###################### #### Mount Script #### ###################### ### Version 0.96.6 ### ###################### ####### EDIT ONLY THESE SETTINGS ####### # INSTRUCTIONS # 1. Change the name of the rclone remote and shares to match your setup # 2. NOTE: enter RcloneRemoteName WITHOUT ':' # 3. Optional: include custom command and bind mount settings # 4. Optional: include extra folders in mergerfs mount # REQUIRED SETTINGS RcloneRemoteName="gdrive_media_vfs" # Name of rclone remote mount WITHOUT ':'. NOTE: Choose your encrypted remote for sensitive data RcloneMountShare="/mnt/user/mount_rclone" # where your rclone remote will be located without trailing slash e.g. /mnt/user/mount_rclone MergerfsMountShare="/mnt/user/mount_mergerfs" # location without trailing slash e.g. /mnt/user/mount_mergerfs. Enter 'ignore' to disable DockerStart="" # list of dockers, separated by space, to start once mergerfs mount verified. Remember to disable AUTOSTART for dockers added in docker settings page LocalFilesShare="/mnt/user/Videos/3D movies" # location of the local files and MountFolders you want to upload without trailing slash to rclone e.g. /mnt/user/local. Enter 'ignore' to disable MountFolders=\{"downloads/complete,downloads/intermediate,downloads/seeds,movies,tv"\} # comma separated list of folders to create within the mount # Note: Again - remember to NOT use ':' in your remote name above # OPTIONAL SETTINGS # Add extra paths to mergerfs mount in addition to LocalFilesShare LocalFilesShare2="ignore" # without trailing slash e.g. /mnt/user/other__remote_mount/or_other_local_folder. Enter 'ignore' to disable LocalFilesShare3="ignore" LocalFilesShare4="ignore" # Add extra commands or filters Command1="--rc" Command2="" Command3="" Command4="" Command5="" Command6="" Command7="" Command8="" CreateBindMount="N" # Y/N. Choose whether to bind traffic to a particular network adapter RCloneMountIP="192.168.1.252" # My unraid IP is 172.30.12.2 so I create another similar IP address NetworkAdapter="eth0" # choose your network adapter. eth0 recommended VirtualIPNumber="2" # creates eth0:x e.g. eth0:1. I create a unique virtual IP addresses for each mount & upload so I can monitor and traffic shape for each of them ####### END SETTINGS ####### ############################################################################### ##### DO NOT EDIT ANYTHING BELOW UNLESS YOU KNOW WHAT YOU ARE DOING ####### ############################################################################### ####### Preparing mount location variables ####### LocalFilesLocation="$LocalFilesShare/$RcloneRemoteName" # Location for local files to be merged with rclone mount RcloneMountLocation="$RcloneMountShare/$RcloneRemoteName" # Location for rclone mount MergerFSMountLocation="$MergerfsMountShare/$RcloneRemoteName" # Rclone data folder location ####### create directories for rclone mount and mergerfs mounts ####### mkdir -p /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName #for script files if [[ $LocalFileShare == 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Not creating local folders as requested." else echo "$(date "+%d.%m.%Y %T") INFO: Creating local folders." eval mkdir -p $LocalFilesLocation/"$MountFolders" fi mkdir -p $RcloneMountLocation mkdir -p $MergerFSMountLocation ####### Check if script is already running ####### echo "$(date "+%d.%m.%Y %T") INFO: *** Starting mount of remote ${RcloneRemoteName}" echo "$(date "+%d.%m.%Y %T") INFO: Checking if this script is already running." if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Exiting script as already running." exit else echo "$(date "+%d.%m.%Y %T") INFO: Script not running - proceeding." touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running fi ####### Create Rclone Mount ####### # Check If Rclone Mount Already Created if [[ -f "$RcloneMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Success ${RcloneRemoteName} remote is already mounted." else echo "$(date "+%d.%m.%Y %T") INFO: Mount not running. Will now mount ${RcloneRemoteName} remote." # Creating mountcheck file in case it doesn't already exist echo "$(date "+%d.%m.%Y %T") INFO: Recreating mountcheck file for ${RcloneRemoteName} remote." touch mountcheck rclone copy mountcheck $RcloneRemoteName: -vv --no-traverse # Check bind option if [[ $CreateBindMount == 'Y' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}" ping -q -c2 $RCloneMountIP > /dev/null # -q quiet, -c number of pings to perform if [ $? -eq 0 ]; then # ping returns exit status 0 if successful echo "$(date "+%d.%m.%Y %T") INFO: *** IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}" else echo "$(date "+%d.%m.%Y %T") INFO: *** Creating IP address ${RCloneMountIP} for remote ${RcloneRemoteName}" ip addr add $RCloneMountIP/24 dev $NetworkAdapter label $NetworkAdapter:$VirtualIPNumber fi echo "$(date "+%d.%m.%Y %T") INFO: *** Created bind mount ${RCloneMountIP} for remote ${RcloneRemoteName}" else RCloneMountIP="" echo "$(date "+%d.%m.%Y %T") INFO: *** Creating mount for remote ${RcloneRemoteName}" fi # create rclone mount rclone mount \ --allow-other \ --buffer-size 256M \ --dir-cache-time 720h \ --drive-chunk-size 512M \ --log-level INFO \ --vfs-read-chunk-size 128M \ --vfs-read-chunk-size-limit off \ --vfs-cache-mode writes \ --bind=$RCloneMountIP \ $RcloneRemoteName: $RcloneMountLocation & # Check if Mount Successful echo "$(date "+%d.%m.%Y %T") INFO: sleeping for 5 seconds" # slight pause to give mount time to finalise sleep 5 echo "$(date "+%d.%m.%Y %T") INFO: continuing..." if [[ -f "$RcloneMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Successful mount of ${RcloneRemoteName} mount." else echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mount failed - please check for problems." rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running exit fi fi ####### Start MergerFS Mount ####### if [[ $MergerfsMountShare == 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Not creating mergerfs mount as requested." else if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount in place." else # check if mergerfs already installed if [[ -f "/bin/mergerfs" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs already installed, proceeding to create mergerfs mount" else # Build mergerfs binary echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs not installed - installing now." mkdir -p /mnt/user/appdata/other/rclone/mergerfs docker run -v /mnt/user/appdata/other/rclone/mergerfs:/build --rm trapexit/mergerfs-static-build mv /mnt/user/appdata/other/rclone/mergerfs/mergerfs /bin # check if mergerfs install successful echo "$(date "+%d.%m.%Y %T") INFO: *sleeping for 5 seconds" sleep 5 if [[ -f "/bin/mergerfs" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs installed successfully, proceeding to create mergerfs mount." else echo "$(date "+%d.%m.%Y %T") ERROR: Mergerfs not installed successfully. Please check for errors. Exiting." rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running exit fi fi # Create mergerfs mount echo "$(date "+%d.%m.%Y %T") INFO: Creating ${RcloneRemoteName} mergerfs mount." # Extra Mergerfs folders if [[ $LocalFilesShare2 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare2} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare2=":$LocalFilesShare2" else LocalFilesShare2="" fi if [[ $LocalFilesShare3 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare3} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare3=":$LocalFilesShare3" else LocalFilesShare3="" fi if [[ $LocalFilesShare4 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare4} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare4=":$LocalFilesShare4" else LocalFilesShare4="" fi # mergerfs mount command mergerfs $LocalFilesLocation:$RcloneMountLocation$LocalFilesShare2$LocalFilesShare3$LocalFilesShare4 $MergerFSMountLocation -o rw,async_read=false,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=partial,dropcacheonclose=true # check if mergerfs mount successful echo "$(date "+%d.%m.%Y %T") INFO: Checking if ${RcloneRemoteName} mergerfs mount created." if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount created." else echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mergerfs mount failed." rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running exit fi fi fi ####### Starting Dockers That Need Mergerfs Mount To Work Properly ####### # only start dockers once if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: dockers already started." else touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started echo "$(date "+%d.%m.%Y %T") INFO: Starting dockers." docker start $DockerStart fi rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running echo "$(date "+%d.%m.%Y %T") INFO: Script complete" exit

Thanks ... moving right along ... I had a failure because Docker wasn't enabled. Did that, and now, I've got an error: FUSE library version: 2.9.7-mergerfs_2.29.0 using FUSE kernel interface version 7.31 'build/mergerfs' -> '/build/mergerfs' 24.05.2020 23:39:55 INFO: *sleeping for 5 seconds 24.05.2020 23:40:00 INFO: Mergerfs installed successfully, proceeding to create mergerfs mount. 24.05.2020 23:40:00 INFO: Creating gdrive_media_vfs mergerfs mount. fuse: bad mount point `movies/gdrive_media_vfs:/mnt/user/mount_rclone/gdrive_media_vfs': No such file or directory 24.05.2020 23:40:00 INFO: Checking if gdrive_media_vfs mergerfs mount created. 24.05.2020 23:40:00 CRITICAL: gdrive_media_vfs mergerfs mount failed. This if from rclone_mount script.

Thanks! i'm still missing something. I went through everything twice. When I tried to configure my remote as a team drive, I get an error: "No team drives found in your account" What could I be missing? When I login to drive.google.com, I have a shared drive that I've created. It's shared with the group that was created during step 2 of the service account setup. I ran "python3 add_to_team_drive.py -d XXXXXX" and added the shared drive ID there.

Thanks. I keep starting this and stopping it. I'm going to go back and re-read everything to make sure I'm not missing anything. Meanwhile... I see a "backup" option what's the difference between that and Copy/Sync mode? My goal is to end up with a mirrored copy of some of my unraid shares (not all of them) on GDrive, and then use that first found option you'd mentioned, to primarily serve files from cloud drive.

I'm still working on getting this setup.. just realized my service_accounts folder doesn't have a sa_gdrive.json file... do i just put the path in rclone? also, confused about team_drive .. I thought the service accounts mean we don't need a team drive? sorry for the newb questions.

Okay - so here's my first actual implementation question. I don't have the appdata folder ... is this because I don't have Docker enabled? I ran my service accounts outside of UnRaid. I'm pretty sure the RClone installation went OK, because if i type in RClone at the command prompt, I get the usage screen. I installed CA user scripts and Rclone via CA Apps. Should I just create the path above? follow-up when I created the service accounts, does that already create a unique client_id? or do I need to create another one?