axeman

Thanks - this does seem cleaner than the whole mergerFS route. I'm partially set up now. your setup does sound pretty clever. I can't be 100% cloud, as my internet is awful. So for now, I'm just putting the 4K stuff up there, and keeping lower quality locally. I created Unraid Shares, and then used the mount script you have, and it works. Do you have a different/modified upload script, or use the one from this thread?

Thanks! Here's my union: [union] type = union upstreams = /mnt/user/Videos /mnt/user/media_vfs action_policy = ff create_policy = ff search_policy = all cache_time = 120 That one works OK. If it says /mnt/user/media_vfs:nc I don't see the cloud files. Okay - I am not using it with Dockers, my media import and presentation tools currently run on a VM. Thanks for your time!

Thanks for typing this up @MowMdown with your limited time. I got it started, but have a couple of questions (for you or anyone else here). I don't have any unassigned disks, so I put mine in /mnt/user/media_vfs and /mnt/user/media. I can't seem to see it as a "share" in unraid. Is that why? If i go into MC, I see everything like I'd want to see it. And MC even recognizes the cloud file (it's in green). When I add that, I don't see the cloud files. I too want to avoid accidentally creating files there, because I'm guessing Emby/Plex would write unencrypted metadata, defeating the whole point of encrypting. Can you clarify a bit ? I'm not sure I understand where to do this. Thanks again!

Yes Please! I had too many other things going on and am finally getting back into this thread... Almost feel like holding off again until this gets done, so that I don't have to go back and re-do it.

Are you running it using the "Run Script in Background" option? If not, do it that way.

So forgive me if this is dumb - but I can use the same service account files across all the team drives, right? I should just grant access to the google group that I created? For some reason I get a 404 when trying to setup the rclone remote for a new team drive.

Fancy - I started out with a 14.4 - and it was like magic to get a 28.8 Zoom modem. That sucker even did voicemail. Bit something forget the name of the software. By the time 56K hit my neighborhood, Cable was rolling out 1, 3, and maybe 5mbit lines.

Do you use the same API/project or do you create separate for each TeamDrive ?

Thanks. this is excellent. I am going to see if I have the cranial capacity to do this. Turns out a LOT of the issues I had at the start of this endeavor was due to running it in the in the foreground. I saw the user scripts postthat you can force a script to be background only by setting the #backgroundOnly=true directive Maybe add that to the next update of your scripts so that other fools like me don't repeat.

Thanks, Can you share some of your organization/folder structure? Also, do you have a bunch of mount scripts and a bunch of upload scripts? Or is it possible to use the extra sections on the script to manage these too? Thank you!

I had to put this down for a bit ... but coming back around... anyone else concerned about that 400K limit on the team drive? I think i'm at already 125K objects on my array. Will it muck things up if you split your stuff into various team drives? say 1 for each media type in your library? 3D movies, Animation, TV Shows, etc? Or is that adding unnecessary complication?

Sorry - that's about the end of my knowledge in this arena. @DZMM - when using the scripts with the Backup option, is there a way to mount it so that it doesn't look like this? mount_rclone/gdrive_media_vfs/backup/gdrive_media_vfs/Movies/Movie1/ ?

did you run as sudo?

Not sure this will help you at all, but I just ran all the optional/pre-req stuff in a Linux Mint VM, and just copied over the service accounts files. Only because I wanted to test this thing out without really making too many changes to my UnRaid install. Just a thought.

@DZMM So been trying some tinkering... and noticed something IF I specify LocalFilesShare="/mnt/user/Videos" the script won't upload anything. However, if i create a new folder at: "/mnt/user/local" and do this: LocalFilesShare="/mnt/user/local" and start putting files in there, those files get happily uploaded. Will I need to reorganize my existing videos in to a new folder for the script to upload? Could this be a permission thing? Thanks!

okay .. is that why I can't seem to shutdown the array safely? for some reason that mergerfs never stops, and without a manual kill command, my array refuses to shutdown. there must be something wrong with my mergerFS setup. it's just not mounting anything at all. mind boggling.

Thanks - backup will be awesome for woopsies. I don't see anything at all. I'm starting to think something is wrong with mergerFS ... I don't see it in my Dockers. Should it be there as a docker? Also whenever I reboot the server, on the first run of the mount script, it says mergerFS not installed and proceeds to "install" it I never see anything in Docker. Maybe I can be test dummy for the union backend you guys are talking about above? I'm ready, willing, and (somewhat) able to test for you, with some guidance.

@DZMM I deleted my rclone remotes and recreated - and now all seems to be fine. Every now and then the upload script seems to crash. Not sure why it's happening. two more questions. 1. my rclone mode is set to "sync" and i'm using the backup = 'Y' option... does "sync' imply that if i go to google drive and manually delete something, it'll delete from my array? So i'm guessing I want "copy". 2. when I look at my mergerfs_mount\videos ... i just see the "MountFolders" that I specified. I thought when I go to the mergerfs_mount, I would see all of my shares/files from local array and google drive? Thanks - I was really close to just giving up on this - and am almost seeing the light at the end of the tunnel.

hmm it is running: 26.05.2020 17:23:24 INFO: Creating local folders. 26.05.2020 17:23:24 INFO: *** Starting mount of remote gdrive_media_vfs 26.05.2020 17:23:24 INFO: Checking if this script is already running. 26.05.2020 17:23:24 INFO: Script not running - proceeding. 26.05.2020 17:23:24 INFO: Mount not running. Will now mount gdrive_media_vfs remote. 26.05.2020 17:23:24 INFO: Recreating mountcheck file for gdrive_media_vfs remote. 2020/05/26 17:23:24 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "mountcheck" "gdrive_media_vfs:" "-vv" "--no-traverse"] 2020/05/26 17:23:24 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf" 2020/05/26 17:23:24 DEBUG : mountcheck: Modification times differ by -57m50.100450373s: 2020-05-26 17:23:24.211100396 -0400 EDT, 2020-05-26 16:25:34.110650023 -0400 EDT 2020/05/26 17:23:24 INFO : mountcheck: Copied (replaced existing) 2020/05/26 17:23:24 INFO : Transferred: 32 / 32 Bytes, 100%, 138.821 kBytes/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 0.0s 2020/05/26 17:23:24 DEBUG : 4 go routines active 2020/05/26 17:23:24 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "mountcheck" "gdrive_media_vfs:" "-vv" "--no-traverse"] 26.05.2020 17:23:24 INFO: *** Creating mount for remote gdrive_media_vfs 26.05.2020 17:23:24 INFO: sleeping for 5 seconds 2020/05/26 17:23:24 INFO : Encrypted drive 'gdrive_media_vfs:': poll-interval is not supported by this remote 26.05.2020 17:23:29 INFO: continuing... 26.05.2020 17:23:29 INFO: Successful mount of gdrive_media_vfs mount. 26.05.2020 17:23:29 INFO: Check successful, gdrive_media_vfs mergerfs mount in place. 26.05.2020 17:23:29 INFO: dockers already started. 26.05.2020 17:23:29 INFO: Script complete How do I check that mergerfs is installed? Is it supposed to be a docker? When I check in dockers, I don't see anything (before this, I haven't been using Docker). We're almost there - i can almost taste it!

/appdata/other/rclone has the service_accounts folder. I kept that and deleted everything else. The rest of my settings look OK right? I have my appdata, mount_mergerfs and mount_rclone set to "cache only" that can't cause this, can it? After the deletes, it's not detecting rclone: "26.05.2020 16:26:05 INFO: Checking if rclone installed successfully. 26.05.2020 16:26:05 INFO: rclone not installed - will try again later." I looked and it seems it's just checking for the existence of a file right? or is there more to it?

Mount: # REQUIRED SETTINGS RcloneRemoteName="gdrive_media_vfs" # Name of rclone remote mount WITHOUT ':'. NOTE: Choose your encrypted remote for sensitive data RcloneMountShare="/mnt/user/mount_rclone" # where your rclone remote will be located without trailing slash e.g. /mnt/user/mount_rclone MergerfsMountShare="/mnt/user/mount_mergerfs" # location without trailing slash e.g. /mnt/user/mount_mergerfs. Enter 'ignore' to disable DockerStart="" # list of dockers, separated by space, to start once mergerfs mount verified. Remember to disable AUTOSTART for dockers added in docker settings page LocalFilesShare="/mnt/user/Videos" # location of the local files and MountFolders you want to upload without trailing slash to rclone e.g. /mnt/user/local. Enter 'ignore' to disable MountFolders=\{"downloads/complete,downloads/intermediate,downloads/seeds,movies,tv,Videos/Animation_shorts,Trailers"\} # comma separated list of folders to create within the mount Upload: # REQUIRED SETTINGS RcloneCommand="sync" # choose your rclone command e.g. move, copy, sync RcloneRemoteName="gdrive_media_vfs" # Name of rclone remote mount WITHOUT ':'. RcloneUploadRemoteName="gdrive_media_vfs" # If you have a second remote created for uploads put it here. Otherwise use the same remote as RcloneRemoteName. LocalFilesShare="/mnt/user/Videos" # location of the local files without trailing slash you want to rclone to use RcloneMountShare="/mnt/user/mount_rclone" # where your rclone mount is located without trailing slash e.g. /mnt/user/mount_rclone MinimumAge="15m" # sync files suffix ms|s|m|h|d|w|M|y ModSort="ascending" # "ascending" oldest files first, "descending" newest files first #kept all the rest default, except this stuff: # Is this a backup job BackupJob="Y" # Y/N. Syncs or Copies files from LocalFilesLocation to BackupRemoteLocation, rather than moving from LocalFilesLocation/RcloneRemoteName BackupRemoteLocation="backup" # choose location on mount for deleted sync files BackupRemoteDeletedLocation="backup_deleted" # choose location on mount for deleted sync files BackupRetention="2M" # How long to keep deleted sync files suffix ms|s|m|h|d|w|M|y

@DZMM I'm so close, yet so far... the upload script runs fine, without any errors, but for some reason "nothing" is happening... I see folders and files in this folder: /appdata/other/rclone/remotes/gdrive_media_vfs/gdrive/a24321dsfasdfafadsf132/ but those are physical files on my HD, taking up about 200GB of space.

do you have anything in /mnt/user/appdata/other/rclone/remotes/ ? any other folder?

My unmount wasn't working because of the checker file name... i added a * to the end of the find command. committed in my fork and did a pull request for you. my first ever pull request woo.

Weird I have it run on startup. It doesn't seem to run at all. I wonder if something got messed up in the copy/pasta. Any idea why the array doesn't stop? It seems to get stuck on : Array Stopping•Retry unmounting user share(s)... MergerFS was the culprit. I followed this: