watchmeexplode5

@bedpan Check out: https://github.com/xyou365/AutoRclone That will auto-create all the service accounts and download all the .json files. AutoRclone also has a script to add them to your group for the teamdrive. From there you are kinda left to edit your rclone conf by yourself. I'm sure somebody could script it but I just used some keyboard macros for quick adding. Kinda a lazy way but I'm not the best with scripting and it only took about 5-10 minutes to add 100 so I can't really complain.

@DZMM Awesome work on the new changes (and thanks @senpaibox). It makes the setup and updating to your most recent git scripts so much easier. This script just keeps getting better! I ended up using service accounts instead of unique gdrive accounts like @senpaibox hinted at earlier. Couldn't get as creative as you to get a bunch of unique accounts and needing a phone number for creation seemed like a hassle. I've used cloudbox and plexguide prior so I kinda knew the idea behind service accounts and figured it would translate over to your script well. By using service accounts I didn't need to bother creating a bunch of unique accounts and go through the steps of authorizing every single one. With Service Accounts you can add them to a google group and add that single group into your shared drive. For rclone authentication, you just have to reference the .json account in the config. Example: [gdrive_counter1] type = drive scope = drive service_account_file = /mnt/user/appdata/Rclone/accounts/SERVICEACCOUNT01.json team_drive = SAME TEAM DRIVE [gdrive_counter1_vfs] type = crypt remote = gdrive_counter1:/encrypt filename_encryption = standard directory_name_encryption = true password = PASSWORD 1 password2 = PASSWORD 2 ----------------------------------------------------------------------------------------------------------------------------- Initially I had issues regarding the script not properly referencing the counter. (PSA: my knowledge is limited on this subject but I know enough to read the script and get the jist but not enough to fix issues). Rclone was getting fed the wrong updated $RcloneUploadRemoteName when using the counter. After seeing @Thel1988 post if figured I'd see if changing the field from 3 --> 4 would make any difference. Seems like that fixed all the issues and now everything functions as intended. ----------------------------------------------------------------------------------------------------------------------------- Included some logs so hopefully you can track down what was going on: For reference (RcloneRemoteName="gdrive_media_vfs", RcloneUploadRemoteName="gdrive_media_vfs", RcloneUploadRemoteStart="gdrive_counter", RcloneUploadRemoteEnd="_vfs") With: CounterNumber=$(find /mnt/user/appdata/other/rclone/$RcloneRemoteName/ -name 'counter_*' | cut -d"_" -f3) 08.02.2020 22:35:34 INFO: rclone installed successfully - proceeding with upload. vfs/counter /tmp/user.scripts/tmpScripts/New_Rclone_Upload/script: line 85: [[: vfs/counter: division by 0 (error token is "counter") 08.02.2020 22:35:34 INFO: No counter file found for gdrive_media_vfs. Creating counter_1. 08.02.2020 22:35:34 INFO: Adjusted upload remote name to gdrive_countervfs/counter_vfs based on counter vfs/counter. 08.02.2020 22:35:34 INFO: *** Uploading to remote gdrive_media_vfs 2020/02/08 22:35:34 DEBUG : --min-age 10m0s to 2020-02-08 22:25:34.878642852 -0500 EST m=-599.992140160 2020/02/08 22:35:34 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "move" "/mnt/user/local/gdrive_media_vfs" "gdrive_countervfs/counter_vfs:" "--user-agent=gdrive_countervfs/counter_vfs" "-vv" "--buffer-size" "512M" "--drive-chunk-size" "512M" "--tpslimit" "8" "--checkers" "8" "--transfers" "4" "--order-by" "modtime,ascending" "--min-age" "10m" "--exclude" ".Recycle.Bin/**" "--exclude" "downloads/**" "--exclude" "*fuse_hidden*" "--exclude" "*_HIDDEN" "--exclude" ".recycle**" "--exclude" "*.backup~*" "--exclude" "*.partial~*" "--delete-empty-src-dirs" "--drive-stop-on-upload-limit" "--bwlimit" "01:00,off 08:00,off 16:00,off"] 2020/02/08 22:35:34 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf" 2020/02/08 22:35:34 INFO : Starting HTTP transaction limiter: max 8 transactions/s with burst 1 2020/02/08 22:35:34 DEBUG : downloads: Excluded 2020/02/08 22:35:34 INFO : Local file system at /usr/local/emhttp/gdrive_countervfs/counter_vfs:: Waiting for checks to finish 2020/02/08 22:35:34 INFO : Local file system at /usr/local/emhttp/gdrive_countervfs/counter_vfs:: Waiting for transfers to finish 2020/02/08 22:35:34 DEBUG : moviedocumentary/09 - The Nurse Who Loved Me.m4a: Can't move: rename /mnt/user/local/gdrive_media_vfs/moviedocumentary/09 - The Nurse Who Loved Me.m4a /usr/local/emhttp/gdrive_countervfs/counter_vfs:/moviedocumentary/09 - The Nurse Who Loved Me.m4a: invalid cross-device link: trying copy 2020/02/08 22:35:34 DEBUG : moviedocumentary/09 - The Nurse Who Loved Me.m4a: Can't move, switching to copy 2020/02/08 22:35:34 DEBUG : moviedocumentary/09 - The Nurse Who Loved Me.m4a: MD5 = e1434506ef58266cfe03aedd9a6ae72d OK 2020/02/08 22:35:34 INFO : moviedocumentary/09 - The Nurse Who Loved Me.m4a: Copied (new) 2020/02/08 22:35:34 INFO : moviedocumentary/09 - The Nurse Who Loved Me.m4a: Deleted 2020/02/08 22:35:34 DEBUG : moviedocumentary: Removing directory 2020/02/08 22:35:34 DEBUG : Local file system at /mnt/user/local/gdrive_media_vfs: deleted 1 directories 2020/02/08 22:35:34 INFO : Transferred: 20.048M / 20.048 MBytes, 100%, 458.734 MBytes/s, ETA 0s Checks: 2 / 2, 100% Deleted: 1 Transferred: 1 / 1, 100% Elapsed time: 0.0s 2020/02/08 22:35:34 DEBUG : 5 go routines active 2020/02/08 22:35:34 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "move" "/mnt/user/local/gdrive_media_vfs" "gdrive_countervfs/counter_vfs:" "--user-agent=gdrive_countervfs/counter_vfs" "-vv" "--buffer-size" "512M" "--drive-chunk-size" "512M" "--tpslimit" "8" "--checkers" "8" "--transfers" "4" "--order-by" "modtime,ascending" "--min-age" "10m" "--exclude" ".Recycle.Bin/**" "--exclude" "downloads/**" "--exclude" "*fuse_hidden*" "--exclude" "*_HIDDEN" "--exclude" ".recycle**" "--exclude" "*.backup~*" "--exclude" "*.partial~*" "--delete-empty-src-dirs" "--drive-stop-on-upload-limit" "--bwlimit" "01:00,off 08:00,off 16:00,off"] rm: cannot remove '/mnt/user/appdata/other/rclone/gdrive_media_vfs/counter_vfs/counter': No such file or directory /tmp/user.scripts/tmpScripts/New_Rclone_Upload/script: line 170: vfs/counter: division by 0 (error token is "counter") 08.02.2020 22:35:34 INFO: Script complete With: CounterNumber=$(find /mnt/user/appdata/other/rclone/$RcloneRemoteName/ -name 'counter_*' | cut -d"_" -f4) 09.02.2020 11:35:42 INFO: Counter file found for gdrive_media_vfs. 09.02.2020 11:35:42 INFO: Adjusted upload remote name to gdrive_counter1_vfs based on counter 1. 09.02.2020 11:35:42 INFO: *** Uploading to remote gdrive_media_vfs 2020/02/09 11:35:42 DEBUG : --min-age 10m0s to 2020-02-09 11:25:42.791100876 -0500 EST m=-599.991213270 2020/02/09 11:35:42 DEBUG : rclone: Version "v1.51.0" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "move" "/mnt/user/local/gdrive_media_vfs" "gdrive_counter1_vfs:" "--user-agent=gdrive_counter1_vfs" "-vv" "--buffer-size" "512M" "--drive-chunk-size" "512M" "--tpslimit" "8" "--checkers" "8" "--transfers" "4" "--order-by" "modtime,ascending" "--min-age" "10m" "--exclude" ".Recycle.Bin/**" "--exclude" "downloads/**" "--exclude" "*fuse_hidden*" "--exclude" "*_HIDDEN" "--exclude" ".recycle**" "--exclude" "*.backup~*" "--exclude" "*.partial~*" "--delete-empty-src-dirs" "--drive-stop-on-upload-limit" "--bwlimit" "01:00,off 08:00,off 16:00,off"] 2020/02/09 11:35:42 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf" 2020/02/09 11:35:42 INFO : Starting HTTP transaction limiter: max 8 transactions/s with burst 1 2020/02/09 11:35:43 DEBUG : downloads: Excluded 2020/02/09 11:35:44 INFO : Encrypted drive 'gdrive_counter1_vfs:': Waiting for checks to finish 2020/02/09 11:35:44 INFO : Encrypted drive 'gdrive_counter1_vfs:': Waiting for transfers to finish 2020/02/09 11:35:46 INFO : testdir/Alcatraz (45)/cover.jpg: Copied (new) 2020/02/09 11:35:46 INFO : testdir/Alcatraz (45)/cover.jpg: Deleted 2020/02/09 11:35:47 INFO : testdir/Alcatraz (45)/Alcatraz - Brandon Sanderson.mobi: Copied (new) 2020/02/09 11:35:47 INFO : testdir/Alcatraz (45)/Alcatraz - Brandon Sanderson.mobi: Deleted 2020/02/09 11:35:47 INFO : testdir/Alcatraz (45)/Alcatraz - Brandon Sanderson.pdf: Copied (new) 2020/02/09 11:35:47 INFO : testdir/Alcatraz (45)/Alcatraz - Brandon Sanderson.pdf: Deleted 2020/02/09 11:35:47 INFO : testdir/Alcatraz (45)/Alcatraz - Brandon Sanderson.epub: Copied (new) 2020/02/09 11:35:47 INFO : testdir/Alcatraz (45)/Alcatraz - Brandon Sanderson.epub: Deleted 2020/02/09 11:35:48 INFO : testdir/Alcatraz (45)/metadata.opf: Copied (new) 2020/02/09 11:35:48 INFO : testdir/Alcatraz (45)/metadata.opf: Deleted 2020/02/09 11:35:48 INFO : testdir/Alcatraz (45)/Alcatraz - Brandon Sanderson.opf: Copied (new) 2020/02/09 11:35:48 INFO : testdir/Alcatraz (45)/Alcatraz - Brandon Sanderson.opf: Deleted 2020/02/09 11:35:48 DEBUG : testdir/Alcatraz (45): Removing directory 2020/02/09 11:35:48 DEBUG : testdir: Removing directory 2020/02/09 11:35:48 DEBUG : Local file system at /mnt/user/local/gdrive_media_vfs: deleted 2 directories 2020/02/09 11:35:48 INFO : Transferred: 1.695M / 1.695 MBytes, 100%, 368.139 kBytes/s, ETA 0s Checks: 12 / 12, 100% Deleted: 6 Transferred: 6 / 6, 100% Elapsed time: 4.7s 2020/02/09 11:35:48 DEBUG : 14 go routines active 2020/02/09 11:35:48 DEBUG : rclone: Version "v1.51.0" finishing with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "move" "/mnt/user/local/gdrive_media_vfs" "gdrive_counter1_vfs:" "--user-agent=gdrive_counter1_vfs" "-vv" "--buffer-size" "512M" "--drive-chunk-size" "512M" "--tpslimit" "8" "--checkers" "8" "--transfers" "4" "--order-by" "modtime,ascending" "--min-age" "10m" "--exclude" ".Recycle.Bin/**" "--exclude" "downloads/**" "--exclude" "*fuse_hidden*" "--exclude" "*_HIDDEN" "--exclude" ".recycle**" "--exclude" "*.backup~*" "--exclude" "*.partial~*" "--delete-empty-src-dirs" "--drive-stop-on-upload-limit" "--bwlimit" "01:00,off 08:00,off 16:00,off"] 09.02.2020 11:35:48 INFO: Created counter_2 for next upload run. 09.02.2020 11:35:48 INFO: Script complete Again, I don't know what exactly is going on with changing the field but it seems to have fixed my issues. In both cases the counter would be created, however when f3 was used the command would reference $RcloneUploadRemoteName as "gdrive_countervfs/counter_vfs:" regardless of counter value in appdata/other/rclone/$RcloneRemoteName/counter_X When f3 was used the files/folders would be moved to /usr/local/emhttp/gdrive_countervfs/ 🤔 The script would proceed to delete the file from /mnt/local. Nothing would actually be moved to the drive and it was not seen in the merger_fs (because it was moved to /usr/local.....) Sorry for the long post...... I wanted to include enough info so hopefully its helpful ----------------------------------------------------------------------------------------------------------------------------- TL:DR Changing f3 --> f4 fixed my issues with the counter value being interpreted wrong. Service Accounts work fine when using this script. Simply modify the [gdrive_counterX] in rclone conf to reflect the location of the account.json -----------------------------------------------------------------------------------------------------------------------------

@DZMM Hey, I haven't been on here for a bit to see the changes you've made. Just looked over them and wanted to say that all the revisions line up with the fixes I had made to mine. So everything should work smoothly. ____________________________________________ If anybody is on the fence, migration should be error/headache free now I've been running mine 24/7 for 2+ weeks now without a single issue. Much cleaner script and even though I wasn't getting bottlenecks or utilizing hardlinks... More optimized is always a plus in my books (+ a minor bump in pull/push speed is always appreciated). ____________________________________________ And as always -- Thank you so much @DZMM for the work you have done.

@nuhll When I ran the script with the command, the script would finish but due to the docker error it would never pull and build mergerfs. At the end I would have my rclone share mounted but the local and rclone mounts were not merged into my unionfs/mergerfs folder (because mergerfs was unknown due to never having the file pulled in the first place). I'm not sure why yours worked with -it and mine didn't. Mines been working with the arguments removed. I'm also unsure if the mkdir is needed but it doesn't hurt to have it there 😛

@bedpan Somebody correct me if I'm wrong.... but --dir-cache-time is only for how long rclone will keep directory entities cached. It shouldn't affect rclone from picking up newly added content. That variable can be modified via --poll-interval (default is something like 1m). You can see a reference for rclone commands here: https://rclone.org/commands/rclone_mount/

@DZMM I think I figured out my issues. These are the updates I made to your script. Now I'm able to mount and unmount with no issues. When I looked into the errors I was receiving it was due to "the input device is not a TTY" causing the docker run command to not run (and thus never getting mergerfs). I simply removed the flags. I'm not extremely versed in docker so hopefully there is no downside/issues with removing the interactive and TTY flags. Also edited your new mkdir. I was having the file drop in the wrong place which later interfered with moving it to bin. ### Avoids placing *mergerfs file in /mergerfs/mergerfs/*mergerfs file structure ### Line 30: mkdir -p /mnt/user/appdata/other/mergerfs/mergerfs ---> mkdir -p /mnt/user/appdata/other/mergerfs/ ### Minor Fixed ### Line 50: cho "$(date "+%d.%m.%Y %T") INFO: continuing..." ---> echo "$(date "+%d.%m.%Y %T") INFO: continuing..." ### Remove Interactive and TTY flag from Docker Run ### Line 81: docker run -v /mnt/user/appdata/other/mergerfs:/build --rm -it trapexit/mergerfs-static-build ---> docker run -v /mnt/user/appdata/other/mergerfs:/build --rm trapexit/mergerfs-static-build

Huge thank you to @DZMM for all the work you have done on this. Been running without issues for 6+ months with your scripts! I just migrated to mergerfs and with your instructions it's working smoothly. I did have one issue when trying to migrate. I followed your migration guide, using your new scripts from github and replaced any occurance of mount_mergerfs with mount_unionfs. The mount script initially was failing at the mv mergerfs /bin portion due to file/directory not existing. I didn't look into the error much but it appeared the script wasn't running the docker for some reason. I resolved the issue by manually running the commands below via ssh: # Build mergerfs binary and delete old binary as precaution rm /bin/mergerfs # Create Docker docker run -v /mnt/user/appdata/other/mergerfs:/build --rm -it trapexit/mergerfs-static-build After that, all the scripts work great. Again, thanks so much @DZMM for making and maintaining these scripts!

@DZMM Just to clarify, I am running 1.49.5 via the rclone plugin (Not the rclone beta plugin). No issues and mount seems stable (24+ hrs and no drops/errors). The issue described above occurred on the rclone beta build (not sure the version but if I recall it was 1.49.5-XXXXX-beta) If anybody is having issues it might be worth a shot to uninstall the rclone-beta and install the rclone stable (1.49.5) And thank you DZMM for putting the scripts together, they made the install/setup experience a breeze.

@mestep, what plugin are you using (beta or non-beta)? I recently installed rclone to my unraid build. I had previously been using rclone in Ubuntu/windows without issue. I experienced the same issue you described (over this weekend). Mounts and reads fine however, during very high load (plex-scan) the mount would become unresponsive. Directories still available but if I entered an end directory it would freeze my terminal. Only solution was a hard-reset. No real issues noted logs, no excessive cpu use, and plenty of memory still available. I believe the issue I was experiencing was caused by the rclone-beta pluggin and the version it pulled (see link discussed issue): https://forum.rclone.org/t/1-49-4-plex-internal-errors-on-google-drive/12108 When combing my logs I also getting this "failed to get starttokenpage" error. Although a fix has been pushed (1.49.5) - I still seemed to experience the issue on the newest rclone-beta plugin build. After uninstalling rclone-beta and installing the rclone (non-beta) plugin, I am no longer having issues. Under drive load the mount stays responsive and behaves as expected.