DZMM

@Thel1988 and @watchmeexplode5 I found an error with the counter which messed it up on the first run. Can you change: else echo "$(date "+%d.%m.%Y %T") INFO: No counter file found for ${RcloneRemoteName}. Creating counter_1." touch /mnt/user/appdata/other/rclone/$RcloneRemoteName/counter_1 fi to: else echo "$(date "+%d.%m.%Y %T") INFO: No counter file found for ${RcloneRemoteName}. Creating counter_1." touch /mnt/user/appdata/other/rclone/$RcloneRemoteName/counter_1 CounterNumber="1" fi Or just use the latest script on github. @watchmeexplode5 I've tested this with the service accounts (copying passwords worked perfectly!) using this config and it all worked perfectly for sa_tdrive1_vfs, sa_tdrive2_vfs etc etc: UseMultipleUploadRemotes="Y" RemoteNumber="16" RcloneUploadRemoteStart="sa_tdrive" RcloneUploadRemoteEnd="_vfs"

@Spladge Excellent! @watchmeexplode5 I have a question about the service account script. It created hundreds of json files named gobbledygook.json. Did you just rename the ones you used to SERVICEACCOUNT01.json, SERVICEACCOUNT02.json ...SERVICEACCOUNTXX.json?

@watchmeexplode5 wow thanks for checking. It seems a weird way to do it, otherwise why as you confirmed would rclone config split out different text when the passwords are the same. Weird! Edit: found answer here. Wish I'd known this before, would have saved me a lot of time 😞 https://forum.rclone.org/t/crypt-remote-generating-different-hash-each-time-for-the-same-password/13154/11

100% certain that you need to run rclone config to create each of the additional remotes and during config enter the same real 'readable' PASSWORD1 and PASSWORD2, and then rclone obscurates them in the config - otherwise your password would be visible in the config file. If you want to test, upload a new file e.g. test.txt to the root of your crypt with new gdrive_counter1_vfs and then see if it appears in your main mount. It should appear obscured in the team drive folder but it won't appear in your mount decrypted as the passwords won't match.

I really don't know why as it works for me (just tested again). Glad you figured it out though. @watchmeexplode5 I was curious and even though I've already got 17 remotes I created my way, this way was definitely quickly. It's nice to know I've got 500 accounts sitting there if I need them! How did you create 100 remotes so quickly though? I'm just checking you didn't cut and paste the same gobbledygook PASSWORD1 and PASSWORD2 into each of the 100 remotes? You still have to create each one individually don't you, so that each remote has a unique gobbledygook string for the password? You still have to enter the actual passwords into rclone config for rclone to obfuscate, right?

The new scripts are a lot easier to use - it's handy for me as I have 3-4 mounts going, so being able to just edit the config section is great. You've got me confused though about editing the CUT - that just extracts the number from the counter_# tracking file that the script creates and should have nothing to do with the remote name. What's the name or format of the multiple upload remotes you're using? It shouldn't matter what they are called as long as you enter the right values for what comes before and after the counter number in the remote name: RcloneUploadRemoteStart="gdrive_counter" # Enter characters before counter in your remote names ## i.e. for gdrive_counter1_vfs, gdrive_counter2_vfs, ...gdrive_counter15_vfs,gdrive_counter16_vfs enter 'gdrive_counter' RcloneUploadRemoteEnd="_vfs" # Enter characters after counter ## i.e. for gdrive_counter1_vfs, gdrive_counter2_vfs, ...gdrive_counter15_vfs,gdrive_counter16_vfs enter '_vfs' so if your remotes are name_media1_vfs, name_media2_vfs etc: RcloneUploadRemoteStart="name_media" RcloneUploadRemoteEnd="_vfs" which creates RcloneUploadRemoteStart+Counter+RcloneUploadRemoteEnd as the remote to use on each run

@Xaero - you're a star, thanks.

I forgot to add. Because you can create multiple teamdrives, this is a good way to give a mate an unlimited 'gdrive' account i.e. create another teamdrive and share it with them

To move files between your gdrive and new teamdrive, the easiest way is to: Stop your current rclone mount + plex, radarr etc - any dockers that need to access the mount Log into gdrive with your 'master' account i.e. one that can access both the gdrive folder and the teamdrive click on 'crypt' in the gdrive folder and use the move command to move the folder to the teamdrive The files will then get moved fairly quickly Adjust your rclone mount and upload scripts to use the new tdrive based remotes It's best to wait until the move is finished before remounting, because rclone might not see any server side changes made after mounting for a while Once the old gdrive folder is empty, start the mount with the new tdrive remote

Update: an update to getting creative with making google accounts to create the additional remotes is to create a service account using this guide and then using as many of the credentials as you need to create additional remotes - rclone guide here I've just made some more updates to the script that will be of interest to any users who have an upload speed > 70Mbps and want to upload more than the 750GB/day limit set by Google (per remote and per user/team drive), or just want to upload without a --bwlimit and not get locked out for 24 hours. The new script now allows daily theoretical uploads/day of nearly 11TB with a Gbps connection. I say theoretically as with my Gbps connection I got max upload speeds to Google of around 700-800Mbps giving a daily potential of around 8TB, but I had other things going on. I probably could have gone faster as I did some tdrive->tdrive transfers last month and rclone was reporting 1.7Gbps. I hadn't shared how I did this before as my script was quite clunky and a couple of us got it working, but I've now managed to make it easier for anyone else to setup in the new scripts. I also didn't share because my old script only worked if you had less than 750GB/day in the upload queue. Otherwise, the script would get stuck for up to 24 hours. Now thanks to the --drive-stop-on-upload-limit command added to rclone 1.5.1 the behaviour is much better - if the upload run exceeds 750GB/day it now stops rather than hammering away at google for up to 24 hours. My script takes advantage of this and uses a different account for the next run i.e. in 5 mins, or whatever cron schedule you've set. Setup now should take a maximum 30-60 mins (stage 3 below) if you need the full 14-15 accounts for a 1Gbps upload. You could just dabble with a few and then add more when needed. E.g. 1 extra account would allow 1.5TB/day which (should) be enough for most users. How It Works 1. Your remote needs to mount a team drive NOT a normal gdrive folder. Create One if you don't have one If you haven't done this yet, creating a team drive is easy and moving the files from your gdrive-->tdrive will be mega fast as you can do it server side using server_side_across_configs = true in your remote settings and this new updated script - - just follow these instructions to do quickly: 2. Share your new team drive with other UNIQUE google accounts Google's 750GB/day quota is not only per remote, but also per user per team drive i.e. if you have 2 people sharing a team drive, they both can upload 750GB each = 1.5TB/day, 4 users = 3TB/day and so on. So, to max out your upload you just need to decide how many users you need accessing the team drive based on how fast your connection is, how much you might upload daily and how long your upload job is scheduled for. E.g. for a 1Gbps connection: - 24x7 upload: 14-15 users (1000/8 x 60 x 60 x 24 = 10.8TB / 0.75TB = 14.4) = 14-15 extra users and remotes - Uploading for 8 hours overnight: 5 users (3.6TB) = 5 extra users and remotes - Script running every 5 mins with no --bwlimit: As many accounts/remotes to cover as much data downloaded UPDATE: I advise creating NOT using your existing mounted remote to upload this way to avoid it getting locked out. Use your existing remote just to mount If you want to add 14-15 google accounts with access to the teamdrive you might have to get a bit creative with finding accounts to invite. I had another google apps domain that helped where I gave those users access, plus I had a few gmail.com accounts as well I could use. 3. Create the extra remotes and corresponding encrypted remotes Because each of the accounts in #2 above have access to the new teamdrive, they can all create mounts to access the extra 750GB/day per account. To do this, create rclone mounts as usual - BUT for the client_ID and client_secret for each remote CREATE AND USE a DIFFERENT google account from #2. This is because each user can only upload 750GB, regardless of which remote did it. For each of your new remotes, use the SAME TEAMDRIVE and the same CRYPT LOCATION. i.e. if your main config looks like this: [gdrive] type = drive client_id = UNIQUE CLIENT_ID client_secret = MATCHING_UNIQUE_SECRET scope = drive team_drive = SAME TEAM DRIVE token = {"access_token":Google_Generated"} server_side_across_configs = true [gdrive_media_vfs] type = crypt remote = gdrive:crypt filename_encryption = standard directory_name_encryption = true password = PASSWORD1 password2 = PASSWORD2 then your first new remote for fast uploading should look like this: [gdrive_counter1] type = drive client_id = UNIQUE CLIENT_ID client_secret = MATCHING_UNIQUE_SECRET scope = drive team_drive = SAME TEAM DRIVE token = {"access_token":Google_Generated"} [gdrive_counter1_vfs] type = crypt remote = gdrive_counter1:crypt filename_encryption = standard directory_name_encryption = true password = PASSWORD1 password2 = PASSWORD2 gdrive_counter1: - Recommended: (so you don't lose track!) make sure you give each unencrypted remote the same name before the number (gdrive_counter) - use a unique CLIENT_ID and SECRET - make sure each remote is using the same TEAM DRIVE - when creating the token using rclone config, remember to use the google account that matches the Client_ID and CLient_secret gdrive_counter1_vfs: - IMPORTANT: Each encrypted remote HAS TO HAVE the same characters before the number (gdrive_counter) OR THE SCRIPT WON'T WORK - IMPORTANT: Each encrypted remote HAS TO HAVE the same characters after the number (_vfs) OR THE SCRIPT WON'T WORK - IMPORTANT: remote needs to be :crypt to ensure files go in the same place - IMPORTANT: PASSWORD1 and PASSWORD2 (i.e. what's entered in rclone config not the scrambled versions) need to be the same as used for gdrive_media_vfs That's it. Once finished, your rclone config should look something like this: [gdrive] type = drive client_id = UNIQUE CLIENT_ID client_secret = MATCHING_UNIQUE_SECRET scope = drive team_drive = SAME TEAM DRIVE token = {"access_token":Google_Generated"} server_side_across_configs = true [gdrive_media_vfs] type = crypt remote = gdrive:crypt filename_encryption = standard directory_name_encryption = true password = PASSWORD1 password2 = PASSWORD2 [gdrive_counter1] type = drive client_id = UNIQUE CLIENT_ID client_secret = MATCHING_UNIQUE_SECRET scope = drive team_drive = SAME TEAM DRIVE token = {"access_token":Google_Generated"} [gdrive_counter1_vfs] type = crypt remote = gdrive_counter1:crypt filename_encryption = standard directory_name_encryption = true password = PASSWORD1 password2 = PASSWORD2 [gdrive_counter2] type = drive client_id = UNIQUE CLIENT_ID client_secret = MATCHING_UNIQUE_SECRET scope = drive team_drive = SAME TEAM DRIVE token = {"access_token":Google_Generated"} [gdrive_counter2_vfs] type = crypt remote = gdrive_counter2:crypt filename_encryption = standard directory_name_encryption = true password = PASSWORD1 password2 = PASSWORD2 . . . . . . . . [gdrive_counter15] type = drive client_id = UNIQUE CLIENT_ID client_secret = MATCHING_UNIQUE_SECRET scope = drive team_drive = SAME TEAM DRIVE token = {"access_token":Google_Generated"} [gdrive_counter15_vfs] type = crypt remote = gdrive_counter15:crypt filename_encryption = standard directory_name_encryption = true password = PASSWORD1 password2 = PASSWORD2 4. Enter Values Into Script Once complete, then just fill in this section in the new upload script: # Use Multiple upload remotes for multiple quotas UseMultipleUploadRemotes="Y" # Y/N. Choose whether want to rotate multiple upload remotes for incresed quota (750GB x number of remotes) RemoteNumber="15" # Integer number of remotes to use. RcloneUploadRemoteStart="gdrive_counter" # Enter characters before counter in your remote names ## i.e. for gdrive_counter1_vfs, gdrive_counter2_vfs, ...gdrive_counter15_vfs,gdrive_counter16_vfs enter 'gdrive_counter' RcloneUploadRemoteEnd="_vfs" # Enter characters after counter ## i.e. for gdrive_counter1_vfs, gdrive_counter2_vfs, ...gdrive_counter15_vfs,gdrive_counter16_vfs enter '_vfs'

I'm trying to do: FoldersToCreate="folder1/subfolder,folder2,folder3,folder4" mkdir -p /mnt/user/public/test/{$FoldersToCreate} which ends up making these folders: - {folder1 - subfolder,folder2,folder3,folder4} I've read up on bracket expansion, but I can't find a solution to my problem. Help please. Thanks in advance.

Sorry about that - didn't test that option. Change: $RcloneMountLocation & to: $RcloneRemoteName: $RcloneMountLocation & I've fixed on github. @Porkie apologies as well

Update: Thanks to inspiration from @senpaibox I've made a major revision this evening to the scripts on github: They are now much easier to setup through the use of configurable variables Much better messaging Upload script has better --bwlimit options allowing daily schedules, faster or slower uploads without worrying about daily quotas (rclone 1.5.1 upwards needed) e.g. you can now do a 30MB/s upload job overnight for 7 hours to use up your quota, rather than a slow 10MB/s trickle over the day. Or, schedule a slow trickle over the day and a max speed upload overnight option to bind individual rclone mounts and uploads to different IPs. I use this to put my mount traffic in a high-priority queue on pfsense, and my uploads in a low If you haven't switched from unionfs to mergerfs I really recommend that you do now and the layout of the new scripts should make it easier to do so. These are now the scripts (except for my upload script which is modified which rotates remotes to upload more than 750GB/day) I'm using myself, so it'll be easier for me to maintain. I've also updated the first two posts in this thread to align with the new scripts. Any teething problems, please let me know.

Thanks for this. I started this thread not just to share, but also to find ways to improve my own scripts. I'm going to incorporate how you've created the variables (will rename some as I don't think you've used the best names) and a few other things today. I'm hoping then you'll be able to support pulls for any improvements your end in the future.

sharing a good script I found that lets you control per tracker when torrents are deleted automatically by qbittorrent. You just need to create a .qman file for each tracker e.g. : { "category": "imported", "tracker": "tracker.torrentseeds.org", "public": { "min_seed_ratio": 1, "max_seed_ratio": 20, "min_seed_time": 1, "max_seed_time": 336, "required_seeders": 1 }, "private": { "min_seed_ratio": 1, "max_seed_ratio": 20, "min_seed_time": 1, "max_seed_time": 336, "required_seeders": 1 }, "delete_files": true } https://github.com/Hundter/qBittorrent-Ratio-Manager

Thanks - I wasn't aware of that command. Saved to useful folder

Thanks. Yes, that all works fine - I've added my own mappings and I use those. I hate tracking down docker image growth - I was hoping the unrar thing might be the problem e.g. files getting expanded in /data. Back to the drawing board....

@binhex my docker image keeps filling up and I'm wondering if it's the inbuilt unrar tool that's doing it. Does this use the /data mapping that's part of the default template? I deleted this as I used my own, but I'm wondering if by deleting it I'm causing files to be added to the image? I've recreated /data to see if this fixes the problem.

@bluesky509 did you work out how to do this? I'm in the same boat

@bryansj you can remove the /downloads exclusion if you want. But, I've read (not tried myself) that seeding from gdrive is a bad idea as it can lead to an API ban because of all the calls. Similarly you can add an exclusion for your 4K content folder to keep it local, and then use gdrive for non-4K, non-seeding content on top of your local 84TB i.e. have access to xxxTBs extra storage. This is what I did initially until I went all in, and uploaded everything and sold my HDDs. I would definitely use it for backups.

@bryansj with only 40Mbps upload to play with, plus seeding activity I'm not sure how much you'll achieve. I would definitely consider replacing crash plan though, and moving non seeding media to the cloud. 10Mbps still equals around 100GB/day which adds up over time e.g. 3TB/mth.

@Tabris just did a bit of quick research as I'm stuck somewhere boring and looks like you need to use the epmfs option: epmfs (existing path, most free space) Of all the drives on which the relative path exists choose the drive with the most free space. This should ensure that mergerfs writes /media_array files to the actual array, and then just exclude /media_array from the upload script

Not a mergerfs expert, but that's the way it works in my script i.e. writes to the first directory in the union. I'm not sure how your media folder is setup, but the way I'd handle your problem is to have the files you want uploading in /google_vfs/movies and the ones you want to keep offline (or is it online??!!) in google_vfs/movies_offline and then add an exclusion for /movies_offline to your upload script. --exclude movies_offline/**

Hmm just tested and I saw the popup this time so it all makes sense. Must have been me - sorry for the false alarm. Loving the new multi-config and the ability to run concurrently.

yeah I was worried I wasn't making sense - I'd just woken up! What I was trying to say (hopefully better this time!) is that when you create a profile, it'd be useful if there was an option to 'use current settings'. I thought this was the behaviour when I'd tweaked all my settings and then got to the last tab and created my first profile 'w10' - I thought it was saving my current settings to w10, rather than creating a new profile alongside the existing default profile. The Default button on the first tab needs to say something like 'create new profile' as that's what it does currently i.e. load blank settings, not the default profile that any initial changes are saved to. i.e. if I on first run make changes and don't create as a new profile then those settings are saved to the 'default profile' but clicking the 'Default' button doesn't load those settings, but resets all. Update: clicking the Default button actually reset my 'default' profile not loaded my defaults so it's really a Reset button. It also applied the reset rather than letting me review, so it wiped out my settings