DZMM

I just add to rclone config - since I learnt that you can copy passwords and somehow the encryption still works, I do almost all my config stuff directly in the file.

No need for new project if the SA group has been added to the respective teamdrives - think of SAs as normal accounts, that don't need credentials/client_ids setting up i.e. bans work the same - on the offending SA. They're good for efficiently handling multiple accounts for rotating etc once they are setup

ok, I see where your confusion is coming from Or, like this if using service accounts: [gdrive] type = drive scope = drive service_account_file = /mnt/user/appdata/other/rclone/service_accounts/sa_gdrive.json team_drive = TEAM DRIVE ID server_side_across_configs = true [gdrive_media_vfs] type = crypt remote = gdrive:crypt filename_encryption = standard directory_name_encryption = true password = PASSWORD1 password2 = PASSWORD2 Fixed the readme - glad someone is reading it!

you don't need to use client_id if you're using service accounts.

@Kaizac try the github read.me https://github.com/BinsonBuzz/unraid_rclone_mount

In the script. Mount your other tdrives as normal, but enter 'ignore' for the mergerfs location, so that you don't get a corresponding mergerfs mount. Then for the merged mount add the extra locations: # OPTIONAL SETTINGS # Add extra paths to mergerfs mount in addition to LocalFilesShare LocalFilesShare2="/mnt/user/mount_rclone/gdrive_media_vfs" LocalFilesShare3="/mnt/user/mount_rclone/backup_vfs" LocalFilesShare4="ignore" Above you can see I've added my music and backup teamdrives to my main plex teamdrive. Then run your upload as usual against this mergerfs mount. I worked out what the goobledygook name was in my crypts e.g. if 'music' is crazy_folder_name, it should match up in both teamdrives if you've used the same passwords. You have to use the encrypted remotes to do all server side: rclone move tdrive:crypt/crazy_folder_name gdrive:crypt/crazy_folder_name --user-agent="transfer" -vv --buffer-size 512M --drive-chunk-size 512M --tpslimit 8 --checkers 8 --transfers 4 --order-by modtime,ascending --exclude *fuse_hidden* --exclude *_HIDDEN --exclude .recycle** --exclude .Recycle.Bin/** --exclude *.backup~* --exclude *.partial~* --drive-stop-on-upload-limit --delete-empty-src-dirs I probably don't need all the options, just copied from the main script. Transfer flies at an insane speed and is over in seconds. You do need --drive-stop-on-upload-limit as it abides by the 750GB/day limit. If you need to move more daily, then rotate service accounts by just repeating the command: rclone move tdrive:crypt/crazy_folder_name gdrive:crypt/crazy_folder_name --drive-service-account-file=$ServiceAccountDirectory/SA1.json rclone move tdrive:crypt/crazy_folder_name gdrive:crypt/crazy_folder_name --drive-service-account-file=$ServiceAccountDirectory/SA2.json rclone move tdrive:crypt/crazy_folder_name gdrive:crypt/crazy_folder_name --drive-service-account-file=$ServiceAccountDirectory/SA3.json etc etc Add as many as you need.

You only need 1 project. SAs are associated with your google account, so they can be shared between teamdrives if you want to. Of the 500 or so I created, I assign 16 to each upload script (sa_tdrive1.json ----> sa_tdrive16.json, sa_cloud1.json ----> sa_cloud16.json etc etc) - don't need that many, but means I've got enough to saturate a gigabit line if I need to. All you have to do is rename the file, so you might as well assign 16 to each script. If you want to reduce the number of scripts, you could do what I've done: 1. I've added the additional rclone mounts as extra mergerfs locations, so that I only have one master mergerfs share for say teamdrive1, td2 etc etc - saves a bit of ram 2. I have one upload moving local files to teamdrive1 - saves a bit of ram and easier to manage bandwidth 3. overnight I do a server side move from td1-->td2, td1-->td3 etc etc for the relevant folders - limited ram and no bandwidth hit as done server-side 4. all files still accessible to mergerfs share in #1 - files are just picked up from their respective rclone mounts, rather than local or the td1 mount

Sure - someone send me a pull request and I'll add

Thanks @trapexit. The worst case of both drives having less than 100GB shouldn't happen as I've got a few TBs free on the slower array 'drive'.

Thanks for spotting that - changed: mergerfs /mnt/disks/ud_mx500/local:/mnt/user/local /mnt/user/mount_mergerfs/local -o rw,async_read=true,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=partial,dropcacheonclose=true,moveonenospc=true,minfreespace=100G While you're here, can I check the command above is doing what I want please. I want files to go to my mx500 SSD as long as there's 100GB free, if not write to the array - /mnt/user/local. Have I got it right? #4 on your github confused me a bit about setting minfreespace to the size of the largest cache drive

No - I'm on 6.8.2 DVB. I changed my mergerfs command to: mergerfs /mnt/disks/ud_mx500/local:/mnt/user/local /mnt/user/mount_mergerfs/local -o rw,async_read=false,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=partial,dropcacheonclose=true,moveonenospc=true,minfreespace=100G and retested. Copy is now going to SSD as desired and now I get: 2. Array-2-SSD (Disk 2--> MX500): 184MB/s 3. Array-2-Mergerfs (Disk 2--> MX500): 182MB/s I think this was a better test than before as my Disk 1 is doing lots of other things, which I think is why there was such a slow down before. Maybe something's changed in 6.8.3??

It's slower for me as well. I did a few tests (just copied a big tar file in ssh) to my 'local' mergerfs: Array-2-Array (Disk 2--> Disk 1): 108MB/s Array-2-SSD (Disk 2--> MX500): 188MB/s Array-2-Mergerfs (Disk 2--> Disk 1): 84MB/s 1 and 3 should be roughly the same. Separately, I need to work out and fix what's wrong with my mergerfs command as #3 was supposed to write to my SSD not my array: mergerfs /mnt/disks/ud_mx500/local:/mnt/user/local /mnt/user/mount_mergerfs/local -o rw,async_read=false,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=lus,cache.files=partial,dropcacheonclose=true,moveonenospc=true,minfreespace=150G

@jrdnlc I would add a log file as a custom command https://rclone.org/docs/#log-file-file to see what's going on. As far as I can see rclone should be deleting the files

Are those the folders you want to see deleted? They will get re-created every 10 mins. If not, then something else is recreating your folders, or your folders aren't setup correctly as rclone deletes source folders after uploading.

What do you have set for MountFolders= in the mount script and do you have it running on a cron job? If so, it'll recreate those folders. Otherwise, I'm out of ideas.

The upload script is uploading files from /mnt/cache/mount_upload

@jrdnlc can you post your script options please

Can you give an example and post your script options as it should be deleting source folders.

@neow here are my Plex mappings. Use similar ones for all dockers

1. Created a 2TB UD pool /mnt/disks/ud_mx500 of 2x1TB SSDs 2. Before my rclone and mergerfs mounts are built etc I create an extra mergerfs mount of #1 and my array only share /mnt/user/local mergerfs /mnt/disks/ud_mx500/local:/mnt/user/local /mnt/user/mount_mergerfs/local -o rw,async_read=false,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=lus,cache.files=partial,dropcacheonclose=true,moveonenospc=true,minfreespace=150G The combination of category.create=lus (least used space - SSD always wins as smaller than the array/drive - not sure which is used in the calc, but it works) and minfreespace=150G (don't store on SSD if less than 150G) seems to work the way I want, with new files going onto the SSD and only onto the array if the SSD is full. The SSD sometimes dips to around 70-80GB free, but never lower. It keeps files off my faster cache nvme drive as the SSDs are fast enough to max out my linespeed both up and down. 3. Then I add /mnt/user/mount_mergerfs/local as my local location to my scripts: # REQUIRED SETTINGS RcloneRemoteName="tdrive_vfs" LocalFilesShare="/mnt/user/mount_mergerfs/local" RcloneMountShare="/mnt/user/mount_rclone" MergerfsMountShare="/mnt/user/mount_mergerfs" DockerStart="duplicati nzbget qbittorrentvpn lazylibrarian radarr radarr-uhd radarr-collections sonarr sonarr-uhd plex ombi tautulli LDAPforPlex letsencrypt organizrv2" MountFolders=\{"downloads/complete,downloads/seeds,documentaries/kids,documentaries/adults,movies_adults_gd,movies_kids_gd,tv_adults_gd,tv_kids_gd,uhd/tv_adults_gd,uhd/tv_kids_gd,uhd/documentaries/kids,uhd/documentaries/adults"\} # OPTIONAL SETTINGS LocalFilesShare2="/mnt/user/mount_rclone/gdrive_media_vfs" LocalFilesShare3="" LocalFilesShare4="" /mnt/user/mount_rclone/gdrive_media_vfs is my rclone mount for my music and photos. I don't add these to the tdrive I use for plex media, as combined it pushes me over the 400k object limit.

I had problems back in the day with /mnt/disks - I use /mnt/user/mount_rclone and /mnt/user/mount_mergerfs. 750GB/user/day upload - 10TB/day I think download. I think you're overthinking things - rclone does not add any extra considerations to your local setup, other than bandwidth and enough storage for local files that are pending upload. For my setup I have made the following choices: 1. plex appdata on an unassigned Nvme - probably overkill, but I want my library browsing to be as fast as possible and the drive was on sale. 2. A mergerfs union of 2 old unassigned 1TB SSDs in a pool and /mnt/user/local - if the SSD pool is full then new nzbget/qbittorrent files get added to the array instead i.e. like a 2nd cache pool. I do this to avoid my new nvme cache drive, to try and avoid 'noisy' writes to a HDD and because I need an SSD to keep up with my download speed. Not sure what's going on there.

@Switchblade HomeAssistant Core is the new name for Home Assistant, and Hass.io is now called Home Assistant. https://www.home-assistant.io/blog/2020/01/29/changing-the-home-assistant-brand/

you can just copy the rclone.conf for that system to /boot/config/plugins/rclone - assuming you are using the unraid rclone plugin.

Thanks @trapexit. I think the message is getting out now and those unlucky enough to have installed the build with issues have probably updated by now.

@Urya see a few posts up about mergerfs build issues and how to fix