DZMM

What performance problems have you seen beyond 200k files? I've just checked and I'm at around 70k items for my plex media library tdrive, but I have another tdrive for backups which probably is over 200k files due to all the little system files 06.07.2019 00:50:41 PLEX LIBRARY STATS Media items in Libraries Library = Movies Items = 16147 Library = TV - Adults Items = 34549 Library = TV - Kids Items = 19054

This is for streaming files not backing up. But, yes you could use it for moving your videos to the cloud - I'm not sure about moving photos as I haven't tried that yet.

fixed on github. Sorry about that - not sure how that went missing

I run my mount script on a 5 min cron job */5 * * * * so even if I'm unlucky and the unmount hasn't run yet, it gets fixed on the next run 5 mins later

Run the unmount script at array start rather than shutdown - that's what I do

I started this way and eventually as I got more comfortable I went all in for plex media, and I've also lots of other files to the point where my server is just photos and personal docs. I'm gone from around 44TB in my array to only 16TB, where only around 4TB is permanent. Re seeking, to be my mind it's got better since I started using rclone but it might be because I've got used to it now that I've gone all in. Have you tried experimenting with --buffer-size ?? I think a bigger buffer might help with forward seeking (not backward of course)

This setup supports that - i.e. plex, sonarr, radarr etc all act as normal. Sonarr 'looks' at the mount_unionfs folder so it functions normally i.e. grab->download->move to media folder - all the rclone script behaviour is invisible to sonarr. The upload script 15m delay ensures that Sonarr has finished any post-processing (renaming etc) before the file is uploaded.

Thanks

Hi I'm getting this error in my logs but it still seems to be working: nginx: [alert] detected a LuaJIT version which is not OpenResty's; many optimizations will be disabled and performance will be compromised (see https://github.com/openresty/luajit2 for OpenResty's LuaJIT or, even better, consider using the OpenResty releases from https://openresty.org/en/download.html) nginx: [error] lua_load_resty_core failed to load the resty.core module from https://github.com/openresty/lua-resty-core; ensure you are using an OpenResty release from https://openresty.org/en/download.html (rc: 2, reason: module 'resty.core' not found: no field package.preload['resty.core'] no file './resty/core.lua' no file '/usr/share/luajit-2.1.0-beta3/resty/core.lua' no file '/usr/local/share/lua/5.1/resty/core.lua' no file '/usr/local/share/lua/5.1/resty/core/init.lua' no file '/usr/share/lua/5.1/resty/core.lua' no file '/usr/share/lua/5.1/resty/core/init.lua' no file '/usr/share/lua/common/resty/core.lua' no file '/usr/share/lua/common/resty/core/init.lua' no file './resty/core.so' no file '/usr/local/lib/lua/5.1/resty/core.so' no file '/usr/lib/lua/5.1/resty/core.so' no file '/usr/local/lib/lua/5.1/loadall.so' no file './resty.so' no file '/usr/local/lib/lua/5.1/resty.so' no file '/usr/lib/lua/5.1/resty.so' no file '/usr/local/lib/lua/5.1/loadall.so')

yes - if you copy a file that already exists in mount_rclone I think unionfs overwrites it.

It got pushed back but it looks like the necessary changes to rclone union allowing unionfs to be dropped will be in the next release - 1.4.9 https://github.com/ncw/rclone/milestone/36

Hmm not sure what's going on there. Maybe set the episode to unmonitored in sonarr?

Yes!! If you organise your files in one big folder, when Plex is told there is a change it will scan all files in that folder. Having a folder per movie is more efficient as Plex will only scan that folder for changes

post your logs. I've done full library scans/update metadata and not had problems. The bans only last 24 hours

not sure. I've never done that. I assume if you change the rclone config then it will kill any necessary processes.

it took me a few hours of trial and error to sort the counters. Once you've finished moving your existing content you should be able to upload from just one folder like me if your upload and download speeds are the same (i.e. content is shifted just as fast as it's added) - you just need enough accounts to ensure no individual account will upload more than 750GB/day and mess up the script for up to 24 hours until the ban lifts. I cap my upload scripts at 70MB/s so if I uploaded 24/7 I'd do about 6TB/day so I'd need at a min 6000/750=8 users... but I use about double that just in case something goes wrong.

sorted - Domain A record had the wrong IP address

Yes that'd be the problem. Within gdrive you need to share the teamdrive with each user email and then when you create the remote use that account to create the token. A few tips: 1. Don't use the remote/user account you mount for uploading to make sure your mount always works for playback 2. If you're bulk uploading and you are confident there is no more than 750GB in each of your sub upload folders, I would run your move commands sequentially rather than at all at the same time ONCE A DAY, with the bwlimit set at say 80% of your max. Running multiple rclone move commands at the same time uses up more memory. You'll still get the same max transfer per day, with less ram usage

@neow if all you want to do is backup read this post

Hmm it all looks ok. And for each upload command you've got: rclone move /mnt/user/rclone_upload/tdrive_01_vfs/ tdrive_01_vfs: ......... rclone move /mnt/user/rclone_upload/tdrive_02_vfs/ tdrive_02_vfs: ......... Etc etc

here's what my config looks like: [user1] type = drive client_id = id1 client_secret = secret1 scope = drive team_drive = SAME_TDRIVE token = token1 [user1_vfs] type = crypt remote = user1:crypt filename_encryption = standard directory_name_encryption = true password = pass1 password2 = pass2 [user2] type = drive client_id = id2 client_secret = secret2 scope = drive team_drive = SAME_TDRIVE token = token2 [user2_vfs] type = crypt remote = user2:crypt filename_encryption = standard directory_name_encryption = true password = pass1 password2 = pass2 and my move commands: rclone move /mnt/disks/ud_mx500/rclone_upload/google_vfs user1_vfs: --user-agent="unRAID" -vv --buffer-size 512M --drive-chunk-size 512M --checkers 8 --transfers 6 --exclude .unionfs/** --exclude *fuse_hidden* --exclude *_HIDDEN --exclude .recycle** --exclude *.backup~* --exclude *.partial~* --bwlimit 70M --tpslimit 8 --min-age 10m --fast-list rclone move /mnt/disks/ud_mx500/rclone_upload/google_vfs user2_vfs: --user-agent="unRAID" -vv --buffer-size 512M --drive-chunk-size 512M --checkers 8 --transfers 6 --exclude .unionfs/** --exclude *fuse_hidden* --exclude *_HIDDEN --exclude .recycle** --exclude *.backup~* --exclude *.partial~* --bwlimit 70M --tpslimit 8 --min-age 10m --fast-list

Help please - my cert won't renew. It's been so long since I've had problems with LE I can't work out how to fix: Brought to you by linuxserver.io We gratefully accept donations at: https://www.linuxserver.io/donate/ ------------------------------------- GID/UID ------------------------------------- User uid: 99 User gid: 100 ------------------------------------- [cont-init.d] 10-adduser: exited 0. [cont-init.d] 20-config: executing... [cont-init.d] 20-config: exited 0. [cont-init.d] 30-keygen: executing... using keys found in /config/keys [cont-init.d] 30-keygen: exited 0. [cont-init.d] 50-config: executing... Variables set: PUID=99 PGID=100 TZ=Europe/London URL=MyDOMAIN.com SUBDOMAINS=www,unifi,ha,nextcloud,office,home,heimdall EXTRA_DOMAINS= ONLY_SUBDOMAINS=false DHLEVEL=2048 VALIDATION=http DNSPLUGIN= [email protected] STAGING= 2048 bit DH parameters present SUBDOMAINS entered, processing SUBDOMAINS entered, processing Sub-domains processed are: -d www.MyDOMAIN.com -d unifi.MyDOMAIN.com -d ha.MyDOMAIN.com -d nextcloud.MyDOMAIN.com -d office.MyDOMAIN.com -d home.MyDOMAIN.com -d heimdall.MyDOMAIN.com E-mail address entered: [email protected] http validation is selected Generating new certificate Saving debug log to /var/log/letsencrypt/letsencrypt.log Plugins selected: Authenticator standalone, Installer None Obtaining a new certificate Performing the following challenges: http-01 challenge for MyDOMAIN.com Waiting for verification... Challenge failed for domain MyDOMAIN.com http-01 challenge for MyDOMAIN.com Cleaning up challenges Some challenges have failed. IMPORTANT NOTES: - The following errors were reported by the server: Domain: MyDOMAIN.com Type: connection Detail: Fetching http://MyDOMAIN.com/.well-known/acme-challenge/r_lFlfJYMg2gmnwGbgo-4gqRceo17BLkfJUj8CXnK2A: Timeout during connect (likely firewall problem) To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address. Additionally, please check that your computer has a publicly routable IP address and that no firewalls are preventing the server from communicating with the client. If you're using the webroot plugin, you should also verify that you are serving files from the webroot path you provided. Challenge failed for domain MyDOMAIN.com http-01 challenge for MyDOMAIN.com Cleaning up challenges Some challenges have failed. IMPORTANT NOTES: - The following errors were reported by the server: Domain: MyDOMAIN.com Type: connection Detail: Fetching http://MyDOMAIN.com/.well-known/acme-challenge/r_lFlfJYMg2gmnwGbgo-4gqRceo17BLkfJUj8CXnK2A: Timeout during connect (likely firewall problem) To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address. Additionally, please check that your computer has a publicly routable IP address and that no firewalls are preventing the server from communicating with the client. If you're using the webroot plugin, you should also verify that you are serving files from the webroot path you provided. ERROR: Cert does not exist! Please see the validation error above. The issue may be due to incorrect dns or port forwarding settings. Please fix your settings and recreate the container

glad you got it almost right first time. Re your teamdrive setup, I'm assuming: 1. all files are being loaded to the same teamdrive 2. then the teamdrive is shared with x different users with unique email addresses 3. for each user you've created a unique rclone remote, with unique client IDs each time (all loading to same teamdrive) 4. you've then created a encrypted version of each unique remote 5. you've then created unique rclone_move commands for each user i.e rclone move /mnt/user/rclone_upload/google_vfs USER1_remote: ................ rclone move /mnt/user/rclone_upload/google_vfs USER2_remote: ................ rclone move /mnt/user/rclone_upload/google_vfs USER3_remote: ................

If you put files in mount_unionfs (which you should!) they will behind the scenes really be added to rclone_upload and once uploaded, removed from rclone_upload - all at the same time as always being available and not 'moving' in mount_unionfs. My setup isn't designed for copies or syncs - it's for moving files to gdrive for seamless playback and library management aka plex. If this is what you want, you'll need to start a new thread to get help doing that - or read the rclone help pages which are good. Using my setup to do this is overkill as you would only need to mount if you lose files.

@neow not quite..... think of the folders in this order: 1. mount_rclone are the files that are on gdrive and are accessed via this mount 2. rclone_upload are files on your local server waiting to be uploaded to gdrive. once uploaded they no longer exist locally and are now on gdrive and are accessible via mount_rclone 3. mount_unionfs is a merged or combined folder that shows both files that are in both locations - mount_rclone for gdrive and rclone_upload for local files think of 3. this way - you have already uploaded back to the future I and back to the future II is in the queue to be uploaded: - in mount_rclone you will only see ..../movies/back_to_future_I - in rclone_upload you will see ..../movies/back_to_future_II - in mount_unionfs you will see ..../movies/back_to_future_I AND ..../movies/back_to_future_II that is why you should map all dockers to mount_unionfs as it lets them see files that are on gdrive, plus those haven't been uploaded yet AND hides when files move from rclone_upload to mount_rclone i.e. to mount_unionfs it's a non-event. If you want to free up space on your server - move files to rclone_upload - they will still be playable until they move If you want to backup files, then tweak the upload script to use "rclone sync" rather than "rclone move" and add the folders you want to backup/sync