June 20, 20206 yr @DZMM Is your upgrade post from unionfs to mergefs on page 46 still correct? Just so i can type it out to sanity check myself, to upgrade i need too. 1.Unmount the drive and finish and current uploads. 2.Copy the new mount script and unmount scripts and replace my current mount script, using ctrl f to find and replace mount_mergerfs to mount_unions (does this also apply to the new upload script?, i still have a lot of data left in rclone_upload left to upload, but no transfer currently in progress) 3. run mount script and adjust upload script to only upload at 1AM every day.
June 21, 20206 yr Author 11 hours ago, Bolagnaise said: @DZMM Is your upgrade post from unionfs to mergefs on page 46 still correct? Just so i can type it out to sanity check myself, to upgrade i need too. 1.Unmount the drive and finish and current uploads. 2.Copy the new mount script and unmount scripts and replace my current mount script, using ctrl f to find and replace mount_mergerfs to mount_unions (does this also apply to the new upload script?, i still have a lot of data left in rclone_upload left to upload, but no transfer currently in progress) 3. run mount script and adjust upload script to only upload at 1AM every day. Probably not up to date. There are comments on the new scripts which make moving easy - just: 1. make sure you haven't got any rclone activity going on - stop old scripts, uploads and any dockers using the mount 2. setup new paths - but /mount_unionfs etc as your mergerfs mount paths etc if that's what you have setup now. Be careful to put your existing paths in 3. choose other script options 4. run scripts and if all ok, launch dockers
June 22, 20206 yr I had to put this down for a bit ... but coming back around... anyone else concerned about that 400K limit on the team drive? I think i'm at already 125K objects on my array. Will it muck things up if you split your stuff into various team drives? say 1 for each media type in your library? 3D movies, Animation, TV Shows, etc? Or is that adding unnecessary complication?
June 22, 20206 yr Hi in my mountfolders, I just want media\tv When I do: MountFolders=\{"media/tv"\} I get 2 folders. 1 media\tv which is correct but also a {media with subfolder tv} How do I stop that?
June 22, 20206 yr Author 1 hour ago, Bjur said: Hi in my mountfolders, I just want media\tv When I do: MountFolders=\{"media/tv"\} I get 2 folders. 1 media\tv which is correct but also a {media with subfolder tv} How do I stop that? MountFolders=\{"downloads/complete,downloads/intermediate,downloads/seeds,movies,tv"\} # comma separated list of folders to create within the mount I guess it fails if you only add one folder. just create folder manually in your mergerfs mount.
June 22, 20206 yr 12 hours ago, BRiT said: That's what many already do, use multiple team drives. Thanks, Can you share some of your organization/folder structure? Also, do you have a bunch of mount scripts and a bunch of upload scripts? Or is it possible to use the extra sections on the script to manage these too? Thank you!
June 22, 20206 yr I dont use this, yet, but DZMM has shared many of his setups and others have said they have separate setup because the number of files, especially for music or personal backups.
June 22, 20206 yr Author 8 hours ago, axeman said: Thanks, Can you share some of your organization/folder structure? Also, do you have a bunch of mount scripts and a bunch of upload scripts? Or is it possible to use the extra sections on the script to manage these too? Thank you! Multiple mounts, one upload and one tidy-up script. @watchmeexplode5 did some testing and performance gets worse as you get closer to the 400k mark, so you'll need to do something like below soon: 1. My folder structure looks something like this: mount_mergerfs/tdrive_vfs/movies mount_mergerfs/tdrive_vfs/music mount_mergerfs/tdrive_vfs/uhd mount_mergerfs/tdrive_vfs/tv_adults mount_mergerfs/tdrive_vfs/tv_kids 2. I created separate tdrives / rclone mounts for some of the bigger folders e.g. mount_rclone/tdrive_vfs/movies mount_rclone/tdrive_vfs/music mount_rclone/tdrive_vfs/uhd mount_rclone/tdrive_vfs/adults_tv for each of those I created a mount script instance where I do NOT create a mergerfs mount 3. I mount each in turn and for the final main mount add the extra tdrive rclone mounts as extra mergerfs folders: ############################################################### ###################### mount tdrive ######################### ############################################################### # REQUIRED SETTINGS RcloneRemoteName="tdrive_vfs" RcloneMountShare="/mnt/user/mount_rclone" LocalFilesShare="/mnt/user/local" MergerfsMountShare="/mnt/user/mount_mergerfs" # OPTIONAL SETTINGS # Add extra paths to mergerfs mount in addition to LocalFilesShare LocalFilesShare2="/mnt/user/mount_rclone/music" LocalFilesShare3="/mnt/user/mount_rclone/uhd" LocalFilesShare4="/mnt/user/mount_rclone/adults_tv" 4. Run the single upload script - everything initially gets moved from /mnt/user/local/tdrive_vfs to the tdrive_vfs teamdrive 5. Overnight I run another script to move files from the folders that are in tdrive_vfs: to the correct teamdrive. You have to work out the encrypted folder names for this to work. Because rclone is moving the files, the mergerfs mount gets updated i.e. it looks to plex etc like they haven't moved #!/bin/bash rclone move tdrive:crypt/music_tdrive_encrypted_folder_name gdrive:crypt/music_tdrive_encrypted_folder_name \ --user-agent="transfer" \ -vv \ --buffer-size 512M \ --drive-chunk-size 512M \ --tpslimit 8 \ --checkers 8 \ --transfers 4 \ --order-by modtime,ascending \ --exclude *fuse_hidden* \ --exclude *_HIDDEN \ --exclude .recycle** \ --exclude .Recycle.Bin/** \ --exclude *.backup~* \ --exclude *.partial~* \ --drive-stop-on-upload-limit \ --delete-empty-src-dirs rclone move tdrive:crypt/tv_tdrive_encrypted_folder_name tdrive_t_adults:crypt/tv_tdrive_encrypted_folder_name \ --user-agent="transfer" \ -vv \ --buffer-size 512M \ --drive-chunk-size 512M \ --tpslimit 8 \ --checkers 8 \ --transfers 4 \ --order-by modtime,ascending \ --exclude *fuse_hidden* \ --exclude *_HIDDEN \ --exclude .recycle** \ --exclude .Recycle.Bin/** \ --exclude *.backup~* \ --exclude *.partial~* \ --drive-stop-on-upload-limit \ --delete-empty-src-dirs rclone move tdrive:crypt/uhd_tdrive_encrypted_folder_name tdrive_uhd:crypt/uhd_tdrive_encrypted_folder_name \ --user-agent="transfer" \ -vv \ --buffer-size 512M \ --drive-chunk-size 512M \ --tpslimit 8 \ --checkers 8 \ --transfers 4 \ --order-by modtime,ascending \ --exclude *fuse_hidden* \ --exclude *_HIDDEN \ --exclude .recycle** \ --exclude .Recycle.Bin/** \ --exclude *.backup~* \ --exclude *.partial~* \ --drive-stop-on-upload-limit \ --delete-empty-src-dirs exit Edited June 22, 20206 yr by DZMM
June 22, 20206 yr 14 minutes ago, DZMM said: Multiple mounts, one upload and one tidy-up script. @watchmeexplode5 did some testing and performance gets worse as you get closer to the 400k mark, so you'll need to do something like below soon: 1. My folder structure looks something like this: mount_mergerfs/tdrive_vfs/movies mount_mergerfs/tdrive_vfs/music mount_mergerfs/tdrive_vfs/uhd mount_mergerfs/tdrive_vfs/tv_adults mount_mergerfs/tdrive_vfs/tv_kids 2. I created separate tdrives / rclone mounts for some of the bigger folders e.g. mount_rclone/tdrive_vfs/movies mount_rclone/tdrive_vfs/music mount_rclone/tdrive_vfs/uhd mount_rclone/tdrive_vfs/adults_tv for each of those I created a mount script instance where I do NOT create a mergerfs mount 3. I mount each in turn and for the final main mount add the extra tdrive rclone mounts as extra mergerfs folders: 4. Run the single upload script - everything initially gets moved from /mnt/user/local/tdrive_vfs to the tdrive_vfs teamdrive 5. Overnight I run another script to move files from the folders that should be in tdrive_vfs: to the correct teamdrive. You have to work out the encrypted folder names for this to work. Because rclone is moving the files, the mergerfs mount gets updated i.e. it looks to plex etc like they haven't moved Thanks. this is excellent. I am going to see if I have the cranial capacity to do this. Turns out a LOT of the issues I had at the start of this endeavor was due to running it in the in the foreground. I saw the user scripts postthat you can force a script to be background only by setting the #backgroundOnly=true directive Maybe add that to the next update of your scripts so that other fools like me don't repeat.
June 24, 20206 yr @DZMM Yup, found out about that performance hit 2 weeks ago and already migrated to a bunch of t drives. I took the same approach with server side move after figuring out the corresponding encrypted name 👍 Currently playing around with a modified version of rclone's backend that rotates sa based on error callbacks. Developer has done a lot of performance tweaks, bypass 10tb dl per day limit on mounts, and circumvents API limit troubles (though that's less of an issue these days). I'd post it but it's fairly unstable and I don't think the developer wants it being tossed around till it's all ready but I'll keep you updated on it
June 24, 20206 yr So I want to move to doing thins with service accounts but of course I hit every hurdle imaginable. I am literally stuck at step 1. This step requires me to input sudo git clone https://github.com/xyou365/AutoRclone && cd AutoRclone && sudo pip3 install -r requirements.txt into the terminal which I do very obediently. Here is my output. ----@Keep:~/AutoRclone# sudo pip3 install -r requirements.txt Collecting oauth2client Using cached oauth2client-4.1.3-py2.py3-none-any.whl (98 kB) Collecting google-api-python-client Using cached google_api_python_client-1.9.3-py3-none-any.whl (59 kB) Collecting progress Using cached progress-1.5.tar.gz (5.8 kB) ERROR: Command errored out with exit status 1: command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-rcm7bl3g/progress/setup.py'"'"'; __file__='"'"'/tmp/pip-install-rcm7bl3g/progress/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-rcm7bl3g/progress/pip-egg-info cwd: /tmp/pip-install-rcm7bl3g/progress/ Complete output (11 lines): Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib64/python3.8/site-packages/setuptools/__init__.py", line 19, in <module> from setuptools.dist import Distribution, Feature File "/usr/lib64/python3.8/site-packages/setuptools/dist.py", line 36, in <module> from setuptools import windows_support File "/usr/lib64/python3.8/site-packages/setuptools/windows_support.py", line 2, in <module> import ctypes File "/usr/lib64/python3.8/ctypes/__init__.py", line 7, in <module> from _ctypes import Union, Structure, Array ImportError: libffi.so.7: cannot open shared object file: No such file or directory ---------------------------------------- ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output. Some smacking around and sending me in the correct direction would be much appreciated. Thank you kind souls.
June 24, 20206 yr Sorry if this has been explained before, it's a long thread... What is the reason for the upload script if I can just place files in the mounted folder and it'll upload on its own? I just use the simple mount command below. Thanks mntpoint="/mnt/disks/gdrive" remoteshare="gsuite:" rclone mount --allow-other --drive-chunk-size 512M --dir-cache-time 5m --log-level INFO --vfs-read-chunk-size 128M --vfs-read-chunk-size-limit off --fast-list --vfs-cache-mode writes --max-read-ahead 2G $remoteshare $mntpoint &
June 24, 20206 yr Author 37 minutes ago, DisposableHero said: Sorry if this has been explained before, it's a long thread... What is the reason for the upload script if I can just place files in the mounted folder and it'll upload on its own? More control over how files are uploaded and when
June 25, 20206 yr Hello! I've been trying to get this set up for a good portion of today. I don't usually upload from my Unraid box so I just want the remote mounted so I can dockerize plex (it's currently running on a VM). I don't need to merge any folders or start/stop any docker containers yet. The script successfully mounts the remote but it always gives me the mount failed message. I can see the mountcheck file in the directory it is checking. Here are my settings: # REQUIRED SETTINGS RcloneRemoteName="GSuiteDrive_Storage" # Name of rclone remote mount WITHOUT ':'. NOTE: Choose your encrypted remote for sensitive data RcloneMountShare="/mnt/user/rclone_mount" # where your rclone remote will be located without trailing slash e.g. /mnt/user/mount_rclone LocalFilesShare="ignore" # location of the local files and MountFolders you want to upload without trailing slash to rclone e.g. /mnt/user/local. Enter 'ignore' to disable MergerfsMountShare="ignore" # location without trailing slash e.g. /mnt/user/mount_mergerfs. Enter 'ignore' to disable DockerStart="" # list of dockers, separated by space, to start once mergerfs mount verified. Remember to disable AUTOSTART for dockers added in docker settings page MountFolders=c\{""\} # comma separated list of folders to create within the mount And script execution: Script location: /tmp/user.scripts/tmpScripts/mount_rclone_gsuite_storage/script Note that closing this window will abort the execution of this script 25.06.2020 12:22:03 INFO: Creating local folders. 25.06.2020 12:22:03 INFO: *** Starting mount of remote GSuiteDrive_Storage 25.06.2020 12:22:03 INFO: Checking if this script is already running. 25.06.2020 12:22:03 INFO: Script not running - proceeding. 25.06.2020 12:22:03 INFO: *** Checking if online 25.06.2020 12:22:10 PASSED: *** Internet online 25.06.2020 12:22:10 INFO: Mount not running. Will now mount GSuiteDrive_Storage remote. 25.06.2020 12:22:10 INFO: Recreating mountcheck file for GSuiteDrive_Storage remote. 2020/06/25 12:22:10 DEBUG : rclone: Version "v1.52.2" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "mountcheck" "GSuiteDrive_Storage:" "-vv" "--no-traverse"] 2020/06/25 12:22:10 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf" 2020/06/25 12:22:10 DEBUG : fs cache: renaming cache item "mountcheck" to be canonical "/" 2020/06/25 12:22:21 DEBUG : mountcheck: Modification times differ by -58.115950247s: 2020-06-25 12:22:10.156950247 -0700 PDT, 2020-06-25 19:21:12.041 +0000 UTC 2020/06/25 12:22:40 INFO : mountcheck: Copied (replaced existing) 2020/06/25 12:22:40 INFO : Transferred: 32 / 32 Bytes, 100%, 1 Bytes/s, ETA 0s Transferred: 1 / 1, 100% Elapsed time: 19.6s 2020/06/25 12:22:40 DEBUG : 6 go routines active 25.06.2020 12:22:40 INFO: *** Creating mount for remote GSuiteDrive_Storage 25.06.2020 12:22:40 INFO: sleeping for 5 seconds 25.06.2020 12:22:50 INFO: continuing... 25.06.2020 12:22:50 CRITICAL: GSuiteDrive_Storage mount failed - please check for problems. Stopping dockers "docker stop" requires at least 1 argument. See 'docker stop --help'. Usage: docker stop [OPTIONS] CONTAINER [CONTAINER...] Stop one or more running containers Edited June 25, 20206 yr by psycho_asylum
June 27, 20206 yr On 6/24/2020 at 2:37 AM, Hypner said: So I want to move to doing thins with service accounts but of course I hit every hurdle imaginable. I am literally stuck at step 1. This step requires me to input sudo git clone https://github.com/xyou365/AutoRclone && cd AutoRclone && sudo pip3 install -r requirements.txt into the terminal which I do very obediently. Here is my output. ----@Keep:~/AutoRclone# sudo pip3 install -r requirements.txt Collecting oauth2client Using cached oauth2client-4.1.3-py2.py3-none-any.whl (98 kB) Collecting google-api-python-client Using cached google_api_python_client-1.9.3-py3-none-any.whl (59 kB) Collecting progress Using cached progress-1.5.tar.gz (5.8 kB) ERROR: Command errored out with exit status 1: command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-rcm7bl3g/progress/setup.py'"'"'; __file__='"'"'/tmp/pip-install-rcm7bl3g/progress/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-rcm7bl3g/progress/pip-egg-info cwd: /tmp/pip-install-rcm7bl3g/progress/ Complete output (11 lines): Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib64/python3.8/site-packages/setuptools/__init__.py", line 19, in <module> from setuptools.dist import Distribution, Feature File "/usr/lib64/python3.8/site-packages/setuptools/dist.py", line 36, in <module> from setuptools import windows_support File "/usr/lib64/python3.8/site-packages/setuptools/windows_support.py", line 2, in <module> import ctypes File "/usr/lib64/python3.8/ctypes/__init__.py", line 7, in <module> from _ctypes import Union, Structure, Array ImportError: libffi.so.7: cannot open shared object file: No such file or directory ---------------------------------------- ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output. Some smacking around and sending me in the correct direction would be much appreciated. Thank you kind souls. Can anyone help me with this? Sorry dont mean to bump something but I didnt see a response. Much appreciated.
June 27, 20206 yr Author 21 minutes ago, Hypner said: Can anyone help me with this? Sorry dont mean to bump something but I didnt see a response. Much appreciated. Sorry I don't really understand python and I think I fluked completing this step as I didn't really understand what I was doing! Hopefully someone else can help. Or, have you tried asking the autorclone author?
June 27, 20206 yr 14 minutes ago, DZMM said: Sorry I don't really understand python and I think I fluked completing this step as I didn't really understand what I was doing! Hopefully someone else can help. Or, have you tried asking the autorclone author? I haven't tried asking the autoclone author. Thank you though. Edited June 27, 20206 yr by Hypner Correction to post
June 28, 20206 yr Hey so while restarting my server today I ran into an issue where after 5-10 minutes or so sometimes longer my mount_mergerfs doesn't show everything. As in it only shows my downloads folder and not anything from mount_rclone. I am able to run the mount script and things work again but I dont know what is stopped the mount_mergfs to stop showing anything outside of whats on local. Let me know what logs are needed. Thank you.
June 28, 20206 yr On 6/22/2020 at 6:14 PM, DZMM said: Multiple mounts, one upload and one tidy-up script. Do you use the same API/project or do you create separate for each TeamDrive ?
June 28, 20206 yr Author 3 hours ago, Hypner said: Hey so while restarting my server today I ran into an issue where after 5-10 minutes or so sometimes longer my mount_mergerfs doesn't show everything. As in it only shows my downloads folder and not anything from mount_rclone. I am able to run the mount script and things work again but I dont know what is stopped the mount_mergfs to stop showing anything outside of whats on local. Let me know what logs are needed. Thank you. Logs from mount script and script options please
June 28, 20206 yr Author 2 hours ago, axeman said: Do you use the same API/project or do you create separate for each TeamDrive ? If you're using SAs you don't need APIs. If you're not, then unique client IDs is recommended
June 29, 20206 yr On 6/27/2020 at 11:32 PM, DZMM said: Logs from mount script and script options please I actually ended up figuring it out. Thanks.
June 29, 20206 yr Alright so I have tried the AutoRclone to create multiple service accounts but just ran into a brick wall and didn't get any support author. Anyway no biggie. I was hoping someone here could help me here instead as it looks like a lot of users are running service accounts. I have created one service account already just need to know what to really do to use them once I create them. I know dumb question but I want to be sure I'm not going to just jack up my current setup.
June 30, 20206 yr Author 11 hours ago, Hypner said: I have created one service account already just need to know what to really do to use them once I create them. I know dumb question but I want to be sure I'm not going to just jack up my current setup. It's all explained on my github
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.