privateer

6.8.3 I had no changes in my software when this started occuring.

Well it worked for about 4 hours and now has gone back to being broken again. ?gdrive_media_vfs in red for me. Noticed when all my dockers turned back off.

I had the same marking. It's like the connection to the gdrive was timing out - had nothing to do with the script running. I tried rebooting a few times, including manually unmounting everything and removing the local folders and remounting, all to no avail. Waited 6 hours and tried again and works like a charm. Must have been on Google's side.

Rebooted unraid earlier today and am now getting an error. Everything loads fine. Mounts correctly on the first attempt. Next two times the script runs it looks good. Third time it runs I get this error: 17.12.2020 21:00:02 INFO: Creating gdrive_media_vfs mergerfs mount. mv: cannot stat '/mnt/user/mount_unionfs/gdrive_media_vfs': Transport endpoint is not connected mkdir: cannot stat '/mnt/user/mount_unionfs/gdrive_media_vfs': Transport endpoint is not connected fuse: bad mount point `/mnt/user/mount_unionfs/gdrive_media_vfs': Socket not connected 17.12.2020 21:00:02 INFO: Checking if gdrive_media_vfs mergerfs mount created. 17.12.2020 21:00:02 CRITICAL: gdrive_media_vfs mergerfs mount failed. Stopping dockers. Any attempt to re-run the script results in the same error. Any idea what's causing this and how to prevent it in the future? EDIT: noticed in my system log the following: `mergerfs[20393]: segfault at 14966c44cff8 ip 000014966de79151 sp 000014966c44cff0 error 6 in mergerfs[14966de46000+b5000]`

Inside /mount_mergerfs/ there is another folder created called gdrive_media_vfs. Why is this here? This looks like something is misconfigured right? That folder is also inside the /rclone_upload/ Is something wrong here?

Thanks - I don't see anything there I missed. I think the issue was when I migrated my data from the My Drive to the Team Drive. I used rclone lsd to look through my old mounts and new mounts, and it looks like the issue might be when I moved the data using the gdrive web ui. Is there a "right" way to move data from the original location ("My Drive") to the new location (Team Drive)?

Up until recently I've been using the OG scripts (no team drive, unionfs etc). I've migrated over to the new way of doing things and had some difficulty, I cobbled together something that worked for a small specific use, but when I start from scratch using the scripts I can't get my whole gdrive mounted. Inside mount_rclone I have /cache/ and /gdrive_media_vfs/. However, inside the gdrive folder I only have a mountcheck and one of the many folders I have in my gdrive. I'm sure I've just made an error somewhere, but I migrated my data from my drive to the team drive, and I'm unable to see those folders and files. Any thoughts? EDIT: I migrated files from "My Drive" to the Team Drive section and those files aren't showing up when I use rclone lsd gdrive_media_vfs. I only see an existing folder I had in the team drive and all the data/folders inside. I migrated using the move folder command on gdrive's web interface. When I use rclone gdrive lsd (unencrypted) it shows 4 encrypted folders...anyone know what's happening or could be happening to those 3 folders?

The setup I'm using right now doesn't involve auto-uploading, but instead manually moving files to be uploaded. Is the ignore setting on both what I'm looking for? Using the OG scripts I just manually dumped things into the upload folder.

#!/bin/bash ###################### #### Mount Script #### ###################### ### Version 0.96.7 ### ###################### ####### EDIT ONLY THESE SETTINGS ####### # INSTRUCTIONS # 1. Change the name of the rclone remote and shares to match your setup # 2. NOTE: enter RcloneRemoteName WITHOUT ':' # 3. Optional: include custom command and bind mount settings # 4. Optional: include extra folders in mergerfs mount # REQUIRED SETTINGS RcloneRemoteName="gdrive_media_vfs" # Name of rclone remote mount WITHOUT ':'. NOTE: Choose your encrypted remote for sensitive data RcloneMountShare="/mnt/user/mount_rclone" # where your rclone remote will be located without trailing slash e.g. /mnt/user/mount_rclone LocalFilesShare="ignore" # location of the local files and MountFolders you want to upload without trailing slash to rclone e.g. /mnt/user/local. Enter 'ignore' to disable MergerfsMountShare="/mnt/user/mount_unionfs" # location without trailing slash e.g. /mnt/user/mount_mergerfs. Enter 'ignore' to disable DockerStart="binhex-rtorrentvpn plex ombi tautulli radarr radarr4k sonarr sonarr4k varken" # list of dockers, separated by space, to start once mergerfs mount verified. Remember to disable AUTOSTART for dockers added in docker settings page MountFolders=\{"downloads/complete,downloads/intermediate,downloads/seeds,movies,tv_shows,4kmovies"\} # comma separated list of folders to create within the mount # Note: Again - remember to NOT use ':' in your remote name above # OPTIONAL SETTINGS # Add extra paths to mergerfs mount in addition to LocalFilesShare LocalFilesShare2="ignore" # without trailing slash e.g. /mnt/user/other__remote_mount/or_other_local_folder. Enter 'ignore' to disable LocalFilesShare3="ignore" LocalFilesShare4="ignore" # Add extra commands or filters Command1="--rc" Command2="" Command3="" Command4="" Command5="" Command6="" Command7="" Command8="" CreateBindMount="N" # Y/N. Choose whether to bind traffic to a particular network adapter RCloneMountIP="192.168.1.123" # My unraid IP is 172.30.12.2 so I create another similar IP address NetworkAdapter="eth0" # choose your network adapter. eth0 recommended VirtualIPNumber="2" # creates eth0:x e.g. eth0:1. I create a unique virtual IP addresses for each mount & upload so I can monitor and traffic shape for each of them ####### END SETTINGS ####### ############################################################################### ##### DO NOT EDIT ANYTHING BELOW UNLESS YOU KNOW WHAT YOU ARE DOING ####### ############################################################################### ####### Preparing mount location variables ####### RcloneMountLocation="$RcloneMountShare/$RcloneRemoteName" # Location for rclone mount LocalFilesLocation="$LocalFilesShare/$RcloneRemoteName" # Location for local files to be merged with rclone mount MergerFSMountLocation="$MergerfsMountShare/$RcloneRemoteName" # Rclone data folder location ####### create directories for rclone mount and mergerfs mounts ####### mkdir -p /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName #for script files if [[ $LocalFileShare == 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Not creating local folders as requested." else echo "$(date "+%d.%m.%Y %T") INFO: Creating local folders." eval mkdir -p $LocalFilesLocation/"$MountFolders" fi mkdir -p $RcloneMountLocation mkdir -p $MergerFSMountLocation ####### Check if script is already running ####### echo "$(date "+%d.%m.%Y %T") INFO: *** Starting mount of remote ${RcloneRemoteName}" echo "$(date "+%d.%m.%Y %T") INFO: Checking if this script is already running." if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Exiting script as already running." exit else echo "$(date "+%d.%m.%Y %T") INFO: Script not running - proceeding." touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running fi ####### Checking have connectivity ####### echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if online" ping -q -c2 google.com > /dev/null # -q quiet, -c number of pings to perform if [ $? -eq 0 ]; then # ping returns exit status 0 if successful echo "$(date "+%d.%m.%Y %T") PASSED: *** Internet online" else echo "$(date "+%d.%m.%Y %T") FAIL: *** No connectivity. Will try again on next run" rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running exit fi ####### Create Rclone Mount ####### # Check If Rclone Mount Already Created if [[ -f "$RcloneMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Success ${RcloneRemoteName} remote is already mounted." else echo "$(date "+%d.%m.%Y %T") INFO: Mount not running. Will now mount ${RcloneRemoteName} remote." # Creating mountcheck file in case it doesn't already exist echo "$(date "+%d.%m.%Y %T") INFO: Recreating mountcheck file for ${RcloneRemoteName} remote." touch mountcheck rclone copy mountcheck $RcloneRemoteName: -vv --no-traverse # Check bind option if [[ $CreateBindMount == 'Y' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}" ping -q -c2 $RCloneMountIP > /dev/null # -q quiet, -c number of pings to perform if [ $? -eq 0 ]; then # ping returns exit status 0 if successful echo "$(date "+%d.%m.%Y %T") INFO: *** IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}" else echo "$(date "+%d.%m.%Y %T") INFO: *** Creating IP address ${RCloneMountIP} for remote ${RcloneRemoteName}" ip addr add $RCloneMountIP/24 dev $NetworkAdapter label $NetworkAdapter:$VirtualIPNumber fi echo "$(date "+%d.%m.%Y %T") INFO: *** Created bind mount ${RCloneMountIP} for remote ${RcloneRemoteName}" else RCloneMountIP="" echo "$(date "+%d.%m.%Y %T") INFO: *** Creating mount for remote ${RcloneRemoteName}" fi # create rclone mount rclone mount \ --allow-other \ --buffer-size 256M \ --dir-cache-time 720h \ --drive-chunk-size 512M \ --log-level INFO \ --vfs-read-chunk-size 128M \ --vfs-read-chunk-size-limit off \ --vfs-cache-mode writes \ --bind=$RCloneMountIP \ $RcloneRemoteName: $RcloneMountLocation & # Check if Mount Successful echo "$(date "+%d.%m.%Y %T") INFO: sleeping for 5 seconds" # slight pause to give mount time to finalise sleep 5 echo "$(date "+%d.%m.%Y %T") INFO: continuing..." if [[ -f "$RcloneMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Successful mount of ${RcloneRemoteName} mount." else echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mount failed - please check for problems. Stopping dockers" docker stop $DockerStart rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running exit fi fi ####### Start MergerFS Mount ####### if [[ $MergerfsMountShare == 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Not creating mergerfs mount as requested." else if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount in place." else # check if mergerfs already installed if [[ -f "/bin/mergerfs" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs already installed, proceeding to create mergerfs mount" else # Build mergerfs binary echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs not installed - installing now." mkdir -p /mnt/user/appdata/other/rclone/mergerfs docker run -v /mnt/user/appdata/other/rclone/mergerfs:/build --rm trapexit/mergerfs-static-build mv /mnt/user/appdata/other/rclone/mergerfs/mergerfs /bin # check if mergerfs install successful echo "$(date "+%d.%m.%Y %T") INFO: *sleeping for 5 seconds" sleep 5 if [[ -f "/bin/mergerfs" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs installed successfully, proceeding to create mergerfs mount." else echo "$(date "+%d.%m.%Y %T") ERROR: Mergerfs not installed successfully. Please check for errors. Exiting." rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running exit fi fi # Create mergerfs mount echo "$(date "+%d.%m.%Y %T") INFO: Creating ${RcloneRemoteName} mergerfs mount." # Extra Mergerfs folders if [[ $LocalFilesShare2 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare2} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare2=":$LocalFilesShare2" else LocalFilesShare2="" fi if [[ $LocalFilesShare3 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare3} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare3=":$LocalFilesShare3" else LocalFilesShare3="" fi if [[ $LocalFilesShare4 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare4} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare4=":$LocalFilesShare4" else LocalFilesShare4="" fi # make sure mergerfs mount point is empty mv $MergerFSMountLocation $LocalFilesLocation mkdir -p $MergerFSMountLocation # mergerfs mount command mergerfs $LocalFilesLocation:$RcloneMountLocation$LocalFilesShare2$LocalFilesShare3$LocalFilesShare4 $MergerFSMountLocation -o rw,async_read=false,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=partial,dropcacheonclose=true # check if mergerfs mount successful echo "$(date "+%d.%m.%Y %T") INFO: Checking if ${RcloneRemoteName} mergerfs mount created." if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount created." else echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mergerfs mount failed. Stopping dockers." docker stop $DockerStart rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running exit fi fi fi ####### Starting Dockers That Need Mergerfs Mount To Work Properly ####### # only start dockers once if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: dockers already started." else touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started echo "$(date "+%d.%m.%Y %T") INFO: Starting dockers." docker start $DockerStart fi rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running echo "$(date "+%d.%m.%Y %T") INFO: Script complete" exit

Any clue what's going on here? I pulled my server back up and get this error - no changes to my scripts from when it was running... 13.10.2020 10:53:56 INFO: Creating local folders. 13.10.2020 10:53:56 INFO: *** Starting mount of remote gdrive_media_vfs 13.10.2020 10:53:56 INFO: Checking if this script is already running. 13.10.2020 10:53:56 INFO: Script not running - proceeding. 13.10.2020 10:53:56 INFO: *** Checking if online 13.10.2020 10:53:57 PASSED: *** Internet online 13.10.2020 10:53:57 INFO: Success gdrive_media_vfs remote is already mounted. 13.10.2020 10:53:57 INFO: Mergerfs already installed, proceeding to create mergerfs mount 13.10.2020 10:53:57 INFO: Creating gdrive_media_vfs mergerfs mount. * ERROR: unable to parse 'branches' - ignore/gdrive_media_vfs:/mnt/user/mount_rclone/gdrive_media_vfs * ERROR: mountpoint not set 13.10.2020 10:53:57 INFO: Checking if gdrive_media_vfs mergerfs mount created. 13.10.2020 10:53:57 CRITICAL: gdrive_media_vfs mergerfs mount failed. Stopping dockers.

Thanks - this helped. I was expecting a file with that name was going to tell rclone how to use the numbered files. I didn't understand that the cycling of remotes didn't require it's own json file. The first time this came out I figured it out quickly, but adding team drives and moving from unionfs turned me into a struggler! Thanks for yours (and everyone else's) patience. It's appreciated. Let me know if you have any thoughts on how I can help pay all the help I've recently been given forward.

Yes I read it, and yes I ran those commands. the files I have are named sa_gdrive_upload[X].json, but there's no sa_gdrive.json file in there. They are in the correct folder and there's 100 of them. This is the error I've been getting: Failed to create file system for "gdrive_media_vfs:": failed to make remote gdrive:"crypt" to wrap: drive: failed when making oauth client: error opening service account credentials file: open /mnt/user/appdata/other/rclone/service_accounts/sa_gdrive.json: no such file or directory

Hopefully progressing onward...but with a new issue. Where should I get a copy of the sa_gdrive.json file? I have the remotes but not sure about that file... I don't know what the SharedTeamDriveSrcID or the SharedTeamDriveDstID are. Is the DstID the folder inside the teamdrive where I'm going to store things (e.g. teamdrivefolder/crypt)? What should go here...wondering if this is why I don't have the .json file.

well that was foolish.

Just tried that route. I get much further, but when I put the auth key in from google I get an error: File "gen_sa_accounts.py", line 23, in <module> download_keys=args.download_keys File "gen_sa_accounts.py", line 175, in serviceaccountfactory with open(token, 'wb') as t: PermissionError: [Errno 13] Permission denied: 'token.pickle'

Newest problem: Can't install everything in the requirements package. I went to install the packages manually, one at a time and I get the following error on several Using cached rsa-4.2.tar.gz (46 kB) ERROR: Command errored out with exit status 1: command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-8jf7sxhg/rsa/setup.py'"'"'; __file__='"'"'/tmp/pip-install-8jf7sxhg/rsa/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-8jf7sxhg/rsa/pip-egg-info cwd: /tmp/pip-install-8jf7sxhg/rsa/ Complete output (11 lines): Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib64/python3.8/site-packages/setuptools/__init__.py", line 19, in <module> from setuptools.dist import Distribution, Feature File "/usr/lib64/python3.8/site-packages/setuptools/dist.py", line 36, in <module> from setuptools import windows_support File "/usr/lib64/python3.8/site-packages/setuptools/windows_support.py", line 2, in <module> import ctypes File "/usr/lib64/python3.8/ctypes/__init__.py", line 7, in <module> from _ctypes import Union, Structure, Array ImportError: libffi.so.7: cannot open shared object file: No such file or directory ---------------------------------------- ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output. Am I missing something else from the nerdpack? I've got python2, python 3, python-pip, and python-setuptools. Thanks for hand holding me along - can't believe I'm struggling so much... Once I'm done I'll condense my posts down to a single post with steps I took to solve, so if others run into these issues they can see how I solved it, and hopefully find the post using the search function.

Looks like it's installed despite the errors. Thanks! (sure I'll have more questions)

Edited my above post as I figured out the dependency for the pip and now generating a new issue. Sorry for all this back and forth. Appreciate you.

python-pip-20.0.2-x86_64-1.txz is installed on my server. Only one I see with pip in it (unless I've missed something). Shouldn't need to reboot or anything after an install right? pip3 returns this error: Traceback (most recent call last): File "/usr/bin/pip3", line 6, in <module> from pkg_resources import load_entry_point ModuleNotFoundError: No module named 'pkg_resources' installed the python-setuptools package as well and now get this error when running 'sudo pip3 install -r requirements.txt' ERROR: Command errored out with exit status 1: command: /usr/bin/python3 -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-4jslat6w/progress/setup.py'"'"'; __file__='"'"'/tmp/pip-install-4jslat6w/progress/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' egg_info --egg-base /tmp/pip-install-4jslat6w/progress/pip-egg-info cwd: /tmp/pip-install-4jslat6w/progress/ Complete output (11 lines): Traceback (most recent call last): File "<string>", line 1, in <module> File "/usr/lib64/python3.8/site-packages/setuptools/__init__.py", line 19, in <module> from setuptools.dist import Distribution, Feature File "/usr/lib64/python3.8/site-packages/setuptools/dist.py", line 36, in <module> from setuptools import windows_support File "/usr/lib64/python3.8/site-packages/setuptools/windows_support.py", line 2, in <module> import ctypes File "/usr/lib64/python3.8/ctypes/__init__.py", line 7, in <module> from _ctypes import Union, Structure, Array ImportError: libffi.so.7: cannot open shared object file: No such file or directory ---------------------------------------- ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

Sorry for the long delay but want to make sure I don't leave a dangling question. I did restore the appdata by using a backup created by using the CA backup plugin. I reinstalled without using the previous apps feature as I didn't know it existed when I did it! I know exactly what corrected the fill-up, and I think now it should be harder to do.

I've been successfully running the original version (unionfs) for a while and finally decided to make the plunge to team drive, service accounts, and mergerfs. While trying to upgrade, I ran the following command as listed on the AutoRclone git page: sudo git clone https://github.com/xyou365/AutoRclone && cd AutoRclone && sudo pip3 install -r requirements.txt The output for this command resulted in an error: sudo: pip3: command not found The rest of the command worked fine. Any idea what's going on here?

Cache drive filled up all the way and corrupted my plex installation. I have CA backup and extracted the tar and have all the files from a working backup back in the correct folder. However, it recognizes this as a different server/plex install. For instance, remote access is disabled, and it shows my old server as inaccessable. Is there anyway to fix this?

They tend to start and freeze etc. right from the beginning with no improvement. Can't make it more than a few mins. I have the same copy local to try and make sure everything is fine with the file etc. Right now I have 16GB of RAM.

I've had some issues streaming 4K remuxes, and have pulled them onto the local drives from gdrive. Any thoughts on the amount of memory / CPU I need (I'm on symmetrical gigabit)? The streams are incredibly choppy and freeze. I didn't see anything that looked like it was getting jammed up, but its been a while since I tested.

New problem popped up last night. Updated to 6.7, rebooted a few times, and ran into a new problem /mount_rclone/google_vfs/[all shares] loads the few files I've put there, but /mount_unionfs/google_vfs/ loads the share folders (movies, tv shows) but doesn't load the contents. I tried deleting the folders in google_vfs but that didn't help. Here is the new error (from the mount script): INFO: mounting rclone vfs. Fatal error: Can not open: /mnt/user/mount_rclone/google_vfs: open /mnt/user/mount_rclone/google_vfs: transport endpoint is not connected CRITICAL: rclone gdrive vfs mount failed - please check for problems. rm: cannot remove '/mnt/user/appdata/other/rclone/rclone_mount_running': No such file or directory Script Finished Mon, 03 Jun 2019 after a reboot I get this error: INFO: mounting rclone vfs. INFO: Check rclone gdrive vfs mount success. fuse: mountpoint is not empty fuse: if you are sure this is safe, use the 'nonempty' mount option CRITICAL: unionfs Remount failed. rm: cannot remove '/mnt/user/appdata/other/rclone/rclone_mount_running': No such file or directory Script Finished Mon, 03 Jun 2019 It looks like I can see the mounted files inside /mount_clone but they aren't appearing in /mount_unionfs.