Kaizac Posted August 12, 2020 Share Posted August 12, 2020 On 8/11/2020 at 5:32 AM, Emilio5639 said: Scripts You're erroring on this part: ####### check if rclone installed ########## echo "$(date "+%d.%m.%Y %T") INFO: Checking if rclone installed successfully." if [[ -f "$RcloneMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: rclone installed successfully - proceeding with upload." else echo "$(date "+%d.%m.%Y %T") INFO: rclone not installed - will try again later." rm /mnt/user/appdata/other/rclone/remotes/$RcloneUploadRemoteName/upload_running$JobName exit So it can't find the $RcloneMountLocation/mountcheck file. RcloneMountLocation is the same as your RcloneShare, so I would start tracing back to there to check if you can find that file and whether all the $'s are correctly in this script. Quote Link to comment
bugster Posted August 18, 2020 Share Posted August 18, 2020 On 8/6/2020 at 3:54 AM, DZMM said: You need to run the unmount script or manually delete the /mnt/cache/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started file - somehow you've got the scripts running slightly out of sync Thank you. Quote Link to comment
Nomad007 Posted August 19, 2020 Share Posted August 19, 2020 Hi guys just reading back the past pages trying to understand how to setup team drives and if there is any benefit for me. Current setup is using the latest scripts. Its working great. Now with teams I understand it allows me to upload more data. Currently though and near future I probably wont be able to surpass the google enforced limits due to the limits enforced by our crappy ISP’s here in my country. Are there any more befits im not seeing. Are there any negatives to sticking to what is described in the current gdrive scripts thanks Quote Link to comment
testdasi Posted August 19, 2020 Share Posted August 19, 2020 1 hour ago, Nomad007 said: Hi guys just reading back the past pages trying to understand how to setup team drives and if there is any benefit for me. Current setup is using the latest scripts. Its working great. Now with teams I understand it allows me to upload more data. Currently though and near future I probably wont be able to surpass the google enforced limits due to the limits enforced by our crappy ISP’s here in my country. Are there any more befits im not seeing. Are there any negatives to sticking to what is described in the current gdrive scripts thanks There are 2 limits. 750 GB / day transfer limit API limit I can't remember the exact number anymore but Google measures how many API call is done every minute and if you exceed it, you will be locked out. I think your ISP probably can only limit your overall transfer and not API call. Using mergerfs (or unionfs) + Team drive allows you to quickly switch to a user if one is locked out (or have multiple users, each for a certain purpose). API call is actually quite hard to exceed (with subtitle app being the common exception) so you may not care but keep that in mind. Quote Link to comment
KeyBoardDabbler Posted August 19, 2020 Share Posted August 19, 2020 (edited) I have a strange issue when trying to move/ copy files into a folder within my mount_mergerfs dir. via windows i get the error "0x8007003A". error "5" via krusader and fails the transfer. Some files transfer and some don't into that dir. The ones that fail are able to move into other dir within the mount_mergerfs share. ***edit** for now it is working. i will report back if the fault persists again. Edited August 20, 2020 by KeyBoardDabbler Quote Link to comment
DZMM Posted August 22, 2020 Author Share Posted August 22, 2020 On 8/19/2020 at 12:14 PM, Nomad007 said: Are there any negatives to sticking to what is described in the current gdrive scripts I'm not sure if gdrive based mounts have the same performance issues, but once a certain number of files are stored the IO performance of tdrives starts to degrade - there's a post in here somewhere from @watchmeexplode5 where he did some testing. One of the advantages of moving to a tdrive, is once you've setup your first one it's easy to create multiple tdrives so that you can spread out your files. Quote Link to comment
Lucka Posted August 27, 2020 Share Posted August 27, 2020 Hey @DZMM, do you provide any kind of support if using your script on Debian instead of Unraid? In any case, here's my issue: I created the 3 scripts into /usr/local/bin, and runned them manually from there for testing purpose, before making cron job. These script, when executed, are showing no error so far, but the folder /mount_mergerfs does not seems to works like intended, I can't create any folder in it, the folders are always deleted, even if they are not empty, and the mount script is supposed to create /mount_mergerfs/movies and /mount_mergerfs/tv right? These folder are there for like 10 seconds and are deleted right away. Any help would be appreciated! Quote Link to comment
DZMM Posted August 27, 2020 Author Share Posted August 27, 2020 17 minutes ago, Lucka said: Hey @DZMM, do you provide any kind of support if using your script on Debian instead of Unraid? In any case, here's my issue: I created the 3 scripts into /usr/local/bin, and runned them manually from there for testing purpose, before making cron job. These script, when executed, are showing no error so far, but the folder /mount_mergerfs does not seems to works like intended, I can't create any folder in it, the folders are always deleted, even if they are not empty, and the mount script is supposed to create /mount_mergerfs/movies and /mount_mergerfs/tv right? These folder are there for like 10 seconds and are deleted right away. Any help would be appreciated! sorry, no idea what's going on there and I've no experience of Debian. Somebody else might help you though! Quote Link to comment
HonkyKONG22 Posted August 29, 2020 Share Posted August 29, 2020 (edited) Do I have to run a mergefs command for my local media to sync with my drive media in the mergefs folder? Currently /mergefs only has gdrive data showing in it. Also, where do I go in Unraid to edit my rclone mount settings to match the front page? Edited August 29, 2020 by HonkyKONG22 Quote Link to comment
HonkyKONG22 Posted August 29, 2020 Share Posted August 29, 2020 Actually I think I figured it out. Moved all my media to the /user/mergerfs/movies,tv,music,etc, files. Have my "local" storage at /user/local/google_vfs/movies,tv,music,etc. Again, cut and pasted my "local" storage to the mergerfs file, it's still local, but now a part of the file system, I have all my apps pointing to the mergerfs file as well, and I am playing stuff from gdrive and from local on plex just fine! Thank you for this script, this is truly amazing! last question, on the Upload Script, the last setting is "minimum age" if I set that to "90d", it will only upload files that have been on my system for over 90 days, right? If that's the case, how does it handle upgrades? My plex tends to see upgrades as new additions. Quote Link to comment
DZMM Posted August 29, 2020 Author Share Posted August 29, 2020 34 minutes ago, HonkyKONG22 said: last question, on the Upload Script, the last setting is "minimum age" if I set that to "90d", it will only upload files that have been on my system for over 90 days, right? If that's the case, how does it handle upgrades? My plex tends to see upgrades as new additions. Yes, files older than 90 days. Mergerfs handles all the upgrades. If it's a local file it just replaces, if it's in the cloud it deletes the cloud version and puts the new copy in the local folder for upload. Quote Link to comment
HonkyKONG22 Posted August 29, 2020 Share Posted August 29, 2020 (edited) Nice, thanks. So upon further inspection, it's doing something strange, I renamed all my paths and folders to match your script so I wouldn't need to edit anything. This is what it should look like, and how I have my file structure setup: /mnt/user -- /local/gdrive_vfs/movies,tv,music /mount_mergerfs/gdrive_vfs/movies,tv,music (where I moved all my media, and have Plex, Radarr, Sonarr, Sabnzbd pointed) /mount_rclone/gdrive_vfs/movies,tv,music However, when I ran the script, it added another /gdrive_vfs folder inside, and moved my media into that one, but it is read only?? I am confused to say the least. now my structure looks like this: /mnt/user -- /local/gdrive_vfs/gdrive_vfs/movies,tv,music /mount_mergerfs/gdrive_vfs/gdrive_vfs(read-only)/movies,tv,music /mount_rclone/gdrive_vfs/gdrive_vfs/movies,tv,music Plex points to it just fine and reads and plays everything perfectly, on gdrive or local storage doesn't matter. since that extra gdrive_vfs folder is read only, I can not get sonarr/radarr to map to it... not sure why this is. edit: I have all my dockers that need media access mapped /user --> /mnt/user in case that makes a difference Edit 2: Restarted my server and sure enough it went and created a gdrive folder in my gdrive folder, wtf... I deleted it from mergerfs manually and everything is working great now. I'll need to dive into the script and figure out why it thinks it "needs" to make that folder vs just seeing it's already there. Edited August 30, 2020 by HonkyKONG22 Quote Link to comment
HonkyKONG22 Posted August 29, 2020 Share Posted August 29, 2020 (edited) For now I fixed it by moving my media from the extra /gdrive_vfs folder to the first /gdrive_vfs folder, and deleted the extra one. it mirrored the structure in /local and /mount_rclone. I'm curious if it recreates that file and moves everything again later when my 750gb limit is expired... One more question: I am about done with creating service accounts, I created them on my fedora workstation and copied them over, renamed them, and I edit my "gdrive" profile so it points to /mnt/user/appdata/other/rclone/service_accounts/sa_gdrive1.json (that's how I named them) I put "yes" to setup as teamdrive, and I get this error: "Failed to configure team drive: listing team drives failed: googleapi: Error 404: File not found: ., notFound" Any ideas? Another Edit: I configured the upload script to use the service accounts anyways, dispite the error in the config, and it seems to be working now, I'll report back, if there's any issues. My setup has to be the most duct taped, roadkill solution there is to getting this to work, but it does work... hopefully Edit: Fixed the remote issue: deleted the original gdrive remote, and recreated it with the service account, now I'm back to uploading. everything seems much smoother, less duct tape, more superglue, don't mean to brag or anything Edited August 30, 2020 by HonkyKONG22 add another question Quote Link to comment
DZMM Posted August 30, 2020 Author Share Posted August 30, 2020 10 hours ago, HonkyKONG22 said: For now I fixed it by moving my media from the extra /gdrive_vfs folder to the first /gdrive_vfs folder, and deleted the extra one. it mirrored the structure in /local and /mount_rclone. I'm curious if it recreates that file and moves everything again later when my 750gb limit is expired... One more question: I am about done with creating service accounts, I created them on my fedora workstation and copied them over, renamed them, and I edit my "gdrive" profile so it points to /mnt/user/appdata/other/rclone/service_accounts/sa_gdrive1.json (that's how I named them) I put "yes" to setup as teamdrive, and I get this error: "Failed to configure team drive: listing team drives failed: googleapi: Error 404: File not found: ., notFound" Any ideas? Another Edit: I configured the upload script to use the service accounts anyways, dispite the error in the config, and it seems to be working now, I'll report back, if there's any issues. My setup has to be the most duct taped, roadkill solution there is to getting this to work, but it does work... hopefully Edit: Fixed the remote issue: deleted the original gdrive remote, and recreated it with the service account, now I'm back to uploading. everything seems much smoother, less duct tape, more superglue, don't mean to brag or anything We can't really help you if you don't post your scripts or your logs Quote Link to comment
HonkyKONG22 Posted August 30, 2020 Share Posted August 30, 2020 (edited) As of now, my only issue, is it keeps making a new /gdrive_vfs folder inside the one already there in /mergerfs. I tried deleting the /gdrive_vfs folder but then the script fails... Mount logs attached- Mount Script: Quote #!/bin/bash ###################### #### Mount Script #### ###################### ### Version 0.96.7 ### ###################### ####### EDIT ONLY THESE SETTINGS ####### # INSTRUCTIONS # 1. Change the name of the rclone remote and shares to match your setup # 2. NOTE: enter RcloneRemoteName WITHOUT ':' # 3. Optional: include custom command and bind mount settings # 4. Optional: include extra folders in mergerfs mount # REQUIRED SETTINGS RcloneRemoteName="gdrive_vfs" # Name of rclone remote mount WITHOUT ':'. NOTE: Choose your encrypted remote for sensitive data RcloneMountShare="/mnt/user/mount_rclone" # where your rclone remote will be located without trailing slash e.g. /mnt/user/mount_rclone LocalFilesShare="/mnt/user/local" # location of the local files and MountFolders you want to upload without trailing slash to rclone e.g. /mnt/user/local. Enter 'ignore' to disable MergerfsMountShare="/mnt/user/mount_mergerfs" # location without trailing slash e.g. /mnt/user/mount_mergerfs. Enter 'ignore' to disable DockerStart="sabnzbd plex sonarr radarr ombi bazarr" # list of dockers, separated by space, to start once mergerfs mount verified. Remember to disable AUTOSTART for dockers added in docker settings page MountFolders=\{"movies,tv,music,youtube"\} # comma separated list of folders to create within the mount # Note: Again - remember to NOT use ':' in your remote name above # OPTIONAL SETTINGS # Add extra paths to mergerfs mount in addition to LocalFilesShare LocalFilesShare2="ignore" # without trailing slash e.g. /mnt/user/other__remote_mount/or_other_local_folder. Enter 'ignore' to disable LocalFilesShare3="ignore" LocalFilesShare4="ignore" # Add extra commands or filters Command1="--rc" Command2="" Command3="" Command4="" Command5="" Command6="" Command7="" Command8="" CreateBindMount="N" # Y/N. Choose whether to bind traffic to a particular network adapter RCloneMountIP="192.168.1.23" # My unraid IP is 172.30.12.2 so I create another similar IP address NetworkAdapter="eth0" # choose your network adapter. eth0 recommended VirtualIPNumber="2" # creates eth0:x e.g. eth0:1. I create a unique virtual IP addresses for each mount & upload so I can monitor and traffic shape for each of them ####### END SETTINGS ####### ############################################################################### ##### DO NOT EDIT ANYTHING BELOW UNLESS YOU KNOW WHAT YOU ARE DOING ####### ############################################################################### ####### Preparing mount location variables ####### RcloneMountLocation="$RcloneMountShare/$RcloneRemoteName" # Location for rclone mount LocalFilesLocation="$LocalFilesShare/$RcloneRemoteName" # Location for local files to be merged with rclone mount MergerFSMountLocation="$MergerfsMountShare/$RcloneRemoteName" # Rclone data folder location ####### create directories for rclone mount and mergerfs mounts ####### mkdir -p /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName #for script files if [[ $LocalFileShare == 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Not creating local folders as requested." else echo "$(date "+%d.%m.%Y %T") INFO: Creating local folders." eval mkdir -p $LocalFilesLocation/"$MountFolders" fi mkdir -p $RcloneMountLocation mkdir -p $MergerFSMountLocation ####### Check if script is already running ####### echo "$(date "+%d.%m.%Y %T") INFO: *** Starting mount of remote ${RcloneRemoteName}" echo "$(date "+%d.%m.%Y %T") INFO: Checking if this script is already running." if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Exiting script as already running." exit else echo "$(date "+%d.%m.%Y %T") INFO: Script not running - proceeding." touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running fi ####### Checking have connectivity ####### echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if online" ping -q -c2 google.com > /dev/null # -q quiet, -c number of pings to perform if [ $? -eq 0 ]; then # ping returns exit status 0 if successful echo "$(date "+%d.%m.%Y %T") PASSED: *** Internet online" else echo "$(date "+%d.%m.%Y %T") FAIL: *** No connectivity. Will try again on next run" rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running exit fi ####### Create Rclone Mount ####### # Check If Rclone Mount Already Created if [[ -f "$RcloneMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Success ${RcloneRemoteName} remote is already mounted." else echo "$(date "+%d.%m.%Y %T") INFO: Mount not running. Will now mount ${RcloneRemoteName} remote." # Creating mountcheck file in case it doesn't already exist echo "$(date "+%d.%m.%Y %T") INFO: Recreating mountcheck file for ${RcloneRemoteName} remote." touch mountcheck rclone copy mountcheck $RcloneRemoteName: -vv --no-traverse # Check bind option if [[ $CreateBindMount == 'Y' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}" ping -q -c2 $RCloneMountIP > /dev/null # -q quiet, -c number of pings to perform if [ $? -eq 0 ]; then # ping returns exit status 0 if successful echo "$(date "+%d.%m.%Y %T") INFO: *** IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}" else echo "$(date "+%d.%m.%Y %T") INFO: *** Creating IP address ${RCloneMountIP} for remote ${RcloneRemoteName}" ip addr add $RCloneMountIP/24 dev $NetworkAdapter label $NetworkAdapter:$VirtualIPNumber fi echo "$(date "+%d.%m.%Y %T") INFO: *** Created bind mount ${RCloneMountIP} for remote ${RcloneRemoteName}" else RCloneMountIP="" echo "$(date "+%d.%m.%Y %T") INFO: *** Creating mount for remote ${RcloneRemoteName}" fi # create rclone mount rclone mount \ --allow-other \ --buffer-size 256M \ --dir-cache-time 720h \ --drive-chunk-size 512M \ --log-level INFO \ --vfs-read-chunk-size 128M \ --vfs-read-chunk-size-limit off \ --vfs-cache-mode writes \ --bind=$RCloneMountIP \ $RcloneRemoteName: $RcloneMountLocation & # Check if Mount Successful echo "$(date "+%d.%m.%Y %T") INFO: sleeping for 5 seconds" # slight pause to give mount time to finalise sleep 5 echo "$(date "+%d.%m.%Y %T") INFO: continuing..." if [[ -f "$RcloneMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Successful mount of ${RcloneRemoteName} mount." else echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mount failed - please check for problems. Stopping dockers" docker stop $DockerStart rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running exit fi fi ####### Start MergerFS Mount ####### if [[ $MergerfsMountShare == 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Not creating mergerfs mount as requested." else if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount in place." else # check if mergerfs already installed if [[ -f "/bin/mergerfs" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs already installed, proceeding to create mergerfs mount" else # Build mergerfs binary echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs not installed - installing now." mkdir -p /mnt/user/appdata/other/rclone/mergerfs docker run -v /mnt/user/appdata/other/rclone/mergerfs:/build --rm trapexit/mergerfs-static-build mv /mnt/user/appdata/other/rclone/mergerfs/mergerfs /bin # check if mergerfs install successful echo "$(date "+%d.%m.%Y %T") INFO: *sleeping for 5 seconds" sleep 5 if [[ -f "/bin/mergerfs" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs installed successfully, proceeding to create mergerfs mount." else echo "$(date "+%d.%m.%Y %T") ERROR: Mergerfs not installed successfully. Please check for errors. Exiting." rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running exit fi fi # Create mergerfs mount echo "$(date "+%d.%m.%Y %T") INFO: Creating ${RcloneRemoteName} mergerfs mount." # Extra Mergerfs folders if [[ $LocalFilesShare2 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare2} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare2=":$LocalFilesShare2" else LocalFilesShare2="" fi if [[ $LocalFilesShare3 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare3} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare3=":$LocalFilesShare3" else LocalFilesShare3="" fi if [[ $LocalFilesShare4 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare4} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare4=":$LocalFilesShare4" else LocalFilesShare4="" fi # make sure mergerfs mount point is empty mv $MergerFSMountLocation $LocalFilesLocation mkdir -p $MergerFSMountLocation # mergerfs mount command mergerfs $LocalFilesLocation:$RcloneMountLocation$LocalFilesShare2$LocalFilesShare3$LocalFilesShare4 $MergerFSMountLocation -o rw,async_read=false,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=partial,dropcacheonclose=true # check if mergerfs mount successful echo "$(date "+%d.%m.%Y %T") INFO: Checking if ${RcloneRemoteName} mergerfs mount created." if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount created." else echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mergerfs mount failed. Stopping dockers." docker stop $DockerStart rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running exit fi fi fi ####### Starting Dockers That Need Mergerfs Mount To Work Properly ####### # only start dockers once if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: dockers already started." else touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started echo "$(date "+%d.%m.%Y %T") INFO: Starting dockers." docker start $DockerStart fi rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running echo "$(date "+%d.%m.%Y %T") INFO: Script complete" exit log.txt Edited August 30, 2020 by HonkyKONG22 Quote Link to comment
DZMM Posted August 30, 2020 Author Share Posted August 30, 2020 27 minutes ago, HonkyKONG22 said: I tried deleting the /gdrive_vfs folder but then the script fails... Deleting a folder shouldn't make the script fail as it's designed to create all the folders it needs. What I would do is stop the relevant dockers, stop the script and manually unmount everything (fusermount /mnt/user/mount_rclone/gdrive_vfs and fusermount /mnt/user/mount_mergerfs/gdrive_vfs) and eyeball each folder to make sure they are empty. I would then manually run just the rclone mount command as I think when you were trying to get up and running, you created another gdrive_vfs folder by mistake that you don't need - the script isn't doing it. If so, delete it. rclone mount gdrive_vfs: \mnt\user\mount_rclone Quote Link to comment
HonkyKONG22 Posted August 30, 2020 Share Posted August 30, 2020 (edited) I cleared all the folders, so it looked like this: /local/gdrive_vfs/movies,tv,music /mount_mergerfs/ /mount_rclone/ ran the mount command, and it didn't work, just hung in the terminal and never completed after about an hour of waiting... ran the mount script, and once again, it created: /local/gdrive_vfs | /gdrive_vfs | /music,tv,movies /mount_mergerfs/gdrive_vfs | /movies,tv,music | /gdrive_vfs(read-only) /mount_rclone/gdrive_vfs | /movies,tv,music | /gdrive_vfs the /movies,tv,music folders get created in the second /gdrive_vfs folders if i run the upload script without deleting those second folders first. Edited August 30, 2020 by HonkyKONG22 Quote Link to comment
DZMM Posted August 31, 2020 Author Share Posted August 31, 2020 17 hours ago, HonkyKONG22 said: I cleared all the folders, so it looked like this: /local/gdrive_vfs/movies,tv,music /mount_mergerfs/ /mount_rclone/ ran the mount command, and it didn't work, just hung in the terminal and never completed after about an hour of waiting... ran the mount script, and once again, it created: /local/gdrive_vfs | /gdrive_vfs | /music,tv,movies /mount_mergerfs/gdrive_vfs | /movies,tv,music | /gdrive_vfs(read-only) /mount_rclone/gdrive_vfs | /movies,tv,music | /gdrive_vfs the /movies,tv,music folders get created in the second /gdrive_vfs folders if i run the upload script without deleting those second folders first. Just to be clear, is it creating /mount_mergerfs/gdrive_vfs/music,tv,movies or 3 folders - /mount_mergerfs/gdrive_vfs/music, /mount_mergerfs/gdrive_vfs/tv and /mount_mergerfs/gdrive_vfs/movies? I'm suspecting that you have a /gdrive_vfs folder in your gdrive when you 'shouldn't' which is creating the other folder. Can you post your rclone config, without any keys/passwords please. Quote Link to comment
Lucka Posted September 1, 2020 Share Posted September 1, 2020 I'm running the Mount script every 15 minutes, and the Upload script every 10 minutes. But after a certain time, the Upload script won't work anymore, I need to use the clean/unmount and then mount again. Any fix for that? Quote Link to comment
DZMM Posted September 1, 2020 Author Share Posted September 1, 2020 1 hour ago, Lucka said: I'm running the Mount script every 15 minutes, and the Upload script every 10 minutes. But after a certain time, the Upload script won't work anymore, I need to use the clean/unmount and then mount again. Any fix for that? logs Quote Link to comment
sauso Posted September 2, 2020 Share Posted September 2, 2020 Guys thought I might post a problem I was having. I spun up a GCP server to download some backlog content quickly. Basically i wrote SAB post processing script to upload the file to my gdrive downloads directory. As the gdrive is part of my mergerfs location locally it should just move to the correct folder and rename. Well this wasn't happening for some reason. Turns out the version of Mono that binhex-radarr is using is 6.4 which has a bug that prevents renaming and moving. What it was doing was downloading each file locally rather than keeping it in the mount. (not ideal if you are downloading old content and want to download it direct to cloud) Upgrading Mono in the binhex-radarr fixed it and it now works a treat. I've asked binhex if we can upgrade mono in the container so this works correctly. Quote Link to comment
sauso Posted September 2, 2020 Share Posted September 2, 2020 Just a follow up that Binhex has updated the base image to include the latest version of Mono. Issue is resolved. Quote Link to comment
arizonaslim Posted September 2, 2020 Share Posted September 2, 2020 I keep getting this error when i run the rclone_mount script. Ive kept all naming conventions the same as in the guide Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_mount/log.txt 02.09.2020 14:07:30 INFO: Creating local folders. 02.09.2020 14:07:30 INFO: *** Starting mount of remote gdrive_vfs 02.09.2020 14:07:30 INFO: Checking if this script is already running. 02.09.2020 14:07:30 INFO: Script not running - proceeding. 02.09.2020 14:07:30 INFO: *** Checking if online 02.09.2020 14:07:32 PASSED: *** Internet online 02.09.2020 14:07:32 INFO: Mount not running. Will now mount gdrive_vfs remote. 02.09.2020 14:07:32 INFO: Recreating mountcheck file for gdrive_vfs remote. 2020/09/02 14:07:32 DEBUG : rclone: Version "v1.53.0" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "mountcheck" "gdrive_vfs:" "-vv" "--no-traverse"] 2020/09/02 14:07:32 DEBUG : Creating backend with remote "mountcheck" 2020/09/02 14:07:32 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf" 2020/09/02 14:07:32 DEBUG : fs cache: adding new entry for parent of "mountcheck", "/usr/local/emhttp" 2020/09/02 14:07:32 DEBUG : Creating backend with remote "gdrive_vfs:" 2020/09/02 14:07:32 Failed to create file system for "gdrive_vfs:": didn't find section in config file 02.09.2020 14:07:32 INFO: *** Creating mount for remote gdrive_vfs 02.09.2020 14:07:32 INFO: sleeping for 5 seconds 2020/09/02 14:07:32 Failed to create file system for "gdrive_vfs:": didn't find section in config file 02.09.2020 14:07:37 INFO: continuing... 02.09.2020 14:07:37 CRITICAL: gdrive_vfs mount failed - please check for problems. Stopping dockers binhex-sabnzbd binhex-radarr binhex-sonarr plex tautulli Script Finished Sep 02, 2020 14:09.53 Heres my mout script #!/bin/bash ###################### #### Mount Script #### ###################### ### Version 0.96.7 ### ###################### ####### EDIT ONLY THESE SETTINGS ####### # INSTRUCTIONS # 1. Change the name of the rclone remote and shares to match your setup # 2. NOTE: enter RcloneRemoteName WITHOUT ':' # 3. Optional: include custom command and bind mount settings # 4. Optional: include extra folders in mergerfs mount # REQUIRED SETTINGS RcloneRemoteName="gdrive_vfs" # Name of rclone remote mount WITHOUT ':'. NOTE: Choose your encrypted remote for sensitive data RcloneMountShare="/mnt/user/mount_rclone" # where your rclone remote will be located without trailing slash e.g. /mnt/user/mount_rclone LocalFilesShare="/mnt/user/local" # location of the local files and MountFolders you want to upload without trailing slash to rclone e.g. /mnt/user/local. Enter 'ignore' to disable MergerfsMountShare="/mnt/user/mount_mergerfs" # location without trailing slash e.g. /mnt/user/mount_mergerfs. Enter 'ignore' to disable DockerStart="binhex-sabnzbd binhex-radarr binhex-sonarr plex tautulli" # list of dockers, separated by space, to start once mergerfs mount verified. Remember to disable AUTOSTART for dockers added in docker settings page MountFolders=\{"downloads/complete,downloads/intermediate,downloads/seeds,movies,tv"\} # comma separated list of folders to create within the mount # Note: Again - remember to NOT use ':' in your remote name above # OPTIONAL SETTINGS # Add extra paths to mergerfs mount in addition to LocalFilesShare LocalFilesShare2="ignore" # without trailing slash e.g. /mnt/user/other__remote_mount/or_other_local_folder. Enter 'ignore' to disable LocalFilesShare3="ignore" LocalFilesShare4="ignore" # Add extra commands or filters Command1="--rc" Command2="" Command3="" Command4="" Command5="" Command6="" Command7="" Command8="" CreateBindMount="N" # Y/N. Choose whether to bind traffic to a particular network adapter RCloneMountIP="192.168.1.252" # My unraid IP is 172.30.12.2 so I create another similar IP address NetworkAdapter="eth0" # choose your network adapter. eth0 recommended VirtualIPNumber="2" # creates eth0:x e.g. eth0:1. I create a unique virtual IP addresses for each mount & upload so I can monitor and traffic shape for each of them ####### END SETTINGS ####### ############################################################################### ##### DO NOT EDIT ANYTHING BELOW UNLESS YOU KNOW WHAT YOU ARE DOING ####### ############################################################################### ####### Preparing mount location variables ####### RcloneMountLocation="$RcloneMountShare/$RcloneRemoteName" # Location for rclone mount LocalFilesLocation="$LocalFilesShare/$RcloneRemoteName" # Location for local files to be merged with rclone mount MergerFSMountLocation="$MergerfsMountShare/$RcloneRemoteName" # Rclone data folder location ####### create directories for rclone mount and mergerfs mounts ####### mkdir -p /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName #for script files if [[ $LocalFileShare == 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Not creating local folders as requested." else echo "$(date "+%d.%m.%Y %T") INFO: Creating local folders." eval mkdir -p $LocalFilesLocation/"$MountFolders" fi mkdir -p $RcloneMountLocation mkdir -p $MergerFSMountLocation ####### Check if script is already running ####### echo "$(date "+%d.%m.%Y %T") INFO: *** Starting mount of remote ${RcloneRemoteName}" echo "$(date "+%d.%m.%Y %T") INFO: Checking if this script is already running." if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Exiting script as already running." exit else echo "$(date "+%d.%m.%Y %T") INFO: Script not running - proceeding." touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running fi ####### Checking have connectivity ####### echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if online" ping -q -c2 google.com > /dev/null # -q quiet, -c number of pings to perform if [ $? -eq 0 ]; then # ping returns exit status 0 if successful echo "$(date "+%d.%m.%Y %T") PASSED: *** Internet online" else echo "$(date "+%d.%m.%Y %T") FAIL: *** No connectivity. Will try again on next run" rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running exit fi ####### Create Rclone Mount ####### # Check If Rclone Mount Already Created if [[ -f "$RcloneMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Success ${RcloneRemoteName} remote is already mounted." else echo "$(date "+%d.%m.%Y %T") INFO: Mount not running. Will now mount ${RcloneRemoteName} remote." # Creating mountcheck file in case it doesn't already exist echo "$(date "+%d.%m.%Y %T") INFO: Recreating mountcheck file for ${RcloneRemoteName} remote." touch mountcheck rclone copy mountcheck $RcloneRemoteName: -vv --no-traverse # Check bind option if [[ $CreateBindMount == 'Y' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: *** Checking if IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}" ping -q -c2 $RCloneMountIP > /dev/null # -q quiet, -c number of pings to perform if [ $? -eq 0 ]; then # ping returns exit status 0 if successful echo "$(date "+%d.%m.%Y %T") INFO: *** IP address ${RCloneMountIP} already created for remote ${RcloneRemoteName}" else echo "$(date "+%d.%m.%Y %T") INFO: *** Creating IP address ${RCloneMountIP} for remote ${RcloneRemoteName}" ip addr add $RCloneMountIP/24 dev $NetworkAdapter label $NetworkAdapter:$VirtualIPNumber fi echo "$(date "+%d.%m.%Y %T") INFO: *** Created bind mount ${RCloneMountIP} for remote ${RcloneRemoteName}" else RCloneMountIP="" echo "$(date "+%d.%m.%Y %T") INFO: *** Creating mount for remote ${RcloneRemoteName}" fi # create rclone mount rclone mount \ --allow-other \ --buffer-size 256M \ --dir-cache-time 720h \ --drive-chunk-size 512M \ --log-level INFO \ --vfs-read-chunk-size 128M \ --vfs-read-chunk-size-limit off \ --vfs-cache-mode writes \ --bind=$RCloneMountIP \ $RcloneRemoteName: $RcloneMountLocation & # Check if Mount Successful echo "$(date "+%d.%m.%Y %T") INFO: sleeping for 5 seconds" # slight pause to give mount time to finalise sleep 5 echo "$(date "+%d.%m.%Y %T") INFO: continuing..." if [[ -f "$RcloneMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Successful mount of ${RcloneRemoteName} mount." else echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mount failed - please check for problems. Stopping dockers" docker stop $DockerStart rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running exit fi fi ####### Start MergerFS Mount ####### if [[ $MergerfsMountShare == 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Not creating mergerfs mount as requested." else if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount in place." else # check if mergerfs already installed if [[ -f "/bin/mergerfs" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs already installed, proceeding to create mergerfs mount" else # Build mergerfs binary echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs not installed - installing now." mkdir -p /mnt/user/appdata/other/rclone/mergerfs docker run -v /mnt/user/appdata/other/rclone/mergerfs:/build --rm trapexit/mergerfs-static-build mv /mnt/user/appdata/other/rclone/mergerfs/mergerfs /bin # check if mergerfs install successful echo "$(date "+%d.%m.%Y %T") INFO: *sleeping for 5 seconds" sleep 5 if [[ -f "/bin/mergerfs" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Mergerfs installed successfully, proceeding to create mergerfs mount." else echo "$(date "+%d.%m.%Y %T") ERROR: Mergerfs not installed successfully. Please check for errors. Exiting." rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running exit fi fi # Create mergerfs mount echo "$(date "+%d.%m.%Y %T") INFO: Creating ${RcloneRemoteName} mergerfs mount." # Extra Mergerfs folders if [[ $LocalFilesShare2 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare2} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare2=":$LocalFilesShare2" else LocalFilesShare2="" fi if [[ $LocalFilesShare3 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare3} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare3=":$LocalFilesShare3" else LocalFilesShare3="" fi if [[ $LocalFilesShare4 != 'ignore' ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Adding ${LocalFilesShare4} to ${RcloneRemoteName} mergerfs mount." LocalFilesShare4=":$LocalFilesShare4" else LocalFilesShare4="" fi # make sure mergerfs mount point is empty mv $MergerFSMountLocation $LocalFilesLocation mkdir -p $MergerFSMountLocation # mergerfs mount command mergerfs $LocalFilesLocation:$RcloneMountLocation$LocalFilesShare2$LocalFilesShare3$LocalFilesShare4 $MergerFSMountLocation -o rw,async_read=false,use_ino,allow_other,func.getattr=newest,category.action=all,category.create=ff,cache.files=partial,dropcacheonclose=true # check if mergerfs mount successful echo "$(date "+%d.%m.%Y %T") INFO: Checking if ${RcloneRemoteName} mergerfs mount created." if [[ -f "$MergerFSMountLocation/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check successful, ${RcloneRemoteName} mergerfs mount created." else echo "$(date "+%d.%m.%Y %T") CRITICAL: ${RcloneRemoteName} mergerfs mount failed. Stopping dockers." docker stop $DockerStart rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running exit fi fi fi ####### Starting Dockers That Need Mergerfs Mount To Work Properly ####### # only start dockers once if [[ -f "/mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: dockers already started." else touch /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/dockers_started echo "$(date "+%d.%m.%Y %T") INFO: Starting dockers." docker start $DockerStart fi rm /mnt/user/appdata/other/rclone/remotes/$RcloneRemoteName/mount_running echo "$(date "+%d.%m.%Y %T") INFO: Script complete" exit Will someone kindly tell me where I'm screwing up. Thanks in advsance Quote Link to comment
DZMM Posted September 3, 2020 Author Share Posted September 3, 2020 11 hours ago, arizonaslim said: Failed to create file system for "gdrive_vfs:": didn't find section in config file you don't have a gdrive_vfs remote in your rclone config Quote Link to comment
arizonaslim Posted September 3, 2020 Share Posted September 3, 2020 (edited) @DZMM appreciate it. Thanks. Edited September 3, 2020 by arizonaslim Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.