DZMM Posted May 15, 2019 Author Share Posted May 15, 2019 (edited) 4 hours ago, Abe87 said: I was having 429 Rate Limit Exceeded errors in my log. It was taking 20-30 seconds to access files. I found this thread on the RClone with other users having the same problem. The fix is to add the user-agent option to your mount script. I added this --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36" which is the Chrome user-agent. This has fixed the problem for me. There are reports that you can choose any user agent you want and it will work. Thanks for posting the thread. I hadn't noticed any problems with playback and none of my users had mentioned anything, but my upload job was getting 429 errors. Adding the user agent to my upload script seems to have got it going again but it seems a bit slower - about 75% of the previous speed. It's a bit worrying if they are targetting the rclone user agent..... Edited May 15, 2019 by DZMM Quote Link to comment
francrouge Posted May 16, 2019 Share Posted May 16, 2019 (edited) On 5/13/2019 at 12:56 PM, DZMM said: Manually remove this file /mnt/user/appdata/other/rclone/rclone_upload It should get removed at array start/end depending on how you've setup your unmount script It seem to be working i'm waiting for a transfert to start will see Edit: Its working now thx again Edited May 17, 2019 by francrouge Quote Link to comment
francrouge Posted May 17, 2019 Share Posted May 17, 2019 (edited) Hi all still having some random issue Full logs for this script are available at /tmp/user.scripts/tmpScripts/test rclone/log.txt /usr/sbin/rclone: line 22: 21762 Killed rcloneorig --config $config "$@" Script Starting Fri, 17 May 2019 17:20:01 -0400 Full logs for this script are available at /tmp/user.scripts/tmpScripts/test rclone/log.txt mkdir: cannot stat '/mnt/user/mount_rclone/google_vfs': Transport endpoint is not connected 17.05.2019 17:20:01 INFO: mounting rclone vfs. 2019/05/17 17:20:03 Fatal error: Can not open: /mnt/user/mount_rclone/google_vfs: open /mnt/user/mount_rclone/google_vfs: transport endpoint is not connected 17.05.2019 17:20:06 CRITICAL: rclone gdrive vfs mount failed - please check for problems. Script Finished Fri, 17 May 2019 17:20:06 -0400 Full logs for this script are available at /tmp/user.scripts/tmpScripts/test rclone/log.txt Script Starting Fri, 17 May 2019 17:25:01 -0400 Full logs for this script are available at /tmp/user.scripts/tmpScripts/test rclone/log.txt mkdir: cannot stat '/mnt/user/mount_rclone/google_vfs': Transport endpoint is not connected 17.05.2019 17:25:01 INFO: mounting rclone vfs. 2019/05/17 17:25:02 Fatal error: Can not open: /mnt/user/mount_rclone/google_vfs: open /mnt/user/mount_rclone/google_vfs: transport endpoint is not connected 17.05.2019 17:25:06 CRITICAL: rclone gdrive vfs mount failed - please check for problems. Script Finished Fri, 17 May 2019 17:25:06 -0400 Can someone tell me what should i do please thx Also in plex every media has a red trash what can i do ? Edited May 17, 2019 by francrouge Quote Link to comment
DZMM Posted May 18, 2019 Author Share Posted May 18, 2019 8 hours ago, francrouge said: Hi all still having some random issue Full logs for this script are available at /tmp/user.scripts/tmpScripts/test rclone/log.txt /usr/sbin/rclone: line 22: 21762 Killed rcloneorig --config $config "$@" Script Starting Fri, 17 May 2019 17:20:01 -0400 Full logs for this script are available at /tmp/user.scripts/tmpScripts/test rclone/log.txt mkdir: cannot stat '/mnt/user/mount_rclone/google_vfs': Transport endpoint is not connected 17.05.2019 17:20:01 INFO: mounting rclone vfs. 2019/05/17 17:20:03 Fatal error: Can not open: /mnt/user/mount_rclone/google_vfs: open /mnt/user/mount_rclone/google_vfs: transport endpoint is not connected 17.05.2019 17:20:06 CRITICAL: rclone gdrive vfs mount failed - please check for problems. Script Finished Fri, 17 May 2019 17:20:06 -0400 Full logs for this script are available at /tmp/user.scripts/tmpScripts/test rclone/log.txt Script Starting Fri, 17 May 2019 17:25:01 -0400 Full logs for this script are available at /tmp/user.scripts/tmpScripts/test rclone/log.txt mkdir: cannot stat '/mnt/user/mount_rclone/google_vfs': Transport endpoint is not connected 17.05.2019 17:25:01 INFO: mounting rclone vfs. 2019/05/17 17:25:02 Fatal error: Can not open: /mnt/user/mount_rclone/google_vfs: open /mnt/user/mount_rclone/google_vfs: transport endpoint is not connected 17.05.2019 17:25:06 CRITICAL: rclone gdrive vfs mount failed - please check for problems. Script Finished Fri, 17 May 2019 17:25:06 -0400 Can someone tell me what should i do please thx Also in plex every media has a red trash what can i do ? Does /mnt/user/mount_rclone/google_vfs exist? The Plex behaviour is correct as rclone isn't mounted, so it can't see the files. It will self correct once you fix the mount. Quote Link to comment
francrouge Posted May 18, 2019 Share Posted May 18, 2019 (edited) 9 hours ago, DZMM said: Does /mnt/user/mount_rclone/google_vfs exist? The Plex behaviour is correct as rclone isn't mounted, so it can't see the files. It will self correct once you fix the mount. Hi yes google_vfs exist but its only a file not folder I didn't not restart krusader now its a folder And for plex do you have to scan the library again or not ? thx Edited May 18, 2019 by francrouge Quote Link to comment
francrouge Posted May 18, 2019 Share Posted May 18, 2019 Does /mnt/user/mount_rclone/google_vfs exist? The Plex behaviour is correct as rclone isn't mounted, so it can't see the files. It will self correct once you fix the mount.Do you have an idea why its doing that ? ThxEnvoyé de mon Pixel 2 XL en utilisant Tapatalk Quote Link to comment
DZMM Posted May 18, 2019 Author Share Posted May 18, 2019 39 minutes ago, francrouge said: Do you have an idea why its doing that ? Thx Envoyé de mon Pixel 2 XL en utilisant Tapatalk It does it when it can't see the files ie the mount isn't mounted. If you want fix ahead of schedule tell Plex to do a manual scan Is all working now? Quote Link to comment
francrouge Posted May 18, 2019 Share Posted May 18, 2019 It does it when it can't see the files ie the mount isn't mounted. If you want fix ahead of schedule tell Plex to do a manual scan Is all working now?I did run unmout and mount script and ist wokring but it crashed 3 times today so far .ThxEnvoyé de mon Pixel 2 XL en utilisant Tapatalk Quote Link to comment
francrouge Posted May 18, 2019 Share Posted May 18, 2019 It does it when it can't see the files ie the mount isn't mounted. If you want fix ahead of schedule tell Plex to do a manual scan Is all working now?Could it be unraid that is killing the process ? Because i can see that my ram usage is very high.. i dont know why but it seem to be thatEnvoyé de mon Pixel 2 XL en utilisant Tapatalk Quote Link to comment
privateer Posted May 19, 2019 Share Posted May 19, 2019 (edited) Very basic question. When I go into my gdrive/crypt folder online the only thing I see a 32 byte file with a jumbled name. The scripts etc don't seem to be working for me. I used the ones loaded into github. I've pasted some log files and data below. /mnt/user/mount_rclone/google_vfs contains 1 file, mountcheck, and no folders. /mnt/user/mount_unionfs/google_vfs contains the two folders I created on my local system, movies and tv_shows. I put a file in the movies folder but I don't see any indication that it has uploaded/is uploading (file is >30 min old in the folder). /mnt/user/rclone_upload/google_vfs is empty. Mount script log: INFO: Check rclone vfs already mounted. fuse: mountpoint is not empty fuse: if you are sure this is safe, use the 'nonempty' mount option CRITICAL: unionfs Remount failed. rm: cannot remove '/mnt/user/appdata/other/rclone/rclone_mount_running': No such file or directory Upload script log: /tmp/user.scripts/tmpScripts/rclone_upload/script: line 4: $'\357\273\277': command not found INFO: rclone installed successfully - proceeding with upload. DEBUG : --min-age 30m0s to [TIMESTAMP AND TIMEZONE] m=-1799.989256277 DEBUG : rclone: Version "v1.47.0-073-gcff85f0b-beta" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone-beta/.rclone.conf" "move" "/mnt/user/rclone_upload/google_vfs/" "gdrive_media_vfs:" "-vv" "--drive-chunk-size" "512M" "--checkers" "3" "--fast-list" "--transfers" "2" "--exclude" ".unionfs/**" "--exclude" "*fuse_hidden*" "--exclude" "*_HIDDEN" "--exclude" ".recycle**" "--exclude" "*.backup~*" "--exclude" "*.partial~*" "--delete-empty-src-dirs" "--fast-list" "--bwlimit" "9500k" "--tpslimit" "3" "--min-age" "30m"] DEBUG : Using config file from "/boot/config/plugins/rclone-beta/.rclone.conf" INFO : Starting bandwidth limiter at 9.277MBytes/s INFO : Starting HTTP transaction limiter: max 3 transactions/s with burst 1 INFO : Encrypted drive 'gdrive_media_vfs:': Waiting for checks to finish INFO : Encrypted drive 'gdrive_media_vfs:': Waiting for transfers to finish INFO : Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA - Errors: 0 Checks: 0 / 0, - Transferred: 0 / 0, - Elapsed time: 1.2s DEBUG : 7 go routines active DEBUG : rclone: Version "v1.47.0-073-gcff85f0b-beta" finishing with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone-beta/.rclone.conf" "move" "/mnt/user/rclone_upload/google_vfs/" "gdrive_media_vfs:" "-vv" "--drive-chunk-size" "512M" "--checkers" "3" "--fast-list" "--transfers" "2" "--exclude" ".unionfs/**" "--exclude" "*fuse_hidden*" "--exclude" "*_HIDDEN" "--exclude" ".recycle**" "--exclude" "*.backup~*" "--exclude" "*.partial~*" "--delete-empty-src-dirs" "--fast-list" "--bwlimit" "9500k" "--tpslimit" "3" "--min-age" "30m"] Edited May 19, 2019 by privateer updated info Quote Link to comment
DZMM Posted May 19, 2019 Author Share Posted May 19, 2019 On 5/18/2019 at 7:39 PM, francrouge said: Could it be unraid that is killing the process ? Because i can see that my ram usage is very high.. i dont know why but it seem to be that Envoyé de mon Pixel 2 XL en utilisant Tapatalk how much ram do you have? Does rebooting help? Quote Link to comment
francrouge Posted May 19, 2019 Share Posted May 19, 2019 how much ram do you have? Does rebooting help?16go and reboot help so maybe its that ? Envoyé de mon Pixel 2 XL en utilisant Tapatalk Quote Link to comment
DZMM Posted May 19, 2019 Author Share Posted May 19, 2019 6 hours ago, privateer said: fuse: mountpoint is not empty fuse: if you are sure this is safe, use the 'nonempty' mount option CRITICAL: unionfs Remount failed. rm: cannot remove '/mnt/user/appdata/other/rclone/rclone_mount_running': No such file or directory your unionfs mount is failing, I think because you created your movies and tv_shows folder incorrectly. you need to delete those folders and then run your mount command/script again, then recreate the folders and they'll be copied to gdrive. Quote Link to comment
DZMM Posted May 19, 2019 Author Share Posted May 19, 2019 5 minutes ago, francrouge said: 16go and reboot help so maybe its that ? Envoyé de mon Pixel 2 XL en utilisant Tapatalk 16gb should be enough. I'd disable the scripts and reboot. Then delete any residual checker files form /mnt/user/appdata/rclone (i think that's where all the script checks are) and then enable the scripts. Quote Link to comment
francrouge Posted May 19, 2019 Share Posted May 19, 2019 16gb should be enough. I'd disable the scripts and reboot. Then delete any residual checker files form /mnt/user/appdata/rclone (i think that's where all the script checks are) and then enable the scripts.Ok cool thxEnvoyé de mon Pixel 2 XL en utilisant Tapatalk Quote Link to comment
privateer Posted May 19, 2019 Share Posted May 19, 2019 21 minutes ago, DZMM said: your unionfs mount is failing, I think because you created your movies and tv_shows folder incorrectly. you need to delete those folders and then run your mount command/script again, then recreate the folders and they'll be copied to gdrive. Any thoughts on the upload issue: /tmp/user.scripts/tmpScripts/rclone_upload/script: line 4: $'\357\273\277': command not found Quote Link to comment
privateer Posted May 20, 2019 Share Posted May 20, 2019 3 hours ago, privateer said: Any thoughts on the upload issue: /tmp/user.scripts/tmpScripts/rclone_upload/script: line 4: $'\357\273\277': command not found Found the error as referenced earlier in this thread. While no text shows, there are blank/not visible entries in the upload script. Manually deleting the blank spaces and re-adding fixed this. I've added a file several hours ago into /mnt/user/mount_unionfs/google_vfs/movies/ which has not uploaded to my drive. Also the blank folders /movies/ and /tv_shows/ in /mnt/user/mount_unionfs/google_vfs/ have not uploaded. None of these show up in /mnt/user/rclone_upload/google_vfs/ either. Any thoughts or logs I can provide to help? Quote Link to comment
DZMM Posted May 20, 2019 Author Share Posted May 20, 2019 5 hours ago, privateer said: Found the error as referenced earlier in this thread. While no text shows, there are blank/not visible entries in the upload script. Manually deleting the blank spaces and re-adding fixed this. I've added a file several hours ago into /mnt/user/mount_unionfs/google_vfs/movies/ which has not uploaded to my drive. Also the blank folders /movies/ and /tv_shows/ in /mnt/user/mount_unionfs/google_vfs/ have not uploaded. None of these show up in /mnt/user/rclone_upload/google_vfs/ either. Any thoughts or logs I can provide to help? Can you post your rclone config and scripts please Quote Link to comment
privateer Posted May 20, 2019 Share Posted May 20, 2019 Thanks for all the help. I'm (obviously) a newbie so this is much appreciated. When I create the movies and tv_shows folders (and moving files into them) I'm using Krusader. Could this be causing problems? Do I need to use CLI? rclone config: Quote [gdrive] type = drive client_id = [removed].apps.googleusercontent.com client_secret = [removed] scope = drive token = [removed] [gdrive_media_vfs] type = crypt remote = gdrive:crypt filename_encryption = standard directory_name_encryption = true password = [removed] password2 = [removed] Mount script Quote #!/bin/bash ####### Start rclone gdrive mount ########## # check if gdrive mount already created if [[ -f "/mnt/user/mount_rclone/google_vfs/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check rclone vfs already mounted." else echo "$(date "+%d.%m.%Y %T") INFO: mounting rclone vfs." # create directories for rclone mount and unionfs mount mkdir -p /mnt/user/appdata/other/rclone mkdir -p /mnt/user/mount_unionfs/google_vfs mkdir -p /mnt/user/rclone_upload/google_vfs rclone mount --allow-other --buffer-size 256M --dir-cache-time 72h --drive-chunk-size 512M --fast-list --log-level INFO --vfs-read-chunk-size 128M --vfs-read-chunk-size-limit off gdrive_media_vfs: /mnt/user/mount_rclone/google_vfs & # check if mount successful # slight pause to give mount time to finalise sleep 5 if [[ -f "/mnt/user/mount_rclone/google_vfs/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check rclone gdrive vfs mount success." else echo "$(date "+%d.%m.%Y %T") CRITICAL: rclone gdrive vfs mount failed - please check for problems." rm /mnt/user/appdata/other/rclone/rclone_mount_running exit fi fi ####### End rclone gdrive mount ########## ####### Start unionfs mount ########## if [[ -f "/mnt/user/mount_unionfs/google_vfs/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check successful, unionfs already mounted." else unionfs -o cow,allow_other,direct_io,auto_cache,sync_read /mnt/user/rclone_upload/google_vfs=RW:/mnt/user/mount_rclone/google_vfs=RO /mnt/user/mount_unionfs/google_vfs if [[ -f "/mnt/user/mount_unionfs/google_vfs/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Check successful, unionfs mounted." else echo "$(date "+%d.%m.%Y %T") CRITICAL: unionfs Remount failed." rm /mnt/user/appdata/other/rclone/rclone_mount_running exit fi fi ####### End Mount unionfs ########## ############### starting dockers that need unionfs mount ###################### # only start dockers once if [[ -f "/mnt/user/appdata/other/rclone/dockers_started" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: dockers already started" else touch /mnt/user/appdata/other/rclone/dockers_started echo "$(date "+%d.%m.%Y %T") INFO: Starting dockers." docker start plex docker start ombi docker start tautulli docker start radarr docker start radarr4k docker start sonarr docker start sonarr4k fi ############### end dockers that need unionfs mount ###################### exit Upload script Quote #!/bin/bash ####### Check if script already running ########## if [[ -f "/mnt/user/appdata/other/rclone/rclone_upload" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Exiting as script already running." exit else touch /mnt/user/appdata/other/rclone/rclone_upload fi ####### End Check if script already running ########## ####### check if rclone installed ########## if [[ -f "/mnt/user/mount_rclone/google_vfs/mountcheck" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: rclone installed successfully - proceeding with upload." else echo "$(date "+%d.%m.%Y %T") INFO: rclone not installed - will try again later." rm /mnt/user/appdata/other/rclone/rclone_upload exit fi ####### end check if rclone installed ########## # move files rclone move /mnt/user/rclone_upload/google_vfs/ gdrive_media_vfs: -vv --drive-chunk-size 512M --checkers 3 --fast-list --transfers 2 --exclude .unionfs/** --exclude *fuse_hidden* --exclude *_HIDDEN --exclude .recycle** --exclude *.backup~* --exclude *.partial~* --delete-empty-src-dirs --fast-list --bwlimit 9500k --tpslimit 3 --min-age 30m # remove dummy file rm /mnt/user/appdata/other/rclone/rclone_upload exit Unmount script Quote #!/bin/bash # unmount to be safe fusermount -uz /mnt/user/mount_unionfs/google_vfs fusermount -uz /mnt/user/mount_rclone/google_vfs # Remove other dummy files if [[ -f "/mnt/user/appdata/other/rclone/rclone_upload" ]]; then echo "upload file present - removing dummy file" rm /mnt/user/appdata/other/rclone/rclone_upload else echo "rclone upload already exited properly" fi if [[ -f "/mnt/user/appdata/other/rclone/rclone_cleanup" ]]; then echo "cleanup file present - removing dummy file" rm /mnt/user/appdata/other/rclone/rclone_cleanup else echo "cleanup already exited properly" fi if [[ -f "/mnt/user/appdata/other/rclone/dockers_started" ]]; then echo "removing dummy docker check file" rm /mnt/user/appdata/other/rclone/dockers_started else echo "docker run once already removed" fi exit Cleanup script (haven't paid much attention as I haven't got anything to upload yet) Quote #!/bin/bash ####### Check if script already running ########## if [[ -f "/mnt/user/appdata/other/rclone/rclone_cleanup" ]]; then echo "$(date "+%d.%m.%Y %T") INFO: Exiting as script already running." exit else touch /mnt/user/appdata/other/rclone/rclone_cleanup fi ####### End Check if script already running ########## ################### Clean-up UnionFS Folder ######################### echo "$(date "+%d.%m.%Y %T") INFO: starting unionfs cleanup." find /mnt/user/mount_unionfs/google_vfs/.unionfs -name '*_HIDDEN~' | while read line; do oldPath=${line#/mnt/user/mount_unionfs/google_vfs/.unionfs} newPath=/mnt/user/mount_rclone/google_vfs${oldPath%_HIDDEN~} rm "$newPath" rm "$line" done find "/mnt/user/mount_unionfs/google_vfs/.unionfs" -mindepth 1 -type d -empty -delete rm /mnt/user/appdata/other/rclone/rclone_cleanup exit Quote Link to comment
DZMM Posted May 20, 2019 Author Share Posted May 20, 2019 1 hour ago, privateer said: Thanks for all the help. I'm (obviously) a newbie so this is much appreciated. When I create the movies and tv_shows folders (and moving files into them) I'm using Krusader. Could this be causing problems? Do I need to use CLI? rclone config: Mount script Upload script Unmount script Cleanup script (haven't paid much attention as I haven't got anything to upload yet) All looks ok to me - maybe someone else will spot something @Kaizac ??. What happens if you create a few test folders in: (i) /mnt/user/mount_unionfs/google_vfs (ii) directly in /mnt/user/rclone_upload/google_vfs Do they appear in /mnt/user/mount_rclone/google_vfs? (change --min-age 30m to --min-age 1m or delete in the upload script to test) The only thing I can think of is there was a weird gdrive issue last week where uploads from Europe were blocked for a day, but this has been resolved as far as I know Quote Link to comment
privateer Posted May 20, 2019 Share Posted May 20, 2019 Want to respond here in case other people run into this problem in the future. I'm using unraid as my OS. I was creating folders and moving files with a Krusader docker. This appears to be the source of the problem. When I used CLI to create the folders and move the files everything appeared to run correctly. File is uploading now. Quote Link to comment
DZMM Posted May 20, 2019 Author Share Posted May 20, 2019 (edited) 4 minutes ago, privateer said: Want to respond here in case other people run into this problem in the future. I'm using unraid as my OS. I was creating folders and moving files with a Krusader docker. This appears to be the source of the problem. When I used CLI to create the folders and move the files everything appeared to run correctly. File is uploading now. oh yeah, Krusader seems to have problems if you don't restart it. I don't use so I'm not sure why - I use mc to move files Edited May 20, 2019 by DZMM Quote Link to comment
francrouge Posted May 21, 2019 Share Posted May 21, 2019 On 5/19/2019 at 4:43 PM, DZMM said: 16gb should be enough. I'd disable the scripts and reboot. Then delete any residual checker files form /mnt/user/appdata/rclone (i think that's where all the script checks are) and then enable the scripts. Hi DZMM, Would it be possible to add in the script if the mount failed an email is sent ? Is it something difficult to do. I don't know nothing about scripting thx Quote Link to comment
DZMM Posted May 21, 2019 Author Share Posted May 21, 2019 1 hour ago, francrouge said: I don't know nothing about scripting thx Neither do I really! I cobbled these scripts together using other guides and Google. Quote Link to comment
francrouge Posted May 21, 2019 Share Posted May 21, 2019 Neither do I really! I cobbled these scripts together using other guides and Google. O ok thx Envoyé de mon Pixel 2 XL en utilisant Tapatalk Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.