Guide: How To Use Rclone To Mount Cloud Drives And Play Files


DZMM

Recommended Posts

4 hours ago, Abe87 said:

I was having 429 Rate Limit Exceeded errors in my log. It was taking 20-30 seconds to access files. I found this thread on the RClone with other users having the same problem. The fix is to add the user-agent option to your mount script.

 

I added this --user-agent "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36" which is the Chrome user-agent. This has fixed the problem for me. There are reports that you can choose any user agent you want and it will work.

 

 

Thanks for posting the thread.  I hadn't noticed any problems with playback and none of my users had mentioned anything, but my upload job was getting 429 errors.  Adding the user agent to my upload script seems to have got it going again but it seems a bit slower - about 75% of the previous speed.

 

It's a bit worrying if they are targetting the rclone user agent.....

Edited by DZMM
Link to comment
On 5/13/2019 at 12:56 PM, DZMM said:

Manually remove this file /mnt/user/appdata/other/rclone/rclone_upload

 

It should get removed at array start/end depending on how you've setup your unmount script

It seem to be working i'm waiting for a transfert to start will see

 

 

 

Edit: Its working now thx again

Edited by francrouge
Link to comment

Hi all still having some random issue

 

Full logs for this script are available at /tmp/user.scripts/tmpScripts/test rclone/log.txt

/usr/sbin/rclone: line 22: 21762 Killed                  rcloneorig --config $config "$@"
Script Starting Fri, 17 May 2019 17:20:01 -0400

Full logs for this script are available at /tmp/user.scripts/tmpScripts/test rclone/log.txt

mkdir: cannot stat '/mnt/user/mount_rclone/google_vfs': Transport endpoint is not connected
17.05.2019 17:20:01 INFO: mounting rclone vfs.
2019/05/17 17:20:03 Fatal error: Can not open: /mnt/user/mount_rclone/google_vfs: open /mnt/user/mount_rclone/google_vfs: transport endpoint is not connected
17.05.2019 17:20:06 CRITICAL: rclone gdrive vfs mount failed - please check for problems.
Script Finished Fri, 17 May 2019 17:20:06 -0400

Full logs for this script are available at /tmp/user.scripts/tmpScripts/test rclone/log.txt

Script Starting Fri, 17 May 2019 17:25:01 -0400

Full logs for this script are available at /tmp/user.scripts/tmpScripts/test rclone/log.txt

mkdir: cannot stat '/mnt/user/mount_rclone/google_vfs': Transport endpoint is not connected
17.05.2019 17:25:01 INFO: mounting rclone vfs.
2019/05/17 17:25:02 Fatal error: Can not open: /mnt/user/mount_rclone/google_vfs: open /mnt/user/mount_rclone/google_vfs: transport endpoint is not connected
17.05.2019 17:25:06 CRITICAL: rclone gdrive vfs mount failed - please check for problems.
Script Finished Fri, 17 May 2019 17:25:06 -0400

Can someone tell me what should i do please thx

 

 

Also in plex every media has a red trash what can i do ?

Edited by francrouge
Link to comment
8 hours ago, francrouge said:

Hi all still having some random issue

 


Full logs for this script are available at /tmp/user.scripts/tmpScripts/test rclone/log.txt

/usr/sbin/rclone: line 22: 21762 Killed                  rcloneorig --config $config "$@"
Script Starting Fri, 17 May 2019 17:20:01 -0400

Full logs for this script are available at /tmp/user.scripts/tmpScripts/test rclone/log.txt

mkdir: cannot stat '/mnt/user/mount_rclone/google_vfs': Transport endpoint is not connected
17.05.2019 17:20:01 INFO: mounting rclone vfs.
2019/05/17 17:20:03 Fatal error: Can not open: /mnt/user/mount_rclone/google_vfs: open /mnt/user/mount_rclone/google_vfs: transport endpoint is not connected
17.05.2019 17:20:06 CRITICAL: rclone gdrive vfs mount failed - please check for problems.
Script Finished Fri, 17 May 2019 17:20:06 -0400

Full logs for this script are available at /tmp/user.scripts/tmpScripts/test rclone/log.txt

Script Starting Fri, 17 May 2019 17:25:01 -0400

Full logs for this script are available at /tmp/user.scripts/tmpScripts/test rclone/log.txt

mkdir: cannot stat '/mnt/user/mount_rclone/google_vfs': Transport endpoint is not connected
17.05.2019 17:25:01 INFO: mounting rclone vfs.
2019/05/17 17:25:02 Fatal error: Can not open: /mnt/user/mount_rclone/google_vfs: open /mnt/user/mount_rclone/google_vfs: transport endpoint is not connected
17.05.2019 17:25:06 CRITICAL: rclone gdrive vfs mount failed - please check for problems.
Script Finished Fri, 17 May 2019 17:25:06 -0400

Can someone tell me what should i do please thx

 

 

Also in plex every media has a red trash what can i do ?

Does /mnt/user/mount_rclone/google_vfs exist?

 

The Plex behaviour is correct as rclone isn't mounted, so it can't see the files.  It will self correct once you fix the mount.

Link to comment
9 hours ago, DZMM said:

Does /mnt/user/mount_rclone/google_vfs exist?

 

The Plex behaviour is correct as rclone isn't mounted, so it can't see the files.  It will self correct once you fix the mount.

Hi yes google_vfs exist but its only a file not folder

 

I didn't not restart krusader now its a folder

 

And for plex do you have to scan the library again or not ?

 

thx

Edited by francrouge
Link to comment
39 minutes ago, francrouge said:

Do you have an idea why its doing that ? Thx

Envoyé de mon Pixel 2 XL en utilisant Tapatalk
 

It does it when it can't see the files ie the mount isn't mounted. If you want fix ahead of schedule tell Plex to do a manual scan

 

Is all working now?

Link to comment
It does it when it can't see the files ie the mount isn't mounted. If you want fix ahead of schedule tell Plex to do a manual scan
 
Is all working now?
I did run unmout and mount script and ist wokring but it crashed 3 times today so far .

Thx

Envoyé de mon Pixel 2 XL en utilisant Tapatalk

Link to comment
It does it when it can't see the files ie the mount isn't mounted. If you want fix ahead of schedule tell Plex to do a manual scan
 
Is all working now?
Could it be unraid that is killing the process ?

Because i can see that my ram usage is very high.. i dont know why but it seem to be that

Envoyé de mon Pixel 2 XL en utilisant Tapatalk

Link to comment

Very basic question. When I go into my gdrive/crypt folder online the only thing  I see a 32 byte file with a jumbled name. The scripts etc don't seem to be working for me. I used the ones loaded into github. I've pasted some log files and data below.

 

/mnt/user/mount_rclone/google_vfs  contains 1 file, mountcheck, and no folders.

/mnt/user/mount_unionfs/google_vfs  contains the two folders I created on my local system, movies and tv_shows. I put a file in the movies folder but I don't see any indication that it has uploaded/is uploading (file is >30 min old in the folder).

/mnt/user/rclone_upload/google_vfs  is empty.

 

Mount script log:

INFO: Check rclone vfs already mounted.
fuse: mountpoint is not empty
fuse: if you are sure this is safe, use the 'nonempty' mount option
CRITICAL: unionfs Remount failed.
rm: cannot remove '/mnt/user/appdata/other/rclone/rclone_mount_running': No such file or directory
 

Upload script log:

/tmp/user.scripts/tmpScripts/rclone_upload/script: line 4: $'\357\273\277': command not found
INFO: rclone installed successfully - proceeding with upload.
DEBUG : --min-age 30m0s to [TIMESTAMP AND TIMEZONE] m=-1799.989256277
DEBUG : rclone: Version "v1.47.0-073-gcff85f0b-beta" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone-beta/.rclone.conf" "move" "/mnt/user/rclone_upload/google_vfs/" "gdrive_media_vfs:" "-vv" "--drive-chunk-size" "512M" "--checkers" "3" "--fast-list" "--transfers" "2" "--exclude" ".unionfs/**" "--exclude" "*fuse_hidden*" "--exclude" "*_HIDDEN" "--exclude" ".recycle**" "--exclude" "*.backup~*" "--exclude" "*.partial~*" "--delete-empty-src-dirs" "--fast-list" "--bwlimit" "9500k" "--tpslimit" "3" "--min-age" "30m"]
DEBUG : Using config file from "/boot/config/plugins/rclone-beta/.rclone.conf"
INFO : Starting bandwidth limiter at 9.277MBytes/s
INFO : Starting HTTP transaction limiter: max 3 transactions/s with burst 1
INFO : Encrypted drive 'gdrive_media_vfs:': Waiting for checks to finish
INFO : Encrypted drive 'gdrive_media_vfs:': Waiting for transfers to finish
INFO : 
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 0, -
Elapsed time: 1.2s

DEBUG : 7 go routines active
DEBUG : rclone: Version "v1.47.0-073-gcff85f0b-beta" finishing with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone-beta/.rclone.conf" "move" "/mnt/user/rclone_upload/google_vfs/" "gdrive_media_vfs:" "-vv" "--drive-chunk-size" "512M" "--checkers" "3" "--fast-list" "--transfers" "2" "--exclude" ".unionfs/**" "--exclude" "*fuse_hidden*" "--exclude" "*_HIDDEN" "--exclude" ".recycle**" "--exclude" "*.backup~*" "--exclude" "*.partial~*" "--delete-empty-src-dirs" "--fast-list" "--bwlimit" "9500k" "--tpslimit" "3" "--min-age" "30m"]

Edited by privateer
updated info
Link to comment
On 5/18/2019 at 7:39 PM, francrouge said:

Could it be unraid that is killing the process ?

Because i can see that my ram usage is very high.. i dont know why but it seem to be that

Envoyé de mon Pixel 2 XL en utilisant Tapatalk
 

how much ram do you have?  Does rebooting help?

Link to comment
6 hours ago, privateer said:


fuse: mountpoint is not empty
fuse: if you are sure this is safe, use the 'nonempty' mount option
CRITICAL: unionfs Remount failed.
rm: cannot remove '/mnt/user/appdata/other/rclone/rclone_mount_running': No such file or directory

your unionfs mount is failing, I think because you created your movies and tv_shows folder incorrectly.  you need to delete those folders and then run your mount command/script again, then recreate the folders and they'll be copied to gdrive.

Link to comment
5 minutes ago, francrouge said:

16go and reboot help so maybe its that ?

Envoyé de mon Pixel 2 XL en utilisant Tapatalk
 

16gb should be enough.  I'd disable the scripts and reboot.  Then delete any residual checker files form /mnt/user/appdata/rclone (i think that's where all the script checks are) and then enable the scripts.

Link to comment
21 minutes ago, DZMM said:

your unionfs mount is failing, I think because you created your movies and tv_shows folder incorrectly.  you need to delete those folders and then run your mount command/script again, then recreate the folders and they'll be copied to gdrive.

 

Any thoughts on the upload issue:

 

/tmp/user.scripts/tmpScripts/rclone_upload/script: line 4: $'\357\273\277': command not found

Link to comment
3 hours ago, privateer said:

 

Any thoughts on the upload issue:

 

/tmp/user.scripts/tmpScripts/rclone_upload/script: line 4: $'\357\273\277': command not found

 

Found the error as referenced earlier in this thread. While no text shows, there are blank/not visible entries in the upload script. Manually deleting the blank spaces and re-adding fixed this.

 

I've added a file several hours ago into /mnt/user/mount_unionfs/google_vfs/movies/  which has not uploaded to my drive. Also the blank folders /movies/ and /tv_shows/ in  /mnt/user/mount_unionfs/google_vfs/ have not uploaded. None of these show up in /mnt/user/rclone_upload/google_vfs/ either. Any thoughts or logs I can provide to help?

Link to comment
5 hours ago, privateer said:

 

Found the error as referenced earlier in this thread. While no text shows, there are blank/not visible entries in the upload script. Manually deleting the blank spaces and re-adding fixed this.

 

I've added a file several hours ago into /mnt/user/mount_unionfs/google_vfs/movies/  which has not uploaded to my drive. Also the blank folders /movies/ and /tv_shows/ in  /mnt/user/mount_unionfs/google_vfs/ have not uploaded. None of these show up in /mnt/user/rclone_upload/google_vfs/ either. Any thoughts or logs I can provide to help?

Can you post your rclone config and scripts please

Link to comment

Thanks for all the help. I'm (obviously) a newbie so this is much appreciated. When I create the movies and tv_shows folders (and moving files into them) I'm using Krusader. Could this be causing problems? Do I need to use CLI?

 

rclone config:

Quote

[gdrive]
type = drive
client_id = [removed].apps.googleusercontent.com
client_secret = [removed]
scope = drive
token = [removed]

 

[gdrive_media_vfs]
type = crypt
remote = gdrive:crypt
filename_encryption = standard
directory_name_encryption = true
password = [removed]
password2 = [removed]

 

Mount script

Quote

#!/bin/bash

#######  Start rclone gdrive mount  ##########

# check if gdrive mount already created

if [[ -f "/mnt/user/mount_rclone/google_vfs/mountcheck" ]]; then

echo "$(date "+%d.%m.%Y %T") INFO: Check rclone vfs already mounted."

else

echo "$(date "+%d.%m.%Y %T") INFO: mounting rclone vfs."

# create directories for rclone mount and unionfs mount

mkdir -p /mnt/user/appdata/other/rclone
mkdir -p /mnt/user/mount_unionfs/google_vfs
mkdir -p /mnt/user/rclone_upload/google_vfs

rclone mount --allow-other --buffer-size 256M --dir-cache-time 72h --drive-chunk-size 512M --fast-list --log-level INFO --vfs-read-chunk-size 128M --vfs-read-chunk-size-limit off gdrive_media_vfs: /mnt/user/mount_rclone/google_vfs &

# check if mount successful

# slight pause to give mount time to finalise

sleep 5

if [[ -f "/mnt/user/mount_rclone/google_vfs/mountcheck" ]]; then

echo "$(date "+%d.%m.%Y %T") INFO: Check rclone gdrive vfs mount success."

else

echo "$(date "+%d.%m.%Y %T") CRITICAL: rclone gdrive vfs mount failed - please check for problems."

rm /mnt/user/appdata/other/rclone/rclone_mount_running

exit

fi

fi

#######  End rclone gdrive mount  ##########

#######  Start unionfs mount   ##########

if [[ -f "/mnt/user/mount_unionfs/google_vfs/mountcheck" ]]; then

echo "$(date "+%d.%m.%Y %T") INFO: Check successful, unionfs already mounted."

else

unionfs -o cow,allow_other,direct_io,auto_cache,sync_read /mnt/user/rclone_upload/google_vfs=RW:/mnt/user/mount_rclone/google_vfs=RO /mnt/user/mount_unionfs/google_vfs

if [[ -f "/mnt/user/mount_unionfs/google_vfs/mountcheck" ]]; then

echo "$(date "+%d.%m.%Y %T") INFO: Check successful, unionfs mounted."

else

echo "$(date "+%d.%m.%Y %T") CRITICAL: unionfs Remount failed."

rm /mnt/user/appdata/other/rclone/rclone_mount_running

exit

fi

fi

#######  End Mount unionfs   ##########

############### starting dockers that need unionfs mount ######################

# only start dockers once

if [[ -f "/mnt/user/appdata/other/rclone/dockers_started" ]]; then

echo "$(date "+%d.%m.%Y %T") INFO: dockers already started"

else

touch /mnt/user/appdata/other/rclone/dockers_started

echo "$(date "+%d.%m.%Y %T") INFO: Starting dockers."

docker start plex
docker start ombi
docker start tautulli
docker start radarr
docker start radarr4k
docker start sonarr
docker start sonarr4k

fi

############### end dockers that need unionfs mount ######################

exit

 

Upload script

Quote

#!/bin/bash

#######  Check if script already running  ##########

if [[ -f "/mnt/user/appdata/other/rclone/rclone_upload" ]]; then

echo "$(date "+%d.%m.%Y %T") INFO: Exiting as script already running."

exit

else

touch /mnt/user/appdata/other/rclone/rclone_upload

fi

#######  End Check if script already running  ##########

#######  check if rclone installed  ##########

if [[ -f "/mnt/user/mount_rclone/google_vfs/mountcheck" ]]; then

echo "$(date "+%d.%m.%Y %T") INFO: rclone installed successfully - proceeding with upload."

else

echo "$(date "+%d.%m.%Y %T") INFO: rclone not installed - will try again later."

rm /mnt/user/appdata/other/rclone/rclone_upload

exit

fi

#######  end check if rclone installed  ##########

# move files

rclone move /mnt/user/rclone_upload/google_vfs/ gdrive_media_vfs: -vv --drive-chunk-size 512M --checkers 3 --fast-list --transfers 2 --exclude .unionfs/** --exclude *fuse_hidden* --exclude *_HIDDEN --exclude .recycle** --exclude *.backup~* --exclude *.partial~*  --delete-empty-src-dirs --fast-list --bwlimit 9500k --tpslimit 3 --min-age 30m 

# remove dummy file

rm /mnt/user/appdata/other/rclone/rclone_upload

exit

 

Unmount script

Quote

#!/bin/bash

# unmount to be safe

fusermount -uz /mnt/user/mount_unionfs/google_vfs
fusermount -uz /mnt/user/mount_rclone/google_vfs

# Remove other dummy files

if [[ -f "/mnt/user/appdata/other/rclone/rclone_upload" ]]; then

echo "upload file present - removing dummy file"

rm /mnt/user/appdata/other/rclone/rclone_upload

else

echo "rclone upload already exited properly"

fi

if [[ -f "/mnt/user/appdata/other/rclone/rclone_cleanup" ]]; then

echo "cleanup file present - removing dummy file"

rm /mnt/user/appdata/other/rclone/rclone_cleanup

else

echo "cleanup already exited properly"

fi

if [[ -f "/mnt/user/appdata/other/rclone/dockers_started" ]]; then

echo "removing dummy docker check file"

rm /mnt/user/appdata/other/rclone/dockers_started

else

echo "docker run once already removed"

fi

exit

 

 

Cleanup script (haven't paid much attention as I haven't got anything to upload yet)

Quote

#!/bin/bash

#######  Check if script already running  ##########

if [[ -f "/mnt/user/appdata/other/rclone/rclone_cleanup" ]]; then

echo "$(date "+%d.%m.%Y %T") INFO: Exiting as script already running."

exit

else

touch /mnt/user/appdata/other/rclone/rclone_cleanup

fi

#######  End Check if script already running  ##########


################### Clean-up UnionFS Folder  #########################

echo "$(date "+%d.%m.%Y %T") INFO: starting unionfs cleanup."

find /mnt/user/mount_unionfs/google_vfs/.unionfs -name '*_HIDDEN~' | while read line; do
oldPath=${line#/mnt/user/mount_unionfs/google_vfs/.unionfs}
newPath=/mnt/user/mount_rclone/google_vfs${oldPath%_HIDDEN~}
rm "$newPath"
rm "$line"
done
find "/mnt/user/mount_unionfs/google_vfs/.unionfs" -mindepth 1 -type d -empty -delete

rm /mnt/user/appdata/other/rclone/rclone_cleanup

exit

 

Link to comment
1 hour ago, privateer said:

Thanks for all the help. I'm (obviously) a newbie so this is much appreciated. When I create the movies and tv_shows folders (and moving files into them) I'm using Krusader. Could this be causing problems? Do I need to use CLI?

 

rclone config:

 

Mount script

 

Upload script

 

Unmount script

 

Cleanup script (haven't paid much attention as I haven't got anything to upload yet)

 

All looks ok to me - maybe someone else will spot something @Kaizac ??. 

 

What happens if you create a few test folders in:

 

(i) /mnt/user/mount_unionfs/google_vfs

(ii) directly in /mnt/user/rclone_upload/google_vfs

 

Do they appear in /mnt/user/mount_rclone/google_vfs?  (change --min-age 30m to --min-age 1m or delete in the upload script to test)

 

The only thing I can think of is there was a weird gdrive issue last week where uploads from Europe were blocked for a day, but this has been resolved as far as I know

 

Link to comment

Want to respond here in case other people run into this problem in the future.

 

I'm using unraid as my OS. I was creating folders and moving files with a Krusader docker. This appears to be the source of the problem. When I used CLI to create the folders and move the files everything appeared to run correctly. File is uploading now.

Link to comment
4 minutes ago, privateer said:

Want to respond here in case other people run into this problem in the future.

 

I'm using unraid as my OS. I was creating folders and moving files with a Krusader docker. This appears to be the source of the problem. When I used CLI to create the folders and move the files everything appeared to run correctly. File is uploading now.

oh yeah, Krusader seems to have problems if you don't restart it.  I don't use so I'm not sure why - I use mc to move files

Edited by DZMM
Link to comment
On 5/19/2019 at 4:43 PM, DZMM said:

16gb should be enough.  I'd disable the scripts and reboot.  Then delete any residual checker files form /mnt/user/appdata/rclone (i think that's where all the script checks are) and then enable the scripts.

Hi DZMM,

 

Would it be possible to add in the script if the mount failed an email is sent ? 

 

Is it something difficult to do.

 

I don't know nothing about scripting

 

thx

 

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.