Jump to content
DZMM

Guide: How To Use Rclone To Mount Cloud Drives And Play Files

357 posts in this topic Last Reply

Recommended Posts

That totally depends on your download speed and your processing power of your server. And if you want to stream outside of your home it also depends on your upload speed. If you can mostly direct stream (no transcoding) it will depend 90% on your download/upload speed.
Ok i understand thx for info

Envoyé de mon Pixel 2 XL en utilisant Tapatalk

Share this post


Link to post

Back home after holiday travels and again my mount seems to have failed after a reboot / power outage. I've fixed it manually and haven't tested a reboot yet but now have a problem with ```/var/log is getting full (currently 100 % used)``` my syslog being full of rclone entries is all I can really notice.

I've changed log level from DEBUG to WARN but do I need to reboot to test this or will it use changed log level on next run? I don't really want to reboot for a few days right now if I can avoid it. Or I should say, due to the size I do not really want to rebuild it if it fails again.

Share this post


Link to post
7 hours ago, Spladge said:

Back home after holiday travels and again my mount seems to have failed after a reboot / power outage. I've fixed it manually and haven't tested a reboot yet but now have a problem with ```/var/log is getting full (currently 100 % used)``` my syslog being full of rclone entries is all I can really notice.

I've changed log level from DEBUG to WARN but do I need to reboot to test this or will it use changed log level on next run? I don't really want to reboot for a few days right now if I can avoid it. Or I should say, due to the size I do not really want to rebuild it if it fails again.

if you unmount and remount your log level will refresh

Share this post


Link to post
2 hours ago, DZMM said:

if you unmount and remount your log level will refresh

What do you have your remount script set on? I think when an "de-mount" happens it's pretty hard to get a working mount again. Dockers will be moving files to the mount_unionfs folders and rclone will see it as not being empty thus failing. I've put my remount script on 1 minute, to try to mitigate this, but I think there is a big chance it would fail anyways.

Share this post


Link to post
Posted (edited)

Unmount then run mount script in background? Or will this require a server restart?

 

Kaizac Is this the

'sleep 5'

setting, ie sleep 60.

Edited by Spladge

Share this post


Link to post
4 minutes ago, Spladge said:

Unmount then run mount script in background? Or will this require a server restart?

 

Kaizac Is this the


'sleep 5'

setting, ie sleep 60.

When dockers are not active/moving files in the union folder then you can unmount and remount through running the script in the background. The script will notify you if it succeeded or not. Most reliable is a full restart of the server of course, but not necessary when you are just remounting previous mounts.

 

The sleep is a waiting time build in the execution of the script. Some things need some time to load up properly (like the mounting). This depends on your own server speed and plugins you run. For example I run the Recycle Bin plugin which can take up to 5-10 minutes to load properly. So if you run docker start you need to put in sleeps for that period or that part of the script won't execute.

 

So you just need to experiment a bit with that.

 

What I was asking DZMM is the cron job timing for the mounting script after it started at array. So I have a mount script at start of array. Then I also run the same script every minute again in case of an unmount.

Share this post


Link to post

AH - thank you very much, answered a question I didn't even know I needed to ask yet.

Share this post


Link to post
Posted (edited)

When I want to manually remount I create a temp script to run in the background:

 

fusermount -uz /mnt/user/mount_rclone/google_vfs

rclone mount --allow-other --buffer-size 512M --dir-cache-time 72h --drive-chunk-size 512M --fast-list --log-level INFO --vfs-read-chunk-size 128M --vfs-read-chunk-size-limit off tdrive1_vfs: /mnt/user/mount_rclone/google_vfs &

exit

I don't add any sleeps in and I've not had any problems as the mount is remounted before any dockers realise.  Ongoing uploads are fine as they don't use the mount.

 

The sleep in the main mount script is just to avoid any potential problems - it's probably not needed anymore, but I keep it there for insurance.

 

Like @Kaizac I have my script running every 5 mins to check if the mount has dropped.  I think this has only happened once before to me in the early days, but it's there if I have e.g. a network failure in the future

Edited by DZMM

Share this post


Link to post
1 hour ago, Spladge said:

What is the checker script?

just the mount script - it's designed to remount if the mount drops automatically if run on a schedule e.g. every 5 mins

Share this post


Link to post

What you guys think about a small script which checkes if mountcheck file is there, and if not closes all related dockers (atleast plex)? :)

Share this post


Link to post
33 minutes ago, nuhll said:

What you guys think about a small script which checkes if mountcheck file is there, and if not closes all related dockers (atleast plex)? :)

I guess you could add that in, but if you set my mount script to run on a schedule it should remount automatically. 

 

I've been running my setup for over 6 months and I think my mount has only dropped once and that was probably my fault when I was figuring stuff out.

Share this post


Link to post
Posted (edited)
1 hour ago, DZMM said:

I guess you could add that in, but if you set my mount script to run on a schedule it should remount automatically. 

 

I've been running my setup for over 6 months and I think my mount has only dropped once and that was probably my fault when I was figuring stuff out.

 

Yes ive only use ur scripts, just changed here and there, like upload script, still the question is, if it fails to mount for whatever reason, and this whatever reason doenst get automaticly resolved, what happens e.g. to the plex library.

 

Ive set the "dont auto remove" like u suggested, but i already found a flaw in it... if e.g. a series gets updated from xxxP to 1080p, plex shows it as missing (the old file) which is kinda annoying... so i wanna enable it again.

 

I guess you could add it to your script like

 

if not mounted, mount, if that still not helps, close dockers, sleep 10min. Since your script also starts the dockers, this would jsut make sense.

Edited by nuhll

Share this post


Link to post
49 minutes ago, nuhll said:

Yes ive only use ur scripts, just changed here and there, like upload script, still the question is, if it fails to mount for whatever reason, and this whatever reason doenst get automaticly resolved, what happens e.g. to the plex library.

 

Ive set the "dont auto remove" like u suggested, but i already found a flaw in it... if e.g. a series gets updated from xxxP to 1080p, plex shows it as missing (the old file) which is kinda annoying... so i wanna enable it again.

I just click 'empty trash' every day or so.  IMO that's a much better solution than plex automatically removing all media if a mount fails and you loose all manual metadata corrections, manual artwork changes etc

 

51 minutes ago, nuhll said:

if not mounted, mount, if that still not helps, close dockers, sleep 10min. Since your script also starts the dockers, this would jsut make sense.

I don't think any script could do this fast enough 100% of the time before Plex decides a file is missing.  As I stated I think my mount has only gone down once in 6 months, so if you hate the plex behaviour and want to take the chance then turn on auto remove.

Share this post


Link to post
Posted (edited)

We could make a script which runs all the time and just checks for the mountcheck file, this would instant stop plex if neccessary. 

 

while [ ! -f /tmp/list.txt ];

do

sleep 1;

done;

 

Would be like 1 or 2 sec delay, so fast he cant delete sometzhing, and even if, its  unlikly its something gets deleted which was manually added.

Edited by nuhll

Share this post


Link to post
Posted (edited)

DZMM I noticed that you have two different rclone mount's on your github page and the first page of this discussion. Which one are you currently using?

Secondly I'm having a issue where when I start my array, unionfs doesn't seem to be mounting, but when I run the script manually in the background it works fine. Here it is.

Quote

 

#!/bin/bash

#######  Check if script already running  ##########
 

if [[ -f "/mnt/user/Media/Cloud/rclone_install_running" ]]; then
 

echo "$(date "+%d.%m.%Y %T") INFO: Exiting script already running."
 

exit
 

else
 

touch /mnt/user/Media/Cloud/rclone_install_running
 

fi
 

## End Check ##
 

mkdir -p /mnt/disks/crypt
 

#######  Check if rclone vfs mount is mounted  ##########
 

if [[ -f "/mnt/disks/crypt/Movies/mountcheck" ]] && [[ -f "/mnt/disks/crypt/Series/mountcheck" ]]; then

echo "$(date "+%d.%m.%Y %T") INFO: Check rclone_vfs mounted success."
 

else
 

echo "$(date "+%d.%m.%Y %T") INFO: mounting rclone."
 

# Mount rclone vfs mount
 

rclone mount --allow-other --dir-cache-time 72h --vfs-read-chunk-size 64M --vfs-read-chunk-size-limit off --buffer-size 1G --log-level INFO --stats 1m uploadcrypt: /mnt/disks/crypt --stats 1m &
 

sleep 5
 

if [[ -f "/mnt/disks/crypt/Movies/mountcheck" ]] && [[ -f "/mnt/disks/crypt/Series/mountcheck" ]]; then

echo "$(date "+%d.%m.%Y %T") INFO: rclone_vfs mount success."
 

else
 

echo "$(date "+%d.%m.%Y %T") CRITICAL: rclone_vfs mount failed - please check for problems."
 

rm /mnt/user/Media/Cloud/rclone_install_running
 

exit
 

fi
 

fi
 

#######  Mount unionfs   ##########
 

# check if mounted
 

if [[ -f "/mnt/user/Media/Cloud/Series/mountcheck" ]] && [[ -f "/mnt/user/Media/Cloud/Movies/mountcheck" ]]; then
echo "$(date "+%d.%m.%Y %T") INFO: Check successful, unionfs Movies & Series mounted."
 

else
 

# Unmount before remounting
fusermount -uz /mnt/user/Media/Cloud/Series
fusermount -uz /mnt/user/Media/Cloud/Movies


unionfs -o cow,allow_other,direct_io,auto_cache,sync_read /mnt/user/Media/Cloud/Seriestmp=RW:/mnt/disks/crypt/Series=RO /mnt/user/Media/Cloud/Series
unionfs -o cow,allow_other,direct_io,auto_cache,sync_read /mnt/user/Media/Cloud/Moviestmp=RW:/mnt/disks/crypt/Movies=RO /mnt/user/Media/Cloud/Movies
 

if [[ -f "/mnt/user/Media/Cloud/Series/mountcheck" ]] && [[ -f "/mnt/user/Media/Cloud/Movies/mountcheck" ]]; then
echo "$(date "+%d.%m.%Y %T") INFO: Check successful, unionfs Movies & Series mounted."

else

echo "$(date "+%d.%m.%Y %T") CRITICAL: unionfs Movies & Series Remount failed."
 

rm /mnt/user/Media/Cloud/rclone_install_running
 

exit
 

fi
 

fi
 

#######  End Mount unionfs   ##########
 

rm /mnt/user/Media/Cloud/rclone_install_running
 

exit

 




Got this error after adding 

rclone mount --rc --allow-other --dir-cache-time 72h --vfs-read-chunk-size 128M --drive-chunk-size 256M --fast-list --vfs-read-chunk-size-limit off --buffer-size 1G --log-level INFO uploadcrypt: /mnt/disks/crypt &

# Populate rclone dir-cache 
rclone rc --timeout=1h vfs/refresh recursive=true &

exit

Quote

2019/01/05 18:27:45 NOTICE: Serving remote control on http://127.0.0.1:5572/
2019/01/05 18:27:45 ERROR : rc: "vfs/refresh": error: couldn't find method "vfs/refresh"
2019/01/05 18:27:45 Failed to rc: Failed to read rc response: 404 Not Found: {
"error": "couldn't find method \"vfs/refresh\"",
"input": {
"recursive": "true"
},
"path": "vfs/refresh",
"status": 404
}

 

Edited by slimshizn

Share this post


Link to post
Posted (edited)

Okay I ran the unmount script and tried it again. This is what I'm getting. 

I ran the script, then saw it said please check for problems, I ran it a SECOND time and thats when it showed the syntax error...BUT I checked my mount and everything appeared. 

 

Quote

Script Starting Sat, 05 Jan 2019 18:34:11 -0500

Full logs for this script are available at /tmp/user.scripts/tmpScripts/Rclone_install_DZMM/log.txt

05.01.2019 18:34:11 INFO: mounting rclone.
2019/01/05 18:34:11 NOTICE: Serving remote control on http://127.0.0.1:5572/
05.01.2019 18:34:16 CRITICAL: rclone_vfs mount failed - please check for problems.
Script Finished Sat, 05 Jan 2019 18:34:16 -0500

Full logs for this script are available at /tmp/user.scripts/tmpScripts/Rclone_install_DZMM/log.txt

2019/01/05 18:34:20 INFO : 
Transferred: 0 / 0 Bytes, -, 0 Bytes/s, ETA -
Errors: 0
Checks: 0 / 0, -
Transferred: 0 / 0, -
Elapsed time: 5h6m10.6s

Script Starting Sat, 05 Jan 2019 18:34:57 -0500

Full logs for this script are available at /tmp/user.scripts/tmpScripts/Rclone_install_DZMM/log.txt

05.01.2019 18:34:57 INFO: Check rclone_vfs mounted success.
fusermount: failed to unmount /mnt/user/Media/Cloud/Series: Invalid argument
fusermount: failed to unmount /mnt/user/Media/Cloud/Movies: Invalid argument
05.01.2019 18:34:58 INFO: Check successful, unionfs Movies & Series mounted.
/tmp/user.scripts/tmpScripts/Rclone_install_DZMM/script: line 85: syntax error near unexpected token `fi'
/tmp/user.scripts/tmpScripts/Rclone_install_DZMM/script: line 85: `fi'
Script Finished Sat, 05 Jan 2019 18:34:58 -0500

Full logs for this script are available at /tmp/user.scripts/tmpScripts/Rclone_install_DZMM/log.txt

{
"result": {
"": "OK"
}
}


I found the extra "fi" and removed it. 

 

Quote

05.01.2019 18:42:52 INFO: mounting rclone.
2019/01/05 18:42:52 NOTICE: Serving remote control on http://127.0.0.1:5572/
05.01.2019 18:42:58 INFO: rclone_vfs mount success.
fusermount: failed to unmount /mnt/user/Media/Cloud/Series: Invalid argument
fusermount: failed to unmount /mnt/user/Media/Cloud/Movies: Invalid argument
05.01.2019 18:42:58 INFO: Check successful, unionfs Movies & Series mounted.
Script Finished Sat, 05 Jan 2019 18:42:58 -0500

Full logs for this script are available at /tmp/user.scripts/tmpScripts/Rclone_install_DZMM/log.txt

{
"result": {
"": "OK"
}



Oddly, the file that's created to show that it's running didn't delete. I had to go in and remove it manually. After that It ran like above and seems to be working fine again! Will report back if there's errors again after a reboot in the future.

Edited by slimshizn

Share this post


Link to post
33 minutes ago, slimshizn said:

DZMM I noticed that you have two different rclone mount's on your github page and the first page of this discussion

Ignore the one in the first post - the github is the up to date version.

 

34 minutes ago, slimshizn said:

Secondly I'm having a issue where when I start my array,

how soon do you run the script after an array start?  Mine's on a 5 min cycle - does it work the second time automatically?  Maybe it starts too close to the array starting

 

35 minutes ago, slimshizn said:

Got this error after adding 

rclone mount --rc --allow-other --dir-cache-time 72h --vfs-read-chunk-size 128M --drive-chunk-size 256M --fast-list --vfs-read-chunk-size-limit off --buffer-size 1G --log-level INFO uploadcrypt: /mnt/disks/crypt &

# Populate rclone dir-cache 
rclone rc --timeout=1h vfs/refresh recursive=true &

exit

I've ditched rc as the refresh was just a nice to have and rc and wasn't worth the hassle

 

13 minutes ago, slimshizn said:

I ran the script, then saw it said please check for problems, I ran it a SECOND time and thats when it showed the syntax error...BUT I checked my mount and everything appeared. 

what's on line 85?

Share this post


Link to post

"fi" was on line 85, has been removed and working fine now. 

Seems to be all good, I'll set a custom time for it to run again after 10 minutes.

Editing some scripts to update them and came across this portion of your upload..

 

Quote

--delete-empty-src-dirs --fast-list --bwlimit 9500k --tpslimit 3 --min-age 30

Now I don't know if you remember our discussion where rclone would "download" files that were deleted from my server after an upload. Really odd thing that was going on. I believe you had me remove the delete portion and it seemed to work fine but I see it's changed to checkers 3, tpslimit 3 which I don't know what that is and min-age 30. 

Share this post


Link to post

Current rclone script

Quote

 

#!/bin/bash

#######  Check if script already running  ##########

if [[ -f "/mnt/user/Media/Cloud/rclone_install_running" ]]; then

echo "$(date "+%d.%m.%Y %T") INFO: Exiting script already running."

exit

else

touch /mnt/user/Media/Cloud/rclone_install_running

fi

## End Check ##

mkdir -p /mnt/disks/crypt

#######  Check if rclone vfs mount is mounted  ##########

if [[ -f "/mnt/disks/crypt/Movies/mountcheck" ]] && [[ -f "/mnt/disks/crypt/Series/mountcheck" ]]; then
echo "$(date "+%d.%m.%Y %T") INFO: Check rclone_vfs mounted success."

else

echo "$(date "+%d.%m.%Y %T") INFO: mounting rclone."

# Mount rclone vfs mount

rclone mount --rc --allow-other --dir-cache-time 72h --vfs-read-chunk-size 128M --drive-chunk-size 256M --fast-list --vfs-read-chunk-size-limit off --buffer-size 1G --log-level INFO uploadcrypt: /mnt/disks/crypt &

sleep 5

if [[ -f "/mnt/disks/crypt/Movies/mountcheck" ]] && [[ -f "/mnt/disks/crypt/Series/mountcheck" ]]; then
echo "$(date "+%d.%m.%Y %T") INFO: rclone_vfs mount success."

else

echo "$(date "+%d.%m.%Y %T") CRITICAL: rclone_vfs mount failed - please check for problems."

rm /mnt/user/Media/Cloud/rclone_install_running

exit

fi

fi

#######  Mount unionfs   ##########

# check if mounted

if [[ -f "/mnt/user/Media/Cloud/Series/mountcheck" ]] && [[ -f "/mnt/user/Media/Cloud/Movies/mountcheck" ]]; then
echo "$(date "+%d.%m.%Y %T") INFO: Check successful, unionfs Movies & Series mounted."

else

# Unmount before remounting
fusermount -uz /mnt/user/Media/Cloud/Series
fusermount -uz /mnt/user/Media/Cloud/Movies


unionfs -o cow,allow_other,direct_io,auto_cache,sync_read /mnt/user/Media/Cloud/Seriestmp=RW:/mnt/disks/crypt/Series=RO /mnt/user/Media/Cloud/Series
unionfs -o cow,allow_other,direct_io,auto_cache,sync_read /mnt/user/Media/Cloud/Moviestmp=RW:/mnt/disks/crypt/Movies=RO /mnt/user/Media/Cloud/Movies

if [[ -f "/mnt/user/Media/Cloud/Series/mountcheck" ]] && [[ -f "/mnt/user/Media/Cloud/Movies/mountcheck" ]]; then
echo "$(date "+%d.%m.%Y %T") INFO: Check successful, unionfs Movies & Series mounted."
else
echo "$(date "+%d.%m.%Y %T") CRITICAL: unionfs Movies & Series Remount failed."

rm /mnt/user/Media/Cloud/rclone_install_running

exit

fi

fi

# Populate rclone dir-cache

rclone rc --timeout=1h vfs/refresh recursive=true &

#######  End Mount unionfs   ##########

rm /mnt/user/Media/Cloud/rclone_install_running

exit

 


Current upload script

 

Quote

 

#!/bin/bash

#######  Check if script already running  ##########

if [[ -f "/mnt/user/Media/Cloud/rclone_upload_running" ]]; then
exit
else
touch /mnt/user/Media/Cloud/rclone_upload_running

fi

# set folders

uploadfolderMovies="/mnt/user/Media/Cloud/Moviestmp" 
uploadfolderSeries="/mnt/user/Media/Cloud/Seriestmp" 

# move files

rclone move $uploadfolderSeries uploadcrypt:/Series -vv --drive-chunk-size 512M --checkers 10 --fast-list --transfers 4 --exclude .unionfs/** --exclude *fuse_hidden* --exclude *_HIDDEN --exclude .recycle/** --exclude *.backup~* --exclude *.partial~* --bwlimit 1.25M
rclone move $uploadfolderMovies uploadcrypt:/Movies -vv --drive-chunk-size 512M --checkers 10 --fast-list --transfers 4 --exclude .unionfs/** --exclude *fuse_hidden* --exclude *_HIDDEN --exclude .recycle/** --exclude *.backup~* --exclude *.partial~* --bwlimit 1.25M

rm /mnt/user/Media/Cloud/rclone_upload_running

exit


 

 

Share this post


Link to post
10 hours ago, slimshizn said:

Now I don't know if you remember our discussion where rclone would "download" files that were deleted from my server after an upload. Really odd thing that was going on. I believe you had me remove the delete portion and it seemed to work fine but I see it's changed to checkers 3, tpslimit 3 which I don't know what that is and min-age 30. 

rclone fixed delete-empty-src-dirs a few releases ago, so I put it back in.  min-age 30 stops files younger than 30 mins being uploaded.  I reduced my checkers and tpslimits to reduce the odds of API bans - they probably could be increased.

Share this post


Link to post

the create and remove directorys is also a known problem... 

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now