Guide: How To Use Rclone To Mount Cloud Drives And Play Files


DZMM

Recommended Posts

34 minutes ago, mestep said:

 

I don't mean to sound stupid or anything. If I put stuff in the union folder instead of the upload folder, it will still get uploaded to the cloud, right? Right now I have sabnzbd set to download to the upload folder, can/should I have it set to download to union?

No it will not. In your union you define which folder is Read Only (RO) and which one is Read Write (RW). So if you followed the tutorial you will have your local folder as your RW folder and your remote is RO. When you then move files to the union folder it will be written to your local folder. So all your dockers like Sab you can point to your union folder as through the union it knows it has to move the files to the rclone_upload folder.

 

In this setup files only get moved through using a script. If you want to direct write to your mount, you will have to do that through the mount_rclone folder. Through which you are directly accessing your cloud folders.

 

Makes sense?

Link to comment

Alright, I think I finally understand everything now. Thank you for all your help!

 

It just took me too long to understand the simple concept of:

 

mount_rclone = cloud files

rclone_upload = local files

mount_unionfs = combines mount_rclone and rclone_upload

 

One final thing. I saw someone post this earlier, but never got a definitive answer. The cleanup script isn't working:

 

rm: cannot remove '/mnt/user/mount_rclone/google_vfs/4k Movies/FOLDER': Is a directory
rm: cannot remove '/mnt/user/mount_unionfs/google_vfs/.unionfs/4k_ Movies/FOLDER_HIDDEN~': Is a directory

 

Not sure what is going wrong with it.  It deletes the files but leaves the directories.

 

Edited by mestep
Link to comment
On 2/14/2019 at 3:13 PM, mestep said:

Alright, I think I finally understand everything now. Thank you for all your help!

 

It just took me too long to understand the simple concept of:

 

mount_rclone = cloud files

rclone_upload = local files

mount_unionfs = combines mount_rclone and rclone_upload

 

One final thing. I saw someone post this earlier, but never got a definitive answer. The cleanup script isn't working:

 

rm: cannot remove '/mnt/user/mount_rclone/google_vfs/4k Movies/FOLDER': Is a directory
rm: cannot remove '/mnt/user/mount_unionfs/google_vfs/.unionfs/4k_ Movies/FOLDER_HIDDEN~': Is a directory

 

Not sure what is going wrong with it.  It deletes the files but leaves the directories.

 

Not sure what's going wrong.  Does your cleanup script look like this?  (i use a teamdrive)

 

################### Clean-up UnionFS Folder  #########################

echo "$(date "+%d.%m.%Y %T") INFO: starting unionfs cleanup."

find /mnt/user/mount_unionfs/google_vfs/.unionfs -name '*_HIDDEN~' | while read line; do
oldPath=${line#/mnt/user/mount_unionfs/google_vfs/.unionfs}
newPath=/mnt/user/mount_rclone/tdrive_vfs${oldPath%_HIDDEN~}
rm "$newPath"
rm "$line"
done

find "/mnt/user/mount_unionfs/google_vfs/.unionfs" -mindepth 1 -type d -empty -delete

rm /mnt/user/appdata/other/rclone/rclone_mount_cleanup

exit

 

Link to comment

I have also a question. Cleanups cript is also not working for me, u gave me a changed script, no change.

 

But i wonder if i really need it.

 

All mountpoints where i write data to are RW mounted, so a DEL should delete it on all points (local and remote), correct?
 

Quote

 


unionfs -o cow,allow_other,direct_io,auto_cache,sync_read /mnt/user/rclone_upload/google_vfs=RO:/mnt/user/Archiv=RW:/mnt/user/mount_rclone/google_vfs=RO /mnt/user/mount_unionfs/google_vfs

 

 

I only add data to /mnt/user/Archiv

 

and upload it later via upload script.

Edited by nuhll
Link to comment
25 minutes ago, nuhll said:

I have also a question. Cleanups cript is also not working for me, u gave me a changed script, no change.

 

But i wonder if i really need it.

 

All mountpoints where i write data to are RW mounted, so a DEL should delete it on all points (local and remote), correct?
 

 

I only add data to /mnt/user/Archiv

 

and upload it later via upload script.

depends what you are uploading.  If you've uploaded a movie and then replace it with a better copy, this will be hidden and 'deleted' from the unionfs folder, but not from the cloud folder.  So, if you ever mount the cloud folder elsewhere or differently, suddenly all the files you thought you'd deleted would reappear.

Link to comment
4 hours ago, DZMM said:

Not sure what's going wrong.  Does your cleanup script look like this?  (i use a teamdrive)

 

Mine is basically the same. I am not using teamdrive:

 

 

#!/bin/bash
touch /mnt/user/appdata/other/rclone/rclone_mount_cleanup
################### Clean-up UnionFS Folder  #########################
echo "$(date "+%d.%m.%Y %T") INFO: starting unionfs cleanup."
find /mnt/user/mount_unionfs/google_vfs/.unionfs -name '*_HIDDEN~' | while read line; do
oldPath=${line#/mnt/user/mount_unionfs/google_vfs/.unionfs}
newPath=/mnt/user/mount_rclone/google_vfs${oldPath%_HIDDEN~}
rm "$newPath"
rm "$line"
done
find "/mnt/user/mount_unionfs/google_vfs/.unionfs" -mindepth 1 -type d -empty -delete
rm /mnt/user/appdata/other/rclone/rclone_mount_cleanup
exit

I delete the "test" folder in the root of mount_unionfs/google_vfs/ and run the script, this is the log:

15.02.2019 15:37:19 INFO: starting unionfs cleanup.
rm: cannot remove '/mnt/user/mount_rclone/google_vfs/test': Is a directory
rm: cannot remove '/mnt/user/mount_unionfs/google_vfs/.unionfs/test_HIDDEN~': Is a directory
Script Finished Fri, 15 Feb 2019 15:37:19 -0500

Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_cleanup/log.txt

 

I think it is a permissions issue, but I don't know enough about linux to figure it out.

Edited by mestep
Link to comment
22 hours ago, DZMM said:

depends what you are uploading.  If you've uploaded a movie and then replace it with a better copy, this will be hidden and 'deleted' from the unionfs folder, but not from the cloud folder.  So, if you ever mount the cloud folder elsewhere or differently, suddenly all the files you thought you'd deleted would reappear.

Okay, so this would be a problem, or lets say inconviniece then.

 

Which folder i need to check if it gets deleted correct? (like where they are hidden?`)

 

I get this error on running (but it might be an old file which was not uploaded by upload script)

16.02.2019 10:38:43 INFO: starting unionfs cleanup.
rm: cannot remove '/mnt/user/mount_rclone/google_vfs/Filme/some/thing.mkv': No such file or directory
rm: cannot remove '/mnt/user/rclone_upload/google_vfs/Filme/some/thing.mkv': No such file or directory
rm: cannot remove '/mnt/user/mount_unionfs/google_vfs/.unionfs/Filme/some/thing_HIDDEN~': Permission denied

 

But i cant manually RM it because it says no file... :D

 

Edited by nuhll
Link to comment
37 minutes ago, nuhll said:

Okay, so this would be a problem, or lets say inconviniece then.

 

Which folder i need to check if it gets deleted correct? (like where they are hidden?`)

 

I get this error on running (but it might be an old file which was not uploaded by upload script)

16.02.2019 10:38:43 INFO: starting unionfs cleanup.
rm: cannot remove '/mnt/user/mount_rclone/google_vfs/Filme/1922 (2017)/1922 (2017).mkv': No such file or directory
rm: cannot remove '/mnt/user/rclone_upload/google_vfs/Filme/1922 (2017)/1922 (2017).mkv': No such file or directory
rm: cannot remove '/mnt/user/mount_unionfs/google_vfs/.unionfs/Filme/1922 (2017)/1922 (2017).mkv_HIDDEN~': Permission denied

 

But i cant manually RM it because it says no file... :D

 

I'm not sure why people are getting this error - try running the fix permissions tool and check your radarr permissions settings - mine's at 777. 


To manually check if working browse the mount mount_rclone folder directly - test by moving a file from the unionfs folder to a non-mount folder, run the script, check the rclone mount to see if deleted and then move the file back to rclone_upload folder so it gets uploaded again

Link to comment

I unfortunately had a powercut when the script was uploading to gdrive, now I keep getting this in the logs:

2019/02/16 23:22:28 NOTICE: Serving remote control on http://127.0.0.1:5572/
2019/02/16 23:22:29 NOTICE: pacer: Invalid error in fs.Pacer: %!t()
2019/02/16 23:22:29 NOTICE: pacer: Invalid error in fs.Pacer: %!t()

2019/02/16 23:22:29 NOTICE: pacer: Invalid error in fs.Pacer: %!t()

 

So I ran all the unmount/cleanup etc scripts, uninstalled the rclone beta (including the config files in the boot folder) and redid my gdrive and crypt remotes but it keeps on spamming this message still, I cannot find a single entry on google about it.

I get this message even after starting again - ie deleting the remotes and recreating them, then upon entering the 2nd command with your setup routine (ie "rclone copy mountcheck gdrive_media_vfs: -vv --no-traverse", i get the same error message - so it must be an rclone issue 

 

I did notice that it is showing port 5572 for the remote control above (at top of my post) and when I recreated a gdrive remote and did the token refresh it shows a different port ie 53682   http://127.0.0.1:53682/auth but not sure if that is relevant to the error or not..

 

 

 

Edited by shimlapinks
Link to comment
  • 4 weeks later...

ok, so I got everything configured .. however when I run the upload script I get the following: 

 

Script Starting Tue, 12 Mar 2019 22:35:24 -0400

Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_gdrive_upload/log.txt

/tmp/user.scripts/tmpScripts/rclone_gdrive_upload/script: line 4: $'\357\273\277': command not found
/tmp/user.scripts/tmpScripts/rclone_gdrive_upload/script: line 26: $'\357\273\277': command not found
12.03.2019 22:35:24 INFO: rclone not installed - will try again later.
Script Finished Tue, 12 Mar 2019 22:35:24 -0400

Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_gdrive_upload/log.txt

 

I also get this when running the cleanup script: 

 

Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_gdrive_cleanup/log.txt

12.03.2019 22:44:55 INFO: starting unionfs cleanup.
find: `/mnt/user/mount_unionfs/google_vfs/.unionfs': No such file or directory
find: `/mnt/user/mount_unionfs/google_vfs/.unionfs': No such file or directory
Script Finished Tue, 12 Mar 2019 22:44:55 -0400

Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone_gdrive_cleanup/log.txt

 

both the mount and unmount script seems to run fine.

 

any advice ? 

 

thanks

 

edit ------------------------------

 

so if I just change the entries to remove one of the bracket set, (if [ -f "/mnt/user/appdata/other/rclone/rclone_upload" ]; then) script seems to run.

Edited by scloutier
issue
Link to comment

If u remove that bracket the script will run over and over again starting unlimited concurrent uploads. (bad idea)

 



/tmp/user.scripts/tmpScripts/rclone_gdrive_upload/script: line 4: $'\357\273\277': command not found
/tmp/user.scripts/tmpScripts/rclone_gdrive_upload/script: line 26: $'\357\273\277': command not found

 

Its due to wrong text encoding.

 

Be sure to copy it from github and place it in a e.g. vi shell, then it should work. (like "vi" your script file and replace it with the one from github)

OR use config editor plugin.

 

 



12.03.2019 22:35:24 INFO: rclone not installed - will try again later.
 

also this means that the mount didnt work correct.

Edited by nuhll
Link to comment

I tried to copy both from github in raw directly into putty, and also tried to paste it in the unraid config editor.  results are the same.  will see if I can download it directly from the shell then.

 

I'm also confused as the if man page only shows single bracket (https://ss64.com/bash/syntax-file-operators.html

 

their example: 

 

Check if a file exists:

if [ -f ~/some-file ]; then
 Echo the file exists 
fi

which also has no " " around the file path"

 

thanks

Edited by scloutier
Link to comment

alright .. I used VI to paste the script.  same issue, running out of idea .. will try to wget the file directly from the shell and rename see if that works, however running out of ideas.

 

so what's weird is that part of it works .. 

 

/tmp/user.scripts/tmpScripts/rclone_gdrive_upload/script: line 4: $'\357\273\277': command not found
13.03.2019 15:00:01 INFO: rclone installed successfully - proceeding with upload.

 

the first if file check for the rclone_upload fails .. the second one that looks for mountcheck seems to work 

 

Edited by scloutier
Link to comment

ok, so using cat -v filename actually shows where the issues are .. in the upload script, issue was inside two blank lines.

 

will go through the other script.  really weird issue.

 

 

edit

 

seems like going through and removing any broken portion and retyping worked.

Edited by scloutier
  • Like 1
Link to comment

Locally I've been using unionfs and rclone mount as stated in this guide. My cloud dedicated server is running plexdrive V4 and unionfs. So far this has been a great combination but I think i'd like to move onto the mergefs and rclone_vfs and off of plexdrive/unionfs due to some of the benefits.

Has anyone been running this setup?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.