Guide: How To Use Rclone To Mount Cloud Drives And Play Files

[Kaizac]

Thanks! I was already transferring it from mount_rclone, but then I transferred it to the mount_unionfs which caused the problem. Transferring with Krusader from mount_rclone to mount_rclone works perfect.

[NewDisplayName]

On 11/29/2018 at 6:59 PM, DZMM said:

It's saying rclone isn't installed - check plugins

Yes, but its just telling it bc:

#######  check if rclone installed  ##########

if [[ -f "/mnt/user/mount_rclone/google_vfs/mountcheck" ]]; then

echo "$(date "+%d.%m.%Y %T") INFO: rclone installed successfully - proceeding with upload."

else

echo "$(date "+%d.%m.%Y %T") INFO: rclone not installed - will try again later."

rm /mnt/user/appdata/other/rclone/rclone_upload

 

Even the mounting went find - besides the RC failure. Also i could upload the mountcheck file via rsync just fine.

[Kaizac]

1 hour ago, nuhll said:

Yes, but its just telling it bc:

#######  check if rclone installed  ##########

if [[ -f "/mnt/user/mount_rclone/google_vfs/mountcheck" ]]; then

echo "$(date "+%d.%m.%Y %T") INFO: rclone installed successfully - proceeding with upload."

else

echo "$(date "+%d.%m.%Y %T") INFO: rclone not installed - will try again later."

rm /mnt/user/appdata/other/rclone/rclone_upload

 

Even the mounting went find - besides the RC failure. Also i could upload the mountcheck file via rsync just fine.

I'm having difficulty understanding what exactly your problem is. But seeing an earlier post of you it appears to me you execute the scripts by clicking "Run script" which you shouldn't do. Use the "Run in the background" button and then check log.

[NewDisplayName]

Yes, i use run script. Didnt know, that ic ouldnt test it this way.

 

My problem is just that after it initiatlly worked one time, i couldnt get it back to upload something.

[Kaizac]

Just now, nuhll said:

Yes, i use run script. Didnt know, that ic ouldnt test it this way.

 

My problem is just that after it initiatlly worked one time, i couldnt get it back to upload something.

Run it in the background and then on the right you can download the log. Can you share that here? And your mounting config and upload config?

[NewDisplayName]

20 minutes ago, Kaizac said:

Run it in the background and then on the right you can download the log. Can you share that here? And your mounting config and upload config?

As in the tutorial, its all standard now. I just changed RC port, but thats optional anyway.

 

root@Unraid-Server:~# rclone copy mountcheck gdrive_media_vfs: -vv --no-traverse

2018/12/01 16:27:19 NOTICE: --no-traverse is obsolete and no longer needed - please remove
2018/12/01 16:27:19 DEBUG : rclone: Version "v1.45-009-g4b15c421-beta" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone-beta/.rclone.conf" "copy" "mountcheck" "gdrive_media_vfs:" "-vv" "--no-traverse"]
2018/12/01 16:27:19 DEBUG : Using config file from "/boot/config/plugins/rclone-beta/.rclone.conf"
2018/12/01 16:27:24 DEBUG : pacer: Rate limited, sleeping for 1.806655311s (1 consecutive low level retries)
2018/12/01 16:27:24 DEBUG : pacer: low level retry 1/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2018/12/01 16:27:24 DEBUG : pacer: Rate limited, sleeping for 2.209989177s (2 consecutive low level retries)
2018/12/01 16:27:24 DEBUG : pacer: low level retry 2/10 (error googleapi: Error 403: Rate Limit Exceeded, rateLimitExceeded)
2018/12/01 16:27:26 DEBUG : pacer: Resetting sleep to minimum 10ms on success
2018/12/01 16:27:27 DEBUG : mountcheck: Modification times differ by -45h52m26.035137444s: 2018-12-01 16:25:43.600137444 +0100 CET, 2018-11-29 17:33:17.565 +0000 UTC
2018/12/01 16:27:28 INFO  : mountcheck: Copied (replaced existing)
2018/12/01 16:27:28 INFO  :
Transferred:            32 / 32 Bytes, 100%, 3 Bytes/s, ETA 0s
Errors:                 0
Checks:                 0 / 0, -
Transferred:            1 / 1, 100%
Elapsed time:        8.4s

2018/12/01 16:27:28 DEBUG : 4 go routines active
2018/12/01 16:27:28 DEBUG : rclone: Version "v1.45-009-g4b15c421-beta" finishing with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone-beta/.rclone.conf" "copy" "mountcheck" "gdrive_media_vfs:" "-vv" "--no-traverse"]

 

 

 

 

-> 

 

Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone upload/log.txt

/tmp/user.scripts/tmpScripts/rclone upload/script: line 4: $'\357\273\277': command not found
/tmp/user.scripts/tmpScripts/rclone upload/script: line 26: $'\357\273\277': command not found
01.12.2018 16:27:41 INFO: rclone not installed - will try again later.
Script Finished Sat, 01 Dec 2018 16:27:41 +0100

Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone upload/log.txt

Script Starting Sat, 01 Dec 2018 16:39:30 +0100

Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone upload/log.txt

/tmp/user.scripts/tmpScripts/rclone upload/script: line 4: $'\357\273\277': command not found
/tmp/user.scripts/tmpScripts/rclone upload/script: line 26: $'\357\273\277': command not found
01.12.2018 16:39:30 INFO: rclone not installed - will try again later.
Script Finished Sat, 01 Dec 2018 16:39:30 +0100

Full logs for this script are available at /tmp/user.scripts/tmpScripts/rclone upload/log.txt

 

Mouint was working, besides the RC port (i did a restart before)

Edited December 1, 2018 by nuhll

[NewDisplayName]

 

Thats my upload script, fresh copied from github:

Quote

#!/bin/bash

 

######  Check if script already running  ##########

if [[ -f "/mnt/user/appdata/other/rclone/rclone_upload" ]]; then

echo "$(date "+%d.%m.%Y %T") INFO: Exiting as script already running."

exit

else

touch /mnt/user/appdata/other/rclone/rclone_upload

fi

######  End Check if script already running  ##########

######  check if rclone installed  ##########

if [[ -f "/mnt/user/mount_rclone/google_vfs/mountcheck" ]]; then

echo "$(date "+%d.%m.%Y %T") INFO: rclone installed successfully - proceeding with upload."

else

echo "$(date "+%d.%m.%Y %T") INFO: rclone not installed - will try again later."

rm /mnt/user/appdata/other/rclone/rclone_upload

exit

fi

######  end check if rclone installed  ##########

# move files

rclone move /mnt/user/rclone_upload/google_vfs/ gdrive_media_vfs: -vv --drive-chunk-size 512M --checkers 3 --fast-list --transfers 2 --exclude .unionfs/** --exclude *fuse_hidden* --exclude *_HIDDEN --exclude .recycle** --exclude *.backup~* --exclude *.partial~*  --delete-empty-src-dirs --fast-list --bwlimit 9500k --tpslimit 3 --min-age 1m 

# remove dummy file

rm /mnt/user/appdata/other/rclone/rclone_upload

exit
 

 

There is some encoding issue, which i dont know how to fix, as i just copied it from github.... anyway i dont think thats the problem, it cant reach that check file (and i cant in terminal also)

 

 

The Problem is now that the mountcheck file is not reachable, but i have no clue why. I can upload it perfectly.

 

Quote

 

/tmp/user.scripts/tmpScripts/rclone upload/script: line 4: $'\357\273\277': command not found
/tmp/user.scripts/tmpScripts/rclone upload/script: line 26: $'\357\273\277': command not found
 

 

Edited December 1, 2018 by nuhll

[Kaizac]

Ok lets check a few things first:

 

- You installed the plugin Rclone beta from Waseh?

- You installed the nerpack plugin?

- Within nerdpack you installed unionfs?

- You created the shares mount_rclone, mount_unionfs and rclone_upload?

- You created a rclone mount in rclone and when you browse to mount_rclone you see your Gdrive mount there and you can copy a file through it and you see it appear on your Gdrive?

- You've created these scripts under Settings > User Scripts in Unraid?

 

If so try my scripts, they are adapted versions of DZMM and maybe a bit more for general purpose.

 

Mount:
 

#!/bin/bash

#######  Check if script already running  ##########

if [[ -f "/mnt/user/mount_rclone/rclone_install_running" ]]; then

echo "$(date "+%d.%m.%Y %T") INFO: Exiting script already running."

exit

else

touch /mnt/user/mount_rclone/rclone_install_running

fi

#######  End Check if script already running  ##########

mkdir -p /mnt/user/mount_rclone/Gdrive
mkdir -p /mnt/user/mount_unionfs/Gdrive

#######  Start rclone_vfs mounted  ##########

if [[ -f "/mnt/user/mount_rclone/Gdrive/mountcheck" ]]; then

echo "$(date "+%d.%m.%Y %T") INFO: Check rclone vfs mount success."

else

echo "$(date "+%d.%m.%Y %T") INFO: installing and mounting rclone."

rclone mount --rc --rc-addr=10.0.0.60:5572 --allow-other --buffer-size 1G --dir-cache-time 72h --drive-chunk-size 32M --fast-list --log-level INFO --vfs-read-chunk-size 128M --vfs-read-chunk-size-limit off gdrive_crypt: /mnt/user/mount_rclone/Gdrive --stats 1m &

rclone rc vfs/refresh recursive=true

# pausing briefly to give mount time to initialise
sleep 10

if [[ -f "/mnt/user/mount_rclone/Gdrive/mountcheck" ]]; then

echo "$(date "+%d.%m.%Y %T") INFO: Check rclone vfs mount success."

else

echo "$(date "+%d.%m.%Y %T") CRITICAL: rclone_vfs mount failed - please check for problems."

rm /mnt/user/mount_rclone/rclone_install_running

exit

fi

fi

#######  End rclone_vfs mount  ##########

#######  Start Mount unionfs   ##########

if [[ -f "/mnt/user/mount_unionfs/Gdrive/mountcheck" ]]; then

echo "$(date "+%d.%m.%Y %T") INFO: Check successful, unionfs Movies & Series mounted."

else

# Unmount before remounting

fusermount -uz /mnt/user/mount_unionfs/Gdrive
#fusermount -uz /mnt/user/mount_unionfs/Gdrive_Backup

unionfs -o cow,allow_other,direct_io,auto_cache,sync_read /mnt/user/rclone_upload/Gdrive=RW:/mnt/user/mount_rclone/Gdrive=RO /mnt/user/mount_unionfs/Gdrive

if [[ -f "/mnt/user/mount_unionfs/Gdrive/mountcheck" ]]; then

echo "$(date "+%d.%m.%Y %T") INFO: Check successful, unionfs Movies & Series mounted."

else

echo "$(date "+%d.%m.%Y %T") CRITICAL: unionfs Movies & Series Remount failed."

rm /mnt/user/mount_rclone/rclone_install_running

exit

fi

fi

#######  End Mount unionfs   ##########

rm /mnt/user/mount_rclone/rclone_install_running

exit

Upload:
 

#!/bin/bash

#######  Check if script already running  ##########

if [[ -f "/mnt/user/mount_rclone/rclone_upload" ]]; then

echo "$(date "+%d.%m.%Y %T") INFO: Exiting as script already running."

exit

else

touch /mnt/user/mount_rclone/rclone

fi

#######  End Check if script already running  ##########

#######  check if rclone installed  ##########

if [[ -f "/mnt/user/mount_rclone/Gdrive/mountcheck" ]]; then

echo "$(date "+%d.%m.%Y %T") INFO: rclone installed successfully - proceeding with upload."

else

echo "$(date "+%d.%m.%Y %T") INFO: rclone not installed - will try again later."

rm /mnt/user/mount_rclone/rclone_upload

exit

fi

#######  end check if rclone installed  ##########
# move media

rclone move /mnt/user/rclone_upload/Gdrive/ gdrive_upload: -vv --drive-chunk-size 512M --checkers 4 --transfers 1 --exclude .unionfs/** --exclude *fuse_hidden* --exclude *_HIDDEN --exclude .recycle** --exclude *.backup~* --exclude *.partial~* --bwlimit 8500k

# remove dummy file

rm /mnt/user/mount_rclone/rclone_upload

exit

 

[NewDisplayName]

Yes, i did the setup step by step. 

 

The first time it was working fine, also the upload script.

 

But i abort it mid air and restartet it. This did not work, since then i have problems.

 

I dont really want to use new directorys now, im confused enaught.

Edited December 2, 2018 by nuhll

[NewDisplayName]

The mountcheck checks for

 

/mnt/user/mount_rclone/Gdrive/mountcheck

 

But i only have 

/mnt/user/mount_rclone/google_vfs

 

why?

[Kaizac]

DZMM has google_vfs I use Gdrive. So in your case you can change it to google_vfs if use the file structure of DZMM.

[NewDisplayName]

1 hour ago, Kaizac said:

DZMM has google_vfs I use Gdrive. So in your case you can change it to google_vfs if use the file structure of DZMM.

Ah sorry, you see, im already confused. xD.

 

So that upload command for mountcheck on the first page is working. BUT it doesnt appear in the 

/mnt/user/mount_rclone/google_vfs directory.... why?

[Kaizac]

2 hours ago, nuhll said:

Ah sorry, you see, im already confused. xD.

 

So that upload command for mountcheck on the first page is working. BUT it doesnt appear in the 

/mnt/user/mount_rclone/google_vfs directory.... why?

After the mount the mountcheck file should be created if the script is working. When mount again it would say you're already mounted. These checks are more for a server that is running well. If you are still first getting the mount/upload to work you can remove the check parts from the script and just use the rclone commands.

[markrudling]

Firstly, massive thank you for this guide. I migrated from rclone cache to this today and streaming is far better. 👏

 

Im having one issue. With sonarr/radarr i have the renamer section set to update file time to airdate of the media. This is fine for new files, but when doing a scan/library update, it seems to pull the entire file to my rclone_upload folder, then sets the time and waits for it to upload again with the mover script.

 

Has anyone else seen this behavior?

[DZMM]

21 minutes ago, markrudling said:

This is fine for new files, but when doing a scan/library update, it seems to pull the entire file to my rclone_upload folder, then sets the time and waits for it to upload again with the mover script.

 

Has anyone else seen this behavior?

If you are renaming files you've already uploaded, unfortunately this is what unionfs does.  Hopefully rclone union in a future release will allow unraid users to ditch unionfs

[markrudling]

9 minutes ago, DZMM said:

If you are renaming files you've already uploaded, unfortunately this is what unionfs does.  Hopefully rclone union in a future release will allow unraid users to ditch unionfs

Ok great - I can live with that knowing its not due to a setup issue on my side. I will create clones of my dockers and have them point at the rclone mount to preform cleanup of my now uploaded media.

 

Thank you again, looking forward to seeing how rclone will manage unions.

 

Cheers

Mark

[DZMM]

48 minutes ago, markrudling said:

Thank you again, looking forward to seeing how rclone will manage unions.

It's earmarked for the next release, so fingers crossed

 

https://github.com/ncw/rclone/milestone/33

[NewDisplayName]

On 12/3/2018 at 6:30 PM, Kaizac said:

After the mount the mountcheck file should be created if the script is working. When mount again it would say you're already mounted. These checks are more for a server that is running well. If you are still first getting the mount/upload to work you can remove the check parts from the script and just use the rclone commands.

If im correct the moutncheck file doesnt get created via the mount script, it gets created the first time u install this manually....?

 

I can manually use the command to upload it, but it doesnt appear in that directory where the scripts are looking for it. so something on this part must be the problem.

 

How can i check the unionfs log or something?

 

Legendary would be if someone could put this together in a plugin or docker....

 

 

2.       Create Mountcheck files

 

This blank file is used in the following scripts to verify if the mounts have been created properly.  Run these commands:

 

touch mountcheck

 

rclone copy mountcheck gdrive_media_vfs: -vv --no-traverse

Edited December 10, 2018 by nuhll

[DZMM]

9 hours ago, nuhll said:

If im correct the moutncheck file doesnt get created via the mount script, it gets created the first time u install this manually....?

 

I can manually use the command to upload it, but it doesnt appear in that directory where the scripts are looking for it. so something on this part must be the problem.

 

How can i check the unionfs log or something?

 

Legendary would be if someone could put this together in a plugin or docker....

 

 

 

 

yes, the mountcheck file is created outside of rclone by running these two commands which creates the file and then moves it to the root of the rclone mount:

 

touch mountcheck

rclone copy mountcheck gdrive_media_vfs: -vv --no-traverse

once done you should have something looking like this when looking locally at mount_rclone and mount_unionfs:

 

 

and on drive the mountcheck file is encrypted so it will have a funny name.  I went back through this thread and your rclone remote names are the same as mine, so running the two commands above in terminal will definitely do the job.

 

 

 

Edited December 10, 2018 by DZMM

[NewDisplayName]

So guys, i didnt got it working till now, but i worked very hard (and with help of a friend) to get the "only upload when no one is using his pc part working".

 

 

So you see im not just taking, ill also give back! For slower internets this is pretty good i guess!

 

Quote

 

#!/bin/sh
#rm /mnt/user/downloads/pingtest
#touch /mnt/user/downloads/pingtest

host=192.168.86.42
host2=192.168.86.48
host3=192.168.86.154

current_date_time="`date "+%Y-%m-%d %H:%M:%S"`";
echo $current_date_time >> /mnt/user/downloads/pingtest;

echo "host 1" >> /mnt/user/downloads/pingtest;
ping -c 1 -W1 -q $host > /dev/null
echo "$?" >> /mnt/user/downloads/pingtest;

echo "host 2" >> /mnt/user/downloads/pingtest;
ping -c 1 -W1 -q $host2 > /dev/null
echo "$?" >> /mnt/user/downloads/pingtest;

echo "host 3" >> /mnt/user/downloads/pingtest;
ping -c 1 -W1 -q $host3 > /dev/null
echo "$?" >> /mnt/user/downloads/pingtest;

ping -c 1 -W1 -q $host || ping -c 1 -W1 -q $host2 || ping -c 1 -W1 -q $host3 > /dev/null
      if [ $? == 0 ]; then
      logger ping erfolgreich, upload stoppen
      else
          logger ping fehlgeschlagen, upload starten
      exit;

      fi
exit;
 

 

 

 

everything betweenn current date and the last echo can be removed, its just to see if its working.

 

You just need to put the upload script at where "logger ping fehlgeschlagen, upload starten" is, and you need to put a stop command at "      logger ping erfolgreich, upload stoppen".

 

Im not quite sure how to stop effectly maybe killall -9 rclone?

 

Then its as simply as put this in a cronjob run every 5 minutes.

 

 

 

Edited December 11, 2018 by nuhll

[DZMM]

@nuhll is everything else working ok now?

[NewDisplayName]

45 minutes ago, DZMM said:

@nuhll is everything else working ok now?

Hi, like said, till now not.

 

BUT i have good news.

 

I got a succesfull mount after a long time now..

 

Quote

 

cript location: /tmp/user.scripts/tmpScripts/rclone mount/script
Note that closing this window will abort the execution of this script
11.12.2018 19:04:51 INFO: mounting rclone vfs.
2018/12/11 19:04:51 NOTICE: Serving remote control on http://192.168.86.2:5572/
11.12.2018 19:05:10 INFO: Check rclone vfs mount success.
fusermount: failed to unmount /mnt/user/mount_unionfs/google_vfs: Invalid argument
11.12.2018 19:05:10 INFO: Check successful, unionfs mounted.
11.12.2018 19:05:10 INFO: dockers already started
11.12.2018 19:05:10 Info: populating dir cache - this could take a while.
2018/12/11 19:05:10 Failed to rc: connection failed: Post http://localhost:5572/vfs/refresh: dial tcp 127.0.0.1:5572: connect: connection refused
11.12.2018 19:05:10 Info: populating dir cache complete.
 

 

 

AND (!!!!!) the mountcheck file is at /mnt/user/mount_rclone/google_vfs/mountcheck

 

Also, i found out what was the problem... im complete retarded acutally.

 

Kaizac was actually correct. The problem was (is) that i use run script, as long as i have that window open, the file is there and its working, but when i close it, its over...^^ (thanks for that hint!)

Edited December 11, 2018 by nuhll

[NewDisplayName]

im currently testing.

 

 

killall -9 rclone seems to work fine, just kills rclone transfer, script continue and exit, if u restart it it resumes..

Edited December 11, 2018 by nuhll

[NewDisplayName]

On 11/29/2018 at 3:58 PM, DZMM said:

it doesn't matter as no real files exist on your array for those folders - just virtual files.

 

Just add the current folders to your unionfs mount and then move them manually behind the scenes to your rclone_upload folder where they'll get uploaded over time e.g.

 

unionfs -o cow,allow_other,direct_io,auto_cache,sync_read /mnt/user/rclone_upload/google_vfs=RW:/mnt/user/mount_rclone/google_vfs=RO:/mnt/user/OLD_MEDIA_LOCATION=RO /mnt/user/mount_unionfs/google_vfs

Then add the unionfs folders to sonarr etc.  You just have to make sure that the filepaths matchup e.g.

 

- /mnt/user/mount_rclone/google_vfs/TV Show 1

- /mnt/user/Old/Media_location/TV Show 1

 

 

Thats the last thing i need to do. And i dont want to make a mistake and ruin my library, so please help me...

 

 

 

Currently i have the following setup:

 

1 Share "Archiv"

/mnt/user/Archiv

 

there are:

/mnt/user/Archiv/Filme

/mnt/user/Archiv/Serien

(there are other directorys, but it would be ok if only these 2 folders work) (or whole share Archiv, idc)

 

All my applications are setup to work with this directorys.

 

Whats the easiest way to get it working? I would really like to not change anything on the structure. Also i still want to be able to brose thru it via SMB and so on....

 

I can move files to the upload directory thats ok.

 

 

Edited December 11, 2018 by nuhll

[Kaizac]

What is your desired file structure/topology? Do you want to upload everything to the cloud eventually or do you have part of your library you want to keep local>