Guide: How To Use Rclone To Mount Cloud Drives And Play Files


DZMM

2468 posts in this topic Last Reply

Recommended Posts

Sometimes the connection fails and I can see unionfs is left running. I think this is preventing the mount script from successfully running whenever it retries. Even running the cleanup or shutdown scripts don't fix it. I have to killall unionfs, then re-run the mount script... I'm using your scripts from github.

Edited by GorOps
Link to post
  • Replies 2.5k
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

3/1/20 UPDATE: TO MIGRATE FROM UNIONFS TO MERGERFS READ THIS POST.  New users continue reading 13/3/20 Update: For a clean version of the 'How To' please use the github site https://github.com/B

Multiple mounts, one upload and one tidy-up script.   @watchmeexplode5 did some testing and performance gets worse as you get closer to the 400k mark, so you'll need to do something like bel

so ive been reading for ages but theres so many pages here, my issue is that when the array is stopped the mergerfs mount is not unmounted casuing unraid to keep retrying indefinately to unmount share

Posted Images

7 hours ago, Kaizac said:

Thanks you're right. My backup Share is just using a lot of files. How do you do this yourself? All the pictures and small files you want to keep safe, don't amount to a lot of size, but do to number of files

 

Some way to autozip them would be best I think. Just like the CA backup does but then for your own shares.

I'm ok I'm way under the 400k limit - maybe use a backup utility to create a zip then upload?

Link to post
2 hours ago, GorOps said:

Sometimes the connection fails and I can see unionfs is left running. I think this is preventing the mount script from successfully running whenever it retries. Even running the cleanup or shutdown scripts don't fix it. I have to killall unionfs, then re-run the mount script... I'm using your scripts from github.

must be something wrong with your internet connection as my gdrive connection never drops.

 

the mount script if run on a cron job auto-remounts the rclone and unionfs mounts, so maybe increase the frequency

 

 

Link to post
3 hours ago, yendi said:

With this rclone setup and no cache, everything is completely streamed as soon as you access the media? How long does it conserve the downloaded data on the server? 

Thanks (uploading on-going)

Yes, everything is streamed - nothing is preserved/cached

Link to post

Thanks !

Still have some hicups with like the mountcheck file being duplicated so I have to finish a full upload to see why I have it on both on the upload and google folder (two identical files in Union folder with 2 different creation date), but except that it seems pretty solid. 

Quick question that you probably missed, do you have a backup strategy with this or you just have 1 Gsuite account and if he gets closed you will just re-dl all medias?

Anyway, you really have provided great scripts !

Link to post
On 7/27/2019 at 8:04 PM, GorOps said:

Sometimes the connection fails and I can see unionfs is left running. I think this is preventing the mount script from successfully running whenever it retries. Even running the cleanup or shutdown scripts don't fix it. I have to killall unionfs, then re-run the mount script... I'm using your scripts from github.

If you need to do a "reset" without having to restart, you can see the PID in Tools -> Processes and then in SSH just kill all the rclone and unionfs processes using the kill command (PID separated by space) e.g.

kill 12345 23456 34567 

I actually have never had the mount drop on me. Even when I unplug the router, the mount just reconnects itself automatically once replug (without the need to run the mount script). So your issue sounds more like rclone mount crashing / got killed / kicked. That is usually due to insufficient memory.

 

30 minutes ago, yendi said:

Thanks !

Still have some hicups with like the mountcheck file being duplicated so I have to finish a full upload to see why I have it on both on the upload and google folder (two identical files in Union folder with 2 different creation date), but except that it seems pretty solid. 

Quick question that you probably missed, do you have a backup strategy with this or you just have 1 Gsuite account and if he gets closed you will just re-dl all medias?

Anyway, you really have provided great scripts !

That sounds like you "touch mountcheck" the unionfs mount + the rclone mount.

Link to post
58 minutes ago, yendi said:

Quick question that you probably missed, do you have a backup strategy with this or you just have 1 Gsuite account

No backup strategy - it's been working so well for me for over a year now, that if I wasn't posting in this thread, I would honestly have forgotten all about it and wouldn't even remember how I set it up! 

 

One thing that I considered with one person was to backup our tdrives with each other, so as long as we both didn't get booted at the same time, we'd have access to our content - we might get around to sorting this one day.

 

I guess I'm assuming that if Google ever pulled the plug I'd have a few days to download what I want and to decide if I want to build a new server with 350TB+ of storage to hold everything....

Edited by DZMM
Link to post

Were doing nothing wrong, its not against their EULA/TOS.

 

They cant just close it from day to day, if they cancel our subscription they have to give us a good amount of time to save our files (new account.. ) :D

 

just btw, ive got an email today abotut ehy close "access for less secure apps" does that mean anything to us?

Edited by nuhll
Link to post

I found a big bug ! 

If I go to Sonarr to change the path of a tv-show (did not try a movie yet) my mount is gone as soon as I click on the path that I configured in Sonarr:

My previous show was local, I copied everything in the rclone union folder, it uploaded successfully. I went to change the path in Sonarr to point it to the union folder and as soon as I click on it, the mount is gone and the only files I see in the Union folder are the one in the Upload folder.1870682349_2019-07-2921_28_58-VeronicaMars-Sonarr.thumb.jpg.7e48f5a08e9bc7ef9f9086aeb10e0df7.jpg

Edited by yendi
Link to post
35 minutes ago, yendi said:

I found a big bug ! 

If I go to Sonarr to change the path of a tv-show (did not try a movie yet) my mount is gone as soon as I click on the path that I configured in Sonarr:

My previous show was local, I copied everything in the rclone union folder, it uploaded successfully. I went to change the path in Sonarr to point it to the union folder and as soon as I click on it, the mount is gone and the only files I see in the Union folder are the one in the Upload folder.1870682349_2019-07-2921_28_58-VeronicaMars-Sonarr.thumb.jpg.7e48f5a08e9bc7ef9f9086aeb10e0df7.jpg

Have you looked in the rclone_moubt folder?  There's a small delay between files being uploaded and appearing there.

 

To be honest though it sounds like you've set up your unionfs folder wrong somewhere if you're not seeing any cloud files

Link to post
3 minutes ago, DZMM said:

Have you looked in the rclone_moubt folder?  There's a small delay between files being uploaded and appearing there.

 

To be honest though it sounds like you've set up your unionfs folder wrong somewhere if you're not seeing any cloud files

I can assure you its a bug; everything is working fine when I reboot: all script launch perfectly and upload is working. Bug when I go to sonarr to change the path of ANY tv show by using the path browser, the "mount_rclone" folder is suddenly empty thus, the Union folder aswell. 

If I change path using the bulk editor, the mount is preserved.

When the mount disapears, I tried to unmount + remount and it fails (can not connect to end point) and when I reboot, everything is fine. Can someone try ?

Edited by yendi
Link to post

Have you tried restarting sonarr?  I change the paths of shows/movies frequently without any problems, both from mount folders to another mount folder and in the past from local-2-mount and vice versa

Link to post
24 minutes ago, DZMM said:

Have you tried restarting sonarr?  I change the paths of shows/movies frequently without any problems, both from mount folders to another mount folder and in the past from local-2-mount and vice versa

Well, not it seems to be working correctly. The only thing I changed is disabling a plugin (Unbalance) and restarting the server.

I try to browse an other show and it seems to work perfectly 🤷‍♂️

Link to post
4 hours ago, yendi said:

Do you config that I can use the cache disk without issue with the 3 mounts? 
 

As they are created at boot with a script, I prefer to ask.

Sorry but your question is very ambiguous. What kind of issue? What kind of "use"?

Link to post
2 minutes ago, testdasi said:

Sorry but your question is very ambiguous. What kind of issue? What kind of "use"?

Sorry, I typed on my cellphone so it made a typo:

My question was: Is it ok to add a cache to the 3 shares ? As they are unmounted and recreated each boot via the scripts I wanted to know if it would work/be safe?

 

Link to post
10 minutes ago, yendi said:

Sorry, I typed on my cellphone so it made a typo:

My question was: Is it ok to add a cache to the 3 shares ? As they are unmounted and recreated each boot via the scripts I wanted to know if it would work/be safe?

You mean changing the "Cache" box to Yes in the share settings in the Unraid GUI? That should be safe as long as your mount scripts are all /mnt/user.

Link to post
18 minutes ago, testdasi said:

You mean changing the "Cache" box to Yes in the share settings in the Unraid GUI? That should be safe as long as your mount scripts are all /mnt/user.

Yes, that's it ! Thanks

Edited by yendi
Link to post
On 7/29/2019 at 9:59 PM, DZMM said:

Have you tried restarting sonarr?  I change the paths of shows/movies frequently without any problems, both from mount folders to another mount folder and in the past from local-2-mount and vice versa

I had multiple rclone crash since yesterday (mount is suddenly empty, not only when using Sonarr, also with Plex Scans) but I think I found the issue: I "only" have 12gb of RAM, and the upload + services running on unRAID are eating up 70% of the memory. I read that rclone tend to crash when no RAM is available and I saw that when I do a library scan, memory tend to go to the 90%ish so I ordered 8 gb of more ram... I will give a feedback when my RAM arrive to tell you if its the shortage that make rclone crash.

Edited by yendi
Link to post
2 hours ago, yendi said:

I had multiple rclone crash since yesterday (mount is suddenly empty, not only when using Sonarr, also with Plex Scans) but I think I found the issue: I "only" have 12gb of RAM, and the upload + services running on unRAID are eating up 70% of the memory. I read that rclone tend to crash when no RAM is available and I saw that when I do a library scan, memory tend to go to the 90%ish so I ordered 8 gb of more ram... I will give a feedback when my RAM arrive to tell you if its the shortage that make rclone crash.

you could try reducing the number of uploads and checkers in the upload script to reduce ram usage.  An extra 8GB might do the trick, but you might need to change some of the buffer settings in the mount command if you anticipate having a lot of concurrent streams

Link to post

Sorry if this has been answered already. But I only want to mount a Google Drive folder to be able to play media using plex. I don't need any other of the fancy stuff the script does. I tried following the directions and have created the gdrive and vfs drive. But when I try to run the script I get errors about tdrive and unionfs cant be found. Is there some instructions just for the basic mount folder in unraid? 

Link to post

The union fs part allow you to use the upload system and to have both cloud and upload folder merged. So no you dont need that if yu just want to mount a gdrive: You just need "rcloune mount" command. You should check rclone help to chekc what argument to add to your command so it suits your need or just try with one like this:

rclone mount  --buffer-size 256M --dir-cache-time 72h --drive-chunk-size 32M --fast-list --log-level INFO --vfs-read-chunk-size 128M --vfs-read-chunk-size-limit off gdrive_media_vfs: /mnt/user/mount_rclone/google_vfs

 

Link to post
4 hours ago, yendi said:

The union fs part allow you to use the upload system and to have both cloud and upload folder merged. So no you dont need that if yu just want to mount a gdrive: You just need "rcloune mount" command. You should check rclone help to chekc what argument to add to your command so it suits your need or just try with one like this:


rclone mount  --buffer-size 256M --dir-cache-time 72h --drive-chunk-size 32M --fast-list --log-level INFO --vfs-read-chunk-size 128M --vfs-read-chunk-size-limit off gdrive_media_vfs: /mnt/user/mount_rclone/google_vfs

 

Thanks, I guess the command should be: 

rclone mount gdrive_media_vfs: /mnt/user/mount_rclone/google_vfs --buffer-size 256M --dir-cache-time 72h --drive-chunk-size 32M --fast-list --log-level INFO --vfs-read-chunk-size 128M --vfs-read-chunk-size-limit off

 as that was the only way I could get rclone to accept the command.

 

But when I run the command nothing happens.

Edited by Cliff
Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.