Guide: How To Use Rclone To Mount Cloud Drives And Play Files


DZMM

2383 posts in this topic Last Reply

Recommended Posts

Is anyone else having problems where hardlinked torrents are being uploaded and deleting the torrents, rather than just a copy being uploaded?  Over the last week my torrents keep disappearing and I think it's rclone/mergerfs uploading the real file.

Link to post
  • Replies 2.4k
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

3/1/20 UPDATE: TO MIGRATE FROM UNIONFS TO MERGERFS READ THIS POST.  New users continue reading 13/3/20 Update: For a clean version of the 'How To' please use the github site https://github.com/B

Multiple mounts, one upload and one tidy-up script.   @watchmeexplode5 did some testing and performance gets worse as you get closer to the 400k mark, so you'll need to do something like bel

I just made a very useful change to my scripts that has solved my problem with the limit of only being able to upload 750GB/day, which was creating bottlenecks on my local server as I couldn't upload

Posted Images

Is anyone else having problems where hardlinked torrents are being uploaded and deleting the torrents, rather than just a copy being uploaded?  Over the last week my torrents keep disappearing and I think it's rclone/mergerfs uploading the real file.
Me i got the problem

Envoyé de mon Pixel 2 XL en utilisant Tapatalk

Link to post
25 minutes ago, francrouge said:

Me i got the problem

Envoyé de mon Pixel 2 XL en utilisant Tapatalk
 

 

Just in 2021?  It's been driving me mad this year.  I've just turned off "use hardlinks instead of copy" which should confirm my suspicion 

Link to post
Just in 2021?  It's been driving me mad this year.  I've just turned off "use hardlinks instead of copy" which should confirm my suspicion 
No i noticed it like 2month ago

Envoyé de mon Pixel 2 XL en utilisant Tapatalk

Link to post

With my current setup I can't get hardlinks or atomic-moves to work. Currently Radarr/Sonarr/Lidarr do a copy instead of an instant move. So I would like to setup a new folder structure in order to make it work.

The below structure is mounted as folder /data under /mnt/user

 

Deluge will be linked to /mnt/user/data/torrents

Sabnzbd will be linked to /mnt/user/data/usenet

Completed files will be moved to /mnt/user/data/media

Radarr/Sonarr/Lidarr will take the files and move them to /mnt/user/data/gdrive/mount_mergerfs

In mount_mergerfs gdrive_vfs1 (movies) gdrive_vfs2 (tv) and gdrive_vfs3 (music).

 

Would the below work? If not any advise on how I should adjust it?

 

data
├── torrents
    ├── movies
    ├── music
     └── tv
├── usenet
    ├── movies
    ├── music
    └── tv
└── media
    ├── movies
    ├── music
    └── tv
├── gdrive
    ├── mount_mergerfs
        ├── gdrive_vfs1
        ├── gdrive_vfs2
        ├── gdrive_vfs3
    ├── mount_rclone
        ├── gdrive_vfs1
        ├── gdrive_vfs2
        ├── gdrive_vfs3

Link to post
  • 2 weeks later...

@DZMM
 Any idea? Can you share your folder structure?

On 1/10/2021 at 1:16 PM, Ericsson said:

With my current setup I can't get hardlinks or atomic-moves to work. Currently Radarr/Sonarr/Lidarr do a copy instead of an instant move. So I would like to setup a new folder structure in order to make it work.

The below structure is mounted as folder /data under /mnt/user

 

Deluge will be linked to /mnt/user/data/torrents

Sabnzbd will be linked to /mnt/user/data/usenet

Completed files will be moved to /mnt/user/data/media

Radarr/Sonarr/Lidarr will take the files and move them to /mnt/user/data/gdrive/mount_mergerfs

In mount_mergerfs gdrive_vfs1 (movies) gdrive_vfs2 (tv) and gdrive_vfs3 (music).

 

Would the below work? If not any advise on how I should adjust it?

 

data
├── torrents
    ├── movies
    ├── music
     └── tv
├── usenet
    ├── movies
    ├── music
    └── tv
└── media
    ├── movies
    ├── music
    └── tv
├── gdrive
    ├── mount_mergerfs
        ├── gdrive_vfs1
        ├── gdrive_vfs2
        ├── gdrive_vfs3
    ├── mount_rclone
        ├── gdrive_vfs1
        ├── gdrive_vfs2
   

 

 

Link to post
1 hour ago, Ericsson said:

Deluge will be linked to /mnt/user/data/torrents

Sabnzbd will be linked to /mnt/user/data/usenet

Completed files will be moved to /mnt/user/data/media

Radarr/Sonarr/Lidarr will take the files and move them to /mnt/user/data/gdrive/mount_mergerfs

This is the problem - you need all your dockers using the same mappings, otherwise hardlinks won't work as they will be seen as different volumes.

 

I.e in your docker settings in unraid just create /user -->/mnt/user , then in the docker applications link to where on unraid you want to use e.g. /user/data/torrents for files you want storing in /mnt/user/data/torrents on unraid.

 

Read the thread (search for hardlinks) if it's not clear as I must have gone over this at least a dozen times.

Link to post
On 1/18/2021 at 8:38 PM, DZMM said:

This is the problem - you need all your dockers using the same mappings, otherwise hardlinks won't work as they will be seen as different volumes.

 

I.e in your docker settings in unraid just create /user -->/mnt/user , then in the docker applications link to where on unraid you want to use e.g. /user/data/torrents for files you want storing in /mnt/user/data/torrents on unraid.

 

Read the thread (search for hardlinks) if it's not clear as I must have gone over this at least a dozen times.

@DZMM Thanks for the pointer. I have it working now :)

Link to post

I have an issue when running rclone_mount script. It can't create gdrive_media_vfs inside mount_rclone folder. I check mount_rclone from krusade. It has ghost gdrive_media_vfs existed. What should I do?

 

Script location: /tmp/user.scripts/tmpScripts/rclone_mount/script
Note that closing this window will abort the execution of this script
21.01.2021 10:12:03 INFO: Creating local folders.
mkdir: cannot stat '/mnt/user/mount_rclone/gdrive_media_vfs': Transport endpoint is not connected
21.01.2021 10:12:03 INFO: Creating MergerFS folders.
21.01.2021 10:12:03 INFO: *** Starting mount of remote gdrive_media_vfs
21.01.2021 10:12:03 INFO: Checking if this script is already running.
21.01.2021 10:12:03 INFO: Script not running - proceeding.
21.01.2021 10:12:03 INFO: *** Checking if online
21.01.2021 10:12:04 PASSED: *** Internet online
21.01.2021 10:12:04 INFO: Mount not running. Will now mount gdrive_media_vfs remote.
21.01.2021 10:12:04 INFO: Recreating mountcheck file for gdrive_media_vfs remote.
2021/01/21 10:12:04 DEBUG : rclone: Version "v1.53.4" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "mountcheck" "gdrive_media_vfs:" "-vv" "--no-traverse"]
2021/01/21 10:12:04 DEBUG : Creating backend with remote "mountcheck"
2021/01/21 10:12:04 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf"
2021/01/21 10:12:04 DEBUG : fs cache: adding new entry for parent of "mountcheck", "/"
2021/01/21 10:12:04 DEBUG : Creating backend with remote "gdrive_media_vfs:"
2021/01/21 10:12:04 DEBUG : Creating backend with remote "gdrive:crypt"
2021/01/21 10:12:05 DEBUG : mountcheck: Modification times differ by -5m13.245271688s: 2021-01-21 10:12:04.210271688 -0800 PST, 2021-01-21 18:06:50.965 +0000 UTC
2021/01/21 10:12:06 INFO : mountcheck: Copied (replaced existing)
2021/01/21 10:12:06 INFO :
Transferred: 32 / 32 Bytes, 100%, 23 Bytes/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 2.4s

2021/01/21 10:12:06 DEBUG : 4 go routines active
21.01.2021 10:12:06 INFO: *** Creating mount for remote gdrive_media_vfs
21.01.2021 10:12:06 INFO: sleeping for 5 seconds
2021/01/21 10:12:06 NOTICE: Serving remote control on http://127.0.0.1:5572/
2021/01/21 10:12:07 Fatal error: Can not open: /mnt/user/mount_rclone/gdrive_media_vfs: open /mnt/user/mount_rclone/gdrive_media_vfs: transport endpoint is not connected
21.01.2021 10:12:11 INFO: continuing...
21.01.2021 10:12:11 CRITICAL: gdrive_media_vfs mount failed - please check for problems. Stopping dockers
plex

 

Link to post

I think the problem is rclone_unmount script not unmount gdrive_media_vfs folder. If I ran,

fusermount -u /mnt/user/mount_rclone/gdrive_media_vfs 

 

I can rerun the rclone_mount after rclound_unmount again. Anybody see this issue.

 

unRaid version: 6.9.0-rc2

Link to post

I'm trying to use this script to mount multiple shared drives. From everything I've read here I should just be able to run multiple version of this script. But when I do I get the following error:

Failed to start remote control: start server failed: listen tcp 127.0.0.1:5572: bind: address already in use

 

I have rebooted. I'm only mounting the mergerfs in one of the versions of the script. Whichever script I run first will work but I can't get a second to work due to the above error. 

Any clues?

Link to post
3 hours ago, bnevets27 said:

I'm trying to use this script to mount multiple shared drives. From everything I've read here I should just be able to run multiple version of this script. But when I do I get the following error:


Failed to start remote control: start server failed: listen tcp 127.0.0.1:5572: bind: address already in use

 

I have rebooted. I'm only mounting the mergerfs in one of the versions of the script. Whichever script I run first will work but I can't get a second to work due to the above error. 

Any clues?

You can only use the --rc command once.  I think you've got it in each mount instance

Link to post
On 1/21/2021 at 6:16 PM, tamito said:

I have an issue when running rclone_mount script. It can't create gdrive_media_vfs inside mount_rclone folder. I check mount_rclone from krusade. It has ghost gdrive_media_vfs existed. What should I do?

 


Script location: /tmp/user.scripts/tmpScripts/rclone_mount/script
Note that closing this window will abort the execution of this script
21.01.2021 10:12:03 INFO: Creating local folders.
mkdir: cannot stat '/mnt/user/mount_rclone/gdrive_media_vfs': Transport endpoint is not connected
21.01.2021 10:12:03 INFO: Creating MergerFS folders.
21.01.2021 10:12:03 INFO: *** Starting mount of remote gdrive_media_vfs
21.01.2021 10:12:03 INFO: Checking if this script is already running.
21.01.2021 10:12:03 INFO: Script not running - proceeding.
21.01.2021 10:12:03 INFO: *** Checking if online
21.01.2021 10:12:04 PASSED: *** Internet online
21.01.2021 10:12:04 INFO: Mount not running. Will now mount gdrive_media_vfs remote.
21.01.2021 10:12:04 INFO: Recreating mountcheck file for gdrive_media_vfs remote.
2021/01/21 10:12:04 DEBUG : rclone: Version "v1.53.4" starting with parameters ["rcloneorig" "--config" "/boot/config/plugins/rclone/.rclone.conf" "copy" "mountcheck" "gdrive_media_vfs:" "-vv" "--no-traverse"]
2021/01/21 10:12:04 DEBUG : Creating backend with remote "mountcheck"
2021/01/21 10:12:04 DEBUG : Using config file from "/boot/config/plugins/rclone/.rclone.conf"
2021/01/21 10:12:04 DEBUG : fs cache: adding new entry for parent of "mountcheck", "/"
2021/01/21 10:12:04 DEBUG : Creating backend with remote "gdrive_media_vfs:"
2021/01/21 10:12:04 DEBUG : Creating backend with remote "gdrive:crypt"
2021/01/21 10:12:05 DEBUG : mountcheck: Modification times differ by -5m13.245271688s: 2021-01-21 10:12:04.210271688 -0800 PST, 2021-01-21 18:06:50.965 +0000 UTC
2021/01/21 10:12:06 INFO : mountcheck: Copied (replaced existing)
2021/01/21 10:12:06 INFO :
Transferred: 32 / 32 Bytes, 100%, 23 Bytes/s, ETA 0s
Transferred: 1 / 1, 100%
Elapsed time: 2.4s

2021/01/21 10:12:06 DEBUG : 4 go routines active
21.01.2021 10:12:06 INFO: *** Creating mount for remote gdrive_media_vfs
21.01.2021 10:12:06 INFO: sleeping for 5 seconds
2021/01/21 10:12:06 NOTICE: Serving remote control on http://127.0.0.1:5572/
2021/01/21 10:12:07 Fatal error: Can not open: /mnt/user/mount_rclone/gdrive_media_vfs: open /mnt/user/mount_rclone/gdrive_media_vfs: transport endpoint is not connected
21.01.2021 10:12:11 INFO: continuing...
21.01.2021 10:12:11 CRITICAL: gdrive_media_vfs mount failed - please check for problems. Stopping dockers
plex

 

Post your script and rclone config without passwords

Link to post
26 minutes ago, privateer said:

I've done a few tests and noticed that on reboot the mount script fails the first time, and works the second time. Anyone else encountered this?

Same here. I even tried adjusting the sleep time to 120 seconds. Didn't seem to help. 

Link to post
3 hours ago, axeman said:

Same here. I even tried adjusting the sleep time to 120 seconds. Didn't seem to help. 

With the introduction of the cache, rclone does a bit of extra checking before the mount is fully live and accessible.  Just keep repeating the script and it gets there.  It takes me a while to complete my mount (about 5-10 mins) with 7 tdrvies each with a 300GB cache.  I've found that bigger caches take even longer.

 

One option I'm considering adding is to choose whether or not to flush the cache on unmount.

Link to post

 

On 1/26/2021 at 1:20 AM, DZMM said:

With the introduction of the cache, rclone does a bit of extra checking before the mount is fully live and accessible.  Just keep repeating the script and it gets there.  It takes me a while to complete my mount (about 5-10 mins) with 7 tdrvies each with a 300GB cache.  I've found that bigger caches take even longer.

 

One option I'm considering adding is to choose whether or not to flush the cache on unmount.

 

Yeah it works on the second one. I adjusted the cache size as well. Not a problem for me, but when I see something like that I want to make sure I know why it's happening or that I'm not the only one!

 

EDIT: Deleted question about API calls - I set up something wrong a long time ago

Edited by privateer
Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.