Jump to content
Waseh

[Plugin] rclone

449 posts in this topic Last Reply

Recommended Posts

rclone Plugin

hHXQqTb.png

 

Hey everyone

 

This is a simple plugin which installs rclone on your unraid system.

I've released a version for the stable branch as well as the beta branch of rclone

You can install them from CA or from the plugin menu using these links: Stable and Beta

 

rclone is a command line tool which enables you to mount a cloud share as well as to sync files and directories to and from:

* Google Drive

* Amazon S3

* Openstack Swift / Rackspace cloud files / Memset Memstore

* Dropbox

* Google Cloud Storage

* Amazon Drive

* Microsoft One Drive

* Hubic

* Backblaze B2

* Yandex Disk

* The local filesystem

 

rclone also supports encryption and decryption of files synced to these services.

More information can be found at http://rclone.org/

 

 

After installation, setup rclone by using the command:

rclone config
 

 

The config file can be found on your flash drive in the rclone plugin folder

Further documentation can be found at: http://rclone.org/docs/

 

 

To mount a remote share with rclone

It's possible to mount a remote share with rclone as a filesystem so you can access your files directly from your unraid machine.

The rclone beta versions even allow seeking which allows you to playback media as if it was stored directly on your machine.

Use the following command:

rclone mount --allow-other remote:path/to/files /path/to/local/mount &
 

 

The local mount point should be inside /mnt/disks/ if you want to share the files with your docker containers.

The --allow-other mount option is also important if you want to share the remote share with your docker containers or for example through samba.

 

 

Making your rclone mount accessible to your docker containers

For your docker containers to be able to access your remote share mount you have to specify the path to the mount point in the docker container interface.

Set the container/host volume with a mode of Read Write,Slave, else the files will not show up inside the container.

Also remember to use the --allow-other mount option, as well as mounting the remote share inside /mnt/disks/. This is due to the fact that only mount points within /mnt/disks/ support Slave modes.

 

 

 

For automation i recommend using the excellent user scripts plugin by Squid.

 

Any feedback and suggestions are more than welcome!

I want to thank aschamberger for the original plugin as well as the people who helped me test the initial versions!

Edited by Waseh
  • Upvote 1

Share this post


Link to post

Video guide by gridrunner:

Sync daemon by bobbintb:

 

Edited by Waseh

Share this post


Link to post

Thanks for this. I'm interested but cautious. I need to do some reading before I install it.

 

  • Upvote 1

Share this post


Link to post

Thanks for this. I'm interested but cautious. I need to do some reading before I install it.

 

There has been a lot of discussion and experimentation on it. Check here for starters:

 

http://lime-technology.com/forum/index.php?topic=52033.msg512568

 

If you have questions or concerns, feel free to ask as there are plenty of people that have been tinkering that can answer you.

 

I'm excited to have this "official" now. Thanks to everyone involved.

Share this post


Link to post

Really awesome thanks for making this.

 

 

Could someone explain what I should be using for my rclone copy command?

 

 

The rclone docs say:

 

 

rclone copy remote:test.jpg /tmp/download

But I don't know what to put for my paths.

Share this post


Link to post

Great work putting this together Waseh!

 

Can you show a link to the source code? I will eventually make a simple GUI for this to make it even more user friendly.

Share this post


Link to post

Great work putting this together Waseh!

 

Can you show a link to the source code? I will eventually make a simple GUI for this to make it even more user friendly.

 

A GUI would be great. I thought about doing it but I don't have the time and it would take me even more time learning it than someone that is already familiar.

 

I think rclone will eventually have a daemon but until then this might be usefulle to add to the plugin:

https://techarena51.com/index.php/inotify-tools-example/

Share this post


Link to post

Really awesome thanks for making this.

 

 

Could someone explain what I should be using for my rclone copy command?

 

 

The rclone docs say:

 

 

rclone copy remote:test.jpg /tmp/download

But I don't know what to put for my paths.

 

the syntax is:

rclone copy source:sourcepath dest:destpath

 

The syntax for the remote can be confusing though. Here is my example:

 

rclone copy /mnt/user/server/backup/myfile.bin amazon:

 

The root of a remote share is the name of the share followed by a colon. So if I wanted to backup to a folder named "documents" on my Amazon Cloud Drive (named "amazon" in my rclone config), that would be as follows:

 

amazon:documents/

Share this post


Link to post

Great work putting this together Waseh!

 

Can you show a link to the source code? I will eventually make a simple GUI for this to make it even more user friendly.

 

A GUI would be great. I thought about doing it but I don't have the time and it would take me even more time learning it than someone that is already familiar.

 

I think rclone will eventually have a daemon but until then this might be usefulle to add to the plugin:

https://techarena51.com/index.php/inotify-tools-example/

 

You can add inotify from the nerd tools plugin - I think thats a better solution for the people who need it :)

Share this post


Link to post

Great work putting this together Waseh!

 

Can you show a link to the source code? I will eventually make a simple GUI for this to make it even more user friendly.

 

A GUI would be great. I thought about doing it but I don't have the time and it would take me even more time learning it than someone that is already familiar.

 

I think rclone will eventually have a daemon but until then this might be usefulle to add to the plugin:

https://techarena51.com/index.php/inotify-tools-example/

 

You can add inotify from the nerd tools plugin - I think thats a better solution for the people who need it :)

 

I could have sworn I already looked there...

 

Anyway, I'll probably give that a go and give a writeup on it for those that want use that instead of a cronjob.

Share this post


Link to post

If anyone is interested in testing the daemon:

 

https://github.com/bobbintb/backup-bash

 

I haven't had a chance to test it yet myself but really it was just a swap from the original rsync command to rclone. You need to install inotify-tools from NerdTools. Some other plugins also have it installed. Let me know how it works.

Share this post


Link to post

I've submitted a PR to upgrade to the latest rclone version. Apperantly it fixes some issues with corrupt files when seeking and some ACD file limit warnings.

Share this post


Link to post

Yea I would have updated it yesterday when I saw the new beta but I didn't want multiple releases on the same day  ;D

I'll accept the PR when I get to a computer

Share this post


Link to post

Thanks for uploading this to CA and offering to support it. 

 

 

I've been using an older version which came with a 'helper' file and a file to add cron jobs.  How do I add cron jobs to this version.

 

 

Old version:

 

 

# daily backup of test folder to blackblaze b2 mytest bucket (with log option)
#0 0 * * * /boot/config/plugins/rclone/rclone_helper --log sync /mnt/user/Public/test b2:mytest

 

 

Do I now just do:




# daily backup of test folder to blackblaze b2 mytest bucket (with log option)
#0 0 * * * /boot/config/plugins/rclone-beta --log sync /mnt/user/Public/test b2:mytest

 

 

Also, are the rclone config settings preserved on reboot?

 

 

Thanks

Share this post


Link to post

You can't use the same method as the would require editing  the plg each time an update is released.

I would use the user scripts plugin as I state in the OP or manually adding the cron job to your go file

The config is preserved :)

Share this post


Link to post

You can't use the same method as the would require editing  the plg each time an update is released.

I would use the user scripts plugin as I state in the OP or manually adding the cron job to your go file

The config is preserved :)

 

Makes sense.  For the user script do I just use the following format:

 

 

rclone copy /mnt/user/server/backup/myfile.bin amazon:

 

So happy someone is working on this plugin now.

Share this post


Link to post

I can't seem to find any sort of changelog for rclone beta. Not the plugin but the program itself. I can find the regular changelog but nothing for the beta version. Can someone direct me?

Share this post


Link to post

I can't seem to find any sort of changelog for rclone beta. Not the plugin but the program itself. I can find the regular changelog but nothing for the beta version. Can someone direct me?

 

Beta releases are generated from each commit to master. Note these are named like

 

Source: http://rclone.org/downloads/

Share this post


Link to post

The stable branch of rclone has been updated today. Because of this i have now released a version of the plugin which follows the stable branch. You can use this if you dont want to be on bleeding edge beta releases :)

I will keep both versions updated as new releases are made :)

Share this post


Link to post

The stable branch of rclone has been updated today. Because of this i have now released a version of the plugin which follows the stable branch. You can use this if you dont want to be on bleeding edge beta releases :)

I will keep both versions updated as new releases are made :)

 

Oh, that's good. The only real reason for using the beta in the first place was that the new features were necessary for our particular usage scenario. Good to have the peace of mind of a stable ranch.

Share this post


Link to post

Is there a list somewhere of what files types work when mounted?  E.g I can't open ms office files.

 

 

Loving the possibilities though - just wish my upload was faster so I could seriously consider uploading my Plex files.  At the moment I'm uploading to cloud sync so I won't need my server on all the time.... eventually

Share this post


Link to post

Is there a list somewhere of what files types work when mounted?  E.g I can't open ms office files.

 

 

Loving the possibilities though - just wish my upload was faster so I could seriously consider uploading my Plex files.  At the moment I'm uploading to cloud sync so I won't need my server on all the time.... eventually

 

Strange. I don't know why Office files wouldn't work. Maybe because they aren't really streamable, but I doubt that should matter, especially given their size. I haven't come across anything about it. There is a community forum on rclone that might be worth a look if no one here know.

 

But, yeah, I've been looking for a long time for a solution like this and there is finally a viable solution, as long as you have the speed. I'm just going to have to spend the next few years uploading until we finally get some cost effective fiber in my area, which I am told will be soon. We already have one provider but they are ridiculously expensive and have very small caps and the upload is restricted to 50mbps. Lame. We will eventually be getting some competition from what I can find and they have the same price and speeds as Google Fiber. Ironically enough, I have 3 siblings that live in Google Fiber areas, or soon to be Google Fiber areas, and my parents are moving to another. Yet none of them actually have Google Fiber, except my brother but he gets the slow, free, option. Tthe one person (me) that would actually get use out of it is stuck in the boondocks. I might either wait to see if my parents get it and just work out some sort of arrangement where I mail a 512gb flash drive back and forth or pay for my brother's gigabit for a month and drive down there with my server in tow and upload everything within that month and come back for it. First world problems.

Share this post


Link to post

Is there a list somewhere of what files types work when mounted?  E.g I can't open ms office files.

 

 

Loving the possibilities though - just wish my upload was faster so I could seriously consider uploading my Plex files.  At the moment I'm uploading to cloud sync so I won't need my server on all the time.... eventually

 

Huh - I just tested it and have the same problem. I agree with bobbintb that you should consult the rclone forum or the rclone github repo. Maybe the developer can shed some light on this.

Edit: I'm using Google Drive

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now