[Plugin] rclone


Waseh

906 posts in this topic Last Reply

Recommended Posts

5 minutes ago, rragu said:

Thanks! I'll look into the resources you posted.

 

As for not writing to the rclone Google Drive mount, (1) it's a slightly more widely known tip now 😅, (2) while I'll switch to using "rclone copy", is there any particular negative effect to transferring data to Google Drive in the way I've been doing (e.g. data loss/corruption) or is it just lower performance?

The biggest issue with writing to the rclone mount is just flat out reliability. People pretty much ALWAYS complain about it. Either it gives errors, or its slow, or it doesn't copy everything you told it. Pretty much, it isn't something you'd wanna use/trust. I know its convenient......sorry.

It is such a well-known thing that there are a ton of very popular scripts around GitHub that basically monitor directories and perform regular rclone sync/copy jobs for you in the background on a regular schedule (like every 2 minutes or whatever you set). One such script is called "Cloudplow". Very well documented/mature. Easy to find on github.

 

Now, rclone mount is absolutely remarkable as a read source. It is excellent for reading.

Link to post
  • Replies 905
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

rclone Plugin   Hey everyone   This is a simple plugin which installs rclone on your unraid system. The plugins have now been merged so both the stable and beta branch are

Hey guys Sorry for the lack of updates in a (long) while ? Real life has been taking up a lot of time and my own install of rclone has been sufficient for my needs. However both the stable branch

Someone beat me to it. And probably explained it in a nicer way than i would have Reinstalling the plugin will also result in the newest version being pulled Take a look at the change log

Posted Images

2 hours ago, Stupifier said:

The biggest issue with writing to the rclone mount is just flat out reliability. People pretty much ALWAYS complain about it. Either it gives errors, or its slow, or it doesn't copy everything you told it. Pretty much, it isn't something you'd wanna use/trust. I know its convenient......sorry.

It is such a well-known thing that there are a ton of very popular scripts around GitHub that basically monitor directories and perform regular rclone sync/copy jobs for you in the background on a regular schedule (like every 2 minutes or whatever you set). One such script is called "Cloudplow". Very well documented/mature. Easy to find on github.

 

Now, rclone mount is absolutely remarkable as a read source. It is excellent for reading.

Just tried out "rclone copy"....the difference is night and day

 

Test files: 4 files (total of 12.3 GB; between 2.3-3.6GB each)

Average transfer speed using rclone mount: 19.4MB/s

Average transfer speed using "rclone copy": 60.9MB/s

Average transfer speed using "rclone copy" and chunk-size 256M: 78.1MB/s

 

The only drawback is heightened CPU/RAM usage but I'm sure I can manage that with a script like you mentioned.

 

Thanks very much for all your help!

Link to post
2 hours ago, rragu said:

Just tried out "rclone copy"....the difference is night and day

 

1.) Check the destination documentation of rclone. Every destination has its own special parameters and behaviour like Google Drive has its own, too:

https://rclone.org/drive/

 

2.) Do not copy/sync to the mount of the destination like /mnt/disks/gdrive. Use only the rclone alias like gdrive:subfolder. rclone does not work properly if the target is a local path. Example: If you use the mount path and your destination is a webdav server it does not presere the file modification time.

 

3.) If it does not preserve the file modifcation time it must use checksums to verify that the destination file is the same (or not) before being able to skip (or overwrite it). This means it downloads and calculates the checksums (high cpu usage) and uploads at the same time (rclone uses 8 parallel running --checkers for this).

 

4.) There are minor performance differences between sync and copy as they act a little bit different, but finally this should not influence the transfer speed of files:

sync = deletes already existing files from destination that are not present on source

copy = ignores existing files on destination

 

Conclusion:

Check if your transfer preserves the timestamps. If yes, then disable the checksum check through --ignore-checksum (if you are ok with that) and test different --checkers and --transfers (4 is the default) values until you reach the best performance. For example I use "--checkers 2 --transfers 1" for my Nextcloud destination as there was no benefit uploading multiple files and finally it does not raise the total transfer speed if I have two parallel uploads instead of one.

 

 

 

 

My command as an example:

rclone sync /mnt/user/Software owncube:software --create-empty-src-dirs --ignore-checksum --bwlimit 3M --checkers 2 --transfers 1 -vv --stats 10s

 

 

 

Link to post
On 8/29/2020 at 5:25 AM, mgutt said:

1.) Check the destination documentation of rclone. Every destination has its own special parameters and behaviour like Google Drive has its own, too:

https://rclone.org/drive/

 

2.) Do not copy/sync to the mount of the destination like /mnt/disks/gdrive. Use only the rclone alias like gdrive:subfolder. rclone does not work properly if the target is a local path. Example: If you use the mount path and your destination is a webdav server it does not presere the file modification time.

 

3.) If it does not preserve the file modifcation time it must use checksums to verify that the destination file is the same (or not) before being able to skip (or overwrite it). This means it downloads and calculates the checksums (high cpu usage) and uploads at the same time (rclone uses 8 parallel running --checkers for this).

 

4.) There are minor performance differences between sync and copy as they act a little bit different, but finally this should not influence the transfer speed of files:

sync = deletes already existing files from destination that are not present on source

copy = ignores existing files on destination

 

Conclusion:

Check if your transfer preserves the timestamps. If yes, then disable the checksum check through --ignore-checksum (if you are ok with that) and test different --checkers and --transfers (4 is the default) values until you reach the best performance. For example I use "--checkers 2 --transfers 1" for my Nextcloud destination as there was no benefit uploading multiple files and finally it does not raise the total transfer speed if I have two parallel uploads instead of one.

 

My command as an example:


rclone sync /mnt/user/Software owncube:software --create-empty-src-dirs --ignore-checksum --bwlimit 3M --checkers 2 --transfers 1 -vv --stats 10s

 

Thanks! I lowered the checkers to 2 and transfers to 1. Combined with a chunk-size of 256M, I get the same ~80MBps with half the CPU utilization as before, even without --ignore checksum

Link to post
  • 2 weeks later...

Hi Guys

 

So I’ve finally started using rclone mount again after several years with only doing weekly scripted backups.

This prompted me to give the plugin a bit of an overhaul to fix some of my own long-standing nuisances with the plugin.

 

Gone are the default scripts and the script editor (You should really use UserScripts for all your script needs). The config file can still be edited.

The settings page has been overhauled and i intend to deprecate the beta branch of the plugin for this new unified model.

You are now able to change branch at will in the settings page as well as update rclone if a new version is available. Current version as well as newest version will now be shown in the settings page.

 

If you want to check it out you can install/update the plugin manually by inserting THIS URL in the plugins/install plugin settings menu. It's a drop in replacement for the stable branch.

 

Whenever this version has proven itself without any major bugs i will merge it with the master branch and deprecate the beta branch.

 

@Stupifier @DZMM if you get the chance please test out the new version and see if there's any glaring bugs I’ve missed.

 

Edit: A picture to show the new settings page

BJ1JRoi.png

Edited by Waseh
Link to post

@Waseh
Just did an install.....really basic look over. Everything looks good so far. I did NOT experience your beta version number issue though. First image shows the update to beta rclone popup. The next image shows a version check on stable branch, then another version check once flipped over to beta branch.

All I've done so far is copy over my .rclone.conf file and run a few simple rclone copy and rclone lsd commands and rclone mount commands. Ran into zero issues.

8dAtwRy.png

 

EQfJf4C.png

Edited by Stupifier
Link to post
52 minutes ago, mgutt said:

Does this help? It's related to Google drive and happens because it's possible to have the same filename two times:

https://forum.rclone.org/t/rclone-mount-google-drive-shows-empty-directory-which-is-not-empty/3838/6

I start to see the error once the mergerfs command finishes mounting i think, I also have been recently moving alot of files around so i will check to see if i have any dupes as per the link you provided. I will report back. thanks

 

Link to post

@Waseh I've just updated and spotted a few things.

 

- I'm currently on 1.53.0 and when I tried to update to 1.53.1 it didn't work and asked if rclone was running 

- The config editor seems better as before I couldn't use it as it wouldn't show my whole file.  However, even though it shows my file I can't apply any edits as it says

 

please shorten this text to 4096 characters or less (you are currently using 5648 characters)

 

Link to post

I never tried or used the config editor stuff....I dunno....just seemed like it was more straight forward to edit the .rclone.config file directly or as rclone devs intend by using the 'rclone config' command in a terminal.

But yes....I did not attempt to update while rclone processes were active. Verified using htop command. Given that, on Ubuntu, rclone is capable of updating seemlessly while rclone mounts are active. Don't ask me how...but I know for a fact it does. I'm sure the same applies to other modern OS's as well.

Link to post
On 9/17/2020 at 11:29 AM, Waseh said:

@DZMM with the risk of sounding like a broken record, is rclone running? (Including a rclone mount 😉)

 

I'll bump up the max chars on the editor 

yes it works fine.  I just did a reboot and the plugin disappeared - I don't know if it was because of my failed upgrade to 1.53.1

Link to post

@DZMM

Yea actually the update script might actually ruin the ability for plugin to survive a reboot (if no internet connection is present at boot)

 

I'll fix that for the release.

I'm pushing the update to the main branch in an hour or so it will be live for everyone

Link to post

@DZMM
I can't seem to reproduce it.

Oh and my question about rclone running was not a question about if it was working but related to the fact that if the rclone binary probably was busy.

With that being said i have updated the way the binary is replaced so we shouldn't run into the same problem again.

Edited by Waseh
Link to post

Im curious if there is any head-way (or keeping track of progress) on Bi-directional Sync functionality. I say this not even sure if its being pursued at the moment, so forgive me if i've mis-assumed. I remember a few months back reading some discussion on this functionality, but havent seen anything lately.

Link to post

That's because you are on the beta branch. As mentioned that one is getting deprecated while the old stable branch now provides the ability to install both stable and beta.

 

So to sum up that is as expected 😄

Edited by Waseh
Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.