[Plugin] rclone


Waseh

Recommended Posts

5 minutes ago, rragu said:

Thanks! I'll look into the resources you posted.

 

As for not writing to the rclone Google Drive mount, (1) it's a slightly more widely known tip now 😅, (2) while I'll switch to using "rclone copy", is there any particular negative effect to transferring data to Google Drive in the way I've been doing (e.g. data loss/corruption) or is it just lower performance?

The biggest issue with writing to the rclone mount is just flat out reliability. People pretty much ALWAYS complain about it. Either it gives errors, or its slow, or it doesn't copy everything you told it. Pretty much, it isn't something you'd wanna use/trust. I know its convenient......sorry.

It is such a well-known thing that there are a ton of very popular scripts around GitHub that basically monitor directories and perform regular rclone sync/copy jobs for you in the background on a regular schedule (like every 2 minutes or whatever you set). One such script is called "Cloudplow". Very well documented/mature. Easy to find on github.

 

Now, rclone mount is absolutely remarkable as a read source. It is excellent for reading.

Link to comment
2 hours ago, Stupifier said:

The biggest issue with writing to the rclone mount is just flat out reliability. People pretty much ALWAYS complain about it. Either it gives errors, or its slow, or it doesn't copy everything you told it. Pretty much, it isn't something you'd wanna use/trust. I know its convenient......sorry.

It is such a well-known thing that there are a ton of very popular scripts around GitHub that basically monitor directories and perform regular rclone sync/copy jobs for you in the background on a regular schedule (like every 2 minutes or whatever you set). One such script is called "Cloudplow". Very well documented/mature. Easy to find on github.

 

Now, rclone mount is absolutely remarkable as a read source. It is excellent for reading.

Just tried out "rclone copy"....the difference is night and day

 

Test files: 4 files (total of 12.3 GB; between 2.3-3.6GB each)

Average transfer speed using rclone mount: 19.4MB/s

Average transfer speed using "rclone copy": 60.9MB/s

Average transfer speed using "rclone copy" and chunk-size 256M: 78.1MB/s

 

The only drawback is heightened CPU/RAM usage but I'm sure I can manage that with a script like you mentioned.

 

Thanks very much for all your help!

Link to comment
2 hours ago, rragu said:

Just tried out "rclone copy"....the difference is night and day

 

1.) Check the destination documentation of rclone. Every destination has its own special parameters and behaviour like Google Drive has its own, too:

https://rclone.org/drive/

 

2.) Do not copy/sync to the mount of the destination like /mnt/disks/gdrive. Use only the rclone alias like gdrive:subfolder. rclone does not work properly if the target is a local path. Example: If you use the mount path and your destination is a webdav server it does not presere the file modification time.

 

3.) If it does not preserve the file modifcation time it must use checksums to verify that the destination file is the same (or not) before being able to skip (or overwrite it). This means it downloads and calculates the checksums (high cpu usage) and uploads at the same time (rclone uses 8 parallel running --checkers for this).

 

4.) There are minor performance differences between sync and copy as they act a little bit different, but finally this should not influence the transfer speed of files:

sync = deletes already existing files from destination that are not present on source

copy = ignores existing files on destination

 

Conclusion:

Check if your transfer preserves the timestamps. If yes, then disable the checksum check through --ignore-checksum (if you are ok with that) and test different --checkers and --transfers (4 is the default) values until you reach the best performance. For example I use "--checkers 2 --transfers 1" for my Nextcloud destination as there was no benefit uploading multiple files and finally it does not raise the total transfer speed if I have two parallel uploads instead of one.

 

 

 

 

My command as an example:

rclone sync /mnt/user/Software owncube:software --create-empty-src-dirs --ignore-checksum --bwlimit 3M --checkers 2 --transfers 1 -vv --stats 10s

 

 

 

Link to comment
On 8/29/2020 at 5:25 AM, mgutt said:

1.) Check the destination documentation of rclone. Every destination has its own special parameters and behaviour like Google Drive has its own, too:

https://rclone.org/drive/

 

2.) Do not copy/sync to the mount of the destination like /mnt/disks/gdrive. Use only the rclone alias like gdrive:subfolder. rclone does not work properly if the target is a local path. Example: If you use the mount path and your destination is a webdav server it does not presere the file modification time.

 

3.) If it does not preserve the file modifcation time it must use checksums to verify that the destination file is the same (or not) before being able to skip (or overwrite it). This means it downloads and calculates the checksums (high cpu usage) and uploads at the same time (rclone uses 8 parallel running --checkers for this).

 

4.) There are minor performance differences between sync and copy as they act a little bit different, but finally this should not influence the transfer speed of files:

sync = deletes already existing files from destination that are not present on source

copy = ignores existing files on destination

 

Conclusion:

Check if your transfer preserves the timestamps. If yes, then disable the checksum check through --ignore-checksum (if you are ok with that) and test different --checkers and --transfers (4 is the default) values until you reach the best performance. For example I use "--checkers 2 --transfers 1" for my Nextcloud destination as there was no benefit uploading multiple files and finally it does not raise the total transfer speed if I have two parallel uploads instead of one.

 

My command as an example:


rclone sync /mnt/user/Software owncube:software --create-empty-src-dirs --ignore-checksum --bwlimit 3M --checkers 2 --transfers 1 -vv --stats 10s

 

Thanks! I lowered the checkers to 2 and transfers to 1. Combined with a chunk-size of 256M, I get the same ~80MBps with half the CPU utilization as before, even without --ignore checksum

Link to comment
  • 2 weeks later...

Hi Guys

 

So I’ve finally started using rclone mount again after several years with only doing weekly scripted backups.

This prompted me to give the plugin a bit of an overhaul to fix some of my own long-standing nuisances with the plugin.

 

Gone are the default scripts and the script editor (You should really use UserScripts for all your script needs). The config file can still be edited.

The settings page has been overhauled and i intend to deprecate the beta branch of the plugin for this new unified model.

You are now able to change branch at will in the settings page as well as update rclone if a new version is available. Current version as well as newest version will now be shown in the settings page.

 

If you want to check it out you can install/update the plugin manually by inserting THIS URL in the plugins/install plugin settings menu. It's a drop in replacement for the stable branch.

 

Whenever this version has proven itself without any major bugs i will merge it with the master branch and deprecate the beta branch.

 

@Stupifier @DZMM if you get the chance please test out the new version and see if there's any glaring bugs I’ve missed.

 

Edit: A picture to show the new settings page

BJ1JRoi.png

Edited by Waseh
  • Like 2
Link to comment

@Waseh
Just did an install.....really basic look over. Everything looks good so far. I did NOT experience your beta version number issue though. First image shows the update to beta rclone popup. The next image shows a version check on stable branch, then another version check once flipped over to beta branch.

All I've done so far is copy over my .rclone.conf file and run a few simple rclone copy and rclone lsd commands and rclone mount commands. Ran into zero issues.

8dAtwRy.png

 

EQfJf4C.png

Edited by Stupifier
Link to comment
52 minutes ago, mgutt said:

Does this help? It's related to Google drive and happens because it's possible to have the same filename two times:

https://forum.rclone.org/t/rclone-mount-google-drive-shows-empty-directory-which-is-not-empty/3838/6

I start to see the error once the mergerfs command finishes mounting i think, I also have been recently moving alot of files around so i will check to see if i have any dupes as per the link you provided. I will report back. thanks

 

Link to comment

@Waseh I've just updated and spotted a few things.

 

- I'm currently on 1.53.0 and when I tried to update to 1.53.1 it didn't work and asked if rclone was running 

- The config editor seems better as before I couldn't use it as it wouldn't show my whole file.  However, even though it shows my file I can't apply any edits as it says

 

please shorten this text to 4096 characters or less (you are currently using 5648 characters)

 

Link to comment

I never tried or used the config editor stuff....I dunno....just seemed like it was more straight forward to edit the .rclone.config file directly or as rclone devs intend by using the 'rclone config' command in a terminal.

But yes....I did not attempt to update while rclone processes were active. Verified using htop command. Given that, on Ubuntu, rclone is capable of updating seemlessly while rclone mounts are active. Don't ask me how...but I know for a fact it does. I'm sure the same applies to other modern OS's as well.

Link to comment
On 9/17/2020 at 11:29 AM, Waseh said:

@DZMM with the risk of sounding like a broken record, is rclone running? (Including a rclone mount 😉)

 

I'll bump up the max chars on the editor 

yes it works fine.  I just did a reboot and the plugin disappeared - I don't know if it was because of my failed upgrade to 1.53.1

Link to comment

@DZMM
I can't seem to reproduce it.

Oh and my question about rclone running was not a question about if it was working but related to the fact that if the rclone binary probably was busy.

With that being said i have updated the way the binary is replaced so we shouldn't run into the same problem again.

Edited by Waseh
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.