Jump to content
BDPM

Rclone/rsync checking.....

4 posts in this topic Last Reply

Recommended Posts

Posted (edited)

If I rclone copy or rsync to a disk that's outside the array, unassigned, it always checks full list of files again every time. Takes as long as the first time.  I can work around with size or time only, but...

 

A) I dont know why it does this and I'm curious.

B) I'm wondering if there's a better method.

 

I have pictures, documents, music on the array, and a backup drive formatted ntfs outside the array for backups only. 

 

Rsync or rclone every night from the array to the backup drive and then to cloud storage.

 

My current script uses rclone but I've tried rsync and get same result. They both copy new or changed files, then check every single other file in the destination. 

 

rclone copy /mnt/user/Documents/ /mnt/disks/3tb-backup/Documents/

 

Sending to cloud storage with rclone only takes seconds. No checking destination again. 

 

Any info and/or advise will be greatly appreciated. 

 

Thanks in advance 

BDPM

Edited by BDPM

Share this post


Link to post

rsync will always check the full list of files everytime and the non-default option of using checksums will definitely cause all files on both sides to be read, to generate the checksums for comparison.

rclone sending to cloud storage I think uses the size and timestamp check first, before falling back to the checksums (depends on the cloud provider)

Share this post


Link to post

I always get a full compare both sides but it takes only seconds for 70,000 files between USA and China. How many files are you rsyncing in one go?

My experience with rclone is similar if a bit faster since it is USA to Google drive. No China delays.

That a look at your rsync command. Do a smaller group of files to test. It may just be a typo...

Sent from my chisel, carved into granite

Share this post


Link to post
On 5/15/2019 at 4:13 PM, ken-ji said:

rsync will always check the full list of files everytime and the non-default option of using checksums will definitely cause all files on both sides to be read, to generate the checksums for comparison.

rclone sending to cloud storage I think uses the size and timestamp check first, before falling back to the checksums (depends on the cloud provider)

Thanks.  Yeah Im far from a linux guru, but I believe that the very nature of rsync or rclone require checks on both source and destination. It just baffles me as to why when I do a rsync or rclone test from array to array its nearly instantaneous, unless it needs to add or update file(s), but when I do the same test from array to the unassigned drive it takes nearly as long to simply check as it did to write the files to an empty directory. 

 

Im using --size-only and its all good. Adding or updating files as it should.

 

On 5/16/2019 at 5:02 AM, tr0910 said:

I always get a full compare both sides but it takes only seconds for 70,000 files between USA and China. How many files are you rsyncing in one go?

My experience with rclone is similar if a bit faster since it is USA to Google drive. No China delays.

That a look at your rsync command. Do a smaller group of files to test. It may just be a typo...

Sent from my chisel, carved into granite
 

Im doing 47,034 files for about 296.9 GB.  Took time to put it up there, but when the daily rclone runs, it takes maybe 3 seconds. Works great. I use backblaze b2.

 

I'll just keep using --size-only for my local backups. Works just fine.

 

Thanks both of you for your input.

 

Have a great day!

 

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.