Rclone/rsync checking.....

[BDPM]

If I rclone copy or rsync to a disk that's outside the array, unassigned, it always checks full list of files again every time. Takes as long as the first time.  I can work around with size or time only, but...

 

A) I dont know why it does this and I'm curious.

B) I'm wondering if there's a better method.

 

I have pictures, documents, music on the array, and a backup drive formatted ntfs outside the array for backups only. 

 

Rsync or rclone every night from the array to the backup drive and then to cloud storage.

 

My current script uses rclone but I've tried rsync and get same result. They both copy new or changed files, then check every single other file in the destination. 

 

rclone copy /mnt/user/Documents/ /mnt/disks/3tb-backup/Documents/

 

Sending to cloud storage with rclone only takes seconds. No checking destination again. 

 

Any info and/or advise will be greatly appreciated. 

 

Thanks in advance 

BDPM

Edited May 15, 2019 by BDPM

[ken-ji]

rsync will always check the full list of files everytime and the non-default option of using checksums will definitely cause all files on both sides to be read, to generate the checksums for comparison.

rclone sending to cloud storage I think uses the size and timestamp check first, before falling back to the checksums (depends on the cloud provider)

[tr0910]

I always get a full compare both sides but it takes only seconds for 70,000 files between USA and China. How many files are you rsyncing in one go?

My experience with rclone is similar if a bit faster since it is USA to Google drive. No China delays.

That a look at your rsync command. Do a smaller group of files to test. It may just be a typo...

Sent from my chisel, carved into granite

[BDPM]

On 5/15/2019 at 4:13 PM, ken-ji said:

rsync will always check the full list of files everytime and the non-default option of using checksums will definitely cause all files on both sides to be read, to generate the checksums for comparison.

rclone sending to cloud storage I think uses the size and timestamp check first, before falling back to the checksums (depends on the cloud provider)

Thanks.  Yeah Im far from a linux guru, but I believe that the very nature of rsync or rclone require checks on both source and destination. It just baffles me as to why when I do a rsync or rclone test from array to array its nearly instantaneous, unless it needs to add or update file(s), but when I do the same test from array to the unassigned drive it takes nearly as long to simply check as it did to write the files to an empty directory. 

 

Im using --size-only and its all good. Adding or updating files as it should.

 

On 5/16/2019 at 5:02 AM, tr0910 said:

I always get a full compare both sides but it takes only seconds for 70,000 files between USA and China. How many files are you rsyncing in one go?

My experience with rclone is similar if a bit faster since it is USA to Google drive. No China delays.

That a look at your rsync command. Do a smaller group of files to test. It may just be a typo...

Sent from my chisel, carved into granite
 

Im doing 47,034 files for about 296.9 GB.  Took time to put it up there, but when the daily rclone runs, it takes maybe 3 seconds. Works great. I use backblaze b2.

 

I'll just keep using --size-only for my local backups. Works just fine.

 

Thanks both of you for your input.

 

Have a great day!