I will be syncing (rsyncing, btsyncing, syncthing) between parts of the world where the reliability of the internet connection is very iffy.  To validate the sync , but I also want to validate that the files actually all arrived first doing a simple filename, size, date compare for a transfer that is initially about 100,000 files and constantly changing.  We would output a text file of the mismatched files.  I want to create an automated script that can be run via cron on a regular basis.

 

I know there are a million ways to do this, but is there any best practices?  (is bash sufficient or do we need to go to awk, perl, or python)

 

Is there any scripts that would be a good starting point?  (my experience with all the above is noob for bash and extreme noob for the rest...)

 

I plan on using bunker and the file integrity plugin to check the file contents after the transfer is complete.

 

But first I need to verify that the files arrived first.

 

(both servers are unRaid 6.1.9 at present)

 

I think rsync takes care of that for you, completely, so long as you use the right options.  The built-in checksumming works down to the device, which maybe isn't the last inch, but very close.  Your followup hash check would confirm.  I don't think you'll see *any* mismatches.

It seems that rsync can't break through all the firewalls successfully.  So far bt sync is doing best. 

 

But the status of the sync can be a mystery.  If it breaks, I want to know that it has stopped syncing so I can restart the sync.

As of today, this sync is still progressing, but slowly.  I will hold my breath that it continues to be stable.

 

Still I would like to have a verification script that give the status of the ongoing sync.