tr0910 Posted June 10, 2016 Share Posted June 10, 2016 I will be syncing (rsyncing, btsyncing, syncthing) between parts of the world where the reliability of the internet connection is very iffy. To validate the sync , but I also want to validate that the files actually all arrived first doing a simple filename, size, date compare for a transfer that is initially about 100,000 files and constantly changing. We would output a text file of the mismatched files. I want to create an automated script that can be run via cron on a regular basis. I know there are a million ways to do this, but is there any best practices? (is bash sufficient or do we need to go to awk, perl, or python) Is there any scripts that would be a good starting point? (my experience with all the above is noob for bash and extreme noob for the rest...) I plan on using bunker and the file integrity plugin to check the file contents after the transfer is complete. But first I need to verify that the files arrived first. (both servers are unRaid 6.1.9 at present) Quote Link to comment
RobJ Posted June 10, 2016 Share Posted June 10, 2016 I think rsync takes care of that for you, completely, so long as you use the right options. The built-in checksumming works down to the device, which maybe isn't the last inch, but very close. Your followup hash check would confirm. I don't think you'll see *any* mismatches. Quote Link to comment
tr0910 Posted June 11, 2016 Author Share Posted June 11, 2016 It seems that rsync can't break through all the firewalls successfully. So far bt sync is doing best. But the status of the sync can be a mystery. If it breaks, I want to know that it has stopped syncing so I can restart the sync. Quote Link to comment
tr0910 Posted June 17, 2016 Author Share Posted June 17, 2016 As of today, this sync is still progressing, but slowly. I will hold my breath that it continues to be stable. Still I would like to have a verification script that give the status of the ongoing sync. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.