jonnypajamas Posted May 1, 2018 Share Posted May 1, 2018 Hi Everyone, I've searched and searched but can't find a how to for my precise issue. I have 2 unraid servers for television shows but one of them is offsite with very very slow internet. When I copy to one I also copy off to an external hd which gets plugged into the remote machine to update it. I do this every few months as its a 4 hour drive away. I got tired of having to duplicate everything on the external and had the idea of comparing the two shares and then only offloading the missing files to the external as a one shot deal rather than a little every day. I need the results in some form that I can use excel to build a script to offload the missing files to the external. what ideas can you guys think of? I can load and run anything on both machines but I can't sync over the internet due to speed and bandwidth caps. Any help will be most appreciated. - Jonny Quote Link to comment
remotevisitor Posted May 1, 2018 Share Posted May 1, 2018 Just to clarify .... when you say you can run anything on both machines, do you mean you have remote access to the 2nd machine or you can run something when at the location of the 2nd machine? Quote Link to comment
remotevisitor Posted May 1, 2018 Share Posted May 1, 2018 I’m away from my unRaid system until tomorrow evening, so this is just a suggestion until I can double check it ... log into unraid2 system. change directory to your TV share, eg cd /mnt/user/TV run the following command to produce a list of files into the file list2 find . -type f -print >list2 log into unraid1 system and do the same commands but put the output into the file list1 cd /mnt/user/TV find . -type f -print > list1 you will need to somehow copy the file list2 from the unraid2 system onto unraid1 and put it in the same directory as the list1 file then run the command comm -2 -3 list1 list2 >list the file list should now contain those files that are on your unraid1 system but not on your unraid2 system. Hopefully you can do something useful with this list. Quote Link to comment
jonnypajamas Posted May 2, 2018 Author Share Posted May 2, 2018 I have remote access to the distant machine so I can install any apps or run any scripts needed. I'll try your suggestion and let you know. Thanks! Quote Link to comment
jonnypajamas Posted May 2, 2018 Author Share Posted May 2, 2018 I'm getting: comm: file 1 is not in sorted order comm: file 2 is not in sorted order When I vi the files the remote machine has half as much as the local machine but I think its all the .actor files which I can ignore. Quote Link to comment
remotevisitor Posted May 2, 2018 Share Posted May 2, 2018 Ah. The dangers of scripting when you are not at a machine where you can check it out. Add the sort command to the command lines doing the find, eg find . -type f -print | sort >list1 This will ensure the contents of the files being compared by the comm command are sorted. Quote Link to comment
jonnypajamas Posted May 2, 2018 Author Share Posted May 2, 2018 That worked great. and fast! this is exactly what I needed. thanks again! Quote Link to comment
remotevisitor Posted May 2, 2018 Share Posted May 2, 2018 That’s great. Happy to have helped. Quote Link to comment
Hobbesthecalvinist Posted April 16, 2020 Share Posted April 16, 2020 You can also use the rsync command to only copy the missing files. rsync -avzh --progress /mnt/user/ShareSource /mnt/disks/ShareDestination You can also add the --delete-before tag at the end if you want to remove old files from the destination share. Careful with this those. If you set up automatic sync, it can bite you by deleting data that may have been lost or corrupted. (Came here searching for something and thought my addition might help) Quote Link to comment
fishface Posted April 16, 2020 Share Posted April 16, 2020 I'll second rsync, use it daily and for years, it's rock solid. There are many options, -avzh -progress gets the job done for most people. rsync is single threaded, if you have an extremely large data-set you can use msrsync ( https://github.com/jbd/msrsync ), it's a multi-thread version of rsync, I used many years ago to migrate a database. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.