Hi Everyone,

 

I've searched and searched but can't find a how to for my precise issue.

 

I have 2 unraid servers for television shows but one of them is offsite with very very slow internet.  When I copy to one I also copy off to an external hd which gets plugged into the remote machine to update it.  I do this every few months as its a 4 hour drive away.  I got tired of having to duplicate everything on the external and had the idea of comparing the two shares and then only offloading the missing files to the external as a one shot deal rather than a little every day.  I need the results in some form that I can use excel to build a script to offload the missing files to the external.  what ideas can you guys think of?  I can load and run anything on both machines but I can't sync over the internet due to speed and bandwidth caps.  Any help will be most appreciated.

 

- Jonny

Just to clarify ....

 

when you say you can run anything on both machines, do you mean you have remote access to the 2nd machine or you can run something when at the location of the 2nd machine?

 

I’m away from my unRaid system until tomorrow evening, so this is just a suggestion until I can double check it ...

 

log into unraid2 system.

change directory to your TV share, eg

  cd /mnt/user/TV

run the following command to produce a list of files into the file list2

  find . -type f -print >list2

 

log into unraid1 system and do the same commands but put the output into the file list1

  cd /mnt/user/TV

  find . -type f -print > list1

 

you will need to somehow copy the file list2 from the unraid2 system onto unraid1 and put it in the same directory as the list1 file

 

then run the command

  comm -2 -3 list1 list2 >list

 

the file list should now contain those files that are on your unraid1 system but not on your unraid2 system.   Hopefully you can do something useful with this list.

I have remote access to the distant machine so I can install any apps or run any scripts needed.

 

I'll try your suggestion and let you know.  Thanks!
 

I'm getting:

 

comm: file 1 is not in sorted order

comm: file 2 is not in sorted order

 

When I vi the files the remote machine has half as much as the local machine but I think its all the .actor files which I can ignore.

Ah.   The dangers of scripting when you are not at a machine where you can check it out.

 

Add the sort command to the command lines doing the find, eg

 

    find . -type f -print | sort >list1

 

This will ensure the contents of the files being compared by the comm command are sorted.

 

That worked great.  and fast!

 

this is exactly what I needed.  thanks again!

That’s great.   Happy to have helped.

You can also use the rsync command to only copy the missing files.

rsync -avzh --progress /mnt/user/ShareSource /mnt/disks/ShareDestination

 

You can also add the --delete-before tag at the end if you want to remove old files from the destination share. Careful with this those. If you set up automatic sync, it can bite you by deleting data that may have been lost or corrupted.

 

(Came here searching for something and thought my addition might help)

I'll second rsync, use it daily and for years, it's rock solid. There are many options, -avzh -progress gets the job done for most people.

 

rsync is single threaded, if you have an extremely large data-set you can use  msrsync ( https://github.com/jbd/msrsync ), it's a multi-thread version of rsync, I used many years ago to migrate a database.