Syncing files between Unraid and multiple Laptops without the Cloud / DropBox


Recommended Posts

I just started using Rsync and Deltacopy to sync files between my laptop and a network share on my unraid server.

http://lime-technology.com/forum/index.php?topic=2165.msg16145#msg16145 '>http://lime-technology.com/forum/index.php?topic=2165.msg16145#msg16145  ( Thanks WeeboTech and Tibbar )

http://lime-technology.com/forum/index.php?topic=25619.0 ( Thanks Trurl )

 

I wanted to know a little more about how this syncing works:

 

I am syncing \\tower\Backup to D:\Backup

 

If I create a file in D:\Backup it is appearing in \\tower\Backup which is great.

 

My question is do most people using this only edit files at D:\Backup or is it ok to edit the file at  \\tower\Backup ?

 

As a test I created a test file in D:\Backup and then attempted to delete the file by browsing to \\tower\Backup to delete it.  I got a message saying I needed permission to do this.  Is this rsync stopping me from deleting the file?

 

I also was wondering if I can have Computer-1, computer-2, and Computer-3 successfully syncing to \\tower\Backup

I would like to have deltacopy on all three computers with the ability to add and edit files at each location, and have the ability to add a forth computer at a latter date and have all the past files propagate at the new location.

 

Is the above possible?

 

First when you setup deltacopy it seems if you want to sync a folder from your computer to the server you have to use the "Add Folder" option.  (You can't just specify a folder), Meaning deltacopy will create a folder on the server when you say "add folder" so instead of all your files ending up at \\tower\Backup they will end up at \\tower\Backup\UnraidBackup  ("UnraidBackup" is the name of the folder created with deltacopy when using "add folder"

 

With that said I tried to add a second computer and use the same name on the second computer (each computer has a folder called UnraidBackup if the names are different like "UnraidBackup1"and "UnraidBackup2" then you end up with:

\\tower\Backup\UnraidBackup1

\\tower\Backup\UnraidBackup2

and no files from computer 1 will end up at computer 2 or vice versa.

 

 

Is there another way to approach this with deltacopy or do I need to use, unison to make this work?

http://lime-technology.com/forum/index.php?topic=2165.msg16145#msg16145

 

I am trying to make this work a little like dropbox, but locally, meaning I am not going from computer1 to cloud, then to computer2  (I just want to go from computer1 to server, to computer2 ) just want a solution without the cloud having to be involved.

 

Thanks for any help on this.

 

Albin

Link to comment

It should work. Make sure all of the machines are using the same NTP source. You'll have to test to determine the best rsync settings for your requirements.

 

I really never thought to look at that and honestly I had to look up what NTP Source was "Network Time Protocol"

 

I really have not messed with that much, other than trying to using this to set the time on my routers.

I went to the Unraid time settings to see what time zone I am using and I am using (UTC-05:00) eastern Time (US & Canada) and the current date and time look fine, but the "Use NTP" option is set to "NO"

 

I set the "Use NTP" to "Yes", and set "NTP Server 1" to: 0.us.pool.ntp.org

I then set the computers NTP Servers to: 0.us.pool.ntp.org

(right clicked on the clock and clicked "adjust date/time", then picked the "Internet time" TAB and clicked change settings)

 

I need to do this son my other unraid server, I think the time is off on that one.  Thanks for the tip

 

I then tried to force a sync, (right clicked the deltacopy profile name and clicked "run now"). Syncing this way give you a line by line view of what deltacopy is doing.

 

I was getting a no such directory error.

 

I deleted the profile, then recreated the profile and syncing started to semi work.

 

I say semi because if I create a file (file1) on computer1, and then run a sync on computer1, the file is copied to the network share, but if I go to computer2 and run a sync, it deletes the file netwrok shares copy of file1 I just created.  If I then create a file (file2) on computer2 and sync, the file2 shows on network share.  At this point if I go to computer1 and run a sync, file2 is deleted and file1 is recopied to the network share. 

 

Are the above symptoms due to improper steps to setting up deltacopy.  Do I need to set up the folder on each computer, leave them empty, then setup the deltacopy profiles, and then run a sync at both locations prior to putting any files in either place?

 

Thanks for any help on this.  Looks like I am one step close and I am excited that this is something that could work.

 

Thanks

 

Albin

 

 

Link to comment

In DeltaCopy, select the profile and then go to the options tab. You probably have "Remove deleted files from server" checked. This will make it delete any files that don't exist on the pc. Uncheck this box.

 

If you right-click on a profile you can "Display run command". This will show you the exact rsync command that is used based on the options you have selected. You can also put additional options in the "Additional parameters" box on the options tab.

 

You may have to do a little research and try some things out to get the rsync just like you need it. I am not using it the way you want so I may not be much help.

 

Also, what I do for NTP is let unRAID get the time from the internet and then make everything on my local network get the time from my unRAID so there is less delay in getting everything else in synch.

 

Link to comment

In DeltaCopy, select the profile and then go to the options tab. You probably have "Remove deleted files from server" checked. This will make it delete any files that don't exist on the pc. Uncheck this box.

 

If you right-click on a profile you can "Display run command". This will show you the exact rsync command that is used based on the options you have selected. You can also put additional options in the "Additional parameters" box on the options tab.

 

You may have to do a little research and try some things out to get the rsync just like you need it. I am not using it the way you want so I may not be much help.

 

Also, what I do for NTP is let unRAID get the time from the internet and then make everything on my local network get the time from my unRAID so there is less delay in getting everything else in synch.

 

Thanks for the suggestion. 

It looks like deltaCopy will work fine for me if I have one client and Unraid as the server.

 

I do not think it will do 2 clients and Unraid as the Server (maybe if I was a programmer I could do it.... I am not a programmer)

 

If I uncheck "Remove deleted files from Server", I am guessing that would keep all files on the server are on Client 1 or Client 2.

 

Let say that client 1, and Client 2 have files 1, 2, 3, 4 and the server has 1,2,3,4, If I delete file 1 from Client 2 with "Remove deleted files from Server" unchecked, then I would end up with Client 1 and the server having files 1,2,3,4 and client 2 having 2,3,4

 

Now if Client 1 changes file 2, (now 2rev1) (and then did a sync), Client 1 would have 1,2rev1,3,4 the server would have 1,2rev1,3,4, but Client 2 would have 2 (original),3,4.  At this point if I do a sync on client 2, file 2rev1 would be over written and replaced with 2 (original), so the server would now have files 1,2 (original),3,4, going back to client 1 and doing a sync would over right 2 (original) with 2 rev1 on the server.

 

Basically Client 1 and Client 2 are not working together together to have the same files (with the settings I currently have they will continue to fight over what version of file 2 will reside on the server). I am guessing that Client 1 does not know that client 2 even exists and vice versa, so they are just doing what they are told, and that is to make sure that the files on the server are the same as the ones on self, and do not remove deleted files from server) (I am sure there is a more technical way to explain this, but this is more the layman's terms for what the end result seems to be.  Again, I understand rsync is powerful, but I do not have the knowledge to tackle writing the additional scripts, or parameters to make sure that client 1, client 2, and the server are all mirror each other.

 

I did notice a nice feature where  rysnc / deltacopy adjust the permissions of the files on the server in such a way that it seems you can not browse to the network share and delete or change files there.  This is nice seeing it would force the use to make changes to the files on the client and then sync the changes to the unraid server.

 

 

I did look at the "Additional Parameters" for deltacopy.  DeltaCopy's site pointed me to a samba site

http://www.samba.org/ftp/rsync/rsync.html

There is a lot of stuff there to explain rsync and all the different commands that can be used.

 

You mentioned that you are using DeltaCopy but not the way I am trying to use it.  Do you mind explaining how you are using DeltaCopy?

 

 

I did see that there is another product (other than deltacopy) that synametrics offers called Syncrify

http://web.synametrics.com/SyncrifyVsDeltaCopy.htm

http://web.synametrics.com/SyncrifyInstallationLinux.htm

 

It looks like it can be installed on FreeNAS

http://web.synametrics.com/syncrifyfreenas.htm

 

I see RussellinSacto started a post on this product back in 2011 and ProStuff1 said he would look at some, but I do not see where any more post were made to indicate there was any more progress:

http://lime-technology.com/forum/index.php?topic=13728.msg130496#msg130496

 

Syncrify seems to be more suited for what I am looking for.  Has anyone looked at getting Syncrify server on an unraid system.  Is it possible?

 

Trurl

By the way do you have a link to a lime-tech post explaining how to point a windows machine to the UNraid servers NTP / time to fetch the date and time?

 

Thanks

 

Albin

Link to comment

...

You mentioned that you are using DeltaCopy but not the way I am trying to use it.  Do you mind explaining how you are using DeltaCopy?

...

By the way do you have a link to a lime-tech post explaining how to point a windows machine to the UNraid servers NTP / time to fetch the date and time?

...

I am only using it to do scheduled backups. I know I could have used something else for this but I wanted to play around with rsync in unRAID. I am also using rsync on unRAID to produce monthly backups of critical files to ntfs-3g mounted esata for storage offsite.

 

I don't remember having to do anything on unRAID to enable ntpd. I just tell my windows machines to get the time from my unRAID server like this:

time.png.5ddbd03682a643bf7a4a764f4e5ae4f9.png

Link to comment

...

You mentioned that you are using DeltaCopy but not the way I am trying to use it.  Do you mind explaining how you are using DeltaCopy?

...

By the way do you have a link to a lime-tech post explaining how to point a windows machine to the UNraid servers NTP / time to fetch the date and time?

...

I am only using it to do scheduled backups. I know I could have used something else for this but I wanted to play around with rsync in unRAID. I am also using rsync on unRAID to produce monthly backups of critical files to ntfs-3g mounted esata for storage offsite.

 

I don't remember having to do anything on unRAID to enable ntpd. I just tell my windows machines to get the time from my unRAID server like this:

 

I think deltacopy will be a good solution for the backups.

 

I did try entering the server name tower2 for my windows machine to fetch the time.  It worked on the new version of unraid, but when trying to fetch the time from my 4.7 version unraid (server name tower ), I got an error from windows.  (so it works with the newer version anyway... not sure if there is a setting for 4.7 to make it work, but I do not need it to sync with that one.)

 

Thanks again,

 

Albin

Link to comment

I did not know that Unraid had a NPT server.  Just synced my PC to my unraid server using its IP address.  I hard code the IP in unraid as I noticed in the past that it appears to stop/start the SMB share when renewing the IP address.

 

I did not try using the hard assigned IP address of my tower.  Glad to know that is an option.  I think I will keep mine pointed to tower opposed to the ip address.  I do not see changing the IP address, but I think there is a greater chance of me changing the IP address than there is a chance of me changing the server name.

 

I just set "tower" to point to:

 

0.us.pool.ntp.org

1.us.pool.ntp.org

2.us.pool.ntp.org

 

And set "tower2" to point to:

 

tower

1.us.pool.ntp.org  (I am guessing these are helpful if "tower" is down)

2.us.pool.ntp.org  (I am guessing these are helpful if "tower" is down)

 

And My laptop points to:  tower

(I adjusted windows to check the server every 12 hours (instead of every 7 days))

 

I found this link to be helpful with making the adjustments:

http://www.cumps.be/nl/blog/read/howto-set-up-ntp-on-windows-vista

 

Begin EDIT:

 

I originally used the above link to point to 4 peers.  I could only see one peer when doing:

 

w32tm /query /peers

 

after doing some of the setup work, and struggling to see all the peers, I tried:

 

net stop w32time
net start w32time

 

and then all the peers were showing.

 

End EDIT

 

 

Does anyone know how often the Unraid server is set to fetch the time and date?  Is there instructions on adjusting this setting?

 

Albin

Link to comment

It should work. Make sure all of the machines are using the same NTP source. You'll have to test to determine the best rsync settings for your requirements.

 

I really never thought to look at that and honestly I had to look up what NTP Source was "Network Time Protocol"

 

I really have not messed with that much, other than trying to using this to set the time on my routers.

I went to the Unraid time settings to see what time zone I am using and I am using (UTC-05:00) eastern Time (US & Canada) and the current date and time look fine, but the "Use NTP" option is set to "NO"

 

I set the "Use NTP" to "Yes", and set "NTP Server 1" to: 0.us.pool.ntp.org

I then set the computers NTP Servers to: 0.us.pool.ntp.org

(right clicked on the clock and clicked "adjust date/time", then picked the "Internet time" TAB and clicked change settings)

 

I need to do this son my other unraid server, I think the time is off on that one.  Thanks for the tip

 

I then tried to force a sync, (right clicked the deltacopy profile name and clicked "run now"). Syncing this way give you a line by line view of what deltacopy is doing.

 

I was getting a no such directory error.

 

I deleted the profile, then recreated the profile and syncing started to semi work.

 

I say semi because if I create a file (file1) on computer1, and then run a sync on computer1, the file is copied to the network share, but if I go to computer2 and run a sync, it deletes the file netwrok shares copy of file1 I just created.  If I then create a file (file2) on computer2 and sync, the file2 shows on network share.  At this point if I go to computer1 and run a sync, file2 is deleted and file1 is recopied to the network share. 

 

Are the above symptoms due to improper steps to setting up deltacopy.  Do I need to set up the folder on each computer, leave them empty, then setup the deltacopy profiles, and then run a sync at both locations prior to putting any files in either place?

 

Thanks for any help on this.  Looks like I am one step close and I am excited that this is something that could work.

 

Thanks

 

Albin

 

You'll need to get rsync to always keep the file with the newest time stamp. This is why setting a common ntp source is important.

Link to comment

  I use batch files on my windows machines to make and sync files between machines and unRAID storage much like what you are trying to do.  And yes, the most important thing to make it all work is a consistent time between the various computers.  My batch files run on each computer on start-up/user log-on.  When I am ready to shut off a computer, I have another batch file I run to again sync files, then shut-down the computers.  Perhaps not the most elegant solution, but it works well for me.  :-)  I think you should be able to get what you want, with DELTACOPY, with the possible exception of two un-related files that may have the same name... could be a real problem there...

 

I did not know that Unraid had a NPT server.  Just synced my PC to my unraid server using its IP address.  I hard code the IP in unraid as I noticed in the past that it appears to stop/start the SMB share when renewing the IP address.

 

  WOW!  I had thought about how nice it would be to use an internal NPT source, I just never went further than thinking about it because I did not think about where I wanted to put it.

 

  I had not even thought that it might already be there in my unRAID machines!  COOL! :-)

 

 

Link to comment
  • 2 years later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.