Hi Everyone,

 

I'm looking for advice on the best way to synchronize two UnRaid systems (for onsite backup; I use Crashplan for offsite backup).

 

I have tried BTSync, but it seems to crash often.

 

I have tried FreeFileSync (on Windows) to sync the two UnRaids - but for whatever reason, some file changes aren't being detected (based on Time, I think - Samba?  I don't know).

 

I haven't found anything that is making my syncs perfect - despite lots of tinkering with them.  (SyncBack, Pathsync, and a few others).

 

Your suggestions?  (Yes - TB of data - and some HUGE files)  Prefer something that somehow notices changes and doesn't scan the entire array on both sites (drive wear).

 

Thanks,

 

Russell

I don't know why I couldn't find this before I posted...  But there's lots of good answers here:  https://lime-technology.com/forum/index.php?topic=42269.0

 

I've been using PureSync Personal Edition (Free) with more success than the solutions I've tried (BTSync, FreeFileSync, SyncBack, and PathSync).

 

Thanks,

 

Russell

http://lime-technology.com/forum/index.php?topic=13432.0

I would suggest rsync, it's excellent but is only one way

I would suggest rsync, it's excellent but is only one way

 

I second this.. huge fan of rsync..

If you plan is to do onsite backups then I third rsync.

 

Rsycn is good for remote backups too, but the process is a lot harder to accomplish securely so I don't feel like it's good to recomend to new rsync users.

If you plan is to do onsite backups then I third rsync.

 

Rsycn is good for remote backups too, but the process is a lot harder to accomplish securely so I don't feel like it's good to recomend to new rsync users.

 

I've been toying with the idea of syncing two unRAID servers with rsync and one being offsite. Any suggetions on the correct commands to use with rsync for offsite backups?

 

P.S. I use openVPN to connect to the offsite server so the securely part isn't an issue. I am nore wondering what happens if rsync is in the middle of syncing a large file and then all of the sudden the connection drops for a bit ut then reconnects, will rsync fail instantly or will it resume after a bit?

Can someone please make a plugin or something for those of us who don't know RSync (or Linux) to sync our UnRaid machines?  :-)

 

Thanks,

 

Russell

Can someone please make a plugin or something for those of us who don't know RSync (or Linux) to sync our UnRaid machines?  :-)

 

Thanks,

 

Russell

 

I've been working diligently on a rsync server plugin. At least the destination rsync server part.

Unfortunately I've come across a strange bug where daemons forked via emhttp in events, hang emhttp. 

Hang on, it's a bumpy ride that's kill -9'ing me!  I'll figure something out soon.

 

Can someone please make a plugin or something for those of us who don't know RSync (or Linux) to sync our UnRaid machines?  :-)

 

Thanks,

 

Russell

 

I've been working diligently on a rsync server plugin. At least the destination rsync server part.

Unfortunately I've come across a strange bug where daemons forked via emhttp in events, hang emhttp. 

Hang on, it's a bumpy ride that's kill -9'ing me!  I'll figure something out soon.

 

Awesome! Looking forward to testing this out [emoji1]

Unfortunately I've come across a strange bug where daemons forked via emhttp in events, hang emhttp. 

Hang on, it's a bumpy ride that's kill -9'ing me!  I'll figure something out soon.

That bug is probably the reason for a pretty good proportion of the webgui hanging complaints. It would be nice if the bug were squashed instead of you having to find a workaround.

Unfortunately I've come across a strange bug where daemons forked via emhttp in events, hang emhttp. 

Hang on, it's a bumpy ride that's kill -9'ing me!  I'll figure something out soon.

That bug is probably the reason for a pretty good proportion of the webgui hanging complaints. It would be nice if the bug were squashed instead of you having to find a workaround.

 

 

I've tried everything with setsid,  disown, even setsid multiple times.

For some reason a zombie is left around and emhttp is waiting for the zombie.

Do you know of a support thread where this bug is posted so I can read along and possibly add my 2c, logs and findings?

It sounds like the service start issue from a PLG file, see this.

 

Any relation ?

 

It sounds like the service start issue from a PLG file, see this.

 

Any relation ?

 

Yeah that seems to be it.  I don't really want to start it via at/batch, but I may have to.

I tried disown, setsid, etc, etc.

 

Issue for me is, I don't want people getting mailed when the daemon is started.

That means I have to customize the rc script to capture it into syslog.

 

I never throw stdout/stderr.  You just never know if something went wrong.

Even in my event scripts I redirect it all to syslog.

Does anyone know a Sync solution that would look at a directory, notice that only a single folder has had a modified date change, open that file, and repeat the process, syncing only files that have changed?

 

Every solution I've found seems to spin those drives like crazy, doing serious comparison - and not just the first time I sync - but forever.

 

I'm trying to avoid the drive wear of these solutions.

 

I'm excited to see what you guys come up with for RSync - especially if it will behave this way.

 

Thanks,

 

Russell

Rsync will behave this way. Mine has been doing this with 40tb every night for 3 years.

Damn 43,800 TB server! Impressive.

Lol

 

Not quite. Some days we would generate up to 100 GB of new data, but many days we had very few changes to replicate to the backup server. On days with few changes, rsync could complete in just over one hour using rsync -av

 

This year I was concerned that having no file verification done, that my backup server might not contain good data. I did a complete file verification via bunker, Bitrot & Weebotech's method and found the backups were perfect.

 

(Both the main server and backup server were connected to the same gigabit LAN and my 2 offsite backups were small suitcase like servers connected for backup then carried offsite)

 

Lol yeah it's funny how your brain reads things. I was like whoa that's so much data in one day and then I was like oh I'm being slow this morning...

If you plan is to do onsite backups then I third rsync.

 

Rsycn is good for remote backups too, but the process is a lot harder to accomplish securely so I don't feel like it's good to recomend to new rsync users.

 

I've been toying with the idea of syncing two unRAID servers with rsync and one being offsite. Any suggetions on the correct commands to use with rsync for offsite backups?

 

P.S. I use openVPN to connect to the offsite server so the securely part isn't an issue. I am nore wondering what happens if rsync is in the middle of syncing a large file and then all of the sudden the connection drops for a bit ut then reconnects, will rsync fail instantly or will it resume after a bit?

 

 

rsync --progress -avz -e ssh user@ipaddress:/home .

 

so after ssh you would use a user that has ssh to the box.  IE if I were to use this command I would run it from the backup server so  backupuser@primaryserver:/mnt/user/Movies  /mnt/user/Movies

So it would connect to the primary server and sync /mnt/user/Movies to /mnt/user/Movies on the backup server.