RussellinSacto Posted August 25, 2015 Share Posted August 25, 2015 Hi Everyone, I'm looking for advice on the best way to synchronize two UnRaid systems (for onsite backup; I use Crashplan for offsite backup). I have tried BTSync, but it seems to crash often. I have tried FreeFileSync (on Windows) to sync the two UnRaids - but for whatever reason, some file changes aren't being detected (based on Time, I think - Samba? I don't know). I haven't found anything that is making my syncs perfect - despite lots of tinkering with them. (SyncBack, Pathsync, and a few others). Your suggestions? (Yes - TB of data - and some HUGE files) Prefer something that somehow notices changes and doesn't scan the entire array on both sites (drive wear). Thanks, Russell Link to comment
RussellinSacto Posted August 27, 2015 Author Share Posted August 27, 2015 I don't know why I couldn't find this before I posted... But there's lots of good answers here: https://lime-technology.com/forum/index.php?topic=42269.0 I've been using PureSync Personal Edition (Free) with more success than the solutions I've tried (BTSync, FreeFileSync, SyncBack, and PathSync). Thanks, Russell Link to comment
dgaschk Posted August 27, 2015 Share Posted August 27, 2015 http://lime-technology.com/forum/index.php?topic=13432.0 Link to comment
Dimtar Posted August 27, 2015 Share Posted August 27, 2015 I would suggest rsync, it's excellent but is only one way Link to comment
ntrlsur Posted August 27, 2015 Share Posted August 27, 2015 I would suggest rsync, it's excellent but is only one way I second this.. huge fan of rsync.. Link to comment
gundamguy Posted August 27, 2015 Share Posted August 27, 2015 If you plan is to do onsite backups then I third rsync. Rsycn is good for remote backups too, but the process is a lot harder to accomplish securely so I don't feel like it's good to recomend to new rsync users. Link to comment
archedraft Posted August 28, 2015 Share Posted August 28, 2015 If you plan is to do onsite backups then I third rsync. Rsycn is good for remote backups too, but the process is a lot harder to accomplish securely so I don't feel like it's good to recomend to new rsync users. I've been toying with the idea of syncing two unRAID servers with rsync and one being offsite. Any suggetions on the correct commands to use with rsync for offsite backups? P.S. I use openVPN to connect to the offsite server so the securely part isn't an issue. I am nore wondering what happens if rsync is in the middle of syncing a large file and then all of the sudden the connection drops for a bit ut then reconnects, will rsync fail instantly or will it resume after a bit? Link to comment
RussellinSacto Posted August 28, 2015 Author Share Posted August 28, 2015 Can someone please make a plugin or something for those of us who don't know RSync (or Linux) to sync our UnRaid machines? :-) Thanks, Russell Link to comment
WeeboTech Posted August 28, 2015 Share Posted August 28, 2015 Can someone please make a plugin or something for those of us who don't know RSync (or Linux) to sync our UnRaid machines? :-) Thanks, Russell I've been working diligently on a rsync server plugin. At least the destination rsync server part. Unfortunately I've come across a strange bug where daemons forked via emhttp in events, hang emhttp. Hang on, it's a bumpy ride that's kill -9'ing me! I'll figure something out soon. Link to comment
archedraft Posted August 28, 2015 Share Posted August 28, 2015 Can someone please make a plugin or something for those of us who don't know RSync (or Linux) to sync our UnRaid machines? :-) Thanks, Russell I've been working diligently on a rsync server plugin. At least the destination rsync server part. Unfortunately I've come across a strange bug where daemons forked via emhttp in events, hang emhttp. Hang on, it's a bumpy ride that's kill -9'ing me! I'll figure something out soon. Awesome! Looking forward to testing this out [emoji1] Link to comment
JonathanM Posted August 28, 2015 Share Posted August 28, 2015 Unfortunately I've come across a strange bug where daemons forked via emhttp in events, hang emhttp. Hang on, it's a bumpy ride that's kill -9'ing me! I'll figure something out soon. That bug is probably the reason for a pretty good proportion of the webgui hanging complaints. It would be nice if the bug were squashed instead of you having to find a workaround. Link to comment
WeeboTech Posted August 28, 2015 Share Posted August 28, 2015 Unfortunately I've come across a strange bug where daemons forked via emhttp in events, hang emhttp. Hang on, it's a bumpy ride that's kill -9'ing me! I'll figure something out soon. That bug is probably the reason for a pretty good proportion of the webgui hanging complaints. It would be nice if the bug were squashed instead of you having to find a workaround. I've tried everything with setsid, disown, even setsid multiple times. For some reason a zombie is left around and emhttp is waiting for the zombie. Do you know of a support thread where this bug is posted so I can read along and possibly add my 2c, logs and findings? Link to comment
bonienl Posted August 28, 2015 Share Posted August 28, 2015 It sounds like the service start issue from a PLG file, see this. Any relation ? Link to comment
WeeboTech Posted August 29, 2015 Share Posted August 29, 2015 It sounds like the service start issue from a PLG file, see this. Any relation ? Yeah that seems to be it. I don't really want to start it via at/batch, but I may have to. I tried disown, setsid, etc, etc. Issue for me is, I don't want people getting mailed when the daemon is started. That means I have to customize the rc script to capture it into syslog. I never throw stdout/stderr. You just never know if something went wrong. Even in my event scripts I redirect it all to syslog. Link to comment
RussellinSacto Posted September 1, 2015 Author Share Posted September 1, 2015 Does anyone know a Sync solution that would look at a directory, notice that only a single folder has had a modified date change, open that file, and repeat the process, syncing only files that have changed? Every solution I've found seems to spin those drives like crazy, doing serious comparison - and not just the first time I sync - but forever. I'm trying to avoid the drive wear of these solutions. I'm excited to see what you guys come up with for RSync - especially if it will behave this way. Thanks, Russell Link to comment
tr0910 Posted September 1, 2015 Share Posted September 1, 2015 Rsync will behave this way. Mine has been doing this with 40tb every night for 3 years. Link to comment
archedraft Posted September 1, 2015 Share Posted September 1, 2015 Damn 43,800 TB server! Impressive. Link to comment
tr0910 Posted September 1, 2015 Share Posted September 1, 2015 Lol Not quite. Some days we would generate up to 100 GB of new data, but many days we had very few changes to replicate to the backup server. On days with few changes, rsync could complete in just over one hour using rsync -av This year I was concerned that having no file verification done, that my backup server might not contain good data. I did a complete file verification via bunker, Bitrot & Weebotech's method and found the backups were perfect. (Both the main server and backup server were connected to the same gigabit LAN and my 2 offsite backups were small suitcase like servers connected for backup then carried offsite) Link to comment
archedraft Posted September 1, 2015 Share Posted September 1, 2015 Lol yeah it's funny how your brain reads things. I was like whoa that's so much data in one day and then I was like oh I'm being slow this morning... Link to comment
ntrlsur Posted September 3, 2015 Share Posted September 3, 2015 If you plan is to do onsite backups then I third rsync. Rsycn is good for remote backups too, but the process is a lot harder to accomplish securely so I don't feel like it's good to recomend to new rsync users. I've been toying with the idea of syncing two unRAID servers with rsync and one being offsite. Any suggetions on the correct commands to use with rsync for offsite backups? P.S. I use openVPN to connect to the offsite server so the securely part isn't an issue. I am nore wondering what happens if rsync is in the middle of syncing a large file and then all of the sudden the connection drops for a bit ut then reconnects, will rsync fail instantly or will it resume after a bit? rsync --progress -avz -e ssh user@ipaddress:/home . so after ssh you would use a user that has ssh to the box. IE if I were to use this command I would run it from the backup server so backupuser@primaryserver:/mnt/user/Movies /mnt/user/Movies So it would connect to the primary server and sync /mnt/user/Movies to /mnt/user/Movies on the backup server. Link to comment
Recommended Posts
Archived
This topic is now archived and is closed to further replies.