Good day fellow Unraiders

 

Currently I have Tower1 and Tower2 shares Mapped Read/Write on MY Win 8.1 machine. 

Tower1 has some Mapped Read/Write shares on my LOML's 8.1 machine.

Tower2 is Read only from the other machines on our network.

Tower1 broadcasts SMB while Tower2 does not.

 

Tower2's Job is to maintain a full backup copy of Tower1 and to host the crashplan docker for our VITAL documents.

From MY Win 8.1 machine I manually launch a script that contains:

ROBOCOPY \\TOWER1\SHARE\SUBFOLDER \\TOWER2\SHARE\SUBFOLDER /MIR /R:0 /W:1

 

Now I have a few issues with this setup

1 - Occasionally I or my wife will accidentally move a Subfolder into another Subfolder on tower1 (crappy laptop trackpads).  This is not captured by the otherwise fantastic Recycle Bin Plugin.

2 - I am vastly concerned about cryptolockers locking up both Tower1 and Tower2 from MY Win 8.1 machine.

 

Question 1:

I would like to remove all Write access and Windows Mappings to tower2 and use a Docker/Rsync/Ubuntu VM/etc to backup Tower1 to Tower2 FROM Tower2.  I would like this backup to be done either on Demand or on Schedule depending on the share in question.

 

So far my 2 options appear to be the Beta WebSync Docker (with some scary warnings) or an Ubuntu VM running Rsync.  Which would you recommend?  Perhaps another method entirely?

 

Question 2:

What is the Rsync equivalent to

ROBOCOPY \\TOWER1\SHARE\SUBFOLDER \\TOWER2\SHARE\SUBFOLDER /MIR /R:0 /W:1

 

Question 3:

I really do NOT want subfolders/files that no longer exist on Tower1 to be deleted from Tower2 upon the backup job.  These deletions or moves are accidental in nature.  How to modify the Rysnc script to accomplish this?

 

Question 4:

For share\subfolder\file.ext not backed up to crashplan the files are written once and never updated.

How to script the backup to only copy when new, not when modified?  This should prevent cryptolockers from messing up these.

 

Thanks crew!

As this goes above normal system support, IF a Global Mod wants to answer all 4 questions to completion in 1 go I will offer a bounty via paypal.  Is $50 kosher?

It is VERY important to me that this is nailed the first time as a great deal of this data is only on the 2 machines.

I think the rsync approach will work, but you've got to understand a few things about Rsync. (Also why run it under a VM, when it runs natively on unraid...?)

 

1. There are two modes to Rsync: Rsync and Rsycn as a deamon. When you use Rsync you'll be prompted to sign into tower you are connecting to via SSH. If you don't have preshared keys set up, this means that you'll physically have to type the password in when your script rusn. Rsycn as a deamon avoids this, but also sends all it's data unencrypted across the LAN or internet...

 

2. Rsync is designed to update files as they are modified, but you can work around this by using symbolic links to create a "Rsycn Timemachine" there are a few tutorals out there...

 

I would love to create a plugin that makes this easier... but I am not that skilled in php... and I am le tired....

 

Also your basic Rsync call would be

rsync OPTIONS SOURCE DESTINATION

 

if your source or destination isn't on the same system you would use user@host[sOURCE or DESTINATION] to preceed them.

 

You could set this up either way... running on Tower1 to push to Tower2, or running on Tower2 to pull from Tower1.

 

:My own question:

 

Suppose I start worring about Cyptolockers today... and want to change my system (which has one unraid tower that is shared via SMB to a windows machine). If I were to create a "DropBox" share and only give Windows Write access to that share (Read access to the others) and then using command line move the files from that share to there final resting place in the shares that have read only access... whould that improve my protection?

 

it adds setps but it seems more secure... unless I am missing something.

Are these machines co-located? If so, rsync as a server would work well and be fast.

 

 

You can also keep dated backups and only rsync new data from the main tower, thus saving lots of space.

There is an option called    --link-dest=DIR        hardlink to files in DIR when unchanged.

 

If you keep dated directories, and use the last directory as the name to this parameter, then a newly dated directory hardlinks all files from the prior directory. This equates to a ghost of the prior backup, Now only modified files are copied across the network replacing the prior files.

 

This equates to 1x the FULL space of the first copy and an incremental update per date.

Each date can pretty much stand on it's own, so it's more of a differential backup.

 

i.e. if you were to remove the last 6 months of backup dates, then the most current directory would still contain 1 full backup.

 

Here is what it looks like from this month's backups in one of my host.

 

root@rgclws:/storage/backups/npgvm7 # du -hs 20151201

1.6G    20151201

 

root@rgclws:/storage/backups/npgvm7 # du -hs 20151205

1.6G    20151205

 

root@rgclws:/storage/backups/npgvm7 # du -hs 201512*

1.6G    20151201

17M    20151202

17M    20151203

17M    20151204

17M    20151205

17M    20151206

18M    20151207

17M    20151208

17M    20151209

25M    20151210

17M    20151211

17M    20151212

17M    20151213

18M    20151214

24M    20151215

17M    20151216

17M    20151217

17M    20151218

17M    20151219

17M    20151220

18M    20151221

17M    20151222

17M    20151223

 

root@rgclws:/storage/backups/npgvm7 # find 20151201 -type f | wc -l

31260

 

root@rgclws:/storage/backups/npgvm7 # find 20151223 -type f | wc -l

31318

 

root@rgclws:/storage/backups/npgvm7 # du -hs 20151223

1.6G    20151223

 

root@rgclws:/storage/backups/npgvm7 # ls -l 20151223/home/rcotrone/.bash_profile

-rw-r--r-- 44 10350 20506 546 Jan 30  2009 20151223/home/rcotrone/.bash_profile

 

So there are 44 links to the same file here.

Granted if the source file or one of the links is modified directly on the backup, they all change.

So this may not be something you want to prevent visibility and/or write access to.

 

Without the link option it would be 1.6GB per day for this backup.

 

What I personally do to age out backups is keep the first of the month for 6-9 months, keep the sunday backups for 6-8 weeks. and age off the other directories with a remove.

 

FWIW, this can also be done on some other type of rotation. That's where I got the idea from only I changed it to be date specific, instead if count specific.  In my use, if I needed hourly backups, I would change the date so it was YYYYMMDD-HH.

on some backups I use the Week so it's YYYY-WWW.

 

This works when the backup server is pulling the data via rsync over ssh or an rsync server.

I would prefer for this to be a pull. 

 

If no Web GUI access to manually kick off I would prefer to have this automated to run nightly.

 

1 - Deamon sounds like the path for me.  Both machines are on the same local network in my home behind a PFsense router/firewall.  An unencrypted copy across the network is what I am doing now (I believe).

 

2 - I am not overeager to have a different filename or foldername for any date that does change...of course if this is the only solution...

 

So in the script I would have (http://www.computerhope.com/unix/rsync.htm):

rsync -r -u --modify-window=1 \\192.168.1.155\share1\subfolder1  mnt\share1\subfolder1 

and this would copy from tower1\share1\subfolder1 all additional subfolders and files only those files that are older or non-existent from what is already on tower2, right?

 

So then next in the script I need a way to modify all the files with date/timestamp in the future, right?

Would this work and would it capture any potential subfolder/subfile?

touch -a -m t 205001010130.09 \\192.168.1.156\share1\subfolder1\*.*

or maybe

find \mnt\share1\subfolder1\* -exec touch -d '30 August 2050' *.* {} \;

 

Your question ***

This is essentially what I am looking to do... Tower1 is mapped for our laptops and it is the ONLY place we save/store files.  Tower2 is the backup to Tower1... so in your example Tower1 = Windows computer (for all intents).

 

Hoping to automate this and not require command line copies while removing the Windows robocopy batch scripts/tower2 write access/etc

 

 

 

@Weeboo... really hoping to NOT have DateName Directories.

 

For ShareDocuments I am fine overriding modified documents as I can revert from crashplan

For ShareStatic I do not want to override with modified documents as these are not backed up to crashplan

 

I do NOT want to delete a Share\Subfolder or Share\Subfolder\File that is no longer in Tower1 from Tower2

@Weeboo... really hoping to NOT have DateName Directories.

 

For ShareDocuments I am fine overriding modified documents as I can revert from crashplan

For ShareStatic I do not want to override with modified documents as these are not backed up to crashplan

 

I do NOT want to delete a Share\Subfolder or Share\Subfolder\File that is no longer in Tower1 from Tower2

 

You do not have to use the dated directory naming scheme. I only mentioned as it provides a way to capture a file that is accidentally deleted.

 

You would set up an rsyncd.conf file in /etc with the shares you want to export via rsync.

then pull them with

 

 

rsync -a rsync://tower1/sharename /mnt/sharenae

or

rsync -a rsync://tower1/disk1/ /mnt/disk1

 

There are plenty of examples on setting up a rsync server on the forum.

I would suggest searching the forum a bit and then jumping in with our assistance.

 

... and I am stuck again!

 

I am setting this up on Tower2 to do a Pull from a read only share on Tower1.  I am doing this because for now I believe working with a GUI is a little easier... and because I will want to move the modified flag of some shares' contents forward with another script.

 

For now I have the Tower2 shares passed through to Ubuntu VM as defined here:  http://lime-technology.com/forum/index.php?topic=45039.0

 

Tower1 shares are NOT mounted in the ubuntu VM

 

In the terminal:  rsync -r -u --modify-window=1 rsync://192.168.1.155/share1/  /home/UserName/tower2/share1

Results in:  rsync failed to connect to 192.168.1.155

 

do one of you fine folks happen to see what is wrong with my source link?  Thanks

 

 

Did you pay the $50 to the helping Mod?

 

J/K! 

@ Danioj... If you or another Mod wants to hand me the answer (set), then the bounty is open   [hopefully not bad form!]

 

I will be posting the final answer over here: http://lime-technology.com/forum/index.php?topic=45039.0

 

Gundamguy and WeebTech helped to point me in the rsync direction... now that I have spent about 10+ hours reading and doing local tests I am again stuck!

 

Leifgg helped point me in the direction for mounting unraid's shares within the VM here: https://lime-technology.com/forum/index.php?topic=45032.msg429957#msg429957

@ Danioj... If you or another Mod wants to hand me the answer (set), then the bounty is open   [hopefully not bad form!]

 

I will be posting the final answer over here: http://lime-technology.com/forum/index.php?topic=45039.0

 

Gundamguy and WeebTech helped to point me in the rsync direction... now that I have spent about 10+ hours reading and doing local tests I am again stuck!

 

Leifgg helped point me in the direction for mounting unraid's shares within the VM here: https://lime-technology.com/forum/index.php?topic=45032.msg429957#msg429957

 

I am not a Mod .... BUT I "sort of" have a setup running similar to what you want I think.

 

I have two Towers. Both located at the same site.

 

I have a Windows 10 VM running on Tower 2 (which is the backup server) and the primary purpose of this is to backup Tower 1 to Tower 2.

 

I use a program called SyncBack (free) running on this VM which is mapped to Tower 1 and Tower 2 shares (although because they are on a LAN I don't use security). It is an excellent tool which is used by a few people here including a senior Mod (who is very Pro Windows) and put me onto this setup.

 

Synback has profiles in place for daily, weekly, monthly and yearly backups. Each backup has different parameters set based on what I want (e.g. the first 3 are incremental with no delete and the yearly is a complete mirror including delete).

 

The VM chugs along in the background hardly every being checked (unless I want to glance at the reports casually to ensure there are no issues) and keeps all my files in sync with the two boxes. All or free (minus OS cost).

 

For me this is just as simple as creating VM on Backup Server. Installing Syncback. Putting Source in one Box (which is either a mapped drive or a direct network link e.g \\unraid\share) and Destination in another Box (which is either a mapped drive or a direct network link e.g \\unraid\share) and then choose your file transfer parameters (overwrite, increment etc) choose when you want it to run (e.g. daily, weekly etc). Add any others you want to work with it. Click OK. Test. If it works, leave it. Done.

 

Happy to give you a more detailed walkthrough if you choose this option.

Close to what I want... but I am strongly trying to remove Windows from any interaction with the backup/backup shares... I have actually had a robocopy script doing 75% of what I wanted up until recently.

Close to what I want... but I am strongly trying to remove Windows from any interaction with the backup/backup shares... I have actually had a robocopy script doing 75% of what I wanted up until recently.

 

Understood. Good look in your efforts. I personally wanted what you wanted too which is why I think there is an Rsync docker in beta somewhere at the moment (I think by sparklyballs) and also why I put in a feature request for a native backup feature in unRAID GUI. All because I can't get my head around rsync and I really don't want to have to drop to the command line to manage my backups. I want a GUI.

 

Anyway, thats how I ended up with the solution I have BUT can understand what you are trying to do. Ill keep this thread on notify as I am interested to see what your final solution is going to be and more importantly if it is as simple as I want mine to be!

I have figured out in Ubuntu VM how to

1 - create an executable script file

2 - how to pass through  and mount Tower2 shares in full read/write

3 - confirm that Tower1 shares are readable via IP/ShareName

4 - how to set up a cron 'aka task schedule'

5 - how to rsync from Local Mount A To Local Mount B.

 

Steps missing

1 - rsync from Tower1 network (read only & not mounted) to Local Mount

2 - setup touch script to alter created on / modified on property of any file in ShareX (and all ShareX subfolders) <> 20750101 to 20750101

3 - move rsync scripts, touch scripts, and cron schedules from Ubuntu VM to Unraid itself

Ill keep this thread on notify as I am interested to see what your final solution is going to be and more importantly if it is as simple as I want mine to be!

 

As I am cobling a few different things together trying to get this to work... set your notify on this one http://lime-technology.com/forum/index.php?topic=45039.0

Oh.. and for those coming to the bottom .... I am stuck and need help (see reply 7 up above).  Thanks!

Don't see any reason to get a VM involved. rsync is builtin to unRAID. Maybe I misunderstand what you are trying to do.

 

See this old thread where Weebotech shows us how to configure rsyncd. I have had this setup almost since I first started using unRAID and use an rsync client on Windows called DeltaCopy to backup Windows folders to unRAID shares. If you are just trying to backup from unRAID to unRAID then just setup rsyncd on the receiving unRAID and put an rsync command in cron on the sending unRAID.

trurl --- correct me if I am reading this wrong... but it looks to take configuration on both servers?  If at all possible I want the servers segregated so that the backup server only has access to read only shares and that no other machine has write access (or even read) to the backup server.

 

I found out the correct syntax for what I am attempting to accomplish here:  http://www.tecmint.com/rsync-local-remote-file-synchronization-commands/

 

rsync -r -u --modify-windows=1 [email protected]:/Share1/ /home/UserName/Share1

 

BUT... this insists on a password... and any password I give (even blank) results in a permission denied... any ideas?

 

I assume you mean the backup server should have read-only access to shares not that the shares should be read-only for everyone.

 

I'm not sure what logins would apply here (probably none) because the users you set up in unRAID webUI are really only for user share / disk share access over SMB/NFS/AFP.

 

Maybe look into rsync with ssh if you really need this security between the servers. I think you should be able to keep other users out of the backup server with the usual user and share setup provided by unRAID on that server.

This whole thing would be a very nice thing to see implemented in unraid itself... Or a plugin... One would run on both sides.. Anyone ?

Trurl -

Main/Tower1 has read/write for certain users and read only for guests.

Backup/Tower2 NOW has SMB/NFS/AFP turned off - the shares are secure/not visible on the network.  The shares are currently only passed through to the the Ubuntu VM (and this works great)

 

As Main/Tower1 shares are read only for all guests/anonymous, I SHOULD be able to use an rsync to read this FROM Backup/Tower2, be it directly in unraid or via a VM.

 

I dont need this secure between the 2 servers (both on the same LAN), but want to have the backup/Tower2 data untouchable via Windows, Mac, Android, etc. 

This whole thing would be a very nice thing to see implemented in unraid itself... Or a plugin... One would run on both sides.. Anyone ?

 

This would be fantastic

Scratch this for now... added read only user to all the shares on tower1, adding a cifs fstab mount to the ubuntu VM.  Tracking here: http://lime-technology.com/forum/index.php?topic=45039.msg429967#msg429967

 

Hopefully not too much longer for an invisible backup server

trurl --- correct me if I am reading this wrong... but it looks to take configuration on both servers?  If at all possible I want the servers segregated so that the backup server only has access to read only shares and that no other machine has write access (or even read) to the backup server.

 

I found out the correct syntax for what I am attempting to accomplish here:  http://www.tecmint.com/rsync-local-remote-file-synchronization-commands/

 

rsync -r -u --modify-windows=1 [email protected]:/Share1/ /home/UserName/Share1

 

BUT... this insists on a password... and any password I give (even blank) results in a permission denied... any ideas?

 

Well I think the problem with your rsync call here is that it's not Rsync as a daemon, it's Rsycn over SSH, so it's going to ask for a password.... well maybe.

 

The user, is root I think... and should have a black password... but it might still prompt you for one... since it's the Rsynce via SSH approach.

 

Also why can't you just mount the SMB share as Read-only?

Thanks gundam - tried root as well, no joy.

 

I want to keep all link info off of Tower1 in regards to the backup (paranoia with cryptolocker sophistication really)...  and was hoping if I could crack this nut I would be able to migrate from the Tower2 VM to unraid Crons... for now I have made peace with keeping the VM as the backup manager. 

 

I have added a read only user to all the shares on Tower1 and have mounted the SMB shares in the VM via CIFS. 

 

Initial Rsync of SMB/CIFS mount to Passthrough mount looks promising... but was cut short last night due to an over aggressive initial Dynamix File Integrity build of all disks on my part on Tower1 (all disks vs 1 disk at a time on an Atom cpu).