rsync for dummies


Recommended Posts

 

Let me flip the question back: why do you need rsync?

There are simpler apps out there to keep 2 folders in sync without needing endless loops (i.e. only run upon changes) e.g. Syncthing, Resilio Sync, etc.

 

Sounds to me like you picked a tool beyond your skill level to solve a problem it may not be the the most suitable tool for.

Link to comment
2 hours ago, nicksphone said:

Is it possible to get an rsync plugin or docker that makes it simple for us newbs so its source, destination lines and which copy options you want with a check box or a yes or no field. and an option for it to loop after its done?

 

 

If you want/need to use rsync and you need it to run "continually" you can use the User Scripts plugin to schedule your rsync task every minute with a * * * * * custom cron setting.

 

I back up all my unRAID shares from one server to another unattended via an rsync script.   I only do it once a week, but you can set the frequency to anything cron allows and it can be as granular as every minute.

Link to comment
18 hours ago, testdasi said:

 

Let me flip the question back: why do you need rsync?

There are simpler apps out there to keep 2 folders in sync without needing endless loops (i.e. only run upon changes) e.g. Syncthing, Resilio Sync, etc.

 

Sounds to me like you picked a tool beyond your skill level to solve a problem it may not be the the most suitable tool for.

syncthing and resilo sync i end up with duplicated files like File1 and file1 is there a setting im missing?

Link to comment
1 hour ago, testdasi said:

That's not duplicated file. Linux is a case-sensitive system, unlike Windows. So File1 and file1 are 2 different files.

which is fine in a perfect world but when a windows user changes the capital letters in a file name linux sees it as a diffrent file on sync and windows has a hissy fit when there are 2 files named the same as it sees 

Link to comment
5 hours ago, nicksphone said:

which is fine in a perfect world but when a windows user changes the capital letters in a file name linux sees it as a diffrent file on sync and windows has a hissy fit when there are 2 files named the same as it sees 

But rsync on Unraid, which is Linux based, will do the same.

Link to comment
23 hours ago, nicksphone said:

which is fine in a perfect world but when a windows user changes the capital letters in a file name linux sees it as a diffrent file on sync and windows has a hissy fit when there are 2 files named the same as it sees 

But the file on unRaid will be changed from say "test" to be "TEST".  There will not be 2 files on unRaid.  Your backup solutions will see that there is only a single file, but the history / old versions of the files will still exist.  

Link to comment

I use a switch in my rsync backup script called "--delete-before" and it will drop/delete any file or folder in the target location that was either changed or doesn't exist anymore prior to rsync making a new backup from source. So if a file was renamed file1.txt to File1.txt since the last backup, file1.txt on my backup target will be deleted and a new copy of File1.txt will take place.

Edited by cybrnook
Link to comment

Well I use the wonderful recycle bin plug-in for accidental deletes, and to be honest I have been using unraid for around 10 years and have never once accidently deleted anything.

 

And for corruption I have redundant backup servers that take backups at different times. Again, have yet to run into corruption due to anything like bit rot etc.

 

Key is I write my own scripts, so I understand what they are doing and create my own assumed and acceptable risk factors trading things like versioned backups for smaller backup footprints.

Edited by cybrnook
Link to comment
  • 3 years later...

@cybrnook

I know this thread is way old, Pre-Pandemic one might say, but I'm now dealing with something similar and your approach seems like something that would be beneficial to me. Can you give me an idea of a script you would use for my situation?

Basically, I'm a photographer; I have three 2TB external drives where I store all of the photos from each photoshoot and event that I photograph. The way I have organized their directory structure is by Year, then by project, then raw files along with folder(s) broken out by the concepts of edited, Full-Res, Low-Res, Watermarked Low-Res, etc. 
On each drive I have a 2022 folder, then inside that folder on one drive might be folders 2022-03-19 and 2022-07-21, then the other drive might have 2022-03-21 and 2022-05-11, and the third drive has several other 2022 project folders.

My goal is for my destination folder on the NAS to have all 2022 folders from each drive copied into a single 2022 destination folder. I have sporadically been moving one to two folders at a time from each drive into the single 2022 destination folder. What I feel is my problem now is, if I use RSYNC to move all sub folders under each external drive's 2022 folder but some of those folders are already on the NAS, how do I do it so it moves only new files and folders into the combined folder and skips over any files that are already on the nas and the latest version?

I hope this makes sense. Thanks for your willingness to read through this.
 

Link to comment
1 hour ago, DSPhoto said:

@cybrnook

I know this thread is way old, Pre-Pandemic one might say, but I'm now dealing with something similar and your approach seems like something that would be beneficial to me. Can you give me an idea of a script you would use for my situation?

Basically, I'm a photographer; I have three 2TB external drives where I store all of the photos from each photoshoot and event that I photograph. The way I have organized their directory structure is by Year, then by project, then raw files along with folder(s) broken out by the concepts of edited, Full-Res, Low-Res, Watermarked Low-Res, etc. 
On each drive I have a 2022 folder, then inside that folder on one drive might be folders 2022-03-19 and 2022-07-21, then the other drive might have 2022-03-21 and 2022-05-11, and the third drive has several other 2022 project folders.

My goal is for my destination folder on the NAS to have all 2022 folders from each drive copied into a single 2022 destination folder. I have sporadically been moving one to two folders at a time from each drive into the single 2022 destination folder. What I feel is my problem now is, if I use RSYNC to move all sub folders under each external drive's 2022 folder but some of those folders are already on the NAS, how do I do it so it moves only new files and folders into the combined folder and skips over any files that are already on the nas and the latest version?

I hope this makes sense. Thanks for your willingness to read through this.
 

I think "--ignore-existing" would be the switch you want in your rssync command.

 

https://linux.die.net/man/1/rsync

 

"--ignore-existing skip updating files that exist on receiver"

Link to comment
  • 7 months later...
On 7/18/2019 at 1:02 PM, Hoopster said:

If you want/need to use rsync and you need it to run "continually" you can use the User Scripts plugin to schedule your rsync task every minute with a * * * * * custom cron setting.

 

I back up all my unRAID shares from one server to another unattended via an rsync script.   I only do it once a week, but you can set the frequency to anything cron allows and it can be as granular as every minute.

Came across this thread and this is eventually exactly what I would want to implement. Going to continue research to understand how to deploy this scheduled task.

 

Thank you.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.