rsync Incremental Backup


119 posts in this topic Last Reply

Recommended Posts

I just lost 7 days of progress on a self hosted game server and this will be a life saver.  I do have 2 questions. 

 

1.  I created a share for the backup itself.  The source points to my server folder, but then in the backup folder is has the entire path instead of just the single folder (/mnt/user/appdata/valheim/.config/unity3d/IronGate/Valheim/worlds).  So every folder is backed up in the specified end folder.  Is this normal or did I do something wrong?  If it's normal then no big deal since there are no additional files.  Just a bunch of folders.  Was just making sure I had it right.

 

2.  If I change the dropdown on UserScript plugin to schedule daily, is that all I need to do or is there another section in the script to change?

Edited by tonsofguns
Link to post
  • Replies 118
  • Created
  • Last Reply

Top Posters In This Topic

Top Posters In This Topic

Popular Posts

The following script creates incremental backups of a share to a target path by using rsync. As an example the default setting is creating backups of the shares "Music" and "Photos" to the path "/mnt/

My plan is to realize a plugin out of it. So it's not a good idea to publish it at Github at this time. It will be done if it reaches the plugin state.

Good point. I will upgrade the script so it supports multiple and complete paths (not only a share name).

Posted Images

3 hours ago, tonsofguns said:

Is this normal or did I do something wrong?

Yes it's normal. If I would use only the last dir "valheim", a second backup of the path "/mnt/user/valheim" would write in the same folder. But usually my script replaces /mnt/user against /Shares. Does it not happen to you? Please post your set source paths 1:1. Maybe a whitespace in front of the slash?

 

Quote

is that all I need

Yes

Link to post
6 hours ago, mgutt said:

Yes it's normal. If I would use only the last dir "valheim", a second backup of the path "/mnt/user/valheim" would write in the same folder. But usually my script replaces /mnt/user against /Shares. Does it not happen to you? Please post your set source paths 1:1. Maybe a whitespace in front of the slash?

source_paths=(
    "/mnt/user/appdata/valheim/.config/unity3d/IronGate/Valheim/worlds"
)
backup_path="/mnt/user/valheimbackup"

 

Link to post
On 11/5/2020 at 9:43 AM, mgutt said:

Nothing wrong with that. The script should work with all paths (and not only "shares").

 

This would cause a huge problem if a user backups different paths with the same last subdir name. Example:


/mnt/user/Moritz/Notizen
/mnt/user/Max/Notizen

Both would target "/backup/Notizen". I know its "ugly" having super long paths, but how could we solve this? Maybe an optional setting like "force last subdir name"?

 

The permission problem is not related to rsync. It's only related to the "mv" command. The "mv" renames (or "moves") the backup folder from the hidden ".20201105" to "20201105" if the backup was successful:


mv "${backup_path}/.${new_backup}" "${backup_path}/${new_backup}"

It's a really basic linux command so I wonder why it does not work for you.

 

How did you mount "/mnt/hawi/192.168.178.101_disk17/Backup" and what could be the reason why read & write is allowed, but not renaming?

 

Please manually repeat the command through the WebTerminal:


mv '/mnt/hawi/192.168.178.101_disk17/Backup/disk17/Notizen/.20201105_130528' '/mnt/hawi/192.168.178.101_disk17/Backup/disk17/Notizen/20201105_130528'

If it works now, then something in this path locked the directory or a file inside of it. Maybe an index service on the external location or similar?
 

 

First, thanks for creating this.  It's exactly what I've been trying to accomplish.

 

With regards to the mv permissions issue, it seems the log file is locking the .hidden directory:

 

# lsof +D .20210216_191026/
COMMAND   PID USER   FD   TYPE DEVICE SIZE/OFF    NODE NAME
tee     10992 root    4w   REG  0,152    42261 1376475 .20210216_191026/backup.log

 

Any idea how to prevent this?

 

 

Link to post
  • 3 weeks later...

Hi,

 

how can i run the Backup to a external Share like a Diskstation?

I have a SMB Share for it, but i run in some problems 🙂

 

Script location: /tmp/user.scripts/tmpScripts/Unraid Backup/script
Note that closing this window will abort the execution of this script
Create backup of /mnt/user/Musik/Mischmasch/Led
Backup path has been set to /mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led
Create full backup 20210308_172541
sending incremental file list
Led/
Led/Controller-1/
Led/Controller-1/00_program1.led

Number of files: 3 (reg: 1, dir: 2)
Number of created files: 3 (reg: 1, dir: 2)
Number of deleted files: 0
Number of regular files transferred: 1
Total file size: 455,680 bytes
Total transferred file size: 455,680 bytes
Literal data: 455,680 bytes
Matched data: 0 bytes
File list size: 0
File list generation time: 0.001 seconds
File list transfer time: 0.000 seconds
Total bytes sent: 455,971
Total bytes received: 47

sent 455,971 bytes received 47 bytes 912,036.00 bytes/sec
total size is 455,680 speedup is 1.00
Try #1 to make backup visible
mv: cannot move '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/.20210308_172541' to '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/20210308_172541': Permission denied
Try #2 to make backup visible
mv: cannot move '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/.20210308_172541' to '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/20210308_172541': Permission denied
Try #3 to make backup visible
mv: cannot move '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/.20210308_172541' to '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/20210308_172541': Permission denied
Try #4 to make backup visible
mv: cannot move '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/.20210308_172541' to '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/20210308_172541': Permission denied
Try #5 to make backup visible
mv: cannot move '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/.20210308_172541' to '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/20210308_172541': Permission denied
Try #6 to make backup visible
mv: cannot move '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/.20210308_172541' to '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/20210308_172541': Permission denied
Try #7 to make backup visible
mv: cannot move '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/.20210308_172541' to '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/20210308_172541': Permission denied
Try #8 to make backup visible
mv: cannot move '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/.20210308_172541' to '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/20210308_172541': Permission denied
Try #9 to make backup visible
mv: cannot move '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/.20210308_172541' to '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/20210308_172541': Permission denied
Try #10 to make backup visible
mv: cannot move '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/.20210308_172541' to '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/20210308_172541': Permission denied
Try #11 to make backup visible
mv: cannot move '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/.20210308_172541' to '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/20210308_172541': Permission denied
Try #12 to make backup visible
mv: cannot move '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/.20210308_172541' to '/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/20210308_172541': Permission denied
Try #13 to make backup visible

 

Link to post
12 hours ago, AndreasNe said:

Try #13 to make backup visible

Was this the last try and finally successful or did it finally return a "Backup failed" error?
 

If it returned an error, are you able to execute this command successful through the WebTerminal?

mv "/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/.20210308_172541" "/mnt/remotes/DISKSTATION1511_UNRAID-BACKUPS/Shares/Musik/Mischmasch/Led/20210308_172541"

 

 

Link to post
9 minutes ago, mgutt said:
12 hours ago, AndreasNe said:

Try #13 to make backup visible

Was this the last try and finally successful or did it finally return a "Backup failed" error?

 

I have cut the LOG here. At the End it returned "Backup failed" error.

 

I will try the command today. Thank you.

Link to post

It is possible that the script had some problems with big initial backups?

I started yesterday to backup my 4 TB of Data to an external drive and it stopped after 1,5 TB. Now I started again and I am excited to see what will happen.

But very cool script, really!

I had a look how the hard links and inodes work, very clever!

Link to post

Then the script itself was killed. If only resync is killed, the final cleanup of the script still occurs. This is not a normal behaviour. Server restart or execution in a terminal which was closed can be excluded?

 

Link to post

I presume the problem for just a big job was that I ran the script manually in the web gui user script plugin.

At the moment it's still running. When done I will schedule an automatic job.

Link to post
2 hours ago, toasti said:

is there any chance to use this script locally on a Windows PC

Nope. This script is written in Bash/Shell = Linux. I add in my Windows Clients a new user names "Unraid" and by right-clicking on a folder I share this folder to this user read-only. Now I mount my Windows Client through Unassigned Disks and use my script to backup this Windows Client ;)

Link to post

@hawihoney @cgram23 @AndreasNe

 

I think I found the problem with the renaming. Because I forward the terminal output to a log file which is in the same subdirectory as the backup directory, it is not possible to rename the backup directory, because the log file is still open for writing.

 

I solved this in version 0.5.

Link to post

Version 0.4 released

# - timeout for backup renaming removed (as it did not solve the renaming problem)
# - changed log file location to solve renaming problem
# - skip permission errors if permission_error_treshold is under 20% (default)
# - optional: skip "host is down" error
# - optional: skip "wrong source path" error

 

Hopefully moving the logs out of the backup directory into a separate "logs" directory will solve the renaming problem.

 

In addition some soft errors can be skipped:

 

skip permission errors

The backup is valid if 20% or less of the files are not copied because of permission errors. Can be changed through the permission_error_treshold setting.

 

skip "Host is down (112)" error

The backup is invalid, but does not return an error, if rsync returns this message. This happens if the source direcory is an SMB server which is mounted, but offline.

 

skip "No such file or directory (2)" error

The backup is invalid, but does not return an error, if rsync returns this message. This happens if the source direcory is an SMB server which is offline (or if the source path is simply wrong).

 

 

Link to post

Version 0.5 released

# - preserve logs directory on clean up
# - ignore logs directory on last backup check

 

Sorry, v0.4 compared the new backup with the logs directory instead of the last backup, so the new backup was a full backup instead of incremental.

Link to post
10 hours ago, toasti said:

But how to automate / trigger this from the Windows Client to make backup now?

 

 


There are many ways to realize this. But at first the explanation, why I don't create backups through my client. My server has a share "backups" and no client is able to write into this folder. This is important because:

a) No access means no access for Ransomware

b) If a client changes the file backups/20190101/document.txt, which is old, it will overwrite this file in all backups, because rsync used hardlinks to save storage space and hardlinks are only different file names for the same file.

 

So it's more secure to execute the script on the server to fetch the files from the client (read-only) and not pushing them to the server.

 

Now the question how to start the script through the client:

 

a) Schedule

One idea would be to execute it every x hours, which creates multiple backups per day. As they are incremental, they won't consume much space. To start the script directly or for example 10 minutes after the client was powered on, we could add a "ping client check" before executing the script. This means the script starts every x hours and waits for the client as long as needed. As my script is atomic, it won't re-execute as long its waiting. This method does not need any interaction by the client. This is how I solved it.

 

b) Ping / send request to unraid server

A script could monitor incoming traffic on port X and if it was found it starts the backup script. The traffic could be created by a simple startbackup.bat file which will be placed on the client's desktop.

 

c) Create file or update file on client

The script permanently checks if a file on the client is updated / created and if, it starts the backup. 

 

Link to post

Aahhhh, changed 5 min ago and startet backup ;-)

 

On 3/14/2021 at 10:42 PM, mgutt said:

Now I mount my Windows Client through Unassigned Disks and use my script to backup this Windows Client

 

That works ;-)

 

Thanks for letting me know the possible ways to trigger a backup. The question is what would be the easiest and fastest to implement?

Link to post
18 hours ago, toasti said:

The question is what would be the easiest and fastest to implement?

Executing every 3 hours, is the easiest solution:

1576146876_2021-03-1617_30_13.thumb.png.0b62baa6a16a9e20921aae0df2ab1c60.png

 

Another solution is to search for this line in my script:

if [[ -d "/tmp/${0///}" ]] || ! mkdir "/tmp/${0///}"; then exit 1; fi; trap 'rmdir "/tmp/${0///}"' EXIT;

 

And add this after it:

if tcpdump -i eth0 -c1 port 8133; then
    echo "Traffic detected!"
fi

 

Now set a custom cron to execute the script every 5 minutes:

*/5 * * * *

 

Now the script is executed every 5 minutes, but only one session stays open and monitors traffic on port 8133 (you can change this port of course).

 

Now you only need to open the link "http://tower:8133" in your browser, which returns an error, because the page does not exist, but it will execute the backup script as it detected this connection attempt:

1162072652_2021-03-1617_52_29.png.e2438763a07f703d17b26eb8809f2e30.png

 

Opening a website, which does not return anything is strange? Ok, then create a startbackup.bat file on your windows desktop and use a text editor with the following content:

curl http://tower:8133

 

Screenshot:

683011678_2021-03-1617_55_49.png.4f85bb9a35740203d9b93fdf623bfafe.png

 

Double-click the file and it tries to open this link. After some seconds the window automatically closes. The script on the server detected the traffic and starts the backup.

 

And there is even more crazy stuff possible, like sending an answer ^^

 

848246484_2021-03-1618_29_27.png.d73b40d4ad826d5d5d5d0f5fca02cff0.png

 

By that it would be even possible to output the rsync logs to the client, but this would need some extra work. 

 

 

Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.