rsync Incremental Backup


Recommended Posts

Thank you for the Script and Update!

 

I would like to change the Skript Version from v0.6 to 1.3 and the Path.

 

What´s the best Way to change the Backup Paht and preserve my old Rsync Backups?

 

"/mnt/user/backup/shares/daten"    >>>       "/mnt/user/backup/daten"

 

 

Backups.JPG

Edited by schoppehermann
Link to comment
6 hours ago, schoppehermann said:

What´s the best Way to change the Backup Paht and preserve my old Rsync Backups?

 

You could move everything to a new path as follows (change diskX as needed):

mkdir /mnt/diskX/backup/daten
mv /mnt/diskX/backup/shares/daten/* /mnt/diskX/backup/daten/

 

 

The old backups will be preserved, but on the next backup hardlinking (space saving) does not work. If you even want that, you need to move the files of your most recent backup in the upper dir as follows:

mv /mnt/diskX/backup/daten/20220102_044001/daten/* /mnt/diskX/backup/daten/20220102_044001

 

 

 

 

Link to comment

Thank´s for your fast Reply, it works werry well!

 

I moved the last Backup one Level up:

Quote

mv /mnt/user/backup/daten/20220103_044001/daten/* /mnt/user/backup/daten/20220103_044001

 

 

And for the same Structure in my Monthly Backups:

Quote

mv /mnt/user/backup/daten/20210701_044001/daten/* /mnt/user/backup/daten/20210701_044001/
mv /mnt/user/backup/daten/20210801_044001/daten/* /mnt/user/backup/daten/20210801_044001/
mv /mnt/user/backup/daten/20210901_044001/daten/* /mnt/user/backup/daten/20210901_044001/
mv /mnt/user/backup/daten/20211001_044001/daten/* /mnt/user/backup/daten/20211001_044001/
mv /mnt/user/backup/daten/20211101_044001/daten/* /mnt/user/backup/daten/20211101_044001/

 

I hope second Step works to ;-) 

Edited by schoppehermann
Link to comment
8 hours ago, schoppehermann said:

And for the same Structure in my Monthly Backups:

Was not necessary (but does not hurt) as hardlinks won't break if they are in a subdir and my script uses only the most recent backup to compare and decide which files only need hardlinks. That's why I posted only one mv command.

Link to comment
2 hours ago, Shantarius said:

thank you for the superb script. I have a feature request: Is it possible to have an option for excluding files or paths?

 

Its already there:

  --exclude="[Tt][Ee][Mm][Pp]/" # exclude dirs with the name "temp" or "Temp" or "TEMP"
  --exclude="[Tt][Mm][Pp]/" # exclude dirs with the name "tmp" or "Tmp" or "TMP"
  --exclude="Cache/" # exclude dirs with the name "Cache"

 

 

Link to comment

Hi mgutt,

 

i have changed the rsync command in the script (V0.6) to

 

rsync -av --stats --exclude-from="/boot/config/scripts/exclude.list" --delete --link-dest="${backup_path}/${last_backup}" "${source_path}" "${backup_path}/.${new_backup}"

 

The file exclude.list contains this:

 

/mnt/zpool/VMs/Debian11_105/
/mnt/zpool/VMs/Ubuntu21_102/
/mnt/zpool/VMs/Win10_106/
/mnt/zpool/Docker/Dockerimage/

 

But the script don't ignore the directorys in the exclude.list

Can you say why?

 

Thanks!

Edited by Shantarius
Link to comment
37 minutes ago, Shantarius said:

Can you say why?

 

I'd say you need to use the relative path and not the absolute path. This means if you backup from "/mnt/zpool/" to /mnt/user/Backups, you need to add "VMs/Debian11_105/" to your exclude list file.

Link to comment

I got myself moderately tangled up changing my parameters when copying over your latest version (user stupidity on my half), so do you fancy making these changes so that it can be config based (on a yaml file)

 

if you call the script with no parameters then it uses whatever values are hardcoded in the script. Otherwise it'll parse the YAML file you specify and use those values. It also allows you to more easily have multiple backup sets running in parallel with a single backup script

 

# #####################################
# Settings
# #####################################


# rsync options which are used while creating the full and incremental backup
rsync_options=(
#  --dry-run
  --archive # same as --recursive --links --perms --times --group --owner --devices --specials
  --human-readable # output numbers in a human-readable format
  --itemize-changes # output a change-summary for all updates
  --exclude="[Tt][Ee][Mm][Pp]/" # exclude dirs with the name "temp" or "Temp" or "TEMP"
  --exclude="[Tt][Mm][Pp]/" # exclude dirs with the name "tmp" or "Tmp" or "TMP"
  --exclude="[Cc][Aa][Cc][Hh][Ee]/" # exclude dirs with the name "Cache" or "cache"
)
# set empty dir
empty_dir="/tmp/${0//\//_}"

if [ "${1}" == "" ]; then

  backupBase="root@10.1.2.2:/mnt/user/backup/beast"

  # backup source to destination
  backup_jobs=(
    # source                          # destination
      "/boot"             "$backupBase/boot"
      "/mnt/user/scan"    "$backupBase/scan"
  )

  # keep backups of the last X days
  keep_days=14

  # keep multiple backups of one day for X days
  keep_days_multiple=1

  # keep backups of the last X months
  keep_months=12

  # keep backups of the last X years
  keep_years=3

  # keep the most recent X failed backups
  keep_fails=3


  # notify if the backup was successful (1 = notify)
  notification_success=0

  # notify if last backup is older than X days
  notification_backup_older_days=30

  # create destination if it does not exist
  create_destination=1

  # backup does not fail if files vanished during transfer https://linux.die.net/man/1/rsync#:~:text=vanished
  skip_error_vanished_source_files=1

  # backup does not fail if source path returns "host is down".
  # This could happen if the source is a mounted SMB share, which is offline.
  skip_error_host_is_down=1

  # backup does not fail if file transfers return "host is down"
  # This could happen if the source is a mounted SMB share, which went offline during transfer
  skip_error_host_went_down=1

  # backup does not fail, if source path does not exist, which for example happens if the source is an unmounted SMB share
  skip_error_no_such_file_or_directory=1

  # a backup fails if it contains less than X files
  backup_must_contain_files=2

  # a backup fails if more than X % of the files couldn't be transfered because of "Permission denied" errors
  permission_error_treshold=20
else

  if ! [ -f yaml.sh ]; then
    wget https://raw.githubusercontent.com/jasperes/bash-yaml/master/script/yaml.sh 
  fi
  if ! [ -f "${1}" ]; then
	echo "File \"${1}\" not found"
	exit 1
  fi

  source yaml.sh
  create_variables "${1}"

  empty_dir+="."
  empty_dir+=$(basename "${1}")
fi

 

 

my YAML file

backup_jobs:
  -/boot
  -root@10.1.2.2:/mnt/user/backup/beast/boot
  -/mnt/user/scan
  -root@10.1.2.2:/mnt/user/backup/beast/scan
keep:
  days: 14
  days_multiple:  14
  months: 12
  years: 3
  fails: 3
notification:
  # notify if the backup was successful (1 = notify)
  success: 0 
  # notify if last backup is older than X days
  backup_older_days: 30
# create destination if it does not exist
create_destination: 1 
# a backup fails if it contains less than X files
backup_must_contain_files: 2
# a backup fails if more than X % of the files couldn't be transfered because of "Permission denied" errors
permission_error_treshold: 20
skip_error:
  # backup does not fail if files vanished during transfer https://linux.die.net/man/1/rsync#:~:text=vanished
  vanished_source_files: 1
  # backup does not fail, if source path does not exist, which for example happens if the source is an unmounted SMB share
  no_such_file_or_directory: 1
  host:
    # backup does not fail if source path returns "host is down".
    # This could happen if the source is a mounted SMB share, which is offline.
    is_down: 1
    # backup does not fail if file transfers return "host is down".
    # This could happen if the source is a mounted SMB share, which went offline during transfer
    went_down: 1

 

 

also

# make script race condition safe
if [[ -d "${empty_dir}" ]] || ! mkdir "${empty_dir}"; then echo "Script is already running!" && exit 1; fi; trap 'rmdir "${empty_dir}"' EXIT;

 

 

and obv remove the setting of "empty_dir" just above the loop

Edited by Meles Meles
Link to comment

Also...

 

i'm backing up from one unRAID server to another (via ssh), and for some reason I get errors pop up every time it makes a SSH connection.

 

Quote

hostfile_replace_entries: link /root/.ssh/known_hosts to /root/.ssh/known_hosts.old: Operation not permitted
update_known_hosts: hostfile_replace_entries failed for /root/.ssh/known_hosts: Operation not permitted

 

 

so that I can get rid of these in the (final) log files, i'm "sed"ding them out just before I move the log to the destination

  

  # hostfile_replace_entries: link /root/.ssh/known_hosts to /root/.ssh/known_hosts.old: Operation not permitted
  # update_known_hosts: hostfile_replace_entries failed for /root/.ssh/known_hosts: Operation not permitted
  sed -i '/hostfile_replace_entries/d' "$log_file"
  sed -i '/update_known_hosts/d' "$log_file"

  # move log file to destination
  rsync --remove-source-files "$log_file" "$dst_path/$new_backup/" || rsync --remove-source-files "$log_file" "$dst_path/.$new_backup/"

 

 

could you either 

a) pop this into your code

or 

b) give me some sort of clue as to why the blazes i'm getting this message!

 

 

ta :D

Edited by Meles Meles
Link to comment

Had an idea, which I need to note for myself:

Before rsync starts, create reflink copies of vdisks in the new backup dir.

 

Needs testing, but I hope it will update only some bytes, instead if copying the whole file. But I don't know if rsync compares the destination dir at all, if link-dest is used. Or isn't this needed at all, because rsync already transfers only the changed bytes?

Link to comment
On 1/4/2022 at 10:24 PM, mgutt said:

I'd say you need to use the relative path and not the absolute path. This means if you backup from "/mnt/zpool/" to /mnt/user/Backups, you need to add "VMs/Debian11_105/" to your exclude list file.

 

Hi mgutt,

 

thank you for your answer. This workes for me!

 

source_paths=(
        "/mnt/zpool/Docker"
        "/mnt/zpool/VMs"
        "/mnt/zpool/nextcloud_data"
        )
backup_path="/mnt/user/Backup_zpool"

...

rsync -av --exclude-from="/boot/config/scripts/exclude.list" --stats --delete --link-dest="${backup_path}/${last_backup}" "${source_path}" "${backup_path}/.${new_backup}"

 

excludes.list

Debian11_105/
Ubuntu21_102/
Win10_106/

 

  • Like 1
Link to comment
On 1/5/2022 at 4:33 AM, Meles Meles said:

hostfile_replace_entries: link /root/.ssh/known_hosts to /root/.ssh/known_hosts.old: Operation not permitted
update_known_hosts: hostfile_replace_entries failed for /root/.ssh/known_hosts: Operation not permitted

This happens because /root/.ssh is linked to the flash drive and the flash drive does not support hardlinks (as it uses FAT):

ls -lah | grep .ssh
lrwxrwxrwx  1 root root  21 Apr  7  2021 .ssh -> /boot/config/ssh/root/

 

On 1/5/2022 at 4:33 AM, Meles Meles said:

why the blazes i'm getting this message!

It is produced by OpenSSH and is patched in various projects:

https://github.com/termux/termux-packages/issues/2909

https://github.com/haikuports/haikuports/issues/6018

 

 

 

I don't know if it is patched in OpenSSH itself?! As Unraid 6.9.3 uses a relatively OpenSSH version, it does not seem so:

ssh -V
OpenSSH_8.4p1, OpenSSL 1.1.1j  16 Feb 2021

 

 

 

One solution would be to remove the link and copy the file from the usb flash drive:

rm /root/.ssh
cp /boot/config/ssh/root/ /root/.ssh

 

The downside of this solution is that any changes to /root/.ssh are not permanently saved in your unraid configuration and are lost on reboot. So you maybe should add an daily check as a script which copies the file back to your flash drive if the timestamp differs.

 

 

 

Link to comment
On 1/5/2022 at 3:52 AM, Meles Meles said:

if you call the script with no parameters

At first this script is created to use it as an Unraid users script which does not allow the usage of parameters, but yes, it would be nice having an external configuration file or being able to pass parameters to use it as an stand-alone script. I'm annoyed by myself changing the settings on each update and once I even forgot a setting as you did😅

 

But I don't think we should use bash and YAML at the same time as this would confuse the user. Instead we should use only bash like "source settings.conf" or change the settings area so it uses YAML, too. For example (untested):

 

cat > /tmp/${0//\//_}_settings.yaml << EOF
# #####################################
# Settings
# #####################################

backup_jobs:
  -/boot
  -root@10.1.2.2:/mnt/user/backup/beast/boot
  -/mnt/user/scan
  -root@10.1.2.2:/mnt/user/backup/beast/scan
keep:
  days: 14
  days_multiple:  14
  months: 12
  years: 3
  fails: 3

# #####################################
# Script
# #####################################
EOF

# parse parameters
for i in "$@"; do
  case $i in
    --include=*) settings_file="${i#*=}"; shift ;;
    *) echo "Unknown parameter passed: $1"; exit 1 ;;
  esac
done

# use inline settings
if [[ ! "$settings_file" ]]; then
  $settings_file="/tmp/${0//\//_}_settings.yaml"
fi

# parse settings
source yaml.sh
create_variables "$settings_file"

 

 

Example to use an external config file:

rsynclink.sh include=/boot/config/backupjob.yaml

 

 

Then we only need to decide if we use only bash or only YAML.

 

Link to comment

Greetings. I have been happily using .60 for a while now and noticed I am a few versions behind in added features. I ran the 1.3 script on a spare 8TB HD to see what changed. These are my backup paths at total about 4 TB of data. The 1.3 script takes about 2 days to run with the drive in a USB cradle. I was wondering if there was a way to have all backup paths included in one destination folder. When the script finished, I had 3 separate folders since the backup time spanned 3 days. In other words, can I change something in the script to place all 5 backup locations in one destination folder even if the backup takes multiple days? So, if I start the backup on 1-12-22 I will have one folder on the backup disk with all 5 backups in that folder (1-12-22)? (similar to the way script .6 named backup destination folders?)

 

 

backup paths 

    "/mnt/user/data"        "/mnt/disks/backup3/data"
    "/mnt/disk2"        "/mnt/disks/backup3/disk2"
    "/mnt/user/backups"        "/mnt/disks/backup3/backups"
    "/mnt/user/domains"        "/mnt/disks/backup3/domains"
    "/mnt/user/isos"        "/mnt/disks/backup3/isos"

 

Edited by bclinton
Link to comment
4 hours ago, bclinton said:

, if I start the backup on 1-12-22 I will have one folder on the backup disk with all 5 backups in that folder

Sorry, not possible. Even if your jobs would start at the same day: the timestamp is accurate to the second. It will be never the same.

 

Maybe it's possible by passing only /mnt as source and using the rsync options to include / exclude what you want in your backup. I'll test that.

 

PS thank you for your donation!

Link to comment
5 hours ago, mgutt said:

Sorry, not possible. Even if your jobs would start at the same day: the timestamp is accurate to the second. It will be never the same.

 

Maybe it's possible by passing only /mnt as source and using the rsync options to include / exclude what you want in your backup. I'll test that.

 

PS thank you for your donation!

 

Ah. that didn't occur to me.....I think I got it working using the exclude option like you suggested....did a few dry runs to test and it is now backing up the data. Will take about a 30 hours so I'll know tomorrow. I'm wondering what will happen when I do another backup next week to the data. Should I see another folder with the new date along with the most recent incremental copy of the data inside that folder?

 

This is a screen grab of my root on the back up drive along with my change to the script. It looks like the data is being placed in the .01-12-22 folder in their correct sub folders. (I removed the H:M:S: variable so it only used the date) The folder name starts with a "." - is that normal?826022489_Screenshot2022-01-12163259.thumb.png.30f51802d1256d9400860a322e09f4fd.png

backup_jobs=(
    # source                    # destination
    "/mnt/user"        "/mnt/disks/backup3"

 

Excludes - 

rsync_options=(
  --exclude="[Tt][Ee][Mm][Pp]/" # exclude dirs with the name "temp" or "Temp" or "TEMP"
  --exclude="[Tt][Mm][Pp]/" # exclude dirs with the name "tmp" or "Tmp" or "TMP"
  --exclude="Cache/" # exclude dirs with the name "Cache"
  --exclude="[Pp][Ll][Ee][Xx]/"
  --exclude="[Aa][Pp][Pp][Dd][Aa][Tt][Aa]/"
  --exclude="[Dd][Oo][Ww][Nn][Ll][Oo][Aa][Dd][Ss]/"
  --exclude="[Ss][Yy][Ss][Tt][Ee][Mm]/"

 

backup drive....

826022489_Screenshot2022-01-12163259.thumb.png.30f51802d1256d9400860a322e09f4fd.png

 

Thanks for the help!

 

 

Edited by bclinton
Link to comment

Yeah exclude works, but by that you need to add every new folder which is added under /mnt/user.

 

Instead try this if you use /mnt/user as source:

--include="data**"
--include="backups**"
--include="isos**"
--include="domains**"
--exclude="*"

 

By that everything is excluded except the includes

 

Or if you use /mnt as source:

--include="disk2**"
--include="user/"
--include="user/backups**"
--include="user/data**"
--include="user/isos**"
--include="user/domains**"
--exclude="*"

 

Note: Each parent dir needs a separate include as you can see for "user/".

 

Note: You don't need the other exclude entries for TMP etc. Those are only examples if you like to match folder names case insensitive.

  • Like 1
Link to comment
On 7/7/2021 at 6:17 PM, mgutt said:

a) enable unix extensions in the SMB settings of the Synology NAS

or

b) mount unraid on your Synology NASand execute the script on the Synology NAS

Hi, love this script. But i ran into the the same problem as someone before when creating an Backup on a Synology NAS. Hardlinks wont get created and so there is alway a full backup. I searched for an "unix extensions" option in the Synology SMB settings but coudnt find one. So i added "unix extensions=yes" into the smb.conf under /etc/samba via Telnet and restarted the NAS, but without success...
Does someone have any idea what i am missing? Thanks

Link to comment
5 hours ago, schwarzzrawhcs said:

Does someone have any idea what i am missing?

I'm not sure, but are unix extensions only available for SMB1? There seems to be a special option for SMB2, but it seems to be unavailable by now:

https://www.reddit.com/r/archlinux/comments/fx33rc/posix_smb311_extensions_unsupported_with_current/

 

 

I would test to mount the Synology share manually and set version 1.

Link to comment
4 hours ago, mgutt said:

I'm not sure, but are unix extensions only available for SMB1? There seems to be a special option for SMB2, but it seems to be unavailable by now:

https://www.reddit.com/r/archlinux/comments/fx33rc/posix_smb311_extensions_unsupported_with_current/

 

 

I would test to mount the Synology share manually and set version 1.

Ok thanks for the response. I did that by restricting the maximal SMB protocol to SMB1. But now i cant connect it as a remote share in Unraid. In windows i was able to install support for SMB1 and got it working, but not in Unraid even if NetBIOS is set to yes in SMB settings...

Sorry i am a really new to this stuff, thanks for the help!

Link to comment

Because I am going unRAID -> unRAID via SSH i'm getting the issue with the "hostfile_replace_entries" error (notwithstanding the workaround you suggested above).

 

As such the setting of "last_backup" is not working properly (as the errors are coming out in the rsync listing.

  # obtain last backup
  if last_backup=$(rsync --dry-run --recursive --itemize-changes --exclude="*/*/" --include="[0-9]*/" --exclude="*" "$dst_path/" "$empty_dir" 2>&1); then
    last_backup=$(echo "$last_backup" | grep -v Operation | grep -oP "[0-9_/]*" | sort -r | head -n1)

 

 

putting the "| grep -v Operation " in there cures this. Can you incorporate this into your next version please. Cheers!

Link to comment
On 1/7/2022 at 8:42 AM, mgutt said:

This happens because /root/.ssh is linked to the flash drive and the flash drive does not support hardlinks (as it uses FAT):

ls -lah | grep .ssh
lrwxrwxrwx  1 root root  21 Apr  7  2021 .ssh -> /boot/config/ssh/root/

 

It is produced by OpenSSH and is patched in various projects:

https://github.com/termux/termux-packages/issues/2909

https://github.com/haikuports/haikuports/issues/6018

 

 

 

I don't know if it is patched in OpenSSH itself?! As Unraid 6.9.3 uses a relatively OpenSSH version, it does not seem so:

ssh -V
OpenSSH_8.4p1, OpenSSL 1.1.1j  16 Feb 2021

 

 

 

One solution would be to remove the link and copy the file from the usb flash drive:

 

The downside of this solution is that any changes to /root/.ssh are not permanently saved in your unraid configuration and are lost on reboot. So you maybe should add an daily check as a script which copies the file back to your flash drive if the timestamp differs.

 

 

 

 

 

 

For posterity here's what i've done to fix it...

 

at the bottom of /boot/config/go i added

 

rm /root/.ssh
cp -R /boot/config/ssh/root/ /root/.ssh

 

 

 

and then i've done a User Script (scheduled hourly) to sync the data back to the flash drive

 

rsync -au /root/.ssh/ /boot/config/ssh/root/

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.