rsync Incremental Backup

[Naitor]

Hi, Thank you for this awesome script! I use it against ransomware - one question: can you use chattr +i after the incrimental Backup on the backup server and the script will add more hardlinks with the next backup? Or are there no more hardlinks possible after chattr+i ? ('i' A file with the 'i' attribute cannot be modified: it cannot be deleted or renamed, no link can be created to this file, most of the file's metadata can not be modified, and the file can not be opened in write mode. Only the superuser or a process possessing the CAP_LINUX_IMMUTABLE capability can set or clear this attribute.)

 

Same in German:

Hallo, danke für das tolle Script! Ich benutze das Script bei mir gegen Ransomeware und natürlich für ein Backup. Meine Frage jetzt; kann ich als zusätzlichen Schutz noch chattr+i über die bereits gesicherten Files drüberlaufen lassen? Oder werden dann keine Hardlinks mehr erstellt? 

Edited February 15 by Naitor

[Jaytie]

Hi!

Great script, I am planning on using this for main daily filebackup.

 

If I check the filesizes via the console like

du -d1 -h /mnt/disks/backup | sort -k2

13M     /mnt/disks/backup_hdd/#testbackup
11M     /mnt/disks/backup_hdd/#testbackup/20240228_171307
4.0K    /mnt/disks/backup_hdd/#testbackup/20240228_171556
4.0K    /mnt/disks/backup_hdd/#testbackup/20240228_171932
4.0K    /mnt/disks/backup_hdd/#testbackup/20240228_175035
1.8M    /mnt/disks/backup_hdd/#testbackup/20240228_175310

 

I can see looking at the foldersizes, that only changes/new files are copied, so hardlinks seem to work.

 

But when calculating the foldersize with the unraid file manager it adds all folders up (as if the backups are all full backups).

Same behaviour within the webui of my Synology NAS.

 

How can I prevent Unraid to show wrong values of used disk space? Or is it only shown wrong within Unraid's file manager?

 

Thanks for your help!

 

 

edit:

Since I'm quite new to the behaviour of hardlinks I was a bit hesitant and wanted to make sure, my backup storage won't be eaten up because of a wrong config on my side.

But I tested it and the used disc space on the Unraid 'Main' page doen't change, when doing an incremental backup without any new data. But the backupfolder size gets bigger, which is irritating at first

 

 

Thanks again for sharing this script

Edited March 18 by Jaytie

[Naitor]

Please help 🙂 Can somebody explain please how to change the Path where the logfile is saved after the Backup? I want one Folder outside the destination path with all the log files from all my backups, so i can go through and check if all the backups are made.  Would be very kind, im a newbie when it comes to bash!

 

# move log file to destination

log_path=$(rsync --dry-run --itemize-changes --include=".$new_backup/" --include="$new_backup/" --exclude="*" --recursive "$dst_path/" "$empty_dir" | cut -d " " -f 2)

[[ $log_path ]] && rsync "${dryrun[@]}" --remove-source-files "$log_file" "$dst_path/$log_path/$new_backup.log"

[[ -f "$log_file" ]] && rm "$log_file" 

 

[bclinton]

Hi friends. What setting would I change to have the script only keep the latest backup? In other words, after it backs up it would delete the previous backup folder and my backup drive would simply have 1 (the latest) backup folder. I changed the days retention to 1, will that do it? I backup once a week on Mondays.

 

Would this do the trick?

 

# keep backups of the last X days
keep_days=1

# keep multiple backups of one day for X days
keep_days_multiple=0

# keep backups of the last X months
keep_months=0

# keep backups of the last X years
keep_years=0

[Naitor]

Yes. Try it on a linux vm with any small folder. That is how i test my setzings 🙂

[cloudyhome]

Hey, 

 

Thank you for the wonderful script! Saved me a lot of time.

 

I tried to figure out how I can ensure that the backed up data is not acessible anymore once the backup is done. My understanding is that spinning it down and detaching it will do the trick. This can be achieved by using the unassigned device plugin as mentioned by OP.

 

Here is a short manual which works for me. Any feedback is appreciated.

 

Updating Unassigned Devices Default Script

1. First, you want to adapt the unassigned devices script by uncommenting the sections of the default script. Click on the settings button in the unassigned devices section (Main tab).

 

 

2. Copy Disk Serial and Disk Name to a notepad. You will need them later. My Disk Serial in this example ends at "74". Do not copy blank space or the brackets.

 

Note: @dlandon pointed out: Do not use "devX" device names. Assign a unique name to the 'Disk Name' field as an alias and use that in your script. Take the time to update it here and save the change before proceeding to the next steps.

 

3. Scroll down to find the script section. Click on "Default". It will load the default script to the Device Script content. At this stage, the script does nothing.

 

 

4. Uncomment the "remove" section activities by removing the hashtag. Effect: Once unmounted, the disk spins down and deattaches. Click on Save.

 

 

5. Scroll up and select the device script. Yours will have a different name, but there is probably only one to select.

 

--------------------------------------------------

Switch to the User Scripts:

 

6. Open the script which you created for the backup based on OPs script.

 

7. Add lines at the beginning and the end of the script to attach, mount and unmount the usb device. You will need the saved information from the first step: Disk Serial and Device Name. Note: I am using different devices than at the beginning here, do not let this confuse you. Just copy & paste the information you saved before.

 

 

### mount external USB drive

/usr/local/sbin/rc.unassigned attach 'device serial number'
/usr/local/sbin/rc.unassigned mount name='device name'

 

# unmount the usb device
/usr/local/sbin/rc.unassigned umount name='device name'

 

You can test the command without processing the full script by copy and pasting them in the command line of your unraid system. Note that attaching and mounting will take some time (~10 seconds for me). Just wait and see if it pops an error or hopefully just works.

 

Source: 

 

 

Edited March 21 by cloudyhome
added note regarding proper choice of disk name.

[dlandon]

@cloudyhome Nice writeup!  One suggestion.  Don't use the 'devX' designation from the 'Disk Name' field.  UD defaults this to the 'devX' designation Unraid assigns to unassigned disks and it can change.  Assign a unique name to the 'Disk Name' field as an alias and use that in your script.

[cloudyhome]

Thanks a lot for your feedback. Appreciate it! I was not aware of that I'll update the guide above accordingly with a hint to the chosen device name.

[myxxmikeyxx]

On 10/18/2020 at 9:14 AM, mgutt said:

The following script creates incremental backups by using rsync. Check the settings to define your own paths.

 

Explanations

• All created backups are full backups with hardlinks to already existing files (~ incremental backup)
• All backups use the most recent backup to create hardlinks or new files. Deleted files are not copied (1:1 backup)
• There are no dependencies between the most recent backup and the previous backups. You can delete as many backups as you like. All backups that are left, are still full backups. This could be confusing as most incremental backup softwares need the previous backups for restoring the data. But this is not valid for rsync and hardlinks. Read here if you need more informations about links, inodes and files.
• After a backup has been created the script purges the backup dir and keeps only the backups of the last 14 days, 12 month and 3 years, which can be defined through the settings
• logs can be found inside of each backup folder
• Sends notifications after job execution
• Unraid exclusive: Stops docker containers if the source path is the appdata path, to create consistent backups
• Unraid exclusive: Creates a snapshot of the docker container source path, before creating a backup of it. This allows an extremely short downtime of the containers (usually only seconds).

How to execute this script?

• Use the User Scripts Plugin (Unraid Apps) to execute it by schedule
• Use the Unassigned Devices Plugin (Unraid Apps) to execute it after mounting a USB drive
• Call the script manually (Example: /usr/local/bin/incbackup /mnt/cache/appdata /mnt/disk6/Backups/Shares/appdata)

Hello @mgutt! I was looking at your script and really like it. I think this would be great for backups from my Unraid to a NAS or some other external storage source. I have a question and I might be missing some information. Can this script be modified to not make incremental backups but backup one Unraid system to a second off site Unraid that I have running? My home unraid is my primary (now) but I still have files on my old unraid off site. I would like to backup all appdata and containers and plugins from my off site to home first and have it merge and keep newer files (assuming my home Unraid has more recent files). The problem I ran into is that some files are too large and are timing and erroring out (my nextcloud data). I also have manually tried backing up my containers and then restoring them on my home one. I have the appdata folder and data but the container is not showing up in my home unraid to start. I tried installing the container and then running it but it is still isn't working right and throwing errors (thinking something with my database isn't syncing correctly leading to problems).

 

Sorry for the long post and any confusion, if I need to I can try to clearify but the gist is:

Off site unraid full backup to Home unraid (including docker containers and configs, etc.)

Need the home unraid to be able to run all containers from sync (ex MariaDB for NExtcloud user data both synced from off site)

Need to keep newer and existing data on Home unraid.

After above is working, have my Home Unraid as my primary that then syncs back to offsite. 

Keep off site as a failover if Home unraid dies (I have an idea on how to do this with a load balancer in the cloud ran through cloudflare) -- This means the failover needs to have all my docker containers and auto start settings and appdata backup.

 

 

Edit:

On your script for the custom command would I be able to change the rsync custom one to something like this?

rsync -Pav -e "ssh -i $HOME/.ssh/key.pubkey" /from/dir/ username@hostname:/to/dir/

I don't understand your script well enough to know if this will break something since your script custom command is:

alias rsync='sshpass -p "<password>" rsync -e "ssh -o StrictHostKeyChecking=no"'

 

Edited April 3 by myxxmikeyxx

[UnKwicks]

On 7/30/2023 at 1:00 PM, mgutt said:

The target filesystem has to be BTRFS or XFS or ZFS. The source filesystem does not matter except it contains the appdata directory. Then it should be BTRFS or XFS (At the moment ZFS does not support reflink files, which is a feature used by my script.

Is ZFS still not supported?

My pool where my appdata folder is, is a zfs pool.

The script runs fine (I guess) but after completing the backup the container do not start up automatically.

 

EDIT:

I guess I found a bug in the script. When appdata is on a ZFS pool then there is the following error:

rm: cannot remove '/mnt/cache/.appdata_snapshot': No such file or directory

 

As far as I understand this is because $snapshot_path is always set:

       else
          # set snapshot path
          snapshot_dirname=$(dirname "$src_path")
          snapshot_basename=$(basename "$src_path")
          snapshot_path="$snapshot_dirname/.${snapshot_basename}_snapshot"

and because of this it always tries to delete the snapshot instead of starting the container

    # remove snapshot (containers are already running)
    if [[ $snapshot_path ]]; then
      rm -r "$snapshot_path"
    # start containers
    else
      echo "Start containers (slow method):"
      docker container start "${container_ids[@]}"
    fi

 

Am I right?

 

EDIT 2:

It seems I am right. I added a unset snapshot_path here, and it works now:

else
   notify "Backup causes long docker downtime!" "The source path $src_path is located on a filesystem which does not support reflink copies!"
   unset snapshot_path

This should be fixed in the next version.

OK, the script is running now, but only in "slow mode" because of ZFS. May I better switch to btrfs? I was hoping I can use automatic snapshots for my pool and array.

Edited April 7 by UnKwicks
possible reason/bug/fix added

[UnKwicks]

On 4/7/2024 at 10:09 PM, UnKwicks said:

This should be fixed in the next version.

I realized that my fix above is not a clean solution, because even with ZFS the script creates a snapshot that is not getting deleted if I unset the variable.

So my fix for now is to just remove the else where the container needs to get startet:

# final container steps (unraid exclusive)
  if [[ ${#container_ids[@]} -gt 0 ]]; then
	
    # remove snapshot (containers are already running)
    if [[ $snapshot_path ]]; then
      rm -r "$snapshot_path"
	  unset snapshot_path
      # start containers
	  echo "Start containers (slow method):"
	  docker container start "${container_ids[@]}"
    fi
    
  fi

 

So if containers were stopped and if a snapshot_path is set (so we are in the loop for appdata right now) the snapshot is deleted and the containers get started. For the loop that backups other backup locations container do not get stopped, no snapshot is being created so no need to start the container.

 

Not sure if this is how @mgutt  meant the script to work, but maybe you can bring light into the dark?

As well to answere the question if it is even possible/recommended to run the script when appdata is on a ZFS share.

 

Thanks!

Edited April 9 by UnKwicks

[millercb]

I am just finding this solution as I am trying to do remote backups. I have a Raspberry Pi with a USB hard drive connected over Wireguard to my Unraid server. I want to back up folders on my unraid drive to the remote hard drive, which is shared with SMB. Right now it is NTFS formatted, but I can reformat it to another format if necessary. On the first page it was said that the destination and link-dest have to be on the same volume, but I don't quite understand. I found a link to this page from serverfault.com and in that answer it said you can do remote backups. Will this meet my needs? Any particular requirements I need to be aware of? Thanks in advance.