borgsnap - Automated borg backups using ZFS snapshots


Recommended Posts

This is a guide for users of ZFS Plugin who want to backup their ZFS pools using borgbackup.  pre/post scripts can be used to briefly shutdown dockers/VMs while snapshots are taken.  The backup can then proceed using the ZFS snapshot while the dockers/VMs are running. (personally I haven't bothered with this, a crash-consistent snapshot backup is good enough for me)

 

Quote

Data doesn't really exist unless you have at least two copies of it.

   - Schofield's Second Law of Computing

 

Use case

 

I use sanoid/syncoid but also wanted an encrypted copy of my non-encrypted zfs pool backed up incrementally off-site.

 

I came across borgsnap that was further improved here.  borgsnap wasn't suitable for large numbers of nested datasets so I've created a fork that adds recursive ZFS snapshots, some modifications for Unraid and other features here:
https://github.com/jortan/borgsnap/blob/master/borgsnap

 

borgsnap can backup to a local destination, a remote destination (via borg over SSH) or both.  It will:

 

- Read configuration file and encryption key file

- Validate output directory exists and a few other basics

- For each ZFS filesystem configured:

  • Initialize borg repository if it doesn't exist
  • Take a ZFS snapshot of the filesystem (recursively if enabled)
  • Run borg for the local output if configured
  • Run borg for the rsync.net output if configured
  • Delete old ZFS snapshots (recursively if enabled)
  • Prune local and remote borg backups if configured and needed

 

Disclaimer

 

I barely speak bash and have modified a script written by someone who does.  Use at your own risk!

 

Remove existing borgbackup from NerdPack

 

The "borgbackup" package in NerdPack has broken a couple of times (including as of now).  I recommend uninstalling borgbackup from NerdPack if you've installed this previously.  The instructions below include downloading a linux binary of borgbackup from here - as far as I can tell works fine, though I'm not sure if it's relying on python / other packages I have installed via NerdPack.  Please post any requirements you come across and I'll update this post.

 

Create location for borg respository

 

You can store the borgsnap repository on an array share, unassigned disk, remote mount, or another ZFS pool.  You just need an empty folder somewhere.  You don't need to create a borg respository, borgsnap will handle that for you.

 

Create single disk ZFS pool for LOCAL borgsnap backup (optional)

 

wipefs -a /dev/sdxx
zpool create -o ashift=12 -m /mnt/borgpool borgpool /dev/sdxx
zfs create borgpool/borgrepo


Configure ZFS tunables to taste - borgbackup is going to compress backups, so no need to compress this pool/dataset.
 

zfs set compression=zle atime=off recordsize=1m xattr=sa borgpool

 

Download borgbackup/borgsnap

 

Check for latest 1.x release here, copy URL for "borg-linux64" binary.
 

Download the borgbackup binary:
 

mkdir /boot/config/borgsnap
wget https://github.com/borgbackup/borg/releases/download/1.1.16/borg-linux64 /boot/config/borgsnap/borg-linux64


Download borgsnap scripts to unRAID
 

wget 'https://github.com/jortan/borgsnap/raw/master/borgsnap' /boot/config/borgsnap/
wget 'https://github.com/jortan/borgsnap/raw/master/borgwrapper' /boot/config/borgsnap/


Install binaries/scripts
 

cp /boot/config/borgsnap/borg-linux64 /usr/local/sbin/borg
cp /boot/config/borgsnap/borgsnap /usr/local/sbin/
cp /boot/config/borgsnap/borgwrapper /usr/local/sbin/
chmod +x /usr/local/sbin/borg
chmod +x /usr/local/sbin/borgsnap
chmod +x /usr/local/sbin/borgwrapper
echo "alias borgwrap='/usr/local/sbin/borgwrapper /boot/config/borgsnap/borgsnap.conf'">>/etc/profile


Install borgbackup/borgsnap on startup

 

Add commands to /boot/config/go
 

# borgbackup / borgsnap setup
#
cp /boot/config/borgsnap/borg-linux64 /usr/local/sbin/borg
cp /boot/config/borgsnap/borgsnap /usr/local/sbin/
cp /boot/config/borgsnap/borgwrapper /usr/local/sbin/
chmod +x /usr/local/sbin/borg
chmod +x /usr/local/sbin/borgsnap
chmod +x /usr/local/sbin/borgwrapper
echo "alias borgwrap='/usr/local/sbin/borgwrapper /boot/config/borgsnap/borgsnap.conf'">>/etc/profile

 

Create a borg passphrase file

 

IMPORTANT: borgbackup stores the decryption key with the backups (inside "config" file) but requires a passphrase to decrypt the key.

 

You will not be able to restore your borg backups if you lose access to the passphrase

 

Enter a long random passphrase here:

/boot/config/borgsnap/passphrase.key

 

Or generate a passphrase:

cat /dev/urandom | tr -dc 'a-zA-Z0-9' | head -c 128 >/boot/config/borgsnap/passphrase.key

 

Store a copy of this somewhere accessible in a DR scenario!  No really, do it now!

 

Create borgsnap config file.  Note: I have only tested this with first-level zfs datasets (ie. poolname/dataset)

 

/boot/config/borgsnap/borgsnap.conf

FS="pool/appdata pool/data pool/vm"
BASEDIR="/mnt/user/appdata/borgsnap"
LOCAL="/mnt/borgpool/borgrepo"
LOCAL_READABLE_BY_OTHERS=false
LOCALSKIP=false
RECURSIVE=true
COMPRESS=zstd
CACHEMODE="mtime,size"
REMOTE=""
REMOTE_BORG_COMMAND=""
PASS="/boot/config/borgsnap/passphrase.key"
MONTH_KEEP=1
WEEK_KEEP=4
DAY_KEEP=7
PRE_SCRIPT=""
POST_SCRIPT=""


IMPORTANT:  All options must be present in the borgsnap.conf file, even if they have no value.

 

BASEDIR is required for Unraid.  This may take a few gigabytes, put this anywhere persistent.  It's not required for a restore, but you don't want to lose borgbackup's cache every reboot.

 

If you don't specify a "LOCAL" path, borgsnap will create a repository inside the dataset being backed up.  You can skip doing LOCAL backups entirely and only create a REMOTE backup by setting LOCALSKIP=true

 

If you want to create borg backups on rsync.net:

 

REMOTE="[email protected]:myhost"

 

In my case I installed borg on a remote Ubuntu machine, configured ssh key to access the remote server.  From unraid:
 

ssh-keygen
ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected]

 

You can then specify the remote backup destination like this:

 

REMOTE="[email protected]:/remote/path/borgrepo"

 

By default, the remote borg command is configured for rsync.net - if using your own borg installation, you will probably need to set:

 

REMOTE_BORG_COMMAND="borg"

 

Run borgsnap!

 

You can run borgsnap manually with this command:

 

/usr/local/sbin/borgsnap run /boot/config/borgsnap/borgsnap.conf

 

Note that if borgsnap is interrupted, it won't run again properly until the next day.  You may want to add this to User Scripts and use "Run in background" so that closing your shell won't stop borgsnap.

 

If something didn't work and you want to try again, you can use the "tidy" command - this will attempt to unmount borgsnap's ZFS snapshots and remove any local/remote backups for the current day.  This isn't perfect but helps when initially trying to configure borgsnap:

 

/usr/local/sbin/borgsnap tidy /boot/config/borgsnap/borgsnap.conf

 

Pre/Post Scripts

 

You can nominate a script to run before taking a ZFS snapshot and after a ZFS snapshot.  Specify the full path to the script:

PRE_SCRIPT="/mnt/user/appdata/borgsnap/prescript.sh"
POST_SCRIPT="/mnt/user/appdata/borgsnap/postscript.sh"

 

Note:  You can't make files executable in /boot/config so these need to go somewhere else persistant.  Keep in mind these scripts are going to be run as root so you may want to set appropriate permissions with something like:

 

chmod 700 /mnt/user/appdata/borgsnap/prescript.sh
chmod 700 /mnt/user/appdata/borgsnap/postscript.sh

 

The same script is run for every FS (pool/dataset) configured in borgsnap.conf.  The FS name is passed to the scripts as $1, so you can use this to run commands specific to a pool/dataset.  Example:

 

/mnt/user/appdata/borgsnap/prescript.sh

#!/bin/bash

if [[ "$1" = "pool/appdata" ]]; then
  echo pool/appdata - Stopping all dockers
  docker stop $(docker ps -aq)
  sleep 10
fi

 

/mnt/user/appdata/borgsnap/postscript.sh

#!/bin/bash

if [[ "$1" = "pool/appdata" ]]; then
  echo pool/appdata - Starting all dockers
  docker start $(docker ps -aq)
  sleep 10
fi

Note: This starts all containers, not just those configured to auto-start in the docker service.  There's presumably a better command for this.

 

User Scripts

 

borgsnap-run

 

Used to run borgsnap on a schedule.

 

/usr/local/sbin/borgsnap run /boot/config/borgsnap/borgsnap.conf

 

borgsnap-local-backup-summary


Inside the LOCAL or REMOTE repo folder, borgsnap creates

  • A folder for each zfs pool being backed up
  • A sub-folder for each dataset being backed up

 

Each of those sub-folders contains a borg repository.

This script will parse LOCAL folder and give a summary of borg backups and repo sizes:

 

#!/bin/bash
source /boot/config/borgsnap/borgsnap.conf
echo $'\n Summary of all borgsnap backups\n'
echo " Source datasets:   $FS"
echo $''
echo " Backup repository: $LOCAL"
echo $''
echo " Archives sizes: Original size, Compressed size, Deduplicated size"
echo $''
for z in $LOCAL/*; do
    for f in $z/*; do
        if [ -d "$f" ]; then
            echo $'-------------------------------------------------------\n'
            /usr/local/sbin/borgwrapper /boot/config/borgsnap/borgsnap.conf list "$f" | cut -d" " -f1 | xargs -d "\n" -L1 echo $f:: | tr -d ' '
            echo $''
            /usr/local/sbin/borgwrapper /boot/config/borgsnap/borgsnap.conf info "$f" | awk 'NR==8'
            echo $''
        fi
    done
done

 

Example output:

Summary of all borgsnap backups

Source datasets: pool/appdata pool/vm

Backup repository: /mnt/borgpool

Archives sizes: Original size, Compressed size, Deduplicated size

-------------------------------------------------------

/mnt/borgpool/pool/appdata::month-20210328
/mnt/borgpool/pool/appdata::week-20210328
/mnt/borgpool/pool/appdata::day-20210329

All archives: 7.63 MB 134.86 kB 46.04 kB

-------------------------------------------------------

/mnt/borgpool/pool/vm::month-20210328
/mnt/borgpool/pool/vm::week-20210328
/mnt/borgpool/pool/vm::day-20210329

All archives: 209.32 GB 105.99 GB 27.76 GB

 

I'll add a REMOTE version of this script at some point.


Run borgbackup commands interactively

 

You can use "borgwrapper" to run arbitrary borg commands using the keyfile passphrase in borgsnap.conf:

 

borgwrapper /boot/config/borgsnap/borgsnap.conf <borg commands>

 

We added an alias in /boot/config/go to make this even easier to run (restart your shell if this doesn't work yet)

 

borgwrap <borg commands>

 

Example commands:

 

List contents of a borg repository:

 

borgwrap list /mnt/borgpool/pool/appdata

 

Edited by jortan
corrections
Link to comment
  • 2 months later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.