jortan Posted May 27, 2021 Share Posted May 27, 2021 (edited) This is a guide for users of ZFS Plugin who want to backup their ZFS pools using borgbackup. pre/post scripts can be used to briefly shutdown dockers/VMs while snapshots are taken. The backup can then proceed using the ZFS snapshot while the dockers/VMs are running. (personally I haven't bothered with this, a crash-consistent snapshot backup is good enough for me) Quote Data doesn't really exist unless you have at least two copies of it. - Schofield's Second Law of Computing Use case I use sanoid/syncoid but also wanted an encrypted copy of my non-encrypted zfs pool backed up incrementally off-site. I came across borgsnap that was further improved here. borgsnap wasn't suitable for large numbers of nested datasets so I've created a fork that adds recursive ZFS snapshots, some modifications for Unraid and other features here: https://github.com/jortan/borgsnap/blob/master/borgsnap borgsnap can backup to a local destination, a remote destination (via borg over SSH) or both. It will: - Read configuration file and encryption key file - Validate output directory exists and a few other basics - For each ZFS filesystem configured: Initialize borg repository if it doesn't exist Take a ZFS snapshot of the filesystem (recursively if enabled) Run borg for the local output if configured Run borg for the rsync.net output if configured Delete old ZFS snapshots (recursively if enabled) Prune local and remote borg backups if configured and needed Disclaimer I barely speak bash and have modified a script written by someone who does. Use at your own risk! Remove existing borgbackup from NerdPack The "borgbackup" package in NerdPack has broken a couple of times (including as of now). I recommend uninstalling borgbackup from NerdPack if you've installed this previously. The instructions below include downloading a linux binary of borgbackup from here - as far as I can tell works fine, though I'm not sure if it's relying on python / other packages I have installed via NerdPack. Please post any requirements you come across and I'll update this post. Create location for borg respository You can store the borgsnap repository on an array share, unassigned disk, remote mount, or another ZFS pool. You just need an empty folder somewhere. You don't need to create a borg respository, borgsnap will handle that for you. Create single disk ZFS pool for LOCAL borgsnap backup (optional) wipefs -a /dev/sdxx zpool create -o ashift=12 -m /mnt/borgpool borgpool /dev/sdxx zfs create borgpool/borgrepo Configure ZFS tunables to taste - borgbackup is going to compress backups, so no need to compress this pool/dataset. zfs set compression=zle atime=off recordsize=1m xattr=sa borgpool Download borgbackup/borgsnap Check for latest 1.x release here, copy URL for "borg-linux64" binary. Download the borgbackup binary: mkdir /boot/config/borgsnap wget https://github.com/borgbackup/borg/releases/download/1.1.16/borg-linux64 /boot/config/borgsnap/borg-linux64 Download borgsnap scripts to unRAID wget 'https://github.com/jortan/borgsnap/raw/master/borgsnap' /boot/config/borgsnap/ wget 'https://github.com/jortan/borgsnap/raw/master/borgwrapper' /boot/config/borgsnap/ Install binaries/scripts cp /boot/config/borgsnap/borg-linux64 /usr/local/sbin/borg cp /boot/config/borgsnap/borgsnap /usr/local/sbin/ cp /boot/config/borgsnap/borgwrapper /usr/local/sbin/ chmod +x /usr/local/sbin/borg chmod +x /usr/local/sbin/borgsnap chmod +x /usr/local/sbin/borgwrapper echo "alias borgwrap='/usr/local/sbin/borgwrapper /boot/config/borgsnap/borgsnap.conf'">>/etc/profile Install borgbackup/borgsnap on startup Add commands to /boot/config/go # borgbackup / borgsnap setup # cp /boot/config/borgsnap/borg-linux64 /usr/local/sbin/borg cp /boot/config/borgsnap/borgsnap /usr/local/sbin/ cp /boot/config/borgsnap/borgwrapper /usr/local/sbin/ chmod +x /usr/local/sbin/borg chmod +x /usr/local/sbin/borgsnap chmod +x /usr/local/sbin/borgwrapper echo "alias borgwrap='/usr/local/sbin/borgwrapper /boot/config/borgsnap/borgsnap.conf'">>/etc/profile Create a borg passphrase file IMPORTANT: borgbackup stores the decryption key with the backups (inside "config" file) but requires a passphrase to decrypt the key. You will not be able to restore your borg backups if you lose access to the passphrase Enter a long random passphrase here: /boot/config/borgsnap/passphrase.key Or generate a passphrase: cat /dev/urandom | tr -dc 'a-zA-Z0-9' | head -c 128 >/boot/config/borgsnap/passphrase.key Store a copy of this somewhere accessible in a DR scenario! No really, do it now! Create borgsnap config file. Note: I have only tested this with first-level zfs datasets (ie. poolname/dataset) /boot/config/borgsnap/borgsnap.conf FS="pool/appdata pool/data pool/vm" BASEDIR="/mnt/user/appdata/borgsnap" LOCAL="/mnt/borgpool/borgrepo" LOCAL_READABLE_BY_OTHERS=false LOCALSKIP=false RECURSIVE=true COMPRESS=zstd CACHEMODE="mtime,size" REMOTE="" REMOTE_BORG_COMMAND="" PASS="/boot/config/borgsnap/passphrase.key" MONTH_KEEP=1 WEEK_KEEP=4 DAY_KEEP=7 PRE_SCRIPT="" POST_SCRIPT="" IMPORTANT: All options must be present in the borgsnap.conf file, even if they have no value. BASEDIR is required for Unraid. This may take a few gigabytes, put this anywhere persistent. It's not required for a restore, but you don't want to lose borgbackup's cache every reboot. If you don't specify a "LOCAL" path, borgsnap will create a repository inside the dataset being backed up. You can skip doing LOCAL backups entirely and only create a REMOTE backup by setting LOCALSKIP=true If you want to create borg backups on rsync.net: REMOTE="[email protected]:myhost" In my case I installed borg on a remote Ubuntu machine, configured ssh key to access the remote server. From unraid: ssh-keygen ssh-copy-id -i /root/.ssh/id_rsa.pub [email protected] You can then specify the remote backup destination like this: REMOTE="[email protected]:/remote/path/borgrepo" By default, the remote borg command is configured for rsync.net - if using your own borg installation, you will probably need to set: REMOTE_BORG_COMMAND="borg" Run borgsnap! You can run borgsnap manually with this command: /usr/local/sbin/borgsnap run /boot/config/borgsnap/borgsnap.conf Note that if borgsnap is interrupted, it won't run again properly until the next day. You may want to add this to User Scripts and use "Run in background" so that closing your shell won't stop borgsnap. If something didn't work and you want to try again, you can use the "tidy" command - this will attempt to unmount borgsnap's ZFS snapshots and remove any local/remote backups for the current day. This isn't perfect but helps when initially trying to configure borgsnap: /usr/local/sbin/borgsnap tidy /boot/config/borgsnap/borgsnap.conf Pre/Post Scripts You can nominate a script to run before taking a ZFS snapshot and after a ZFS snapshot. Specify the full path to the script: PRE_SCRIPT="/mnt/user/appdata/borgsnap/prescript.sh" POST_SCRIPT="/mnt/user/appdata/borgsnap/postscript.sh" Note: You can't make files executable in /boot/config so these need to go somewhere else persistant. Keep in mind these scripts are going to be run as root so you may want to set appropriate permissions with something like: chmod 700 /mnt/user/appdata/borgsnap/prescript.sh chmod 700 /mnt/user/appdata/borgsnap/postscript.sh The same script is run for every FS (pool/dataset) configured in borgsnap.conf. The FS name is passed to the scripts as $1, so you can use this to run commands specific to a pool/dataset. Example: /mnt/user/appdata/borgsnap/prescript.sh #!/bin/bash if [[ "$1" = "pool/appdata" ]]; then echo pool/appdata - Stopping all dockers docker stop $(docker ps -aq) sleep 10 fi /mnt/user/appdata/borgsnap/postscript.sh #!/bin/bash if [[ "$1" = "pool/appdata" ]]; then echo pool/appdata - Starting all dockers docker start $(docker ps -aq) sleep 10 fi Note: This starts all containers, not just those configured to auto-start in the docker service. There's presumably a better command for this. User Scripts borgsnap-run Used to run borgsnap on a schedule. /usr/local/sbin/borgsnap run /boot/config/borgsnap/borgsnap.conf borgsnap-local-backup-summary Inside the LOCAL or REMOTE repo folder, borgsnap creates A folder for each zfs pool being backed up A sub-folder for each dataset being backed up Each of those sub-folders contains a borg repository. This script will parse LOCAL folder and give a summary of borg backups and repo sizes: #!/bin/bash source /boot/config/borgsnap/borgsnap.conf echo $'\n Summary of all borgsnap backups\n' echo " Source datasets: $FS" echo $'' echo " Backup repository: $LOCAL" echo $'' echo " Archives sizes: Original size, Compressed size, Deduplicated size" echo $'' for z in $LOCAL/*; do for f in $z/*; do if [ -d "$f" ]; then echo $'-------------------------------------------------------\n' /usr/local/sbin/borgwrapper /boot/config/borgsnap/borgsnap.conf list "$f" | cut -d" " -f1 | xargs -d "\n" -L1 echo $f:: | tr -d ' ' echo $'' /usr/local/sbin/borgwrapper /boot/config/borgsnap/borgsnap.conf info "$f" | awk 'NR==8' echo $'' fi done done Example output: Summary of all borgsnap backups Source datasets: pool/appdata pool/vm Backup repository: /mnt/borgpool Archives sizes: Original size, Compressed size, Deduplicated size ------------------------------------------------------- /mnt/borgpool/pool/appdata::month-20210328 /mnt/borgpool/pool/appdata::week-20210328 /mnt/borgpool/pool/appdata::day-20210329 All archives: 7.63 MB 134.86 kB 46.04 kB ------------------------------------------------------- /mnt/borgpool/pool/vm::month-20210328 /mnt/borgpool/pool/vm::week-20210328 /mnt/borgpool/pool/vm::day-20210329 All archives: 209.32 GB 105.99 GB 27.76 GB I'll add a REMOTE version of this script at some point. Run borgbackup commands interactively You can use "borgwrapper" to run arbitrary borg commands using the keyfile passphrase in borgsnap.conf: borgwrapper /boot/config/borgsnap/borgsnap.conf <borg commands> We added an alias in /boot/config/go to make this even easier to run (restart your shell if this doesn't work yet) borgwrap <borg commands> Example commands: List contents of a borg repository: borgwrap list /mnt/borgpool/pool/appdata Edited August 7, 2021 by jortan corrections Quote Link to comment
OneMeanRabbit Posted August 6, 2021 Share Posted August 6, 2021 Thank you for supporting ZFS on unRAID!! I'll check this out, I've had ok luck with sanoid/syncoid/auto-snap/etc - but I was just learning zfs during that time. Either way, always good to try something else. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.