Jump to content
catapultam_habeo

btrfs incremental snapshots

9 posts in this topic Last Reply

Recommended Posts

Hi, this is mostly a WIP thread, but as of the first post it does work up to my relatively limited testing. I plan on expanding this to a fully featured plugin, but this script is a working foundation, and I'd like to make this available to people to play with asap.

Bottom Line Up Front: This script only works on your btrfs-formatted array drives. By default, it will keep 8760 snapshots (1 year of hourly snapshots), this value can be adjusted by changing the MAX_SNAPS variable in the script. This script does not handle cache drives, but would be trivial to extend to do so - I just think it is a bad idea. Detection of this is minimal but present. Running this script for the first time will remove and recreate all of your existing array shares (moved to temporary path, original path converted to subvolume, moved back), no data should be lost, but I have only tested this with my own data and configuration, and cannot account for all edge cases and I absolutely cannot be held accountable for your data if it is lostNo script is provided to revert these changes.

 

Goals: I wanted to have delta snapshot recovery as part of my NAS feature set. FreeNAS is appealing, but I dislike FreeBSD's ecosystem (weird problems with bhyve, don't really need ZFS performance improvements), tried ProxMox and didn't care for it, didn't want to roll my own (likely debian-based) setup, very much like unRAID's GUI and asynchronous drive upgrade process, and wasn't interested in the crazy ZFS-on-unRAID frankenstein config by Wendell at L1T. I noted that unRAID can be configured to use BTRFS, and in theory it should be able to do this, given enough scripting to keep everything in sync. I also want to leverage unRAID's GUI as heavily as possible, and do as little command-line work on the regular as possible. Adding a new drive, creating a new share, etc, should all be possible through the GUI, and this setup should automagically adjust.


Step 1) Adjust your Settings -> SMB -> SMB Extras field to include the following line. This will publish your snapshots to windows SMB clients as 'previous versions'.

vfs objects = shadow_copy2

Notes: This config does work at the global scope (where adding it to extras puts it by default), and will apply to all of your shares. You just don't get to configure any of the other options for this feature. Unfortunately, duplicating the UnRAID team's work to build share configs on the fly is outside of my ambition, so I'm willing to live with that compromise. This is going to get us into an interesting situation, where the only place samba seems to be able to find our snapshots directory is at '/mnt/user/.snapshots'. The directory needs to be created on each storage device and then let unraid aggregate it later, so we are going to do it in the script so we can handle new drives and new shares correctly.

Step 2) Install the CA Userscripts Plugin. Details of this step are outside the scope of this post.

 

Step 3) Settings -> User Scripts. Add New Script. Click on script name to edit it. Add the following code to the script. Adjust MAX_SNAPS to your preference. Schedule it as you desire. Adjust EXCLUDE to your preference.

Random Notes: This provides some, but minimal protection from ransomware and bit-rot. In particular, ransomware which understands a linux system and actually gets access to the server could purge snapshots.

Edit 1/6/20: Added options to exclude some shares from being snapshotted. -e\--exclude <Comma seperated list of shortnames>

#!/bin/bash
#description=This script implements incremental snapshots on btrfs array drives.
#arrayStarted=true
#argumentDescription= -n|--number <MAXIMUM NUMBER OF SNAPSHOTS TO RETAIL>
#argumentDefault=-s 8760

shopt -s nullglob #make empty directories not freak out
date=$(TZ=GMT date +@GMT-%Y.%m.%d-%H.%M.%S) #standardized datestamp
MAX_SNAPS=8760
EXCLUDE=

is_btrfs_subvolume() {
    local dir=$1
    [ "$(stat -f --format="%T" "$dir")" == "btrfs" ] || return 1
    inode="$(stat --format="%i" "$dir")"
    case "$inode" in
        2|256)
            return 0;;
        *)
            return 1;;
    esac
}

POSITIONAL=()
while [[ $# -gt 0 ]]
do
key="$1"

case $key in
    -n|--number)
    MAX_SNAPS="$2"
    shift # past argument
    shift # past value
    ;;
    -e|--exclude)
    EXCLUDE="$2"
    shift # past argument
    shift # past value
    ;;
    *)
    POSITIONAL+=("$1") # save it in an array for later
    shift # past argument
    ;;
esac
done
set -- "${POSITIONAL[@]}" # restore positional parameters

#ADJUST MAX_SNAPS to prevent off-by-1
MAX_SNAPS=$((MAX_SNAPS+1))

#Tokenize exclude list
declare -A excludes
for token in ${EXCLUDE//,/ }; do
	excludes[$token]=1
done

#iterate over all disks on array
for disk in /mnt/disk*[0-9]* ; do
#examine disk for btrfs-formatting (MOSTLY UNTESTED)
 if is_btrfs_subvolume $disk ; then 
  #check for .snapshots directory prior to generating snapshot
  if [ -d "$disk" ]; then
   if [ ! -d "$disk/.snapshots/" ] ; then
    mkdir -v $disk/.snapshots
   fi
   if [ ! -d "$disk/.snapshots/$date/" ] ; then
    mkdir -v $disk/.snapshots/$date
   fi
  fi
 #iterate over shares present on disk
  for share in ${disk}/* ; do
   #test for exclusion
   if [ ! -n "${excludes[$(basename $share)]}" ]; then
    #echo "Examining $share on $disk"
    is_btrfs_subvolume $share
    if [ ! "$?" -eq 0 ]; then
     #echo "$share is likely not a subvolume"
     mv -v ${share} ${share}_TEMP
     btrfs subvolume create $share
     cp -avT --reflink=always ${share}_TEMP $share
     rm -vrf ${share}_TEMP
    fi
    #make new snap
    btrfs subvolume snap -r ${share} /mnt/$(basename $disk)/.snapshots/${date}/$(basename $share)
   else
    echo "$share is on the exclusion list. Skipping..."
   fi
  done
  #find old snaps
  echo "Found $(find ${disk}/.snapshots/ -maxdepth 1 -mindepth 1 | sort -nr | tail -n +$MAX_SNAPS | wc -l) old snaps"
  for snap in $(find ${disk}/.snapshots/ -maxdepth 1 -mindepth 1 | sort -nr | tail -n +$MAX_SNAPS); do
   for share_snap in ${snap}/*; do
    btrfs subvolume delete $share_snap
   done
   rm -rfv $snap
  done
 fi
done

 

Edited by catapultam_habeo
updated script

Share this post


Link to post
Posted (edited)

Here is a housekeeping script. By default just deletes empty timestamp directories. Optionally, it can delete snapshots.

 

-a purges all snapshots

-i <Comma seperated list of shares> purges the selected shares. example -i Downloads,Test1,Test2 will purge all snapshots for Downloads, Test1, and Test2.

 

#!/bin/bash
shopt -s nullglob

POSITIONAL=()
while [[ $# -gt 0 ]]
do
key="$1"

case $key in
    -i|--include)
    INCLUDE="$2"
    shift # past argument
    shift # past value
    ;;
    -a|--all)
    ALL=YES
    shift
    ;;
    *)    # unknown option
    POSITIONAL+=("$1") # save it in an array for later
    shift # past argument
    ;;
esac
done
set -- "${POSITIONAL[@]}" # restore positional parameters

#Tokenize include list
declare -A includes
for token in ${INCLUDE//,/ }; do
        includes[$token]=1
done

#iterate over all disks on array
for disk in /mnt/disk*[0-9]* ; do
 #iterate over each timestamp
 for timestamp in ${disk}/.snapshots/* ; do
  #iterate over each share in the timestamp
  for snap in $timestamp/* ; do
   if [ -n "${includes[$(basename $snap)]}" ] || [ "$ALL" = "YES" ] ; then
    echo "Purging - $snap"
    btrfs subvolume delete $snap
  fi
  #check for empty timestamp
  if [ ! "$(ls -A $timestamp)" ] ; then
   echo "Purging empty directory - $timestamp"
   rmdir $timestamp
  fi
  done
 done
done

 

Edited by catapultam_habeo

Share this post


Link to post

Hello, Thank you for your great job.

For snapshot disks, it works.

I can see a list by command "btrfs sub list /mnt/diskN", but it doesn't work via windows 10 access to check the previous version.

Does it conflict with "Enhanced macOS interoperability" option? If it is, may I just restore the disks to some specific time point?

And what do we do with the cache drive? Seems there're no snapshot on it.

Sry, I'm not very good at linux, and do need snapshot function for data safety.

Thank you so much.

Share this post


Link to post
Posted (edited)

I have most of this script working but I don't think the arguments are, it looks like the arguments are being treated as a single argument and not as individual arguments. If I echo $#, regardless of how many space separated values I put in it always comes out as 1.

 

User Scripts plugin is version 2020.03.19

Edited by Supermillhouse

Share this post


Link to post
On 3/18/2020 at 5:35 AM, afon said:

Hello, Thank you for your great job.

For snapshot disks, it works.

I can see a list by command "btrfs sub list /mnt/diskN", but it doesn't work via windows 10 access to check the previous version.

Does it conflict with "Enhanced macOS interoperability" option? If it is, may I just restore the disks to some specific time point?

And what do we do with the cache drive? Seems there're no snapshot on it.

Sry, I'm not very good at linux, and do need snapshot function for data safety.

Thank you so much.


You probably didn't do Step 1 in the original post.

 

 

On 4/16/2020 at 9:36 AM, Supermillhouse said:

I have most of this script working but I don't think the arguments are, it looks like the arguments are being treated as a single argument and not as individual arguments. If I echo $#, regardless of how many space separated values I put in it always comes out as 1.

 

User Scripts plugin is version 2020.03.19

I'll check on this. It seems to be working on my local version, but admittedly I don't parse the args, I just set them in the EXCLUDE var directly.

Share this post


Link to post

Hi, I did.

I paste the SMB configure field here:

Quote

#unassigned_devices_start
#Unassigned devices share includes
   include = /tmp/unassigned.devices/smb-settings.conf
#unassigned_devices_end

#vfs_recycle_start
#Recycle bin configuration
[global]
   syslog only = Yes
   log level = 0 vfs:0

#vfs_recycle_end
#shadow_copy_enable
vfs objects = shadow_copy2
#shadow_copy_end

 

Share this post


Link to post
Posted (edited)

I am shocked this has not gotten more attention.

 

Lack of snapshots has been one of the only shortcomings to using unraid I have noticed. This includes VM snapshots, I already miss those in a big way to the point I think I am going to setup a VM for esxi to use vmware for VM's that don't need passthrough. It is a sad day when windows has a feature and linux doesn't lol.

 

btrfs seems to be stable as long as not used in raid 56, so I am seriously considering going with Bbtrfs for my array IF snapshots can be made to work.

 

Maybe setting it up for cache snapshots would get it more attention since most users are still on xfs arrays?

 

A few questions:

 

1: I read online that at some point in the past excessive snapshots could make btrfs slow down / have performance issues, do you know if this has been fixed?

 

2: I do not need hourly snapshots, daily / weekly would be more then enough. Is there a simple way to change the frequency?

 

3: It would be really cool if the housekeeping script could not only purge certain shares snapshots but also prune from hourly down to daily when older then X time. And from daily to weekly snapshots when older then Y time and weekly to monthly after Z time etc.

 

For example hourly for a week, then daily for a month and weekly for 3 months and monthly for a year would be a good starting point IMHO.

 

 

I would LOVE to see a plugin for this with a simple GUI.

Edited by TexasUnraid

Share this post


Link to post

Been getting closer to putting my server into service and was setting up this snapshot script when I noticed it did not support folders with spaces in them.

 

After some trial and error I got it working on folders with spaces, although the exclude option still doesn't work with spaces and I can't figure out how to get it working.

 

The argument option of user scripts also seems to only support a single argument passed to the script and does not support spaces either.

 

Here is the updated script with all the $share variables wrapped in quotes and timezone changed to CST:

 

#!/bin/bash
#description=This script implements incremental snapshots on btrfs array drives.
#arrayStarted=true
#argumentDescription= -n|--number <MAXIMUM NUMBER OF SNAPSHOTS TO RETAIL>
#argumentDefault=-s 3

shopt -s nullglob #make empty directories not freak out
date=$(TZ=America/Chicago date +@CST-%Y.%m.%d-%H.%M.%S) #standardized datestamp
MAX_SNAPS=3
EXCLUDE=

is_btrfs_subvolume() {
    local dir=$1
    [ "$(stat -f --format="%T" "$dir")" == "btrfs" ] || return 1
    inode="$(stat --format="%i" "$dir")"
    case "$inode" in
        2|256)
            return 0;;
        *)
            return 1;;
    esac
}

POSITIONAL=()
while [[ $# -gt 0 ]]
do
key="$1"

case $key in
    -n|--number)
    MAX_SNAPS="$2"
    shift # past argument
    shift # past value
    ;;
    -e|--exclude)
    EXCLUDE="$2"
    shift # past argument
    shift # past value
    ;;
    *)
    POSITIONAL+=("$1") # save it in an array for later
    shift # past argument
    ;;
esac
done
set -- "${POSITIONAL[@]}" # restore positional parameters

#ADJUST MAX_SNAPS to prevent off-by-1
MAX_SNAPS=$((MAX_SNAPS+1))

#Tokenize exclude list
declare -A excludes
for token in ${EXCLUDE//,/ }; do
	excludes[$token]=1
done

#iterate over all disks on array
for disk in /mnt/disk*[0-9]* ; do
#examine disk for btrfs-formatting (MOSTLY UNTESTED)
 if is_btrfs_subvolume $disk ; then 
  #check for .snapshots directory prior to generating snapshot
  if [ -d "$disk" ]; then
   if [ ! -d "$disk/.snapshots/" ] ; then
    mkdir -v $disk/.snapshots
   fi
   if [ ! -d "$disk/.snapshots/$date/" ] ; then
    mkdir -v $disk/.snapshots/$date
   fi
  fi
 #iterate over shares present on disk
  for share in ${disk}/* ; do
   #test for exclusion
   if [ ! -n "${excludes[$(basename "$share")]}" ]; then
    #echo "Examining $share on $disk"
    is_btrfs_subvolume "$share"
    if [ ! "$?" -eq 0 ]; then
     #echo "$share is likely not a subvolume"
     mv -v "${share}" "${share}_TEMP"
     btrfs subvolume create "$share"
     cp -avT --reflink=always "${share}_TEMP" "$share"
     rm -vrf "${share}_TEMP"
    fi
    #make new snap
    btrfs subvolume snap -r "${share}" "/mnt/$(basename $disk)/.snapshots/${date}/$(basename "$share")"
   else
    echo "$share is on the exclusion list. Skipping..."
   fi
  done
  #find old snaps
  echo "Found $(find ${disk}/.snapshots/ -maxdepth 1 -mindepth 1 | sort -nr | tail -n +$MAX_SNAPS | wc -l) old snaps"
  for snap in $(find ${disk}/.snapshots/ -maxdepth 1 -mindepth 1 | sort -nr | tail -n +$MAX_SNAPS); do
   for share_snap in ${snap}/*; do
    btrfs subvolume delete "$share_snap"
   done
   rm -rfv "$snap"
  done
 fi
done

 

Share this post


Link to post

Oh, I found the cleanup script needed a similar update with quotes as well when trying to clean up the mess I made while sorting this out lol

 

#!/bin/bash
#description=This script cleans up snapshots.
#arrayStarted=true
#argumentDescription= -i <Comma separated list of shares to purge> or -a to purge all
#argumentDefault=-i share_name_here

shopt -s nullglob
INCLUDE=

POSITIONAL=()
while [[ $# -gt 0 ]]
do
key="$1"

case $key in
    -i|--include)
    INCLUDE="$2"
    shift # past argument
    shift # past value
    ;;
    -a|--all)
    ALL=YES
    shift
    ;;
    *)    # unknown option
    POSITIONAL+=("$1") # save it in an array for later
    shift # past argument
    ;;
esac
done
set -- "${POSITIONAL[@]}" # restore positional parameters

#Tokenize include list
declare -A includes
for token in ${INCLUDE//,/ }; do
        includes[$token]=1
done

#iterate over all disks on array
for disk in /mnt/disk*[0-9]* ; do
 #iterate over each timestamp
 for timestamp in ${disk}/.snapshots/* ; do
  #iterate over each share in the timestamp
  for snap in $timestamp/* ; do
   if [ -n "${includes[$(basename "$snap")]}" ] || [ "$ALL" = "YES" ] ; then
    echo "Purging - $snap"
    btrfs subvolume delete "$snap"
  fi
  #check for empty timestamp
  if [ ! "$(ls -A $timestamp)" ] ; then
   echo "Purging empty directory - $timestamp"
   rmdir $timestamp
  fi
  done
 done
done

Thanks again for this script!

 

Can't wait to see snapshots added to unraid GUI, either officially or in plugin form.

 

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.