catapultam_habeo Posted December 30, 2019 Share Posted December 30, 2019 (edited) Hi, this is mostly a WIP thread, but as of the first post it does work up to my relatively limited testing. I plan on expanding this to a fully featured plugin, but this script is a working foundation, and I'd like to make this available to people to play with asap. Bottom Line Up Front: This script only works on your btrfs-formatted array drives. By default, it will keep 8760 snapshots (1 year of hourly snapshots), this value can be adjusted by changing the MAX_SNAPS variable in the script. This script does not handle cache drives, but would be trivial to extend to do so - I just think it is a bad idea. Detection of this is minimal but present. Running this script for the first time will remove and recreate all of your existing array shares (moved to temporary path, original path converted to subvolume, moved back), no data should be lost, but I have only tested this with my own data and configuration, and cannot account for all edge cases and I absolutely cannot be held accountable for your data if it is lost. No script is provided to revert these changes. Goals: I wanted to have delta snapshot recovery as part of my NAS feature set. FreeNAS is appealing, but I dislike FreeBSD's ecosystem (weird problems with bhyve, don't really need ZFS performance improvements), tried ProxMox and didn't care for it, didn't want to roll my own (likely debian-based) setup, very much like unRAID's GUI and asynchronous drive upgrade process, and wasn't interested in the crazy ZFS-on-unRAID frankenstein config by Wendell at L1T. I noted that unRAID can be configured to use BTRFS, and in theory it should be able to do this, given enough scripting to keep everything in sync. I also want to leverage unRAID's GUI as heavily as possible, and do as little command-line work on the regular as possible. Adding a new drive, creating a new share, etc, should all be possible through the GUI, and this setup should automagically adjust. Step 1) Adjust your Settings -> SMB -> SMB Extras field to include the following line. This will publish your snapshots to windows SMB clients as 'previous versions'. vfs objects = shadow_copy2 Notes: This config does work at the global scope (where adding it to extras puts it by default), and will apply to all of your shares. You just don't get to configure any of the other options for this feature. Unfortunately, duplicating the UnRAID team's work to build share configs on the fly is outside of my ambition, so I'm willing to live with that compromise. This is going to get us into an interesting situation, where the only place samba seems to be able to find our snapshots directory is at '/mnt/user/.snapshots'. The directory needs to be created on each storage device and then let unraid aggregate it later, so we are going to do it in the script so we can handle new drives and new shares correctly. Step 2) Install the CA Userscripts Plugin. Details of this step are outside the scope of this post. Step 3) Settings -> User Scripts. Add New Script. Click on script name to edit it. Add the following code to the script. Adjust MAX_SNAPS to your preference. Schedule it as you desire. Adjust EXCLUDE to your preference. Random Notes: This provides some, but minimal protection from ransomware and bit-rot. In particular, ransomware which understands a linux system and actually gets access to the server could purge snapshots. Edit 1/6/20: Added options to exclude some shares from being snapshotted. -e\--exclude <Comma seperated list of shortnames> #!/bin/bash #description=This script implements incremental snapshots on btrfs array drives. #arrayStarted=true #argumentDescription= -n|--number <MAXIMUM NUMBER OF SNAPSHOTS TO RETAIL> #argumentDefault=-s 8760 shopt -s nullglob #make empty directories not freak out date=$(TZ=GMT date +@GMT-%Y.%m.%d-%H.%M.%S) #standardized datestamp MAX_SNAPS=8760 EXCLUDE= is_btrfs_subvolume() { local dir=$1 [ "$(stat -f --format="%T" "$dir")" == "btrfs" ] || return 1 inode="$(stat --format="%i" "$dir")" case "$inode" in 2|256) return 0;; *) return 1;; esac } POSITIONAL=() while [[ $# -gt 0 ]] do key="$1" case $key in -n|--number) MAX_SNAPS="$2" shift # past argument shift # past value ;; -e|--exclude) EXCLUDE="$2" shift # past argument shift # past value ;; *) POSITIONAL+=("$1") # save it in an array for later shift # past argument ;; esac done set -- "${POSITIONAL[@]}" # restore positional parameters #ADJUST MAX_SNAPS to prevent off-by-1 MAX_SNAPS=$((MAX_SNAPS+1)) #Tokenize exclude list declare -A excludes for token in ${EXCLUDE//,/ }; do excludes[$token]=1 done #iterate over all disks on array for disk in /mnt/disk*[0-9]* ; do #examine disk for btrfs-formatting (MOSTLY UNTESTED) if is_btrfs_subvolume $disk ; then #check for .snapshots directory prior to generating snapshot if [ -d "$disk" ]; then if [ ! -d "$disk/.snapshots/" ] ; then mkdir -v $disk/.snapshots fi if [ ! -d "$disk/.snapshots/$date/" ] ; then mkdir -v $disk/.snapshots/$date fi fi #iterate over shares present on disk for share in ${disk}/* ; do #test for exclusion if [ ! -n "${excludes[$(basename $share)]}" ]; then #echo "Examining $share on $disk" is_btrfs_subvolume $share if [ ! "$?" -eq 0 ]; then #echo "$share is likely not a subvolume" mv -v ${share} ${share}_TEMP btrfs subvolume create $share cp -avT --reflink=always ${share}_TEMP $share rm -vrf ${share}_TEMP fi #make new snap btrfs subvolume snap -r ${share} /mnt/$(basename $disk)/.snapshots/${date}/$(basename $share) else echo "$share is on the exclusion list. Skipping..." fi done #find old snaps echo "Found $(find ${disk}/.snapshots/ -maxdepth 1 -mindepth 1 | sort -nr | tail -n +$MAX_SNAPS | wc -l) old snaps" for snap in $(find ${disk}/.snapshots/ -maxdepth 1 -mindepth 1 | sort -nr | tail -n +$MAX_SNAPS); do for share_snap in ${snap}/*; do btrfs subvolume delete $share_snap done rm -rfv $snap done fi done Edited January 6, 2020 by catapultam_habeo updated script 6 Quote Link to comment
catapultam_habeo Posted January 6, 2020 Author Share Posted January 6, 2020 (edited) Here is a housekeeping script. By default just deletes empty timestamp directories. Optionally, it can delete snapshots. -a purges all snapshots -i <Comma seperated list of shares> purges the selected shares. example -i Downloads,Test1,Test2 will purge all snapshots for Downloads, Test1, and Test2. #!/bin/bash shopt -s nullglob POSITIONAL=() while [[ $# -gt 0 ]] do key="$1" case $key in -i|--include) INCLUDE="$2" shift # past argument shift # past value ;; -a|--all) ALL=YES shift ;; *) # unknown option POSITIONAL+=("$1") # save it in an array for later shift # past argument ;; esac done set -- "${POSITIONAL[@]}" # restore positional parameters #Tokenize include list declare -A includes for token in ${INCLUDE//,/ }; do includes[$token]=1 done #iterate over all disks on array for disk in /mnt/disk*[0-9]* ; do #iterate over each timestamp for timestamp in ${disk}/.snapshots/* ; do #iterate over each share in the timestamp for snap in $timestamp/* ; do if [ -n "${includes[$(basename $snap)]}" ] || [ "$ALL" = "YES" ] ; then echo "Purging - $snap" btrfs subvolume delete $snap fi #check for empty timestamp if [ ! "$(ls -A $timestamp)" ] ; then echo "Purging empty directory - $timestamp" rmdir $timestamp fi done done done Edited January 6, 2020 by catapultam_habeo Quote Link to comment
afon Posted March 18, 2020 Share Posted March 18, 2020 Hello, Thank you for your great job. For snapshot disks, it works. I can see a list by command "btrfs sub list /mnt/diskN", but it doesn't work via windows 10 access to check the previous version. Does it conflict with "Enhanced macOS interoperability" option? If it is, may I just restore the disks to some specific time point? And what do we do with the cache drive? Seems there're no snapshot on it. Sry, I'm not very good at linux, and do need snapshot function for data safety. Thank you so much. Quote Link to comment
Supermillhouse Posted April 16, 2020 Share Posted April 16, 2020 (edited) I have most of this script working but I don't think the arguments are, it looks like the arguments are being treated as a single argument and not as individual arguments. If I echo $#, regardless of how many space separated values I put in it always comes out as 1. User Scripts plugin is version 2020.03.19 Edited April 16, 2020 by Supermillhouse Quote Link to comment
catapultam_habeo Posted April 20, 2020 Author Share Posted April 20, 2020 On 3/18/2020 at 5:35 AM, afon said: Hello, Thank you for your great job. For snapshot disks, it works. I can see a list by command "btrfs sub list /mnt/diskN", but it doesn't work via windows 10 access to check the previous version. Does it conflict with "Enhanced macOS interoperability" option? If it is, may I just restore the disks to some specific time point? And what do we do with the cache drive? Seems there're no snapshot on it. Sry, I'm not very good at linux, and do need snapshot function for data safety. Thank you so much. You probably didn't do Step 1 in the original post. On 4/16/2020 at 9:36 AM, Supermillhouse said: I have most of this script working but I don't think the arguments are, it looks like the arguments are being treated as a single argument and not as individual arguments. If I echo $#, regardless of how many space separated values I put in it always comes out as 1. User Scripts plugin is version 2020.03.19 I'll check on this. It seems to be working on my local version, but admittedly I don't parse the args, I just set them in the EXCLUDE var directly. Quote Link to comment
afon Posted April 20, 2020 Share Posted April 20, 2020 Hi, I did. I paste the SMB configure field here: Quote #unassigned_devices_start #Unassigned devices share includes include = /tmp/unassigned.devices/smb-settings.conf #unassigned_devices_end #vfs_recycle_start #Recycle bin configuration [global] syslog only = Yes log level = 0 vfs:0 #vfs_recycle_end #shadow_copy_enable vfs objects = shadow_copy2 #shadow_copy_end Quote Link to comment
TexasUnraid Posted June 6, 2020 Share Posted June 6, 2020 (edited) I am shocked this has not gotten more attention. Lack of snapshots has been one of the only shortcomings to using unraid I have noticed. This includes VM snapshots, I already miss those in a big way to the point I think I am going to setup a VM for esxi to use vmware for VM's that don't need passthrough. It is a sad day when windows has a feature and linux doesn't lol. btrfs seems to be stable as long as not used in raid 56, so I am seriously considering going with Bbtrfs for my array IF snapshots can be made to work. Maybe setting it up for cache snapshots would get it more attention since most users are still on xfs arrays? A few questions: 1: I read online that at some point in the past excessive snapshots could make btrfs slow down / have performance issues, do you know if this has been fixed? 2: I do not need hourly snapshots, daily / weekly would be more then enough. Is there a simple way to change the frequency? 3: It would be really cool if the housekeeping script could not only purge certain shares snapshots but also prune from hourly down to daily when older then X time. And from daily to weekly snapshots when older then Y time and weekly to monthly after Z time etc. For example hourly for a week, then daily for a month and weekly for 3 months and monthly for a year would be a good starting point IMHO. I would LOVE to see a plugin for this with a simple GUI. Edited June 8, 2020 by TexasUnraid Quote Link to comment
TexasUnraid Posted July 4, 2020 Share Posted July 4, 2020 Been getting closer to putting my server into service and was setting up this snapshot script when I noticed it did not support folders with spaces in them. After some trial and error I got it working on folders with spaces, although the exclude option still doesn't work with spaces and I can't figure out how to get it working. The argument option of user scripts also seems to only support a single argument passed to the script and does not support spaces either. Here is the updated script with all the $share variables wrapped in quotes and timezone changed to CST: #!/bin/bash #description=This script implements incremental snapshots on btrfs array drives. #arrayStarted=true #argumentDescription= -n|--number <MAXIMUM NUMBER OF SNAPSHOTS TO RETAIL> #argumentDefault=-s 3 shopt -s nullglob #make empty directories not freak out date=$(TZ=America/Chicago date +@CST-%Y.%m.%d-%H.%M.%S) #standardized datestamp MAX_SNAPS=3 EXCLUDE= is_btrfs_subvolume() { local dir=$1 [ "$(stat -f --format="%T" "$dir")" == "btrfs" ] || return 1 inode="$(stat --format="%i" "$dir")" case "$inode" in 2|256) return 0;; *) return 1;; esac } POSITIONAL=() while [[ $# -gt 0 ]] do key="$1" case $key in -n|--number) MAX_SNAPS="$2" shift # past argument shift # past value ;; -e|--exclude) EXCLUDE="$2" shift # past argument shift # past value ;; *) POSITIONAL+=("$1") # save it in an array for later shift # past argument ;; esac done set -- "${POSITIONAL[@]}" # restore positional parameters #ADJUST MAX_SNAPS to prevent off-by-1 MAX_SNAPS=$((MAX_SNAPS+1)) #Tokenize exclude list declare -A excludes for token in ${EXCLUDE//,/ }; do excludes[$token]=1 done #iterate over all disks on array for disk in /mnt/disk*[0-9]* ; do #examine disk for btrfs-formatting (MOSTLY UNTESTED) if is_btrfs_subvolume $disk ; then #check for .snapshots directory prior to generating snapshot if [ -d "$disk" ]; then if [ ! -d "$disk/.snapshots/" ] ; then mkdir -v $disk/.snapshots fi if [ ! -d "$disk/.snapshots/$date/" ] ; then mkdir -v $disk/.snapshots/$date fi fi #iterate over shares present on disk for share in ${disk}/* ; do #test for exclusion if [ ! -n "${excludes[$(basename "$share")]}" ]; then #echo "Examining $share on $disk" is_btrfs_subvolume "$share" if [ ! "$?" -eq 0 ]; then #echo "$share is likely not a subvolume" mv -v "${share}" "${share}_TEMP" btrfs subvolume create "$share" cp -avT --reflink=always "${share}_TEMP" "$share" rm -vrf "${share}_TEMP" fi #make new snap btrfs subvolume snap -r "${share}" "/mnt/$(basename $disk)/.snapshots/${date}/$(basename "$share")" else echo "$share is on the exclusion list. Skipping..." fi done #find old snaps echo "Found $(find ${disk}/.snapshots/ -maxdepth 1 -mindepth 1 | sort -nr | tail -n +$MAX_SNAPS | wc -l) old snaps" for snap in $(find ${disk}/.snapshots/ -maxdepth 1 -mindepth 1 | sort -nr | tail -n +$MAX_SNAPS); do for share_snap in ${snap}/*; do btrfs subvolume delete "$share_snap" done rm -rfv "$snap" done fi done Quote Link to comment
TexasUnraid Posted July 4, 2020 Share Posted July 4, 2020 Oh, I found the cleanup script needed a similar update with quotes as well when trying to clean up the mess I made while sorting this out lol #!/bin/bash #description=This script cleans up snapshots. #arrayStarted=true #argumentDescription= -i <Comma separated list of shares to purge> or -a to purge all #argumentDefault=-i share_name_here shopt -s nullglob INCLUDE= POSITIONAL=() while [[ $# -gt 0 ]] do key="$1" case $key in -i|--include) INCLUDE="$2" shift # past argument shift # past value ;; -a|--all) ALL=YES shift ;; *) # unknown option POSITIONAL+=("$1") # save it in an array for later shift # past argument ;; esac done set -- "${POSITIONAL[@]}" # restore positional parameters #Tokenize include list declare -A includes for token in ${INCLUDE//,/ }; do includes[$token]=1 done #iterate over all disks on array for disk in /mnt/disk*[0-9]* ; do #iterate over each timestamp for timestamp in ${disk}/.snapshots/* ; do #iterate over each share in the timestamp for snap in $timestamp/* ; do if [ -n "${includes[$(basename "$snap")]}" ] || [ "$ALL" = "YES" ] ; then echo "Purging - $snap" btrfs subvolume delete "$snap" fi #check for empty timestamp if [ ! "$(ls -A $timestamp)" ] ; then echo "Purging empty directory - $timestamp" rmdir $timestamp fi done done done Thanks again for this script! Can't wait to see snapshots added to unraid GUI, either officially or in plugin form. Quote Link to comment
TexasUnraid Posted September 2, 2020 Share Posted September 2, 2020 (edited) I was inspired by this script so I started out trying to adapt it to work for my UD backup drive but didn't understand how parts of it worked (still new to linux, the exclude and BTRFS check sections made no sense to me). Like normal, it is at your own risk and make sure you have a backup. So I ended up basically re-writing those sections but thus far it seems to work. You will enter the path to the drive(s) you want to snapshot in the UDDRIVE= variables. It will then scan the paths to make sure they are valid and that they are BTRFS drives. It then scans the folders to make sure they are subvolumes and not just normal folders. If the folders are not subvolumes it will let you know that it needs to be put into conversion mode before it can make snapshots. I decided to make this separate as I was not sure of any downsides to automatically doing this and better to error on the safe side. To use conversion mode simply enter "convert" in the argument box and it will convert the folders to subvolumes, it can take some time if you have a lot of files or slow drives. Once the conversion is finished it will exit and you can restart it in normal mode to take a snapshot. The argument is used for the number of snapshot history to keep, 0 will erase all snapshots, good for clearing out space if you deleted a lot of files. My plan is to use it on my weekly and monthly cold backups to give me the ability to get old version of files if it were needed. #!/bin/bash #description=This script will create a snapshot on unassigned device, perfect for use with a backup drive for example. Simply enter the path drives you wish to snapsshot in the DRIVEX variables. Example: /mnt/disks/backup. Do not include a tailing /. only enter drives that are connected, it will not run if the drive is entered but not detected. The argument is normally used for the number of snapshot history you want to keep. It also includes the ability to convert folders to subvolumes which is needed for snapshots to work. To convert your exsisting folders first make sure you have a backup of your data and then run the script with "convert" in the argument. #backgroundOnly= #arrayStarted= #noParity= #argumentDescription= Number of snapshots to keep, set to 0 to remove all snapshots (allows you to free up space if you deleted a lot of things). Or put script into Subvolume conversion mode by entering "convert". #argumentDefault= 12 UDDRIVE1=/mnt/disks/Personal_Data_Backup UDDRIVE2= UDDRIVE3= UDDRIVE4= DATE=$(TZ=America/Chicago date +@CST-%m.%d.%Y-%H.%M.%S) #standardized datestamp CONVERSIONPATH=$(echo "$1" | sed -e "s/^convert//") #check to make sure drive is connected and all folders are converted to subvolumes for DRIVE in $UDDRIVE1 $UDDRIVE2 $UDDRIVE3 $UDDRIVE4 ; do echo Checking that $DRIVE is connected... if [ -d "$DRIVE" ] ; then echo "Drive $DRIVE exists" echo "Checking that $DRIVE is a BTRFS volume" btrfs subvolume show "$DRIVE" if [ $? -eq 0 ] ; then echo echo "Drive $DRIVE is a BTRFS volume" echo echo ----------------------------------------------------------------------------------------- echo "Now checking that each subfolder is valid BTRFS subvolume" echo echo #check that each folder is a BTRFS subvolume, if not warn and suggest convert mode for BTRFSCHECK in $DRIVE/* ; do btrfs subvolume show "$BTRFSCHECK" if [ $? -eq 0 ] ; then echo echo "Folder $BTRFSCHECK is BTRFS subvolume, ready for snapshots" echo echo ----------------------------------------------------------------------------------------- else echo "Folder $BTRFSCHECK is NOT a subvolume, can not create snapshot, use convert mode to convert it to subvolume" if [ "$1" = "convert" ] ; then echo "-----------------Folder to subvolume conversion mode is Active, DO NOT interrupt, it might take a few minutes-------------------" echo mv -v "$BTRFSCHECK" "${BTRFSCHECK}_TEMP" btrfs subvolume create "$BTRFSCHECK" cp -avT --reflink=always "${BTRFSCHECK}_TEMP" "$BTRFSCHECK" rm -vrf "${BTRFSCHECK}_TEMP" btrfs subvolume show "$BTRFSCHECK" if [ $? -eq 0 ] ; then echo echo "Subvolume conversion complete for $BTRFSCHECK. Remove convert from argument now and restart script to take snapshot." echo else echo Something went wrong with subvolume conversion, see logs. Exiting. exit 99 fi else echo "Folder to subvolume conversion mode not active, Not all folders are subvolumes, can not continue..." echo echo "Enter '"convert"' without quotesin the argument to enable conversion mode" echo "DO NOT interrupt folder to subvolume conversion once started, it can take several minutes if you have a lot of files" echo "Don't run convert mode unless you are sure the drive is a BTRFS drive" exit 99 fi fi done else echo "Folder $BTRFSCHECK is NOT a subvolume, can not create snapshot, use convert mode to convert it to subvolume" exit 99 fi else echo "$DRIVE does not exists. Exiting" ; exit 99 fi done #if convert mode active, stop script so that max snaps history can be entered in argument if [ "$1" = "convert" ] ; then echo Conversion mode ended, Exiting. If successful Restart script without convert mode to take snapshot exit else MAX_SNAPS=$(("$1"+1)) fi echo Starting Snapshots for all folders on drive, echo if you get errors make sure that all of the folders have been converted to subvolumes by entering '"convert"' in the argument echo for BTRFSDRIVE in $UDDRIVE1 $UDDRIVE2 $UDDRIVE3 $UDDRIVE4 ; do if [ -d "$BTRFSDRIVE" ]; then if [ ! -d "$BTRFSDRIVE/.snapshots/" ] ; then mkdir -v "$BTRFSDRIVE/.snapshots" fi if [ ! -d "$BTRFSDRIVE/.snapshots/$DATE/" ] ; then mkdir -v "$BTRFSDRIVE/.snapshots/$DATE" fi fi for SHARE in $BTRFSDRIVE/* ; do #make new snapshot btrfs subvolume snap -r "${SHARE}" "/mnt/disks/$(basename "$BTRFSDRIVE")/.snapshots/${DATE}/$(basename "$SHARE")" echo done #Cleanup old snapshots echo "Found $(find ${BTRFSDRIVE}/.snapshots/ -maxdepth 1 -mindepth 1 | sort -nr | tail -n +$MAX_SNAPS | wc -l) old snaps" for SNAP in $(find "${BTRFSDRIVE}/.snapshots/" -maxdepth 1 -mindepth 1 | sort -nr | tail -n +$MAX_SNAPS); do for SHARE_SNAP in ${SNAP}/*; do btrfs subvolume delete "$SHARE_SNAP" done rm -rfv "$SNAP" done done echo echo Finished taking snapshots and cleaning up old snapshots. Edited September 2, 2020 by TexasUnraid Quote Link to comment
segator Posted September 15, 2020 Share Posted September 15, 2020 this defintelly will need a plugin.. it will be so nice a plugin with a UI with all the snapshots stuff and cronjob configurable. also this plugin could enable shadow copy smb feature Quote Link to comment
TexasUnraid Posted September 15, 2020 Share Posted September 15, 2020 (edited) On 9/15/2020 at 5:30 AM, segator said: this defintelly will need a plugin.. it will be so nice a plugin with a UI with all the snapshots stuff and cronjob configurable. also this plugin could enable shadow copy smb feature Yep, to be honest I am shocked that a server OS is lacking such basic features as snapshots as a built in feature. Really surprised a plugin does not exists for it either. Edited September 16, 2020 by TexasUnraid Quote Link to comment
segator Posted September 15, 2020 Share Posted September 15, 2020 I'm trying to find out documentation about unraid plugin creation, but i can not find too many things. I have clear idea how to create the page and package the plugin but for exemple if I want to deploy a service that boot just after array start? Quote Link to comment
TexasUnraid Posted September 15, 2020 Share Posted September 15, 2020 I am sure this exists but no idea where and it is way over my head. I know I found some details on how to make scripts use the unraid notification system purely by chance at some point and it is actually quite simple to do. Got to think there is a thread about the plugin system floating around as well. It could be from a decade ago though. Quote Link to comment
primeval_god Posted September 16, 2020 Share Posted September 16, 2020 Unfortunately so far as i am aware this is the best there is. When last i asked about how to get started creating plugins it was suggested that the best way is to look at the github repos of some of the more prolific plugin writers to see how they constructed their plugins. Quote Link to comment
Tomr Posted October 13, 2020 Share Posted October 13, 2020 (edited) I'm evaluating UnRAID for my new NAS, I modified your script a bit. With it, one can create periodic snapshots with various retention policy (eg. keep hourly backups for a day, daily backups for a week etc). You just need to schedule more copy of the script. I think you can't pass arguments to scheduled script, so I deleted them. (Optional) Add this to your Settings->SMB->SMB Extra. This enables file versioning for smb clients. vfs objects = shadow_copy2 shadow:sort = desc shadow:format = _UTC_%Y.%m.%d-%H.%M.%S shadow:localtime = no shadow:snapprefix = ^\(monthly\)\{0,1\}\(weekly\)\{0,1\}\(daily\)\{0,1\}\(hourly\)\{0,1\}$ shadow:delimiter = _UTC_ Here's the code: #!/bin/bash #description=This script implements incremental snapshots on btrfs array drives. Credits @catapultam_habeo #arrayStarted=true #If you change the type you'll have to delete the old snapshots manually #valid values are: hourly, daily, weekly, monthly SNAPSHOT_TYPE=daily #How many days should these snapshots last. RETENTION_MINUTES=$((7*60*24)) #7 days #RETENTION_MINUTES=5 #Name of the shares to snapshot, can be comma separated like "medias,valuables" INCLUDE=valuables #name of the snapshot folder and delimeter. Do not change. #https://www.samba.org/samba/docs/current/man-html/vfs_shadow_copy2.8.html SNAPSHOT_DELIMETER="_UTC_" SNAPSHOT_FORMAT="$(TZ=UTC date +${SNAPSHOT_TYPE}${SNAPSHOT_DELIMETER}%Y.%m.%d-%H.%M.%S)" shopt -s nullglob #make empty directories not freak out is_btrfs_subvolume() { local dir=$1 [ "$(stat -f --format="%T" "$dir")" == "btrfs" ] || return 1 inode="$(stat --format="%i" "$dir")" case "$inode" in 2|256) return 0;; *) return 1;; esac } #Tokenize include list declare -A includes for token in ${INCLUDE//,/ }; do includes[$token]=1 done echo "RETENTION_MINUTES: ${RETENTION_MINUTES}" #iterate over all disks on array for disk in /mnt/disk*[0-9]* ; do #examine disk for btrfs-formatting (MOSTLY UNTESTED) if is_btrfs_subvolume $disk ; then #iterate over shares present on disk for share in ${disk}/* ; do declare baseShare=$(basename $share) #test for exclusion if [ ! -n "${includes[$baseShare]}" ]; then echo "$share is not in the inclusion list. Skipping..." else #check for .snapshots directory prior to generating snapshot #only before we make the actual snapshot #so we don't create empty folders if [ -d "$disk" ]; then if [ ! -d "$disk/.snapshots/$SNAPSHOT_FORMAT/" ] ; then #echo "new" mkdir -v -p $disk/.snapshots/$SNAPSHOT_FORMAT fi fi #echo "Examining $share on $disk" is_btrfs_subvolume $share if [ ! "$?" -eq 0 ]; then echo "$share is likely not a subvolume" mv -v ${share} ${share}_TEMP btrfs subvolume create $share cp -avT --reflink=always ${share}_TEMP $share rm -vrf ${share}_TEMP fi #make new snap btrfs subvolume snap -r ${share} $disk/.snapshots/${SNAPSHOT_FORMAT}/$baseShare #find old snaps #echo "Found $(find ${disk}/.snapshots/${SNAPSHOT_TYPE}${SNAPSHOT_DELIMETER}*/${baseShare} -maxdepth 0 -mindepth 0 -type d -mmin +${RETENTION_MINUTES} | wc -l) old snaps" for snap in $(find ${disk}/.snapshots/${SNAPSHOT_TYPE}${SNAPSHOT_DELIMETER}*/ -maxdepth 0 -mindepth 0 -type d -mmin +${RETENTION_MINUTES}); do #echo "Deleting: ${snap}${baseShare}" [ -d ${snap}${baseShare} ] && btrfs subvolume delete ${snap}${baseShare} #this will only delete empty directory so it's safe to call #even if there are other backups there rmdir $snap --ignore-fail-on-non-empty done fi done fi done Edited October 14, 2020 by Tomr Fixes Quote Link to comment
TexasUnraid Posted October 17, 2020 Share Posted October 17, 2020 (edited) Just wanted to say that I like where you are going with that, it is one of the features I would like to see in a plugin if it is ever made. I have not had time to test it out yet myself though as I am in the process of pre-clearing some drives and don't want to risk having to start that over lol. FYI, You can pass 1 argument to the script, just not more then one as spaces are not allowed in the argument with user scripts. It would be nice to be able to pass a "delete all snapshots" argument of some kind that will remove all snapshots so you can reclaim space if needed. I use this on a fairly regular basis. I suppose it is not possible to change the timezone of the snapshots? It would be handy for it to match the local timezone for hourly snapshots. EDIT: personally I find that an exclude more user friendly as if you ever add a share later it will be included without having to remember to add it manually. With an exclude list, worst case a new share gets snapshotted mistakenly and you have to delete it, reminding you to add it to the exclude list at the same time. Edited October 18, 2020 by TexasUnraid Quote Link to comment
slize Posted October 19, 2020 Share Posted October 19, 2020 I would also like to see this in the plugins section Quote Link to comment
BlackRockSoul Posted October 21, 2020 Share Posted October 21, 2020 Same. Can't wait to see a plugin for it. Also, I'd love to see a feature to exclude drives just not to spin up empty ones all the time. Quote Link to comment
TexasUnraid Posted October 28, 2020 Share Posted October 28, 2020 I finally had a little time to try out this script, had the same issues with it not being able to handle shares with spaces in the name. Oh well, I will just stick to the setup I am using now and see if someday a plugin is made. I do wish I could get the previous versions working in windows though. Quote Link to comment
TD2779 Posted October 28, 2020 Share Posted October 28, 2020 TexasUnraid's post from July 4th has an updated script that handles spaces. I don't recall if I altered the script after that, but it was my starting point and working well as far as I can tell. I have NOT gotten shadow copy/preview versions working yet though. Quote Link to comment
slize Posted December 23, 2020 Share Posted December 23, 2020 @Squid, @bonienl, @dlandon sorry to ping you guys, but i really think this feature would add so much value for UNRAID especially in those two points: #1 Humans do make mistakes and a snapshotted version of an office document would help a lot when overwritten accidentally. #2 Ransomware has become an serious issue around the world from large companies to very small networks. As long as a user cannot delete a snapshot and the server is not compromised it would not be as big of an issue if you get hit at one day. (Sure you can create backups every day but snapshots would be much more convenient.) It would be awesome to get some prominent attention on this topic and maybe see an plugin coming in 2021. I think this topic would also fit very well on @SpaceInvaderOne's YouTube channel. Quote Link to comment
ChrisYx511 Posted January 20, 2021 Share Posted January 20, 2021 (edited) I've modified @Tomr's snapshot with retention policy script to merge in @catapultam_habeo's original snapshot pruning method in order to improve the reliability of snapsot pruning. I've changed it back to a number of snapshots being retained instead of retention minutes. I found that going off minutes was fairly unreliable and often failed to auto-prune. EDIT: Forgot I also added back the exclusion list instead of an inclusion list. Instructions for shadow copy remains unchanged: On 10/13/2020 at 7:51 PM, Tomr said: (Optional) Add this to your Settings->SMB->SMB Extra. This enables file versioning for smb clients. vfs objects = shadow_copy2 shadow:sort = desc shadow:format = _UTC_%Y.%m.%d-%H.%M.%S shadow:localtime = no shadow:snapprefix = ^\(monthly\)\{0,1\}\(weekly\)\{0,1\}\(daily\)\{0,1\}\(hourly\)\{0,1\}$ shadow:delimiter = _UTC_ Here's the script: #!/bin/bash #description=This script implements incremental snapshots on btrfs array drives. #arrayStarted=true ## Credits # catapultam_habeo - Initial script # Tomr - Modified version with SNAPSHOT_TYPE retention policy #If you change the type you'll have to delete the old snapshots manually #valid values are: hourly, daily, weekly, monthly SNAPSHOT_TYPE=hourly #How many snapshots should be kept. MAX_SNAPS= #Name of the shares to exclude, can be comma separated like "medias,valuables" EXCLUDE= #name of the snapshot folder and delimeter. Do not change. #https://www.samba.org/samba/docs/current/man-html/vfs_shadow_copy2.8.html SNAPSHOT_DELIMETER="_UTC_" SNAPSHOT_FORMAT="$(TZ=UTC date +${SNAPSHOT_TYPE}${SNAPSHOT_DELIMETER}%Y.%m.%d-%H.%M.%S)" shopt -s nullglob #make empty directories not freak out is_btrfs_subvolume() { local dir=$1 [ "$(stat -f --format="%T" "$dir")" == "btrfs" ] || return 1 inode="$(stat --format="%i" "$dir")" case "$inode" in 2|256) return 0;; *) return 1;; esac } #ADJUST MAX_SNAPS to prevent off-by-1 MAX_SNAPS=$((MAX_SNAPS+1)) #Tokenize exclude list declare -A excludes for token in ${EXCLUDE//,/ }; do excludes[$token]=1 done #iterate over all disks on array for disk in /mnt/disk*[0-9]* ; do #examine disk for btrfs-formatting (MOSTLY UNTESTED) if is_btrfs_subvolume $disk ; then #iterate over shares present on disk for share in ${disk}/* ; do declare baseShare=$(basename $share) #test for exclusion if [ -n "${excludes[$baseShare]}" ]; then echo "$share is on the exclusion list. Skipping..." else #check for .snapshots directory prior to generating snapshot #only before we make the actual snapshot #so we don't create empty folders if [ -d "$disk" ]; then if [ ! -d "$disk/.snapshots/$SNAPSHOT_FORMAT/" ] ; then #echo "new" mkdir -v -p $disk/.snapshots/$SNAPSHOT_FORMAT fi fi #echo "Examining $share on $disk" is_btrfs_subvolume $share if [ ! "$?" -eq 0 ]; then echo "$share is likely not a subvolume" mv -v ${share} ${share}_TEMP btrfs subvolume create $share cp -avT --reflink=always ${share}_TEMP $share rm -vrf ${share}_TEMP fi #make new snap btrfs subvolume snap -r ${share} $disk/.snapshots/${SNAPSHOT_FORMAT}/$baseShare fi done #find old snaps echo "Found $(find ${disk}/.snapshots/${SNAPSHOT_TYPE}${SNAPSHOT_DELIMETER}*/ -maxdepth 0 -mindepth 0 | sort -nr | tail -n +$MAX_SNAPS | wc -l) old snaps" for snap in $(find ${disk}/.snapshots/${SNAPSHOT_TYPE}${SNAPSHOT_DELIMETER}*/ -maxdepth 0 -mindepth 0 | sort -nr | tail -n +$MAX_SNAPS); do for share_snap in ${snap}/*; do btrfs subvolume delete $share_snap done rmdir $snap --ignore-fail-on-non-empty done fi done Hope it's useful! Edited January 20, 2021 by ChrisYx511 Forgot an additional feature Quote Link to comment
bwestpha Posted June 5, 2021 Share Posted June 5, 2021 On 12/23/2020 at 3:13 AM, slize said: @Squid, @bonienl, @dlandon sorry to ping you guys, but i really think this feature would add so much value for UNRAID especially in those two points: #1 Humans do make mistakes and a snapshotted version of an office document would help a lot when overwritten accidentally. #2 Ransomware has become an serious issue around the world from large companies to very small networks. As long as a user cannot delete a snapshot and the server is not compromised it would not be as big of an issue if you get hit at one day. (Sure you can create backups every day but snapshots would be much more convenient.) It would be awesome to get some prominent attention on this topic and maybe see an plugin coming in 2021. I think this topic would also fit very well on @SpaceInvaderOne's YouTube channel. Totally agreed! Quote Link to comment
pyrater Posted June 6, 2021 Share Posted June 6, 2021 (edited) I would love a video or plugin for this. This has to be on the road map no? @SpaceInvaderOne please... Edited June 6, 2021 by pyrater 1 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.