HyperWorx

Just thought I would share my appdata backup script that I have been using for a couple of years now. It can be modified for any other accessible directory. I have used the Appdata backup plugin on community apps as well and I do highly recommend it. The way my brain works and even though less feature filled, I decided to just write a script to do exactly what I wanted and streamlined for my particular workflow. Hopefully this is the place to share and I do so in the off-chance someone may find some use for it or inspiration for their own scripts. How to use the script is given in the code comments. In short, it is designed to simplify and automate rolling, incremental, and permanent backups of the appdata folder. With customisable settings like the number of rolling backups to retain, the frequency of incremental backups, and intervals for permanent backups. The script automatically excludes user specified directories and filenames using regex patterns. Note that the script uses hard-linking to reduce storage utilisation, something I wanted that wasn't available in the Appdata backup plugin. Please use your due diligence and run a test case for your needs to see if it works. I have included cursory error checks, but nothing exhaustive. #!/bin/bash ###################################################################### # MyAppDataBackup v1 # ·•˚°○.●|HyperWorx - 2023|●.○°˚•· # # Description # ------------ # This script automates rolling, incremental, and permanent backups # for the specified appdata folder. It keeps a user-specified number # of rolling backups, performs incremental rolling backups and # permanent backups at defined intervals. # # Usage for automated backups: # Modify the user-defined variables to configure the script behavior. # Ensure the excluded_directories array includes directories to exclude # from the backup. # Run this script using the user scripts plugin or as a cron job. # Ensure proper execution permissions for the script. # # Explanation of User-defined Variables # -------------------------------------- # appdata_folder - Path to the source AppData folder to be backed up. # backup_parent - Parent directory for storing backups. # max_rolling_backups - The most recent number of rolling backups to retain. # incremental_backup_freq - Frequency of script execution for incremental backups. # Set to 0 for not utilising this functionality. # max_incremental_backups - The most recent number of incremental backups to retain. # permanent_backup_freq - Frequency of script execution for permanent backups. # Set to 0 for not utilising this functionality. # excluded_directories - List of directories to exclude from the backup. # If undefined, all appdata folders will be included. # excluded_file_patterns - List of regex compatible patterns to match filenames for # excluding from the backup. If undefined, all files included. # # Default Settings Example: # -------------------------_ # Executing the script every month and keeping 2 rolling backups (the last 2 months), # keep incremental backups at every 3 script executions (or 3 months) and storing the 4 of # last 4 most recent incremental backups, and keep a permanent backup snapshotted every # 12 script executions (or every 12 months). # # NOTE: # IT IS USER'S RESPONSIBILITY TO ENSURE SCRIPT WORKS FOR THEIR INTENDED USE CASE. ###################################################################### ################ USER DEFINED VARIABLES ################### appdata_folder="/mnt/user/appdata/" backup_parent="/mnt/user/backups/unraid/MyAppDataBackup/" max_rolling_backups=2 # Adjust the number of rolling backups to keep incremental_backup_freq=3 # Adjust the script execution frequency for incremental backups max_incremental_backups=4 # Adjust the number of incremental backups to keep permanent_backup_freq=12 # Adjust the script execution frequency for permanent backups # Define excluded directory basenames - e.g. excluded_directories=("mariadb" "jackett" "plex") # If undefined or an empty list, no folders will be excluded. excluded_directories= # Define excluded filename patterns - e.g., excluded_file_patterns=("pattern1" "pattern2") # If undefined or an empty list, no filenames will be excluded. # E.g. Exclude all files with '.tmp' extension and exclude files # starting with 'backup_' and ending with '.bak': # excluded_file_patterns=("*.tmp" "backup_.*\.bak") excluded_file_patterns= ########################################################## # Check if required command-line tools are installed command -v rsync >/dev/null 2>&1 || { echo "Error: rsync is not installed. Install it and try again." exit 1 } # Ensure the appdata folder exists if [ ! -d "${appdata_folder}" ]; then echo "Error: AppData folder does not exist." exit 1 fi # Path declaration incremental_folder="${backup_parent}incremental/" permanent_folder="${backup_parent}permanent/" # Create appdata backup folder backup_folder="${backup_parent}appdata_$(date +'%y%m%d')" counter=1 # Create if needed and Give write permission to the backup parent directory mkdir -p "${backup_parent}" || { echo "Error: Failed to create backup parent directory." exit 1 } chmod +w "${backup_parent}" # Manage rolling backups cd "${backup_parent}" || exit mapfile -t backups < <(find . -maxdepth 1 -type d -regex './appdata_[0-9]\{6\}\(_[0-9]\+\)?' -printf '%T@ %p\n' | sort -nr | awk 'NR>1 {print $2}') for ((i = max_rolling_backups; i < ${#backups[@]}; i++)); do rm -rf "${backups[i]}" echo -e "\nRemoved the rolling backup:\n${backups[i]}" done # Add a time suffix if the folder already exists while [ -e "${backup_folder}" ]; do backup_name=appdata_$(date +'%y%m%d_%H%M') backup_folder="${backup_parent}${backup_name}" done # Check if the mkdir command was successful mkdir -p "${backup_folder}" || { echo "Error: Failed to create backup folder." exit 1 } # Create Rolling Backup and exclude directories and filenames based on regex patterns if [ -d "${backup_folder}" ]; then # Prepare --exclude options for each directory exclude_dir_options=() for dir in "${excluded_directories[@]}"; do exclude_dir_options+=("--exclude=$dir") done # Prepare --exclude options for each regex pattern exclude_pattern_options=() for pattern in "${excluded_file_patterns[@]}"; do exclude_pattern_options+=("--exclude=$pattern") done rsync -a --link-dest="${backup_parent}$(ls -t "${backup_parent}" | grep -E 'appdata_[0-9]{6}' | grep -Eo 'appdata_[0-9]{6}_?[0-9]*' | head -n 1)" \ "${appdata_folder}" "${exclude_dir_options[@]}" "${exclude_pattern_options[@]}" "${backup_folder}/" else rsync -a "${appdata_folder}" "${exclude_dir_options[@]}" "${exclude_pattern_options[@]}" "${backup_folder}/" fi echo -e "\nCreated rolling backup:\n${backup_folder}" # Counter to check if it's time for incremental or permanent backups if [ ! -f "${backup_parent}.counter" ]; then echo 1 >"${backup_parent}.counter" else counter=$(<"${backup_parent}.counter") fi ## Manage incremental backups if [ $((incremental_backup_freq > 0)) -eq 1 ] && [ $((counter % incremental_backup_freq)) -eq 0 ]; then mkdir -p "${incremental_folder}" || { echo "Error: Failed to create incremental backup directory." exit 1 } # Rotate incremental backups cd "${incremental_folder}" || exit mapfile -t inc_backups < <(find . -maxdepth 1 -type d -regex './appdata_[0-9]\{6\}\(_[0-9]\+\)?' -printf '%T@ %p\n' | sort -nr | awk 'NR>1 {print $2}') for ((i = max_incremental_backups; i < ${#inc_backups[@]}; i++)); do rm -rf "${incremental_folder:?}/${inc_backups[i]:?}" echo -e "Removed the incremental backup:\n${inc_backups[i]}" done # Create incremental backup folder incremental_backup_folder="${incremental_folder}${backup_name}" # Add a time suffix if the folder already exists count=1 while [ -e "${incremental_backup_folder}" ]; do incremental_backup_folder="${incremental_folder}${backup_name}_${count}" count=$((counter + 1)) done rsync -a --link-dest="${backup_parent}$(ls -t "${backup_parent}" | grep -E 'appdata_[0-9]{6}' | grep -Eo 'appdata_[0-9]{6}_?[0-9]*' | head -n 1)" "${appdata_folder}" "${incremental_backup_folder}/" echo -e "Created incremental backup:\n${incremental_backup_folder}" fi # Manage permanent backups if [ $((permanent_backup_freq > 0)) -eq 1 ] && [ $((counter % permanent_backup_freq)) -eq 0 ]; then mkdir -p "${permanent_folder}" || { echo "Error: Failed to create permanent backup directory." exit 1 } # Create permanent backup folder permanent_backup_folder="${permanent_folder}${backup_name}" # Add a time suffix if the folder already exists just for the sake of sanity count=1 while [ -e "${permanent_backup_folder}" ]; do permanent_backup_folder="${permanent_folder}${backup_name}_${count}" count=$((counter + 1)) done rsync -a --link-dest="${backup_folder}" "${backup_folder}/" "${permanent_backup_folder}/" echo -e "Created permanent backup:\n${permanent_backup_folder}" fi # Increment the script execution counter echo $((counter + 1)) >"${backup_parent}.counter" # Display the directory structure echo -e "\nDirectory Structure:" tree --dirsfirst "${backup_parent}"