catapultam_habeo

Members
  • Posts

    7
  • Joined

  • Last visited

Recent Profile Visitors

The recent visitors block is disabled and is not being shown to other users.

catapultam_habeo's Achievements

Newbie

Newbie (1/14)

7

Reputation

  1. First post mentions docker containers. How do we apply this to containers? ie plex transcoding.
  2. You probably didn't do Step 1 in the original post. I'll check on this. It seems to be working on my local version, but admittedly I don't parse the args, I just set them in the EXCLUDE var directly.
  3. I noticed that it isn't supported. I'm asking if things could be rejiggered so that they could be supported? It looks like the reason why it doesn't work is that your init scripts rebuild the remote entry based on dns returns already - can we get a flag to disable this behavior?
  4. Hey Binhex, Is there any chance I can get support for passing in a ovpn config with multiple remote options configured? I'd like my geolocation to be randomized within a specified range on vpn start. As it stands, if I use this kind of config, all of the extra remote options are forcibly removed from the file by your startup scripts.
  5. Can we please get LDAP authentication support? AD integration is super flaky, and there seem to be pretty mature linux solutions for using LDAP to authenticate (for samba, at least)...
  6. Here is a housekeeping script. By default just deletes empty timestamp directories. Optionally, it can delete snapshots. -a purges all snapshots -i <Comma seperated list of shares> purges the selected shares. example -i Downloads,Test1,Test2 will purge all snapshots for Downloads, Test1, and Test2. #!/bin/bash shopt -s nullglob POSITIONAL=() while [[ $# -gt 0 ]] do key="$1" case $key in -i|--include) INCLUDE="$2" shift # past argument shift # past value ;; -a|--all) ALL=YES shift ;; *) # unknown option POSITIONAL+=("$1") # save it in an array for later shift # past argument ;; esac done set -- "${POSITIONAL[@]}" # restore positional parameters #Tokenize include list declare -A includes for token in ${INCLUDE//,/ }; do includes[$token]=1 done #iterate over all disks on array for disk in /mnt/disk*[0-9]* ; do #iterate over each timestamp for timestamp in ${disk}/.snapshots/* ; do #iterate over each share in the timestamp for snap in $timestamp/* ; do if [ -n "${includes[$(basename $snap)]}" ] || [ "$ALL" = "YES" ] ; then echo "Purging - $snap" btrfs subvolume delete $snap fi #check for empty timestamp if [ ! "$(ls -A $timestamp)" ] ; then echo "Purging empty directory - $timestamp" rmdir $timestamp fi done done done
  7. Hi, this is mostly a WIP thread, but as of the first post it does work up to my relatively limited testing. I plan on expanding this to a fully featured plugin, but this script is a working foundation, and I'd like to make this available to people to play with asap. Bottom Line Up Front: This script only works on your btrfs-formatted array drives. By default, it will keep 8760 snapshots (1 year of hourly snapshots), this value can be adjusted by changing the MAX_SNAPS variable in the script. This script does not handle cache drives, but would be trivial to extend to do so - I just think it is a bad idea. Detection of this is minimal but present. Running this script for the first time will remove and recreate all of your existing array shares (moved to temporary path, original path converted to subvolume, moved back), no data should be lost, but I have only tested this with my own data and configuration, and cannot account for all edge cases and I absolutely cannot be held accountable for your data if it is lost. No script is provided to revert these changes. Goals: I wanted to have delta snapshot recovery as part of my NAS feature set. FreeNAS is appealing, but I dislike FreeBSD's ecosystem (weird problems with bhyve, don't really need ZFS performance improvements), tried ProxMox and didn't care for it, didn't want to roll my own (likely debian-based) setup, very much like unRAID's GUI and asynchronous drive upgrade process, and wasn't interested in the crazy ZFS-on-unRAID frankenstein config by Wendell at L1T. I noted that unRAID can be configured to use BTRFS, and in theory it should be able to do this, given enough scripting to keep everything in sync. I also want to leverage unRAID's GUI as heavily as possible, and do as little command-line work on the regular as possible. Adding a new drive, creating a new share, etc, should all be possible through the GUI, and this setup should automagically adjust. Step 1) Adjust your Settings -> SMB -> SMB Extras field to include the following line. This will publish your snapshots to windows SMB clients as 'previous versions'. vfs objects = shadow_copy2 Notes: This config does work at the global scope (where adding it to extras puts it by default), and will apply to all of your shares. You just don't get to configure any of the other options for this feature. Unfortunately, duplicating the UnRAID team's work to build share configs on the fly is outside of my ambition, so I'm willing to live with that compromise. This is going to get us into an interesting situation, where the only place samba seems to be able to find our snapshots directory is at '/mnt/user/.snapshots'. The directory needs to be created on each storage device and then let unraid aggregate it later, so we are going to do it in the script so we can handle new drives and new shares correctly. Step 2) Install the CA Userscripts Plugin. Details of this step are outside the scope of this post. Step 3) Settings -> User Scripts. Add New Script. Click on script name to edit it. Add the following code to the script. Adjust MAX_SNAPS to your preference. Schedule it as you desire. Adjust EXCLUDE to your preference. Random Notes: This provides some, but minimal protection from ransomware and bit-rot. In particular, ransomware which understands a linux system and actually gets access to the server could purge snapshots. Edit 1/6/20: Added options to exclude some shares from being snapshotted. -e\--exclude <Comma seperated list of shortnames> #!/bin/bash #description=This script implements incremental snapshots on btrfs array drives. #arrayStarted=true #argumentDescription= -n|--number <MAXIMUM NUMBER OF SNAPSHOTS TO RETAIL> #argumentDefault=-s 8760 shopt -s nullglob #make empty directories not freak out date=$(TZ=GMT date +@GMT-%Y.%m.%d-%H.%M.%S) #standardized datestamp MAX_SNAPS=8760 EXCLUDE= is_btrfs_subvolume() { local dir=$1 [ "$(stat -f --format="%T" "$dir")" == "btrfs" ] || return 1 inode="$(stat --format="%i" "$dir")" case "$inode" in 2|256) return 0;; *) return 1;; esac } POSITIONAL=() while [[ $# -gt 0 ]] do key="$1" case $key in -n|--number) MAX_SNAPS="$2" shift # past argument shift # past value ;; -e|--exclude) EXCLUDE="$2" shift # past argument shift # past value ;; *) POSITIONAL+=("$1") # save it in an array for later shift # past argument ;; esac done set -- "${POSITIONAL[@]}" # restore positional parameters #ADJUST MAX_SNAPS to prevent off-by-1 MAX_SNAPS=$((MAX_SNAPS+1)) #Tokenize exclude list declare -A excludes for token in ${EXCLUDE//,/ }; do excludes[$token]=1 done #iterate over all disks on array for disk in /mnt/disk*[0-9]* ; do #examine disk for btrfs-formatting (MOSTLY UNTESTED) if is_btrfs_subvolume $disk ; then #check for .snapshots directory prior to generating snapshot if [ -d "$disk" ]; then if [ ! -d "$disk/.snapshots/" ] ; then mkdir -v $disk/.snapshots fi if [ ! -d "$disk/.snapshots/$date/" ] ; then mkdir -v $disk/.snapshots/$date fi fi #iterate over shares present on disk for share in ${disk}/* ; do #test for exclusion if [ ! -n "${excludes[$(basename $share)]}" ]; then #echo "Examining $share on $disk" is_btrfs_subvolume $share if [ ! "$?" -eq 0 ]; then #echo "$share is likely not a subvolume" mv -v ${share} ${share}_TEMP btrfs subvolume create $share cp -avT --reflink=always ${share}_TEMP $share rm -vrf ${share}_TEMP fi #make new snap btrfs subvolume snap -r ${share} /mnt/$(basename $disk)/.snapshots/${date}/$(basename $share) else echo "$share is on the exclusion list. Skipping..." fi done #find old snaps echo "Found $(find ${disk}/.snapshots/ -maxdepth 1 -mindepth 1 | sort -nr | tail -n +$MAX_SNAPS | wc -l) old snaps" for snap in $(find ${disk}/.snapshots/ -maxdepth 1 -mindepth 1 | sort -nr | tail -n +$MAX_SNAPS); do for share_snap in ${snap}/*; do btrfs subvolume delete $share_snap done rm -rfv $snap done fi done