Additional Scripts For User.Scripts Plugin


Recommended Posts

On 9/3/2016 at 7:06 PM, RobJ said:

 

#!/bin/bash
# A script to clear an unRAID array drive.  It first checks the drive is completely empty,
# except for a marker indicating that the user desires to clear the drive.  The marker is
# that the drive is completely empty except for a single folder named 'clear-me'.
#
# Array must be started, and drive mounted.  There's no other way to verify it's empty.
# Without knowing which file system it's formatted with, I can't mount it.
#
# Quick way to prep drive: format with ReiserFS, then add 'clear-me' folder.
#
# 1.0  first draft
# 1.1  add logging, improve comments
# 1.2  adapt for User.Scripts, extend wait to 60 seconds
# 1.3  add progress display; confirm by key (no wait) if standalone; fix logger
# 1.4  only add progress display if unRAID version >= 6.2

version="1.4"
marker="clear-me"
found=0
wait=60
p=${0%%$P}              # dirname of program
p=${p:0:18}
q="/tmp/user.scripts/"

echo -e "*** Clear an unRAID array data drive ***  v$version\n"

# Check if array is started
ls /mnt/disk[1-9]* 1>/dev/null 2>/dev/null
if [ $? -ne 0 ]
then
   echo "ERROR:  Array must be started before using this script"
   exit
fi

# Look for array drive to clear
n=0
echo -n "Checking all array data drives (may need to spin them up) ... "
if [ "$p" == "$q" ] # running in User.Scripts
then
   echo -e "\n"
   c="<font color=blue>"
   c0="</font>"
else #set color teal
   c="\x1b[36;01m"
   c0="\x1b[39;49;00m"
fi

for d in /mnt/disk[1-9]*
do
   x=`ls -A $d`
   z=`du -s $d`
   y=${z:0:1}
#   echo -e "d:"$d "x:"${x:0:20} "y:"$y "z:"$z

   # the test for marker and emptiness
   if [ "$x" == "$marker" -a "$y" == "0" ]
   then
      found=1
      break
   fi
   let n=n+1
done

#echo -e "found:"$found "d:"$d "marker:"$marker "z:"$z "n:"$n

# No drives found to clear
if [ $found == "0" ]
then
   echo -e "\rChecked $n drives, did not find an empty drive ready and marked for clearing!\n"
   echo "To use this script, the drive must be completely empty first, no files"
   echo "or folders left on it.  Then a single folder should be created on it"
   echo "with the name 'clear-me', exactly 8 characters, 7 lowercase and 1 hyphen."
   echo "This script is only for clearing unRAID data drives, in preparation for"
   echo "removing them from the array.  It does not add a Preclear signature."
   exit
fi

# check unRAID version
v1=`cat /etc/unraid-version`
# v1 is 'version="6.2.0-rc5"' (fixme if 6.10.* happens)
v2="${v1:9:1}${v1:11:1}"
if [[ $v2 -ge 62 ]]
then
   v=" status=progress"
else
   v=""
fi
#echo -e "v1=$v1  v2=$v2  v=$v\n"

# First, warn about the clearing, and give them a chance to abort
echo -e "\rFound a marked and empty drive to clear: $c Disk ${d:9} $c0 ( $d ) "
echo -e "* Disk ${d:9} will be unmounted first."
echo "* Then zeroes will be written to the entire drive."
echo "* Parity will be preserved throughout."
echo "* Clearing while updating Parity takes a VERY long time!"
echo "* The progress of the clearing will not be visible until it's done!"
echo "* When complete, Disk ${d:9} will be ready for removal from array."
echo -e "* Commands to be executed:\n***** $c umount $d $c0\n***** $c dd bs=1M if=/dev/zero of=/dev/md${d:9} $v $c0\n"
if [ "$p" == "$q" ] # running in User.Scripts
then
   echo -e "You have $wait seconds to cancel this script (click the red X, top right)\n"
   sleep $wait
else
   echo -n "Press ! to proceed. Any other key aborts, with no changes made. "
   ch=""
   read -n 1 ch
   echo -e -n "\r                                                                  \r"
   if [ "$ch" != "!" ];
   then
      exit
   fi
fi

# Perform the clearing
logger -tclear_array_drive "Clear an unRAID array data drive  v$version"
echo -e "\rUnmounting Disk ${d:9} ..."
logger -tclear_array_drive "Unmounting Disk ${d:9}  (command: umount $d ) ..."
umount $d
echo -e "Clearing   Disk ${d:9} ..."
logger -tclear_array_drive "Clearing Disk ${d:9}  (command: dd bs=1M if=/dev/zero of=/dev/md${d:9} $v ) ..."
dd bs=1M if=/dev/zero of=/dev/md${d:9} $v
#logger -tclear_array_drive "Clearing Disk ${d:9}  (command: dd bs=1M if=/dev/zero of=/dev/md${d:9} status=progress count=1000 seek=1000 ) ..."
#dd bs=1M if=/dev/zero of=/dev/md${d:9} status=progress count=1000 seek=1000

# Done
logger -tclear_array_drive "Clearing Disk ${d:9} is complete"
echo -e "\nA message saying \"error writing ... no space left\" is expected, NOT an error.\n"
echo -e "Unless errors appeared, the drive is now cleared!"
echo -e "Because the drive is now unmountable, the array should be stopped,"
echo -e "and the drive removed (or reformatted)."
exit

 

 The version check for unraid 6.2 is broken in 6.10+
 
Here is an alternative check:
function version { echo "$@" | awk -F '[.]|-rc' '{ printf("%d%03d%03d%03d\n", $1,$2,$3,$4); }'; }

# check unRAID version
v1=`cat /etc/unraid-version | awk -F= '{ print $2 }' | tr -d '"'`

if [[ $(version $v1) -ge $(version "6.2") ]]
then
   v=" status=progress"
else
   v=""
fi

 

Edited by veri745
fix quote, typo
Link to comment

For me it's always worked at normal array write speed. You can see the write speed on the main page and estimate from that. If your parity is SMR it might take a while (but so would rebuilding parity).

Edited by Kilrah
Link to comment
2 hours ago, Kilrah said:

For me it's always worked at normal array write speed.

If the disk slot shows free and used space in Unraid's GUI, it takes forever, with write speeds of a few hundred k. If the GUI shows unmountable, the dd runs at full speed. Every time I umount the disk with the array running, the GUI still has something showing up in that slot. If I stop the array and start it again, it shows unmountable and runs fine.

 

The script depends on being able to read the disk to find the text, so by definition it's mountable at the start of the script.

 

If you umount a disk, does the GUI immediately show unmountable? Mine doesn't.

Link to comment
  • 2 weeks later...

Has anyone tried to make a script that consolidated files that were scattered across different disks? For example, I had tv shows, music and movies in a "media" share (Media/TV/Shows/Show_name/Season_X/file, Media/TV/Anime/Anime_name/Season_X/file, Media/Movies/Movie_name/file.

 

My split level kept all the episodes of a season together on a single disk. My issue was the movie subtitles were being downloaded and split onto whatever disk had the space, sometimes having more than one drive spin up to access the movie and the subtitles.

 

Tried using ChatGPT to help with a script and this is what it came up with:

 

#!/bin/bash

# Set the path to your movie directory
MOVIE_DIR="/mnt/user/Movies"

# Find all movie files and loop through them
find "$MOVIE_DIR" -type f \( -iname "*.mkv" -o -iname "*.mp4" -o -iname "*.avi" \) -print0 | while IFS= read -r -d '' movie; do
    movie_dir=$(dirname "$movie")
    movie_name=$(basename "$movie")
    movie_name="${movie_name%.*}"

    # Find the corresponding .srt file
    srt_file=$(find "$movie_dir" -type f -iname "$movie_name.srt" -print -quit)

    # Move the .srt file to the same directory as the movie file
    if [ -n "$srt_file" ]; then
        mv "$srt_file" "$movie_dir"
        echo "Moved $srt_file to $movie_dir"
    fi
done

 

 

Link to comment
  • 1 month later...
On 10/29/2016 at 6:56 PM, Squid said:

Run A Custom Script At Parity Check / Rebuild Start And Stop

Use it to run a custom script to (as an example), shut down various docker applications, etc.  Adjust the variables within the script file.

 

Note that you either need to run this in the background or at array start.  Running this in the foreground will not work.

 

 

#!/usr/bin/php
<?PHP
# A simple script to allow you to run a custom script when a parity check starts or stops
# Adjust the following variables to suit:

$checkInterval = 300;                             # Number of seconds in between checks
$startScript   = "full path to the script";       # The full path to the script to run when a parity check starts
$stopScript    = "full path to the stop script";  # The full path to the script to run when a parity check stops

# Don't touch anything below

while (true) {
  $vars = parse_ini_file("/var/local/emhttp/var.ini");
  if ( $vars['mdState'] == "STOPPED" ) {
    break;
  }
  if ( ($vars['mdResyncPos'] != 0) && $vars ) {
    echo "Parity Check / Sync / Rebuild in progress.  Executing the start script ($startScript)";
    exec($startScript,$output);
    foreach ($output as $line) {
      echo $line."\n";
    }
    while (true) {
      $vars = parse_ini_file("var/local/emhttp/var.ini");
      if ( ($vars['mdResyncPos'] == 0) && $vars ) {
        echo "Parity Check / Sync / Rebuild finished.  Executing the stop script ($stopScript)";
        exec($stopScript,$output);
        foreach ($output as $line) {
          echo $line."\n";
        }
        break;
      } else {
        sleep($checkInterval);
      }
    }
  } else { 
    sleep($checkInterval);
  }  
}
?>
 

 

custom_script_parity_check_start_stop.zip 1010 B · 58 downloads

 

So another update on this one..
Thanks for the script! Has been working fine for me, after i added the missing / in /var/local.... and i also had to add "bash" infront of my path to script, since it didn't get permission.

But, since 6.12.3, the script dosen't work anymore...
I know it's an old script, but do you maybe know why it dosen't work anymore? I have tried searching my self, but i can't find anything...

 

Thanks!

Link to comment
On 4/13/2022 at 5:26 AM, Bolagnaise said:

Just a quick update for anyone looking to implement autoheal in docker, i have found that adding -i to the curl command correctly return the 2OO OK http status for the /identify page, the http return is much shorter so i recommend it. 

 

Heres my advanced docker config for my plex server to add the health cmd too. Many thanks again to @Meles Meles

 

--health-cmd 'curl -i --connect-timeout 15 --silent --show-error --fail 192.168.1.100:32400/identify'

 

Nvm, figured out how to keep it up and running.....how but how do you know if it's running correctly, since the log doesnt stay open on it...Guess just wait and see if plex crashes, or more like doesn't crash again.....

Edited by coltonc18
Link to comment
  • 3 weeks later...

So, I tried using the clear script to shrink my array. It found the correct disk and started clearing that. But it then runs the command:
dd bs=1M if=/dev/zero of=/dev/md3 status=progress
Which is correct, as I want to clear disk 3.

However, clearing stops with an "No space left on device" after only 8.4MB. The disk is 3TB

Running it manually in the console indeed shows the same:

 

root@Tower:~# dd bs=1M if=/dev/zero of=/dev/md3 status=progress
dd: error writing '/dev/md3': No space left on device
9+0 records in
8+0 records out
8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.00286115 s, 2.9 GB/s

Right  now, I have a disk of only 15GB according to unraid, which is normal after using this script, as I did use it before. Right now I can only try to fix this, or just  remove it and let it rebuild parity right? Parity wouldn't be valid by removing this disk.

Strange thing is, I only see a /dev/md3, no /dev/md1 (disk 1). But I do have /dev/md1pl and /dev/md3pl

Running on unraid 6.12.3 btw.

Link to comment
  • 1 month later...
  • 2 months later...
On 9/21/2023 at 8:15 AM, isvein said:

Used Notepad++ to find and replace all 5 instances of "md${d:9}" with "md${d:9}p1" in the script and it looks to work just fine now on 6.12.4 :)
(just need to wait and see when it finishes to make sure it actually worked)

Did this work for you?

I recently upgraded to 6.12.4 and after running the original script, it would finish running almost instantly. After making the changes you suggested, it is now running fine. I'll check in the morning and if all goes well, I'll make an update here.

Link to comment
7 hours ago, Daburgers said:

Did this work for you?

I recently upgraded to 6.12.4 and after running the original script, it would finish running almost instantly. After making the changes you suggested, it is now running fine. I'll check in the morning and if all goes well, I'll make an update here.

I remember I did the change and it did work, but I have not had use for that script for a long time now so dont know :(

Edited by isvein
Link to comment
1 hour ago, isvein said:

I remember I did the change and it did work, but I have not had use for that script for a long time now so dont know :(

All good,

I have just checked it and it looks like it worked!

 

Only issue is I am trying to stop the array and its stuck "Array Stopping - Retry unmounting disk shares". Even though I dockers and VM's disabled


Update:

Ok so the script ran fine using those changes in the script: replace all 5 instances of "md${d:9}" with "md${d:9}p1"

When I went to stop the array, it just hung there saying "Array Stopping - Retry unmounting disk shares". I read some forums about this issue on 6.12.# and they suggested the below which worked for some on 6.12.3:  
image.png.d61de8f1c19c28f40f81fe13bf9da7ce.png
For me, I didn't have loop2 on my list which meant it was not running. Those on 6.12.4 also seemed to not have success with the above.


Someone else suggested to run:
image.png.431ca73e5c0ffb240433602e85a46b04.png

This also didn't work for me. it just said it was "not mounted" which was correct.

The Fix for me:
In the end I just shutdown the array and powered it back on. The array was stopped already.

I then went to Tools > new config > preserved all current assignments > then i was able to unassign the drive and start the array again. Everything seems to be in place as well. happy days!

Edited by Daburgers
update on fix
Link to comment

I'm trying to just use the commands and sometimes after I run the umount command, and run the dd command, the progress will run at ~400kb/s with no way to stop it. I end up having to reboot.

 

Has anyone run into this before and found a better way to unmount reliably? I found the same issue occurred with the script.

 

umount /mnt/disk8

 

dd bs=1M if=/dev/zero of=/dev/md8 status=progress

Link to comment
On 12/13/2023 at 8:03 PM, jkexbx said:

I'm trying to just use the commands and sometimes after I run the umount command, and run the dd command, the progress will run at ~400kb/s with no way to stop it. I end up having to reboot.

 

Has anyone run into this before and found a better way to unmount reliably? I found the same issue occurred with the script.

 

umount /mnt/disk8

 

dd bs=1M if=/dev/zero of=/dev/md8 status=progress

 

Did you ever figure out how to unmount reliably?  i was able to run the script, but when i tried to stop the array, it hung up on unmounting.  Ended up having to do a non-clean shutdown and had to run a 38-hr parity check.  I need to remove 2 more drives from my array and want it to be smoother next time

 

Edit:  this terminal command at the end of this post allowed me to unmount my disk.  Hope it helps someone

 

 

Edited by omartian
more info
Link to comment
  • 2 weeks later...

Just thought I would share my appdata backup script that I have been using for a couple of years now. It can be modified for any other accessible directory. I have used the Appdata backup plugin on community apps as well and I do highly recommend it. The way my brain works and even though less feature filled, I decided to just write a script to do exactly what I wanted and streamlined for my particular workflow.  Hopefully this is the place to share and I do so in the off-chance someone may find some use for it or inspiration for their own scripts. 

How to use the script is given in the code comments. In short, it is designed to simplify and automate rolling, incremental, and permanent backups of the appdata folder. With customisable settings like the number of rolling backups to retain, the frequency of incremental backups, and intervals for permanent backups. The script automatically excludes user specified directories and filenames using regex patterns.

 

Note that the script uses hard-linking to reduce storage utilisation, something I wanted that wasn't available in the Appdata backup plugin.  

 

Please use your due diligence and run a test case for your needs to see if it works. I have included cursory error checks, but nothing exhaustive. 

 

#!/bin/bash
######################################################################
#                       MyAppDataBackup v1
#                ·•˚°○.●|HyperWorx - 2023|●.○°˚•·
#
# Description
# ------------
#  This script automates rolling, incremental, and permanent backups
#  for the specified appdata folder. It keeps a user-specified number
#  of rolling backups, performs incremental rolling backups and
#  permanent backups at defined intervals.
#
# Usage for automated backups:
#  Modify the user-defined variables to configure the script behavior.
#  Ensure the excluded_directories array includes directories to exclude
#  from the backup.
#  Run this script using the user scripts plugin or as a cron job.
#  Ensure proper execution permissions for the script.
#
# Explanation of User-defined Variables
# --------------------------------------
# appdata_folder - Path to the source AppData folder to be backed up.
# backup_parent - Parent directory for storing backups.
# max_rolling_backups - The most recent number of rolling backups to retain.
# incremental_backup_freq - Frequency of script execution for incremental backups.
#                           Set to 0 for not utilising this functionality.
# max_incremental_backups - The most recent number of incremental backups to retain.
# permanent_backup_freq - Frequency of script execution for permanent backups.
#                         Set to 0 for not utilising this functionality.
# excluded_directories - List of directories to exclude from the backup.
#                        If undefined, all appdata folders will be included.
# excluded_file_patterns - List of regex compatible patterns to match filenames for
#                          excluding from the backup. If undefined, all files included.
#
# Default Settings Example:
# -------------------------_
#   Executing the script every month and keeping 2 rolling backups (the last 2 months),
#   keep incremental backups at every 3 script executions (or 3 months) and storing the 4 of
#   last 4 most recent incremental backups, and keep a permanent backup snapshotted every
#   12 script executions (or every 12 months).
#
# NOTE:
# IT IS USER'S RESPONSIBILITY TO ENSURE SCRIPT WORKS FOR THEIR INTENDED USE CASE.
######################################################################

################ USER DEFINED VARIABLES ###################
appdata_folder="/mnt/user/appdata/"
backup_parent="/mnt/user/backups/unraid/MyAppDataBackup/"
max_rolling_backups=2     # Adjust the number of rolling backups to keep
incremental_backup_freq=3 # Adjust the script execution frequency for incremental backups
max_incremental_backups=4 # Adjust the number of incremental backups to keep
permanent_backup_freq=12  # Adjust the script execution frequency for permanent backups

# Define excluded directory basenames - e.g. excluded_directories=("mariadb" "jackett" "plex")
# If undefined or an empty list, no folders will be excluded.
excluded_directories=

# Define excluded filename patterns - e.g., excluded_file_patterns=("pattern1" "pattern2")
# If undefined or an empty list, no filenames will be excluded.
# E.g. Exclude all files with '.tmp' extension and exclude files
#      starting with 'backup_' and ending with '.bak':
#      excluded_file_patterns=("*.tmp" "backup_.*\.bak")
excluded_file_patterns=

##########################################################

# Check if required command-line tools are installed
command -v rsync >/dev/null 2>&1 || {
  echo "Error: rsync is not installed. Install it and try again."
  exit 1
}

# Ensure the appdata folder exists
if [ ! -d "${appdata_folder}" ]; then
  echo "Error: AppData folder does not exist."
  exit 1
fi

# Path declaration
incremental_folder="${backup_parent}incremental/"
permanent_folder="${backup_parent}permanent/"

# Create appdata backup folder
backup_folder="${backup_parent}appdata_$(date +'%y%m%d')"
counter=1

# Create if needed and Give write permission to the backup parent directory
mkdir -p "${backup_parent}" || {
  echo "Error: Failed to create backup parent directory."
  exit 1
}

chmod +w "${backup_parent}"

# Manage rolling backups
cd "${backup_parent}" || exit
mapfile -t backups < <(find . -maxdepth 1 -type d -regex './appdata_[0-9]\{6\}\(_[0-9]\+\)?' -printf '%T@ %p\n' | sort -nr | awk 'NR>1 {print $2}')
for ((i = max_rolling_backups; i < ${#backups[@]}; i++)); do
  rm -rf "${backups[i]}"
  echo -e "\nRemoved the rolling backup:\n${backups[i]}"
done

# Add a time suffix if the folder already exists
while [ -e "${backup_folder}" ]; do
  backup_name=appdata_$(date +'%y%m%d_%H%M')
  backup_folder="${backup_parent}${backup_name}"
done

# Check if the mkdir command was successful
mkdir -p "${backup_folder}" || {
  echo "Error: Failed to create backup folder."
  exit 1
}

# Create Rolling Backup and exclude directories and filenames based on regex patterns
if [ -d "${backup_folder}" ]; then
  # Prepare --exclude options for each directory
  exclude_dir_options=()
  for dir in "${excluded_directories[@]}"; do
    exclude_dir_options+=("--exclude=$dir")
  done

  # Prepare --exclude options for each regex pattern
  exclude_pattern_options=()
  for pattern in "${excluded_file_patterns[@]}"; do
    exclude_pattern_options+=("--exclude=$pattern")
  done

  rsync -a --link-dest="${backup_parent}$(ls -t "${backup_parent}" | grep -E 'appdata_[0-9]{6}' | grep -Eo 'appdata_[0-9]{6}_?[0-9]*' | head -n 1)" \
    "${appdata_folder}" "${exclude_dir_options[@]}" "${exclude_pattern_options[@]}" "${backup_folder}/"
else
  rsync -a "${appdata_folder}" "${exclude_dir_options[@]}" "${exclude_pattern_options[@]}" "${backup_folder}/"
fi

echo -e "\nCreated rolling backup:\n${backup_folder}"

# Counter to check if it's time for incremental or permanent backups
if [ ! -f "${backup_parent}.counter" ]; then
  echo 1 >"${backup_parent}.counter"
else
  counter=$(<"${backup_parent}.counter")
fi

## Manage incremental backups
if [ $((incremental_backup_freq > 0)) -eq 1 ] && [ $((counter % incremental_backup_freq)) -eq 0 ]; then

  mkdir -p "${incremental_folder}" || {
    echo "Error: Failed to create incremental backup directory."
    exit 1
  }

  # Rotate incremental backups
  cd "${incremental_folder}" || exit
  mapfile -t inc_backups < <(find . -maxdepth 1 -type d -regex './appdata_[0-9]\{6\}\(_[0-9]\+\)?' -printf '%T@ %p\n' | sort -nr | awk 'NR>1 {print $2}')
  for ((i = max_incremental_backups; i < ${#inc_backups[@]}; i++)); do
    rm -rf "${incremental_folder:?}/${inc_backups[i]:?}"
    echo -e "Removed the incremental backup:\n${inc_backups[i]}"
  done

  # Create incremental backup folder
  incremental_backup_folder="${incremental_folder}${backup_name}"

  # Add a time suffix if the folder already exists
  count=1
  while [ -e "${incremental_backup_folder}" ]; do
    incremental_backup_folder="${incremental_folder}${backup_name}_${count}"
    count=$((counter + 1))
  done

  rsync -a --link-dest="${backup_parent}$(ls -t "${backup_parent}" | grep -E 'appdata_[0-9]{6}' | grep -Eo 'appdata_[0-9]{6}_?[0-9]*' | head -n 1)" "${appdata_folder}" "${incremental_backup_folder}/"
  echo -e "Created incremental backup:\n${incremental_backup_folder}"
fi

# Manage permanent backups
if [ $((permanent_backup_freq > 0)) -eq 1 ] && [ $((counter % permanent_backup_freq)) -eq 0 ]; then

  mkdir -p "${permanent_folder}" || {
    echo "Error: Failed to create permanent backup directory."
    exit 1
  }

  # Create permanent backup folder
  permanent_backup_folder="${permanent_folder}${backup_name}"

  # Add a time suffix if the folder already exists just for the sake of sanity
  count=1
  while [ -e "${permanent_backup_folder}" ]; do
    permanent_backup_folder="${permanent_folder}${backup_name}_${count}"
    count=$((counter + 1))
  done

  rsync -a --link-dest="${backup_folder}" "${backup_folder}/" "${permanent_backup_folder}/"
  echo -e "Created permanent backup:\n${permanent_backup_folder}"
fi

# Increment the script execution counter
echo $((counter + 1)) >"${backup_parent}.counter"

# Display the directory structure
echo -e "\nDirectory Structure:"
tree --dirsfirst "${backup_parent}"


 

Edited by HyperWorx
Info
  • Upvote 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.