Additional Scripts For User.Scripts Plugin

[Squid]

NOTE: I do not have time to keep the table of contents up to date, so there are going to be other scripts within this thread that are not listed here.

 

 

Just a thread to contain any/all additional scripts created by users for use within the user.scripts plugin.  I'm going to be using this thread for anything that pops into my head that may be of use but is either too simple for a plugin format, or just not worth the time for something that may only get run once.

 

Ideal format to post any contributed scripts would be a zip file containing the script and description (stored within an already named folder for ease of adding to the plugin, and additionally a code block of the script itself for complete openness.

 

See the user.scripts thread for details on how to add these scripts (or any others)

 

 

Default Scripts Included in the plugin

Fix Files Stored on the Array for cache-only shares and the reverse

Clean Docker Logs

Backup MySQL Folder

Run mover at a certain utilization automatically

Record Disk Assignments

Enable / Disable Turbo Write Mode

Auto set turbo mode based on drives spun up

Run Mover At A Threshhold, optional to skip moving if parity check in progress

Clear An unRaid Data Drive

A script to have a file with the folders containing movies and tvshows.

Send Server Status To Phone

Backup vm xml files and ovmf nvram files

Automatically download from repo and install custom VM icons to vm manager

Run A Custom Script At Parity Check / Rebuild Start And Stop

Catalog Drive Contents

Move a folder when disk utilization exceeded

Very simple script which will resume paused/suspended vms or start shut off vms

Scheduled Scrubs

Scheduled checks for Out Of Memory Errors

Play PacMan On Your Server

USB Hotplug for Virtual Machines with no passthrough and a revision HERE

Enable / Disable Nested VM

https://forums.unraid.net/topic/48707-additional-scripts-for-userscripts-plugin/?page=4#comment-547492

RemoveSpacesFromFile

FolderfromFilename

Automatically save syslog onto flash drive

Check Plugin Integrity

Allow unRaid to utilize the full width of the browser instead of limited to 1920px

Get size of running containers

Script to spin up all drives at certain times of day

unRaid GUI Bleeding Edge Toolkit

Enable Hardware Decoding In Plex

Convert files from dos to linux format

 

[Squid]

reserved

 

[Squid]

This zip file contains the default user scripts included with the plugin, should you have deleted them from the flash drive and need to get them back

 

delete.DS_Store Files

#!/bin/bash
echo "Searching for (and deleting) .DS_Store Files"
echo "This may take a awhile"
find /mnt/user -maxdepth 9999 -noleaf -type f -name ".DS_Store" -exec rm "{}" \;

Delete Dangling docker Images

#!/bin/bash

docker rmi $(docker images --quiet --filter "dangling=true")

echo Finished
echo if an error shows above, no dangling images were found to delete

 

View Docker Log Size

du -ah /var/lib/docker/containers/ | grep -v "/$" | sort -rh | head -60 | grep .log

 

User.Scripts_Included_Scripts.zip

[Squid]

Move Cache Only Share and Move Array Only Shares

 

These two scripts will analyse your shares and move any files found on the array (for cache-only shares) to the cache drive, and its reverse script to move any files found on the cache drive to the array (for array-only shares)

 

These two scripts require user.scripts version 2016.07.16+ to operate

 

Move Cache Only Shares

#!/usr/bin/php
<?PHP
function getRsyncReturnValue($returnValue) {
  $returnMessage[0] = "Success";
  $returnMessage[1] = "Syntax or usage error";
  $returnMessage[2] = "Protocol incompatibility";
  $returnMessage[3] = "Errors selecting input/output files, dirs";
  $returnMessage[4] = "Requested action not supported: an attempt was made to manipulate 64-bit files on a platform that cannot support them; or an option was specified that is supported by the client and not by the server.";
  $returnMessage[5] = "Error starting client-server protocol";
  $returnMessage[6] = "Daemon unable to append to log-file";
  $returnMessage[10] = "Error in socket I/O";
  $returnMessage[11] = "Error in file I/O";
  $returnMessage[12] = "Error in rsync protocol data stream";
  $returnMessage[13] = "Errors with program diagnostics";
  $returnMessage[14] = "Error in IPC code";
  $returnMessage[20] = "Received SIGUSR1 or SIGINT";
  $returnMessage[21] = "Some error returned by waitpid()";
  $returnMessage[22] = "Error allocating core memory buffers";
  $returnMessage[23] = "Partial transfer due to error";
  $returnMessage[24] = "Partial transfer due to vanished source files";
  $returnMessage[25] = "The --max-delete limit stopped deletions";
  $returnMessage[30] = "Timeout in data send/receive";
  $returnMessage[35] = "Timeout waiting for daemon connection";
  
  $return = $returnMessage[$returnValue];
  if ( ! $return ) {
    $return = "Unknown Error";
  }
  return $return;
}


$shareList = @array_diff(@scandir("/boot/config/shares"),array(".",".."));

if ( ! is_dir("/mnt/user") ) {
  echo "Array Must Be Started And User Shares Must Be Enabled To Utilize This Script.  Exiting.\n";
  return;
}

if ( ! is_dir("/mnt/cache") ) {
  echo "Cache Drive Not Installed.  Exiting\n";
  return;
}

if ( ! $shareList ) {
  $shareList = array();
}

foreach ( $shareList as $shareCfg ) {
  $config = @parse_ini_file("/boot/config/shares/$shareCfg");
  if ( ! $config ) {
    continue;
  }
  if ( $config['shareUseCache'] != "only" ) {
    continue;
  }
  $share = pathinfo($shareCfg,PATHINFO_FILENAME);
  if ( is_dir("/mnt/user0/$share") ) {
    echo "Share $share set to be cache-only, but files exist within array.  Moving the files.  Depending upon the size and number of files, this may take a bit\n";
    $source = escapeshellarg("/mnt/user0/$share/");
    $destination = escapeshellarg("/mnt/cache/$share");
    $flag = true;
    exec("rsync -avXHq --remove-source-files $source $destination",$output,$returnValue);
    if ( $returnValue ) {
      echo "Errors Occurred: ".getRsyncReturnValue($returnValue)."\n";
      $errors = true;
    } else {
      exec("rm -rf $source");
    }
  }
}
if ( ! $flag ) {
  echo "All cache-only shares already only existed on the cache drive\n";
} else {
  if ( ! $errors ) {
    echo "All shares set to be cache-only have now had their files previously existing on the array to now be on the cache drive\n";
  } else {
    echo "Errors Occurred In The Copying\n";
  }
}
?>

 

Move Array Only Shares

#!/usr/bin/php
<?PHP
function getRsyncReturnValue($returnValue) {
  $returnMessage[0] = "Success";
  $returnMessage[1] = "Syntax or usage error";
  $returnMessage[2] = "Protocol incompatibility";
  $returnMessage[3] = "Errors selecting input/output files, dirs";
  $returnMessage[4] = "Requested action not supported: an attempt was made to manipulate 64-bit files on a platform that cannot support them; or an option was specified that is supported by the client and not by the server.";
  $returnMessage[5] = "Error starting client-server protocol";
  $returnMessage[6] = "Daemon unable to append to log-file";
  $returnMessage[10] = "Error in socket I/O";
  $returnMessage[11] = "Error in file I/O";
  $returnMessage[12] = "Error in rsync protocol data stream";
  $returnMessage[13] = "Errors with program diagnostics";
  $returnMessage[14] = "Error in IPC code";
  $returnMessage[20] = "Received SIGUSR1 or SIGINT";
  $returnMessage[21] = "Some error returned by waitpid()";
  $returnMessage[22] = "Error allocating core memory buffers";
  $returnMessage[23] = "Partial transfer due to error";
  $returnMessage[24] = "Partial transfer due to vanished source files";
  $returnMessage[25] = "The --max-delete limit stopped deletions";
  $returnMessage[30] = "Timeout in data send/receive";
  $returnMessage[35] = "Timeout waiting for daemon connection";
  
  $return = $returnMessage[$returnValue];
  if ( ! $return ) {
    $return = "Unknown Error";
  }
  return $return;
}


$shareList = @array_diff(@scandir("/boot/config/shares"),array(".",".."));

if ( ! is_dir("/mnt/user") ) {
  echo "Array Must Be Started And User Shares Must Be Enabled To Utilize This Script.  Exiting.\n";
  return;
}

if ( ! is_dir("/mnt/cache") ) {
  echo "Cache Drive Not Installed.  Exiting\n";
  return;
}

if ( ! $shareList ) {
  $shareList = array();
}

foreach ( $shareList as $shareCfg ) {
  $config = @parse_ini_file("/boot/config/shares/$shareCfg");
  if ( ! $config ) {
    continue;
  }
  if ( $config['shareUseCache'] != "no" ) {
    continue;
  }
  $share = pathinfo($shareCfg,PATHINFO_FILENAME);
  if ( is_dir("/mnt/cache/$share") ) {
    echo "Share $share set to be array-only, but files exist on the cache drive.  Moving the files.  Depending upon the size and number of files, this may take a bit\n";
    $source = escapeshellarg("/mnt/cache/$share/");
    $destination = escapeshellarg("/mnt/user0/$share");
    $flag = true;
    exec("rsync -avXHq --remove-source-files $source $destination",$output,$returnValue);
    if ( $returnValue ) {
      echo "Errors Occurred: ".getRsyncReturnValue($returnValue)."\n";
      $errors = true;
    } else {
      exec("rm -rf $source");
    }
  }
}
if ( ! $flag ) {
  echo "All cache-only shares already only existed on the cache drive\n";
} else {
  if ( ! $errors ) {
    echo "All shares set to be cache-only have now had their files previously existing on the array to now be on the cache drive\n";
  } else {
    echo "Errors Occurred In The Copying\n";
  }
}
?>

Move_Shares.zip

[hernandito]

(Moved from original script thread.)

 

My mini-contribution:

 

CleanDockerLogSize script.

This will display the log sizes, clean them, and then displays the result after the cleaning.

 

description

Clean-up the Docker Log Sizes on your system.

 

script

echo ""
echo "<font color='red'><b>Before:</b></font>"
echo "====================================================================================================================================================================================="
du -ah /var/lib/docker/containers/ | grep -v "/$" | sort -rh | head -60 | grep .log
echo "====================================================================================================================================================================================="
echo "Cleaning Logs:"
logs=$(find /var/lib/docker/containers/ -name '*.log');for log in $logs; do cat /dev/null > $log;done
sleep 6
echo "...<font color='blue'>cleaning complete!</font>"

echo ""
echo "<font color='green'><b>After:</b></font>"
echo "====================================================================================================================================================================================="
du -ah /var/lib/docker/containers/ | grep -v "/$" | sort -rh | head -60 | grep .log
echo ""

 

Please understand guys, I am not an expert.

[hernandito]

Backup MySQL Database Folder

 

This will backup your MySQL folder. Please edit the script below to make sure your paths are correct.

 

This script is fairly basic and is more geared towards beginers than experts. You can in essence, use it as a basis to backup any folder(s) you want. For example, I use it to backup my appdata folder (stored in cache drive) to a drive on my array. This way if my cache drive fails, I have all the Docker stuff backed up.

 

 

description

Backup <b><font color='green'>MySQL</b></font> folder.

 

script

#!/bin/bash
source="/mnt/disk8/mysql4/"
destination="/mnt/disk4/mysql-backup"

echo "<div style=' width: 40%; -webkit-border-radius: 8px 8px 0 0; border-radius: 8px 8px 0 0; border: solid 1px #cccccc; background-color: #ffe88a; padding-left: 10px;'><br><b><font color='black' size='2'>Backing-Up MySQL Folder </font><b><br>  </div>"
#echo "======================="


#echo "Backing-up from:" 
#echo "  <b><font color='blue'>"$source "</b></font>"
#echo "to:"
#echo "  <b><font color='blue'>"$destination </b></font>"


echo "<div style='width: 40%; -webkit-border-radius:  0 0 8px 8px; border-radius: 0 0 8px 8px; background-color: #ebebeb; margin-top: -14px; padding-left: 10px; padding-top: 6px;border: solid 1px #cccccc; '>Backing-up from: <br><b><font color='blue' size='2'>"$source "</b></font> <br>to:<br><b><font color='blue' size='2'>"$destination "</b></font><br> </div>"
date >/var/log/cache_backup.log
/usr/bin/rsync -avrtH --delete $source $destination >>/var/log/mysql_backup.log
echo ""
echo "<div style='padding-left: 10px; margin-top: -14px; '><font color='green' size='4'><b>Done!</b></font></div>"

 

Fun stuff.

[Squid]

Run mover at a certain threshold of cache drive utilization.

 

Adjust the value to move at within the script.  Really only makes sense to use this script as a scheduled operation, and would have to be set to a frequency (hourly?) more often than how often mover itself runs normally.

 

#!/usr/bin/php
<?PHP

$moveAt = 70;    # Adjust this value to suit.

$diskTotal = disk_total_space("/mnt/cache");
$diskFree = disk_free_space("/mnt/cache");
$percent = ($diskTotal - $diskFree) / $diskTotal * 100;

if ( $percent > $moveAt ) {
  exec("/usr/local/sbin/mover");
}
?>

run_mover_at_threshold.zip

[Squid]

Record Disk Assignments

 

Records your current disk assignments to a file on the flash drive called DISK_ASSIGNMENTS.txt (config folder).  Not necessary if you run CA's appdata backup as that's done automatically

 

#!/usr/bin/php
<?PHP
  $availableDisks = parse_ini_file("/var/local/emhttp/disks.ini",true);
  $txt .= "Disk Assignments as of ".date(DATE_RSS)."\r\n";
  foreach ($availableDisks as $Disk) {
    $txt .= "Disk: ".$Disk['name']."  Device: ".$Disk['id']."  Status: ".$Disk['status']."\r\n";
  }
  file_put_contents("/boot/config/DISK_ASSIGNMENTS.txt",$txt);
  echo "Disk assignments have been saved to the flashdrive (config/DISK_ASSIGNMENTS.txt)\n";
?>

record_disk_assignments.zip

[Squid]

Enable / Disable Turbo Write Mode

 

Enable

#!/bin/bash
/usr/local/sbin/mdcmd set md_write_method 1
echo "Turbo write mode now enabled"

Disable

#!/bin/bash
/usr/local/sbin/mdcmd set md_write_method 0
echo "Turbo write mode now disabled"

turbo_writes.zip

[Squid]

Auto set turbo write mode

 

Note that "auto" setting in disk settings (6.2) does NOT operate like this.  According to the help text, auto mode always turns off turbo mode.

 

Adjusts the turbo write setting depending upon the number of drives spun down.  Adjustable within the script for the number of drives.  Only makes sense to really run this at a frequency of hourly (or the forth coming custom frequency)

 

#!/usr/bin/php
<?PHP

$spinDownAllowed = 0;   # Set this to the maximum number of drives allowed to be spundown in order to enable turbo write mode

function startsWith($haystack, $needle) {
  return $needle === "" || strripos($haystack, $needle, -strlen($haystack)) !== FALSE;
}

$disks = parse_ini_file("/var/local/emhttp/disks.ini",true);
foreach ($disks as $disk) {
  if ( startsWith($disk['color'],"grey") ) {
    continue;
  }
  if ( startsWith($disk['name'],"cache") ) {
    continue;
  }
  if ( ! strpos($disk['color'],"on") ) {
    ++$spunDown;
  }
}

if ( $spinDownAllowed >= $spunDown ) {
  exec("/usr/local/sbin/mdcmd set md_write_method 1");
} else {
  exec("/usr/local/sbin/mdcmd set md_write_method 0");
}

?>

auto_turbo_write.zip

[Squid]

A slightly enhanced version of the run mover at a certain threshold script.  This script additionally will skip running mover (optional) if a parity check / rebuild has already been started.

 

Only makes sense to run this script on a schedule, and disable the built-in schedule by editing the config/share.cfg file on the flash drive.  Look for a like that says something like:

shareMoverSchedule="0 4 * * *"

and change it to:

shareMoverSchedule="#0 4 * * *"

 

Followed by a reboot.  Note that any changes to global share settings ( or mover settings ) is probably going to wind up re-enabling the mover schedule

 

 

#!/usr/bin/php
<?PHP
$moveAt = 0;                 # Adjust this value to suit (% cache drive full to move at)
$runDuringCheck = false;     # change to true to run mover during a parity check / rebuild

$diskTotal = disk_total_space("/mnt/cache");
$diskFree = disk_free_space("/mnt/cache");
$percent = ($diskTotal - $diskFree) / $diskTotal * 100;

if ( $percent > $moveAt ) {
  if ( ! $runDuringCheck ) {
    $vars = parse_ini_file("/var/local/emhttp/var.ini");
    if ( $vars['mdResync'] ) {
      echo "Parity Check / Rebuild Running - Not executing mover\n";
      exec("logger Parity Check / Rebuild Running - Not executing mover");
    } else {
      exec("/usr/local/sbin/mover");
    }
  } else {
    exec("/usr/local/sbin/mover");
  }
}
?>

run_mover_at_threshold_enhanced.zip

[GoChris]

Thanks for the scripts. FYI there is no option to disable the mover, at least not on 6.1. Is there on 6.2?

[Squid]

Thanks for the scripts. FYI there is no option to disable the mover, at least not on 6.1. Is there on 6.2?

Rather than do all the debugging on a simple little script, I modified the instructions on how to disable mover by making a one character change on the config/share.cfg file on the flash drive.

[RobJ]

Clear an unRAID array data drive  (for the Shrink array wiki page)

 

Mod note: this script usually takes a much longer than normal time to clear a drive with newer Unraid releases, recommend using the "Remove Drives Then Rebuild Parity" Method or if you really want to clear the disk do it manually.

 

This script is for use in clearing a drive that you want to remove from the array, while maintaining parity protection.  I've added a set of instructions within the Shrink array wiki page for it.  It is designed to be as safe as possible, and will not run unless specific conditions are met -

- The drive must be a data drive that is a part of an unRAID array

- It must be a good drive, mounted in the array, capable of every sector being zeroed (no bad sectors)

- The drive must be completely empty, no data at all left on it.  This is tested for!

- The drive should have a single root folder named clear-me - exactly 8 characters, 7 lowercase and 1 hyphen.  This is tested for!

 

Because the User.Scripts plugin does not allow interactivity (yet!), some kludges had to be used, one being the clear-me folder, and the other being a 60 second wait before execution to allow the user to abort.  I actually like the clear-me kludge, because it means the user cannot possibly make a mistake and lose data.  The user *has* to empty the drive first, then add this odd folder.

 

#!/bin/bash
# A script to clear an unRAID array drive.  It first checks the drive is completely empty,
# except for a marker indicating that the user desires to clear the drive.  The marker is
# that the drive is completely empty except for a single folder named 'clear-me'.
#
# Array must be started, and drive mounted.  There's no other way to verify it's empty.
# Without knowing which file system it's formatted with, I can't mount it.
#
# Quick way to prep drive: format with ReiserFS, then add 'clear-me' folder.
#
# 1.0  first draft
# 1.1  add logging, improve comments
# 1.2  adapt for User.Scripts, extend wait to 60 seconds
# 1.3  add progress display; confirm by key (no wait) if standalone; fix logger
# 1.4  only add progress display if unRAID version >= 6.2

version="1.4"
marker="clear-me"
found=0
wait=60
p=${0%%$P}              # dirname of program
p=${p:0:18}
q="/tmp/user.scripts/"

echo -e "*** Clear an unRAID array data drive ***  v$version\n"

# Check if array is started
ls /mnt/disk[1-9]* 1>/dev/null 2>/dev/null
if [ $? -ne 0 ]
then
   echo "ERROR:  Array must be started before using this script"
   exit
fi

# Look for array drive to clear
n=0
echo -n "Checking all array data drives (may need to spin them up) ... "
if [ "$p" == "$q" ] # running in User.Scripts
then
   echo -e "\n"
   c="<font color=blue>"
   c0="</font>"
else #set color teal
   c="\x1b[36;01m"
   c0="\x1b[39;49;00m"
fi

for d in /mnt/disk[1-9]*
do
   x=`ls -A $d`
   z=`du -s $d`
   y=${z:0:1}
#   echo -e "d:"$d "x:"${x:0:20} "y:"$y "z:"$z

   # the test for marker and emptiness
   if [ "$x" == "$marker" -a "$y" == "0" ]
   then
      found=1
      break
   fi
   let n=n+1
done

#echo -e "found:"$found "d:"$d "marker:"$marker "z:"$z "n:"$n

# No drives found to clear
if [ $found == "0" ]
then
   echo -e "\rChecked $n drives, did not find an empty drive ready and marked for clearing!\n"
   echo "To use this script, the drive must be completely empty first, no files"
   echo "or folders left on it.  Then a single folder should be created on it"
   echo "with the name 'clear-me', exactly 8 characters, 7 lowercase and 1 hyphen."
   echo "This script is only for clearing unRAID data drives, in preparation for"
   echo "removing them from the array.  It does not add a Preclear signature."
   exit
fi

# check unRAID version
v1=`cat /etc/unraid-version`
# v1 is 'version="6.2.0-rc5"' (fixme if 6.10.* happens)
v2="${v1:9:1}${v1:11:1}"
if [[ $v2 -ge 62 ]]
then
   v=" status=progress"
else
   v=""
fi
#echo -e "v1=$v1  v2=$v2  v=$v\n"

# First, warn about the clearing, and give them a chance to abort
echo -e "\rFound a marked and empty drive to clear: $c Disk ${d:9} $c0 ( $d ) "
echo -e "* Disk ${d:9} will be unmounted first."
echo "* Then zeroes will be written to the entire drive."
echo "* Parity will be preserved throughout."
echo "* Clearing while updating Parity takes a VERY long time!"
echo "* The progress of the clearing will not be visible until it's done!"
echo "* When complete, Disk ${d:9} will be ready for removal from array."
echo -e "* Commands to be executed:\n***** $c umount $d $c0\n***** $c dd bs=1M if=/dev/zero of=/dev/md${d:9} $v $c0\n"
if [ "$p" == "$q" ] # running in User.Scripts
then
   echo -e "You have $wait seconds to cancel this script (click the red X, top right)\n"
   sleep $wait
else
   echo -n "Press ! to proceed. Any other key aborts, with no changes made. "
   ch=""
   read -n 1 ch
   echo -e -n "\r                                                                  \r"
   if [ "$ch" != "!" ];
   then
      exit
   fi
fi

# Perform the clearing
logger -tclear_array_drive "Clear an unRAID array data drive  v$version"
echo -e "\rUnmounting Disk ${d:9} ..."
logger -tclear_array_drive "Unmounting Disk ${d:9}  (command: umount $d ) ..."
umount $d
echo -e "Clearing   Disk ${d:9} ..."
logger -tclear_array_drive "Clearing Disk ${d:9}  (command: dd bs=1M if=/dev/zero of=/dev/md${d:9} $v ) ..."
dd bs=1M if=/dev/zero of=/dev/md${d:9} $v
#logger -tclear_array_drive "Clearing Disk ${d:9}  (command: dd bs=1M if=/dev/zero of=/dev/md${d:9} status=progress count=1000 seek=1000 ) ..."
#dd bs=1M if=/dev/zero of=/dev/md${d:9} status=progress count=1000 seek=1000

# Done
logger -tclear_array_drive "Clearing Disk ${d:9} is complete"
echo -e "\nA message saying \"error writing ... no space left\" is expected, NOT an error.\n"
echo -e "Unless errors appeared, the drive is now cleared!"
echo -e "Because the drive is now unmountable, the array should be stopped,"
echo -e "and the drive removed (or reformatted)."
exit

 

 

The attached zip is 'clear an array drive.zip', containing both the User.Scripts folder and files, but also the script named clear_array_drive (same script) for standalone use.  Either extract the files for User.Scripts, or extract clear_array_drive into the root of the flash, and run it from there.

 

Also attached is 'clear an array drive (test only).zip', for playing with this, testing it.  It contains exactly the same scripts, but writing is turned off, so no changes at all will happen.  It is designed for those afraid of clearing the wrong thing, or not trusting these scripts yet.  You can try it in various conditions, and see what happens, and it will pretend to do the work, but no changes at all will be made.

 

I do welcome examination by bash shell script experts, to ensure I made no mistakes.  It's passed my own testing, but I'm not an expert.  Rather, a very frustrated bash user, who lost many hours with the picky syntax!  I really don't understand why people like type-less languages!  It only *looks* easier.

 

After a while, you'll be frustrated with the 60 second wait (when run in User Scripts).  I did have it at 30 seconds, but decided 60 was better for new users, for now.  I'll add interactivity later, for standalone command line use.  It also really needs a way to provide progress info while it's clearing.  I have ideas for that.

 

The included 'clear_array_drive' script can now be run at the command line within any unRAID v6, and possibly unRAID v5, but is not tested there.  (Procedures for removing a drive are different in v5.)  Progress display is only available in 6.2 or later.  In 6.1 or earlier, it's done when it's done.

 

Update 1.3 - add display of progress; confirm by key '!' (no wait) if standalone; fix logger; add a bit of color

  Really appreciate the tip on 'status=progress', looks pretty good.  Lots of numbers presented, the ones of interest are the second and the last.

Update 1.4 - make progress display conditional for 6.2 or later; hopefully now, the script can be run in any v6, possibly v5

clear_an_array_drive.zip

clear_an_array_drive_test_only.zip

[JorgeB]

Clear an unRAID array data drive

 

Nice! One suggestion, if disable, enable turbo write before starting, clearing can be up to 3 times faster.

 

As for the progress info, there's an easy way for v6.2, if the script can check current unRAID version, you can add status=progress to the dd command.

[RobJ]

Clear an unRAID array data drive

 

Nice! One suggestion, if disable, enable turbo write before starting, clearing can be up to 3 times faster.

It's such a major improvement that you're right, it really should be on.  I do include instructions on the Shrink array wiki page to turn it on (and I'm going to add it to the Tips and Tweaks wiki page), but I think I should find a programmatic way to add it to the script.  Perhaps capture their current setting, turn it on, then restore their setting once the script completes.

 

As for the progress info, there's an easy way for v6.2, if the script can check current unRAID version, you can add status=progress to the dd command.

Ooooh nice!  The man page I checked didn't have that.  I should have checked its own '--help'.  Now to see if I can get it to display correctly and usefully in the User Scripts display box...  That may not be easy.  Be great for standalone script use though.

 

Update: added the progress, works great!  Thanks again!

[dheg]

A script to have a file with the folders containing movies and tvshows.

 

My movies and tv shows are independent shares with the next structure:

 

movies
   -BluRay
        -<movie-title>
   -DVD
        -<movie-title>

 

tvshows
   -<tv show title>
        -<season #>

 

 

You might have to adjust -maxdepth and mindepth according to this.

You can also set the REPORTS_PATH variable to configure where to save the reports.

 

#!/bin/bash

DATE=`date +%Y%m%d`
REPORTS_PATH=/mnt/user/appdata/@reports/media

echo "Searching for movies and TV shows..."

mkdir -p $REPORTS_PATH
find /mnt/disk*/* -maxdepth 2 -mindepth 2 -type d -not -iname '.*' \( -path "*/movies/*" -or -path "*/tvshows/*" \) -print | sort -k1 > $REPORTS_PATH/media.catalog_$DATE.csv

echo
echo "Finished."
echo
echo "File created at "$REPORTS_PATH/media.catalog_$DATE.csv

 

I'm a complete noob, so any improvement to the code would be greatly appreciate it!

report_media_catalog.zip

[JonathanM]

I'm playing with Rob's clear drive script, and noticed something very strange. After his script unmounts the drive, emhttp immediately recreates the /mnt/disk1 by chmod it, I guess in RAM. I don't know if it's something wonky with my system, or a bug in unraid, or what.

 

Sep 19 19:10:30 Testunraid emhttp: cmd: /usr/local/emhttp/plugins/user.scripts/backgroundScript.sh /tmp/user.scripts/tmpScripts/clear an array drive/script
Sep 19 19:10:44 Testunraid emhttp: cmd: /usr/local/emhttp/plugins/user.scripts/showLog.php clear an array drive
Sep 19 19:11:30 Testunraid clear_array_drive: Clear an unRAID array data drive v1.4
Sep 19 19:11:30 Testunraid clear_array_drive: Unmounting Disk 1 (command: umount /mnt/disk1 ) ...
Sep 19 19:11:30 Testunraid clear_array_drive: Clearing Disk 1 (command: dd bs=1M if=/dev/zero of=/dev/md1 status=progress ) ...
Sep 19 19:12:01 Testunraid emhttp: shcmd (807): chmod 0777 '/mnt/disk1'
Sep 19 19:12:01 Testunraid emhttp: shcmd (808): chown 'nobody':'users' '/mnt/disk1'

 

/dev/md1 is clearly no longer mounted at /mnt/disk1

 

[email protected]:~# mount
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
tmpfs on /var/log type tmpfs (rw,size=128m,mode=0755)
/dev/sda1 on /boot type vfat (rw,noatime,nodiratime,umask=0,shortname=mixed)
/mnt on /mnt type none (rw,bind)
/dev/md2 on /mnt/disk2 type xfs (rw,noatime,nodiratime)
/dev/sdf1 on /mnt/cache type xfs (rw,noatime,nodiratime)
shfs on /mnt/user0 type fuse.shfs (rw,nosuid,nodev,noatime,allow_other)
shfs on /mnt/user type fuse.shfs (rw,nosuid,nodev,noatime,allow_other)
/mnt/cache/system/docker.img on /var/lib/docker type btrfs (rw)
/mnt/cache/system/libvirt/libvirt.img on /etc/libvirt type btrfs (rw)
[email protected]:~# 

 

I have disk1 globally excluded from shares.

[RobJ]

I'm playing with Rob's clear drive script, and noticed something very strange. After his script unmounts the drive, emhttp immediately recreates the /mnt/disk1 by chmod it, I guess in RAM. I don't know if it's something wonky with my system, or a bug in unraid, or what.

 

I noticed that every time I used the script to clear a 250GB drive, some time after it unmounted it (didn't watch so don't know how soon), the Main screen showed a valid disk mounted there but with a little over 2GB size drive there, and a little of it used, don't remember the numbers.  I just assumed normal wonkiness when I confused it by unmounting it, and never investigated further.  What you found sounds exactly the same.  I never noticed any issues.  When the clearing finished, I stopped the array and went to New Config, which essentially reset everything.  A parity check found nothing wrong.

 

I saw the same lines in my syslog.

[JonathanM]

I'm playing with Rob's clear drive script, and noticed something very strange. After his script unmounts the drive, emhttp immediately recreates the /mnt/disk1 by chmod it, I guess in RAM. I don't know if it's something wonky with my system, or a bug in unraid, or what.

 

I noticed that every time I used the script to clear a 250GB drive, some time after it unmounted it (didn't watch so don't know how soon), the Main screen showed a valid disk mounted there but with a little over 2GB size drive there, and a little of it used, don't remember the numbers.  I just assumed normal wonkiness when I confused it by unmounting it, and never investigated further.  What you found sounds exactly the same.  I never noticed any issues.  When the clearing finished, I stopped the array and went to New Config, which essentially reset everything.  A parity check found nothing wrong.

 

I saw the same lines in my syslog.

I think this is something that Limetech needs to look into. Not your script, but the auto creation of a disk folder not linked to the underlying md system. I can imagine some very strange things happening if a disk is unmounted and things are written to the phantom drive. I didn't test to see if the newly minted disk1 would participate in the user share system, but I assume it could.

 

Are there any circumstances that could cause the disk to unmount without a direct call from a user intiated command?

[RobJ]

I'm playing with Rob's clear drive script, and noticed something very strange. After his script unmounts the drive, emhttp immediately recreates the /mnt/disk1 by chmod it, I guess in RAM. I don't know if it's something wonky with my system, or a bug in unraid, or what.

 

I noticed that every time I used the script to clear a 250GB drive, some time after it unmounted it (didn't watch so don't know how soon), the Main screen showed a valid disk mounted there but with a little over 2GB size drive there, and a little of it used, don't remember the numbers.  I just assumed normal wonkiness when I confused it by unmounting it, and never investigated further.  What you found sounds exactly the same.  I never noticed any issues.  When the clearing finished, I stopped the array and went to New Config, which essentially reset everything.  A parity check found nothing wrong.

 

I saw the same lines in my syslog.

I think this is something that Limetech needs to look into. Not your script, but the auto creation of a disk folder not linked to the underlying md system. I can imagine some very strange things happening if a disk is unmounted and things are written to the phantom drive. I didn't test to see if the newly minted disk1 would participate in the user share system, but I assume it could.

I don't think this is right.  I think if LimeTech was asked about it, they would correctly reply "Don't do that!  Never manually unmount an array drive".  What the script is doing is completely outside the normal operation, could be called a hack.  I suppose it's like taking a hammer but using it backwards, hitting the nail with the handle.  If the handle cracks, would you complain that the handle is no good?  It wasn't meant to do that.

 

I should probably put disclaimers somewhere that this is a completely unofficial hack, tested and seems to work but unapproved by LimeTech.

 

Are there any circumstances that could cause the disk to unmount without a direct call from a user intiated command?

None that I know of.  Maybe one, if the drive is detected to have failed, unRAID may unmount it.  Perhaps I should somehow cause the drive to appear to have failed...

[JonathanM]

Are there any circumstances that could cause the disk to unmount without a direct call from a user intiated command?

None that I know of.  Maybe one, if the drive is detected to have failed, unRAID may unmount it.  Perhaps I should somehow cause the drive to appear to have failed...

A failed drive's content still appears at the /mnt/diskX location, but the link to the physical drive is gone. We need the physical drive to still be linked to the md device so the script accomplishes what is needed.

 

I still think unraid shouldn't automatically create /mnt/diskX locations that don't correspond to what's on the drive. I would expect the GUI line to show unmountable, reflecting what's actually happening.

 

I guess it all boils down to this process being seen as unnecessary from Limetech's point of view, since we've been talking about it and doing it for years, without input from Tom. So, we continue to hit the nail with the handle and hope we don't break it. 

[RobJ]

You do make me laugh!  I still laugh *every* time I see your 'hoovering' comment!

[IamSpartacus]

Can rsyncs be run using this plugin?  I get the following errors when I try to.

 

rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.2]

[Squid]

Can rsyncs be run using this plugin?  I get the following errors when I try to.

 

rsync: connection unexpectedly closed (0 bytes received so far) [Receiver]
rsync error: unexplained error (code 255) at io.c(226) [Receiver=3.1.2]

No fundamental reason why they couldn't be that I'm aware of