Additional Scripts For User.Scripts Plugin

[Vitek]

#!/bin/bash

DIR="/mnt/user/media/movies/MOVIE_FOLDER/"

find "$DIR" -type f -name '*.mkv' | while read filepath; do
docker exec MKVToolNix /usr/bin/mkvmerge -J "$filepath"
done

 

I am trying to write script that use mkvtoolnix container to process my movie files

 

in line: docker exec MKVToolNix /usr/bin/mkvmerge -J "$filepath" i get error

Quote

"The file '/mnt/user/media/movies/MOVIE_FOLDER/MOVIE_FILE.mkv' could not be opened for reading: open file error.\n"

i don't know why script can't read this file

[Kilrah]

The command is being run inside the container, it can't directly reach the unraid filesystem. You have to map a volume and use the container path.

[NominallySavvyTechPerson]

I read https://docs.unraid.net/legacy/FAQ/shrink-array/ and followed the link for step 8 "Run the clear an array drive script" that took me to 

 the comment that has a download link for the "clear_array_drive" script.

I tried running it but got an error.

```

Found a marked and empty drive to clear:  Disk 3  ( /mnt/disk3 )
+ echo -e '* Disk 3 will be unmounted first.'
* Disk 3 will be unmounted first.
+ echo '* Then zeroes will be written to the entire drive.'
* Then zeroes will be written to the entire drive.
+ echo '* Parity will be preserved throughout.'
* Parity will be preserved throughout.
+ echo '* Clearing while updating Parity takes a VERY long time!'
* Clearing while updating Parity takes a VERY long time!
+ echo '* The progress of the clearing will not be visible until it'\''s done!'
* The progress of the clearing will not be visible until it's done!
+ echo '* When complete, Disk 3 will be ready for removal from array.'
* When complete, Disk 3 will be ready for removal from array.
+ echo -e '* Commands to be executed:\n***** \x1b[36;01m umount /mnt/disk3 \x1b[39;49;00m\n***** \x1b[36;01m dd bs=1M if=/dev/zero of=/dev/md3  \x1b[39;49;00m\n'
* Commands to be executed:
*****  umount /mnt/disk3
*****  dd bs=1M if=/dev/zero of=/dev/md3

+ '[' clear_array_drive == /tmp/user.scripts/ ']'
+ echo -n 'Press ! to proceed. Any other key aborts, with no changes made. '
Press ! to proceed. Any other key aborts, with no changes made. + ch=
+ read -n 1 ch
!+ echo -e -n '\r                                                                  \r'
+ '[' '!' '!=' '!' ']'
+ logger -tclear_array_drive 'Clear an unRAID array data drive  v1.4'
+ echo -e '\rUnmounting Disk 3 ...'
Unmounting Disk 3 ...
+ logger -tclear_array_drive 'Unmounting Disk 3  (command: umount /mnt/disk3 ) ...'
+ umount /mnt/disk3
+ echo -e 'Clearing   Disk 3 ...'
Clearing   Disk 3 ...
+ logger -tclear_array_drive 'Clearing Disk 3  (command: dd bs=1M if=/dev/zero of=/dev/md3  ) ...'
+ dd bs=1M if=/dev/zero of=/dev/md3
dd: error writing '/dev/md3': No space left on device
9+0 records in
8+0 records out
8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.00256501 s, 3.3 GB/s
+ logger -tclear_array_drive 'Clearing Disk 3 is complete'
+ echo -e '\nA message saying "error writing ... no space left" is expected, NOT an error.\n'

A message saying "error writing ... no space left" is expected, NOT an error.

+ echo -e 'Unless errors appeared, the drive is now cleared!'
Unless errors appeared, the drive is now cleared!
+ echo -e 'Because the drive is now unmountable, the array should be stopped,'
Because the drive is now unmountable, the array should be stopped,
+ echo -e 'and the drive removed (or reformatted).'
and the drive removed (or reformatted).
+ exit

```

I ran the script with `sh -x` so there's debug output. I see it uses `/dev/mdX` not `/dev/mdXp1`. Is that the issue? Are people using this script? Do I have an old version of the script? Here's the download link for the script from the comment by RobJ https://s3-us-west-2.amazonaws.com/upload.forums.unraid.net/live/monthly_2016_09/clear_an_array_drive_zip.80cc67c667b13547f490de09b702bf4b

I also searched this topic for "of=/dev/md" and I didn't see any references that included the "p1".

[Zerginator]

I cannot get the script working anymore with Unraid 6.12.8.

I finished with successfull after about 30 seconds, leaving the array with in a state with unmounted drive that prevents clean shutdown of the array and enforces a "unclean shutdown detected" parity check.

After reboot / array restart the drive is back with the clear-me folder inside.

[Kilrah]

If that's about the "clear an array drive" script change references to "/dev/mdX" with "/dev/mdXp1"

[Zerginator]

4 hours ago, Kilrah said:

If that's about the "clear an array drive" script change references to "/dev/mdX" with "/dev/mdXp1"

Thank you, I will try that

[Shomesomesho]

On 10/29/2016 at 12:56 PM, Squid said:

Run A Custom Script At Parity Check / Rebuild Start And Stop

Use it to run a custom script to (as an example), shut down various docker applications, etc.  Adjust the variables within the script file.

 

Note that you either need to run this in the background or at array start.  Running this in the foreground will not work.

 

 

#!/usr/bin/php
<?PHP
# A simple script to allow you to run a custom script when a parity check starts or stops
# Adjust the following variables to suit:

$checkInterval = 300;                             # Number of seconds in between checks
$startScript   = "full path to the script";       # The full path to the script to run when a parity check starts
$stopScript    = "full path to the stop script";  # The full path to the script to run when a parity check stops

# Don't touch anything below

while (true) {
  $vars = parse_ini_file("/var/local/emhttp/var.ini");
  if ( $vars['mdState'] == "STOPPED" ) {
    break;
  }
  if ( ($vars['mdResyncPos'] != 0) && $vars ) {
    echo "Parity Check / Sync / Rebuild in progress.  Executing the start script ($startScript)";
    exec($startScript,$output);
    foreach ($output as $line) {
      echo $line."\n";
    }
    while (true) {
      $vars = parse_ini_file("var/local/emhttp/var.ini");
      if ( ($vars['mdResyncPos'] == 0) && $vars ) {
        echo "Parity Check / Sync / Rebuild finished.  Executing the stop script ($stopScript)";
        exec($stopScript,$output);
        foreach ($output as $line) {
          echo $line."\n";
        }
        break;
      } else {
        sleep($checkInterval);
      }
    }
  } else { 
    sleep($checkInterval);
  }  
}
?>

 

 

custom_script_parity_check_start_stop.zip 1010 B · 64 downloads

Is there a way to modify this script to consider a paused parity operation as stopped/finished?

Edited April 7 by Shomesomesho

[itimpi]

17 hours ago, Shomesomesho said:

Is there a way to modify this script to consider a paused parity operation as stopped/finished?

What is it you actually want to do if the check is paused?   There may be a better way to achieve this.?

 

Also what is the Use Case you have for even running a script at start/end?    Just asking as if it is a good one I could consider adding support for this as a standard feature in the Parity Check Tuning plugin.

[Shomesomesho]

5 hours ago, itimpi said:

What is it you actually want to do if the check is paused?   There may be a better way to achieve this.?

 

Also what is the Use Case you have for even running a script at start/end?    Just asking as if it is a good one I could consider adding support for this as a standard feature in the Parity Check Tuning plugin.

https://old.reddit.com/r/unRAID/comments/1byaqxn/scripts_to_pause_torrents_during_parity/

 

Seeding torrents from the array slows down a parity operation. I'd rather pause the seeding than slow down the parity operation. I don't want to close the torrent client completely, as that stops downloads to the cache. And without pause being recognized as a stop/end, and using the Increments feature of Parity Check Tuning plugin means the torrent docker would remain shut down for days until the parity operation is finished.

Edited April 8 by Shomesomesho

[itimpi]

Just now, Shomesomesho said:

https://old.reddit.com/r/unRAID/comments/1byaqxn/scripts_to_pause_torrents_during_parity/

 

Seeding torrents from the array slows down parity operation. I'd rather pause the seeding than slow down the parity operation. I don't want to close the torrent client completely, as that stops downloads to the cache. And without pause being recognized as a stop/end, and using the Increments feature of Parity Check Tuning plugin means the torrent docker would remain shut down for days until the parity operation is finished.

from an earlier request I have been working on an option to pause (rather than stop) user specified  docker containers during parity operations and resume them when an increment ends.   Sounds like that might satisfy this need without the need for a script?

[Shomesomesho]

1 minute ago, itimpi said:

from an earlier request I have been working on an option to pause (rather than stop) user specified  docker containers during parity operations and resume them when an increment ends.   Sounds like that might satisfy this need without the need for a script?

Unfortunately that would prevent being able to download to a cache pool during an increment.

[Kilrah]

13 hours ago, Shomesomesho said:

Unfortunately that would prevent being able to download to a cache pool during an increment.

 

Some people use separate containers for downloading and seeding, could be a solution.

[cinereus]

What happens if you make writes to the array while the zeroing script is running? Is that okay? 

[cinereus]

On 9/4/2016 at 1:06 AM, RobJ said:

Clear an unRAID array data drive  (for the Shrink array wiki page)

 

Mod note: this script usually takes a much longer than normal time to clear a drive with newer Unraid releases, recommend using the "Remove Drives Then Rebuild Parity" Method or if you really want to clear the disk do it manually.

 

This script is for use in clearing a drive that you want to remove from the array, while maintaining parity protection.  I've added a set of instructions within the Shrink array wiki page for it.  It is designed to be as safe as possible, and will not run unless specific conditions are met -

- The drive must be a data drive that is a part of an unRAID array

- It must be a good drive, mounted in the array, capable of every sector being zeroed (no bad sectors)

- The drive must be completely empty, no data at all left on it.  This is tested for!

- The drive should have a single root folder named clear-me - exactly 8 characters, 7 lowercase and 1 hyphen.  This is tested for!

 

Because the User.Scripts plugin does not allow interactivity (yet!), some kludges had to be used, one being the clear-me folder, and the other being a 60 second wait before execution to allow the user to abort.  I actually like the clear-me kludge, because it means the user cannot possibly make a mistake and lose data.  The user *has* to empty the drive first, then add this odd folder.

 

#!/bin/bash
# A script to clear an unRAID array drive.  It first checks the drive is completely empty,
# except for a marker indicating that the user desires to clear the drive.  The marker is
# that the drive is completely empty except for a single folder named 'clear-me'.
#
# Array must be started, and drive mounted.  There's no other way to verify it's empty.
# Without knowing which file system it's formatted with, I can't mount it.
#
# Quick way to prep drive: format with ReiserFS, then add 'clear-me' folder.
#
# 1.0  first draft
# 1.1  add logging, improve comments
# 1.2  adapt for User.Scripts, extend wait to 60 seconds
# 1.3  add progress display; confirm by key (no wait) if standalone; fix logger
# 1.4  only add progress display if unRAID version >= 6.2

version="1.4"
marker="clear-me"
found=0
wait=60
p=${0%%$P}              # dirname of program
p=${p:0:18}
q="/tmp/user.scripts/"

echo -e "*** Clear an unRAID array data drive ***  v$version\n"

# Check if array is started
ls /mnt/disk[1-9]* 1>/dev/null 2>/dev/null
if [ $? -ne 0 ]
then
   echo "ERROR:  Array must be started before using this script"
   exit
fi

# Look for array drive to clear
n=0
echo -n "Checking all array data drives (may need to spin them up) ... "
if [ "$p" == "$q" ] # running in User.Scripts
then
   echo -e "\n"
   c="<font color=blue>"
   c0="</font>"
else #set color teal
   c="\x1b[36;01m"
   c0="\x1b[39;49;00m"
fi

for d in /mnt/disk[1-9]*
do
   x=`ls -A $d`
   z=`du -s $d`
   y=${z:0:1}
#   echo -e "d:"$d "x:"${x:0:20} "y:"$y "z:"$z

   # the test for marker and emptiness
   if [ "$x" == "$marker" -a "$y" == "0" ]
   then
      found=1
      break
   fi
   let n=n+1
done

#echo -e "found:"$found "d:"$d "marker:"$marker "z:"$z "n:"$n

# No drives found to clear
if [ $found == "0" ]
then
   echo -e "\rChecked $n drives, did not find an empty drive ready and marked for clearing!\n"
   echo "To use this script, the drive must be completely empty first, no files"
   echo "or folders left on it.  Then a single folder should be created on it"
   echo "with the name 'clear-me', exactly 8 characters, 7 lowercase and 1 hyphen."
   echo "This script is only for clearing unRAID data drives, in preparation for"
   echo "removing them from the array.  It does not add a Preclear signature."
   exit
fi

# check unRAID version
v1=`cat /etc/unraid-version`
# v1 is 'version="6.2.0-rc5"' (fixme if 6.10.* happens)
v2="${v1:9:1}${v1:11:1}"
if [[ $v2 -ge 62 ]]
then
   v=" status=progress"
else
   v=""
fi
#echo -e "v1=$v1  v2=$v2  v=$v\n"

# First, warn about the clearing, and give them a chance to abort
echo -e "\rFound a marked and empty drive to clear: $c Disk ${d:9} $c0 ( $d ) "
echo -e "* Disk ${d:9} will be unmounted first."
echo "* Then zeroes will be written to the entire drive."
echo "* Parity will be preserved throughout."
echo "* Clearing while updating Parity takes a VERY long time!"
echo "* The progress of the clearing will not be visible until it's done!"
echo "* When complete, Disk ${d:9} will be ready for removal from array."
echo -e "* Commands to be executed:\n***** $c umount $d $c0\n***** $c dd bs=1M if=/dev/zero of=/dev/md${d:9} $v $c0\n"
if [ "$p" == "$q" ] # running in User.Scripts
then
   echo -e "You have $wait seconds to cancel this script (click the red X, top right)\n"
   sleep $wait
else
   echo -n "Press ! to proceed. Any other key aborts, with no changes made. "
   ch=""
   read -n 1 ch
   echo -e -n "\r                                                                  \r"
   if [ "$ch" != "!" ];
   then
      exit
   fi
fi

# Perform the clearing
logger -tclear_array_drive "Clear an unRAID array data drive  v$version"
echo -e "\rUnmounting Disk ${d:9} ..."
logger -tclear_array_drive "Unmounting Disk ${d:9}  (command: umount $d ) ..."
umount $d
echo -e "Clearing   Disk ${d:9} ..."
logger -tclear_array_drive "Clearing Disk ${d:9}  (command: dd bs=1M if=/dev/zero of=/dev/md${d:9} $v ) ..."
dd bs=1M if=/dev/zero of=/dev/md${d:9} $v
#logger -tclear_array_drive "Clearing Disk ${d:9}  (command: dd bs=1M if=/dev/zero of=/dev/md${d:9} status=progress count=1000 seek=1000 ) ..."
#dd bs=1M if=/dev/zero of=/dev/md${d:9} status=progress count=1000 seek=1000

# Done
logger -tclear_array_drive "Clearing Disk ${d:9} is complete"
echo -e "\nA message saying \"error writing ... no space left\" is expected, NOT an error.\n"
echo -e "Unless errors appeared, the drive is now cleared!"
echo -e "Because the drive is now unmountable, the array should be stopped,"
echo -e "and the drive removed (or reformatted)."
exit

 

 

The attached zip is 'clear an array drive.zip', containing both the User.Scripts folder and files, but also the script named clear_array_drive (same script) for standalone use.  Either extract the files for User.Scripts, or extract clear_array_drive into the root of the flash, and run it from there.

 

Also attached is 'clear an array drive (test only).zip', for playing with this, testing it.  It contains exactly the same scripts, but writing is turned off, so no changes at all will happen.  It is designed for those afraid of clearing the wrong thing, or not trusting these scripts yet.  You can try it in various conditions, and see what happens, and it will pretend to do the work, but no changes at all will be made.

 

I do welcome examination by bash shell script experts, to ensure I made no mistakes.  It's passed my own testing, but I'm not an expert.  Rather, a very frustrated bash user, who lost many hours with the picky syntax!  I really don't understand why people like type-less languages!  It only *looks* easier.

 

After a while, you'll be frustrated with the 60 second wait (when run in User Scripts).  I did have it at 30 seconds, but decided 60 was better for new users, for now.  I'll add interactivity later, for standalone command line use.  It also really needs a way to provide progress info while it's clearing.  I have ideas for that.

 

The included 'clear_array_drive' script can now be run at the command line within any unRAID v6, and possibly unRAID v5, but is not tested there.  (Procedures for removing a drive are different in v5.)  Progress display is only available in 6.2 or later.  In 6.1 or earlier, it's done when it's done.

 

Update 1.3 - add display of progress; confirm by key '!' (no wait) if standalone; fix logger; add a bit of color

  Really appreciate the tip on 'status=progress', looks pretty good.  Lots of numbers presented, the ones of interest are the second and the last.

Update 1.4 - make progress display conditional for 6.2 or later; hopefully now, the script can be run in any v6, possibly v5

clear_an_array_drive.zip 4.37 kB · 2,177 downloads

clear_an_array_drive_test_only.zip 4.61 kB · 214 downloads

Is this still the correct script to user in 2024?

[sosdk]

Hi

 

Just ran the Clear an unRAID array data drive script on a 4TB drive and it only took 2 minuttes. Is this a succesfull run ?

 

Script location: /tmp/user.scripts/tmpScripts/ZERO Drive/script
Note that closing this window will abort the execution of this script
*** Clear an unRAID array data drive *** v1.4

Checking all array data drives (may need to spin them up) ...

Found a marked and empty drive to clear: Disk 2 ( /mnt/disk2 )
* Disk 2 will be unmounted first.
* Then zeroes will be written to the entire drive.
* Parity will be preserved throughout.
* Clearing while updating Parity takes a VERY long time!
* The progress of the clearing will not be visible until it's done!
* When complete, Disk 2 will be ready for removal from array.
* Commands to be executed:
***** umount /mnt/disk2
***** dd bs=1M if=/dev/zero of=/dev/md2

You have 60 seconds to cancel this script (click the red X, top right)

Unmounting Disk 2 ...
umount: /mnt/disk2: target is busy.
Clearing Disk 2 ...
dd: error writing '/dev/md2': No space left on device
9+0 records in
8+0 records out
8388608 bytes (8.4 MB, 8.0 MiB) copied, 0.00268934 s, 3.1 GB/s

A message saying "error writing ... no space left" is expected, NOT an error.

Unless errors appeared, the drive is now cleared!
Because the drive is now unmountable, the array should be stopped,
and the drive removed (or reformatted).

 

 

[Kilrah]

No, the script needs a modification to clear /dev/mdXp1 instead of /dev/mdX for unraid 6.12+

Edited Monday at 10:53 AM by Kilrah

[sosdk]

Hi

 

Tried yo modify script but still doesnt run correct. Is there a link to correct script ?

 

 

[Kilrah]

You'll need to reboot (possibly stop/start array is enough) after the failed attempt with the wrong one

[sosdk]

Ok - I'll do a reboot and try again.