Additional Scripts For User.Scripts Plugin

[Meles Meles]

3 hours ago, Sherk said:

Hello Everyone,

 

thanks for helping the community with great script.

 

however I have a small request, I have one docker container that always fail or crash for no  reason,I tried to solve this issue by making a script that will start container every hour, some time it crash between that time and I lose container function so I need a script to monitor docker container and if it crash/stopped it will directly start again in anytime.

 

Thank you all in advance for your time and effort! 

 

 

[Sherk]

5 hours ago, Meles Meles said:

 

 

thanks for your help

Edited May 4, 2022 by Sherk

[Sherk]

47 minutes ago, Sherk said:

 

thanks for your help

I follow you example but it didn't work for me, zigbee2mqtt is the container that I have the issue with it, could you help

[simo241]

Hello Everyone,

thank you  for helping with this great script.

 

I need a script to copy in automatic all file from an external disk to a folder on unraid.

The script need to mount the disk when detected synchronize all file from the mounted disk to a specific folder and then unmount the disk.

If same file on the external disk have been deleted , delete as well the one on unraid .

I was using till now next cloud but to do that i need to switch on another computer and i would like to avoid it

Thank you for your support

 

[trurl]

On 5/10/2022 at 6:48 AM, simo241 said:

script to copy in automatic all file from an external disk to a folder on unraid

You can probably adapt one of the example scripts for Unassigned Devices.

[simo241]

Can you please indicate the link with all script

 

[wgstarks]

35 minutes ago, simo241 said:

Can you please indicate the link with all script

 

https://forums.unraid.net/topic/92462-unassigned-devices-managing-disk-drives-and-remote-shares-outside-of-the-unraid-array/

 

scroll down to the Scripts section.

[timrettop]

On 9/3/2016 at 5:06 PM, RobJ said:

 

 

 

#!/bin/bash
# A script to clear an unRAID array drive.  It first checks the drive is completely empty,
# except for a marker indicating that the user desires to clear the drive.  The marker is
# that the drive is completely empty except for a single folder named 'clear-me'.
#
# Array must be started, and drive mounted.  There's no other way to verify it's empty.
# Without knowing which file system it's formatted with, I can't mount it.
#
# Quick way to prep drive: format with ReiserFS, then add 'clear-me' folder.
#
# 1.0  first draft
# 1.1  add logging, improve comments
# 1.2  adapt for User.Scripts, extend wait to 60 seconds
# 1.3  add progress display; confirm by key (no wait) if standalone; fix logger
# 1.4  only add progress display if unRAID version >= 6.2

version="1.4"
marker="clear-me"
found=0
wait=60
p=${0%%$P}              # dirname of program
p=${p:0:18}
q="/tmp/user.scripts/"

echo -e "*** Clear an unRAID array data drive ***  v$version\n"

# Check if array is started
ls /mnt/disk[1-9]* 1>/dev/null 2>/dev/null
if [ $? -ne 0 ]
then
   echo "ERROR:  Array must be started before using this script"
   exit
fi

# Look for array drive to clear
n=0
echo -n "Checking all array data drives (may need to spin them up) ... "
if [ "$p" == "$q" ] # running in User.Scripts
then
   echo -e "\n"
   c="<font color=blue>"
   c0="</font>"
else #set color teal
   c="\x1b[36;01m"
   c0="\x1b[39;49;00m"
fi

for d in /mnt/disk[1-9]*
do
   x=`ls -A $d`
   z=`du -s $d`
   y=${z:0:1}
#   echo -e "d:"$d "x:"${x:0:20} "y:"$y "z:"$z

   # the test for marker and emptiness
   if [ "$x" == "$marker" -a "$y" == "0" ]
   then
      found=1
      break
   fi
   let n=n+1
done

#echo -e "found:"$found "d:"$d "marker:"$marker "z:"$z "n:"$n

# No drives found to clear
if [ $found == "0" ]
then
   echo -e "\rChecked $n drives, did not find an empty drive ready and marked for clearing!\n"
   echo "To use this script, the drive must be completely empty first, no files"
   echo "or folders left on it.  Then a single folder should be created on it"
   echo "with the name 'clear-me', exactly 8 characters, 7 lowercase and 1 hyphen."
   echo "This script is only for clearing unRAID data drives, in preparation for"
   echo "removing them from the array.  It does not add a Preclear signature."
   exit
fi

# check unRAID version
v1=`cat /etc/unraid-version`
# v1 is 'version="6.2.0-rc5"' (fixme if 6.10.* happens)
v2="${v1:9:1}${v1:11:1}"
if [[ $v2 -ge 62 ]]
then
   v=" status=progress"
else
   v=""
fi
#echo -e "v1=$v1  v2=$v2  v=$v\n"

# First, warn about the clearing, and give them a chance to abort
echo -e "\rFound a marked and empty drive to clear: $c Disk ${d:9} $c0 ( $d ) "
echo -e "* Disk ${d:9} will be unmounted first."
echo "* Then zeroes will be written to the entire drive."
echo "* Parity will be preserved throughout."
echo "* Clearing while updating Parity takes a VERY long time!"
echo "* The progress of the clearing will not be visible until it's done!"
echo "* When complete, Disk ${d:9} will be ready for removal from array."
echo -e "* Commands to be executed:\n***** $c umount $d $c0\n***** $c dd bs=1M if=/dev/zero of=/dev/md${d:9} $v $c0\n"
if [ "$p" == "$q" ] # running in User.Scripts
then
   echo -e "You have $wait seconds to cancel this script (click the red X, top right)\n"
   sleep $wait
else
   echo -n "Press ! to proceed. Any other key aborts, with no changes made. "
   ch=""
   read -n 1 ch
   echo -e -n "\r                                                                  \r"
   if [ "$ch" != "!" ];
   then
      exit
   fi
fi

# Perform the clearing
logger -tclear_array_drive "Clear an unRAID array data drive  v$version"
echo -e "\rUnmounting Disk ${d:9} ..."
logger -tclear_array_drive "Unmounting Disk ${d:9}  (command: umount $d ) ..."
umount $d
echo -e "Clearing   Disk ${d:9} ..."
logger -tclear_array_drive "Clearing Disk ${d:9}  (command: dd bs=1M if=/dev/zero of=/dev/md${d:9} $v ) ..."
dd bs=1M if=/dev/zero of=/dev/md${d:9} $v
#logger -tclear_array_drive "Clearing Disk ${d:9}  (command: dd bs=1M if=/dev/zero of=/dev/md${d:9} status=progress count=1000 seek=1000 ) ..."
#dd bs=1M if=/dev/zero of=/dev/md${d:9} status=progress count=1000 seek=1000

# Done
logger -tclear_array_drive "Clearing Disk ${d:9} is complete"
echo -e "\nA message saying \"error writing ... no space left\" is expected, NOT an error.\n"
echo -e "Unless errors appeared, the drive is now cleared!"
echo -e "Because the drive is now unmountable, the array should be stopped,"
echo -e "and the drive removed (or reformatted)."
exit

 

 

@RobJ The version check fails with 6.10 (As you predicted when you created this. 

 

This seems to be a valid alternative version check from a variety of version number tests I've tried.  It pulls the major and minor version numbers and then tests with those values to determine whether to add the string.  I initialized $v as well to simplify it slightly. 

 

v="" #Initialize $v
# check unRAID version
v1=`cat /etc/unraid-version`
#v1="version=\"7.0.5\"" # For Testing

v2="$(echo $v1 | cut -d \" -f2)"
v2M="$(echo $v2 | cut -d '.' -f 1)"
v2m="$(echo $v2 | cut -d '.' -f 2)"
if [[ $v2M == 6 ]]
then
   if [[ $v2m -ge 2 ]]
   then
      v=" status=progress"
   fi
elif [[ $v2M -gt 6 ]]
   then
      v=" status=progress"
fi
#echo -e "v1=$v1  v2=$v2 v2M=$v2M v2m=$v2m  v=$v\n"

 

[ss1gohan13]

On 9/3/2016 at 7:06 PM, RobJ said:

Update 1.3 - add display of progress; confirm by key '!' (no wait) if standalone; fix logger; add a bit of color

  Really appreciate the tip on 'status=progress', looks pretty good.  Lots of numbers presented, the ones of interest are the second and the last.

If I have closed my browser, is there a way to still see the status of the progress?

[Darksurf]

What is up with the zero drive script? It immediately gives up.

 

*** Clear an unRAID array data drive ***  v1.4

Checking all array data drives (may need to spin them up) ... 

Checked 10 drives, did not find an empty drive ready and marked for clearing!

To use this script, the drive must be completely empty first, no files
or folders left on it.  Then a single folder should be created on it
with the name 'clear-me', exactly 8 characters, 7 lowercase and 1 hyphen.
This script is only for clearing unRAID data drives, in preparation for
removing them from the array.  It does not add a Preclear signature.
Script Finished Jul 10, 2022  19:21.37

Full logs for this script are available at /tmp/user.scripts/tmpScripts/ZeroDisks_ShrinkArray/log.txt

^C
root@Oceans:~# ls -al /mnt/disk*
/mnt/disk1:
total 16
drwxrwxrwx   1 nobody users   84 Jul 10 04:30 ./
drwxr-xr-x  18 root   root   360 Jul 10 00:12 ../
drwxrwxrwx+  1 nobody users  190 Jul 10 18:28 Docker/
drwxrwxrwx   1 nobody users   14 Jul  4 22:49 Downloads/
drwxrwxrwx   1 nobody users   60 Jul  6 03:26 ZDRIVE/
drwxrwxrwx   1 nobody users    0 Jul 20  2021 appdata/
drwxrwxrwx   1 nobody users   16 Apr 16  2021 home/
drwxrwxrwx   1 nobody users 1884 Jul  9 04:40 system/
drwxrwxrwx   1 nobody users  138 Dec 31  2017 tftp/

/mnt/disk10:
total 16
drwxrwxrwx  1 nobody users  16 Jul 10 18:33 ./
drwxr-xr-x 18 root   root  360 Jul 10 00:12 ../
drwxrwxrwx  1 nobody users   0 Jul 10 18:33 clear-me/

/mnt/disk2:
total 16
drwxrwxrwx   1 nobody users  12 Jul 10 04:30 ./
drwxr-xr-x  18 root   root  360 Jul 10 00:12 ../
drwxrwxrwx+  1 nobody users 260 Jul  9 23:38 Docker/

/mnt/disk3:
total 16
drwxrwxrwx   1 nobody users  84 Jul 10 04:30 ./
drwxr-xr-x  18 root   root  360 Jul 10 00:12 ../
drwxrwxrwx+  1 nobody users 188 Jul  9 23:38 Docker/
drwxrwxrwx   1 nobody users   0 Jul  6 22:31 Downloads/
drwxr-xr-x   1 nobody users   0 May  9 09:08 ISOs/
drwxrwxrwx   1 nobody users  32 Jul  6 22:28 ZDRIVE/
drwxrwxrwx   1 nobody users   0 Jul 20  2021 appdata/
drwxrwxrwx   1 nobody users  16 Jul  6 21:50 home/
drwxrwxrwx   1 nobody users 394 Jul  6 22:31 system/

/mnt/disk4:
total 16
drwxrwxrwx   1 nobody users  66 Jul 10 04:30 ./
drwxr-xr-x  18 root   root  360 Jul 10 00:12 ../
drwxrwxrwx+  1 nobody users 170 Jul  6 12:48 Docker/
drwxrwxrwx   1 nobody users   8 Jun  5  2021 ZDRIVE/
drwxrwxrwx   1 nobody users   0 Jul 20  2021 appdata/
drwxrwxrwx   1 nobody users  38 Jul  6 12:47 home/
drwxrwxrwx   1 nobody users  96 Jul  6 12:48 system/
drwxrwxrwx   1 nobody users   0 Dec 31  2017 tftp/

/mnt/disk5:
total 16
drwxrwxrwx  1 nobody users  16 Jul 10 18:35 ./
drwxr-xr-x 18 root   root  360 Jul 10 00:12 ../
drwxrwxrwx  1 nobody users   0 Jul 10 18:35 clear-me/

/mnt/disk6:
total 16
drwxrwxrwx  1 nobody users  16 Jul 10 18:35 ./
drwxr-xr-x 18 root   root  360 Jul 10 00:12 ../
drwxrwxrwx  1 nobody users   0 Jul 10 18:35 clear-me/

/mnt/disk7:
total 16
drwxrwxrwx  1 nobody users  16 Jul 10 18:34 ./
drwxr-xr-x 18 root   root  360 Jul 10 00:12 ../
drwxrwxrwx  1 nobody users   0 Jul 10 18:34 clear-me/

/mnt/disk8:
total 16
drwxrwxrwx  1 nobody users  16 Jul 10 18:34 ./
drwxr-xr-x 18 root   root  360 Jul 10 00:12 ../
drwxrwxrwx  1 nobody users   0 Jul 10 18:34 clear-me/

/mnt/disk9:
total 16
drwxrwxrwx  1 nobody users  16 Jul 10 18:33 ./
drwxr-xr-x 18 root   root  360 Jul 10 00:12 ../
drwxrwxrwx  1 nobody users   0 Jul 10 18:33 clear-me/

/mnt/disks:
total 0
drwxrwxrwt  2 nobody users  40 Jul 10 00:11 ./
drwxr-xr-x 18 root   root  360 Jul 10 00:12 ../

 

[kizer]

@Darksurf

 

Did you make sure the target drive was completely empty with the exception of a single folder clear-me

[Darksurf]

2 hours ago, kizer said:

@Darksurf

 

Did you make sure the target drive was completely empty with the exception of a single folder clear-me

 

Yes. If you check my post above you'll see ls -al of disks5-10 look like this:

 

Quote

/mnt/disk5:
total 16
drwxrwxrwx  1 nobody users  16 Jul 10 18:35 ./
drwxr-xr-x 18 root   root  360 Jul 10 00:12 ../
drwxrwxrwx  1 nobody users   0 Jul 10 18:35 clear-me/
 

/mnt/disk6:
total 16
drwxrwxrwx  1 nobody users  16 Jul 10 18:35 ./
drwxr-xr-x 18 root   root  360 Jul 10 00:12 ../
drwxrwxrwx  1 nobody users   0 Jul 10 18:35 clear-me/
 

/mnt/disk7:
total 16
drwxrwxrwx  1 nobody users  16 Jul 10 18:34 ./
drwxr-xr-x 18 root   root  360 Jul 10 00:12 ../
drwxrwxrwx  1 nobody users   0 Jul 10 18:34 clear-me/
 

/mnt/disk8:
total 16
drwxrwxrwx  1 nobody users  16 Jul 10 18:34 ./
drwxr-xr-x 18 root   root  360 Jul 10 00:12 ../
drwxrwxrwx  1 nobody users   0 Jul 10 18:34 clear-me/
 

/mnt/disk9:
total 16
drwxrwxrwx  1 nobody users  16 Jul 10 18:33 ./
drwxr-xr-x 18 root   root  360 Jul 10 00:12 ../
drwxrwxrwx  1 nobody users   0 Jul 10 18:33 clear-me/

/mnt/disk10:
total 16
drwxrwxrwx  1 nobody users  16 Jul 10 18:33 ./
drwxr-xr-x 18 root   root  360 Jul 10 00:12 ../
drwxrwxrwx  1 nobody users   0 Jul 10 18:33 clear-me/

 

I ran unbalance on these drives 3 times to be sure, checked there were no files left, then did a full rm -rf /mnt/disk#/* on each drive I planned to wipe and then mkdir -p /mnt/disk#/clear-me on every disk I planned to wipe. I'm 100% positive the drives were empty besides the clear-me folder.

I doubt its a problem, but these drives are all formatted BTRFS not XFS. It could also be some incompatibility with 6.10.3, not sure. I ended up just removing them from the machine and creating a new config for the array and rebuilding the parity drives.

Edited July 12, 2022 by Darksurf

[Majyk Oyster]

Hi there.

 

I needed to use RobJ's "Clear An unRaid Data Drive" script, and noticed it wasn't compatible with Unraid 6.10 (version verification checked only 1 digit). So I dived into the script and found a few things I could update or do differently.

 

Since RobJ doesn't seem to be around anymore, I took the liberty to rewrite most of it.

 

Change log :
    Version check adapted for Unraid  6.10 and up.
    Check if unmount successful before clearing.
    Send mail notification when clearing finished.
    Script adapted for more then 9 drives.
    Dead code removed.
    Code simplified and optimized.
    Ambiguous variables renamed.
    Progression of the dd command sent to the GUI log

    Error message more explicit if more than 1 drive is prepared for clearing.

 

 

#!/bin/bash
# A script to clear an unRAID array drive.  It first checks the drive is completely empty,
# except for a marker indicating that the user desires to clear the drive.  The marker is
# that the drive is completely empty except for a single folder named 'clear-me'.
#
# Array must be started, and drive mounted.  There's no other way to verify it's empty.
# Without knowing which file system it's formatted with, I can't mount it.
#
# Quick way to prep drive: format with ReiserFS, then add 'clear-me' folder.
#
# 1.0first draft
# 1.1add logging, improve comments
# 1.2adapt for User.Scripts, extend wait to 60 seconds
# 1.3add progress display; confirm by key (no wait) if standalone; fix logger
# 1.4only add progress display if unRAID version >= 6.2
# 2.0 - This is an update/fork of RobJ script by user Majyk Oyster on forums.unraid.net.
	#Change log :
	#Version check adapted for Unraid  6.10 and up.
	#Check if unmount successful before clearing.
	#Send mail notification when clearing finished.
	#Script adapted for more then 9 drives.
	#Dead code removed.
	#Code simplified and optimized.
	#Ambiguous variables renamed.
	#Progression of the dd command sent to the GUI log 
# 2.1 - Fix to clear the found disk

  #############
 # Functions #
#############

funDisplay () {
	echo -e "\n`date +%d/%m" "%T` : $1\n"
	logger -t$tag "$2"
}

funMail() {
     /usr/local/emhttp/webGui/scripts/notify -i normal -s "$1" -d "$2"
}

  #############
 # Variables #
#############

scriptVersion="2.0"
marker="clear-me"
tag="clear_array_drive"
started=0
ddArg="" #used for dd command
found=0
wait=60
scriptDir=$(dirname "$0")
userscriptstDir="/tmp/user.scripts"
# Colors management
if [[ "$scriptDir" == "$userscriptstDir"* ]]; then # running from User.Scripts GUI. Sets html colors.
	colorStart="<font color=blue>"
	colorFinish="</font>"
else # running from CLI. Sets shell colors
	colorStart="\x1b[36;01m"
	colorFinish="\x1b[39;49;00m"
fi

  ##########
 # Checks #
##########

funDisplay "Clear an unRAID array data drive v$scriptVersion."

# check unRAID version
unraidVersion=`cat /etc/unraid-version | cut -d '"' -f 2`
majorVersion=`echo $unraidVersion | cut -d "." -f 1`
minorVersion=`echo $unraidVersion | cut -d "." -f 2`

if [ $majorVersion -eq 6 -a $minorVersion -ge 2 ]; then
	ddArg="status=progress"
else
	funDisplay "This script was not validated for this version of Unraid ($majorVersion.$minorVersion.x)."
	if [[ "$scriptDir" == "$userscriptstDir"* ]]; then # running in User.Scripts
		funDisplay "You have $wait seconds to cancel this script (click the red X, top right)\n"
		sleep $wait
	else #running from CLI
		echo  "Press ! to proceed. Any other key aborts, with no changes made. "
		ch=""
		read -n 1 ch
		echo -e -n "\r                                                                  \r"
		if [ "$ch" != "!" ]; then
			exit
		fi
	fi
fi

# checks if array is started
disks=`ls -d /mnt/* | grep disk | grep -v disks`
drivesNumber=`echo "$disks" | wc -l`

if [ $drivesNumber == 0 ]; then
	funDisplay "ERROR:  Array must be started before using this script. Exiting."
	exit
fi

# checks if a disk is ready for clearing

funDisplay "Checking all array data drives (may need to spin them up)..."

for disk in $disks; do
	itemsList=`ls -A $disk 2> /dev/null`
	itemsNumber=`echo "$itemsList" | wc -l`

	# test for marker and emptiness
	if [ $itemsNumber -eq 1 -a "$itemsList" == "$marker" ]; then
		itemsSize=`du -s $disk | awk '{print $1}'`
		if [ $itemsSize -eq 0 ]; then
    		foundDisk=$disk
			((found++))
		fi
	fi
done

# No drive or multiple drives found to clear
if [ $found -eq 0 ]; then
	funDisplay "Checked $drivesNumber drives, did not find an empty drive ready and marked for clearing!
	To use this script, the drive must be completely empty first, no files or folders left on it.
	Then a single folder should be created on it with the name 'clear-me', exactly 8 characters, 7 lowercase and 1 hyphen.
	This script is only for clearing unRAID data drives, in preparation for removing them from the array. It does not add a Preclear signature."
	exit
elif [ $found -ge 2 ]; then
	funDisplay "Checked $drivesNumber drives, found multiple empty drives ready and marked for clearing!
	To use this script, the drive must be completely empty first, no files or folders left on it. 
	Then a single folder should be created on it with the name 'clear-me', exactly 8 characters, 7 lowercase and 1 hyphen.
	This script is only for clearing unRAID data drives, in preparation for removing them from the array. It does not add a Preclear signature."
	exit
else
	disk=$foundDisk
	deviceNumber=`echo $disk | cut -d "/" -f 3 | cut -c 5- `
	device="/dev/md$deviceNumber"
	funDisplay "Device found : $colorStart $device $colorFinish"
fi


  ############
 # Warnings #
############

# First, warn about the clearing, and give them a chance to abort
funDisplay "Found a marked and empty drive to clear: $colorStart Disk $deviceNumber $colorFinish ( $device )
* Disk $disk will be unmounted first.
* Then zeroes will be written to the entire drive.
* Parity will be preserved throughout.
* Clearing while updating Parity takes a VERY long time!
* The progress of the clearing will not be visible until it's done!
* When complete, Disk $disk will be ready for removal from array.
* Commands to be executed:
\t* $colorStart umount $disk $colorFinish
\t* $colorStart dd bs=1M if=/dev/zero of=$device $ddArg $colorFinish"

if [[ "$scriptDir" == "$userscriptstDir"* ]]; then # running in User.Scripts
	funDisplay "You have $wait seconds to cancel this script (click the red X, top right)\n"
	sleep $wait
else #running from CLI
	funDisplay "Press ! to proceed. Any other key aborts, with no changes made. "
	ch=""
	read -n 1 ch
	echo -e -n "\r                                                                  \r"
	if [ "$ch" != "!" ]; then
		exit
	fi
fi

  #############
 # Preparing #
#############

funDisplay "Unmounting $disk ... \r"

funDisplay "Unmounting Disk $disk (command: umount $disk ) ..."
if `umount $disk`; then # unmount success
	funDisplay "Disk $disk unmounted successfully."
else # unmount failure
	funDisplay "Disk $disk unmount failed. Exiting."
	exit
fi

  ############
 # Clearing #
############

funDisplay "Clearing Disk $disk  (command: dd bs=1M if=/dev/zero of=$device $ddArg ) ..."
startTime="`date +%c`"

if [ ! -z `cat $userscriptstDir/tmpScripts/*/log.txt 2> /dev/null | grep "Clear an unRAID array data drive"`]; then 
	logFile="ls $userscriptstDir/tmpScripts/*/log.txt"
	dd bs=1M if=/dev/zero of=$device $ddArg >> $logFile
else
	dd bs=1M if=/dev/zero of=$device $ddArg
fi

endTime="`date +%c`"

  ########
 # Done #
########

funDisplay "Clearing Disk $disk is complete"
funDisplay "Started @ $startTime\nFinished @ $endTime"
echo -e "A message saying \"error writing ... no space left\" is expected, NOT an error.
Unless errors appeared, the drive is now cleared!
Because the drive is now unmountable, the array should be stopped, and the drive removed (or reformatted)."

funMail "Clearing complete on $device." "Clearing complete on $device.\nStarted @ $startTime\nFinished @ $endTime"

 

Edited October 3, 2022 by Majyk Oyster

[Majyk Oyster]

On 7/11/2022 at 2:27 AM, Darksurf said:

What is up with the zero drive script? It immediately gives up.

 

Check my post above, I think it might help. 

[Majyk Oyster]

After removing devices from the array, I needed to shred any trace of previous data to sell the HDDs, so I made this.

 

#!/bin/bash
# The purpose of this script is to remove every trace of the data that was once on a device (aka data shredding), so it can be discarded or sold. Multiple devices can be shredded simultaneously (not one after another).
# How the script works :
#     - Using the Unassigned devices plugin, format and mount (if needed) the devices you want to shred.
#     - If any data remains on the devices, remove it completely.
#     - Create a "shred-me" folder on each device you want to shred. Any other device will be ignored.
#     - Launch the script via the "User Scripts" plugin (preferably in background) or via CLI.
#     - A list of the devices correctly prepared for shredding will be displayed.
#     - Devices to shred will be unmounted, and won't be remounted after shredding.
#     - Shredding will then start on all relevant devices simultaneously. Progression will be visible in real time for each device.
#     - A notification will be sent on completion, using Unraid notification system.
#
# Version 1.0 - This is a script by user Majyk Oyster on forums.unraid.net. Some elements of RobJ's Clear An unRaid Data Drive were used. 

  #############
 # Functions #
#############

funDisplay () {
	echo -e "\n`date +%d/%m" "%T` : $1\n"
	logger -t$tag "$2"
}

funMail() {
     /usr/local/emhttp/webGui/scripts/notify -i normal -s "$1" -d "$2"
}

  #############
 # Variables #
#############

# Sets the number of passes. 3 is standard, you can increase the value for even safer data removal.
passes=3

scriptVersion="1.0"
marker="shred-me"
tag="shred_unassigned_device"
started=0
found=0
wait=60
scriptDir=$(dirname "$0")
userscriptstDir="/tmp/user.scripts"
# Colors management
if [[ "$scriptDir" == "$userscriptstDir"* ]]; then # running from User.Scripts GUI. Sets html colors.
	colorStart="<font color=blue>"
	colorFinish="</font>"
else # running from CLI. Sets shell colors
	colorStart="\x1b[36;01m"
	colorFinish="\x1b[39;49;00m"
fi

  ##########
 # Checks #
##########

funDisplay "Shredder v$scriptVersion."
funDisplay "The purpose of this script is to remove every trace of the data that was once on a device (aka data shredding), so it can be discarded or sold. Multiple devices can be shredded simultaneously."

# checks if a disk is ready for shredding

disks=`ls -d /mnt/disks/* 2> /dev/null`
devicesToShred=""

funDisplay "Checking all unassigned devices..."

for disk in $disks; do
	itemsList=`ls -A $disk 2> /dev/null`
	itemsNumber=`echo "$itemsList" | wc -l`

	# test for marker and emptiness
	if [ $itemsNumber -eq 1 -a "$itemsList" == "$marker" ]; then
		itemsSize=`du -s $disk | awk '{print $1}'`
		if [ $itemsSize -eq 0 ]; then
			disksToShred="$disksToShred $disk"
			device=`df | grep "$disk" | awk '{print $1}' | sed 's/.$//'`
			devicesToShred="$devicesToShred $device"
			((found++))
		fi
	fi
done

# No drive or multiple drives found to clear
if [ $found -eq 0 ] || [ -z "${devicesToShred// }" ]; then
	funDisplay "Did not find an empty drive ready and marked for shredding.
	To use this script, the drive must be completely empty first, no files or folders left on it.
	Then a single folder should be created on it with the name 'shred-me', exactly 8 characters, 7 lowercase and 1 hyphen.
	This script is only for safely removing all data from unassigned devices, typically after removing them from the array."
	exit
else
	funDisplay "Device(s) found : $colorStart $devicesToShred $colorFinish"
fi


  ############
 # Warnings #
############

funDisplay "Found marked and empty drive(s) to shred.
\t* Device(s) will be unmounted first.
\t* Then random data will be written to the entire device(s) $passes time(s).
\t* When complete, all data from device(s) will be cleanly erased."


echo -e "\n Commands to be executed:"
for diskToShred in $disksToShred; do
	echo -e "\t* $colorStart umount $diskToShred $colorFinish"
done

for deviceToShred in $devicesToShred; do
	echo -e "\t* $colorStart shred -vf $deviceToShred $colorFinish"
done

if [[ "$scriptDir" == "$userscriptstDir"* ]]; then # running in User.Scripts
	funDisplay "You have $wait seconds to cancel this script (click the red X, top right)\n"
	sleep $wait
else #running from CLI
	funDisplay "Press ! to proceed. Any other key aborts, with no changes made. "
	ch=""
	read -n 1 ch
	echo -e -n "\r                                                                  \r"
	if [ "$ch" != "!" ]; then
		exit
	fi
fi




  #############
 # Preparing #
#############

funDisplay "Unmounting device(s)... \r"

for diskToShred in $disksToShred; do
	if `umount $diskToShred`; then # unmount success
		funDisplay "Disk $diskToShred unmounted successfully."
	else # unmount failure
		funDisplay "Disk $diskToShred unmount failed. Exiting."
		exit
	fi
done

  #############
 # Shredding #
#############

funDisplay "Shredding Device(s)..."
startTime="`date +%c`"

if [ ! -z `cat $userscriptstDir/tmpScripts/*/log.txt 2> /dev/null | grep "Shredding unassigned devices"`]; then 
	logFile="ls $userscriptstDir/tmpScripts/*/log.txt"
else
	logFile="/tmp/shredLog.txt"
	touch $logFile
fi

for deviceToShred in $devicesToShred; do
	shred -vf $deviceToShred | tee -a $logFile &
done

wait

endTime="`date +%c`"

  ########
 # Done #
########

funDisplay "Shredding is complete"
funDisplay "Started @ $startTime\nFinished @ $endTime"
funDisplay "Unless errors appeared, the device(s) is/are now shredded!"

funMail "Shredding complete on device(s) $devicesToShred." "Shredding complete on following device(s):
$devicesToShred

Started @ $startTime
Finished @ $endTime
"

 

Your devices to shred should look something like that before using the script (formated, mounted) :

 

Here's how the script looks like in User Scripts :

Edited July 31, 2022 by Majyk Oyster

[JonathanM]

10 hours ago, Majyk Oyster said:

I needed to shred any trace of previous data to sell the HDDs, so I made this.

The existing preclear plugin for Unassigned devices does a good job of erasing drives, with the added benefit of ensuring that the final pass actually was written correctly.

[Majyk Oyster]

4 hours ago, JonathanM said:

The existing preclear plugin for Unassigned devices does a good job of erasing drives, with the added benefit of ensuring that the final pass actually was written correctly.

I checked the documentation on that plugin, but if it actually renders the previous data unrecoverable the info eluded me. Zeroing a device doesn't prevent a potentially malicious person from using data recovery tools to obtain potentially private data. The previous magnetic state of the device has to be randomly altered several times to obtain a satisfactory protection.

Edited July 31, 2022 by Majyk Oyster

[JonathanM]

10 minutes ago, Majyk Oyster said:

Zeroing a device doesn't prevent a potentially malicious person from using data recovery tools to obtain potentially private data.

Quote

The Erase All the Disk option uses random data to wipe out the drive; the resulting drive can't be quickly added to the array. If you want to add if after erase, you must select Erase and Clear the Disk.

 

[JonathanM]

14 minutes ago, Majyk Oyster said:

The previous magnetic state of the device has to be randomly altered several times to obtain a satisfactory protection.

You can select multiple erase passes with the preclear plugin that write random data to the drive. As many passes as you feel safe with, and as an added bonus you can do a standard clear with verification, which gives you confidence that the final write was actually honored and not a placebo where the write was accepted but the subsequent read still returned old data.

[Majyk Oyster]

I just watched Spaceinvader One video about the plugin, and I indeed saw that there was an option for multiple cycles (couldn't find it mentioned in the plugin thread).

 

I don't want to sound like I'm arguing for the sake of it, but I just noticed Squid mentioned on the Preclear plugin thread that it was incompatible with Unraid versions 6.9.0+ and should be uninstalled. So there's that. 

 

As for the final check for the actual destruction of the data, the fact that "successful" shred commands could actually fail to destruct the data beyond recognition didn't occur to me. I'd have to look into that.

[JonathanM]

31 minutes ago, Majyk Oyster said:

I just noticed Squid mentioned on the Preclear plugin thread that it was incompatible with Unraid versions 6.9.0+ and should be uninstalled.

This is the replacement option, it remains compatible using mostly the same base code. Read the first post.

 

[JonathanM]

34 minutes ago, Majyk Oyster said:

As for the final check for the actual destruction of the data, the fact that "successful" shred commands could actually fail to destruct the data beyond recognition didn't occur to me. I'd have to look into that.

If the disk is working properly, it won't happen. However... the situations that call for permanent erasure are often when a disk is no longer working reliably. If the software based erasure isn't reliable, the procedure must be escalated to physical destruction. If you like playing with powerful magnets, orderly disassembly is recommended, followed by shattering the platters. If you don't have time for that, 5.65 or larger orifices through the platters is satisfying and quite reliable.

[Majyk Oyster]

14 minutes ago, JonathanM said:

If the disk is working properly, it won't happen. However... the situations that call for permanent erasure are often when a disk is no longer working reliably. If the software based erasure isn't reliable, the procedure must be escalated to physical destruction. If you like playing with powerful magnets, orderly disassembly is recommended, followed by shattering the platters. If you don't have time for that, 5.65 or larger orifices through the platters is satisfying and quite reliable.

 

I actually have access to an actual certified disk shredder. I work for a managed security services provider, specialized in hosting critical data (for hospitals, health insurances, ministry of defense...). I haven't worked as a sys admin for a while, but my colleagues in the datacenters can turn my drives into coarse powder if need be. 

Edited July 31, 2022 by Majyk Oyster

[kizer]

Us Poor Folk would just use a drill or a hammer. lol

[hernandito]

Has anyone created a script that renews the LetsEncrypt certificate? Would love to implement.
 

thank you.