Additional Scripts For User.Scripts Plugin


Recommended Posts

@zacharyd3

 

may add something like this for his script

i added this to my tvheadend script in case my file is in progress writing ... in case of handbrake i would set the seconds pretty high,

only an idea in case the script runs while handbrake is in progress ...

instead using a static value with sleep ...

 

endfile=<file to move>

waittime=<seconds to wait in loop if busy>

 

        while [ -n "$(lsof "$endfile")" ]
            do
            sleep $waittime
            echo "$endfile in use"
        done

        echo "File $endfile free"

Link to comment

Hi,

 

I created my unraid server in January of this year and it blew me! Really.

 

One thing was missing and some of these user scripts filled the gap!

So I wanted to express my gratitude to all who wrote some scripts then shared.

 

Thanks everyone :D

Link to comment

Hi all,

 

I am attempting to shrink my array from 4 x 4tb drives to 3 x 10tb drives. I have replaced 2 of the 4tb drives with 2 10tb drives and each time did a parity sync and am now looking to remove/shrink the one 4tb drive that is totally empty before swapping the last 4tb drive for a 10tb. 

 

I have read the page on doing the shrink via the user scripts plugin and was able to start the script with no problems via the plugin. I would like to make sure I don’t screw anything up in the process and am looking for clarification. 

 

I left the user scripts page open displaying “closing out this window will cancel the script” but I was under the impression that I was supposed to be able to see a progress bar. How do I see that progress bar, do I have to type ! to see it?

 

It looks like there is a parity sync running, how can I verify the script is running properly as well and roughly how long will it take in peoples experience. 

 

Once this script/parity sync is done and I go to tools and do a new config, set the disk I want to remove “disk2”to no device and start the array will that disk disappear?

 

I do not see the checkbox mentioned in the instructions that says “parity is already valid” is this on the new config page or the main page where I do the drive assignments? Will this only show up after the current parity sync completes?

 

thanks in advance for any input. 

 

 

Link to comment
2 hours ago, nanoblock said:

I have replaced 2 of the 4tb drives with 2 10tb drives and each time did a parity sync

I am assuming here you mean you did a data rebuild each time. Parity sync actually means a rebuild of the parity disk. Perhaps you meant you did a parity check after the data rebuild?

 

2 hours ago, nanoblock said:

It looks like there is a parity sync running, how can I verify the script is running properly as well and roughly how long will it take in peoples experience. 

What makes you think a parity sync is running? Are you sure your data rebuilds completed? Do you perhaps mean a parity check is running? 

 

2 hours ago, nanoblock said:

I do not see the checkbox mentioned in the instructions that says “parity is already valid” is this on the new config page or the main page where I do the drive assignments? Will this only show up after the current parity sync completes?

There shouldn't be a parity sync running, but leaving aside that possible confusion. After you do a New Config, it will give you a parity valid checkbox when you go to Start the array.

 

Since there may be some confusion, it's probably a good idea to hang on to the original disks until you make sure all your data is there.

Link to comment

Hi,

 

The entire process i did so far was the following:

 

1. Stop array pull the 4tb parity drive and replace with 10tb drive, reassign in the main screen and start array. it did its rebuild/sync and after about 20hours completed.

 

2. Stop array pulled disk3 drive and replace wiith 10tb drive, reassign disk 3 in the main screen and start array. it did its rebuild/sync and after about 20hours completed.

 

3. I stopped the aray and then set disk2 to no device and then started the array thinking that since its empty and unused it would not be more complicated then that. It didn't remove the drive and then I did some research and found this clear an array drive method so I started the array back up and did step 4.

 

4. Installed  user scripts plugin and downloaded clear_an_array_drive.zip and created a folder in the correct plugins directory "clear an array drive" and put the script and description file in there. Additionally added the clear-me folder in /mnt/users/disk2 which was totally empty and then started the plugin. 

 

Now after all these steps I am at the stage of having a running parity sync which I am not sure if its part of the clear an array process or because I set disk 2 to no device and started the array as well as a 10tb parity drive a 10tb disk 3, an empty 4tb disk 2 which i am trying to remove and a partially full disk 1 which I plan on swapping with my last 10tb drive once disk 2 is full removed. 

 

When you say "After you do a New Config, it will give you a parity valid checkbox when you go to Start the array." where will i see this checkbox, after i hit the button start array, or is it somewhere on the main page or is it on the new config page?

 

Is the way I am doing this, thus far ok and how can I see the progress of the clean an array disk?

 

PS

 

This is not an old array with a lot of data, its pretty new I have just not realized all the features of unraid and now want to use the server for a gaming / nas box and having 3 drives allows me to remove a drive cage so i can fit a full length gpu with the added benefit of 20tb usable vs 12tb usable so the only drive that has data on it are the cache drives and disk1.

 

 

IMG_7430.PNG

IMG_7428.PNG

IMG_7429.PNG

 

PS

 

Sorry if this is long winded but just trying to give all the background info. My end goal is to have just the 3x 10tb drives.

Edited by nanoblock
Link to comment
7 hours ago, nanoblock said:

added the clear-me folder in /mnt/users/disk2

Are you sure this is what you did? Because that can't possibly be the correct instructions. The folder path /mnt/user/disk2 (not users) is actually a user share named "disk2", not the actual disk2. And that user share named disk2 could exist on any disk. The path for disk2 would be /mnt/disk2. And the path /mnt/users/disk2 would actually be in memory.

 

Based on everything else you said, and those screenshots, I would say that you have rebuilt the empty disk2, not cleared it. It's unclear what if anything the script could have actually done.

 

And, in case you don't know, an empty disk is not a clear disk. An empty disk has a filesystem on it that contains an empty top level folder. A clear disk has nothing but zeros on it, so no filesystem.

 

Personally, I usually recommend people not even bother with that "clear then remove script". It should work if you understand how to use it and do everything correctly, but it doesn't really save any time. Instead, you can just New Config and not assign the disk(s) you want to remove, then let it rebuild parity. And it would certainly be simpler and faster than what you have already done, since you still haven't accomplished anything.

Link to comment

In fact, you could have removed disk2 (and moved disk3 to the disk2 slot if you wanted) and then New Config and upsize parity all in step 1 and that part would be done and you could then upsize disk3 (or disk2 if you had moved it to that slot).

 

You still have options on how to proceed from here. You can try to preserve parity by using the script again to clear disk2, and then New Config, remove the disk, move disk3 down to slot 2 if you want, and parity would still be valid.

 

Or, you could just New Config without the empty disk2, move disk3 to slot 2 if you want, and let it rebuild parity.

 

Either way will be a similar amount of time. The script has the advantage that parity is valid throughout, but if everything else is working well, and you have backups of anything important and irreplaceable (you should regardless) then there is little risk to rebuilding parity.

 

Let us know what you want to do, and tell us exactly how you plan to proceed so we can make sure you have it all correct.

Link to comment
On 4/6/2019 at 1:52 AM, alturismo said:

@zacharyd3

 

may add something like this for his script

i added this to my tvheadend script in case my file is in progress writing ... in case of handbrake i would set the seconds pretty high,

only an idea in case the script runs while handbrake is in progress ...

instead using a static value with sleep ...

 

endfile=<file to move>

waittime=<seconds to wait in loop if busy>

 

        while [ -n "$(lsof "$endfile")" ]
            do
            sleep $waittime
            echo "$endfile in use"
        done

        echo "File $endfile free"

 

Where in the script would this go?

Link to comment

Thank you so much for the info this is very helpful. I was mistaken in the directory I mentioned I did actually create the folder in mnt/disk2 not /mnt/user/disk2 as you mentioned.

 

After the parity rebuilt drive2 I was trying to stop the array in order to do the new config but it looks like the array was just hanging in the stopping state. I hard rebooted the server and then canceled the parity check after the reboot. I then stopped the array and did a new config. 

In the drive assignments I set disk 3 to no device and set disk 2 to the new 10tb drive and checked the box that parity is valid and started the array. My current configuration is 10tb parity drive, 4tb disk1 and 10tb disk2. 

 

After all this it seems like all is fine but I decided to do a parity check with the write corrections checked before trying to swap out the last drive which actually has some data on it, disk1. Currently there is this parity check running and it seems to have found a ton of errors. I still can access my shares and all my data seems to be here. 

 

Based on what I have read about the errors it seems like this parity check with the write correction box checked should fix any parity issues so I can continue on to stopping the array and swapping disk1 out for the last 10tb I have. 

 

Does this sound ok?

 

I didn’t see any responses yesterday so I just went ahead with what I mentioned above thus far so I can’t go back and undo how I did things up to now. I wasn’t so concerned if things blew up because although inconvenient I don't have to much on the array yet and have all that data on Dropbox as a backup but would rather not have to go that route. 

 

any suggestions based on my current state would be helpful, and thanks again for helping. 

Edited by nanoblock
Link to comment

You apparently removed a disk that had not been cleared so parity was invalid at that point. A correcting parity check will make it valid again, but a parity rebuild would probably have been faster since it doesn't bother to check before rewriting parity. 

 

After it corrects parity, you should do a noncorrecting parity check, and that one should give you exactly zero sync errors. If not then further investigation is needed.

Link to comment
5 minutes ago, nanoblock said:

How do you do a parity rebuild for future reference?

A parity rebuild is the default if you have just done a New Config.

 

you can force it at any other time by:

  • stop the array
  • unassign a parity disk
  • start the array to make Unraid ‘forget’ the parity disk
  • stop the array
  • reassign the parity disk
  • start the array and the system will start building the parity disk from the contents of the other disks.

This is basically the same as the process that is used to force a data disk to be rebuilt to itself, but just applied to a parity disk instead.

Edited by itimpi
Link to comment
1 minute ago, nanoblock said:

How do you do a parity rebuild for future reference?

If you start the array after assigning a disk to parity, it will build parity on that disk. It can be a new disk, or it can be the same disk.

 

The way you would use the same disk is to start the array with the parity disk unassigned. That would make Unraid update the disk assignments so that no parity is assigned. Then when you assign that disk, it will build parity on it.

 

Stop array

Unassign parity

Start array with parity unassigned

Stop array

Reassign parity

Start array with parity assigned to build parity

 

Unraid also builds parity by default after a New Config, as long as you don't check the parity valid box.

Link to comment
1 minute ago, itimpi said:

This is basically the same as the process that is used to force a data disk to be rebuilt to itself, but just applied to a parity disk instead.

Rebuilding a data disk or rebuilding a parity disk is exactly the same parity calculation.

 

The concept of parity is used in many ways by computers and communications. Parity is simply a bit that allows a missing bit to be calculated from all the other bits. It is really pretty simple and understanding parity makes other things about how Unraid works with the disks easier to understand. The reason we are able to tell you what to do in these situations is mostly because we understand parity, not because we have done exactly the same thing ourselves, or memorized procedures.

 

Here is the wiki on parity:

 

https://wiki.unraid.net/UnRAID_6/Overview#Parity-Protected_Array

 

 

Link to comment

Thank you! Last thing the on this topic based on your last post. Once my parity check is done and everything is ok in order to replace the last 4tb drive should I do the following

 

shutdown the array

unassign disk 1 (the 4tb drive that has my data)

start array 

Stop the array

assign the new 10tb drive to disk 1

start array

Link to comment
3 hours ago, nanoblock said:

OK Thanks. Before I do this i was going to run a parity check again with the write errors unchecked, is there a faster way to do this or will this be another 20hr round before i can safely shutdown and swap the drive out?

Not much can be done about how long it takes to check every bit of 10TB. If 20 hrs is what it takes on your system, that is what it takes. And that time seems good to me for that size parity. I only have 6TB and it takes a little over 14 hours.

Link to comment
16 hours ago, IamSpartacus said:

 

Where in the script would this go?

Hi, before u move to look if file is busy, i updated my tvheadend move and update script now a little and dont use the loop as described, i use now

if file is busy do nothing, next cron will come ;) so i dont touch anything while there is any actions running.

 

i look for new records from tvheadend (.ts files), if there is one and NOT in use i start avidemux to read the file (prepare for my com cut, creating idx2),

if .ts and .idx2 is there i look for the .mkv (after i did my work cutting and remuxing), if the .mkv is there and NOT in use i move it now to a proper folder

(sample, season 02 instead season 2, 2 digit ...), update TVHeadend record path, remove the "old" .ts and .idx2 and empty season 2, update emby and plex.

 

the

if ! [ -n "$(lsof "$sourcefile")" ]; then

...

will trigger if NOT in use

else ...

 

i went away from the while loop cause it could take longer then my cron job and i dont want to run multiply instances now,

so like this i can run the cron like every 5 minutes and im good. sample from log

 

echo Hawaii Five-0 - S09E09 - Das letzte Kapitel.ts.idx2 already there...
Hawaii Five-0 - S09E09 - Das letzte Kapitel.mkv found, processing ...
File Hawaii Five-0 - S09E09 - Das letzte Kapitel.mkv free

... in the end its like i want it actually, i hope i didnt mess anything up ;)

im sure it could be done more simple, but as my scripting skills are very old ... ;)

 

my suggestion was to check if files are busy in case handbrake is still running ...

 

#!/bin/bash

startpath="/mnt/cache/Media/"	### main place where to look for
searchpattern="*.ts"			### what type of files to look for
matchpattern=".mkv"				### compare if exist - work done ?
extradelpattern=".idx2"			### extra delete type - here, avidemux file
seasonnaming="Season "			### Naming to check for season folders
mediafolder="/Media"			### add to path beginning for TVHeadend update
waittime=3						### time to wait if file is in use
embyapikey="embyapikeyhere"		### api key to update emby library



tvhuser="username"
tvhpass="password"
tvhip="192.168.1.2"

cd $startpath

find . -mindepth 2 -iname $searchpattern|sed "s|^\./||"|while read fname; do
	echo "$fname"
	sourcefile=$(basename "$fname")
	filenameonly="${sourcefile%.*}"
	endfile="$filenameonly$matchpattern"
	extrafile="$sourcefile$extradelpattern"
	filepath=$(dirname "$fname")
	seasonpath=$(basename "$filepath")

	if [[ $seasonpath == "$seasonnaming"* ]]; then
		seasonpathnew=$(echo "$seasonpath" | sed -E 's/\<[0-9]\>/0&/')
		seriespath=$(dirname "$filepath")
		tvhfinalpath="$mediafolder/$seriespath/$seasonpathnew/$endfile"
	else
		tvhfinalpath="$mediafolder/$filepath/$endfile"
	fi

	echo "output path is: $tvhfinalpath"

	cd "$startpath/$filepath"

	if ! [ -n "$(lsof "$sourcefile")" ]; then
		if ! [ -f "$extrafile" ]; then
			docker exec Avidemux /usr/bin/avidemux3_cli --load /storage/"$fname" --quit > /dev/null
			echo echo "$sourcefile loading in avidemux ..."
		else
			echo echo "$extrafile already there..."
		fi
	else
		echo echo "$sourcefile in use ..."
	fi

	if [ -f "$endfile" ]; then
		echo "$endfile found, processing ..."
	if ! [ -n "$(lsof "$endfile")" ]; then
		echo "File $endfile free"

		if [[ $seasonpath == "$seasonnaming"* ]]; then
			cd ..
			mkdir "$seasonpathnew"
			mv "$seasonpath/$endfile" "$seasonpathnew/"
		fi

		curl -s "http://"$tvhuser":"$tvhpass"@"$tvhip":9981/api/dvr/entry/filemoved" --data-urlencode "src=$mediafolder/$filepath/$sourcefile" --data-urlencode "dst=$tvhfinalpath"

		cd "$startpath/$filepath"
		rm "$sourcefile"
		rm "$extrafile"
		if [[ ! $seasonpath == "$seasonpathnew" ]]; then
			cd ..
			rmdir "$seasonpath"
		fi
	curl -s "http://192.168.1.2:32400/library/sections/1/refresh"
	curl -s "http://192.168.1.2:8096/emby/Library/Refresh?api_key=$embyapikey" -H "accept: */*" -d ""
	else
		echo "$endfile in use"
	fi
	else
		echo "$endfile not found."
	fi
done

 

Link to comment
  • 2 weeks later...

I found a new/updated plex NVDEC script off the Plex Forums, it allows you to set specific codecs that are allowed to be decoded by the GPU, the rest get CPU decoded.

 

#!/bin/bash

# This should always return the name of the docker container running plex - assuming a single plex docker on the system.
con="$(docker ps --format "{{.Names}}" | grep -i plex)"

echo -n "<b>Applying hardware decode patch... </b>"

# Check to see if patch script exists first.
exists=$(docker exec -i "$con" stat "/usr/lib/plexmediaserver/Plex Transcoder2" >/dev/null 2>&1; echo $?)

if [ "$exists" -eq 1 ]; then 
  # If it doesn't, we run the clause below
  # For Legibility and future updates - if needed, use a heredoc to create the wrapper:
  docker exec -i "$con" /bin/sh -c 'sed "s/\(^\t\t\)//g" > "plex-nvdec-patch.sh";' <<'@EOF'
#!/bin/bash

PLEX_PATH="/usr/lib/plexmediaserver"
CODECS=()
ALLOWED_CODECS=("h264" "hevc" "mpeg2video" "mpeg4" "vc1" "vp8" "vp9")
USAGE="Usage: $(basename $0) [OPTIONS]
  -p, --path        Manually define the path to the folder containing the Plex
                      Transcoder
  -c, --codec       Whitelistes codec to enable NVDEC for. When defined, NVDEC
                      will only be enabled for defined codecs. Use -c once per
                      codec
  -u, --uninstall   Remove the NVDEC patch from Plex

Available codec options are:
  h264 (default)       H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10
  hevc (default)       H.265 / HEVC (High Efficiency Video Coding)
  mpeg2video           MPEG-2 video
  mpeg4                MPEG-4 part 2
  vc1                  SMPTE VC-1
  vp8  (default)       On2 VP8
  vp9  (default)       Google VP9"

contains() {
    typeset _x;
    typeset -n _A="$1"
    for _x in "${_A[@]}" ; do
        [ "$_x" = "$2" ] && return 0
    done
    return 1
}

while (( "$#" )); do
  case "$1" in
    -p|--path)
      PLEX_PATH=$2
      shift 2
      ;;
    -c|--codec)
      if contains ALLOWED_CODECS "$2"; then
        CODECS+=("$2")
      else
        echo "ERROR: Incorrect codec $2, please refer to --help for allowed list" >&2
        exit 1
      fi
      shift 2
      ;;
    -u|--uninstall)
      uninstall=1
      shift 1
      ;;
    -h|--help|*)
      echo "$USAGE"
      exit
      ;;
  esac
done

if [ ${#CODECS[@]} -eq 0 ]; then
  CODECS=("h264" "hevc" "vp8" "vp9")
fi

if [ "$EUID" -ne 0 ]; then
  echo "Please run as root or with sudo"
  exit 1
fi

if [ ! -f "$PLEX_PATH/Plex Transcoder" ]; then
  if [ -f "/usr/lib64/plexmediaserver/Plex Transcoder"]; then
    PLEX_PATH="/usr/lib64/plexmediaserver"
  else
    echo "ERROR: Plex transcoder not found. Please ensure plex is installed and use -p to manually define the path to the Plex Transcoder" >&2
    exit 1
  fi
fi

pcheck=$(tail -n 1 "$PLEX_PATH/Plex Transcoder" | tr -d '\0')
if [ "$uninstall" == "1" ]; then
  if [ "$pcheck" == "##patched" ]; then
    if pgrep -x "Plex Transcoder" >/dev/null ; then
      echo "ERROR: Plex Transcoder is currently running. Please stop any open transcodes and run again" >&2
      exit 1
    fi
    mv "$PLEX_PATH/Plex Transcoder2" "$PLEX_PATH/Plex Transcoder"
    echo "Uninstall of patch complete!"
    exit
  else
    echo "ERROR: NVDEC Patch not detected as installed. Cannot uninstall."
    exit 1
  fi
fi

if [ "$pcheck" == "##patched" ]; then
  echo "Patch has already been applied! Reapplying wrapper script"
else
  if pgrep -x "Plex Transcoder" >/dev/null ; then
    echo "ERROR: Plex Transcoder is currently running. Please stop any open transcodes and run again" >&2
    exit 1
  fi
  mv "$PLEX_PATH/Plex Transcoder" "$PLEX_PATH/Plex Transcoder2"
fi

for i in "${CODECS[@]}"; do
  cstring+='"'"$i"'" '
done
cstring="${cstring::-1}"

cat > "$PLEX_PATH/Plex Transcoder" <<< '#!/bin/bash'
cat >> "$PLEX_PATH/Plex Transcoder" <<< 'PLEX_PATH="'"$PLEX_PATH"'"'
cat >> "$PLEX_PATH/Plex Transcoder" <<< 'ALLOWED_CODECS=('"$cstring"')'
cat >> "$PLEX_PATH/Plex Transcoder" <<< '
contains() {
  typeset _x;
  typeset -n _A="$1"
  for _x in "${_A[@]}" ; do
    [ "$_x" = "$2" ] && return 0
  done
  return 1
}

allowed_codec() {
  for i in "$@"; do
    if [ "-i" == "$i" ]; then
      return 1
    elif contains ALLOWED_CODECS "$i"; then
      return 0
    fi
  done
  return 1
}

if allowed_codec $*; then
  exec "$PLEX_PATH/Plex Transcoder2" -hwaccel nvdec "$@"
else
  exec "$PLEX_PATH/Plex Transcoder2" "$@"
fi

##patched'

chmod +x "$PLEX_PATH/Plex Transcoder"

echo "Patch applied successfully!"

@EOF
  
  # chmod the new script to be executable:
  docker exec -i "$con" chmod +x "./plex-nvdec-patch.sh"

  # now we execute the script:
  docker exec -i "$con" bash -c "./plex-nvdec-patch.sh"

  echo '<font color="green"><b>Done!</b></font>' # Green means go!
else
  # If we ended up here, the patch didn't even try to run, presumably because the patch was already applied.
  echo '<font color="red"><b>Patch already applied!</b></font>' # Red means stop!
fi

 

To alter the defaults simply append "-c codec" to the execution command like so:

docker exec -i "$con" bash -c "./plex-nvdec-patch.sh -c h264 -c hevc -c vp8 -c vp9"

 

Source/Credits: https://github.com/revr3nd/plex-nvdec and @Xaero for his script here: https://gist.github.com/Xaero252/9f81593e4a5e6825c045686d685e2428

Edited by MowMdown
Link to comment
3 hours ago, MowMdown said:

*SNIP*

Had been meaning to implement this myself. I just updated my script on gist. The new script pulls the latest version of revr3nds script and applies it. It also supports a list of codecs in a variable near the top of the script if you need to change them for any reason.

Link to comment
8 hours ago, Xaero said:

Had been meaning to implement this myself. I just updated my script on gist. The new script pulls the latest version of revr3nds script and applies it. It also supports a list of codecs in a variable near the top of the script if you need to change them for any reason.

 There is a bug in your new code specifically:

 

command="/usr/lib/plexmediaserver/plex-nvdec-patch.sh"
if [ "$codec_arguments" ]; then
        #This line is recursive, it duplicates the codecs in the Plex Transcoder file (we don't need it)
        #command+="${codec_arguments}" 
	docker exec -i "$con" /bin/sh -c "${command}${codec_arguments}"
else
	docker exec -i "$con" /bin/sh -c "${command}"
fi

What ends up happening is that in the "Plex Transcoder" file, you end up with 

 

ALLOWED_CODECS=("h264" "hevc" "h264" "hevc")

Removing that line of code fixes the issue. You don't need to concatenate the "command+="${codec_arguments}" because it's already being taken care of at the next line where the docker exec code is taking place.

Edited by MowMdown
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.