IamSpartacus Posted April 5, 2019 Share Posted April 5, 2019 9 minutes ago, alturismo said: and you want to overwrite the old file then ? Correct. Quote Link to comment
alturismo Posted April 6, 2019 Share Posted April 6, 2019 @zacharyd3 may add something like this for his script i added this to my tvheadend script in case my file is in progress writing ... in case of handbrake i would set the seconds pretty high, only an idea in case the script runs while handbrake is in progress ... instead using a static value with sleep ... endfile=<file to move> waittime=<seconds to wait in loop if busy> while [ -n "$(lsof "$endfile")" ] do sleep $waittime echo "$endfile in use" done echo "File $endfile free" Quote Link to comment
cam217 Posted April 6, 2019 Share Posted April 6, 2019 Hi, I created my unraid server in January of this year and it blew me! Really. One thing was missing and some of these user scripts filled the gap! So I wanted to express my gratitude to all who wrote some scripts then shared. Thanks everyone Quote Link to comment
nanoblock Posted April 8, 2019 Share Posted April 8, 2019 Hi all, I am attempting to shrink my array from 4 x 4tb drives to 3 x 10tb drives. I have replaced 2 of the 4tb drives with 2 10tb drives and each time did a parity sync and am now looking to remove/shrink the one 4tb drive that is totally empty before swapping the last 4tb drive for a 10tb. I have read the page on doing the shrink via the user scripts plugin and was able to start the script with no problems via the plugin. I would like to make sure I don’t screw anything up in the process and am looking for clarification. I left the user scripts page open displaying “closing out this window will cancel the script” but I was under the impression that I was supposed to be able to see a progress bar. How do I see that progress bar, do I have to type ! to see it? It looks like there is a parity sync running, how can I verify the script is running properly as well and roughly how long will it take in peoples experience. Once this script/parity sync is done and I go to tools and do a new config, set the disk I want to remove “disk2”to no device and start the array will that disk disappear? I do not see the checkbox mentioned in the instructions that says “parity is already valid” is this on the new config page or the main page where I do the drive assignments? Will this only show up after the current parity sync completes? thanks in advance for any input. Quote Link to comment
trurl Posted April 8, 2019 Share Posted April 8, 2019 2 hours ago, nanoblock said: I have replaced 2 of the 4tb drives with 2 10tb drives and each time did a parity sync I am assuming here you mean you did a data rebuild each time. Parity sync actually means a rebuild of the parity disk. Perhaps you meant you did a parity check after the data rebuild? 2 hours ago, nanoblock said: It looks like there is a parity sync running, how can I verify the script is running properly as well and roughly how long will it take in peoples experience. What makes you think a parity sync is running? Are you sure your data rebuilds completed? Do you perhaps mean a parity check is running? 2 hours ago, nanoblock said: I do not see the checkbox mentioned in the instructions that says “parity is already valid” is this on the new config page or the main page where I do the drive assignments? Will this only show up after the current parity sync completes? There shouldn't be a parity sync running, but leaving aside that possible confusion. After you do a New Config, it will give you a parity valid checkbox when you go to Start the array. Since there may be some confusion, it's probably a good idea to hang on to the original disks until you make sure all your data is there. Quote Link to comment
nanoblock Posted April 8, 2019 Share Posted April 8, 2019 (edited) Hi, The entire process i did so far was the following: 1. Stop array pull the 4tb parity drive and replace with 10tb drive, reassign in the main screen and start array. it did its rebuild/sync and after about 20hours completed. 2. Stop array pulled disk3 drive and replace wiith 10tb drive, reassign disk 3 in the main screen and start array. it did its rebuild/sync and after about 20hours completed. 3. I stopped the aray and then set disk2 to no device and then started the array thinking that since its empty and unused it would not be more complicated then that. It didn't remove the drive and then I did some research and found this clear an array drive method so I started the array back up and did step 4. 4. Installed user scripts plugin and downloaded clear_an_array_drive.zip and created a folder in the correct plugins directory "clear an array drive" and put the script and description file in there. Additionally added the clear-me folder in /mnt/users/disk2 which was totally empty and then started the plugin. Now after all these steps I am at the stage of having a running parity sync which I am not sure if its part of the clear an array process or because I set disk 2 to no device and started the array as well as a 10tb parity drive a 10tb disk 3, an empty 4tb disk 2 which i am trying to remove and a partially full disk 1 which I plan on swapping with my last 10tb drive once disk 2 is full removed. When you say "After you do a New Config, it will give you a parity valid checkbox when you go to Start the array." where will i see this checkbox, after i hit the button start array, or is it somewhere on the main page or is it on the new config page? Is the way I am doing this, thus far ok and how can I see the progress of the clean an array disk? PS This is not an old array with a lot of data, its pretty new I have just not realized all the features of unraid and now want to use the server for a gaming / nas box and having 3 drives allows me to remove a drive cage so i can fit a full length gpu with the added benefit of 20tb usable vs 12tb usable so the only drive that has data on it are the cache drives and disk1. PS Sorry if this is long winded but just trying to give all the background info. My end goal is to have just the 3x 10tb drives. Edited April 8, 2019 by nanoblock Quote Link to comment
nanoblock Posted April 8, 2019 Share Posted April 8, 2019 It looks like the parity sync / data rebuild completed. Is there a way for me to see if the clean disk is done so I can stop the array and do a new configuration? Quote Link to comment
trurl Posted April 9, 2019 Share Posted April 9, 2019 7 hours ago, nanoblock said: added the clear-me folder in /mnt/users/disk2 Are you sure this is what you did? Because that can't possibly be the correct instructions. The folder path /mnt/user/disk2 (not users) is actually a user share named "disk2", not the actual disk2. And that user share named disk2 could exist on any disk. The path for disk2 would be /mnt/disk2. And the path /mnt/users/disk2 would actually be in memory. Based on everything else you said, and those screenshots, I would say that you have rebuilt the empty disk2, not cleared it. It's unclear what if anything the script could have actually done. And, in case you don't know, an empty disk is not a clear disk. An empty disk has a filesystem on it that contains an empty top level folder. A clear disk has nothing but zeros on it, so no filesystem. Personally, I usually recommend people not even bother with that "clear then remove script". It should work if you understand how to use it and do everything correctly, but it doesn't really save any time. Instead, you can just New Config and not assign the disk(s) you want to remove, then let it rebuild parity. And it would certainly be simpler and faster than what you have already done, since you still haven't accomplished anything. Quote Link to comment
trurl Posted April 9, 2019 Share Posted April 9, 2019 In fact, you could have removed disk2 (and moved disk3 to the disk2 slot if you wanted) and then New Config and upsize parity all in step 1 and that part would be done and you could then upsize disk3 (or disk2 if you had moved it to that slot). You still have options on how to proceed from here. You can try to preserve parity by using the script again to clear disk2, and then New Config, remove the disk, move disk3 down to slot 2 if you want, and parity would still be valid. Or, you could just New Config without the empty disk2, move disk3 to slot 2 if you want, and let it rebuild parity. Either way will be a similar amount of time. The script has the advantage that parity is valid throughout, but if everything else is working well, and you have backups of anything important and irreplaceable (you should regardless) then there is little risk to rebuilding parity. Let us know what you want to do, and tell us exactly how you plan to proceed so we can make sure you have it all correct. Quote Link to comment
IamSpartacus Posted April 9, 2019 Share Posted April 9, 2019 On 4/6/2019 at 1:52 AM, alturismo said: @zacharyd3 may add something like this for his script i added this to my tvheadend script in case my file is in progress writing ... in case of handbrake i would set the seconds pretty high, only an idea in case the script runs while handbrake is in progress ... instead using a static value with sleep ... endfile=<file to move> waittime=<seconds to wait in loop if busy> while [ -n "$(lsof "$endfile")" ] do sleep $waittime echo "$endfile in use" done echo "File $endfile free" Where in the script would this go? Quote Link to comment
nanoblock Posted April 9, 2019 Share Posted April 9, 2019 (edited) Thank you so much for the info this is very helpful. I was mistaken in the directory I mentioned I did actually create the folder in mnt/disk2 not /mnt/user/disk2 as you mentioned. After the parity rebuilt drive2 I was trying to stop the array in order to do the new config but it looks like the array was just hanging in the stopping state. I hard rebooted the server and then canceled the parity check after the reboot. I then stopped the array and did a new config. In the drive assignments I set disk 3 to no device and set disk 2 to the new 10tb drive and checked the box that parity is valid and started the array. My current configuration is 10tb parity drive, 4tb disk1 and 10tb disk2. After all this it seems like all is fine but I decided to do a parity check with the write corrections checked before trying to swap out the last drive which actually has some data on it, disk1. Currently there is this parity check running and it seems to have found a ton of errors. I still can access my shares and all my data seems to be here. Based on what I have read about the errors it seems like this parity check with the write correction box checked should fix any parity issues so I can continue on to stopping the array and swapping disk1 out for the last 10tb I have. Does this sound ok? I didn’t see any responses yesterday so I just went ahead with what I mentioned above thus far so I can’t go back and undo how I did things up to now. I wasn’t so concerned if things blew up because although inconvenient I don't have to much on the array yet and have all that data on Dropbox as a backup but would rather not have to go that route. any suggestions based on my current state would be helpful, and thanks again for helping. Edited April 9, 2019 by nanoblock Quote Link to comment
trurl Posted April 9, 2019 Share Posted April 9, 2019 You apparently removed a disk that had not been cleared so parity was invalid at that point. A correcting parity check will make it valid again, but a parity rebuild would probably have been faster since it doesn't bother to check before rewriting parity. After it corrects parity, you should do a noncorrecting parity check, and that one should give you exactly zero sync errors. If not then further investigation is needed. Quote Link to comment
nanoblock Posted April 9, 2019 Share Posted April 9, 2019 How do you do a parity rebuild for future reference? Quote Link to comment
itimpi Posted April 9, 2019 Share Posted April 9, 2019 (edited) 5 minutes ago, nanoblock said: How do you do a parity rebuild for future reference? A parity rebuild is the default if you have just done a New Config. you can force it at any other time by: stop the array unassign a parity disk start the array to make Unraid ‘forget’ the parity disk stop the array reassign the parity disk start the array and the system will start building the parity disk from the contents of the other disks. This is basically the same as the process that is used to force a data disk to be rebuilt to itself, but just applied to a parity disk instead. Edited April 9, 2019 by itimpi Quote Link to comment
trurl Posted April 9, 2019 Share Posted April 9, 2019 1 minute ago, nanoblock said: How do you do a parity rebuild for future reference? If you start the array after assigning a disk to parity, it will build parity on that disk. It can be a new disk, or it can be the same disk. The way you would use the same disk is to start the array with the parity disk unassigned. That would make Unraid update the disk assignments so that no parity is assigned. Then when you assign that disk, it will build parity on it. Stop array Unassign parity Start array with parity unassigned Stop array Reassign parity Start array with parity assigned to build parity Unraid also builds parity by default after a New Config, as long as you don't check the parity valid box. Quote Link to comment
trurl Posted April 9, 2019 Share Posted April 9, 2019 1 minute ago, itimpi said: This is basically the same as the process that is used to force a data disk to be rebuilt to itself, but just applied to a parity disk instead. Rebuilding a data disk or rebuilding a parity disk is exactly the same parity calculation. The concept of parity is used in many ways by computers and communications. Parity is simply a bit that allows a missing bit to be calculated from all the other bits. It is really pretty simple and understanding parity makes other things about how Unraid works with the disks easier to understand. The reason we are able to tell you what to do in these situations is mostly because we understand parity, not because we have done exactly the same thing ourselves, or memorized procedures. Here is the wiki on parity: https://wiki.unraid.net/UnRAID_6/Overview#Parity-Protected_Array Quote Link to comment
nanoblock Posted April 9, 2019 Share Posted April 9, 2019 Thank you! Last thing the on this topic based on your last post. Once my parity check is done and everything is ok in order to replace the last 4tb drive should I do the following shutdown the array unassign disk 1 (the 4tb drive that has my data) start array Stop the array assign the new 10tb drive to disk 1 start array Quote Link to comment
bonienl Posted April 9, 2019 Share Posted April 9, 2019 shutdown system replace disk1 start system assign disk1 start array Quote Link to comment
nanoblock Posted April 9, 2019 Share Posted April 9, 2019 OK Thanks. Before I do this i was going to run a parity check again with the write errors unchecked, is there a faster way to do this or will this be another 20hr round before i can safely shutdown and swap the drive out? Quote Link to comment
trurl Posted April 9, 2019 Share Posted April 9, 2019 3 hours ago, nanoblock said: OK Thanks. Before I do this i was going to run a parity check again with the write errors unchecked, is there a faster way to do this or will this be another 20hr round before i can safely shutdown and swap the drive out? Not much can be done about how long it takes to check every bit of 10TB. If 20 hrs is what it takes on your system, that is what it takes. And that time seems good to me for that size parity. I only have 6TB and it takes a little over 14 hours. Quote Link to comment
alturismo Posted April 10, 2019 Share Posted April 10, 2019 16 hours ago, IamSpartacus said: Where in the script would this go? Hi, before u move to look if file is busy, i updated my tvheadend move and update script now a little and dont use the loop as described, i use now if file is busy do nothing, next cron will come so i dont touch anything while there is any actions running. i look for new records from tvheadend (.ts files), if there is one and NOT in use i start avidemux to read the file (prepare for my com cut, creating idx2), if .ts and .idx2 is there i look for the .mkv (after i did my work cutting and remuxing), if the .mkv is there and NOT in use i move it now to a proper folder (sample, season 02 instead season 2, 2 digit ...), update TVHeadend record path, remove the "old" .ts and .idx2 and empty season 2, update emby and plex. the if ! [ -n "$(lsof "$sourcefile")" ]; then ... will trigger if NOT in use else ... i went away from the while loop cause it could take longer then my cron job and i dont want to run multiply instances now, so like this i can run the cron like every 5 minutes and im good. sample from log echo Hawaii Five-0 - S09E09 - Das letzte Kapitel.ts.idx2 already there... Hawaii Five-0 - S09E09 - Das letzte Kapitel.mkv found, processing ... File Hawaii Five-0 - S09E09 - Das letzte Kapitel.mkv free ... in the end its like i want it actually, i hope i didnt mess anything up im sure it could be done more simple, but as my scripting skills are very old ... my suggestion was to check if files are busy in case handbrake is still running ... #!/bin/bash startpath="/mnt/cache/Media/" ### main place where to look for searchpattern="*.ts" ### what type of files to look for matchpattern=".mkv" ### compare if exist - work done ? extradelpattern=".idx2" ### extra delete type - here, avidemux file seasonnaming="Season " ### Naming to check for season folders mediafolder="/Media" ### add to path beginning for TVHeadend update waittime=3 ### time to wait if file is in use embyapikey="embyapikeyhere" ### api key to update emby library tvhuser="username" tvhpass="password" tvhip="192.168.1.2" cd $startpath find . -mindepth 2 -iname $searchpattern|sed "s|^\./||"|while read fname; do echo "$fname" sourcefile=$(basename "$fname") filenameonly="${sourcefile%.*}" endfile="$filenameonly$matchpattern" extrafile="$sourcefile$extradelpattern" filepath=$(dirname "$fname") seasonpath=$(basename "$filepath") if [[ $seasonpath == "$seasonnaming"* ]]; then seasonpathnew=$(echo "$seasonpath" | sed -E 's/\<[0-9]\>/0&/') seriespath=$(dirname "$filepath") tvhfinalpath="$mediafolder/$seriespath/$seasonpathnew/$endfile" else tvhfinalpath="$mediafolder/$filepath/$endfile" fi echo "output path is: $tvhfinalpath" cd "$startpath/$filepath" if ! [ -n "$(lsof "$sourcefile")" ]; then if ! [ -f "$extrafile" ]; then docker exec Avidemux /usr/bin/avidemux3_cli --load /storage/"$fname" --quit > /dev/null echo echo "$sourcefile loading in avidemux ..." else echo echo "$extrafile already there..." fi else echo echo "$sourcefile in use ..." fi if [ -f "$endfile" ]; then echo "$endfile found, processing ..." if ! [ -n "$(lsof "$endfile")" ]; then echo "File $endfile free" if [[ $seasonpath == "$seasonnaming"* ]]; then cd .. mkdir "$seasonpathnew" mv "$seasonpath/$endfile" "$seasonpathnew/" fi curl -s "http://"$tvhuser":"$tvhpass"@"$tvhip":9981/api/dvr/entry/filemoved" --data-urlencode "src=$mediafolder/$filepath/$sourcefile" --data-urlencode "dst=$tvhfinalpath" cd "$startpath/$filepath" rm "$sourcefile" rm "$extrafile" if [[ ! $seasonpath == "$seasonpathnew" ]]; then cd .. rmdir "$seasonpath" fi curl -s "http://192.168.1.2:32400/library/sections/1/refresh" curl -s "http://192.168.1.2:8096/emby/Library/Refresh?api_key=$embyapikey" -H "accept: */*" -d "" else echo "$endfile in use" fi else echo "$endfile not found." fi done Quote Link to comment
nanoblock Posted April 12, 2019 Share Posted April 12, 2019 thanks for all the help guys, my server is now running on the 3 x 10tb drives and everything is looking healthy. Quote Link to comment
MowMdown Posted April 24, 2019 Share Posted April 24, 2019 (edited) I found a new/updated plex NVDEC script off the Plex Forums, it allows you to set specific codecs that are allowed to be decoded by the GPU, the rest get CPU decoded. #!/bin/bash # This should always return the name of the docker container running plex - assuming a single plex docker on the system. con="$(docker ps --format "{{.Names}}" | grep -i plex)" echo -n "<b>Applying hardware decode patch... </b>" # Check to see if patch script exists first. exists=$(docker exec -i "$con" stat "/usr/lib/plexmediaserver/Plex Transcoder2" >/dev/null 2>&1; echo $?) if [ "$exists" -eq 1 ]; then # If it doesn't, we run the clause below # For Legibility and future updates - if needed, use a heredoc to create the wrapper: docker exec -i "$con" /bin/sh -c 'sed "s/\(^\t\t\)//g" > "plex-nvdec-patch.sh";' <<'@EOF' #!/bin/bash PLEX_PATH="/usr/lib/plexmediaserver" CODECS=() ALLOWED_CODECS=("h264" "hevc" "mpeg2video" "mpeg4" "vc1" "vp8" "vp9") USAGE="Usage: $(basename $0) [OPTIONS] -p, --path Manually define the path to the folder containing the Plex Transcoder -c, --codec Whitelistes codec to enable NVDEC for. When defined, NVDEC will only be enabled for defined codecs. Use -c once per codec -u, --uninstall Remove the NVDEC patch from Plex Available codec options are: h264 (default) H.264 / AVC / MPEG-4 AVC / MPEG-4 part 10 hevc (default) H.265 / HEVC (High Efficiency Video Coding) mpeg2video MPEG-2 video mpeg4 MPEG-4 part 2 vc1 SMPTE VC-1 vp8 (default) On2 VP8 vp9 (default) Google VP9" contains() { typeset _x; typeset -n _A="$1" for _x in "${_A[@]}" ; do [ "$_x" = "$2" ] && return 0 done return 1 } while (( "$#" )); do case "$1" in -p|--path) PLEX_PATH=$2 shift 2 ;; -c|--codec) if contains ALLOWED_CODECS "$2"; then CODECS+=("$2") else echo "ERROR: Incorrect codec $2, please refer to --help for allowed list" >&2 exit 1 fi shift 2 ;; -u|--uninstall) uninstall=1 shift 1 ;; -h|--help|*) echo "$USAGE" exit ;; esac done if [ ${#CODECS[@]} -eq 0 ]; then CODECS=("h264" "hevc" "vp8" "vp9") fi if [ "$EUID" -ne 0 ]; then echo "Please run as root or with sudo" exit 1 fi if [ ! -f "$PLEX_PATH/Plex Transcoder" ]; then if [ -f "/usr/lib64/plexmediaserver/Plex Transcoder"]; then PLEX_PATH="/usr/lib64/plexmediaserver" else echo "ERROR: Plex transcoder not found. Please ensure plex is installed and use -p to manually define the path to the Plex Transcoder" >&2 exit 1 fi fi pcheck=$(tail -n 1 "$PLEX_PATH/Plex Transcoder" | tr -d '\0') if [ "$uninstall" == "1" ]; then if [ "$pcheck" == "##patched" ]; then if pgrep -x "Plex Transcoder" >/dev/null ; then echo "ERROR: Plex Transcoder is currently running. Please stop any open transcodes and run again" >&2 exit 1 fi mv "$PLEX_PATH/Plex Transcoder2" "$PLEX_PATH/Plex Transcoder" echo "Uninstall of patch complete!" exit else echo "ERROR: NVDEC Patch not detected as installed. Cannot uninstall." exit 1 fi fi if [ "$pcheck" == "##patched" ]; then echo "Patch has already been applied! Reapplying wrapper script" else if pgrep -x "Plex Transcoder" >/dev/null ; then echo "ERROR: Plex Transcoder is currently running. Please stop any open transcodes and run again" >&2 exit 1 fi mv "$PLEX_PATH/Plex Transcoder" "$PLEX_PATH/Plex Transcoder2" fi for i in "${CODECS[@]}"; do cstring+='"'"$i"'" ' done cstring="${cstring::-1}" cat > "$PLEX_PATH/Plex Transcoder" <<< '#!/bin/bash' cat >> "$PLEX_PATH/Plex Transcoder" <<< 'PLEX_PATH="'"$PLEX_PATH"'"' cat >> "$PLEX_PATH/Plex Transcoder" <<< 'ALLOWED_CODECS=('"$cstring"')' cat >> "$PLEX_PATH/Plex Transcoder" <<< ' contains() { typeset _x; typeset -n _A="$1" for _x in "${_A[@]}" ; do [ "$_x" = "$2" ] && return 0 done return 1 } allowed_codec() { for i in "$@"; do if [ "-i" == "$i" ]; then return 1 elif contains ALLOWED_CODECS "$i"; then return 0 fi done return 1 } if allowed_codec $*; then exec "$PLEX_PATH/Plex Transcoder2" -hwaccel nvdec "$@" else exec "$PLEX_PATH/Plex Transcoder2" "$@" fi ##patched' chmod +x "$PLEX_PATH/Plex Transcoder" echo "Patch applied successfully!" @EOF # chmod the new script to be executable: docker exec -i "$con" chmod +x "./plex-nvdec-patch.sh" # now we execute the script: docker exec -i "$con" bash -c "./plex-nvdec-patch.sh" echo '<font color="green"><b>Done!</b></font>' # Green means go! else # If we ended up here, the patch didn't even try to run, presumably because the patch was already applied. echo '<font color="red"><b>Patch already applied!</b></font>' # Red means stop! fi To alter the defaults simply append "-c codec" to the execution command like so: docker exec -i "$con" bash -c "./plex-nvdec-patch.sh -c h264 -c hevc -c vp8 -c vp9" Source/Credits: https://github.com/revr3nd/plex-nvdec and @Xaero for his script here: https://gist.github.com/Xaero252/9f81593e4a5e6825c045686d685e2428 Edited April 24, 2019 by MowMdown Quote Link to comment
Xaero Posted April 24, 2019 Share Posted April 24, 2019 3 hours ago, MowMdown said: *SNIP* Had been meaning to implement this myself. I just updated my script on gist. The new script pulls the latest version of revr3nds script and applies it. It also supports a list of codecs in a variable near the top of the script if you need to change them for any reason. Quote Link to comment
MowMdown Posted April 24, 2019 Share Posted April 24, 2019 (edited) 8 hours ago, Xaero said: Had been meaning to implement this myself. I just updated my script on gist. The new script pulls the latest version of revr3nds script and applies it. It also supports a list of codecs in a variable near the top of the script if you need to change them for any reason. There is a bug in your new code specifically: command="/usr/lib/plexmediaserver/plex-nvdec-patch.sh" if [ "$codec_arguments" ]; then #This line is recursive, it duplicates the codecs in the Plex Transcoder file (we don't need it) #command+="${codec_arguments}" docker exec -i "$con" /bin/sh -c "${command}${codec_arguments}" else docker exec -i "$con" /bin/sh -c "${command}" fi What ends up happening is that in the "Plex Transcoder" file, you end up with ALLOWED_CODECS=("h264" "hevc" "h264" "hevc") Removing that line of code fixes the issue. You don't need to concatenate the "command+="${codec_arguments}" because it's already being taken care of at the next line where the docker exec code is taking place. Edited April 24, 2019 by MowMdown Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.