January 25, 20224 yr Author 7 hours ago, badbadntgood said: tr: write error: Broken pipe I'd say your paths are wrong?! Please post what you set.
January 25, 20224 yr I have them set to /mnt/user/data/media/movies and /mnt/user/data/media/tv it says 950 files can be preloaded and runs for about 15 minutes until it freezes and after inspecting the log is when I get that message #!/bin/bash # ##################################### # Script: Plex Preloader v0.9 # Description: Preloads the recent video files of a specific path into the RAM to bypass HDD spinup latency # Author: Marc Gutt # # Changelog: # 0.9 # - Preloads only subtitle files that belong to preloaded video files # 0.8 # - Bug fix: In some situations video files were skipped instead of preloading them # 0.7 # - Unraid dashboard notification added # - Removed benchmark for subtitle preloading # 0.6 # - multiple video path support # 0.5 # - replaced the word "movie" against "video" as this script can be used for TV Shows as well # - reduced preload_tail_size to 1MB # 0.4 # - precleaning cache is now optional # 0.3 # - the read cache is cleaned before preloading starts # 0.2 # - preloading time is measured # 0.1 # - first release # # ######### Settings ################## video_paths=( "/mnt/user/data/media/movies/" "/mnt/user/data/media/tv/" ) video_min_size="200MB" # 2GB, to exclude bonus content preload_head_size="100MB" # 60MB, raise this value if your video buffers after ~5 seconds preload_tail_size="1MB" # 1MB, should be sufficient even for 4K video_ext='avi|mkv|mov|mp4|mpeg' # https://support.plex.tv/articles/203824396-what-media-formats-are-supported/ sub_ext='srt|smi|ssa|ass|vtt' # https://support.plex.tv/articles/200471133-adding-local-subtitles-to-your-media/#toc-1 free_ram_usage_percent=75 preclean_cache=0 notification=1 # ##################################### # # ######### Script #################### # make script race condition safe if [[ -d "/tmp/${0///}" ]] || ! mkdir "/tmp/${0///}"; then exit 1; fi; trap 'rmdir "/tmp/${0///}"' EXIT; # check user settings video_min_size="${video_min_size//[!0-9.]/}" # float filtering https://stackoverflow.com/a/19724571/318765 video_min_size=$(awk "BEGIN { print $video_min_size*1000000}") # convert MB to Bytes preload_head_size="${preload_head_size//[!0-9.]/}" preload_head_size=$(awk "BEGIN { print $preload_head_size*1000000}") preload_tail_size="${preload_tail_size//[!0-9.]/}" preload_tail_size=$(awk "BEGIN { print $preload_tail_size*1000000}") # clean the read cache if [ "$preclean_cache" = "1" ]; then sync; echo 1 > /proc/sys/vm/drop_caches fi # preload preloaded=0 skipped=0 preload_total_size=$(($preload_head_size + $preload_tail_size)) free_ram=$(free -b | awk '/^Mem:/{print $7}') free_ram=$(($free_ram / 100 * $free_ram_usage_percent)) echo "Available RAM in Bytes: $free_ram" preload_amount=$(($free_ram / $preload_total_size)) echo "Amount of Videos that can be preloaded: $preload_amount" # fetch video files while IFS= read -r -d '' file; do if [[ $preload_amount -le 0 ]]; then break; fi size=$(stat -c%s "$file") if [ "$size" -gt "$video_min_size" ]; then TIMEFORMAT=%R benchmark=$(time ( head -c $preload_head_size "$file" ) 2>&1 1>/dev/null ) echo "Preload $file (${benchmark}s)" if awk 'BEGIN {exit !('$benchmark' >= '0.150')}'; then preloaded=$((preloaded + 1)) else skipped=$((skipped + 1)) fi tail -c $preload_tail_size "$file" > /dev/null preload_amount=$(($preload_amount - 1)) video_path=$(dirname "$file") # fetch subtitle files find "$video_path" -regextype posix-extended -regex ".*\.($sub_ext)" -print0 | while IFS= read -r -d '' file; do echo "Preload $file" cat "$file" >/dev/null done fi done < <(find "${video_paths[@]}" -regextype posix-extended -regex ".*\.($video_ext)" -printf "%T@ %p\n" | sort -nr | cut -f2- -d" " | tr '\n' '\0') # notification if [[ $preloaded -eq 0 ]] && [[ $skipped -eq 0 ]]; then /usr/local/emhttp/webGui/scripts/notify -i alert -s "Plex Preloader failed!" -d "No video file has been preloaded (wrong path?)!" elif [ "$notification" = "1" ]; then /usr/local/emhttp/webGui/scripts/notify -i normal -s "Plex Preloader has finished" -d "$preloaded preloaded (from Disk) / $skipped skipped (already in RAM)" fi
January 28, 20224 yr @mgutt Any quick way to resolve this? I'd really like to use this script to optimize my server.
February 8, 20224 yr just want to say thanks, seems to work fairly well. I set it to 100mb (have quite a few 4k hdr stuff) and to use 85% of ram. (have 160gb, i have never seen more that 10gb used, only plex and nextcloud on this nas) was able to do just under 1200 videos. seems good like that, almost anything loads very fast now. I put it in user scripts and it runs at 7am every morning. Edited February 8, 20224 yr by coolbeams
February 9, 20224 yr So got this installed and running, but 0 change to my ram, ram amount never increased which seams odd, and it was not grabbing recent videos, they seem to be very old videos loaded over a year ago. Have 256GB ram and only 12% being used and after 20mins of the script running still shows 12% How can I fix this?
February 9, 20224 yr Author 1 hour ago, almulder said: Have 256GB ram and only 12% being used and after 20mins of the script running still shows 12% Read the notes. It uses your free RAM without preserving it. So you can't see how much of your RAM is finally used, but it does. Every file read is cached through Linux automatically in the RAM. If you reexecute the script it should be finished in seconds as it does not read the file again. Instead it uses the RAM cache.
February 9, 20224 yr Hmm, seems after running consecutive scripts that nothing is being stored in RAM which I suspect means that it's not working for me? Spinning down all disks and then attempting to play still incurs buffering. Is this error in the logs still to be disrgarded?: tr: write error: Broken pipe tr: write error cut: write error: Broken pipe
February 15, 20224 yr Below is all im seeing in the logs. Script Starting Feb 15, 2022 06:00.01 Full logs for this script are available at /tmp/user.scripts/tmpScripts/Plex Precache/log.txt Script Finished Feb 15, 2022 06:00.01 Full logs for this script are available at /tmp/user.scripts/tmpScripts/Plex Precache/log.txt Also not finding the above log location unless I'm really missing something. (more than likely im missing something)
February 16, 20224 yr Hi all i tried the script but i can't run it a second time i'm gettin the same thing as @Jurak any idea @mgutt. Thank you have a nice day
February 21, 20224 yr Is the developer still paying attention to this post? Looks like a lot of us are getting the broken pipe error preventing things from working.
March 26, 20224 yr Author On 2/21/2022 at 4:05 PM, DJ-BrianC said: the broken pipe error preventing things from working. Nope. The script works absolutely fine with this "error". This error was only shown because I choosed a bad coding style to interrupt the loop. Check the script's logs. It successfully preloads the correct amount of movies. But I changed the code a little bit so in the next version this message won't be shown.
March 26, 20224 yr Author @DJ-BrianC @francrouge @badbadntgood V1.0 released # - coding style optimized to remove confusing error message
March 26, 20224 yr Author On 2/9/2022 at 11:50 AM, Irithor said: Hmm, seems after running consecutive scripts that nothing is being stored in RAM which I suspect means that it's not working for me? Spinning down all disks and then attempting to play still incurs buffering. Yes, something is wrong here. The second execution should show 37 skipped movies. Please run the script, than spindown the disks and run the script again. Do the disks spinup?
March 26, 20224 yr Author 1 minute ago, DJ-BrianC said: No it doesn't. It doesn't preload a single movie on my end. What do you see in the logs?
March 26, 20224 yr Author On 2/15/2022 at 5:15 PM, Jurak said: Below is all im seeing in the logs. Script Starting Feb 15, 2022 06:00.01 Full logs for this script are available at /tmp/user.scripts/tmpScripts/Plex Precache/log.txt Script Finished Feb 15, 2022 06:00.01 Full logs for this script are available at /tmp/user.scripts/tmpScripts/Plex Precache/log.txt Also not finding the above log location unless I'm really missing something. (more than likely im missing something) This means the find command did not find a single movie. Are you sure you set the correct source path? Execute this command through the terminal to see which movies the script is able to find: find /mnt/your_path -regextype posix-extended -regex ".*\.(avi|mkv|mov|mp4|mpeg)" -printf "%T@ %p\n"
March 26, 20224 yr In my case it goes thru all the movies and tv shows then ends with an error. On subsequent runs it does exactly the same thing. I've set the script to run hourly just so I could keep an eye on it. It shows how many preload but next run never skips anything. Always the same list of movies. If it preloaded then the next run should be skipping. It's not. Edited March 26, 20224 yr by DJ-BrianC Fix Typo
March 26, 20224 yr Mine's working OK, though I am getting an error message in the start of the run for some reason. Is that the one that's safe to ignore? Script Starting Mar 26, 2022 13:08.47 Full logs for this script are available at /tmp/user.scripts/tmpScripts/Plex Preloader/log.txt Available RAM in Bytes: 94524352512 Amount of Videos that can be preloaded: 1549 sort: write failed: 'standard output': Broken pipe sort: write error Preload /mnt/user/Media/TV/REDACTED.mkv (0.369s) Preload /mnt/user/Media/TV/REDACTED.mkv (0.381s) Preload /mnt/user/Media/TV/REDACTED.mkv (0.453s) Preload /mnt/user/Media/TV/REDACTED.mkv (0.455s) Preload /mnt/user/Media/TV/REDACTED.mkv (0.432s) Preload /mnt/user/Media/TV/REDACTED.mkv (0.501s) You'll see my read times for cached files are about 0.4s on average with the default setting in the script set at 0.15, as such my scans never detected any cached files. Once I figured that out, I adjusted the setting in the script to 1.0s and it worked reliably. I just have a bit of a slower system than yours I guess. As a quick fix, it'd be good to move that variable to the top of the script where other user configurable variables are so the user can tweak it. A longer term fix may be to do an initial benchmark run - first reading a reasonable number of files that are totally uncached, then reading them another time to get a cached result, and you can average the resulting types of results. You could then set up upper and lower bounds for when an object is decided as cached and when it is not. In my case, 0.4 Edited March 26, 20224 yr by ClunkClunk
March 26, 20224 yr Just now, DJ-BrianC said: Where did you make that change? I want to give that a try see if it helps me. Line 58, change 0.15 to something larger. I used 1.0 for good measure because mine seem to read from cache about 0.3s to 0.6s. if awk 'BEGIN {exit !('"$benchmark"' >= '1.0')}'; then
March 26, 20224 yr Okay now I get a write failed standard output broken pipe at the very top it appears to load 12 shows stops. Although it tells me it's capable of loading 182. I don't get it.
March 27, 20224 yr I now notice that it tells me it skip three because they're already in memory. 32 gigs of RAM is set at the default 50% at the start of the scan it's almost 200 that it says it can do yet it only does these three. I really don't know what's going on or how to make this thing work.
March 27, 20224 yr I'm still getting the broken pipe error message as well as it still loading in ALL my srt files. Is the script on the first page updated?
April 18, 20224 yr Author Version 1.1 has been released: # - changed the way how movies are found and sorted to avoid confusing error messages # - added timestamps to the logs # - check if the given video paths are used by a docker container (else RAM preloading would not have any effect) # - better documentation of the settings
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.