Video Preloader (avoids HDD spinup latency when starting a Movie or Episode through Plex, Jellyfin or Emby)

[mgutt]

7 hours ago, badbadntgood said:

tr: write error: Broken pipe

I'd say your paths are wrong?! Please post what you set.

[badbadntgood]

I have them set to /mnt/user/data/media/movies and /mnt/user/data/media/tv it says 950 files can be preloaded and runs for about 15 minutes until it freezes and after inspecting the log is when I get that message

 

#!/bin/bash
# #####################################
# Script:      Plex Preloader v0.9
# Description: Preloads the recent video files of a specific path into the RAM to bypass HDD spinup latency
# Author:      Marc Gutt
# 
# Changelog:
# 0.9
# - Preloads only subtitle files that belong to preloaded video files
# 0.8
# - Bug fix: In some situations video files were skipped instead of preloading them
# 0.7
# - Unraid dashboard notification added
# - Removed benchmark for subtitle preloading
# 0.6
# - multiple video path support
# 0.5
# - replaced the word "movie" against "video" as this script can be used for TV Shows as well
# - reduced preload_tail_size to 1MB
# 0.4
# - precleaning cache is now optional
# 0.3
# - the read cache is cleaned before preloading starts
# 0.2
# - preloading time is measured
# 0.1
# - first release
# 
# ######### Settings ##################
video_paths=(
    "/mnt/user/data/media/movies/"
    "/mnt/user/data/media/tv/"
)
video_min_size="200MB" # 2GB, to exclude bonus content
preload_head_size="100MB" # 60MB, raise this value if your video buffers after ~5 seconds
preload_tail_size="1MB" # 1MB, should be sufficient even for 4K
video_ext='avi|mkv|mov|mp4|mpeg' # https://support.plex.tv/articles/203824396-what-media-formats-are-supported/
sub_ext='srt|smi|ssa|ass|vtt' # https://support.plex.tv/articles/200471133-adding-local-subtitles-to-your-media/#toc-1
free_ram_usage_percent=75
preclean_cache=0
notification=1
# #####################################
# 
# ######### Script ####################
# make script race condition safe
if [[ -d "/tmp/${0///}" ]] || ! mkdir "/tmp/${0///}"; then exit 1; fi; trap 'rmdir "/tmp/${0///}"' EXIT;
# check user settings
video_min_size="${video_min_size//[!0-9.]/}" # float filtering https://stackoverflow.com/a/19724571/318765
video_min_size=$(awk "BEGIN { print $video_min_size*1000000}") # convert MB to Bytes
preload_head_size="${preload_head_size//[!0-9.]/}"
preload_head_size=$(awk "BEGIN { print $preload_head_size*1000000}")
preload_tail_size="${preload_tail_size//[!0-9.]/}"
preload_tail_size=$(awk "BEGIN { print $preload_tail_size*1000000}")
# clean the read cache
if [ "$preclean_cache" = "1" ]; then
    sync; echo 1 > /proc/sys/vm/drop_caches
fi
# preload
preloaded=0
skipped=0
preload_total_size=$(($preload_head_size + $preload_tail_size))
free_ram=$(free -b | awk '/^Mem:/{print $7}')
free_ram=$(($free_ram / 100 * $free_ram_usage_percent))
echo "Available RAM in Bytes: $free_ram"
preload_amount=$(($free_ram / $preload_total_size))
echo "Amount of Videos that can be preloaded: $preload_amount"
# fetch video files
while IFS= read -r -d '' file; do
    if [[ $preload_amount -le 0 ]]; then
        break;
    fi
    size=$(stat -c%s "$file")
    if [ "$size" -gt "$video_min_size" ]; then
        TIMEFORMAT=%R
        benchmark=$(time ( head -c $preload_head_size "$file" ) 2>&1 1>/dev/null )
        echo "Preload $file (${benchmark}s)"
        if awk 'BEGIN {exit !('$benchmark' >= '0.150')}'; then
            preloaded=$((preloaded + 1))
        else
            skipped=$((skipped + 1))
        fi
        tail -c $preload_tail_size "$file" > /dev/null
        preload_amount=$(($preload_amount - 1))
        video_path=$(dirname "$file")
        # fetch subtitle files
        find "$video_path" -regextype posix-extended -regex ".*\.($sub_ext)" -print0 | 
            while IFS= read -r -d '' file; do 
                echo "Preload $file"
                cat "$file" >/dev/null
            done
    fi
done < <(find "${video_paths[@]}" -regextype posix-extended -regex ".*\.($video_ext)" -printf "%T@ %p\n" | sort -nr | cut -f2- -d" " | tr '\n' '\0')
# notification
if [[ $preloaded -eq 0 ]] && [[ $skipped -eq 0 ]]; then
    /usr/local/emhttp/webGui/scripts/notify -i alert -s "Plex Preloader failed!" -d "No video file has been preloaded (wrong path?)!"
elif [ "$notification" = "1" ]; then
    /usr/local/emhttp/webGui/scripts/notify -i normal -s "Plex Preloader has finished" -d "$preloaded preloaded (from Disk) / $skipped skipped (already in RAM)"
fi

 

[badbadntgood]

@mgutt Any quick way to resolve this? I'd really like to use this script to optimize my server.

[coolbeams]

just want to say thanks, seems to work fairly well.

I set it to 100mb (have quite a few 4k hdr stuff)
and to use 85% of ram. (have 160gb, i have never seen more that 10gb used, only plex and nextcloud on this nas)
was able to do just under 1200 videos.
seems good like that, almost anything loads very fast now. 

I put it in user scripts and it runs at 7am every morning.

Edited February 8, 2022 by coolbeams

[almulder]

So got this installed and running, but 0 change to my ram, ram amount never increased which seams odd, and it was not grabbing recent videos, they seem to be very old videos loaded over a year ago.

 

Have 256GB ram and only 12% being used and after 20mins of the script running still shows 12%

 

How can I fix this?

[mgutt]

1 hour ago, almulder said:

Have 256GB ram and only 12% being used and after 20mins of the script running still shows 12%

Read the notes. It uses your free RAM without preserving it. So you can't see how much of your RAM is finally used, but it does. Every file read is cached through Linux automatically in the RAM. If you reexecute the script it should be finished in seconds as it does not read the file again. Instead it uses the RAM cache.

[Irithor]

Hmm, seems after running consecutive scripts that nothing is being stored in RAM which I suspect means that it's not working for me? Spinning down all disks and then attempting to play still incurs buffering.

Is this error in the logs still to be disrgarded?:

tr: write error: Broken pipe
tr: write error
cut: write error: Broken pipe

[DJ-BrianC]

I'm also getting the broken pipe error! What's the solution?

[Jurak]

Below is all im seeing in the logs.

 

Script Starting Feb 15, 2022 06:00.01

Full logs for this script are available at /tmp/user.scripts/tmpScripts/Plex Precache/log.txt

Script Finished Feb 15, 2022 06:00.01

Full logs for this script are available at /tmp/user.scripts/tmpScripts/Plex Precache/log.txt

 

Also not finding the above log location unless I'm really missing something. (more than likely im missing something)

[francrouge]

Hi all i tried the script but i can't run it a second time i'm gettin the same thing as @Jurak any idea @mgutt.

 

Thank you have a nice day

[DJ-BrianC]

Is the developer still paying attention to this post? Looks like a lot of us are getting the broken pipe error preventing things from working.

[mgutt]

On 2/21/2022 at 4:05 PM, DJ-BrianC said:

the broken pipe error preventing things from working.

Nope. The script works absolutely fine with this "error". This error was only shown because I choosed a bad coding style to interrupt the loop. Check the script's logs. It successfully preloads the correct amount of movies.

 

But I changed the code a little bit so in the next version this message won't be shown.

[mgutt]

@DJ-BrianC @francrouge @badbadntgood

 

V1.0 released

# - coding style optimized to remove confusing error message

 

[DJ-BrianC]

No it doesn't. It doesn't preload a single movie on my end.

[mgutt]

On 2/9/2022 at 11:50 AM, Irithor said:

Hmm, seems after running consecutive scripts that nothing is being stored in RAM which I suspect means that it's not working for me? Spinning down all disks and then attempting to play still incurs buffering.

 

 

Yes, something is wrong here. The second execution should show 37 skipped movies. Please run the script, than spindown the disks and run the script again. Do the disks spinup?

[mgutt]

1 minute ago, DJ-BrianC said:

No it doesn't. It doesn't preload a single movie on my end.

What do you see in the logs?

[mgutt]

On 2/15/2022 at 5:15 PM, Jurak said:

Below is all im seeing in the logs.

 

Script Starting Feb 15, 2022 06:00.01

Full logs for this script are available at /tmp/user.scripts/tmpScripts/Plex Precache/log.txt

Script Finished Feb 15, 2022 06:00.01

Full logs for this script are available at /tmp/user.scripts/tmpScripts/Plex Precache/log.txt

 

Also not finding the above log location unless I'm really missing something. (more than likely im missing something)

 

This means the find command did not find a single movie. Are you sure you set the correct source path?

 

Execute this command through the terminal to see which movies the script is able to find:

find /mnt/your_path -regextype posix-extended -regex ".*\.(avi|mkv|mov|mp4|mpeg)" -printf "%T@ %p\n"

 

 

[DJ-BrianC]

In my case it goes thru all the movies and tv shows then ends with an error. On subsequent runs it does exactly the same thing. I've set the script to run hourly just so I could keep an eye on it. It shows how many preload but next run never skips anything. Always the same list of movies. If it preloaded then the next run should be skipping. It's not.

Edited March 26, 2022 by DJ-BrianC
Fix Typo

[ClunkClunk]

Mine's working OK, though I am getting an error message in the start of the run for some reason. Is that the one that's safe to ignore?

 

Script Starting Mar 26, 2022  13:08.47

Full logs for this script are available at /tmp/user.scripts/tmpScripts/Plex Preloader/log.txt

Available RAM in Bytes: 94524352512
Amount of Videos that can be preloaded: 1549
sort: write failed: 'standard output': Broken pipe
sort: write error
Preload /mnt/user/Media/TV/REDACTED.mkv (0.369s)
Preload /mnt/user/Media/TV/REDACTED.mkv (0.381s)
Preload /mnt/user/Media/TV/REDACTED.mkv (0.453s)
Preload /mnt/user/Media/TV/REDACTED.mkv (0.455s)
Preload /mnt/user/Media/TV/REDACTED.mkv (0.432s)
Preload /mnt/user/Media/TV/REDACTED.mkv (0.501s)

 

You'll see my read times for cached files are about 0.4s on average with the default setting in the script set at 0.15, as such my scans never detected any cached files. Once I figured that out, I adjusted the setting in the script to 1.0s and it worked reliably. I just have a bit of a slower system than yours I guess.

 

As a quick fix, it'd be good to move that variable to the top of the script where other user configurable variables are so the user can tweak it.

 

A longer term fix may be to do an initial benchmark run - first reading a reasonable number of files that are totally uncached, then reading them another time to get a cached result, and you can average the resulting types of results. You could then set up upper and lower bounds for when an object is decided as cached and when it is not. In my case, 0.4

Edited March 26, 2022 by ClunkClunk

[DJ-BrianC]

Where did you make that change? I want to give that a try see if it helps me.

[ClunkClunk]

Just now, DJ-BrianC said:

Where did you make that change? I want to give that a try see if it helps me.

Line 58, change 0.15 to something larger. I used 1.0 for good measure because mine seem to read from cache about 0.3s to 0.6s.

    if awk 'BEGIN {exit !('"$benchmark"' >= '1.0')}'; then

 

[DJ-BrianC]

Okay now I get a write failed standard output broken pipe at the very top it appears to load 12 shows stops. Although it tells me it's capable of loading 182. I don't get it.

[DJ-BrianC]

I now notice that it tells me it skip three because they're already in memory. 32 gigs of RAM is set at the default 50% at the start of the scan it's almost 200 that it says it can do yet it only does these three. I really don't know what's going on or how to make this thing work.

[Archangel]

I'm still getting the broken pipe error message as well as it still loading in ALL my srt files.  Is the script on the first page updated?

[mgutt]

Version 1.1 has been released:

# - changed the way how movies are found and sorted to avoid confusing error messages
# - added timestamps to the logs
# - check if the given video paths are used by a docker container (else RAM preloading would not have any effect)
# - better documentation of the settings