Video Preloader (avoids HDD spinup latency when starting a Movie or Episode through Plex, Jellyfin or Emby)


Recommended Posts

This was an idea of @frodr. After several testings I came to the following script which:

  1. Caclulates how many Video files (small parts of them) will fit into 50% of the free RAM (amount can be changed)
  2. Obtains the X recent Movies / Episodes (depending on the used path)
  3. Preloads 60MB of the Video file leader and 1MB of the ending into the RAM
  4. Preloads subtitle files that belong to the preloaded Video file

 

Now, if your disks are sleeping and you start a Movie or Episode through Plex, the client will download the Video parts from RAM and while the buffer is emptied, the HDD spins up and the buffer fills up again. This means all preloaded Movies / Episodes will start directly without delay.

 

Donate? 🤗

 

Notes:

  • It does not reserve any RAM, so your RAM stays fully available
  • RAM preloading is not permanent and will be overwritten through server uploads/downloads, which means if you upload some huge files to your server, your RAM cache will be probably completely overwritten and by that all your nice cached movie files, too. That is the reason why I suggest to execute this script multiple times per day (only missing video files will be touched). As long the movie parts are still in the RAM, the disks do not spin up, while "rebuilding" the cache list, but the cached movie parts become "more popular" in the Linux cache.
  • Applications like backup processes can overwrite the complete RAM cache, too. To avoid that, you should consider executing applications inside of containers and limit their RAM usage through the extra parameter "--memory=1G" (or whatever a container needs)
  • If you suffer from buffering at the beginning of a Movie / Episode, try to raise "video_min_size"
  • All preloaded videos can be found in the script's log (CA User Scripts)

 

Script

#!/bin/bash
# #####################################
# Script:      Video Preloader v1.4
# Description: Preloads the recent video files of a specific path into the RAM to bypass HDD spinup latency
# Author:      Marc Gutt
# ######### Settings ##################

video_paths=(
  "/mnt/user/Movie"
  "/mnt/user/TV"
)

# the size of a video files must be at least 2GB (to exclude bonus content)
video_min_size="2000MB" 

# we preload 60MB of the beginning of the video file into the RAM (raise this value if your video buffers after ~5 seconds)
preload_head_size="60MB"

# we preload 1MB of the end of the video file
preload_tail_size="1MB" 

# if a preload is faster than 0.150 seconds than the video is already preloaded to the RAM
preload_threshold="0.150"

# preload only video files with specific extensions https://support.plex.tv/articles/203824396-what-media-formats-are-supported/
video_ext='avi|mkv|mov|mp4|mpeg'

# preload only subtitle files with specific extensions # https://support.plex.tv/articles/200471133-adding-local-subtitles-to-your-media/#toc-1
sub_ext='srt|smi|ssa|ass|vtt'

# we use 50% of our free RAM for preloading (it will be still free, check https://www.linuxatemyram.com/)
free_ram_usage_percent=50

# set this to "1" to remove all preloads (this is only usefull if you want to force reading from Disk)
preclean_cache=0

# notify if the execution was sucessful (errors produce notifications by default)
notification=1

# #####################################
# 
# ######### Script ####################

# timestamping logs
exec &> >(stdbuf -o0 sed 's/%/%%/g' | xargs -d '\n' -I {} date '+%F %T {}')

# make script race condition safe
if [[ -d "/tmp/${0///}" ]] || ! mkdir "/tmp/${0///}"; then exit 1; fi; trap 'rmdir "/tmp/${0///}"' EXIT;

# check user settings
video_min_size="${video_min_size//[!0-9.]/}" # float filtering https://stackoverflow.com/a/19724571/318765
video_min_size=$(awk "BEGIN { print $video_min_size*1000000}") # convert MB to Bytes
preload_head_size="${preload_head_size//[!0-9.]/}"
preload_head_size=$(awk "BEGIN { print $preload_head_size*1000000}")
preload_tail_size="${preload_tail_size//[!0-9.]/}"
preload_tail_size=$(awk "BEGIN { print $preload_tail_size*1000000}")

# check if paths are used in docker containers
if docker info > /dev/null 2>&1; then
  # get docker mounts of all running containers
  # shellcheck disable=SC2016
  docker_mounts=$(docker ps -q | xargs docker container inspect -f '{{$id := .Id}}{{range .Mounts}}{{if .Source}}{{printf $id}}:{{.Source}}{{println}}{{end}}{{end}}' | grep -v -e "^$")
  for path in "${video_paths[@]}"; do
    if [[ $docker_mounts != *"$path"* ]]; then
      /usr/local/emhttp/webGui/scripts/notify -i alert -s "Plex Preloader failed!" -d "$path is not used by a docker container!"
      exit 1
    fi
  done
fi

# clean the read cache
if [ "$preclean_cache" = "1" ]; then
  sync; echo 1 > /proc/sys/vm/drop_caches
fi

# preload calculation
preloaded=0
skipped=0
preload_total_size=$((preload_head_size + preload_tail_size))
free_ram=$(free -b | awk '/^Mem:/{print $7}')
free_ram=$((free_ram * free_ram_usage_percent / 100))
echo "Available RAM: $(numfmt --to si $free_ram)"
preload_amount=$((free_ram / preload_total_size))
preload_size=$(( preload_head_size + preload_tail_size ))
echo "Amount of videos that can be preloaded: $preload_amount (each video occupies $(numfmt --to si $preload_size))"

# find all video files
while IFS= read -r -d '' file; do
  video_files+=("$file")
done < <(find "${video_paths[@]}" -not -path '*/.*' -size +"$video_min_size"c -regextype posix-extended -regex ".*\.($video_ext)" -printf "%T@ %p\0")

# no video files found
if [[ ${#video_files[@]} -eq 0 ]]; then
  echo "No $video_ext files found in ${video_paths[*]}!"
else

  # sort all video files
  while IFS= read -r -d '' file; do
    video_files_sorted+=("$file")
  done < <(printf '%s\0' "${video_files[@]}" | sort -znr)

  # preload video files into RAM
  for i in "${!video_files_sorted[@]}"; do
    # stop if we reached our preload limit
    if [[ $i -ge $preload_amount ]]; then
      break;
    fi
    # remove modification time
    file=$(echo "${video_files_sorted[$i]}" | cut -f2- -d" ")
    TIMEFORMAT=%R
    seconds=$( { time head -c "$preload_head_size" "$file" >/dev/null; } 2>&1 )
    if awk 'BEGIN {exit !('"$seconds"' >= '"$preload_threshold"')}'; then
      preloaded=$((preloaded + 1))
      echo "Preloaded $file in $seconds seconds"
    else
      echo "Skipped $file as loading needed only $seconds"
      skipped=$((skipped + 1))
    fi
    tail -c "$preload_tail_size" "$file" > /dev/null
    video_path=$(dirname "$file")
    # fetch subtitle files
    find "$video_path" -not -path '*/.*' -regextype posix-extended -regex ".*\.($sub_ext)" -print0 | 
      while IFS= read -r -d '' file; do 
        echo "Preloaded $file"
        cat "$file" >/dev/null
      done
  done

fi

# notification
if [[ $preloaded -eq 0 ]] && [[ $skipped -eq 0 ]]; then
  /usr/local/emhttp/webGui/scripts/notify -i alert -s "Plex Preloader failed!" -d "No video file has been preloaded (wrong path?)!"
elif [ "$notification" == "1" ]; then
  /usr/local/emhttp/webGui/scripts/notify -i normal -s "Plex Preloader has finished" -d "$preloaded preloaded (from Disk) / $skipped skipped (already in RAM)"
fi

 

  • Like 5
  • Thanks 8
Link to comment

I'm still experimenting on this. I like to write the preloaded movie parts directly to a swap file, but I can't find a solution for this at the moment (Linux decides on its own which data is swapped). If this would be possible the complete Movie collection could be preloaded to a "small" swap file located on an SSD.

 

I even played around with vmtouch to make this preloading permanent (as vmtouch can lock it in the RAM), but at the moment vmtouch seems to be buggy in Slackware (vmtouch daemon becomes randomly killed and sometimes its not possible to get any caching status).

 

Edited by mgutt
  • Thanks 1
Link to comment

Here you can see the RAM usage in the column "buff/cache". Before Plex Preloader:

sync; echo 1 > /proc/sys/vm/drop_caches
free -m
              total        used        free      shared  buff/cache   available
Mem:          64358        1154       60873        1027        2330       61580
Swap:             0           0           0

After:

free -m
              total        used        free      shared  buff/cache   available
Mem:          64358        1216       27655        1027       35485       61405
Swap:             0           0           0

This means it used 35485 - 2330 = 33155 MB as Preload RAM Cache.

 

After that I spun down all my disks, opened PMP and started a recently added movie and it started without any latency.

1976079372_2020-10-1911_49_40.png.b589ba86902175b0fdae64e54e749b0e.png

 

I spun down all disks again and started one of my very first added movies and it took 8 seconds until it started playing.

1895310542_2020-10-1911_46_47.png.0a5259e4079ada6119312e6ac0e60285.png

 

Edited by mgutt
Link to comment
2 hours ago, Mushin said:

Is it possible to also get this working with TV files? Maybe just for the ones displayed in On Deck?

Yes, use the same script and change these values:

video_path="/mnt/user/Movie/"
video_min_size="2000MB" # 2GB, to exclude bonus content

as follows (set your TV path of course):

video_path="/mnt/user/TV/"
video_min_size="500MB"

While "500MB" is only an estimate of the minimum size of a TV episode. You could even lower it to "1MB" to cover everything.

 

If you use both scripts parallel (to cache Movies and TV Shows) you should think about reducing this value as well:

free_ram_usage_percent=50

Like 25 for TV shows and 25 for Movies or similar.

Edited by mgutt
  • Like 1
Link to comment

Feature Requests from Reddit:

- determine which disks were involved and spin them down after script execution

- multiple path support (to scan movies and tv shows through one script) realised

- obtain recent tv shows through Plex database and add the X next episodes to the cache (alternative request: move complete episodes to SSD cache which could be part of an additional script)

 

 

 

Edited by mgutt
  • Like 2
Link to comment
2 hours ago, mgutt said:

- obtain recent tv shows through Plex database and add the X next episodes to the cache (alternative request: move complete episodes to SSD cache which could be part of an additional script)

In case it helps, I think this is how to get the On Deck from Plex using the API:

wget -q -O- http://$PLEXURL:32400/library/onDeck?X-Plex-Token=$PLEXTOKEN

Set $PLEXURL to your PMS's ip address, and $PLEXTOKEN to your Plex Auth Token, and it'll return with XML. You'll then need to parse out the 'file=' entries to get the filename and path, and replace the returned filepath with whatever the Docker container is mapped to (for example, I pass /tv to my Plex docker container and it maps to /mnt/user/TV on my unRAID).

  • Thanks 2
Link to comment

- I was reading about the precaching and storing on on SSD on Redditt. That's a very interesting idea which would save everybody money if they have to up their Ram or simply another option.

- Could you put in a pre-test or simply a way to say, "Scan 10 files and test" vs a scan everything you can and find out your existing settings are not a enough?  Depending on some peoples Media and Ram it could scan for a long time just to find out they should either up the amount cached or lower. Could be something as simple as:

 

#####Uncomment below####

#Test Scan only 10 Files

#Full Scan

 

Also 

Could there be a (Cache, Flush and Cache or maybe a Compare and update)? If Changes are made. No idea how you intended to address changes in users scans if they need to update their amounts or if files are added/removed. I'm assuming Plex simply wouldn't show the file, but not sure how removing a file would affect the Cache. 

 

Are you planning on adding this as a Plugin in the future? Something that could be added to CA and updated or keeping it simple as a User.Script? Both work obviously, but just something for you to think about in the future. 

Link to comment

@kizer Check if the movie is listed in the script's logs and how much time it took to read the file part. It will look similar to this:

Available RAM in Bytes: 32628881400
Amount of Videos that can be preloaded: 534
...
Preload Video /mnt/user/Movie/EF/Falling Down - Ein ganz normaler Tag (1993)/Falling Down - Ein ganz normaler Tag (1993) 1080P FSK16 EN JP FR IT ES PT IMDB7.6.mkv

real 0m0.545s
user 0m0.007s
sys 0m0.033s

Now repeat the script and check the logs again.

 

1.) Did the "real" time of this movie fall under 0.100s?

a) If not, then it was re-read from disk and the caching failed. This means other processes are blocking or permanently overwriting your RAM. Possible solutions: More RAM or execute the Plex Preloader script after your processes finished their work

b) If yes, then the video file part is in the RAM and you can try to start the movie while your disks are sleeping.

 

2.) Did the Movie played directly?

a) If not, which Player did you use and was transcoding needed? Which path do you use to include your video files in your docker container, do they use the same path that was used by the Plex Preloader script?

b) If yes. It works.

 

 

 

 

 

Link to comment

When I checked the log in User.Scripts it just said the following and nothing else:

 

Available RAM in Bytes: 16G Here don't know the exact digits so I just paraphrased it. 

Amount of Videos that can be preloaded: 110

 

I used Plex to view the File on my AppleTV.  I'm not 100% sure if any Transcoding was needed. Its a MKV with 6CH/AC3 Audio.  I'll look again this Evening for a 2CH video or try it on TV shows since a lot of mine are ripped to 2CH. I have no issues with buying more Ram. I just want to see this work before I lay out the money to 32GB. 

 

Link to comment

Yes my Given movie path is

/mnt/user/Movies/All/

 

OH NO....... I used

/mnt/user/Movie/All/

 

I just checked my Log. I see a lot more info populating since adding the s.  🤪

 

I'll let that run until I get off work and try again. Talk about feeling kinda dumb. 

  • Haha 1
Link to comment
24 minutes ago, kizer said:

or does it run the first path and then run to the second path until the 50% of ram is used up?

 

The files of all paths are added to one huge list, which is sorted by date and by that the most recent video files of all paths will be preloaded.

 

Example: If your RAM allows preloading of 100 videos and you added 90 new movies to your collection, it will preload only 10 episodes.

 

This means if you prefer 25/25 you still need to use one script per path.

 

Link to comment

Just gave the new Script a try and this is the output.

 

Script location: /tmp/user.scripts/tmpScripts/Media-Preload-Movies/script

Note that closing this window will abort the execution of this script
Available RAM in Bytes: 6726713300
Amount of Videos that can be preloaded: 110

Preload /mnt/user/Movies/All/Rush.Hour.3.(2007)(720p)(DTS)/Rush.Hour.3.(2007)(720p).srt
Preload /mnt/user/Movies/All/Waiting.(2005)(720p)(AC3)/Waiting.(2005)(720p).srt
Preload /mnt/user/Movies/All/Rush.Hour.2.(2001)(720p)(AC3)/Rush.Hour.2.(2001)(720p).srt
Preload /mnt/user/Movies/All/The.Librarian.The.Curse.Of.The.Judas.Chalice.(2008)(720p)(AC3)/The.Librarian.The.Curse.Of.The.Judas.Chalice.(2008)(720p).srt
Preload /mnt/user/Movies/1080-Bray-Rips/Avatar/Avatar.srt
Preload /mnt/user/TV/Shows/Adventure Time/Season 7/Adventure.Time.-.S07E35_eng.srt
Preload /mnt/user/TV/Shows/Adventure Time/Season 7/Adventure.Time.-.S07E34_eng.srt
Preload /mnt/user/TV/Shows/Adventure Time/Season 7/Adventure.Time.-.S07E33_eng.srt
Preload /mnt/user/TV/Shows/Adventure Time/Season 7/Adventure.Time.-.S07E32_eng.srt
Preload /mnt/user/TV/Shows/Adventure Time/Season 7/Adventure.Time.-.S07E31_eng.srt
Preload /mnt/user/TV/Shows/Adventure Time/Season 8/Adventure.Time.-.S08E03_eng.srt
Preload /mnt/user/TV/Shows/Adventure Time/Season 8/Adventure.Time.-.S08E01E01_eng.srt
Preload /mnt/user/TV/Shows/Top Secret Recipe/Season 1/Top.Secret.Recipe.-.S01E08.srt
Preload /mnt/user/TV/Shows/Dexter/Season 4/Dexter.-.S04E01.srt

 

Link to comment

This script looks really good and very useful. Is there a way to modify this script to pre load parts of films (4K movies) but with drives spinning. I don’t have my drives sleep much. But notice when watching 4K movies. I have a 5-10 second delay before it starts. 1080p ones is only 2-3 secs. I have a small collection of 4K movies. Was wondering if I could pre load about 180mb of each one into ram to see if it helps with start time

Link to comment
5 hours ago, Apollopayne35 said:

Is there a way to modify this script to pre load parts of films (4K movies) but with drives spinning.

No need to modify the script. It does nothing special except of preloading data into the RAM.

 

Maybe you suffer from a deep idle state. As an example, Seagate Ironwolf and Exos have 4 different idle states, and some of them even reduce the RPM to save energy, which raises latency:

https://www.seagate.com/files/docs/pdf/en-GB/whitepaper/tp608-powerchoice-tech-provides-gb.pdf

 

If you want to test only 4K movies you could raise "video_min_size" to "50000MB" (50GB) and if you want to raise the preloading size, change "preload_head_size" to "75MB", "150MB" or "500MB". This should be the sizes which are used through the client:

https://forums.plex.tv/t/how-can-i-delay-playing-of-a-video-until-the-cache-is-full/179749/3?u=mgutt

Link to comment
1 hour ago, kizer said:

Nope. I just cut and pasted your script up above and changed the paths to my Movies and TV shows. 

This is strange. I mean I changed the code, but a) it works for me and b) it was not a huge change

 

1.) Are you using unraid 6.8.3?

2.) Do you execute the script through CA User Scripts?

 

Please test this code:

movie_path="/mnt/user/Movies"

echo "Process substitution test"
while IFS= read -r -d '' file; do
    echo "$file"
done < <(find "$movie_path" -iname "*.mkv" | tail -3 | tr '\n' '\0')

echo "Piping test"
find "$movie_path" -iname "*.mkv" | tail -3 | tr '\n' '\0' | 
    while IFS= read -r -d '' file; do
        echo "$file"
    done

You could even execute it through the WebTerminal. Both tests should return 3 filenames. If not, do you get an error message?

Link to comment

Im running 6.9-RC30

I could fire up my other machine to test on 6.8.3 if you think there are significant changes from 6.8.3 and 6.9-RC-30

 

This is the output via User.Scripts:

 

Script location: /tmp/user.scripts/tmpScripts/zzzzzzz-test-zzzzzzzz/script
Note that closing this window will abort the execution of this script
Process substitution test
/mnt/user/Movies/All/Yours,.Mine.&.Ours.(1968)(480)/Yours,.Mine.&.Ours.(1968)(480).mkv
/mnt/user/Movies/All/Zootopia.(2016)(720p)/Zootopia.(2016)(720p).mkv
/mnt/user/Movies/All/xXx -.Return.of.Xander.Cage.(2017)(1080p)/xXx -.Return.of.Xander.Cage.(2017)(1080p).mkv
Piping test
/mnt/user/Movies/All/Yours,.Mine.&.Ours.(1968)(480)/Yours,.Mine.&.Ours.(1968)(480).mkv
/mnt/user/Movies/All/Zootopia.(2016)(720p)/Zootopia.(2016)(720p).mkv
/mnt/user/Movies/All/xXx -.Return.of.Xander.Cage.(2017)(1080p)/xXx -.Return.of.Xander.Cage.(2017)(1080p).mkv
 

Link to comment
4 hours ago, kizer said:

This is the output via User.Scripts:

Ok, now I'm confused as both code variants work.

 

Edit v0.7 and search for:

size=$(stat -c%s "$file")

Replace it against:

size=$(stat -c%s "$file")
echo "$file has a size of $size"

After execution. Do you see an massive amount of entries in the logs or does it still only return srt filenames?

Link to comment
  • mgutt changed the title to Video Preloader (avoids HDD spinup latency when starting a Movie or Episode through Plex, Jellyfin or Emby)

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.