Video Preloader (avoids HDD spinup latency when starting a Movie or Episode through Plex, Jellyfin or Emby)


Recommended Posts

On 3/26/2022 at 9:31 PM, ClunkClunk said:

You'll see my read times for cached files are about 0.4s on average with the default setting in the script set at 0.15, as such my scans never detected any cached files. Once I figured that out, I adjusted the setting in the script to 1.0s and it worked reliably. I just have a bit of a slower system than yours I guess.

@DJ-BrianC

 

At first: Please try the new version. The broken pipe is finally gone (hopefully).

 

Then please test:

How long does it take to preload a video file after you executed this command? (video files are loaded from disk)

sync; echo 1 > /proc/sys/vm/drop_caches

 

And how long does it take to preload the video file after you execute the script directly again? (video files are loaded from RAM)

 

The second execution should be much faster as the file is located in your RAM. If not, then something strange is happening.

 

For me it's as follows:

Script location: /tmp/user.scripts/tmpScripts/plex_preload/script
Note that closing this window will abort the execution of this script
2022-04-18 14:44:24 Available RAM: 22G
2022-04-18 14:44:24 Amount of videos that can be preloaded: 358 (each video occupies 61M)
2022-04-18 14:44:24 Skipped /mnt/disk1/Movie/CD/Captain Marvel (2019)/Captain Marvel (2019).mkv as loading needed only 0.016
2022-04-18 14:44:24 Skipped /mnt/disk1/Movie/CD/Denk bloß nicht, ich heule (1965)/Other/Probeaufnahmen.mkv as loading needed only 0.017
2022-04-18 14:44:24 Skipped /mnt/disk1/Movie/CD/Denk bloß nicht, ich heule (1965)/Denk bloß nicht, ich heule (1965).mkv as loading needed only 0.017
2022-04-18 14:44:24 Skipped /mnt/disk1/Movie/AB/Berlin um die Ecke (1966)/Other/Interview - Drehbuchautor Wolfgang Kohlhaase.mkv as loading needed only 0.017
2022-04-18 14:44:24 Skipped /mnt/disk1/Movie/AB/Berlin um die Ecke (1966)/Berlin um die Ecke (1966).mkv as loading needed only 0.017
2022-04-18 14:44:24 Skipped /mnt/disk1/Movie/CD/Deep Impact (1998)/Deep Impact (1998).mkv as loading needed only 0.017
2022-04-18 14:44:24 Skipped /mnt/disk1/Movie/YZ/Zorn der Titanen (2012)/Zorn der Titanen (2012).mkv as loading needed only 0.016
2022-04-18 14:44:24 Skipped /mnt/disk1/Movie/CD/The Crow - Die Krähe (1994)/The Crow- Die Krähe (1994).mkv as loading needed only 0.016
2022-04-18 14:44:25 Preloaded /mnt/disk1/Movie/CD/Crazy Heart (2009)/Crazy Heart (2009).mkv in 0.291 seconds
2022-04-18 14:44:25 Preloaded /mnt/disk1/Movie/AB/Astro Boy (2009)/Astro Boy (2009).mkv in 0.287 seconds
2022-04-18 14:44:26 Preloaded /mnt/disk1/Movie/AB/Boston (2016)/Boston (2016) 4K.mkv in 0.374 seconds
2022-04-18 14:44:26 Preloaded /mnt/disk1/Movie/CD/The Code (2009)/The Code (2009).mkv in 0.290 seconds
2022-04-18 14:44:26 Preloaded /mnt/disk1/Movie/CD/Crawl (2019)/Crawl (2019).mkv in 0.280 seconds
2022-04-18 14:44:26 Preloaded /mnt/disk1/Movie/CD/Club der roten Bänder - Wie alles begann (2019)/Club der roten Bänder - Wie alles begann (2019).mkv in 0.286 seconds
2022-04-18 14:44:27 Preloaded /mnt/disk1/Movie/CD/Dumbo (2019)/Dumbo (2019).mkv in 0.290 seconds
2022-04-18 14:44:27 Preloaded /mnt/disk1/Movie/09/(500) Days of Summer (2009)/(500) Days of Summer

 

As you can see it takes under 0.02 seconds to load from RAM, while loading from Disk takes ~0.3 seconds. Of course the threshold of 0.150 seconds is only an estimate, but if your RAM really needs 0.4 seconds to load only 61MB of data, then it would be extremely slow.

Link to comment

Thank you for the updated version to prevent the pipe errors.

Unfortunately i'm now getting an error message saying that my paths are not being used by docker.

video_paths=(
    "/mnt/user/data/media/movies"
    "/mnt/user/data/media/tv"
)

My paths in docker are set as /mnt/user/data/media typically but even changing to exclude the movies and tv subfolder and doing /mnt/user/data/media I get the same error message. Should I just remove the below section?

# check if paths are used in docker containers
if docker info > /dev/null 2>&1; then
  # get docker mounts of all running containers
  # shellcheck disable=SC2016
  docker_mounts=$(docker ps -q | xargs docker container inspect -f '{{$id := .Id}}{{range .Mounts}}{{if .Source}}{{printf $id}}:{{.Source}}{{println}}{{end}}{{end}}' | grep -v -e "^$")
  for path in "${video_paths[@]}"; do
    if [[ $docker_mounts != *"$path"* ]]; then
      /usr/local/emhttp/webGui/scripts/notify -i alert -s "Plex Preloader failed!" -d "$path is not used by a docker container!"
      exit 1
    fi
  done
fi

image.thumb.png.bafbbb3a64a6c5489bd231a6703d8da5.png

Link to comment
1 minute ago, badbadntgood said:

Thank you for the updated version to prevent the pipe errors.

Unfortunately i'm now getting an error message saying that my paths are not being used by docker.

video_paths=(
    "/mnt/user/data/media/movies"
    "/mnt/user/data/media/tv"
)

My paths in docker are set as /mnt/user/data/media typically but even changing to exclude the movies and tv subfolder and doing /mnt/user/data/media I get the same error message. Should I just remove the below section?

# check if paths are used in docker containers
if docker info > /dev/null 2>&1; then
  # get docker mounts of all running containers
  # shellcheck disable=SC2016
  docker_mounts=$(docker ps -q | xargs docker container inspect -f '{{$id := .Id}}{{range .Mounts}}{{if .Source}}{{printf $id}}:{{.Source}}{{println}}{{end}}{{end}}' | grep -v -e "^$")
  for path in "${video_paths[@]}"; do
    if [[ $docker_mounts != *"$path"* ]]; then
      /usr/local/emhttp/webGui/scripts/notify -i alert -s "Plex Preloader failed!" -d "$path is not used by a docker container!"
      exit 1
    fi
  done
fi

image.thumb.png.bafbbb3a64a6c5489bd231a6703d8da5.png

I too am getting this error message despite using the exact same path in my dockers.  I've commented out the entire section you posted above with #'s and it's working fine again.  

 

I greatly appreciate the script that the OP created, however, I'm not sure the docker check is necessary as everyone coming here likely IS running Plex/Jellyfin/Emby/etc. to utilize this script and has a grasp of what it's meant to do and how it works.  It seems a bit overkill and hand-holdy if we're manually inputting our paths to media in the first place, aware of what the paths should be to access our media within Dockers.  Maybe just a:

"#This script is only meant to be used if you're using a docker that accesses your media at the specified paths."

 

Would suffice instead of complicating the script.

Link to comment

@badbadntgood @Archangel

Please execute the following and check if it does really not contain /mnt/user/data/media

 

docker ps -q | xargs docker container inspect -f '{{$id := .Id}}{{range .Mounts}}{{if .Source}}{{printf $id}}:{{.Source}}{{println}}{{end}}{{end}}' | grep -v -e "^$"


 

2 hours ago, Archangel said:

It seems a bit overkill

I added this check because some users mounted /mnt/disk2/media, /mnt/disk3/media, ... to their container and if those users would use the path /mnt/user/media in the script, the script wouldn't preload the correct files as in unRAID both paths contain the same files, but with different inodes. But I will change this to a warning if we found out why even /mnt/user/data/media does not work 

 

 

 

 

Link to comment

total used free shared buff/cache available
Mem: 31Gi 9.8Gi 9.0Gi 1.2Gi 12Gi 20Gi
Swap: 0B 0B 0B
vm.dirty_background_bytes = 0
vm.dirty_background_ratio = 10
vm.dirty_bytes = 0
vm.dirty_expire_centisecs = 3000
vm.dirty_ratio = 20
vm.dirty_writeback_centisecs = 500
vm.dirtytime_expire_seconds = 43200
 

Link to comment
  • 4 weeks later...
2 hours ago, Nanobug said:

Does it preload the most recent added content, or the most popular, or a mix?

Most recent. Popularity isn't an available trigger... Or we need to try using the last access time. But this will interfere with backup tools which update the access time, too.

Link to comment
On 5/18/2022 at 1:58 PM, mgutt said:

Most recent. Popularity isn't an available trigger... Or we need to try using the last access time. But this will interfere with backup tools which update the access time, too.

Ah, alright.

Out of curosity, why would you want to load the end of a file?
And if you had a bunch of RAM, like 300 GB for example, wouldn't it be a good idea to put the preload to 1 GB for or even more?

Link to comment
11 hours ago, Nanobug said:

why would you want to load the end of a file?

After several tests I found out that it's needed to preload a small part of the end of a video file or it won't start playing directly from RAM.

 

11 hours ago, Nanobug said:

wouldn't it be a good idea to put the preload to 1 GB for or even more?

The target of this script is to avoid waiting for an HDD spinup when you start a movie. If you raise the preload size, this doesn't change anything for the user. It's still playing directly. The only difference is that it will access the disk some seconds / minutes later, but this doesn't add any benefit for the user.

Link to comment
21 hours ago, mgutt said:

After several tests I found out that it's needed to preload a small part of the end of a video file or it won't start playing directly from RAM.

 

The target of this script is to avoid waiting for an HDD spinup when you start a movie. If you raise the preload size, this doesn't change anything for the user. It's still playing directly. The only difference is that it will access the disk some seconds / minutes later, but this doesn't add any benefit for the user.

Ah, I see.

 

I'm not sure about the subject, so I'm just asking to learn, so don't get me wrong :)
But what if you had like 10 different files that was accessed at the same time, from 10 different users, wouldn't it be beneficial to load more of it into the RAM, so it wouldn't cause it to buffer?

I know it probably wouldn't be a realistic scenario, but still.

Link to comment
4 minutes ago, Nanobug said:

But what if you had like 10 different files that was accessed at the same time, from 10 different users,

This script preloads as much videos as possible. So if they all fit into your RAM, they will all play without buffering.

 

Link to comment
23 hours ago, mgutt said:

This script preloads as much videos as possible. So if they all fit into your RAM, they will all play without buffering.

 

Thanks for the answer!

Do you know if the script only works on Unraid?
I probably will go with Unraid anyway, but I'm just curious.

Link to comment
  • 2 months later...
24 minutes ago, DJ-BrianC said:

It doesn't work. You can see by all my previous posts which have never been addressed. The script needs some serious work before it can be released to the general public.

Works fine for me and has for 7 months now.  Just because you can't figure out what YOU'RE doing wrong, doesn't mean it doesn't work.  Don't be obtuse.  The author is NOT your personal IT support.  Figure out your own issues like everyone else.

  • Like 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.