Video Preloader (avoids HDD spinup latency when starting a Movie or Episode through Plex, Jellyfin or Emby)

[mgutt]

31 minutes ago, DJ-BrianC said:

You can see by all my previous posts which have never been addressed.

I don't really understand why it does not work for you. Feel free to send me a PM if I can access your PC remotely to do some tests.

[bubbadk]

1 hour ago, DJ-BrianC said:

It doesn't work. You can see by all my previous posts which have never been addressed. The script needs some serious work before it can be released to the general public.

it works for me and i don't know a thing about programming and scripting

[DJ-BrianC]

8 hours ago, mgutt said:

I don't really understand why it does not work for you. Feel free to send me a PM if I can access your PC remotely to do some tests.

I don't know but I'd provided all the screenshots from the last bunch of tests back in April. What else do you need me to provide In addition to what was sent in April? I'd love to see this work but for whatever reason it doesn't seem to.

[Nanobug]

Do you know if this would work on another Linux system, Debian or something else?

[mgutt]

1 hour ago, Nanobug said:

would work on another Linux system

Should.

[xyber411]

This is awesome!  Used it just fine after a small bit of tweaking on an Ubuntu server. Running very well. Small suggestion though: I have a few directories which should always be pre-loaded, such as nostalgic shows that are frequently accessed. Is there a way we could add an option to $force-preload=(“dir/to/show” “and/other/show” list/of/shows.txt) to place these at the top of the preload list always?

 

My thoughts are that I’d use tatuli to prepare a list of most accessed or recently played shows, and export that list to run in this script in the future, but for now, I’ll just list them manually. 
 

also, instead of sorting the list by date created / modified, sorting them by date accessed could be more beneficial for some, like this:
 

# find all video files
while IFS= read -r -d '' file; do
  video_files+=("$file")
done < <(find "${video_paths[@]}" -not -path '*/.*' -size +"$video_min_size"c -regextype posix-extended -regex ".*\.($video_ext)" -printf "%a@ %p\0")

 

PS, I’m using this to preload my servers videos from google drive. 😄

Edited August 12, 2022 by xyber411

[mgutt]

31 minutes ago, xyber411 said:

Is there a way we could add an option to $force-preload=(“dir/to/show” “and/other/show” list/of/shows.txt) to place these at the top of the preload list always?

I will consider this for a future release.

 

PS As a workaround you could create two different cronjobs with two different scripts containing those paths.

 

33 minutes ago, xyber411 said:

also, instead of sorting the list by date created / modified, sorting them by date accessed could be more beneficial for some.

Will consider this, too. Maybe its useful to have a "mixture" of access and modify date, like x % last accessed and y % last modified?!

 

37 minutes ago, xyber411 said:

My thoughts are that I’d use tatuli to prepare a list of most accessed or recently played shows

Will consider this, too.

 

38 minutes ago, xyber411 said:

Used it just fine after a small bit of tweaking on an Ubuntu server.

Any other things, which I should consider so it runs without any changes on Ubuntu, too?

 

On 7/28/2022 at 5:44 AM, DJ-BrianC said:

What else do you need me to provide In addition to what was sent in April?

As I said. The very best option would be a remote access, so I can debug the script line by line. At the moment I'm totally out of ideas why it does not work for you.

[xyber411]

My tweaks were to the ubuntu system, such as using /tmp/ for a ramdisk in addition to /run/.  I've also done a few other things to make my ubuntu server much like unraid.

 

For now, I've taken your advice with the two separate jobs.  A question though, using it like this, isn't it more likely that the second to run will overwrite some of the items stored on the first run?  Like, I gave the first run 50% of the RAM to use, then I gave 20% to the second run for a total of 70%, but I'm pretty sure it's not working like that...  I think linux is just going to store whatever it feels like, thus, sometimes that 20% will completely overwrite some of the 50%, and thus, only 50% total memory gets utilized.  Or am I just completely incorrect, and linux will utilize whatever is actually considered free first, then utilize whatever has to be overwritten second?  IDK.

[interwebtech]

Updated to 6.11.1 last night. This morning, Preloader failed with the following error:

 

Script Starting Oct 07, 2022 07:00.01

Full logs for this script are available at /tmp/user.scripts/tmpScripts/Plex_Preloader/log.txt

Script Finished Oct 07, 2022 07:00.03

Full logs for this script are available at /tmp/user.scripts/tmpScripts/Plex_Preloader/log.txt

2022-10-07 07:00:03 Available RAM: 39G
2022-10-07 07:00:03 Amount of videos that can be preloaded: 474 (each video occupies 81M)
2022-10-07 07:00:03 /tmp/user.scripts/tmpScripts/Plex_Preloader/script: line 113: syntax error near unexpected token `head'
2022-10-07 07:00:03 /tmp/user.scripts/tmpScripts/Plex_Preloader/script: line 113: ` seconds=$(time ( head -c "$preload_head_size" "$file" ) 2>&1 1>/dev/null )'

 

Edited October 7, 2022 by interwebtech

[mgutt]

20 minutes ago, interwebtech said:

Updated to 6.11.1 last night. This morning, Preloader failed with the following error:

I updated the script. Should work now.

[interwebtech]

7 minutes ago, mgutt said:

I updated the script. Should work now.

Error moved but still persists

 

Script location: /tmp/user.scripts/tmpScripts/Plex_Preloader/script
Note that closing this window will abort the execution of this script
2022-10-07 10:16:22 Available RAM: 30G
2022-10-07 10:16:22 Amount of videos that can be preloaded: 482 (each video occupies 61M)
2022-10-07 10:16:22 /tmp/user.scripts/tmpScripts/Plex_Preloader/script: line 106: syntax error near unexpected token `head'
2022-10-07 10:16:22 /tmp/user.scripts/tmpScripts/Plex_Preloader/script: line 106: ` seconds=$(time ( head -c "$preload_head_size" "$file" >/dev/null ) 2>&1 )'

 

[Archangel]

23 hours ago, interwebtech said:

Error moved but still persists

 

Script location: /tmp/user.scripts/tmpScripts/Plex_Preloader/script
Note that closing this window will abort the execution of this script
2022-10-07 10:16:22 Available RAM: 30G
2022-10-07 10:16:22 Amount of videos that can be preloaded: 482 (each video occupies 61M)
2022-10-07 10:16:22 /tmp/user.scripts/tmpScripts/Plex_Preloader/script: line 106: syntax error near unexpected token `head'
2022-10-07 10:16:22 /tmp/user.scripts/tmpScripts/Plex_Preloader/script: line 106: ` seconds=$(time ( head -c "$preload_head_size" "$file" >/dev/null ) 2>&1 )'

 

Can confirm, I'm also having this same error.

[mgutt]

On 10/7/2022 at 7:17 PM, interwebtech said:

Error moved but still persists

 

On 10/8/2022 at 6:49 PM, Archangel said:

Can confirm, I'm also having this same error.

 

Please test again

[mgutt]

3 minutes ago, Archangel said:

if there's been no changes?

There are changes. The script in the first post is updated. I changed several lines which are not visible in your screenshot.

[Archangel]

Version 1.3 has fixed the issue and it's running again.  Didn't see the changes at first, apologies.

[Archangel]

Thanks for the update!

Edited October 9, 2022 by Archangel

[mgutt]

Did you really copy the whole script? The "fi" is important as it is the end of the new "if else"-part:

 

[Archangel]

1 minute ago, mgutt said:

Did you really copy the whole script? The "fi" is important as it is the end of the new "if else"-part:

 

As I mentioned before my edit, I don't need the script to check if I have video files in my videos folder, I know they're there so I removed that part of the script, so the fi is not necessary for the edited version I'm running.  Appreciate the reply.

[interwebtech]

9 hours ago, mgutt said:

 

 

Please test again

 

Script completes without error. Thanks!
 

Unraid Status: 2022-10-10 00:48
Plex Preloader has finished
0 preloaded (from Disk) / 475 skipped (already in RAM)

 

Edited October 10, 2022 by interwebtech

[VideoVibrations]

Is there limit to the length of a path?

 

This path works: "/mnt/user/Multimedia/TV"

 

But this does not:  "/mnt/user/Multimedia/TV/Antenna"

 

But the working path will scan the "Antenna" folder and subsequent subfolders.

 

 

 

 

[mgutt]

8 hours ago, VideoVibrations said:

But this does not:  "/mnt/user/Multimedia/TV/Antenna"

 

You tested a single path? Does it contain any special chars / whitespace or is it really 1:1 as you posted?

 

Aren't any messages returned by the script?

[VideoVibrations]

1 hour ago, mgutt said:

You tested a single path? Does it contain any special chars / whitespace or is it really 1:1 as you posted?

 

Aren't any messages returned by the script?

Previously I ran multiple paths but I don't have enough ram for all my files.  I wanted to narrow it down to the shows I use most often. 

 

The paths are exactly as written.  Some of the longer paths that did work like /mnt/user/Multimedia/Movies/PBS Specials work just fine.  Is the script sensitive to white spaces?

 

When it fails it says "Plex Preloader failed! /mnt/user/Multimedia/TV/Antenna is not used by a docker container!"

[mgutt]

1 minute ago, VideoVibrations said:

When it fails it says "Plex Preloader failed! /mnt/user/Multimedia/TV/Antenna is not used by a docker container!"

You need to remove this part:

# check if paths are used in docker containers
if docker info > /dev/null 2>&1; then
  # get docker mounts of all running containers
  # shellcheck disable=SC2016
  docker_mounts=$(docker ps -q | xargs docker container inspect -f '{{$id := .Id}}{{range .Mounts}}{{if .Source}}{{printf $id}}:{{.Source}}{{println}}{{end}}{{end}}' | grep -v -e "^$")
  for path in "${video_paths[@]}"; do
    if [[ $docker_mounts != *"$path"* ]]; then
      /usr/local/emhttp/webGui/scripts/notify -i alert -s "Plex Preloader failed!" -d "$path is not used by a docker container!"
      exit 1
    fi
  done
fi

 

The script tries to verify if the path is used in a docker container, so the correct path is used for preloading videos (some users use /mnt/disk1/Movies and if you preload /mnt/user/Movies it would have no effect). I think I will change this part in one of the next versions, so it checks only the first subdir level.

 

[VideoVibrations]

success!

 

I noticed in testing that the RAM data will hardware transcode in RAM as well.  Pretty awesome script. 

Edited October 12, 2022 by VideoVibrations

[relink]

Quote

If this would be possible the complete Movie collection could be preloaded to a "small" swap file located on an SSD.

Have you had any luck with this? I have been trying to use rclone mount with vfs_cache to do something similar and having no luck at all.