mgutt Posted October 19, 2020 Share Posted October 19, 2020 This was an idea of @frodr. After several testings I came to the following script which: Caclulates how many Video files (small parts of them) will fit into 50% of the free RAM (amount can be changed) Obtains the X recent Movies / Episodes (depending on the used path) Preloads 60MB of the Video file leader and 1MB of the ending into the RAM Preloads subtitle files that belong to the preloaded Video file Now, if your disks are sleeping and you start a Movie or Episode through Plex, the client will download the Video parts from RAM and while the buffer is emptied, the HDD spins up and the buffer fills up again. This means all preloaded Movies / Episodes will start directly without delay. Donate? 🤗 Notes: It does not reserve any RAM, so your RAM stays fully available RAM preloading is not permanent and will be overwritten through server uploads/downloads, which means if you upload some huge files to your server, your RAM cache will be probably completely overwritten and by that all your nice cached movie files, too. That is the reason why I suggest to execute this script multiple times per day (only missing video files will be touched). As long the movie parts are still in the RAM, the disks do not spin up, while "rebuilding" the cache list, but the cached movie parts become "more popular" in the Linux cache. Applications like backup processes can overwrite the complete RAM cache, too. To avoid that, you should consider executing applications inside of containers and limit their RAM usage through the extra parameter "--memory=1G" (or whatever a container needs) If you suffer from buffering at the beginning of a Movie / Episode, try to raise "video_min_size" All preloaded videos can be found in the script's log (CA User Scripts) Script #!/bin/bash # ##################################### # Script: Video Preloader v1.4 # Description: Preloads the recent video files of a specific path into the RAM to bypass HDD spinup latency # Author: Marc Gutt # ######### Settings ################## video_paths=( "/mnt/user/Movie" "/mnt/user/TV" ) # the size of a video files must be at least 2GB (to exclude bonus content) video_min_size="2000MB" # we preload 60MB of the beginning of the video file into the RAM (raise this value if your video buffers after ~5 seconds) preload_head_size="60MB" # we preload 1MB of the end of the video file preload_tail_size="1MB" # if a preload is faster than 0.150 seconds than the video is already preloaded to the RAM preload_threshold="0.150" # preload only video files with specific extensions https://support.plex.tv/articles/203824396-what-media-formats-are-supported/ video_ext='avi|mkv|mov|mp4|mpeg' # preload only subtitle files with specific extensions # https://support.plex.tv/articles/200471133-adding-local-subtitles-to-your-media/#toc-1 sub_ext='srt|smi|ssa|ass|vtt' # we use 50% of our free RAM for preloading (it will be still free, check https://www.linuxatemyram.com/) free_ram_usage_percent=50 # set this to "1" to remove all preloads (this is only usefull if you want to force reading from Disk) preclean_cache=0 # notify if the execution was sucessful (errors produce notifications by default) notification=1 # ##################################### # # ######### Script #################### # timestamping logs exec &> >(stdbuf -o0 sed 's/%/%%/g' | xargs -d '\n' -I {} date '+%F %T {}') # make script race condition safe if [[ -d "/tmp/${0///}" ]] || ! mkdir "/tmp/${0///}"; then exit 1; fi; trap 'rmdir "/tmp/${0///}"' EXIT; # check user settings video_min_size="${video_min_size//[!0-9.]/}" # float filtering https://stackoverflow.com/a/19724571/318765 video_min_size=$(awk "BEGIN { print $video_min_size*1000000}") # convert MB to Bytes preload_head_size="${preload_head_size//[!0-9.]/}" preload_head_size=$(awk "BEGIN { print $preload_head_size*1000000}") preload_tail_size="${preload_tail_size//[!0-9.]/}" preload_tail_size=$(awk "BEGIN { print $preload_tail_size*1000000}") # check if paths are used in docker containers if docker info > /dev/null 2>&1; then # get docker mounts of all running containers # shellcheck disable=SC2016 docker_mounts=$(docker ps -q | xargs docker container inspect -f '{{$id := .Id}}{{range .Mounts}}{{if .Source}}{{printf $id}}:{{.Source}}{{println}}{{end}}{{end}}' | grep -v -e "^$") for path in "${video_paths[@]}"; do if [[ $docker_mounts != *"$path"* ]]; then /usr/local/emhttp/webGui/scripts/notify -i alert -s "Plex Preloader failed!" -d "$path is not used by a docker container!" exit 1 fi done fi # clean the read cache if [ "$preclean_cache" = "1" ]; then sync; echo 1 > /proc/sys/vm/drop_caches fi # preload calculation preloaded=0 skipped=0 preload_total_size=$((preload_head_size + preload_tail_size)) free_ram=$(free -b | awk '/^Mem:/{print $7}') free_ram=$((free_ram * free_ram_usage_percent / 100)) echo "Available RAM: $(numfmt --to si $free_ram)" preload_amount=$((free_ram / preload_total_size)) preload_size=$(( preload_head_size + preload_tail_size )) echo "Amount of videos that can be preloaded: $preload_amount (each video occupies $(numfmt --to si $preload_size))" # find all video files while IFS= read -r -d '' file; do video_files+=("$file") done < <(find "${video_paths[@]}" -not -path '*/.*' -size +"$video_min_size"c -regextype posix-extended -regex ".*\.($video_ext)" -printf "%T@ %p\0") # no video files found if [[ ${#video_files[@]} -eq 0 ]]; then echo "No $video_ext files found in ${video_paths[*]}!" else # sort all video files while IFS= read -r -d '' file; do video_files_sorted+=("$file") done < <(printf '%s\0' "${video_files[@]}" | sort -znr) # preload video files into RAM for i in "${!video_files_sorted[@]}"; do # stop if we reached our preload limit if [[ $i -ge $preload_amount ]]; then break; fi # remove modification time file=$(echo "${video_files_sorted[$i]}" | cut -f2- -d" ") TIMEFORMAT=%R seconds=$( { time head -c "$preload_head_size" "$file" >/dev/null; } 2>&1 ) if awk 'BEGIN {exit !('"$seconds"' >= '"$preload_threshold"')}'; then preloaded=$((preloaded + 1)) echo "Preloaded $file in $seconds seconds" else echo "Skipped $file as loading needed only $seconds" skipped=$((skipped + 1)) fi tail -c "$preload_tail_size" "$file" > /dev/null video_path=$(dirname "$file") # fetch subtitle files find "$video_path" -not -path '*/.*' -regextype posix-extended -regex ".*\.($sub_ext)" -print0 | while IFS= read -r -d '' file; do echo "Preloaded $file" cat "$file" >/dev/null done done fi # notification if [[ $preloaded -eq 0 ]] && [[ $skipped -eq 0 ]]; then /usr/local/emhttp/webGui/scripts/notify -i alert -s "Plex Preloader failed!" -d "No video file has been preloaded (wrong path?)!" elif [ "$notification" == "1" ]; then /usr/local/emhttp/webGui/scripts/notify -i normal -s "Plex Preloader has finished" -d "$preloaded preloaded (from Disk) / $skipped skipped (already in RAM)" fi 5 9 Quote Link to comment
mgutt Posted October 19, 2020 Author Share Posted October 19, 2020 (edited) I'm still experimenting on this. I like to write the preloaded movie parts directly to a swap file, but I can't find a solution for this at the moment (Linux decides on its own which data is swapped). If this would be possible the complete Movie collection could be preloaded to a "small" swap file located on an SSD. I even played around with vmtouch to make this preloading permanent (as vmtouch can lock it in the RAM), but at the moment vmtouch seems to be buggy in Slackware (vmtouch daemon becomes randomly killed and sometimes its not possible to get any caching status). Edited October 19, 2020 by mgutt 1 Quote Link to comment
Mushin Posted October 19, 2020 Share Posted October 19, 2020 Is it possible to also get this working with TV files? Maybe just for the ones displayed in On Deck? Quote Link to comment
mgutt Posted October 19, 2020 Author Share Posted October 19, 2020 (edited) Here you can see the RAM usage in the column "buff/cache". Before Plex Preloader: sync; echo 1 > /proc/sys/vm/drop_caches free -m total used free shared buff/cache available Mem: 64358 1154 60873 1027 2330 61580 Swap: 0 0 0 After: free -m total used free shared buff/cache available Mem: 64358 1216 27655 1027 35485 61405 Swap: 0 0 0 This means it used 35485 - 2330 = 33155 MB as Preload RAM Cache. After that I spun down all my disks, opened PMP and started a recently added movie and it started without any latency. I spun down all disks again and started one of my very first added movies and it took 8 seconds until it started playing. Edited October 19, 2020 by mgutt Quote Link to comment
mgutt Posted October 19, 2020 Author Share Posted October 19, 2020 (edited) 2 hours ago, Mushin said: Is it possible to also get this working with TV files? Maybe just for the ones displayed in On Deck? Yes, use the same script and change these values: video_path="/mnt/user/Movie/" video_min_size="2000MB" # 2GB, to exclude bonus content as follows (set your TV path of course): video_path="/mnt/user/TV/" video_min_size="500MB" While "500MB" is only an estimate of the minimum size of a TV episode. You could even lower it to "1MB" to cover everything. If you use both scripts parallel (to cache Movies and TV Shows) you should think about reducing this value as well: free_ram_usage_percent=50 Like 25 for TV shows and 25 for Movies or similar. Edited October 19, 2020 by mgutt 1 Quote Link to comment
mgutt Posted October 19, 2020 Author Share Posted October 19, 2020 v0.5 released Renamed the script, so it's hopefully more clear, that it covers TV Shows as well. Quote Link to comment
mgutt Posted October 19, 2020 Author Share Posted October 19, 2020 (edited) Feature Requests from Reddit: - determine which disks were involved and spin them down after script execution - multiple path support (to scan movies and tv shows through one script) realised - obtain recent tv shows through Plex database and add the X next episodes to the cache (alternative request: move complete episodes to SSD cache which could be part of an additional script) Edited October 20, 2020 by mgutt 2 1 Quote Link to comment
ClunkClunk Posted October 19, 2020 Share Posted October 19, 2020 2 hours ago, mgutt said: - obtain recent tv shows through Plex database and add the X next episodes to the cache (alternative request: move complete episodes to SSD cache which could be part of an additional script) In case it helps, I think this is how to get the On Deck from Plex using the API: wget -q -O- http://$PLEXURL:32400/library/onDeck?X-Plex-Token=$PLEXTOKEN Set $PLEXURL to your PMS's ip address, and $PLEXTOKEN to your Plex Auth Token, and it'll return with XML. You'll then need to parse out the 'file=' entries to get the filename and path, and replace the returned filepath with whatever the Docker container is mapped to (for example, I pass /tv to my Plex docker container and it maps to /mnt/user/TV on my unRAID). 3 Quote Link to comment
kizer Posted October 19, 2020 Share Posted October 19, 2020 - I was reading about the precaching and storing on on SSD on Redditt. That's a very interesting idea which would save everybody money if they have to up their Ram or simply another option. - Could you put in a pre-test or simply a way to say, "Scan 10 files and test" vs a scan everything you can and find out your existing settings are not a enough? Depending on some peoples Media and Ram it could scan for a long time just to find out they should either up the amount cached or lower. Could be something as simple as: #####Uncomment below#### #Test Scan only 10 Files #Full Scan Also Could there be a (Cache, Flush and Cache or maybe a Compare and update)? If Changes are made. No idea how you intended to address changes in users scans if they need to update their amounts or if files are added/removed. I'm assuming Plex simply wouldn't show the file, but not sure how removing a file would affect the Cache. Are you planning on adding this as a Plugin in the future? Something that could be added to CA and updated or keeping it simple as a User.Script? Both work obviously, but just something for you to think about in the future. Quote Link to comment
kizer Posted October 19, 2020 Share Posted October 19, 2020 Just gave it a try and didn't seem to cache right. Clicked on a movie that was maybe 15movies back from the newest and spooled up a drive just like it always does. No buffering just sat there and waited for the Movie to start. Quote Link to comment
mgutt Posted October 19, 2020 Author Share Posted October 19, 2020 @kizer Check if the movie is listed in the script's logs and how much time it took to read the file part. It will look similar to this: Available RAM in Bytes: 32628881400 Amount of Videos that can be preloaded: 534 ... Preload Video /mnt/user/Movie/EF/Falling Down - Ein ganz normaler Tag (1993)/Falling Down - Ein ganz normaler Tag (1993) 1080P FSK16 EN JP FR IT ES PT IMDB7.6.mkv real 0m0.545s user 0m0.007s sys 0m0.033s Now repeat the script and check the logs again. 1.) Did the "real" time of this movie fall under 0.100s? a) If not, then it was re-read from disk and the caching failed. This means other processes are blocking or permanently overwriting your RAM. Possible solutions: More RAM or execute the Plex Preloader script after your processes finished their work b) If yes, then the video file part is in the RAM and you can try to start the movie while your disks are sleeping. 2.) Did the Movie played directly? a) If not, which Player did you use and was transcoding needed? Which path do you use to include your video files in your docker container, do they use the same path that was used by the Plex Preloader script? b) If yes. It works. Quote Link to comment
kizer Posted October 19, 2020 Share Posted October 19, 2020 When I checked the log in User.Scripts it just said the following and nothing else: Available RAM in Bytes: 16G Here don't know the exact digits so I just paraphrased it. Amount of Videos that can be preloaded: 110 I used Plex to view the File on my AppleTV. I'm not 100% sure if any Transcoding was needed. Its a MKV with 6CH/AC3 Audio. I'll look again this Evening for a 2CH video or try it on TV shows since a lot of mine are ripped to 2CH. I have no issues with buying more Ram. I just want to see this work before I lay out the money to 32GB. Quote Link to comment
mgutt Posted October 19, 2020 Author Share Posted October 19, 2020 1 minute ago, kizer said: just said the following and nothing else: Then the given "video_path" seems to be wrong. Did you change it, so it fits to your movie collection path? Quote Link to comment
kizer Posted October 19, 2020 Share Posted October 19, 2020 Yes my Given movie path is /mnt/user/Movies/All/ OH NO....... I used /mnt/user/Movie/All/ I just checked my Log. I see a lot more info populating since adding the s. 🤪 I'll let that run until I get off work and try again. Talk about feeling kinda dumb. 1 Quote Link to comment
mgutt Posted October 19, 2020 Author Share Posted October 19, 2020 v0.6 released Multiple video path support added v0.7 released Unraid Dashboard notification added (can be disabled) Benchmark for Subtitles removed Compact log output Quote Link to comment
kizer Posted October 19, 2020 Share Posted October 19, 2020 On the Multiple Video Path does it split the Paths 50/50 =50% so 25%/25% of Ram? or does it run the first path and then run to the second path until the 50% of ram is used up? Quote Link to comment
mgutt Posted October 20, 2020 Author Share Posted October 20, 2020 24 minutes ago, kizer said: or does it run the first path and then run to the second path until the 50% of ram is used up? The files of all paths are added to one huge list, which is sorted by date and by that the most recent video files of all paths will be preloaded. Example: If your RAM allows preloading of 100 videos and you added 90 new movies to your collection, it will preload only 10 episodes. This means if you prefer 25/25 you still need to use one script per path. Quote Link to comment
kizer Posted October 20, 2020 Share Posted October 20, 2020 Just gave the new Script a try and this is the output. Script location: /tmp/user.scripts/tmpScripts/Media-Preload-Movies/script Note that closing this window will abort the execution of this script Available RAM in Bytes: 6726713300 Amount of Videos that can be preloaded: 110 Preload /mnt/user/Movies/All/Rush.Hour.3.(2007)(720p)(DTS)/Rush.Hour.3.(2007)(720p).srt Preload /mnt/user/Movies/All/Waiting.(2005)(720p)(AC3)/Waiting.(2005)(720p).srt Preload /mnt/user/Movies/All/Rush.Hour.2.(2001)(720p)(AC3)/Rush.Hour.2.(2001)(720p).srt Preload /mnt/user/Movies/All/The.Librarian.The.Curse.Of.The.Judas.Chalice.(2008)(720p)(AC3)/The.Librarian.The.Curse.Of.The.Judas.Chalice.(2008)(720p).srt Preload /mnt/user/Movies/1080-Bray-Rips/Avatar/Avatar.srt Preload /mnt/user/TV/Shows/Adventure Time/Season 7/Adventure.Time.-.S07E35_eng.srt Preload /mnt/user/TV/Shows/Adventure Time/Season 7/Adventure.Time.-.S07E34_eng.srt Preload /mnt/user/TV/Shows/Adventure Time/Season 7/Adventure.Time.-.S07E33_eng.srt Preload /mnt/user/TV/Shows/Adventure Time/Season 7/Adventure.Time.-.S07E32_eng.srt Preload /mnt/user/TV/Shows/Adventure Time/Season 7/Adventure.Time.-.S07E31_eng.srt Preload /mnt/user/TV/Shows/Adventure Time/Season 8/Adventure.Time.-.S08E03_eng.srt Preload /mnt/user/TV/Shows/Adventure Time/Season 8/Adventure.Time.-.S08E01E01_eng.srt Preload /mnt/user/TV/Shows/Top Secret Recipe/Season 1/Top.Secret.Recipe.-.S01E08.srt Preload /mnt/user/TV/Shows/Dexter/Season 4/Dexter.-.S04E01.srt Quote Link to comment
mgutt Posted October 20, 2020 Author Share Posted October 20, 2020 5 hours ago, kizer said: Just gave the new Script a try and this is the output. This means it did only preload the subtitles or did you shorten the output? Quote Link to comment
Apollopayne35 Posted October 20, 2020 Share Posted October 20, 2020 This script looks really good and very useful. Is there a way to modify this script to pre load parts of films (4K movies) but with drives spinning. I don’t have my drives sleep much. But notice when watching 4K movies. I have a 5-10 second delay before it starts. 1080p ones is only 2-3 secs. I have a small collection of 4K movies. Was wondering if I could pre load about 180mb of each one into ram to see if it helps with start time Quote Link to comment
kizer Posted October 20, 2020 Share Posted October 20, 2020 6 hours ago, mgutt said: This means it did only preload the subtitles or did you shorten the output? Nope. I just cut and pasted your script up above and changed the paths to my Movies and TV shows. Quote Link to comment
mgutt Posted October 20, 2020 Author Share Posted October 20, 2020 5 hours ago, Apollopayne35 said: Is there a way to modify this script to pre load parts of films (4K movies) but with drives spinning. No need to modify the script. It does nothing special except of preloading data into the RAM. Maybe you suffer from a deep idle state. As an example, Seagate Ironwolf and Exos have 4 different idle states, and some of them even reduce the RPM to save energy, which raises latency: https://www.seagate.com/files/docs/pdf/en-GB/whitepaper/tp608-powerchoice-tech-provides-gb.pdf If you want to test only 4K movies you could raise "video_min_size" to "50000MB" (50GB) and if you want to raise the preloading size, change "preload_head_size" to "75MB", "150MB" or "500MB". This should be the sizes which are used through the client: https://forums.plex.tv/t/how-can-i-delay-playing-of-a-video-until-the-cache-is-full/179749/3?u=mgutt Quote Link to comment
mgutt Posted October 20, 2020 Author Share Posted October 20, 2020 1 hour ago, kizer said: Nope. I just cut and pasted your script up above and changed the paths to my Movies and TV shows. This is strange. I mean I changed the code, but a) it works for me and b) it was not a huge change 1.) Are you using unraid 6.8.3? 2.) Do you execute the script through CA User Scripts? Please test this code: movie_path="/mnt/user/Movies" echo "Process substitution test" while IFS= read -r -d '' file; do echo "$file" done < <(find "$movie_path" -iname "*.mkv" | tail -3 | tr '\n' '\0') echo "Piping test" find "$movie_path" -iname "*.mkv" | tail -3 | tr '\n' '\0' | while IFS= read -r -d '' file; do echo "$file" done You could even execute it through the WebTerminal. Both tests should return 3 filenames. If not, do you get an error message? Quote Link to comment
kizer Posted October 20, 2020 Share Posted October 20, 2020 Im running 6.9-RC30 I could fire up my other machine to test on 6.8.3 if you think there are significant changes from 6.8.3 and 6.9-RC-30 This is the output via User.Scripts: Script location: /tmp/user.scripts/tmpScripts/zzzzzzz-test-zzzzzzzz/script Note that closing this window will abort the execution of this script Process substitution test /mnt/user/Movies/All/Yours,.Mine.&.Ours.(1968)(480)/Yours,.Mine.&.Ours.(1968)(480).mkv /mnt/user/Movies/All/Zootopia.(2016)(720p)/Zootopia.(2016)(720p).mkv /mnt/user/Movies/All/xXx -.Return.of.Xander.Cage.(2017)(1080p)/xXx -.Return.of.Xander.Cage.(2017)(1080p).mkv Piping test /mnt/user/Movies/All/Yours,.Mine.&.Ours.(1968)(480)/Yours,.Mine.&.Ours.(1968)(480).mkv /mnt/user/Movies/All/Zootopia.(2016)(720p)/Zootopia.(2016)(720p).mkv /mnt/user/Movies/All/xXx -.Return.of.Xander.Cage.(2017)(1080p)/xXx -.Return.of.Xander.Cage.(2017)(1080p).mkv Quote Link to comment
mgutt Posted October 20, 2020 Author Share Posted October 20, 2020 4 hours ago, kizer said: This is the output via User.Scripts: Ok, now I'm confused as both code variants work. Edit v0.7 and search for: size=$(stat -c%s "$file") Replace it against: size=$(stat -c%s "$file") echo "$file has a size of $size" After execution. Do you see an massive amount of entries in the logs or does it still only return srt filenames? Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.