Video Preloader (avoids HDD spinup latency when starting a Movie or Episode through Plex, Jellyfin or Emby)


Recommended Posts

Updated my script to exactly like yours up above and then just changed the one line you asked me to change. It scanned TV shows, but ignored the movies. Keep in mind my script is just like yours with Paths. You think it would of included the movies since most of my content movies are newer and in this case alot of these are really old.  

 

Script location: /tmp/user.scripts/tmpScripts/zzzzzzz-test-zzzzzzzz/script
Note that closing this window will abort the execution of this script
Available RAM in Bytes: 2507377040
Amount of Videos that can be preloaded: 41
/mnt/user/TV/Shows/NCIS/Season 3/NCIS.-.S03E13.avi has a size of 367577088
/mnt/user/TV/Shows/NCIS/Season 3/NCIS.-.S03E14.avi has a size of 367546368
/mnt/user/TV/Shows/NCIS/Season 3/NCIS.-.S03E15.avi has a size of 367622144
/mnt/user/TV/Shows/NCIS/Season 3/NCIS.-.S03E16.avi has a size of 367663104
/mnt/user/TV/Shows/NCIS/Season 3/NCIS.-.S03E17.avi has a size of 367724544
/mnt/user/TV/Shows/NCIS/Season 3/NCIS.-.S03E18.avi has a size of 367560704
/mnt/user/TV/Shows/NCIS/Season 3/NCIS.-.S03E19.avi has a size of 367579136
/mnt/user/TV/Shows/NCIS/Season 3/NCIS.-.S03E20.avi has a size of 367570944
/mnt/user/TV/Shows/NCIS/Season 3/NCIS.-.S03E21.avi has a size of 367587328
/mnt/user/TV/Shows/NCIS/Season 3/NCIS.-.S03E22.avi has a size of 367566848
/mnt/user/TV/Shows/NCIS/Season 3/NCIS.-.S03E23.avi has a size of 367665152
/mnt/user/TV/Shows/NCIS/Season 3/NCIS.-.S03E24.avi has a size of 364546048
/mnt/user/TV/Shows/The Big Bang Theory/Season 1/The.Big.Bang.Theory.-.S01E01.avi has a size of 183793664
/mnt/user/TV/Shows/The Big Bang Theory/Season 1/The.Big.Bang.Theory.-.S01E02.avi has a size of 183742464
/mnt/user/TV/Shows/The Big Bang Theory/Season 1/The.Big.Bang.Theory.-.S01E03.avi has a size of 183769088
/mnt/user/TV/Shows/The Big Bang Theory/Season 1/The.Big.Bang.Theory.-.S01E04.avi has a size of 183756800
/mnt/user/TV/Shows/The Big Bang Theory/Season 1/The.Big.Bang.Theory.-.S01E05.avi has a size of 183732224
/mnt/user/TV/Shows/The Big Bang Theory/Season 1/The.Big.Bang.Theory.-.S01E06.avi has a size of 183709696
/mnt/user/TV/Shows/The Big Bang Theory/Season 1/The.Big.Bang.Theory.-.S01E07.avi has a size of 183732224
/mnt/user/TV/Shows/The Big Bang Theory/Season 1/The.Big.Bang.Theory.-.S01E08.avi has a size of 183703552
/mnt/user/TV/Shows/The Big Bang Theory/Season 1/The.Big.Bang.Theory.-.S01E09.avi has a size of 183711744
/mnt/user/TV/Shows/The Big Bang Theory/Season 1/The.Big.Bang.Theory.-.S01E10.avi has a size of 183672832
/mnt/user/TV/Shows/The Big Bang Theory/Season 1/The.Big.Bang.Theory.-.S01E11.avi has a size of 183674880
/mnt/user/TV/Shows/The Big Bang Theory/Season 1/The.Big.Bang.Theory.-.S01E12.avi has a size of 183674880
/mnt/user/TV/Shows/The Big Bang Theory/Season 1/The.Big.Bang.Theory.-.S01E13.avi has a size of 183746560
/mnt/user/TV/Shows/The Big Bang Theory/Season 1/The.Big.Bang.Theory.-.S01E14.avi has a size of 183701504
/mnt/user/TV/Shows/The Big Bang Theory/Season 1/The.Big.Bang.Theory.-.S01E15.avi has a size of 183732224
/mnt/user/TV/Shows/The Big Bang Theory/Season 1/The.Big.Bang.Theory.-.S01E16.avi has a size of 183719936
/mnt/user/TV/Shows/The Big Bang Theory/Season 1/The.Big.Bang.Theory.-.S01E17.avi has a size of 183703552
/mnt/user/TV/Shows/The Big Bang Theory/Season 2/The.Big.Bang.Theory.-.S02E07.avi has a size of 183415570
/mnt/user/TV/Shows/The Big Bang Theory/Season 2/The.Big.Bang.Theory.-.S02E08.avi has a size of 182377724
/mnt/user/TV/Shows/The Big Bang Theory/Season 2/The.Big.Bang.Theory.-.S02E09.avi has a size of 182779678
/mnt/user/TV/Shows/Torchwood/Season 1/Torchwood.-.S01E06.avi has a size of 367685632
/mnt/user/TV/Shows/Torchwood/Season 1/Torchwood.-.S01E07.avi has a size of 367747072
/mnt/user/TV/Shows/Torchwood/Season 1/Torchwood.-.S01E08.avi has a size of 367767552
/mnt/user/TV/Shows/Torchwood/Season 1/Torchwood.-.S01E09.avi has a size of 367720448
/mnt/user/TV/Shows/Torchwood/Season 3/Torchwood.-.S03E01.avi has a size of 577108990
/mnt/user/TV/Shows/Torchwood/Season 3/Torchwood.-.S03E02.avi has a size of 577187912
/mnt/user/TV/Shows/Torchwood/Season 3/Torchwood.-.S03E03.avi has a size of 577134280
/mnt/user/TV/Shows/Torchwood/Season 3/Torchwood.-.S03E04.avi has a size of 577226288
/mnt/user/TV/Shows/Torchwood/Season 3/Torchwood.-.S03E05.avi has a size of 576900174
Preload /mnt/user/Movies/All/Rush.Hour.3.(2007)(720p)(DTS)/Rush.Hour.3.(2007)(720p).srt
Preload /mnt/user/Movies/All/Waiting.(2005)(720p)(AC3)/Waiting.(2005)(720p).srt
Preload /mnt/user/Movies/All/Rush.Hour.2.(2001)(720p)(AC3)/Rush.Hour.2.(2001)(720p).srt
Preload /mnt/user/Movies/All/The.Librarian.The.Curse.Of.The.Judas.Chalice.(2008)(720p)(AC3)/The.Librarian.The.Curse.Of.The.Judas.Chalice.(2008)(720p).srt
Preload /mnt/user/Movies/1080-Bray-Rips/Avatar/Avatar.srt
Preload /mnt/user/TV/Shows/Adventure Time/Season 7/Adventure.Time.-.S07E35_eng.srt
Preload /mnt/user/TV/Shows/Adventure Time/Season 7/Adventure.Time.-.S07E34_eng.srt
Preload /mnt/user/TV/Shows/Adventure Time/Season 7/Adventure.Time.-.S07E33_eng.srt
Preload /mnt/user/TV/Shows/Adventure Time/Season 7/Adventure.Time.-.S07E32_eng.srt
Preload /mnt/user/TV/Shows/Adventure Time/Season 7/Adventure.Time.-.S07E31_eng.srt
Preload /mnt/user/TV/Shows/Adventure Time/Season 8/Adventure.Time.-.S08E03_eng.srt
Preload /mnt/user/TV/Shows/Adventure Time/Season 8/Adventure.Time.-.S08E01E01_eng.srt
Preload /mnt/user/TV/Shows/Top Secret Recipe/Season 1/Top.Secret.Recipe.-.S01E08.srt
Preload /mnt/user/TV/Shows/Dexter/Season 4/Dexter.-.S04E01.srt

 

 

 

It also gave me a popup error message saying 

 

No video file has been proloaded (wrong path?)

Link to comment

However if I remove the /mnt/user/TV/ from the line it seems to process just fine. I chose only to use 20% RAM. 

 

However when I look at files being accessed it shows a movie playing, but unraid isn't indicating a drive being spun up even well into the movie. Its like unraid doesn't know a drive is spun up or is simply failing to indicate it. 
 

when I get a chance I’ll fire up my other machine running 6.8. 

Link to comment

Ok you found a bug. I cut the list, before I check the size. This means if "video_min_size" is bigger than the first x video files, it preloads nothing.

 

I will change that in the next version. Feel free to reduce "video_min_size" or do you want to skip all these episodes because of their size?

Link to comment

Hi mgutt

 

Just wanted to let you know that I just tried your script, and I think its working 100%

 

Logfile:

Preload /mnt/user/movies/Movies/Friends with Kids (2012)/Friends with Kids (2012) [Bluray.720p.x264.8Bit.DTS.5.1].mkv (6.835s)
Preload /mnt/user/movies/Movies/Transylvania 6-5000 (1985)/Transylvania 6-5000 (1985) [Bluray.720p.x264.8Bit.DTS.2.0].mkv (0.337s)
Script Starting Oct 21, 2020 20:20.51

Full logs for this script are available at /tmp/user.scripts/tmpScripts/Plex Preloader v0.7/log.txt

Script Finished Oct 21, 2020 20:20.51

Update: Did a test added "TV Shows" and I could see srt files being loaded (Running Unraid 6.8.3)

So how to get 1TB of memory!!! LOL

Great work, you have really made a big impact on my Plex server!

 

Link to comment
2 hours ago, casperse said:

it keeps adding srt files forever - without any TV show clip

Are you sure? SRT files are loaded AFTER all clips haven been preloaded. And yes, it loads ALL SRT files into the RAM (as they are small and I'm too lazy to write a function that checks dependencies between clips and SRT files ^^).

Edited by mgutt
Link to comment
On 10/21/2020 at 10:50 PM, mgutt said:

Are you sure? SRT files are loaded AFTER all clips haven been preloaded. And yes, it loads ALL SRT files into the RAM (as they are small and I'm too lazy to write a function that checks dependencies between clips and SRT files ^^).

All of the srt files? for Movies & TV shows?! WOAW

I have a average of 3-4 srt files (Nordic TV sometime also german close to DK - for each TV show and most movies) how does it have enough space left for loading the right subtitles?

Just did a count on TV shows alone and I have around +40K srt files - so now I understand why it did run "forever"

Average srt file is 30-45K = 1G Ram (TV only)

 

Link to comment
4 minutes ago, casperse said:

Average srt file is 30-45K = 1G Ram (TV only)

No problem as the Preloader default setting is 50% of the free RAM. This means there are still 50% left for the subtitles. Except you set a much higher value or you have much less of free RAM. But you are right, the best would be to parse only those folders, which contain preloaded video files. Will think about that.

Link to comment
3 hours ago, mgutt said:

No problem as the Preloader default setting is 50% of the free RAM. This means there are still 50% left for the subtitles. Except you set a much higher value or you have much less of free RAM. But you are right, the best would be to parse only those folders, which contain preloaded video files. Will think about that.

Maybe it's not the size of the srt files but the fact that they are located across 24 drives? 

Anyway your code is solid my server doesn't crash and it still run perfectly! just take some time LOL (I kept it at default 50%)

Link to comment

WOw = 3 min!

 

Script Starting Oct 24, 2020 11:59.04
Full logs for this script are available at /tmp/user.scripts/tmpScripts/Plex Preloader v0.9/log.txt
Available RAM in Bytes: 11931406300
Amount of Videos that can be preloaded: 195

....*.mkv*.srt 

tr: write error: Broken pipe
tr: write error
cut: write error: Broken pipe
Script Finished Oct 24, 2020 12:02.09

 

Would like to do a test of the impact doing this but struggling how to do accomplish it?

Needs to be "remote" for transcoding to kick in, how did you go about this during your dev/testing?

Also some people have "remote transcode settings" fixed, would they overrule this? like 720P 4Mb/s and make the Preloader obsolete? 

image.png.10421978575890a67e9733171ef8b2ee.png

Sorry for the many Q's again great work! your work in so many ways are really moving the "Media tweak" post forward!

Edited by casperse
Link to comment
14 hours ago, casperse said:

 

....*.mkv*.srt 

tr: write error: Broken pipe
tr: write error
cut: write error: Broken pipe

Hmm need to check this error 😕

 

EDIT: Ok, seems to be no problem. It's only because I interrupt the loop when enough videos have been preloaded. Will try to fix this someday.

Edited by mgutt
Link to comment
14 hours ago, casperse said:

Would like to do a test of the impact doing this but struggling how to do accomplish it?

Needs to be "remote" for transcoding to kick in, how did you go about this during your dev/testing?

Also some people have "remote transcode settings" fixed, would they overrule this? like 720P 4Mb/s and make the Preloader obsolete? 

Live-Transcoding should use the preloaded file part as well. This is independent from the resolution or bitrate as the source file is everytime the same.

 

You could verify it as follows:

 

Execute the following to clean the RAM:

sync; echo 1 > /proc/sys/vm/drop_caches

 

Spindown disks and wait 1 minute to be sure that the HDDs does not rotate anymore.

 

Start a movie through your mobile client without Wifi (to force transcoding) and measure the delay.

 

After that clean the RAM again:

sync; echo 1 > /proc/sys/vm/drop_caches

 

Execute the preload script.

 

Spindown disks, wait 1 minute.

 

Start a movie that was preloaded. This time the movie should start faster which proofs that even transcoding uses the preloaded data.

Edited by mgutt
Link to comment
6 hours ago, mgutt said:

I still was not able to find a solution to move files, that are cached in the RAM to the swap partition/file. This website claims the Swap is used if the RAM is full, but I was not able to provoke writing directly to the Swap. As long this is not possible, we can not use a SSD :(

Theoretically would the script be effective if it was run within a docker container (specifically a long running container that periodically invoked the script rather than spinning up a new container for each invocation)? If so then might it be possible to use the per-container swappiness settings of docker to achieve the above? 

Link to comment
3 hours ago, primeval_god said:

If so then might it be possible to use the per-container swappiness settings of docker to achieve the above?

No, because we still aren't able to control which data is moved to the swap. And finally I think its isolated from the host system (where Plex is reading the data from).

 

This is the explanation when Linux writes to Swap:

https://tldp.org/LDP/tlk/mm/memory.html

Quote

3.8  Swapping Out and Discarding Pages

When physical memory becomes scarce the Linux memory management subsystem must attempt to free physical pages

1.) This refers only to used RAM. But the script does not use RAM. It uses only the Buffer Cache (available or "free" RAM). And those data is never written to the Swap.

2.) The only solution to really reserve the RAM by preloading a file would be vmtouch. But I don't know how to force Linux to move this reserved data to the Swap

3.) Someone mentioned, that cgroups are a solution to have isolated settings for a specfic process. This means it seems to be possible to set for example "100MB" RAM for the process "vmtouch" and after it exceeds this value it should write to swap (theoretically). But I was not able to find the right commands to test this scenario.

4.) I was able to install vmtouch, but it does not work very well. Sometimes it does not output the cached data and sometimes the locking daemon process crashes. I would say its not reliable under Slackware which is a huge problem for this project as its the only tool (I know of) which is able to lock data in the RAM.

Edited by mgutt
Link to comment
14 hours ago, mgutt said:

3.) Someone mentioned, that cgroups are a solution to have isolated settings for a specfic process. This means it seems to be possible to set for example "100MB" RAM for the process "vmtouch" and after it exceeds this value it should write to swap (theoretically). But I was not able to find the right commands to test this scenario.

4.) I was able to install vmtouch, but it does not work very well. Sometimes it does not output the cached data and sometimes the locking daemon process crashes. I would say its not reliable under Slackware which is a huge problem for this project as its the only tool (I know of) which is able to lock data in the RAM.

While I dont know for sure how the docker isolation plays into things, i do know that underneath docker is using Cgroups to do its thing. Just an additional thought, while i know nothing about vmtouch, if it worked inside of a docker container that would enable using a version on top of another base os. 

Link to comment
47 minutes ago, primeval_god said:

While I dont know for sure how the docker isolation plays into things, i do know that underneath docker is using Cgroups to do its thing. Just an additional thought, while i know nothing about vmtouch, if it worked inside of a docker container that would enable using a version on top of another base os. 

You are right, at least we should test it.

Link to comment

Ok increased my Ram to 32GB and gave this a try. I'm getting the following ..

 

Script location: /tmp/user.scripts/tmpScripts/zzzzzzz-test-zzzzzzzz/script
Note that closing this window will abort the execution of this script
Available RAM in Bytes: 7500920825
Amount of Videos that can be preloaded: 122
Preload /mnt/user/Movies/All/Roald.Dahl's.The.Witches.(2020)(1080p)(AC3)/Roald.Dahl's.The.Witches.(2020)(1080p).mkv (0.480s)
Preload /mnt/user/Movies/All/Unhinged.(2020)(1080p)(AC3)/Unhinged.(2020)(1080p).mkv (0.496s)
Preload /mnt/user/Movies/All/My.Spy.(2020)(720p)(DTS)/My.Spy.(2020)(720p).mkv (0.488s)
Preload /mnt/user/Movies/All/The.Banana.Splits.Movie.(2019)(720p)(AC3)/The.Banana.Splits.Movie.(2019)(720p).avi (0.507s)

~

~

~
tr: write error: Broken pipe
tr: write error
cut: write error: Broken pipe

 

I'm still getting drive spin ups right away. So I'm trying to adjust the parameters. Maybe something is right. 

Can you see the cached files in /tmp? I've looked and didn't see anything so I'm just curious to where the files are being stored. 

 

I think setting it to the first 100meg I might see something working here using Plex on AppleTV.  I'll keep tinkering with it and let you know. Its a little excessive, but It's all work in progress right. 😃

 

Link to comment
9 hours ago, kizer said:

tr: write error: Broken pipe
tr: write error
cut: write error: Broken pipe

This error is not important, but of course I will try to fix this.

9 hours ago, kizer said:

Can you see the cached files in /tmp? I've looked and didn't see anything so I'm just curious to where the files are being stored. 

Preloading is done only to available / free RAM. You can't see or measure the preloaded data. If you re-execute the script you should get a notification with the amount of files that were cached / skipped. If all files were skipped, you can be sure that the preloading works.

9 hours ago, kizer said:

I think setting it to the first 100meg I might see something working here using Plex on AppleTV

Which HDDs are you using? This completely depends on the spinup speed. It seems your HDDs are a little bit slow in spinup compared to mine. The slowest HDDs I know of (in spin up) are Ironwolfs (those hated WD SMR drives are freaking fast):

1511691476_2020-06-0813_02_40.png.8fa2b3972e362ac96984cc7c8c0902d7.png

 

I made this test by spinning down the HDD and copying a 500MB file through Teracopy (Windows) to the HDD. Teracopy finally shows the transfer time which is a good value to compare the latency.

 

Important: You need to wait 1 to 2 minutes after a spindown or the HDD is still spinning on your next test = spins up faster than usual.

Edited by mgutt
Link to comment

I know it's working now. I did a little test for fun. I upped the amount it caches to 300MB and had some low quality shows that it appears to completely cache. Watched nearly half of an episode before I got bored and moved on and the drive never spun up. It was kinda cool actually. 

 

I did notice a strange thing thou. It was caching some really old stuff and I couldn't figure out why. Their date was changed to 2038-01-18 for some odd reason. Not the scripts fault, but something I have to figure out because it'll always cache those files that are really old not new. LOL Is there a code change that you could recommend that says to skip those or ignore them? If so it would really really make my experience better. Other wise I have to dig and I do mean dig to change them all. 

 

I think I was trying to test files that was on my IronWolf. Maybe thats why it never appeared to work from the start. By increasing the amount it Cache I think I really saw the power because it gave me a lot more room to work with. Obviously I'm not going to keep the amount at 200. I put it back down to 60 and I might tweak it some, but so far so good. 

 

 

Screen Shot 2020-10-28 at 7.06.02 AM.png

Link to comment
  • mgutt changed the title to Video Preloader (avoids HDD spinup latency when starting a Movie or Episode through Plex, Jellyfin or Emby)

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.