kizer Posted October 21, 2020 Share Posted October 21, 2020 Updated my script to exactly like yours up above and then just changed the one line you asked me to change. It scanned TV shows, but ignored the movies. Keep in mind my script is just like yours with Paths. You think it would of included the movies since most of my content movies are newer and in this case alot of these are really old. Script location: /tmp/user.scripts/tmpScripts/zzzzzzz-test-zzzzzzzz/script Note that closing this window will abort the execution of this script Available RAM in Bytes: 2507377040 Amount of Videos that can be preloaded: 41 /mnt/user/TV/Shows/NCIS/Season 3/NCIS.-.S03E13.avi has a size of 367577088 /mnt/user/TV/Shows/NCIS/Season 3/NCIS.-.S03E14.avi has a size of 367546368 /mnt/user/TV/Shows/NCIS/Season 3/NCIS.-.S03E15.avi has a size of 367622144 /mnt/user/TV/Shows/NCIS/Season 3/NCIS.-.S03E16.avi has a size of 367663104 /mnt/user/TV/Shows/NCIS/Season 3/NCIS.-.S03E17.avi has a size of 367724544 /mnt/user/TV/Shows/NCIS/Season 3/NCIS.-.S03E18.avi has a size of 367560704 /mnt/user/TV/Shows/NCIS/Season 3/NCIS.-.S03E19.avi has a size of 367579136 /mnt/user/TV/Shows/NCIS/Season 3/NCIS.-.S03E20.avi has a size of 367570944 /mnt/user/TV/Shows/NCIS/Season 3/NCIS.-.S03E21.avi has a size of 367587328 /mnt/user/TV/Shows/NCIS/Season 3/NCIS.-.S03E22.avi has a size of 367566848 /mnt/user/TV/Shows/NCIS/Season 3/NCIS.-.S03E23.avi has a size of 367665152 /mnt/user/TV/Shows/NCIS/Season 3/NCIS.-.S03E24.avi has a size of 364546048 /mnt/user/TV/Shows/The Big Bang Theory/Season 1/The.Big.Bang.Theory.-.S01E01.avi has a size of 183793664 /mnt/user/TV/Shows/The Big Bang Theory/Season 1/The.Big.Bang.Theory.-.S01E02.avi has a size of 183742464 /mnt/user/TV/Shows/The Big Bang Theory/Season 1/The.Big.Bang.Theory.-.S01E03.avi has a size of 183769088 /mnt/user/TV/Shows/The Big Bang Theory/Season 1/The.Big.Bang.Theory.-.S01E04.avi has a size of 183756800 /mnt/user/TV/Shows/The Big Bang Theory/Season 1/The.Big.Bang.Theory.-.S01E05.avi has a size of 183732224 /mnt/user/TV/Shows/The Big Bang Theory/Season 1/The.Big.Bang.Theory.-.S01E06.avi has a size of 183709696 /mnt/user/TV/Shows/The Big Bang Theory/Season 1/The.Big.Bang.Theory.-.S01E07.avi has a size of 183732224 /mnt/user/TV/Shows/The Big Bang Theory/Season 1/The.Big.Bang.Theory.-.S01E08.avi has a size of 183703552 /mnt/user/TV/Shows/The Big Bang Theory/Season 1/The.Big.Bang.Theory.-.S01E09.avi has a size of 183711744 /mnt/user/TV/Shows/The Big Bang Theory/Season 1/The.Big.Bang.Theory.-.S01E10.avi has a size of 183672832 /mnt/user/TV/Shows/The Big Bang Theory/Season 1/The.Big.Bang.Theory.-.S01E11.avi has a size of 183674880 /mnt/user/TV/Shows/The Big Bang Theory/Season 1/The.Big.Bang.Theory.-.S01E12.avi has a size of 183674880 /mnt/user/TV/Shows/The Big Bang Theory/Season 1/The.Big.Bang.Theory.-.S01E13.avi has a size of 183746560 /mnt/user/TV/Shows/The Big Bang Theory/Season 1/The.Big.Bang.Theory.-.S01E14.avi has a size of 183701504 /mnt/user/TV/Shows/The Big Bang Theory/Season 1/The.Big.Bang.Theory.-.S01E15.avi has a size of 183732224 /mnt/user/TV/Shows/The Big Bang Theory/Season 1/The.Big.Bang.Theory.-.S01E16.avi has a size of 183719936 /mnt/user/TV/Shows/The Big Bang Theory/Season 1/The.Big.Bang.Theory.-.S01E17.avi has a size of 183703552 /mnt/user/TV/Shows/The Big Bang Theory/Season 2/The.Big.Bang.Theory.-.S02E07.avi has a size of 183415570 /mnt/user/TV/Shows/The Big Bang Theory/Season 2/The.Big.Bang.Theory.-.S02E08.avi has a size of 182377724 /mnt/user/TV/Shows/The Big Bang Theory/Season 2/The.Big.Bang.Theory.-.S02E09.avi has a size of 182779678 /mnt/user/TV/Shows/Torchwood/Season 1/Torchwood.-.S01E06.avi has a size of 367685632 /mnt/user/TV/Shows/Torchwood/Season 1/Torchwood.-.S01E07.avi has a size of 367747072 /mnt/user/TV/Shows/Torchwood/Season 1/Torchwood.-.S01E08.avi has a size of 367767552 /mnt/user/TV/Shows/Torchwood/Season 1/Torchwood.-.S01E09.avi has a size of 367720448 /mnt/user/TV/Shows/Torchwood/Season 3/Torchwood.-.S03E01.avi has a size of 577108990 /mnt/user/TV/Shows/Torchwood/Season 3/Torchwood.-.S03E02.avi has a size of 577187912 /mnt/user/TV/Shows/Torchwood/Season 3/Torchwood.-.S03E03.avi has a size of 577134280 /mnt/user/TV/Shows/Torchwood/Season 3/Torchwood.-.S03E04.avi has a size of 577226288 /mnt/user/TV/Shows/Torchwood/Season 3/Torchwood.-.S03E05.avi has a size of 576900174 Preload /mnt/user/Movies/All/Rush.Hour.3.(2007)(720p)(DTS)/Rush.Hour.3.(2007)(720p).srt Preload /mnt/user/Movies/All/Waiting.(2005)(720p)(AC3)/Waiting.(2005)(720p).srt Preload /mnt/user/Movies/All/Rush.Hour.2.(2001)(720p)(AC3)/Rush.Hour.2.(2001)(720p).srt Preload /mnt/user/Movies/All/The.Librarian.The.Curse.Of.The.Judas.Chalice.(2008)(720p)(AC3)/The.Librarian.The.Curse.Of.The.Judas.Chalice.(2008)(720p).srt Preload /mnt/user/Movies/1080-Bray-Rips/Avatar/Avatar.srt Preload /mnt/user/TV/Shows/Adventure Time/Season 7/Adventure.Time.-.S07E35_eng.srt Preload /mnt/user/TV/Shows/Adventure Time/Season 7/Adventure.Time.-.S07E34_eng.srt Preload /mnt/user/TV/Shows/Adventure Time/Season 7/Adventure.Time.-.S07E33_eng.srt Preload /mnt/user/TV/Shows/Adventure Time/Season 7/Adventure.Time.-.S07E32_eng.srt Preload /mnt/user/TV/Shows/Adventure Time/Season 7/Adventure.Time.-.S07E31_eng.srt Preload /mnt/user/TV/Shows/Adventure Time/Season 8/Adventure.Time.-.S08E03_eng.srt Preload /mnt/user/TV/Shows/Adventure Time/Season 8/Adventure.Time.-.S08E01E01_eng.srt Preload /mnt/user/TV/Shows/Top Secret Recipe/Season 1/Top.Secret.Recipe.-.S01E08.srt Preload /mnt/user/TV/Shows/Dexter/Season 4/Dexter.-.S04E01.srt It also gave me a popup error message saying No video file has been proloaded (wrong path?) Quote Link to comment
kizer Posted October 21, 2020 Share Posted October 21, 2020 However if I remove the /mnt/user/TV/ from the line it seems to process just fine. I chose only to use 20% RAM. However when I look at files being accessed it shows a movie playing, but unraid isn't indicating a drive being spun up even well into the movie. Its like unraid doesn't know a drive is spun up or is simply failing to indicate it. when I get a chance I’ll fire up my other machine running 6.8. Quote Link to comment
mgutt Posted October 21, 2020 Author Share Posted October 21, 2020 Ok you found a bug. I cut the list, before I check the size. This means if "video_min_size" is bigger than the first x video files, it preloads nothing. I will change that in the next version. Feel free to reduce "video_min_size" or do you want to skip all these episodes because of their size? Quote Link to comment
mgutt Posted October 21, 2020 Author Share Posted October 21, 2020 v0.8 released: - Bug fix: In some situations video files were skipped instead of preloading them 2 Quote Link to comment
casperse Posted October 21, 2020 Share Posted October 21, 2020 Hi mgutt Just wanted to let you know that I just tried your script, and I think its working 100% Logfile: Preload /mnt/user/movies/Movies/Friends with Kids (2012)/Friends with Kids (2012) [Bluray.720p.x264.8Bit.DTS.5.1].mkv (6.835s) Preload /mnt/user/movies/Movies/Transylvania 6-5000 (1985)/Transylvania 6-5000 (1985) [Bluray.720p.x264.8Bit.DTS.2.0].mkv (0.337s) Script Starting Oct 21, 2020 20:20.51 Full logs for this script are available at /tmp/user.scripts/tmpScripts/Plex Preloader v0.7/log.txt Script Finished Oct 21, 2020 20:20.51 Update: Did a test added "TV Shows" and I could see srt files being loaded (Running Unraid 6.8.3) So how to get 1TB of memory!!! LOL Great work, you have really made a big impact on my Plex server! Quote Link to comment
casperse Posted October 21, 2020 Share Posted October 21, 2020 (edited) Version 0.8 did something with the srt files it keeps adding srt files forever - without any TV show clip, maybee the size is under the limit? Edited October 21, 2020 by casperse Quote Link to comment
mgutt Posted October 21, 2020 Author Share Posted October 21, 2020 (edited) 2 hours ago, casperse said: it keeps adding srt files forever - without any TV show clip Are you sure? SRT files are loaded AFTER all clips haven been preloaded. And yes, it loads ALL SRT files into the RAM (as they are small and I'm too lazy to write a function that checks dependencies between clips and SRT files ^^). Edited October 21, 2020 by mgutt Quote Link to comment
casperse Posted October 23, 2020 Share Posted October 23, 2020 On 10/21/2020 at 10:50 PM, mgutt said: Are you sure? SRT files are loaded AFTER all clips haven been preloaded. And yes, it loads ALL SRT files into the RAM (as they are small and I'm too lazy to write a function that checks dependencies between clips and SRT files ^^). All of the srt files? for Movies & TV shows?! WOAW I have a average of 3-4 srt files (Nordic TV sometime also german close to DK - for each TV show and most movies) how does it have enough space left for loading the right subtitles? Just did a count on TV shows alone and I have around +40K srt files - so now I understand why it did run "forever" Average srt file is 30-45K = 1G Ram (TV only) Quote Link to comment
mgutt Posted October 23, 2020 Author Share Posted October 23, 2020 4 minutes ago, casperse said: Average srt file is 30-45K = 1G Ram (TV only) No problem as the Preloader default setting is 50% of the free RAM. This means there are still 50% left for the subtitles. Except you set a much higher value or you have much less of free RAM. But you are right, the best would be to parse only those folders, which contain preloaded video files. Will think about that. Quote Link to comment
casperse Posted October 23, 2020 Share Posted October 23, 2020 3 hours ago, mgutt said: No problem as the Preloader default setting is 50% of the free RAM. This means there are still 50% left for the subtitles. Except you set a much higher value or you have much less of free RAM. But you are right, the best would be to parse only those folders, which contain preloaded video files. Will think about that. Maybe it's not the size of the srt files but the fact that they are located across 24 drives? Anyway your code is solid my server doesn't crash and it still run perfectly! just take some time LOL (I kept it at default 50%) Quote Link to comment
mgutt Posted October 24, 2020 Author Share Posted October 24, 2020 v0.9 released: - Preloads only subtitle files that belong to preloaded video files Now the preloading should be much faster if your collection contains many SRT files. Here is an example from my logs: 1 Quote Link to comment
casperse Posted October 24, 2020 Share Posted October 24, 2020 (edited) WOw = 3 min! Script Starting Oct 24, 2020 11:59.04 Full logs for this script are available at /tmp/user.scripts/tmpScripts/Plex Preloader v0.9/log.txt Available RAM in Bytes: 11931406300 Amount of Videos that can be preloaded: 195 ....*.mkv*.srt tr: write error: Broken pipe tr: write error cut: write error: Broken pipe Script Finished Oct 24, 2020 12:02.09 Would like to do a test of the impact doing this but struggling how to do accomplish it? Needs to be "remote" for transcoding to kick in, how did you go about this during your dev/testing? Also some people have "remote transcode settings" fixed, would they overrule this? like 720P 4Mb/s and make the Preloader obsolete? Sorry for the many Q's again great work! your work in so many ways are really moving the "Media tweak" post forward! Edited October 24, 2020 by casperse Quote Link to comment
mgutt Posted October 24, 2020 Author Share Posted October 24, 2020 (edited) 14 hours ago, casperse said: ....*.mkv*.srt tr: write error: Broken pipe tr: write error cut: write error: Broken pipe Hmm need to check this error 😕 EDIT: Ok, seems to be no problem. It's only because I interrupt the loop when enough videos have been preloaded. Will try to fix this someday. Edited October 25, 2020 by mgutt Quote Link to comment
mgutt Posted October 25, 2020 Author Share Posted October 25, 2020 (edited) 14 hours ago, casperse said: Would like to do a test of the impact doing this but struggling how to do accomplish it? Needs to be "remote" for transcoding to kick in, how did you go about this during your dev/testing? Also some people have "remote transcode settings" fixed, would they overrule this? like 720P 4Mb/s and make the Preloader obsolete? Live-Transcoding should use the preloaded file part as well. This is independent from the resolution or bitrate as the source file is everytime the same. You could verify it as follows: Execute the following to clean the RAM: sync; echo 1 > /proc/sys/vm/drop_caches Spindown disks and wait 1 minute to be sure that the HDDs does not rotate anymore. Start a movie through your mobile client without Wifi (to force transcoding) and measure the delay. After that clean the RAM again: sync; echo 1 > /proc/sys/vm/drop_caches Execute the preload script. Spindown disks, wait 1 minute. Start a movie that was preloaded. This time the movie should start faster which proofs that even transcoding uses the preloaded data. Edited October 25, 2020 by mgutt Quote Link to comment
diggforbeer Posted October 26, 2020 Share Posted October 26, 2020 (edited) This is amazing! I got it running and its caching about 8.9% of my movies. Is there any chance we can get it to use a spare ssd? With that I could cache my entire collection. Edited October 26, 2020 by diggforbeer Quote Link to comment
dada051 Posted October 26, 2020 Share Posted October 26, 2020 7 hours ago, diggforbeer said: This is amazing! I got it running and its caching about 8.9% of my movies. Is there any chance we can get it to use a spare ssd? With that I could cache my entire collection. Or maybe use the cache pool? Quote Link to comment
mgutt Posted October 26, 2020 Author Share Posted October 26, 2020 @diggforbeer @dada051 I still was not able to find a solution to move files, that are cached in the RAM to the swap partition/file. This website claims the Swap is used if the RAM is full, but I was not able to provoke writing directly to the Swap. As long this is not possible, we can not use a SSD Quote Link to comment
dada051 Posted October 26, 2020 Share Posted October 26, 2020 Ok. I will have just to buy more RAM Quote Link to comment
primeval_god Posted October 26, 2020 Share Posted October 26, 2020 6 hours ago, mgutt said: I still was not able to find a solution to move files, that are cached in the RAM to the swap partition/file. This website claims the Swap is used if the RAM is full, but I was not able to provoke writing directly to the Swap. As long this is not possible, we can not use a SSD Theoretically would the script be effective if it was run within a docker container (specifically a long running container that periodically invoked the script rather than spinning up a new container for each invocation)? If so then might it be possible to use the per-container swappiness settings of docker to achieve the above? Quote Link to comment
mgutt Posted October 26, 2020 Author Share Posted October 26, 2020 (edited) 3 hours ago, primeval_god said: If so then might it be possible to use the per-container swappiness settings of docker to achieve the above? No, because we still aren't able to control which data is moved to the swap. And finally I think its isolated from the host system (where Plex is reading the data from). This is the explanation when Linux writes to Swap: https://tldp.org/LDP/tlk/mm/memory.html Quote 3.8 Swapping Out and Discarding Pages When physical memory becomes scarce the Linux memory management subsystem must attempt to free physical pages 1.) This refers only to used RAM. But the script does not use RAM. It uses only the Buffer Cache (available or "free" RAM). And those data is never written to the Swap. 2.) The only solution to really reserve the RAM by preloading a file would be vmtouch. But I don't know how to force Linux to move this reserved data to the Swap 3.) Someone mentioned, that cgroups are a solution to have isolated settings for a specfic process. This means it seems to be possible to set for example "100MB" RAM for the process "vmtouch" and after it exceeds this value it should write to swap (theoretically). But I was not able to find the right commands to test this scenario. 4.) I was able to install vmtouch, but it does not work very well. Sometimes it does not output the cached data and sometimes the locking daemon process crashes. I would say its not reliable under Slackware which is a huge problem for this project as its the only tool (I know of) which is able to lock data in the RAM. Edited October 26, 2020 by mgutt Quote Link to comment
primeval_god Posted October 27, 2020 Share Posted October 27, 2020 14 hours ago, mgutt said: 3.) Someone mentioned, that cgroups are a solution to have isolated settings for a specfic process. This means it seems to be possible to set for example "100MB" RAM for the process "vmtouch" and after it exceeds this value it should write to swap (theoretically). But I was not able to find the right commands to test this scenario. 4.) I was able to install vmtouch, but it does not work very well. Sometimes it does not output the cached data and sometimes the locking daemon process crashes. I would say its not reliable under Slackware which is a huge problem for this project as its the only tool (I know of) which is able to lock data in the RAM. While I dont know for sure how the docker isolation plays into things, i do know that underneath docker is using Cgroups to do its thing. Just an additional thought, while i know nothing about vmtouch, if it worked inside of a docker container that would enable using a version on top of another base os. Quote Link to comment
mgutt Posted October 27, 2020 Author Share Posted October 27, 2020 47 minutes ago, primeval_god said: While I dont know for sure how the docker isolation plays into things, i do know that underneath docker is using Cgroups to do its thing. Just an additional thought, while i know nothing about vmtouch, if it worked inside of a docker container that would enable using a version on top of another base os. You are right, at least we should test it. Quote Link to comment
kizer Posted October 28, 2020 Share Posted October 28, 2020 Ok increased my Ram to 32GB and gave this a try. I'm getting the following .. Script location: /tmp/user.scripts/tmpScripts/zzzzzzz-test-zzzzzzzz/script Note that closing this window will abort the execution of this script Available RAM in Bytes: 7500920825 Amount of Videos that can be preloaded: 122 Preload /mnt/user/Movies/All/Roald.Dahl's.The.Witches.(2020)(1080p)(AC3)/Roald.Dahl's.The.Witches.(2020)(1080p).mkv (0.480s) Preload /mnt/user/Movies/All/Unhinged.(2020)(1080p)(AC3)/Unhinged.(2020)(1080p).mkv (0.496s) Preload /mnt/user/Movies/All/My.Spy.(2020)(720p)(DTS)/My.Spy.(2020)(720p).mkv (0.488s) Preload /mnt/user/Movies/All/The.Banana.Splits.Movie.(2019)(720p)(AC3)/The.Banana.Splits.Movie.(2019)(720p).avi (0.507s) ~ ~ ~ tr: write error: Broken pipe tr: write error cut: write error: Broken pipe I'm still getting drive spin ups right away. So I'm trying to adjust the parameters. Maybe something is right. Can you see the cached files in /tmp? I've looked and didn't see anything so I'm just curious to where the files are being stored. I think setting it to the first 100meg I might see something working here using Plex on AppleTV. I'll keep tinkering with it and let you know. Its a little excessive, but It's all work in progress right. 😃 Quote Link to comment
mgutt Posted October 28, 2020 Author Share Posted October 28, 2020 (edited) 9 hours ago, kizer said: tr: write error: Broken pipe tr: write error cut: write error: Broken pipe This error is not important, but of course I will try to fix this. 9 hours ago, kizer said: Can you see the cached files in /tmp? I've looked and didn't see anything so I'm just curious to where the files are being stored. Preloading is done only to available / free RAM. You can't see or measure the preloaded data. If you re-execute the script you should get a notification with the amount of files that were cached / skipped. If all files were skipped, you can be sure that the preloading works. 9 hours ago, kizer said: I think setting it to the first 100meg I might see something working here using Plex on AppleTV Which HDDs are you using? This completely depends on the spinup speed. It seems your HDDs are a little bit slow in spinup compared to mine. The slowest HDDs I know of (in spin up) are Ironwolfs (those hated WD SMR drives are freaking fast): I made this test by spinning down the HDD and copying a 500MB file through Teracopy (Windows) to the HDD. Teracopy finally shows the transfer time which is a good value to compare the latency. Important: You need to wait 1 to 2 minutes after a spindown or the HDD is still spinning on your next test = spins up faster than usual. Edited October 28, 2020 by mgutt Quote Link to comment
kizer Posted October 28, 2020 Share Posted October 28, 2020 I know it's working now. I did a little test for fun. I upped the amount it caches to 300MB and had some low quality shows that it appears to completely cache. Watched nearly half of an episode before I got bored and moved on and the drive never spun up. It was kinda cool actually. I did notice a strange thing thou. It was caching some really old stuff and I couldn't figure out why. Their date was changed to 2038-01-18 for some odd reason. Not the scripts fault, but something I have to figure out because it'll always cache those files that are really old not new. LOL Is there a code change that you could recommend that says to skip those or ignore them? If so it would really really make my experience better. Other wise I have to dig and I do mean dig to change them all. I think I was trying to test files that was on my IronWolf. Maybe thats why it never appeared to work from the start. By increasing the amount it Cache I think I really saw the power because it gave me a lot more room to work with. Obviously I'm not going to keep the amount at 200. I put it back down to 60 and I might tweak it some, but so far so good. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.