mgutt Posted April 18, 2022 Author Share Posted April 18, 2022 On 3/26/2022 at 9:31 PM, ClunkClunk said: You'll see my read times for cached files are about 0.4s on average with the default setting in the script set at 0.15, as such my scans never detected any cached files. Once I figured that out, I adjusted the setting in the script to 1.0s and it worked reliably. I just have a bit of a slower system than yours I guess. @DJ-BrianC At first: Please try the new version. The broken pipe is finally gone (hopefully). Then please test: How long does it take to preload a video file after you executed this command? (video files are loaded from disk) sync; echo 1 > /proc/sys/vm/drop_caches And how long does it take to preload the video file after you execute the script directly again? (video files are loaded from RAM) The second execution should be much faster as the file is located in your RAM. If not, then something strange is happening. For me it's as follows: Script location: /tmp/user.scripts/tmpScripts/plex_preload/script Note that closing this window will abort the execution of this script 2022-04-18 14:44:24 Available RAM: 22G 2022-04-18 14:44:24 Amount of videos that can be preloaded: 358 (each video occupies 61M) 2022-04-18 14:44:24 Skipped /mnt/disk1/Movie/CD/Captain Marvel (2019)/Captain Marvel (2019).mkv as loading needed only 0.016 2022-04-18 14:44:24 Skipped /mnt/disk1/Movie/CD/Denk bloß nicht, ich heule (1965)/Other/Probeaufnahmen.mkv as loading needed only 0.017 2022-04-18 14:44:24 Skipped /mnt/disk1/Movie/CD/Denk bloß nicht, ich heule (1965)/Denk bloß nicht, ich heule (1965).mkv as loading needed only 0.017 2022-04-18 14:44:24 Skipped /mnt/disk1/Movie/AB/Berlin um die Ecke (1966)/Other/Interview - Drehbuchautor Wolfgang Kohlhaase.mkv as loading needed only 0.017 2022-04-18 14:44:24 Skipped /mnt/disk1/Movie/AB/Berlin um die Ecke (1966)/Berlin um die Ecke (1966).mkv as loading needed only 0.017 2022-04-18 14:44:24 Skipped /mnt/disk1/Movie/CD/Deep Impact (1998)/Deep Impact (1998).mkv as loading needed only 0.017 2022-04-18 14:44:24 Skipped /mnt/disk1/Movie/YZ/Zorn der Titanen (2012)/Zorn der Titanen (2012).mkv as loading needed only 0.016 2022-04-18 14:44:24 Skipped /mnt/disk1/Movie/CD/The Crow - Die Krähe (1994)/The Crow- Die Krähe (1994).mkv as loading needed only 0.016 2022-04-18 14:44:25 Preloaded /mnt/disk1/Movie/CD/Crazy Heart (2009)/Crazy Heart (2009).mkv in 0.291 seconds 2022-04-18 14:44:25 Preloaded /mnt/disk1/Movie/AB/Astro Boy (2009)/Astro Boy (2009).mkv in 0.287 seconds 2022-04-18 14:44:26 Preloaded /mnt/disk1/Movie/AB/Boston (2016)/Boston (2016) 4K.mkv in 0.374 seconds 2022-04-18 14:44:26 Preloaded /mnt/disk1/Movie/CD/The Code (2009)/The Code (2009).mkv in 0.290 seconds 2022-04-18 14:44:26 Preloaded /mnt/disk1/Movie/CD/Crawl (2019)/Crawl (2019).mkv in 0.280 seconds 2022-04-18 14:44:26 Preloaded /mnt/disk1/Movie/CD/Club der roten Bänder - Wie alles begann (2019)/Club der roten Bänder - Wie alles begann (2019).mkv in 0.286 seconds 2022-04-18 14:44:27 Preloaded /mnt/disk1/Movie/CD/Dumbo (2019)/Dumbo (2019).mkv in 0.290 seconds 2022-04-18 14:44:27 Preloaded /mnt/disk1/Movie/09/(500) Days of Summer (2009)/(500) Days of Summer As you can see it takes under 0.02 seconds to load from RAM, while loading from Disk takes ~0.3 seconds. Of course the threshold of 0.150 seconds is only an estimate, but if your RAM really needs 0.4 seconds to load only 61MB of data, then it would be extremely slow. Quote Link to comment
badbadntgood Posted April 18, 2022 Share Posted April 18, 2022 Thank you for the updated version to prevent the pipe errors. Unfortunately i'm now getting an error message saying that my paths are not being used by docker. video_paths=( "/mnt/user/data/media/movies" "/mnt/user/data/media/tv" ) My paths in docker are set as /mnt/user/data/media typically but even changing to exclude the movies and tv subfolder and doing /mnt/user/data/media I get the same error message. Should I just remove the below section? # check if paths are used in docker containers if docker info > /dev/null 2>&1; then # get docker mounts of all running containers # shellcheck disable=SC2016 docker_mounts=$(docker ps -q | xargs docker container inspect -f '{{$id := .Id}}{{range .Mounts}}{{if .Source}}{{printf $id}}:{{.Source}}{{println}}{{end}}{{end}}' | grep -v -e "^$") for path in "${video_paths[@]}"; do if [[ $docker_mounts != *"$path"* ]]; then /usr/local/emhttp/webGui/scripts/notify -i alert -s "Plex Preloader failed!" -d "$path is not used by a docker container!" exit 1 fi done fi Quote Link to comment
Archangel Posted April 18, 2022 Share Posted April 18, 2022 1 minute ago, badbadntgood said: Thank you for the updated version to prevent the pipe errors. Unfortunately i'm now getting an error message saying that my paths are not being used by docker. video_paths=( "/mnt/user/data/media/movies" "/mnt/user/data/media/tv" ) My paths in docker are set as /mnt/user/data/media typically but even changing to exclude the movies and tv subfolder and doing /mnt/user/data/media I get the same error message. Should I just remove the below section? # check if paths are used in docker containers if docker info > /dev/null 2>&1; then # get docker mounts of all running containers # shellcheck disable=SC2016 docker_mounts=$(docker ps -q | xargs docker container inspect -f '{{$id := .Id}}{{range .Mounts}}{{if .Source}}{{printf $id}}:{{.Source}}{{println}}{{end}}{{end}}' | grep -v -e "^$") for path in "${video_paths[@]}"; do if [[ $docker_mounts != *"$path"* ]]; then /usr/local/emhttp/webGui/scripts/notify -i alert -s "Plex Preloader failed!" -d "$path is not used by a docker container!" exit 1 fi done fi I too am getting this error message despite using the exact same path in my dockers. I've commented out the entire section you posted above with #'s and it's working fine again. I greatly appreciate the script that the OP created, however, I'm not sure the docker check is necessary as everyone coming here likely IS running Plex/Jellyfin/Emby/etc. to utilize this script and has a grasp of what it's meant to do and how it works. It seems a bit overkill and hand-holdy if we're manually inputting our paths to media in the first place, aware of what the paths should be to access our media within Dockers. Maybe just a: "#This script is only meant to be used if you're using a docker that accesses your media at the specified paths." Would suffice instead of complicating the script. Quote Link to comment
DJ-BrianC Posted April 18, 2022 Share Posted April 18, 2022 It goes through the motions but 2nd run is no quicker and nothing gets skipped. Quote Link to comment
mgutt Posted April 18, 2022 Author Share Posted April 18, 2022 @badbadntgood @Archangel Please execute the following and check if it does really not contain /mnt/user/data/media docker ps -q | xargs docker container inspect -f '{{$id := .Id}}{{range .Mounts}}{{if .Source}}{{printf $id}}:{{.Source}}{{println}}{{end}}{{end}}' | grep -v -e "^$" 2 hours ago, Archangel said: It seems a bit overkill I added this check because some users mounted /mnt/disk2/media, /mnt/disk3/media, ... to their container and if those users would use the path /mnt/user/media in the script, the script wouldn't preload the correct files as in unRAID both paths contain the same files, but with different inodes. But I will change this to a warning if we found out why even /mnt/user/data/media does not work Quote Link to comment
mgutt Posted April 18, 2022 Author Share Posted April 18, 2022 3 minutes ago, DJ-BrianC said: It goes through the motions but 2nd run is no quicker and nothing gets skipped. Please post your results: free -h How much free RAM is calculated by the script? Quote Link to comment
DJ-BrianC Posted April 18, 2022 Share Posted April 18, 2022 like i said it goes thru the motions. movie titles do indeed echo to the screen and logfile. the last script you shared shows every folder on the drive. Quote Link to comment
DJ-BrianC Posted April 18, 2022 Share Posted April 18, 2022 total used free shared buff/cache available Mem: 31Gi 9.7Gi 6.7Gi 1.2Gi 14Gi 20Gi Swap: 0B 0B 0B Quote Link to comment
mgutt Posted April 18, 2022 Author Share Posted April 18, 2022 34 minutes ago, DJ-BrianC said: total used free shared buff/cache available Mem: 31Gi 9.7Gi 6.7Gi 1.2Gi 14Gi 20Gi Swap: 0B 0B 0B I needed only this. Please results of this as well: sysctl -a | grep dirty Quote Link to comment
DJ-BrianC Posted April 18, 2022 Share Posted April 18, 2022 total used free shared buff/cache available Mem: 31Gi 9.8Gi 9.0Gi 1.2Gi 12Gi 20Gi Swap: 0B 0B 0B vm.dirty_background_bytes = 0 vm.dirty_background_ratio = 10 vm.dirty_bytes = 0 vm.dirty_expire_centisecs = 3000 vm.dirty_ratio = 20 vm.dirty_writeback_centisecs = 500 vm.dirtytime_expire_seconds = 43200 Quote Link to comment
DJ-BrianC Posted April 19, 2022 Share Posted April 19, 2022 Any updates since my last post? Why is nothing loaded? Quote Link to comment
Nanobug Posted May 18, 2022 Share Posted May 18, 2022 Awesome script! I haven't read all the replies through, so apologies if it has already been asked. Does it preload the most recent added content, or the most popular, or a mix? Quote Link to comment
mgutt Posted May 18, 2022 Author Share Posted May 18, 2022 2 hours ago, Nanobug said: Does it preload the most recent added content, or the most popular, or a mix? Most recent. Popularity isn't an available trigger... Or we need to try using the last access time. But this will interfere with backup tools which update the access time, too. Quote Link to comment
DJ-BrianC Posted May 18, 2022 Share Posted May 18, 2022 Still no answer why it doesn't work? Quote Link to comment
Nanobug Posted May 20, 2022 Share Posted May 20, 2022 On 5/18/2022 at 1:58 PM, mgutt said: Most recent. Popularity isn't an available trigger... Or we need to try using the last access time. But this will interfere with backup tools which update the access time, too. Ah, alright. Out of curosity, why would you want to load the end of a file? And if you had a bunch of RAM, like 300 GB for example, wouldn't it be a good idea to put the preload to 1 GB for or even more? Quote Link to comment
ChatNoir Posted May 20, 2022 Share Posted May 20, 2022 1 hour ago, Nanobug said: why would you want to load the end of a file? I think it's required to preview some media files. Quote Link to comment
Nanobug Posted May 20, 2022 Share Posted May 20, 2022 8 hours ago, ChatNoir said: I think it's required to preview some media files. Ah, I didn't know that. But that makes sense Quote Link to comment
mgutt Posted May 20, 2022 Author Share Posted May 20, 2022 11 hours ago, Nanobug said: why would you want to load the end of a file? After several tests I found out that it's needed to preload a small part of the end of a video file or it won't start playing directly from RAM. 11 hours ago, Nanobug said: wouldn't it be a good idea to put the preload to 1 GB for or even more? The target of this script is to avoid waiting for an HDD spinup when you start a movie. If you raise the preload size, this doesn't change anything for the user. It's still playing directly. The only difference is that it will access the disk some seconds / minutes later, but this doesn't add any benefit for the user. Quote Link to comment
Nanobug Posted May 21, 2022 Share Posted May 21, 2022 21 hours ago, mgutt said: After several tests I found out that it's needed to preload a small part of the end of a video file or it won't start playing directly from RAM. The target of this script is to avoid waiting for an HDD spinup when you start a movie. If you raise the preload size, this doesn't change anything for the user. It's still playing directly. The only difference is that it will access the disk some seconds / minutes later, but this doesn't add any benefit for the user. Ah, I see. I'm not sure about the subject, so I'm just asking to learn, so don't get me wrong But what if you had like 10 different files that was accessed at the same time, from 10 different users, wouldn't it be beneficial to load more of it into the RAM, so it wouldn't cause it to buffer? I know it probably wouldn't be a realistic scenario, but still. Quote Link to comment
mgutt Posted May 21, 2022 Author Share Posted May 21, 2022 4 minutes ago, Nanobug said: But what if you had like 10 different files that was accessed at the same time, from 10 different users, This script preloads as much videos as possible. So if they all fit into your RAM, they will all play without buffering. Quote Link to comment
Nanobug Posted May 22, 2022 Share Posted May 22, 2022 23 hours ago, mgutt said: This script preloads as much videos as possible. So if they all fit into your RAM, they will all play without buffering. Thanks for the answer! Do you know if the script only works on Unraid? I probably will go with Unraid anyway, but I'm just curious. Quote Link to comment
bubbadk Posted July 27, 2022 Share Posted July 27, 2022 if the script runs in the background. does it then look for newly added movies Quote Link to comment
mgutt Posted July 27, 2022 Author Share Posted July 27, 2022 16 minutes ago, bubbadk said: if the script runs in the background. does it then look for newly added movies It sorts by newest movie files, yes. But it does not "run in the background". It runs whenever you start it. 1 Quote Link to comment
DJ-BrianC Posted July 27, 2022 Share Posted July 27, 2022 It doesn't work. You can see by all my previous posts which have never been addressed. The script needs some serious work before it can be released to the general public. Quote Link to comment
Archangel Posted July 27, 2022 Share Posted July 27, 2022 24 minutes ago, DJ-BrianC said: It doesn't work. You can see by all my previous posts which have never been addressed. The script needs some serious work before it can be released to the general public. Works fine for me and has for 7 months now. Just because you can't figure out what YOU'RE doing wrong, doesn't mean it doesn't work. Don't be obtuse. The author is NOT your personal IT support. Figure out your own issues like everyone else. 1 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.