SpaceInvaderOne Posted November 29, 2018 Share Posted November 29, 2018 Hi Guys been thinking about how to get sonarr and radarr to auto re-encode downloads automatically using handbrake. There will be 2 parts to this and i will post the next under this one in a few days time. This video the first part shows how to automatically re-encode video files downloaded by sonarr or radarr using an advanced docker path mapping sending the media files through handbrake first before then being processed by sonarr/radarr. I use h265 as i want smaller video files but any format can be choosen. This first part goes through the principles of how this works. The second video will go deeper using detailed examples. It is recommended to watch this video first. PART ONE PART 2 6 3 Quote Link to comment
Josh.5 Posted December 1, 2018 Share Posted December 1, 2018 (edited) @SpaceInvaderOne Hey thanks for this video. This helped me a lot! I needed to find a solution for my lack of space. Like you I have a library that is full of un-optimised videos. I think this tutorial is a pretty good idea for new incoming videos. However I think it would be nice to also have a background process that can optimise my current files as well. I also often find that I download a video and want to watch it straight away. Perhaps something went wrong with the episode that was downloaded and I want to mark it as failed and download another copy. The problem I found with your setup here is I need to wait for the conversion after the download before I can watch that video. I believe a better solution is to have a monitor watching your library for new files. When a new file comes in, it will convert it (in a separate folder) and then replace that file once the conversion is complete. Last night I wrote a container which has a monitor that watches your library for file changes. New files that match a list of file extensions are checked to see if they are already 265. If they are not they are converted and then replaced. It also has an optional scheduled event to carry out a full scan of your library. I'm still testing it on my library (it's been running for about 9 hours and is working well so far). Credit to you for pointing me down this road. It's a genius solution over searching for a download in a HEVC format Would you be interested in trying out my container? I should have it finished and pushed today. I would appreciate the feedback. It doesn't have a GUI or anything. Just some variables to specify the quality you want (HEVC) and point it at your library. Josh5 Edited December 1, 2018 by Josh.5 1 Quote Link to comment
SpaceInvaderOne Posted December 2, 2018 Author Share Posted December 2, 2018 Hi, @Josh.5 yes it sounds great. Please let me know when its ready. I would love to try your container. Sounds like a much more elegant solution than mine and much easier. It is annoying having to wait for conversion before watching. Have you seen the unraid port of FallingSnow's H265ize? I had a quick look at that yesterday as it will convert whole folders full of video and was thinking of using it to convert some of my existing media. But your container sounds like it takes that one step further, can't wait to try it. Quote Link to comment
torn8o Posted December 2, 2018 Share Posted December 2, 2018 20 hours ago, Josh.5 said: @SpaceInvaderOne Hey thanks for this video. This helped me a lot! I needed to find a solution for my lack of space. Like you I have a library that is full of un-optimised videos. I think this tutorial is a pretty good idea for new incoming videos. However I think it would be nice to also have a background process that can optimise my current files as well. I also often find that I download a video and want to watch it straight away. Perhaps something went wrong with the episode that was downloaded and I want to mark it as failed and download another copy. The problem I found with your setup here is I need to wait for the conversion after the download before I can watch that video. I believe a better solution is to have a monitor watching your library for new files. When a new file comes in, it will convert it (in a separate folder) and then replace that file once the conversion is complete. Last night I wrote a container which has a monitor that watches your library for file changes. New files that match a list of file extensions are checked to see if they are already 265. If they are not they are converted and then replaced. It also has an optional scheduled event to carry out a full scan of your library. I'm still testing it on my library (it's been running for about 9 hours and is working well so far). Credit to you for pointing me down this road. It's a genius solution over searching for a download in a HEVC format Would you be interested in trying out my container? I should have it finished and pushed today. I would appreciate the feedback. It doesn't have a GUI or anything. Just some variables to specify the quality you want (HEVC) and point it at your library. Josh5 I'd be interested in trying out your docker. I do not seam to have handbreak setup correctly. Handbreak is not converting files or it is just under 100 mbps. Are there any tutorials on setting up handbreak? Quote Link to comment
Josh.5 Posted December 2, 2018 Share Posted December 2, 2018 @SpaceInvaderOne I did look initially into H265ize. However I chose not to follow that path as it did not let me simply monitor my current library. I considered building on top of H265ize, but he effort required to modify it was about equal to starting from scratch. Like I said, I pointed it to my library and have had it running for over a day now. If you wish to try it out/contribute the source is sitting here: https://github.com/Josh5/docker-libraryoptimise Add a new container pulling from the "josh5/libraryoptimise:latest" dockerhub repo. (I've attached a screenshot of my current config) Name: LibraryOptimise Repository: josh5/libraryoptimise:latest Variable: PUID=99 Variable: PGID=100 Path: /library:/mnt/user/TVShows/Kids/Paw Patrol/:rw Path: /cache:/mnt/user/transcodercache/libraryoptimise/:rw Variable: DEBUGGING=false Variable: SCHEDULE_FULL_SCAN_MINS=60 Other options that I've left as defaults (defaults listed): Variable: SUPPORTED_CONTAINERS=mkv,avi,mov,ts,rmvb,mp4 Variable: VIDEO_CODEC=hevc Variable: AUDIO_CODEC=aac Variable: AUDIO_STEREO_STREAM_BITRATE=128k Variable: OUT_CONTAINER=mkv Variable: REMOVE_SUBTITLE_STREAMS=true For my initial test I did not want it to go ham on my library, so I isolated it to on TV series (TVShows/Kids/Paw Patrol). I would recommend that anyone who wants to try this while we are still "testing" it out also points it to an isolated folder. The /cache folder is where the output is during encoding. This should be pointed to our ssd if you have one. This will run on 2 threads with one running the full library scan and the other watching for new files. This means that you can have up to 2 simultaneous encoding jobs at once. Please let me know if this should be improved on to allow configuring more encoding tasks or if you fell that this is fine. I would appreciate feedback on defaults for the variables. Or if you feel like there should be additional configuration options. I'm not sure it is worth creating a GUI for this, that would be a lot of work for no extra functionality on what it does. I think it would be better to refine the defaults for people so for non-advanced users can just point it at their library and go. Thanks for taking a look guys. Really appreciate it. PS. This is tested on a Ryzen setup (no GPU). The config here is not using any special encoder configuration. HEVC is just using a libx265 encoder. If you do have a setup that could utilise HW acceleration and know how this should be executed with ffmpeg, then let me know and I'll add it. Quote Link to comment
trekkiedj Posted December 3, 2018 Share Posted December 3, 2018 14 hours ago, Josh.5 said: @SpaceInvaderOne I did look initially into H265ize. However I chose not to follow that path as it did not let me simply monitor my current library. I considered building on top of H265ize, but he effort required to modify it was about equal to starting from scratch. Like I said, I pointed it to my library and have had it running for over a day now. If you wish to try it out/contribute the source is sitting here: https://github.com/Josh5/docker-libraryoptimise Add a new container pulling from the "josh5/libraryoptimise:latest" dockerhub repo. (I've attached a screenshot of my current config) Name: LibraryOptimise Repository: josh5/libraryoptimise:latest Variable: PUID=99 Variable: PGID=100 Path: /library:/mnt/user/TVShows/Kids/Paw Patrol/:rw Path: /cache:/mnt/user/transcodercache/libraryoptimise/:rw Variable: DEBUGGING=false Variable: SCHEDULE_FULL_SCAN_MINS=60 Other options that I've left as defaults (defaults listed): Variable: SUPPORTED_CONTAINERS=mkv,avi,mov,ts,rmvb,mp4 Variable: VIDEO_CODEC=hevc Variable: AUDIO_CODEC=aac Variable: AUDIO_STEREO_STREAM_BITRATE=128k Variable: OUT_CONTAINER=mkv Variable: REMOVE_SUBTITLE_STREAMS=true For my initial test I did not want it to go ham on my library, so I isolated it to on TV series (TVShows/Kids/Paw Patrol). I would recommend that anyone who wants to try this while we are still "testing" it out also points it to an isolated folder. The /cache folder is where the output is during encoding. This should be pointed to our ssd if you have one. This will run on 2 threads with one running the full library scan and the other watching for new files. This means that you can have up to 2 simultaneous encoding jobs at once. Please let me know if this should be improved on to allow configuring more encoding tasks or if you fell that this is fine. I would appreciate feedback on defaults for the variables. Or if you feel like there should be additional configuration options. I'm not sure it is worth creating a GUI for this, that would be a lot of work for no extra functionality on what it does. I think it would be better to refine the defaults for people so for non-advanced users can just point it at their library and go. Thanks for taking a look guys. Really appreciate it. PS. This is tested on a Ryzen setup (no GPU). The config here is not using any special encoder configuration. HEVC is just using a libx265 encoder. If you do have a setup that could utilise HW acceleration and know how this should be executed with ffmpeg, then let me know and I'll add it. Testing this now. If this works as well as i hope, this could be the solution i have been looking for for ages! Thanks so much! Quote Link to comment
Josh.5 Posted December 4, 2018 Share Posted December 4, 2018 Testing this now. If this works as well as i hope, this could be the solution i have been looking for for ages! Thanks so much!How have did you get on so far? And thoughts?Sent from my ONE E1003 using Tapatalk Quote Link to comment
trekkiedj Posted December 4, 2018 Share Posted December 4, 2018 5 hours ago, Josh.5 said: How have did you get on so far? And thoughts? Sent from my ONE E1003 using Tapatalk I am extremely impressed. I do have a couple of thoughts though: I noticed that you mentioned that it would run as two threads, one for the main library conversion and one to monitor new additions. I haven't seen this is the case as only one transcode is happening at any one time and new additions are remaining unconverted. I like many will likely have separate shares for movies and series. How will this cope with that? I don't want to just enter /mnt/user as the Library path for obvious reasons. I just want /mnt/user/Series and /mnt/user/Movies Got to say though that this is stellar work and i don't know why someone hasn't thought of it before! Well done that man! Quote Link to comment
SpaceInvaderOne Posted December 4, 2018 Author Share Posted December 4, 2018 Hi @Josh.5 Great work. I have had a play with the container and it really nice. My thoughts 1. I can't get the container not to scan the library folder. Setting the variable to zero and when the container starts it will still start going through the files that are there. It would be good if it could be disabled for the scan to happen when the container starts. 2. I am used to looking in the log on the handbrake container to see how far through an encode job is. It would be great if we could see this in the log too. 3. If the container is stopped then the temp file is left. This isn't a problem if the container is started again and it does the same job again. However if for some reason it doesn't (maybe the source file has been moved or deleted) it is left behind. Maybe when the container starts it should erase that temp folder. 4. I agree with @trekkiedj multiple library folders would be great. Quote Link to comment
Josh.5 Posted December 4, 2018 Share Posted December 4, 2018 Thanks for the feedback guys. @trekkiedj Regarding the threads, I will take another look. Yesterday I started re-writing the app to support "workers". This will mean you can configure a number of worker threads (1-10) to handle encoding jobs. Both the inotify watcher and library scan will then just add files to a queue to be handled simultaneously by the workers. This will probably take me a few days to complete as I have a few other projects that need attention this week. @trekkiedj @SpaceInvaderOne You both suggest supporting multiplie library folders. Would it be sufficient to just pass through both locations to the containers /library path? Eg. Path: /library/Series:/mnt/user/Series:rw Path: /library/Movies:/mnt/user/Movies:rw @SpaceInvaderOne Currently there is not a way for the container not to scan the library. This is the primary function that it does on start. Are you suggesting that I change the variable "SCHEDULE_FULL_SCAN_MINS" so that if it is "0" or "off" it will not scan the library at all? If so I'll add this also. The logging issue you raise is very difficult, especially seeing as I am now working on adding up to 10 workers at a time. Having multiple processes writing to a single log file will become very confusing, even if it is piped. You will end up with the logs of all of the processes being mixed together in the log file. I think the most elegant solution would be to write a web UI to display more in-depth progress of the conversion processes. I will rais this as a TODO for when I get time. I will also look into a solution for removing old cache files in this coming week. On the subject of cache files, might I also suggest setting this to a tmpfs on the host rather than an ssd. If you have enough spare ram on your device than this should give you a performance boost and reduce wear on the ssd. Thanks again for all the feedback guys. It is much appreciated. Quote Link to comment
SpaceInvaderOne Posted December 4, 2018 Author Share Posted December 4, 2018 @Josh.5 yes, having the SCHEDULE_FULL_SCAN_MINS set to 0 and it not scanning the library at all, would be great. Most of the time I would just like to point the container at my media and it only convert new things as they are added. However, having the ability to disable the inotify watcher in the template as well may be useful too. I was thinking of using 2 instances of the container. One which would run 24/7 doing only the inotify watcher encodes (probably just limited to using a couple of cores) and the other instance, I would have user scripts start at night and stop in the morning working on only the library encodes but using all the available cores. Yes, using the ram for the cache files is a good idea but I guess one would have to keep the worker threads low otherwise it would use too much ram. I was wondering what the use case of using multiple worker threads beyond one for each library location and one for the inotify watcher would be. I did notice that when running the container on 8 cores it would just about max out those cores. But running it on 16 cores only about 45% of each core is being used. So I guess having more workers would saturate higher core counts? Or is there another use case? Oh also, I forgot to ask before would it be possible to be able to adjust the various video encoding quality settings, please? (although the ones used seem to produce good results that I have seen) Anyway just want to thank you for all you work on this container its great Quote Link to comment
trekkiedj Posted December 4, 2018 Share Posted December 4, 2018 @Josh.5 That path option would work fine for me. Will the container at this point accept more than one path variable? Quote Link to comment
trekkiedj Posted December 4, 2018 Share Posted December 4, 2018 And totally agree with @SpaceInvaderOne. This is amazing work. Thanks so much. Quote Link to comment
Josh.5 Posted December 4, 2018 Share Posted December 4, 2018 5 minutes ago, trekkiedj said: @Josh.5 That path option would work fine for me. Will the container at this point accept more than one path variable? A docker container can accept as many paths as you wish. It will only complain if you attempt to bind them on top of each other. For example, you can bind "/foo:/container/bar" and "/baz:/container/bar/baz" just fine. But you cannot bind "/foo:/container/bar" and "/baz:/container/bar" as the second bind mount will conflict with the first. @SpaceInvaderOne I will look at implementing these features as you suggest this week. I'll let you know when it is done. As for the workers, to be honest that is uncharted territory for me. Like you mentioned, I believe if you have more cores available to encode files, you may find disk read speed will bottleneck the process. My thinking is that most people should see a performance boost in carrying out 2-3 jobs at a time (more if you have a better setup with more cores available). It is all theoretical at the moment. We would need to run tests to know for sure. Could you give me an idea of what you need for config options on the video encoding? At the moment the video stream is simply converted to x265 keeping the same framerate and resolution as the source. Quote Link to comment
trekkiedj Posted December 4, 2018 Share Posted December 4, 2018 2 hours ago, Josh.5 said: A docker container can accept as many paths as you wish. It will only complain if you attempt to bind them on top of each other. For example, you can bind "/foo:/container/bar" and "/baz:/container/bar/baz" just fine. But you cannot bind "/foo:/container/bar" and "/baz:/container/bar" as the second bind mount will conflict with the first. I tried adding both paths as you suggested and nothing really happened. It didn't attempt to find and transcode h264/avi files. Quote Link to comment
Josh.5 Posted December 4, 2018 Share Posted December 4, 2018 I tried adding both paths as you suggested and nothing really happened. It didn't attempt to find and transcode h264/avi files.Are you able to post some details of the container config? Possibly even a screenshot?Sent from my ONE E1003 using Tapatalk Quote Link to comment
Josh.5 Posted December 5, 2018 Share Posted December 5, 2018 I've opened some issues on github: https://github.com/Josh5/docker-libraryoptimise/issues Feel free to open any more as you see fit Quote Link to comment
trekkiedj Posted December 5, 2018 Share Posted December 5, 2018 8 hours ago, Josh.5 said: Are you able to post some details of the container config? Possibly even a screenshot? Sent from my ONE E1003 using Tapatalk OK so i found that the container would crash if i mapped container path /library/Movies to /mnt/user/Movies and then another mapped path of /library/Series to /mnt/user/Series The configuration below though works just fine: Quote Link to comment
trekkiedj Posted December 5, 2018 Share Posted December 5, 2018 Also, i guess the only other thought I have on this would be to have the ability to schedule this. Say overnight or any fixed schedule. Quote Link to comment
Josh.5 Posted December 5, 2018 Share Posted December 5, 2018 These are really good suggestions. I've decided to create a web UI (probably just a simple thing nothing flash). This should solve all of your problems @trekkiedj and give you guys better control over what can be done and what cannot. I have opened issues for the features you have requested: https://github.com/Josh5/docker-libraryoptimise/issues If you think of any others feel free to open them there also. Quote Link to comment
trekkiedj Posted December 5, 2018 Share Posted December 5, 2018 @Josh.5 this for me is a really exciting development. As a huge media fan, this solves a major hurdle in getting my library of some 150 odd series and nearly 2500 movies optimised to the latest standard. Thanks so much for your work on this! Quote Link to comment
trekkiedj Posted December 6, 2018 Share Posted December 6, 2018 @Josh.5 Just an update on the crashing. It seems it is something to do with encoding. It only happens if the Movies path is used. I removed some movies it was having issues with but then became apparent that it was every movie!: Quote Link to comment
Josh.5 Posted December 6, 2018 Share Posted December 6, 2018 [mention=76627]Josh.5[/mention] Just an update on the crashing. It seems it is something to do with encoding. It only happens if the Movies path is used. I removed some movies it was having issues with but then became apparent that it was every movie!: Oh right. Cheers for the log. That is very helpfulSent from my ONE E1003 using Tapatalk Quote Link to comment
trekkiedj Posted December 13, 2018 Share Posted December 13, 2018 Just to add now i have been using for a week: Still cant seem to get the container to encode the Movie library as above. With TV series I did some playing on using the SSD cache pool vs /tmp on the RAM and saw no noticeable performance increase. Other than that, it is rock solid and works a treat. Quote Link to comment
munit85 Posted December 15, 2018 Share Posted December 15, 2018 How hard would it be to setup a minimum file-size target for this? I'm imagining pointing this at my library of tv shows and having it re-encode all files greater than 3GB or whatever in an attempt to get the largest files encoded to smaller file size first. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.