Jump to content
SpaceInvaderOne

***VIDEO GUIDE - How to Automatically Re-encode Video Downloaded by Sonarr and Radarr

23 posts in this topic Last Reply

Recommended Posts

Hi Guys been thinking about how to get sonarr and radarr to auto re-encode downloads automatically using handbrake.

There will be 2 parts to this and i will post the next under this one in a few days time.

 

This video  the first part  shows how to automatically re-encode video files downloaded by sonarr or radarr using an advanced docker path mapping sending the media files through handbrake first before then being processed by sonarr/radarr.
I use h265 as i want smaller video files but any format can be choosen. This first part goes through the principles of how this works. The second video will go deeper using detailed examples. It is recommended to watch this video first.
 

PART ONE

 

  • Like 4
  • Upvote 1

Share this post


Link to post

@SpaceInvaderOne

 

Hey thanks for this video. This helped me a lot! I needed to find a solution for my lack of space. Like you I have a library that is full of un-optimised videos.

I think this tutorial is a pretty good idea for new incoming videos. However I think it would be nice to also have a background process that can optimise my current files as well. I also often find that I download a video and want to watch it straight away. Perhaps something went wrong with the episode that was downloaded and I want to mark it as failed and download another copy. The problem I found with your setup here is I need to wait for the conversion after the download before I can watch that video. I believe a better solution is to have a monitor watching your library for new files. When a new file comes in, it will convert it (in a separate folder) and then replace that file once the conversion is complete.

Last night I wrote a container which has a monitor that watches your library for file changes. New files that match a list of file extensions are checked to see if they are already 265. If they are not they are converted and then replaced. It also has an optional scheduled event to carry out a full scan of your library.

I'm still testing it on my library (it's been running for about 9 hours and is working well so far).

Credit to you for pointing me down this road. It's a genius solution over searching for a download in a HEVC format


Would you be interested in trying out my container? I should have it finished and pushed today. I would appreciate the feedback.
It doesn't have a GUI or anything. Just some variables to specify the quality you want (HEVC) and point it at your library.

 

 

Josh5
 

Edited by Josh.5

Share this post


Link to post

Hi, @Josh.5 yes it sounds great. Please let me know when its ready. I would love to try your container. Sounds like a much more elegant solution than mine and much easier. It is annoying having to wait for conversion before watching.  Have you seen the unraid port of FallingSnow's H265ize? I had a quick look at that yesterday as it will convert whole folders full of video and was thinking of using it to convert some of my existing media. But your container sounds like it takes that one step further, can't wait to try it.

Share this post


Link to post
20 hours ago, Josh.5 said:

@SpaceInvaderOne

 

Hey thanks for this video. This helped me a lot! I needed to find a solution for my lack of space. Like you I have a library that is full of un-optimised videos.

I think this tutorial is a pretty good idea for new incoming videos. However I think it would be nice to also have a background process that can optimise my current files as well. I also often find that I download a video and want to watch it straight away. Perhaps something went wrong with the episode that was downloaded and I want to mark it as failed and download another copy. The problem I found with your setup here is I need to wait for the conversion after the download before I can watch that video. I believe a better solution is to have a monitor watching your library for new files. When a new file comes in, it will convert it (in a separate folder) and then replace that file once the conversion is complete.

Last night I wrote a container which has a monitor that watches your library for file changes. New files that match a list of file extensions are checked to see if they are already 265. If they are not they are converted and then replaced. It also has an optional scheduled event to carry out a full scan of your library.

I'm still testing it on my library (it's been running for about 9 hours and is working well so far).

Credit to you for pointing me down this road. It's a genius solution over searching for a download in a HEVC format


Would you be interested in trying out my container? I should have it finished and pushed today. I would appreciate the feedback.
It doesn't have a GUI or anything. Just some variables to specify the quality you want (HEVC) and point it at your library.

 

 

Josh5
 

I'd be interested in trying out your docker. I do not seam to have handbreak setup correctly. Handbreak is not converting files or it is just under 100 mbps. Are there any tutorials on setting up handbreak?

Share this post


Link to post

@SpaceInvaderOne

I did look initially into H265ize. However I chose not to follow that path as it did not let me simply monitor my current library. I considered building on top of H265ize, but he effort required to modify it was about equal to starting from scratch.

Like I said, I pointed it to my library and have had it running for over a day now.

 

If you wish to try it out/contribute the source is sitting here: https://github.com/Josh5/docker-libraryoptimise

 

Add a new container pulling from the "josh5/libraryoptimise:latest" dockerhub repo.

(I've attached a screenshot of my current config)

    Name: LibraryOptimise
    Repository: josh5/libraryoptimise:latest
    Variable: PUID=99
    Variable: PGID=100
    Path: /library:/mnt/user/TVShows/Kids/Paw Patrol/:rw
    Path: /cache:/mnt/user/transcodercache/libraryoptimise/:rw
    Variable: DEBUGGING=false
    Variable: SCHEDULE_FULL_SCAN_MINS=60

 

Other options that I've left as defaults (defaults listed):

 

    Variable: SUPPORTED_CONTAINERS=mkv,avi,mov,ts,rmvb,mp4
    Variable: VIDEO_CODEC=hevc
    Variable: AUDIO_CODEC=aac
    Variable: AUDIO_STEREO_STREAM_BITRATE=128k
    Variable: OUT_CONTAINER=mkv
    Variable: REMOVE_SUBTITLE_STREAMS=true

 

 

For my initial test I did not want it to go ham on my library, so I isolated it to on TV series (TVShows/Kids/Paw Patrol).  I would recommend that anyone who wants to try this while we are still "testing" it out also points it to an isolated folder.

 

The /cache folder is where the output is during encoding. This should be pointed to our ssd if you have one.

 

This will run on 2 threads with one running the full library scan and the other watching for new files. This means that you can have up to 2 simultaneous encoding jobs at once. Please let me know if this should be improved on to allow configuring more encoding tasks or if you fell that this is fine.

 

I would appreciate feedback on defaults for the variables. Or if you feel like there should be additional configuration options. I'm not sure it is worth creating a GUI for this, that would be a lot of work for no extra functionality on what it does. I think it would be better to refine the defaults for people so for non-advanced users can just point it at their library and go.

 

 

Thanks for taking a look guys. Really appreciate it.

 

 

PS. This is tested on a Ryzen setup (no GPU). The config here is not using any special encoder configuration. HEVC is just using a libx265 encoder. If you do have a setup that could utilise HW acceleration and know how this should be executed with ffmpeg, then let me know and I'll add it.

 

 

Selection_999(024).png

Share this post


Link to post
14 hours ago, Josh.5 said:

@SpaceInvaderOne

I did look initially into H265ize. However I chose not to follow that path as it did not let me simply monitor my current library. I considered building on top of H265ize, but he effort required to modify it was about equal to starting from scratch.

Like I said, I pointed it to my library and have had it running for over a day now.

 

If you wish to try it out/contribute the source is sitting here: https://github.com/Josh5/docker-libraryoptimise

 

Add a new container pulling from the "josh5/libraryoptimise:latest" dockerhub repo.

(I've attached a screenshot of my current config)

    Name: LibraryOptimise
    Repository: josh5/libraryoptimise:latest
    Variable: PUID=99
    Variable: PGID=100
    Path: /library:/mnt/user/TVShows/Kids/Paw Patrol/:rw
    Path: /cache:/mnt/user/transcodercache/libraryoptimise/:rw
    Variable: DEBUGGING=false
    Variable: SCHEDULE_FULL_SCAN_MINS=60

 

Other options that I've left as defaults (defaults listed):

 

    Variable: SUPPORTED_CONTAINERS=mkv,avi,mov,ts,rmvb,mp4
    Variable: VIDEO_CODEC=hevc
    Variable: AUDIO_CODEC=aac
    Variable: AUDIO_STEREO_STREAM_BITRATE=128k
    Variable: OUT_CONTAINER=mkv
    Variable: REMOVE_SUBTITLE_STREAMS=true

 

 

For my initial test I did not want it to go ham on my library, so I isolated it to on TV series (TVShows/Kids/Paw Patrol).  I would recommend that anyone who wants to try this while we are still "testing" it out also points it to an isolated folder.

 

The /cache folder is where the output is during encoding. This should be pointed to our ssd if you have one.

 

This will run on 2 threads with one running the full library scan and the other watching for new files. This means that you can have up to 2 simultaneous encoding jobs at once. Please let me know if this should be improved on to allow configuring more encoding tasks or if you fell that this is fine.

 

I would appreciate feedback on defaults for the variables. Or if you feel like there should be additional configuration options. I'm not sure it is worth creating a GUI for this, that would be a lot of work for no extra functionality on what it does. I think it would be better to refine the defaults for people so for non-advanced users can just point it at their library and go.

 

 

Thanks for taking a look guys. Really appreciate it.

 

 

PS. This is tested on a Ryzen setup (no GPU). The config here is not using any special encoder configuration. HEVC is just using a libx265 encoder. If you do have a setup that could utilise HW acceleration and know how this should be executed with ffmpeg, then let me know and I'll add it.

 

 

Selection_999(024).png

Testing this now. If this works as well as i hope, this could be the solution i have been looking for for ages!

 

Thanks so much!

Share this post


Link to post
Testing this now. If this works as well as i hope, this could be the solution i have been looking for for ages!
 
Thanks so much!
How have did you get on so far? And thoughts?

Sent from my ONE E1003 using Tapatalk

Share this post


Link to post
5 hours ago, Josh.5 said:

How have did you get on so far? And thoughts?

Sent from my ONE E1003 using Tapatalk
 

I am extremely impressed. I do have a couple of thoughts though:

  • I noticed that you mentioned that it would run as two threads, one for the main library conversion and one to monitor new additions. I haven't seen this is the case as only one transcode is happening at any one time and new additions are remaining unconverted.
  • I like many will likely have separate shares for movies and series. How will this cope with that? I don't want to just enter /mnt/user as the Library path for obvious reasons. I just want /mnt/user/Series and /mnt/user/Movies

 

Got to say though that this is stellar work and i don't know why someone hasn't thought of it before! Well done that man!

Share this post


Link to post

Hi @Josh.5

 

Great work. I have had a play with the container and it really nice. :)

My thoughts

1.  I can't get the container not to scan the library folder. Setting the variable to zero and when the container starts it will still start going

through the files that are there. It would be good if it could be disabled for the scan to happen when the container starts.

2. I am used to looking in the log on the handbrake container to see how far through an encode job is. It would be great if we could see this in the log too.

3. If the container is stopped then the temp file is left. This isn't a problem if the container is started again and it does the same job again. However if for some reason it doesn't (maybe the source file has been moved or deleted)  it is left behind. Maybe when the container starts it should erase that temp folder.

4. I agree with @trekkiedj multiple library folders would be great.

 

 

 

Share this post


Link to post


Thanks for the feedback guys.

 

@trekkiedj
Regarding the threads, I will take another look. Yesterday I started re-writing the app to support "workers". This will mean you can configure a number of worker threads (1-10) to handle encoding jobs. Both the inotify watcher and library scan will then just add files to a queue to be handled simultaneously by the workers. This will probably take me a few days to complete as I have a few other projects that need attention this week.

 

@trekkiedj @SpaceInvaderOne You both suggest supporting multiplie library folders. Would it be sufficient to just pass through both locations to the containers /library path? Eg.
    Path: /library/Series:/mnt/user/Series:rw
    Path: /library/Movies:/mnt/user/Movies:rw

 

@SpaceInvaderOne Currently there is not a way for the container not to scan the library. This is the primary function that it does on start. Are you suggesting that I change the variable "SCHEDULE_FULL_SCAN_MINS" so that if it is "0" or "off" it will not scan the library at all? If so I'll add this also.
The logging issue you raise is very difficult, especially seeing as I am now working on adding up to 10 workers at a time. Having multiple processes writing to a single log file will become very confusing, even if it is piped. You will end up with the logs of all of the processes being mixed together in the log file. I think the most elegant solution would be to write a web UI to display more in-depth progress of the conversion processes. I will rais this as a TODO for when I get time.
I will also look into a solution for removing old cache files in this coming week.


On the subject of cache files, might I also suggest setting this to a tmpfs on the host rather than an ssd. If you have enough spare ram on your device than this should give you a performance boost and reduce wear on the ssd.


Thanks again for all the feedback guys. It is much appreciated.

Share this post


Link to post

@Josh.5  yes, having the SCHEDULE_FULL_SCAN_MINS set to 0 and it not scanning the library at all, would be great. Most of the time I would just like to point the container at my media and it only convert new things as they are added.
However, having the ability to disable the inotify watcher in the template as well may be useful too. I was thinking of using 2 instances of the container. One which would run 24/7 doing only the inotify watcher encodes (probably just limited to using a couple of cores) and the other instance, I would have user scripts start at night and stop in the morning working on only the library encodes but using all the available cores.  

Yes, using the ram for the cache files is a good idea but I guess one would have to keep the worker threads low otherwise it would use too much ram.

I was wondering what the use case of using multiple worker threads beyond one for each library location and one for the inotify watcher would be. I did notice that when running the container on 8 cores it would just about max out those cores. But running it on 16  cores only about 45% of each core is being used. So I guess having more workers would saturate higher core counts? Or is there another use case?

 

Oh also, I forgot to ask before would it be possible to be able to adjust the various video encoding quality settings, please? (although the ones used seem to produce good results that I have seen)

 

Anyway just want to thank you for all you work on this container its great

 

Share this post


Link to post

 

5 minutes ago, trekkiedj said:

@Josh.5 That path option would work fine for me. Will the container at this point accept more than one path variable?

A docker container can accept as many paths as you wish. It will only complain if you attempt to bind them on top of each other. For example, you can bind "/foo:/container/bar" and "/baz:/container/bar/baz" just fine. But you cannot bind "/foo:/container/bar" and "/baz:/container/bar" as the second bind mount will conflict with the first.

 

@SpaceInvaderOne

I will look at implementing these features as you suggest this week. I'll let you know when it is done.


As for the workers, to be honest that is uncharted territory for me. Like you mentioned, I believe if you have more cores available to encode files, you may find disk read speed will bottleneck the process. My thinking is that most people should see a performance boost in carrying out 2-3 jobs at a time (more if you have a better setup with more cores available). It is all theoretical at the moment. We would need to run tests to know for sure.

 

Could you give me an idea of what you need for config options on the video encoding? At the moment the video stream is simply converted to x265 keeping the same framerate and resolution as the source. 

Share this post


Link to post
2 hours ago, Josh.5 said:

 

A docker container can accept as many paths as you wish. It will only complain if you attempt to bind them on top of each other. For example, you can bind "/foo:/container/bar" and "/baz:/container/bar/baz" just fine. But you cannot bind "/foo:/container/bar" and "/baz:/container/bar" as the second bind mount will conflict with the first.

 

I tried adding both paths as you suggested and nothing really happened. It didn't attempt to find and transcode h264/avi files.

Share this post


Link to post
I tried adding both paths as you suggested and nothing really happened. It didn't attempt to find and transcode h264/avi files.
Are you able to post some details of the container config? Possibly even a screenshot?

Sent from my ONE E1003 using Tapatalk

Share this post


Link to post
8 hours ago, Josh.5 said:

Are you able to post some details of the container config? Possibly even a screenshot?

Sent from my ONE E1003 using Tapatalk
 

OK so i found that the container would crash if i mapped container path /library/Movies to /mnt/user/Movies and then another mapped path of /library/Series to /mnt/user/Series

 

The configuration below though works just fine:

 

2018-12-05.png

Share this post


Link to post

Also, i guess the only other thought I have on this would be to have the ability to schedule this. Say overnight or any fixed schedule.

Share this post


Link to post

These are really good suggestions.

I've decided to create a web UI (probably just a simple thing nothing flash). This should solve all of your problems @trekkiedj and give you guys better control over what can be done and what cannot.

 

I have opened issues for the features you have requested: https://github.com/Josh5/docker-libraryoptimise/issues

If you think of any others feel free to open them there also.

Share this post


Link to post

@Josh.5 this for me is a really exciting development. As a huge media fan, this solves a major hurdle in getting my library of some 150 odd series and nearly 2500 movies optimised to the latest standard. Thanks so much for your work on this!

Share this post


Link to post

@Josh.5 Just an update on the crashing. 
 

It seems it is something to do with encoding. It only happens if the Movies path is used. I removed some movies it was having issues with but then became apparent that it was every movie!:

 

Untitled.jpg

Share this post


Link to post
[mention=76627]Josh.5[/mention] Just an update on the crashing. 
 
It seems it is something to do with encoding. It only happens if the Movies path is used. I removed some movies it was having issues with but then became apparent that it was every movie!:
 
Untitled.jpg.ecda7a7018dde5d4a8569297a87b1397.jpg
Oh right. Cheers for the log. That is very helpful

Sent from my ONE E1003 using Tapatalk

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now