[Support] Josh5 - Unmanic - Library Optimiser


Recommended Posts

On 9/16/2021 at 5:12 AM, Josh.5 said:

Tonight I have pushed a new release of Unmanic. Version 0.1.0 (because Unmanic will forever be a ZeroVer).

 

If you have any questions I encourage you to join the discord chat and ask there for help. You will likely get responses quicker on things like what Plugin is best used for a specific situation, etc.

 

This release is about 3-4 months in the making. I'm really glad to finally get it out. Thanks to everyone who jumped onto the staging release over the past 3 weeks and provided me with bug reports and suggestions for improvements.

 

There is still plenty that I want to do, this release now allows me to move onto the next feature set....

where are the discord pages? After the update my docker container is maxing out and keep getting warnings on high image disk utilization. 

Link to comment
3 minutes ago, jonfive said:

Are the default nvidia hevc settings in the plugin the same as the old version's defaults? I'd like to keep the quality consistent if at all possible

Yup. Defaults in the previous version were:
Preset "medium"

Profile "main" 

 

These are the defaults for the encoder from NVIDIA.

  • Like 1
Link to comment
23 minutes ago, jonfive said:

that's awesome.
I seem to have an issue with this particular set of avi's
It processes very fast, looks like it completes and immediately gets grabbed by the worker again and continues to count up

(img)

trying to reproduce, but it seems like it's a one-off for some reason. I re-ran the whole folder once again and it didn't do this once and completed successfully

Link to comment

So THIS is why none of my files were being automatically compressed — there was a major update and everything has to be reconfigured. LoL! 

This shows you how bulletproof this docker was.  I set it up once and never again had to look at it.

 

Thanks for the video @Josh.5 - that helped.  It did however take a few start / stop / restart of my docker container before it actually started to do anything.  Not sure why that happened?

 

Strangely that old container seemed to have skipped some files(?) because now I've got an entire season of a watched series undergoing a conversion.  Perhaps now there is subtitle support as before there was not?

 

EDIT: Just noticed that it no longer reports how much the file was "shrunk" like it did previously.  Has that moved somewhere else?

Edited by CoZ
Link to comment
So THIS is why none of my files were being automatically compressed — there was a major update and everything has to be reconfigured. LoL! 
This shows you how bulletproof this docker was.  I set it up once and never again had to look at it.
 
Thanks for the video @Josh.5 - that helped.  It did however take a few start / stop / restart of my docker container before it actually started to do anything.  Not sure why that happened?
 
Strangely that old container seemed to have skipped some files(?) because now I've got an entire season of a watched series undergoing a conversion.  Perhaps now there is subtitle support as before there was not?
 
EDIT: Just noticed that it no longer reports how much the file was "shrunk" like it did previously.  Has that moved somewhere else?
Awesome.

Take a look at that video for the section about data panels
Link to comment
1 hour ago, Josh.5 said:

Awesome.

Take a look at that video for the section about data panels

 

That's the first place I went too (well after installing all the HEVC plugins) and it's just blank.

 

I'll re-watch the video to see what I missed.  I skimmed through it to get the container up and running again and that was priority 1.

Link to comment
 
That's the first place I went too (well after installing all the HEVC plugins) and it's just blank.
 
I'll re-watch the video to see what I missed.  I skimmed through it to get the container up and running again and that was priority 1.
In the description are some time stamps. Watch the one about data panels
  • Like 1
Link to comment

Everything is a Plugin now.  You're like Oprah now;  "You get a plugin — you get a plugin — you get a plugin!"

 

I am actually liking it the more I mess with it.  At first, I thought "why have all the functionality taken out by default" but now I realize why you've gone that route; much easier to customize, much easier to nail down errors.

 

Great Job on this Sir. 

 

I thought this container was a must-have installation for any Unraid build but now, this certifies it!

Link to comment
Everything is a Plugin now.  You're like Oprah now;  "You get a plugin — you get a plugin — you get a plugin!"
 
I am actually liking it the more I mess with it.  At first, I thought "why have all the functionality taken out by default" but now I realize why you've gone that route; much easier to customize, much easier to nail down errors.
 
Great Job on this Sir. 
 
I thought this container was a must-have installation for any Unraid build but now, this certifies it!
Exactly. Everyone uses the app differently. And the requests for features coming in then affected everyone. This way it's possible to give someone a feature that does not affect anyone else. It also gives Unmanic the ability to carry out functions that are not related to FFmpeg or video files.
  • Like 1
Link to comment

About the new worker pause feature.

 

In the future are you planning on exposing the ability to pause workers to an external trigger, or an API of some sort?

 

Until that may happen, is it still the preferred method to pause the entire container, or will that cause issues?

 

When my home theater VM is actively in use, I scripted a container pause to give the processor a break, when the theater is shut down, it resumes . It would be nice to leave Unmanic running but automatically pause and resume the workers with a scripted command.

Link to comment
13 minutes ago, JonathanM said:

About the new worker pause feature.

 

In the future are you planning on exposing the ability to pause workers to an external trigger, or an API of some sort?

 

Until that may happen, is it still the preferred method to pause the entire container, or will that cause issues?

 

When my home theater VM is actively in use, I scripted a container pause to give the processor a break, when the theater is shut down, it resumes . It would be nice to leave Unmanic running but automatically pause and resume the workers with a scripted command.

IMO, pausing containers is FAR better than pausing processes. There is no overhead at all for pausing a container. You have full flexibility and it does not affect the processes running inside that container at all (they have no knowledge that they have been paused). Pausing a process inside the Unmanic container is a request to that process to pause itself. It knows that it is paused and then has to also manage all the things that go with it being paused. In Unmanic's case this is things like accounting for pause time when calculating ETC. Also, you have the issue that pausing a worker only pauses that one sub-process. You still have a small amount of overhead. Whereas pausing a container pauses absolutely everything within that container.

 

However, with that said...

 

You can pause/resume workers with the Unmanic API. See this video for location of API spec.

Link to comment
18 minutes ago, Josh.5 said:

IMO, pausing containers is FAR better than pausing processes.

It's certainly worked well for me so far, I guess I'll stay the course.

 

Totally unrelated, could you possibly post an example of what to put in the extra parameters of the audio encoder plugin to duplicate the "Create a stereo clone of streams" that the old version had?

 

I'll understand if you say RTFM. Just hopeful it's a quick addition to the "write your own custom audio options" section.

 

Link to comment
2 hours ago, JonathanM said:

It's certainly worked well for me so far, I guess I'll stay the course.

 

Totally unrelated, could you possibly post an example of what to put in the extra parameters of the audio encoder plugin to duplicate the "Create a stereo clone of streams" that the old version had?

 

I'll understand if you say RTFM. Just hopeful it's a quick addition to the "write your own custom audio options" section.

 

Actually, I have it on my TODO list to create a plugin for that:

https://github.com/Unmanic/unmanic-plugins/issues/34

 

Watch that issue on GitHub for updates on progress. Hopefully wont be far away.

When it is ready, it will be able to go back over your already processed files and add the stereo track. The issue will be ignoring files that have been processed by the old version of Unmanic... If you have any ideas on that, I'm well open to suggestions.

Link to comment
58 minutes ago, Josh.5 said:

When it is ready, it will be able to go back over your already processed files and add the stereo track. The issue will be ignoring files that have been processed by the old version of Unmanic... If you have any ideas on that, I'm well open to suggestions.

Well, since the intent is to make sure any processed files have stereo, then why not only process files without a stereo track?

 

Maybe I'm thinking too simplistic here, but with the new setup, it wouldn't be that bad to set up a workflow that periodically is invoked to scan through and only add stereo to the files that need it, and reverting to the typical "new file" workflow chewing through the files that need the full treatment.

 

Or, to complicate matters, have multiple workflows simultaneously defined, each one having a certain number of workers assigned to work on the queue of files that fit that particular workflow.

 

Or, what would happen if you have multiple Unmanic containers pointed at the same library with different objectives? Maybe you could add a temporary lock file into the directory when a worker starts to work on a file, signalling any other worker instances to move along to the next file in the list if the lock file exists for that specific file.

 

Actually that could easily enable multiple machines to team up on a single library without collisions.

 

You did say you were open to suggestions 😀

  • Like 1
Link to comment
41 minutes ago, JonathanM said:

 

 

Or, what would happen if you have multiple Unmanic containers pointed at the same library with different objectives? Maybe you could add a temporary lock file into the directory when a worker starts to work on a file, signalling any other worker instances to move along to the next file in the list if the lock file exists for that specific file.

 

Sounds like a good idea TBH. Instantly I can think of a big issue tho. What if you kill the container. You now have some "ignore" flags lying around in your library that are preventing any Unmanic installation from processing those files

 

Link to comment
First off @Josh.5 this docker is fantastic!  just one question, I have been going through a backlog of transcoding..
In the Data Panes 'Total File Size Changed' the scale for the bars imply one number but hovering over it it's different.
 
Is it just the old 1000 vs 1024 or has something been cooked?
1050924674_Screenshot2021-09-20193022.png.c48b3f1cfef19a35005e67211fbd6e29.png
Looks cooked to me!

Do you mind raising an issue on the unmanic-plugins GitHub repo. I'll get that fixed when I get some time
Link to comment
7 hours ago, Josh.5 said:

Sounds like a good idea TBH. Instantly I can think of a big issue tho. What if you kill the container. You now have some "ignore" flags lying around in your library that are preventing any Unmanic installation from processing those files

 

Journaling. Each worker reports in with start and stop status, if Unmanic starts up and shows a worker with a dangling start status it can clean up the lock file. Yes, that means if the container's appdata is reset you could end up where you where saying, but a fresh install is detectable as well, or you could flag detected lock files in the queue. A locked file could be clicked and selected to be unlocked. Come to think of it, it could be a useful function to add a lock toggle to the queue anyway, maybe you want to leave a file in line but untouched for an arbitrary period of time, and come back to unlock it later. 

 

The only issue I see remaining with that is the lock would only show up in a second Unmanic instance queue when the library was rescanned or a locked file was assigned to a worker and passed over, but that's not a huge issue.

Link to comment

Like others I was LIVID that things had changed and things weren't scanning and processing.  Then I watched the video and got a better understanding of what is going on and why things have changed.   I was very comfortable how things were but times change and new options give more depth of control which I'm beginning to come to grips with.  Curmudgeonly and all that jazz.

 

My jobs are now taking ages (upwards of 18hrs) to process when they used to take a few hours. I've set the process' to "SLOW" in order to have maximum bitrate as well as dropping CRF to "20".   I have 18 threads dedicated to these jobs and only 4 workers running.

 

How do I tell unmanic to skip processing all audio and just convert it as is like the options before.  Plus have unmanic skip subtitles and keep them as is (I think this is already enabled by default unless I turn the options off)

 

Any tips or optimizations, plugins, to bring the old way of unmanic was running would be helpful.

 

Edited by CyrixDX4
Link to comment

Great app which I’m finding really useful. I even managed to follow the video and migrate without issue.

 

However (why is there always a however!) I’m not finding the “support future development “ at all intuitive. It just takes me to something called “patreon” but doesn’t explain what I’m supposed to do there, how I contribute or what financial options are available. Perhaps some explanation or a simple cookie jar for simpletons like me would work?

Link to comment
33 minutes ago, page3 said:

Great app which I’m finding really useful. I even managed to follow the video and migrate without issue.

 

However (why is there always a however!) I’m not finding the “support future development “ at all intuitive. It just takes me to something called “patreon” but doesn’t explain what I’m supposed to do there, how I contribute or what financial options are available. Perhaps some explanation or a simple cookie jar for simpletons like me would work?

Sorry to hear that you are finding that a bit difficult. Unfortunately there was no simple way to make that happen. Patreon is one of the best services available that I could find that allows me to hook into it's website for authenticating people and linking that with donations. All their competition were pretty trash at that and required that I be the one who writes a lot of code to make it happen.

 

Perhaps someone here could walk you through the process?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.