[Support] Josh5 - Unmanic - Library Optimiser


Recommended Posts

I get this error repeatedly when I try to process a large library:

 

2021-10-10T14:16:46:WARNING:Unmanic.Foreman - [FORMATTED] - Postprocessor queue is over 4. Halting feeding workers until it drops.

 

When this happens I have to restart unmanic and then it will pickup and start processing files again. 

 

I have 4 workers all sitting idle, nothing processing, no scans or anything. I have plenty of disk space available for use.

 

 

Link to comment
I get this error repeatedly when I try to process a large library:
 
2021-10-10T14:16:46:WARNING:Unmanic.Foreman - [FORMATTED] - Postprocessor queue is over 4. Halting feeding workers until it drops.
 
When this happens I have to restart unmanic and then it will pickup and start processing files again. 
 
I have 4 workers all sitting idle, nothing processing, no scans or anything. I have plenty of disk space available for use.
 
 
I'll need all the logs. There are items in your Post-Processor queue that are either taking a long time or are throwing exceptions and staying in cache.
Link to comment
13 hours ago, Josh.5 said:
18 hours ago, CyrixDX4 said:
I get this error repeatedly when I try to process a large library:
 
2021-10-10T14:16:46:WARNING:Unmanic.Foreman - [FORMATTED] - Postprocessor queue is over 4. Halting feeding workers until it drops.
 
When this happens I have to restart unmanic and then it will pickup and start processing files again. 
 
I have 4 workers all sitting idle, nothing processing, no scans or anything. I have plenty of disk space available for use.
 
 

I'll need all the logs. There are items in your Post-Processor queue that are either taking a long time or are throwing exceptions and staying in cache.

 

I no longer have the logs as I switched back to 0.0.9.  I missed the simplicity of the earlier version with less fiddly bits and it was incredibly stable. 

 

I appreciate your hard work in bringing enhancements to unmanic.  I'll follow along and see what other upgrades come along and try again later.

 

 

For those folks that want to roll back to 0.0.9 (I know it's no longer supported) you can add the following to the docker template under "repository":
josh5/unmanic:0.0.9

 

 

 

 

  • Like 1
Link to comment

@Josh.5  I just stumbled across this and I'm looking forward to trying it out.  I'm running the latest Unraid with an Nvidia RTX.  Do I need to use the --runtime=nvidia and GPU UUID or not?  Your note under deprecated seems to suggest not, but I see it referenced almost everywhere else so it's been confusing to me.  TIA.

Link to comment
15 minutes ago, BurntOC said:

@Josh.5  I just stumbled across this and I'm looking forward to trying it out.  I'm running the latest Unraid with an Nvidia RTX.  Do I need to use the --runtime=nvidia and GPU UUID or not?  Your note under deprecated seems to suggest not, but I see it referenced almost everywhere else so it's been confusing to me.  TIA.

You are quite right. I should update the instructions removing that warning as I highly doubt anyone is still running the older version of Unraid and so there is no need to keep those instructions.

 

Just follow the instructions in the template adding the --runtime=nvidia option and the other things that it says to.

  • Like 1
Link to comment
On 10/14/2021 at 9:23 AM, ZosoPage1963 said:

Probably a dumb question, but also probably an easy answer.  When running this new version using the NVidia plugin, it seems it is using a lot of CPU also.  Is that normal or do I have a configuration problem? 

Mines always under a high CPU load.

Link to comment

Is there any way to force a file to recode, regardless of the format it is in?  I have some h264 files with super high bitrates, but I'd like them re-encoded at a lower clip.  no matter what I try, it always skips them if I don't want to encode them as another format.

Link to comment
  • 2 weeks later...

I can't figure it out. I whipped everything clean and tried to start over but once I get the container up and running it keeps telling me "Unable to connect to Unmanic backend. Please check that it is running". in the WEBUI I don't know what I am doing wrong ?

Link to comment

Unmanic not detecting any files.

 

First up, I am running unmanic on a separate system to my media library - I have a remote share mapped.

I am not sure if remote shares is unsupported, but in case that's the issue, wanted to get that out the way.

 

I used the how-video to setep - essentially all unraid app defaults, except for the medial library mapped through remote share. To make sure I didn't destroy my libraby, I am using some test-data (10 video files in a separate directory). I mapped directly that as my movie library, but unmanic is not seeing the files. I tried mapping a level up, but still nothing. Not sure what I am doing wrong here - the remote share is obviously working as unmanic can navigate the remote file system.

 

Not sure what data I can provide here to help  - let me know

Link to comment

So this is just a feature request. Add the ability for sonarr and radarr to notify Unmanic of new files to convert. I have already added this as a feature request in sonarr. It would be nice to have as my library has already been scanned and everything converted to have sonarr and radarr notify unmanic of new files instead of it just scanning every few hours for new files. My library is very large and it takes a long time for it to scan everything.

Link to comment
1 hour ago, Jurak said:

So this is just a feature request. Add the ability for sonarr and radarr to notify Unmanic of new files to convert. I have already added this as a feature request in sonarr. It would be nice to have as my library has already been scanned and everything converted to have sonarr and radarr notify unmanic of new files instead of it just scanning every few hours for new files. My library is very large and it takes a long time for it to scan everything.

 

Can you post a link to that request you made with Sonarr. I'll keep an eye on it if anything is required on my end. 

 

Since you brought it up.. An upcoming release should contain in it a massive improvement to library scanning times.

You also have the ability to enable the event watcher, which is pretty good at listening out for file movements within your library. Even with millions of files, that event watcher works well.

 

  • Thanks 1
Link to comment
2 minutes ago, Josh.5 said:

 

Can you post a link to that request you made with Sonarr. I'll keep an eye on it if anything is required on my end. 

 

Since you brought it up.. An upcoming release should contain in it a massive improvement to library scanning times.

You also have the ability to enable the event watcher, which is pretty good at listening out for file movements within your library. Even with millions of files, that event watcher works well.

 

They have unfortunately closed my feature request already stating not enough information and as something they wont support. but here is the link maybe coming from you as the developer you might get more traction with it.
https://github.com/Sonarr/Sonarr/issues/4744#issuecomment-965854296

Link to comment
15 minutes ago, Josh.5 said:

Since you brought it up.. An upcoming release should contain in it a massive improvement to library scanning times.

You also have the ability to enable the event watcher, which is pretty good at listening out for file movements within your library. Even with millions of files, that event watcher works well.

I look forward to the new release when it comes. Thanks for all you do on this. It has really reduced the size of my library by over 30tb in size. 

Link to comment
7 hours ago, Jurak said:

They have unfortunately closed my feature request already stating not enough information and as something they wont support. but here is the link maybe coming from you as the developer you might get more traction with it.
https://github.com/Sonarr/Sonarr/issues/4744#issuecomment-965854296

I've added a note to it. They did not look very far for the API docs... The API schema is OpenAPI v3 compliant and is in the project and can be imported into any API development tool they want - Postman, Insomnia, swagger, etc. The API is hosted in the application web UI.
I do need to better document how to find these things. I think that is what I am going to spend my December doing.
Not sure if you can re-open that ticket now that I've added the direction to the API docs...

  • Thanks 1
Link to comment

I stumbled upon this great tool while searching for a ffmpeg docker that supports 2 Pass AV1, is this something that would be possible using unmanic? So far there is no AV1 plugin and i don't know jack about python so i cant make one myself :(

 

AV1 is fairly mature at this point, even tho  it has no Hardware acceleration as far as i know.

Link to comment
On 11/11/2021 at 1:37 AM, Josh.5 said:

I've added a note to it. They did not look very far for the API docs... The API schema is OpenAPI v3 compliant and is in the project and can be imported into any API development tool they want - Postman, Insomnia, swagger, etc. The API is hosted in the application web UI.
I do need to better document how to find these things. I think that is what I am going to spend my December doing.
Not sure if you can re-open that ticket now that I've added the direction to the API docs...

They have responded in that feature request with some questions if you havent already been notified of this already. lol

  • Like 1
Link to comment
5 hours ago, Aderalia said:

I stumbled upon this great tool while searching for a ffmpeg docker that supports 2 Pass AV1, is this something that would be possible using unmanic? So far there is no AV1 plugin and i don't know jack about python so i cant make one myself :(

 

AV1 is fairly mature at this point, even tho  it has no Hardware acceleration as far as i know.

Well if you can give me a detailed run-down on a collection of FFmpeg commands for transcoding to AV1 with various bitrates (VBR or CBR), etc. Then I can do the python work for you...

 

I looked at the docs for AV1 the other day and after about an hour I gave up.

Link to comment

Hey @Josh am I able to configure Unmanic to re-encode existing h265 files that are larger than a size specified?

 

I have many files encoded already as h265 but their size is still very large 40-60+GB… and I did some proof-of-concept re-encodes using Handbrake (eg. h265, CPU encoding, 10-bit, 2160p, RF22), and the file size was dramatically smaller 15-22gb with excellent quality (for my standards).

 

Is this possible to do or will there be a plug-in to provide this kind of flexibility? Maybe it could just be a setting to allow re-encoding on an existing plugin?

Link to comment
Hey @Josh am I able to configure Unmanic to re-encode existing h265 files that are larger than a size specified?
 
I have many files encoded already as h265 but their size is still very large 40-60+GB… and I did some proof-of-concept re-encodes using Handbrake (eg. h265, CPU encoding, 10-bit, 2160p, RF22), and the file size was dramatically smaller 15-22gb with excellent quality (for my standards).
 
Is this possible to do or will there be a plug-in to provide this kind of flexibility? Maybe it could just be a setting to allow re-encoding on an existing plugin?
Not all h265 encoders are equal. You will also need to specify bitrate etc. I personally do not believe that you will get the results you are looking for.

But if you are keen to do as you say, I suggest you fork the h265 plugin from the official repo and modify it to force the encode when the file is already in h265
Link to comment
10 hours ago, Josh.5 said:

Well if you can give me a detailed run-down on a collection of FFmpeg commands for transcoding to AV1 with various bitrates (VBR or CBR), etc. Then I can do the python work for you...

 

I looked at the docs for AV1 the other day and after about an hour I gave up.

I sent you a PM with some commands i hope it helps. AV1 uses a lot of the same parameters as other codecs, you basically pick what you want to achieve, say a smaller filesize with still good Quality -> Use CRF with a upper limit.

The only thing specific to AV1 are presets, since they depend on the actual encoder. For example: The "official" encoder is libaom-av1. It uses numbers from 0-8, with 0 being lossless and 8 being as fast as possible. Default is 2 (i think, not sure on that).

  • Like 1
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.