[Support] Josh5 - Unmanic - Library Optimiser


Recommended Posts

I've pushed an update to the "Remux Video Files" plugin - this is now improved to handle transcoding streams that are not compatible with the destination container.

For example, if you have a wmv and you want it converted to an mp4, this will need to be transcoded. It will default to h264/aac. If the stream type is not compatible, then it will just be removed. This will only affect people coming from unique video containers, if you have a mkv file in h265/ac3 and you run it with this plugin to remux to mp4, it will not transcode the video stream as the plugin knows that mp4 files can handle h265, so that stream will be simply copied. 

For people running into issues with a codec not being compatible with the container they are wanting to move to, can you add this plugin in your plugin flow BEFORE any other FFmpeg tasks - eg, before audio sorting, h265 encoding, etc.

cc: @Aerodb @EdwinZelf

  • Like 1
  • Thanks 1
Link to comment
2 hours ago, Josh.5 said:

I've pushed an update to the "Remux Video Files" plugin - this is now improved to handle transcoding streams that are not compatible with the destination container.

For example, if you have a wmv and you want it converted to an mp4, this will need to be transcoded. It will default to h264/aac. If the stream type is not compatible, then it will just be removed. This will only affect people coming from unique video containers, if you have a mkv file in h265/ac3 and you run it with this plugin to remux to mp4, it will not transcode the video stream as the plugin knows that mp4 files can handle h265, so that stream will be simply copied. 

For people running into issues with a codec not being compatible with the container they are wanting to move to, can you add this plugin in your plugin flow BEFORE any other FFmpeg tasks - eg, before audio sorting, h265 encoding, etc.

cc: @Aerodb @EdwinZelf

thank you for this update. This raises a question, does this remove the need to strip subs from the container/content prior to this plugin?

Link to comment
25 minutes ago, Aerodb said:

thank you for this update. This raises a question, does this remove the need to strip subs from the container/content prior to this plugin?

Yes. But there will still be dragons.

If you want to do that it would be really helpful. There will likely be issues with the container compatibility that is configured. I did my best, but I'm not an expert when it comes to what codecs are supported by what containers.

What I decided to do with this plugin was create a compatibility file for the containers: https://github.com/Josh5/unmanic.plugin.video_remuxer/blob/master/lib/containers.json
If you remux from mkv to mp4, the mp4 container does not list subrip as a compatible subtitle in "codec_names", so it will convert with the default encoder (for mp4 that is mov_text). This will still cause issues with image based subtitles and i'm not sure what else at this point. So I would live feedback if people are getting errors in the ffmpeg command, that will help me build this JSON file and form a better compatibility list.

  • Thanks 1
Link to comment
4 hours ago, Josh.5 said:

Yes, you are doing that wrong.

Did you go to the link and read the README.md file as it explains how to install the repo.

The README links to these instructions: https://docs.unmanic.app/docs/plugins/adding_a_custom_plugin_repo/

Well, I converted a few files, and so far so good! Reduced file size, converting audio to AAC, creating a Stereo clone, checking for container size post conversion.

 

I'll need more files to convert to fully test whether its rejecting for file sizes, audio streams are all working, changing containers etc. 

 

Thanks a ton man

  • Like 1
Link to comment
15 hours ago, Josh.5 said:

The lower-case 'tv' and 'movies' directories are created by the docker template. It has nothing to do with Unmanic itself. Unmanic has no code that is specific to "Movies" or "TVShows" - it is a library optimiser and is completely agnostic of what type of file library you have.

So the only way this is possibly happening is because your template is not correct or another container is creating these folders.

Take note of the docker create/run command that is executed when you apply the template:

uraid-unmanic-template1.thumb.png.45e8b0ee7320f807c36cd238cfbb2142.pnguraid-unmanic-template2.thumb.png.3ff3fa8bfea5b8db7bcd7db1f5f3fbcd.png

 

Make sure your template has the mount removed in the "Container Path:" section - not just the description...

uraid-unmanic-template3.png.8612c2de0579e4ae7acd30b63a33021b.png

 

 

 

Hi.

 

Both folders came back again. This time I blanked the content as you suggest followed by save and then I removed them, rather than just 'remove' from the template page. I've also check the config in /boot/config/plugins/dockerMan/templates-user and they do appear to be gone, so we'll see if they stay gone.

 

Thanks.

Link to comment

Hi,

 

Amazing work on the latest version - I love being able to pick and choose plugins and options, and I think unmanic just went from a "nice to have" app to a "must have" app in my mind.

 

I am still seeing errors processing files quite occasionally - specifically, I see the following:

 

Filtergraph 'format=nv12|vaapi,hwupload' was defined for video output stream 0:0 but codec copy was selected.

Filtering and streamcopy cannot be used together.

 

Is there an issue with my configuration that could cause this, or something I can do to mitigate it?

 

Thanks in advance.

Edited by Valyth
Link to comment

Great app, actually upgraded to the patron level to use more add-ons. here is my problem, it keeps logging me out of the site and then pauses because it doesn't recognize I'm a patron. I have to stop the container and then restart it to login again. anyone else seeing this? it's kinda a pain having to restart it a couple times a day.

Link to comment
Great app, actually upgraded to the patron level to use more add-ons. here is my problem, it keeps logging me out of the site and then pauses because it doesn't recognize I'm a patron. I have to stop the container and then restart it to login again. anyone else seeing this? it's kinda a pain having to restart it a couple times a day.
Do you have any proxies or firewall configurations that would prevent Unmanic from communicating with the Auth server every few hours?

I have not checked the Auth server before, but I have seen in the past where someone was running a firewall or VPN and it was blocking half the data so the Auth server was receiving incomplete data and logging the user out.
Link to comment

I'm having files that are being processed multiple times. It processes the same files every cycle of the library scanner from the looks (set to 15 minutes). Or perhaps when the Library File Monitor is enabled (just disabled it now to test).

 

Each one of those "successes" is the same exact size as the prior time it ran.

Processing files multiple times.png

Link to comment
I'm having files that are being processed multiple times. It processes the same files every cycle of the library scanner from the looks (set to 15 minutes). Or perhaps when the Library File Monitor is enabled (just disabled it now to test).
 
Each one of those "successes" is the same exact size as the prior time it ran.
1219354986_Processingfilesmultipletimes.thumb.png.3d33b6e6e24564814243064605915ea2.png
Why would you configure Unmanic to do that?
Link to comment
15 hours ago, Josh.5 said:
21 hours ago, DaClownie said:
I'm having files that are being processed multiple times. It processes the same files every cycle of the library scanner from the looks (set to 15 minutes). Or perhaps when the Library File Monitor is enabled (just disabled it now to test).
 
Each one of those "successes" is the same exact size as the prior time it ran.
1219354986_Processingfilesmultipletimes.thumb.png.3d33b6e6e24564814243064605915ea2.png

Why would you configure Unmanic to do that?

 

It's not? the prior version would never do it. I had the understanding that once it had processed a file it gave it some sort of internal flag in the file so that unmanic wouldn't want to process it again. It just keeps processing the file over and over and I don't know why lol

Link to comment
On 9/30/2021 at 2:03 PM, Bitz69 said:

Great app, actually upgraded to the patron level to use more add-ons. here is my problem, it keeps logging me out of the site and then pauses because it doesn't recognize I'm a patron. I have to stop the container and then restart it to login again. anyone else seeing this? it's kinda a pain having to restart it a couple times a day.

I had the same thing happen to me when my internet connection went down for a few hours this week. To fix it, once my connection was restored I had to log out and back in to Patreon to force the app to pick up my status again - which is quicker than restarting the Docker. I suspect that it would have fixed itself the next time it checked and my connection was working. 

 

It would be nice if the app was a little forgiving - maybe if it fails to get a response from Patreon, it doesn't immediately deactivate, but instead checks again the next day, and deactivates premium features on the subsequent failure. In my case I am fairly sure this was just bad timing, but for those with unstable connections this could be a quality of life issue. 

Link to comment
I had the same thing happen to me when my internet connection went down for a few hours this week. To fix it, once my connection was restored I had to log out and back in to Patreon to force the app to pick up my status again - which is quicker than restarting the Docker. I suspect that it would have fixed itself the next time it checked and my connection was working. 
 
It would be nice if the app was a little forgiving - maybe if it fails to get a response from Patreon, it doesn't immediately deactivate, but instead checks again the next day, and deactivates premium features on the subsequent failure. In my case I am fairly sure this was just bad timing, but for those with unstable connections this could be a quality of life issue. 
That's interesting...
I've updated staging to be more forgiving for you if you would like to test it out?
Link to comment
 
It's not? the prior version would never do it. I had the understanding that once it had processed a file it gave it some sort of internal flag in the file so that unmanic wouldn't want to process it again. It just keeps processing the file over and over and I don't know why lol
I do not think you are smelling what I'm stepping in...

I was being sarcastic, sorry. You have posted a screenshot of a bunch of completed tasks, possibly in the hopes that someone can help you. However you may not realise that no one here can see your logs or your application configuration and therefore no one can help you. The only reason you may wish to share such a screen shot on its own is if you do not want any help and are just showing the fruits of your own doing.

Maybe this was what you were doing? Hence I would ask, "why would you configure the application to do that?"
Or did you just forget to add the required context such as logs and screenshots of your setup and the add something to the post asking for assistance?
Link to comment

In the library management file test workflow block, are all conditions checked before the file is added, or does it add the file when the first matching condition is met? Is it possible that adding a plugin to that list in the wrong order could cause files to be added that should be excluded by other plugins in the list?

 

I hope I'm being clear enough, I can't seem to construct a sentence that conveys what I'm trying to ask without confusing myself.

Link to comment




In the library management file test workflow block, are all conditions checked before the file is added, or does it add the file when the first matching condition is met? Is it possible that adding a plugin to that list in the wrong order could cause files to be added that should be excluded by other plugins in the list?
 
I hope I'm being clear enough, I can't seem to construct a sentence that conveys what I'm trying to ask without confusing myself.


The file test flow is carried out in order - top to bottom. Each plugin has a function that is called as a file tester to see if the file should be added to the pending tasks.

When a single plugin completes its file test run, it returns one of three values:
1. True - that plugin found the file was needing to be added.
2. False - that plugin found the file should not be added.
3. None - this plugin did not care about this file.

Only in the situation where a plugin returns "none" does it move onto the next file test plugin in the flow list. If a file test runner returns either true or false, it stops all further tests.

There are then 3 overall "states" that can be the results of the file test after the plugins have run:
1. "True" - The file tests were run. At least one of these said the file should be added. No subsequent plugin tests were run after receiving that particular plugin's result.
2. "False" - A single file test stated that the file should not be added. Once one test returns False, no other tests are run and the file is ignored (fails fast).
3. "None" - All file tests completed and none of them stated that the file should or should not be added.

The only time above when a file is added to the task list is when state No. 1 above is met.


So to answer your question, yes. The flow order for tests matters a lot.
Link to comment
47 minutes ago, Josh.5 said:

1. "True" - The file tests were run. At least one of these said the file should be added. No subsequent plugin tests were run after receiving that particular plugin's result.

 So having the file extension test, results in all video files being added to be processed, regardless of whether they meet the conditions of the other tests.

 

If I parsed this correctly, what we really should be doing is putting the video test at the top and removing the file extension filter?

 

I was working under the assumption that all conditions must be met, logic AND, and you are saying the tests are logic OR?

 

Sorry, I'm having a hard time wrapping my head around the logic here, I assumed we wanted to first determine whether the file was a video file, then whether it needed to be video converted, if so add it to the queue.

 

What I'm getting is that the file extension filter shouldn't even be in the list of tests, because we only want to check video status to add it to the list.

 

Maybe I'm just thick, but this is doing my head in. I wanted to avoid running the video test on all the other metadata, image, and subtitle files, and only run the video check on likely candidates.

Link to comment




 So having the file extension test, results in all video files being added to be processed, regardless of whether they meet the conditions of the other tests.
 
If I parsed this correctly, what we really should be doing is putting the video test at the top and removing the file extension filter?
 
I was working under the assumption that all conditions must be met, logic AND, and you are saying the tests are logic OR?
 
Sorry, I'm having a hard time wrapping my head around the logic here, I assumed we wanted to first determine whether the file was a video file, then whether it needed to be video converted, if so add it to the queue.
 
What I'm getting is that the file extension filter shouldn't even be in the list of tests, because we only want to check video status to add it to the list.
 
Maybe I'm just thick, but this is doing my head in. I wanted to avoid running the video test on all the other metadata, image, and subtitle files, and only run the video check on likely candidates.


Not a problem at all. The plugin flow is the most important part of the configuration. It is what adds so many custom options and I need people to understand how it works...

In that video you have pinned, about 35:35 the way through it I start working on the plugin flow. (Just as a follow alone reference)

There are currently no tests which will return either a false or a true state. They all at this point in time are written to produce a false/none or true/none.

You have been correct about putting the file extension test first. That plugin returns a false or none result. It is not testing if a file IS in the list of extensions. It is testing if it IS NOT in that list. If the file IS NOT in that list, it returns false. So if it is in that list of extensions, that plugin runner is returning none, which I just explained in the last post will result in the next test being carried out.

Any of the plugins which say "ignore" or "limit" are as described above.

Does that make sense?
Link to comment
On 9/29/2021 at 10:42 AM, page3 said:

Hi.

 

Both folders came back again. This time I blanked the content as you suggest followed by save and then I removed them, rather than just 'remove' from the template page. I've also check the config in /boot/config/plugins/dockerMan/templates-user and they do appear to be gone, so we'll see if they stay gone.

 

Thanks.

Both 'tv' and 'movies' have come back yet again!

This time instead of deleting them I've mapped both to the same folder that I actually want.

 

I only want 'Library' and have no idea where "Movies' and 'TV' keep coming from, no matter how many times I delete them.

 

1663943721_Screenshot2021-10-08at08_36_58.thumb.jpg.81a07407d882ac65d2baa5a1edccfb41.jpg

Link to comment
3 hours ago, page3 said:

Both 'tv' and 'movies' have come back yet again!

This time instead of deleting them I've mapped both to the same folder that I actually want.

 

I only want 'Library' and have no idea where "Movies' and 'TV' keep coming from, no matter how many times I delete them.

 

1663943721_Screenshot2021-10-08at08_36_58.thumb.jpg.81a07407d882ac65d2baa5a1edccfb41.jpg

I cannot be more clear to you that this is a PICNIC. You need to modify your template to remove the /library/tv and /library/movies directories. This has absolutely nothing to do with Unmanic nor the template being an issue, only your configuration of the Unraid template. Look very carefully at my screenshots of the template below and do that!


template1.thumb.JPG.ebef6b1e1dc082d5c7a61f2df7bee0e1.JPG

 

template2.thumb.JPG.3f1aed19a6cdc2c24303a897a87d631c.JPG

 

template3.thumb.JPG.0af138598ed79ad33bb3466aa182900a.JPG

Link to comment

One of the big pros of Unmanic was the ability to shrink the file size of the base file.  I seem to be missing that feature as I run through my library.   I have set the CRF to 20 and the preset quality to "SLOW" .  I barely see any change in file size. 

 

Do I need to increase CRF to be higher in order to gain a better reduction in size? 

 

Link to comment
On 10/2/2021 at 10:13 AM, Josh.5 said:

That's interesting...
I've updated staging to be more forgiving for you if you would like to test it out?

 

Hey Josh. While this did happen to me, my internet recently has been stable, so I haven't been able to replicate the base issue since updating, sorry!  Thanks for taking the feedback, though.  I'm sure it's a niche issue that will only happen to those with unreliable connections, or bad timing.

 

Unrelated: Is there a way to force Unmanic to rescan/rerun through files it has processed previously? I had a bunch of transcodes fail because the location I was transcoding to became full - this lead to over 100 files failing in succession due to lack of space. Obviously this was a configuration issue on my part, and I've cleaned up space and made changes to my setup so this can't happen again, however when I try and have Umanic run through the files again, it no longer adds them to the queue.

Edited by Valyth
Link to comment
 
Hey Josh. While this did happen to me, my internet recently has been stable, so I haven't been able to replicate the base issue since updating, sorry!  Thanks for taking the feedback, though.  I'm sure it's a niche issue that will only happen to those with unreliable connections, or bad timing.
 
Unrelated: Is there a way to force Unmanic to rescan/rerun through files it has processed previously? I had a bunch of transcodes fail because the location I was transcoding to became full - this lead to over 100 files failing in succession due to lack of space. Obviously this was a configuration issue on my part, and I've cleaned up space and made changes to my setup so this can't happen again, however when I try and have Umanic run through the files again, it no longer adds them to the queue.
No worries.

To make Unmanic rerun a file, remove the failure from your historic tasks list then rerun the scan.
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.