[Support] Josh5 - Unmanic - Library Optimiser


Recommended Posts

5 hours ago, JasonK said:

Hey everyone.  Doing some testing and I'm getting the following errors on a test video.  Happens regardless if I tell it to process audio or just copy it to the converted file:

 


[matroska,webm @ 0x557de5bedfc0] Could not find codec parameters for stream 2 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[matroska,webm @ 0x557de5bedfc0] Could not find codec parameters for stream 3 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[matroska,webm @ 0x557de5bedfc0] Could not find codec parameters for stream 4 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size
Consider increasing the value for the 'analyzeduration' and 'probesize' options
[matroska,webm @ 0x557de5bedfc0] Could not find codec parameters for stream 5 (Subtitle: hdmv_pgs_subtitle (pgssub)): unspecified size
Consider increasing the value for the 'analyzeduration' and 'probesize' options
Input #0, matroska,webm, from '/library/movies/The Lord of the Rings - The Two Towers (2002).mkv':
  Metadata:
    encoder         : libebml v1.3.0 + libmatroska v1.4.0
    creation_time   : 2013-05-14T23:32:04.000000Z
  Duration: 03:55:18.65, start: 0.000000, bitrate: 18284 kb/s

Any ideas?  Searching online came up with this solution, but since we can't type in additional commands for unmanic to pass along to ffmpeg...

 



Adjust -probesize and/or -analyzeduration.
The files I was testing were in fact damaged. But if anyone faces this problem, setting -probesize and -analyzeduration to 100 and 10000000 for example might help.

From a popular comment by @DiverseAndRemote.com

my solution was

ffmpeg -analyzeduration 2147483647 -probesize 2147483647 -i /path/to/video.mp4 
2147483647 is max_int

 

Have you tried the Remove Subtitles option under Container?

  • Like 1
Link to comment
On 6/7/2020 at 1:34 PM, UncleBacon said:

That's what fixed the same problem for me so you'll have to find another tool to convert them for the time being, as far as I know. I've been playing with Handbrake but it's nowhere near as easy.

i have the same issue as JasonK and removing subtitles didn't work for me either. i have on file that just will not convert. i've removed attachments using mkvtoolnix like one user suggested. and removing subtitles option, but still won't go. and its a massive 4k movie! be so nice to shrink it and make it h265 as thats native and i had transcoding it.

Link to comment

Josh5,

 

25 pages late into your Unmanic docker but just wanted to say a big Thumbs Up from me.

 

This is looking like a great docker.

 

The feature I like most is the ability to pass through the sound as is. Handbrake will not currently pass through an Atmos sound track instead re-encoding it which is really annoying!!

 

One feature request if I may - The ability to move an encoded file to an 'output' folder.

 

Keep up the good work and Thanks

Edited by dgs2001
Link to comment

I was curious whether Intel Quick Sync is available in Unmanic, as I have it installed in unRAID and enabled in Plex with fantastic results.  Searching this thread I didn't find what I was looking for, but when installing Unmanic, I noticed that the info for enabling Intel Quick Sync in Unmanic is noted.  Here are the steps;

 

684167235_unmanic-dev_dri.jpeg.bc71bd1b8961c53bce0e07ab8b979a67.jpeg

 

Note:  for a step-by-step guide for enabling Quick Sync and editing the Go file, look at the first post here in the Plex Hardware Acceleration Guide, by @lotetreemedia.

 

Though QuickSync (/dev/dri) is working for Plex and Handbrake, so far I'm unable to get Unmanic to use Quick Sync.  Anyone else get it working?

Edited by DoItMyselfToo
  • Like 1
Link to comment
22 hours ago, DoItMyselfToo said:

I was curious whether Intel Quick Sync is available in Unmanic, as I have it installed in unRAID and enabled in Plex with fantastic results.  Searching this thread I didn't find what I was looking for, but when installing Unmanic, I noticed that the info for enabling Intel Quick Sync in Unmanic is noted.  Here are the steps;

 

684167235_unmanic-dev_dri.jpeg.bc71bd1b8961c53bce0e07ab8b979a67.jpeg

 

Note:  for a step-by-step guide for enabling Quick Sync and editing the Go file, look at the first post here in the Plex Hardware Acceleration Guide, by @lotetreemedia.

 

So far I'm unable to get Unmanic to use Quick Sync.  Anyone else get it working?

Can confirm. However, the provided instructions didn’t work for me exactly. I never could get the modprobe change to the go file to persist after a reboot. But once I mod the file each time, it kicks off without a hitch. 

Link to comment
43 minutes ago, DaClownie said:

Hey @Josh.5, If I wanted to empty my logs of files converted, is there a .db file i should delete and let it recreate? I'm up to 5000 entries, and it cripples the I/O of my whole server trying to open history now :(

I'm not Josh.5 but easy enough. Go in to your appdata folder for unmanic and in the config folder delete the .db file. You can delete all the files in the log folder too. Restart unmanic and Bob's your uncle! 

  • Like 2
  • Thanks 1
Link to comment

I got a gpu the other day (P2000) it's working great. However, my cpu cores are still full throttle at 100% despite the p2000 chugging on the transcodes. I have set the settings to hevc_nvenc and no audio encoding. Any idea what is causing this, or is that normal behavior? I have pinned two cores for unmanic FYI

Screen Shot 2020-06-17 at 9.19.16 AM.png

Screen Shot 2020-06-17 at 9.19.09 AM.png

Link to comment
36 minutes ago, harriedr said:

I got a gpu the other day (P2000) it's working great. However, my cpu cores are still full throttle at 100% despite the p2000 chugging on the transcodes. I have set the settings to hevc_nvenc and no audio encoding. Any idea what is causing this, or is that normal behavior? I have pinned two cores for unmanic FYI

Screen Shot 2020-06-17 at 9.19.16 AM.png

Screen Shot 2020-06-17 at 9.19.09 AM.png

I would probably NOT try to do 7 encodes at the same time with a quad core CPU. The CPU will still need to be used for some things.  I would start at like 2 or 3 encodes at the same time and adjust from there.

Link to comment
16 minutes ago, DaClownie said:

It still needs to utilize some CPU. It also has to work if its removing subtitles from files. if you use watch nvidia-smi command in the unRAID terminal, are you seeing a ffmpg line showing that unmanic is utilizing your card?

Yes as well as plex

 

 

Screen Shot 2020-06-17 at 10.05.34 AM.png

Link to comment
10 minutes ago, scottc said:

I would probably NOT try to do 7 encodes at the same time with a quad core CPU. The CPU will still need to be used for some things.  I would start at like 2 or 3 encodes at the same time and adjust from there.

I dropped this to 2 and even 1 and it has same results 

Link to comment

OK, so there are a couple things that come into play.

1. One encode at a time with unmanic is more efficient than multiple. I showed that in prior posts in this thread where one encode was 6 minutes and change, 2 encodes was taking 13 or so minutes, 3 was taking 19, etc. Actually worked out to a few seconds slower per encode this way.

2. Removing subtitles utilizes a lot of CPU, GPU can't do that.

3. Is your temporary encode on a SSD? or is it on your array? if its on your array, you're going to get a lot of CPU usage for I/O as your parity needs to recalculate constantly.

 

If you go to your docker tab, and look at the CPU usage per docker, is it showing the full 50% going to unmanic? If you're not seeing where all your CPU usage is going from the docker tab, you can install glances and get a full break down. My server encounters some heavy I/O wait when using unmanic, so I have it so that it only runs at night when my server isn't being utilized.

Link to comment
2 minutes ago, DaClownie said:

OK, so there are a couple things that come into play.

1. One encode at a time with unmanic is more efficient than multiple. I showed that in prior posts in this thread where one encode was 6 minutes and change, 2 encodes was taking 13 or so minutes, 3 was taking 19, etc. Actually worked out to a few seconds slower per encode this way.

2. Removing subtitles utilizes a lot of CPU, GPU can't do that.

3. Is your temporary encode on a SSD? or is it on your array? if its on your array, you're going to get a lot of CPU usage for I/O as your parity needs to recalculate constantly.

 

If you go to your docker tab, and look at the CPU usage per docker, is it showing the full 50% going to unmanic? If you're not seeing where all your CPU usage is going from the docker tab, you can install glances and get a full break down. My server encounters some heavy I/O wait when using unmanic, so I have it so that it only runs at night when my server isn't being utilized.

Thanks for the breakdown. I did see the previous posts about the number of transcodes on GPU and agree with you it is more efficient with less. I'm not removing subtitles. tmp is going to cache on ssd. Docker is showing 50% to unmanic on docker tab.

 

So final question, is there an easy way to schedule the start/stop of a docker. I think i remember seeing a script somewhere.

 

Once again thanks for your help/patience on this!

Link to comment
On 5/8/2020 at 9:12 PM, jmmille said:

Do you think different GPUs work better or worse at encoding?

Unless you're running the latest and greatest GPU's from nvidia, CPU gives a better encode with h.265/HEVC. Better quality and better compression. Just takes a lot longer which equates to more electricity being used.

 

GPU can perform the encodes a lot faster, but lacks B Frame Support for anything lower than a GeForce GTX 1660. CPU currently supports B Frame which is why we see such a reduction in file sizes. I would be curious to see 1) the final encode sizes on the Turing family GPU chipset (sans the GTX 1650) and 2) the encode quality coming from the Turing family chipset when compared to a CPU encode.

image.thumb.png.2bb064803ff98b4a02798eb4e5a35a7e.png

image.thumb.png.b785582712d9e9975e10f0e3a0c013b4.png

source: https://developer.nvidia.com/video-encode-decode-gpu-support-matrix

On 5/8/2020 at 7:24 PM, belliott_20 said:

just got errors when it attempted to go over the stream limit (2). The first two workers would process a file but the 3rd just kept showing errors for file after file.

image.thumb.png.4ad0fb41bdb0f2a6872475d8775d565d.pngimage.thumb.png.bab282da2f2bd5828a6db9414d1e7d99.png

 

I noticed this too @belliott_20, which is what brought me back to the forum tonight. I am probably mistaking the Max # of concurrent sessions as it probably doesn't apply to our case when it comes to Unmanic. I did notice that with two workers = no errors; with three workers = errors.

Now I am not running in debugging mode, so I don't have any logs to share; sorry I am of no help in this sense.

 

On the flip side, maybe we are lucky to be able to encode two streams at a time when looking at the Total # of NVENC column?

Link to comment
On 6/7/2020 at 6:06 AM, JasonK said:

Nope - as I don't want to remove the subtitles :)

You will not find success with PGS subtitles when trying to encode video files that have these types of subtitles embedded within the container — until this project matures. Your best bet is to look towards Handbrake or the other encoder that's been mentioned elsewhere in this thread. (Apologies I don't remember off the top of my head the other encoder's name.)

On 6/7/2020 at 1:34 PM, UncleBacon said:

That's what fixed the same problem for me so you'll have to find another tool to convert them for the time being, as far as I know. I've been playing with Handbrake but it's nowhere near as easy.

SpaceInvaderOne has a tutorial on how to automate this with Handbrake. It's best to do some test encodes first to make sure you get your preset the way you want it and then let it loose on your library — should you want to undertake some of that risk. A safer way to automate would be with watch folders in which you manually bring the completed encodes into your library, working in batches to ensure everything goes smoothly

 

I was doing batches at first with the Handbrake automation, and then transitioned over to Unmanic. Yes, I have disabled the subtitles and audio to make my encodes with Unmanic easier, but those are the sacrifices I am willing to make do for now until it comes time to convert my library to 4k; it's a problem I'll revisit when it comes time to upgrade my audio equipment.

 

Link to comment
On 5/9/2020 at 5:06 AM, DaClownie said:

As I've been going through my library, I've had some mixed results. Some files shrink 70%. Some shrink 25%. The overall goal of me reencoding my library was to save some space.

 

For me, saving all that time and some space is much more valuable than the additional space saved vs. the heat/power/time requirement of the libx265.

 

I'm still going to stick with the nvenc. I reencoded one library from 336GB to 206GB. If I can do that to most of my library, I'll save a couple terabytes, which is perfect. Especially with how easily the gpu transcodes h265 content.

@DaClownie - Have you noticed that some of the encodes, you will end up with an output that is larger than the original input? I've been noticing this phenomenon lately especially with some .ts (MPEG-2) encoded streams. But it's not consistent. For the most part the encodes with hevc_nvenc does save me some space (along with time and energy useage.)

On 5/6/2020 at 5:26 AM, DaClownie said:

That's unfortunate but we can play with it. I don't know much about the inner workings, so I have no idea what modifiers to settings you have to play with in the background... In testing NVENC in handbrake, The file sizes were very finicky. I still haven't got it dialed in. Converting a single episode with nvenc was increasing size from 1.6gb to 2.9gb. However, if I turned quality all the way down, it tuned it to 150mb. Ideally I'm going to get a single file, and duplicate it. I'll convert it using your normal docker with the lib265 encoder and see the size, then i'll try your nvenc docker. After that, I'll tune the handbrake file to get me the same relative size as your lib265 encoder to see the quality comparison. 

 

Are the settings with nvenc changeable on your end? or are they locked in?

I've noticed this with Handbrake, you really have to crank down the quality when doing h.265 to get the speeds of the encode to rival that of h.264, but then the files end up being bloated with degraded picture quality than when you first start out with.

 

I'm kind of torn with hevc_nvenc and libx265. On one hand I save a butt load of time using the GPU to process the encodes, but I risk that some files are in fact bloated/enlarged which negates the entire point of re-encoding to h.265.

 

On the other hand, I know my encodes will always net me anywhere from 60 – 80% in file size savings at the expense of heat, energy draw, and thermal load on the CPU, not to mention the extended time it takes to perform the encodes with CPU.

 

I guess I will have to keep an eye on the outputs and if it keeps happening, I may just switch back to libx265 and drop the number of workers; that and wait until we are able to have access to advanced controls / command input that will guarantee that hevc_nvenc encodes will always be smaller than the source file.

Link to comment

I've just gone with the Handbrake docker with a preset that passes through all audio channels, etc, present...some movies I have with subtitles i need to have and while unmanic works fine for most things, it seems to have issues unless you want to remove subtitles (see my earlier error report).

 

Handbrake is pretty much the same speed at converting

Link to comment
On 6/24/2020 at 1:34 AM, JasonK said:

I've just gone with the Handbrake docker with a preset that passes through all audio channels, etc, present...some movies I have with subtitles i need to have and while unmanic works fine for most things, it seems to have issues unless you want to remove subtitles (see my earlier error report).

 

Handbrake is pretty much the same speed at converting

Hi Jason,

 

Have you managed to pass through a Dolby Atmos soundtrack in Handbrake? If yes can you let me know which version you are using?

 

Using the djaydev fork of handbrake which allows NVENC and removes quick sync I have been unable to pass through an Atmos sound track.

 

Thanks dgs2001

Link to comment
On 6/25/2020 at 6:35 AM, dgs2001 said:

Hi Jason,

 

Have you managed to pass through a Dolby Atmos soundtrack in Handbrake? If yes can you let me know which version you are using?

 

Using the djaydev fork of handbrake which allows NVENC and removes quick sync I have been unable to pass through an Atmos sound track.

 

Thanks dgs2001

I have Handbrake passing through all audio:

 

image.thumb.png.136ac36bb881744d99374319ec2a2602.png

 

I'm using the jlwsage docker from the community applications

 

https://hub.docker.com/r/jlesage/handbrake/

 

I'mm only using CPU to transcode - no vid card

 

Link to comment

This is a wonderful project and I have been using it to convert all my files for the last few months. A few weeks ago, the converted files will no longer move and replace the original files. I have redone permissions. I cleared out the database file. I emptied the cache/conversion folder. I delete that folder and created a new folder. Tried pointing the Library only to certain folders. Adjusted settings in within the docker. I'm lost. It worked so well for so long and I don't know what happened. What do I need to provide for any assistance? 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.