[Support] Josh5 - Unmanic - Library Optimiser


Recommended Posts

I'm having a lot of my tasks fail - only new downloads convert successfully and anything else in my library gets various errors...below is an example of one file. Any ideas, am I doing something wrong?

 

[mov,mp4,m4a,3gp,3g2,mj2 @ 0x555e28618780] stream 0, missing mandatory atoms, broken header
[mov,mp4,m4a,3gp,3g2,mj2 @ 0x555e28618780] Could not find codec parameters for stream 0 (Video: h264 (avc1 / 0x31637661), none, 1280x720): unspecified pixel format
Consider increasing the value for the 'analyzeduration' and 'probesize' options
Input #0, mov,mp4,m4a,3gp,3g2,mj2, from '/library/tv/New Girl/Season 505/New Girl - S05E17 - Road Trip.mp4':
  Metadata:
    major_brand     : isom
    minor_version   : 1
    compatible_brands: isomavc1
    creation_time   : 2016-04-29T20:58:52.000000Z
  Duration: 00:21:43.10, bitrate: 1561 kb/s
    Stream #0:0(und): Video: h264 (avc1 / 0x31637661), none, 1280x720, 23.98 fps, 90k tbr, 90k tbn, 180k tbc (default)
    Metadata:
      creation_time   : 2016-04-29T20:58:52.000000Z
Stream mapping:
  Stream #0:0 -> #0:0 (h264 (native) -> hevc (hevc_nvenc))
Press [q] to stop, [?] for help
Cannot determine format of input stream 0:0 after EOF
Error marking filters as finished
Conversion failed!

 

Link to comment

Just wanted to add I am having the CPU/RAM slamming issue too. 

 

Kids asked if internet was down. My piHoles are on my unRaid box so I came here first because I was able to ping 1.1.1.1 from Net Analyzer on my phone. The maxed out RAM was killing all my dockers usability. My DNS and Plex were not happy. 

 

I tried disabling Audio Transcode. Changing cache from RAM to cache. No fix. 

 

It grabs a movie, pegs the CPU and before it's at 10%, 32GB of RAM is at 95% and dockers lock up. 

 

Killing unManic calms everything down. Will turn on debugging and try a few things and share. 

 

EDIT: Using a P2000 and it was working great until recently. I can see the streams in SMI but my CPU and RAM are hit hard. I know CPU still does audio, but it's never been that intensive. 

Edited by rmeaux
Link to comment
On 2/12/2021 at 10:53 PM, rmeaux said:

Just wanted to add I am having the CPU/RAM slamming issue too. 

 

Kids asked if internet was down. My piHoles are on my unRaid box so I came here first because I was able to ping 1.1.1.1 from Net Analyzer on my phone. The maxed out RAM was killing all my dockers usability. My DNS and Plex were not happy. 

 

I tried disabling Audio Transcode. Changing cache from RAM to cache. No fix. 

 

It grabs a movie, pegs the CPU and before it's at 10%, 32GB of RAM is at 95% and dockers lock up. 

 

Killing unManic calms everything down. Will turn on debugging and try a few things and share. 

Enable GPU acceleration if you can, even the built-in Intel GPU if you have a Core i-something. Makes a HUGE difference.

Link to comment
15 hours ago, Snickers said:

Hi, is gpu de/encoding working with this docker? I got a 10th Gen GPU and transcoding in Emby and Plex is working but i doesnt work for me in this docker.

CPU DE/Encoding is working

Mine works, how ever I've noticed that it changes to CPU encoding at some point. I'm unsure of why, however, I'd love know why it keeps stopping using the GPU (Nvidia 1660 SUPER OC.)

 

EDIT: It seems to be after the first video it wants to stop using the GPU.

Edited by eagle470
clarity
Link to comment

Hey team.

 

Today i did a little work on Unmanic. These changes are in the "staging" branch so will only be available to those who are running the "staging" tag on docker (Repository: josh5/unmanic:staging).

 

I've pulled in the latest docker-ffmpeg bin base image in case there were any changes. I believe there may have been an update. But it seems like the FFMPEG version is still a little behind.

 

I've added support for hardware accelerated decoding (both NVDEC/CUDA and VAAPI). To enable it, select the "Enable HW Decoding" checkbox in the advanced tab. I would not recommend mixing and matching CUDA and VAAPI decoding and encoding. At the moment if you have both available, it will default to CUDA. If this is an issue for anyone, open an issue on GitHub and I will make a selector on the advanced tab.

 

 

I see quite a few messages here where people are saying that they have issues with spiking CPU use. I find this odd as I have tried for hours to reproduce this and have not been able to. My only suggestion to those people at this time is:

 

  1. Switch to the staging branch.
  2. Clear out the unmanic settings completely.
  3. Ensure that all files in the /config/ volume (/mnt/user/appdata/unmanic/) is owned by 'nobody:users'.
  4. Enable debugging in unmanic.
  5. Avoid using NTFS partitions for media libraries at this time.
  6. Disable Audio Encoding completely (a recent change last month causes unmatch audio to warrant a fresh transcode)

 

Hopefully whatever is causing the issue is left behind with this.

 

A little note on point 5 above. A friend of mine was using unmanic with a pair of NTFS drives on Ubuntu. I believe there was a bug in the shutil python module that was preventing completed transcodes from being moved back to their original directory in the library. The shutil.move function was not returning an error of any kind, just failing, so unmanic was assuming that the move completed successfully. This was causing files to just keep being re-added to the pending task list every time the library scan was run. I have not yet looked into this issue further. However, we believe we did isolate it down to is NTFS partitions. The permissions for those partitions were fine and unmanic worked as expected on his other XFS drives.

If anyone else has experienced this, or has any insight, I would be interested to know...

  • Like 1
Link to comment
7 hours ago, Josh.5 said:

Hey team.

 

So I nuked my docker image and it's appData folder. I switched to "staging" and built unManic from scratch. 

 

P2000 - Using enc and dec - 32GB of RAM but temp/cache is on appData SSD's - 1 worker

 

The first movie it grabs it appears to work fine. My RAM is at 32% and CPU hangs between 1 - 40%.

 

When that movies hits between 22-27% done The CPU pegs out to 100%, RAM goes to 60% and the unManic GUI is inaccessible. If I wait a few minutes, the GUI comes back, CPU settles down, and RAM returns to 32%. But it is starting over on the same or another movies file. 

 

I will attach or PM my appdata files. I had turned on debugging. Take a look when you can.  

rmeaux.zip

Link to comment
6 hours ago, gacpac said:

intel quicksinc does not seem to work with this container. I know because i'm trying to get away of handbrake that forces SSA subtitles causing transcoding in plex

I'm using my Intel CPU to transcode a file at the moment (i3-6100). 

 

Make sure you have `--device=/dev/dri` set in the "Extra Parameters" section.

image.thumb.png.e40725dc04ae82e4844bd14827657392.png

 

Pull from the staging branch for hardware decoding (not yet in latest):

image.thumb.png.b2fea59f6f06b5c184fa97ba035f9aaa.png

 

Set the encoder to one that ends in "vaapi" like I have here for h264:

image.png.3ee75c02d69c373d539881c8778ee9a8.png

(select "Submit")

 

Then under the "Advanced Options" tab select "enable HW Decoding"

image.thumb.png.d6b4b52e848c1836c185aa68b806b33d.png

(select "Submit")

 

Link to comment
45 minutes ago, Josh.5 said:

I'm using my Intel CPU to transcode a file at the moment (i3-6100). 

 

Make sure you have `--device=/dev/dri` set in the "Extra Parameters" section.

image.thumb.png.e40725dc04ae82e4844bd14827657392.png

 

Pull from the staging branch for hardware decoding (not yet in latest):

image.thumb.png.b2fea59f6f06b5c184fa97ba035f9aaa.png

 

Set the encoder to one that ends in "vaapi" like I have here for h264:

image.png.3ee75c02d69c373d539881c8778ee9a8.png

(select "Submit")

 

Then under the "Advanced Options" tab select "enable HW Decoding"

image.thumb.png.d6b4b52e848c1836c185aa68b806b33d.png

(select "Submit")

 

Oh nice!

so this is new. I have an i5-6400 your same setup most likely. I plan on doing h265 instead of h264 you know saving space. 

 

Question, will this ovewrite my current movies/tvshows because i will like to compare before anything

Link to comment
10 minutes ago, gacpac said:

Oh nice!

so this is new. I have an i5-6400 your same setup most likely. I plan on doing h265 instead of h264 you know saving space. 

 

Question, will this ovewrite my current movies/tvshows because i will like to compare before anything

I use 265 also, you just caught me while running some tests with 264. But both work.

 

Yes it does overwrite. So save a copy of your originals if you wish to compare.

 

In my tests between the defaults using intel and nvidia hardware acceleration, I've noticed that the intel hevc_vaapi encoder's defaults tends to compress a lot more resulting in poorer picture quality. I personally would recommend adding a few additional settings to the advanced options if using intel vaapi. Unless all you are going for is smaller files and are not so worried about the result video quality.

Link to comment
1 minute ago, Josh.5 said:

I use 265 also, you just caught me while running some tests with 264. But both work.

 

Yes it does overwrite. So save a copy of your originals if you wish to compare.

 

In my tests between the defaults using intel and nvidia hardware acceleration, I've noticed that the intel hevc_vaapi encoder's defaults tends to compress a lot more resulting in poorer picture quality. I personally would recommend adding a few additional settings to the advanced options if using intel vaapi. Unless all you are going for is smaller files and are not so worried about the result video quality.

 

Cool, I like a middle of both, I'm trying to read the documentation and I'm lost there 

 

 

Link to comment
22 hours ago, Josh.5 said:

I use 265 also, you just caught me while running some tests with 264. But both work.

 

Yes it does overwrite. So save a copy of your originals if you wish to compare.

 

In my tests between the defaults using intel and nvidia hardware acceleration, I've noticed that the intel hevc_vaapi encoder's defaults tends to compress a lot more resulting in poorer picture quality. I personally would recommend adding a few additional settings to the advanced options if using intel vaapi. Unless all you are going for is smaller files and are not so worried about the result video quality.

a Little help please 

 

I tested the quality with default settings and i can tell the difference. The part i don't know much is the custom settings via extra parameters :S 

Link to comment

Hello there, and thanks for this app!

 

Do I need to use unmanic:staging to enable HW transcoding?

I've added --runtime=nvidia, as well as my gpu uuid via extra parameters, and I dont see anything under watch nvidia-smi

 

Nvidia Quadro m2000 4gb, if it matters. Seems to work fine with emby, and handbrake. Not sure if I fat fingered something ☺️

Link to comment
5 hours ago, tbonedude420 said:

Hello there, and thanks for this app!

 

Do I need to use unmanic:staging to enable HW transcoding?

I've added --runtime=nvidia, as well as my gpu uuid via extra parameters, and I dont see anything under watch nvidia-smi

 

Nvidia Quadro m2000 4gb, if it matters. Seems to work fine with emby, and handbrake. Not sure if I fat fingered something ☺️

You also need to change the unmanic settings to us the nvenc encoder.

 

You don't need to be on staging for he transcoding to work

Link to comment
1 hour ago, Josh.5 said:

Is anyone currently on staging that could provide feedback? I would love to find someone who can help test the staging branch for a week before I merge it into master.

I’ll give it another shot for you tonight. 
 

Ive been running 0.1.0 repository and it’s been blowing through my files. Chopped off about 2TB since you gave me that repository a couple days ago. 
 

This time I’ll wipe out my appdata again and try from scratch. 
 

Do you want my current logs before I switch to compare to what I sent you last time? I believe I turned on debugging. 

Link to comment
1 hour ago, rmeaux said:

I’ll give it another shot for you tonight. 
 

Ive been running 0.1.0 repository and it’s been blowing through my files. Chopped off about 2TB since you gave me that repository a couple days ago. 
 

This time I’ll wipe out my appdata again and try from scratch. 
 

Do you want my current logs before I switch to compare to what I sent you last time? I believe I turned on debugging. 

A fast-forward to latest or staging branch from 0.0.1 should work fine without needing to reset.

Could you please switch to latest without wiping appdata (just back it up) and then let me know if you start having those issues again?

If you are still having issues, I can create a branch just for you to test that we can remove commits until we discover what one is causing problems.

Link to comment
13 hours ago, Josh.5 said:

You also need to change the unmanic settings to us the nvenc encoder.

 

You don't need to be on staging for he transcoding to work

Is there anything I can provide to help? Logs? specs?

Quick spec:, PNY (I think) Quadro, m2000 4gb, 4x Xeon 8880v2, 64gb ecc

 

https://prnt.sc/1082llf Emby show hardware encoding, and working.

https://prnt.sc/1082m94 Unraid showing GPU info

https://prnt.sc/1082mjv Unmanic no longer pulling successfully

docker: Error response from daemon: OCI runtime create failed: container_linux.go:349: starting container process caused "process_linux.go:449: container init caused "process_linux.go:432: running prestart hook 1 caused \"error running hook: exit status 1, stdout: , stderr: nvidia-container-cli: device error: false: unknown device\\n\""": unknown.

 

Steps i've taken... Tried without :staging first, ran into problems, came here.

Since then i've removed the docker, rm -R'd the folder from CLI, and tried the staging version.

 

--runtime=nvidia was added, as well as the others, including a line from my Emby docker (NVIDIA_DRIVER_CAPABILITIES) with the value of ' all '

 

(Tried with, and without that last capabilities line)

 

Thanks for the response! Sorry, I didnt get back sooner... Covid time schedules are wonky. 

 

--==EDIT

So, wiped clean, grabbed staging, put in nvidia guuid, started program, set to my needs, shut down, and restarted the container, all is working. I still don't see progress in nvidia-smi, but its going anywhere from 50x-100x, as to before it was going 10-20x. 

 

I have also enabled debugging incase this is important later. Please feel free to @ me for my logs before pushing the staging update, not sure if it helps but totally willing

https://prnt.sc/1087alz

 

I dont see usage in nvidia-smi, however, the card is 'ramping up' as it were, using pcie3. Normally when idle it shows 1(3). 

 

Question, possibly for its own thread. Handbrake is based on ffmpeg also, so in theory, I could copy over my existing tweaked settings into that advanced/extra arguments field, no?

Edited by tbonedude420
updates
Link to comment

I’d also like to chime in and say something similar, that either on stable or staging branches, I cannot get gpu transcoding to working with intel quick sync either. I’ve added /dev/dri to the extra parameters and enabled in the container settings. I know my gpu transcoding works because I use it for my Plex container (even with Plex off, cannot get Unmanic to use gpu). 

Link to comment
18 hours ago, Josh.5 said:

A fast-forward to latest or staging branch from 0.0.1 should work fine without needing to reset.

Could you please switch to latest without wiping appdata (just back it up) and then let me know if you start having those issues again?

If you are still having issues, I can create a branch just for you to test that we can remove commits until we discover what one is causing problems.

So I tokk a copy of my appData, then changed ":0.0.1" to ":staging" and started unManic. I left HW Decoding unchecked. I also notice the scan takes a bit longer that 0.0.1 to populate a list. It does grab a file right off the bat though. 

 

This time the file got to about 15%, CPU pegged out on dashboard and I lost GUI. 

 

Let it stay active for a bit then stopped unManic. Took another copy of appData. 

 

Went back to 0.0.1 and its happy again. I will PM you nextCloud links to the appData files. Debugging made them too big to add here. 

 

Also, you had asked in a PM for the file that was failing. I restarted unManic multiple times in the past with :staging, which grabbed various files and they all did the same thing. It wasn't just one. Some of those files just went through with :0.0.1. 

 

Thanks for all your work @Josh.5, I know we'll (you) get it. This is my favorite docker. It is so satisfying to see TB's fall off. 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.