[Support] Josh5 - Unmanic - Library Optimiser


Recommended Posts

On 8/2/2021 at 11:14 AM, SuberSeb said:

This is just limit RAM usage by service but this will not fix problem with converting.

If I limit RAM usage then I got this error:
kernel: Memory cgroup out of memory: Killed process 2733 (unmanic-service) total-vm:15112168kB, anon-rss:4151332kB, file-rss:7820kB, shmem-rss:3832kB, UID:99 pgtables:8324kB oom_score_adj:0

 

After this unmanic can't transfer converted file from cache to library directory. Seems that unmanic tries to load full video file in RAM which is awful (tmp folder NOT IN RAM).

 

I too keep getting these issues and it doesn't matter if I limit RAM or not - I'm just trying to figure out what is going on. I have had to completely stop running this docker  because of the OOM and it periodically freezing/locking me out of my system. I've ran multiple test scenarios and this is the only docker I've been having this issues with and it started around my last post (Early July if I recall)...

 

 

Screen Shot 2021-08-04 at 8.54.56 AM.png

Link to comment
1 hour ago, harriedr said:

 

I too keep getting these issues and it doesn't matter if I limit RAM or not - I'm just trying to figure out what is going on. I have had to completely stop running this docker  because of the OOM and it periodically freezing/locking me out of my system. I've ran multiple test scenarios and this is the only docker I've been having this issues with and it started around my last post (Early July if I recall)...

 

 

Screen Shot 2021-08-04 at 8.54.56 AM.png

Are you using custom ffmpeg options or yadif?

Link to comment
2 minutes ago, harriedr said:

Not using any custom settings and based on my quick research I have no idea what yadif is, so I would think I'm not using it...

Yadif is a deinterlacing filter.

I think this is just bug in unmanic. It stucks only at the time when it tries to transfer file from cache to the destination directory. So I think that unmanic transfer entire file to RAM for no reason. Ffmpeg can't waste so much of RAM, so it's an issue in unmanic itself.

Edited by SuberSeb
Link to comment
On 7/25/2021 at 5:33 AM, Meller said:

Why is it trying to use nvenc to encode?  I don't have a GPU in my server.  If I go to Settings > Video Encoding.

I have the Video Codec set to HEVC and the Video Encoder to set to libx265.

I've encoded nearly 25,000 tv show episodes so far, and this is the only one that fails over and over, with a huge log file attached to it. 

  1. Quite the DVR collection you have there! 🏆
  2. I have my server setup to use GPU HEVC encoding (speed at the cost of quality/artifacts). Nothing wrong with libx265; better quality — longer encode times and more power usage as it relies on CPU exclusively.
  3. As far as your episode failing it could be a bad h.264 encode or a small portion of the file is corrupted. This has happened to me before. Throwing those problematic videos into Handbrake allowed me to re-encode those videos into H.265.
    1. Subtitles. Subtitles would cause video encodes to fail 98% of the time with Unmanic. This is being worked on and improved upon in more recent times.
    2. /tmp. Is your transcoding cache using the /tmp directory? I suspect that in my case, either my gaming VM is not releasing the GPU fully and that's what is causing the crashes with Unmanic OR I'm running out of RAM memory when encoding large video files. Either way, a reboot of the server has worked for me — not sure why though.

**Update**

I don't think RAM is a factor in my case at this point in time as 30 minute video files are failing now. I've seen it before though.

Edited by Zer0Nin3r
New news to pass.
Link to comment

I am having some trouble getting started with this container.

 

I have a large TV library and would like to have it all converted as HEVC mp4 files. If I try to convert to mp4 however, I keep getting errors on most of the files and I cannot figure out how to fix it. When I keep all of the settings the same and try to transcode to HEVC .MKV files, it goes without issue.

 

My settings are Mp4 Container / AAC audio + Stereo AAC / HEVC NVENC

Most files output this error, if the file is already an HEVC MP4, it just remuxes it and throws in the stereo AAC track.

 

title : s02e12 - Jolly Old St. Dick
     encoder : Lavf57.56.101
   Duration: 00:22:10.11, start: 0.042989, bitrate: 3256 kb/s
     Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 1280x720, 3052 kb/s, 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc (default)
     Metadata:
       handler_name : VideoHandler
     Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 193 kb/s (default)
     Metadata:
       handler_name : SoundHandler
     Stream #0:2: Video: png, rgb24(pc), 667x500 [SAR 3779:3779 DAR 667:500], 90k tbr, 90k tbn, 90k tbc (attached pic)
Stream mapping:
   Stream #0:0 -> #0:0 (h264 (native) -> hevc (hevc_nvenc))
   Stream #0:2 -> #0:1 (png (native) -> hevc (hevc_nvenc))
   Stream #0:1 -> #0:2 (copy)
Press [q] to stop, [?] for help
[mp4 @ 0x55c86394f0c0] Could not find tag for codec hevc in stream #1, codec not currently supported in container
Could not write header for output file #0 (incorrect codec parameters ?): Invalid argument
Error initializing output stream 0:0 --
Conversion failed!

 

Any help would be greatly appreciated! 

Link to comment
I am having some trouble getting started with this container.
 
I have a large TV library and would like to have it all converted as HEVC mp4 files. If I try to convert to mp4 however, I keep getting errors on most of the files and I cannot figure out how to fix it. When I keep all of the settings the same and try to transcode to HEVC .MKV files, it goes without issue.
 
My settings are Mp4 Container / AAC audio + Stereo AAC / HEVC NVENC
Most files output this error, if the file is already an HEVC MP4, it just remuxes it and throws in the stereo AAC track.
 
title : s02e12 - Jolly Old St. Dick    encoder : Lavf57.56.101  Duration: 00:22:10.11, start: 0.042989, bitrate: 3256 kb/s    Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 1280x720, 3052 kb/s, 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc (default)    Metadata:      handler_name : VideoHandler    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 193 kb/s (default)    Metadata:      handler_name : SoundHandler    Stream #0:2: Video: png, rgb24(pc), 667x500 [sAR 3779:3779 DAR 667:500], 90k tbr, 90k tbn, 90k tbc (attached pic)Stream mapping:  Stream #0:0 -> #0:0 (h264 (native) -> hevc (hevc_nvenc))  Stream #0:2 -> #0:1 (png (native) -> hevc (hevc_nvenc))  Stream #0:1 -> #0:2 (copy)Press [q] to stop, [?] for help[mp4 @ 0x55c86394f0c0] Could not find tag for codec hevc in stream #1, codec not currently supported in containerCould not write header for output file #0 (incorrect codec parameters ?): Invalid argumentError initializing output stream 0:0 --Conversion failed!

 
Any help would be greatly appreciated! 

Have you got access to the plugins? There is one in there that will strip out the PNG stream that is causing the error with Unmanic
Link to comment

I do not, when I click plugins, I get a patreon link :/. I was guessing that might be the error but really wasnt sure. I guess for now I will use MKV.

 

Another question. I have files that have a lot of different audio streams, is it possible to just take 1 audio stream that matches a specified language?

 

Link to comment
2 hours ago, Josh.5 said:
3 hours ago, lukeoslavia said:
I am having some trouble getting started with this container.
 
I have a large TV library and would like to have it all converted as HEVC mp4 files. If I try to convert to mp4 however, I keep getting errors on most of the files and I cannot figure out how to fix it. When I keep all of the settings the same and try to transcode to HEVC .MKV files, it goes without issue.
 
My settings are Mp4 Container / AAC audio + Stereo AAC / HEVC NVENC
Most files output this error, if the file is already an HEVC MP4, it just remuxes it and throws in the stereo AAC track.
 


title : s02e12 - Jolly Old St. Dick    encoder : Lavf57.56.101  Duration: 00:22:10.11, start: 0.042989, bitrate: 3256 kb/s    Stream #0:0(und): Video: h264 (Main) (avc1 / 0x31637661), yuv420p, 1280x720, 3052 kb/s, 29.97 fps, 29.97 tbr, 90k tbn, 59.94 tbc (default)    Metadata:      handler_name : VideoHandler    Stream #0:1(und): Audio: aac (LC) (mp4a / 0x6134706D), 44100 Hz, stereo, fltp, 193 kb/s (default)    Metadata:      handler_name : SoundHandler    Stream #0:2: Video: png, rgb24(pc), 667x500 [sAR 3779:3779 DAR 667:500], 90k tbr, 90k tbn, 90k tbc (attached pic)Stream mapping:  Stream #0:0 -> #0:0 (h264 (native) -> hevc (hevc_nvenc))  Stream #0:2 -> #0:1 (png (native) -> hevc (hevc_nvenc))  Stream #0:1 -> #0:2 (copy)Press [q] to stop, [?] for help[mp4 @ 0x55c86394f0c0] Could not find tag for codec hevc in stream #1, codec not currently supported in containerCould not write header for output file #0 (incorrect codec parameters ?): Invalid argumentError initializing output stream 0:0 --Conversion failed!
 

 
Any help would be greatly appreciated! 

Have you got access to the plugins? There is one in there that will strip out the PNG stream that is causing the error with Unmanic

So I went ahead and supported your patreon to get that plugin, but didnt see any plugins that could help me reduce the number of audio streams. Is there any current way I can do this. I would like to just pick a single English audio track, or even copy all english audio tracks, but remove any other languages. 

Link to comment
So I went ahead and supported your patreon to get that plugin, but didnt see any plugins that could help me reduce the number of audio streams. Is there any current way I can do this. I would like to just pick a single English audio track, or even copy all english audio tracks, but remove any other languages. 
I can make you a plugin to do that this week. No problem.
Did you find the plugin for removing images from the video files?
Link to comment
2 hours ago, Josh.5 said:

I can make you a plugin to do that this week. No problem.
Did you find the plugin for removing images from the video files?

I did find that plugin, and it did work, but I decided just to go ahead and convert everything to MKV since I cant remove the audio files and subs. If that function becomes available I can definitely use it! but I'm also happy enough to just have everything copied over to the new format.

 

The container is super easy to use once you figure out a few things,  I just cant help but wonder how much space I could save by getting rid of extra audio tracks! 

Edited by lukeoslavia
Link to comment

Hi, first thanks for this container, it helped me so much.

 

I have problem with plex, actually just the web version that my wife almost only use.
It can't read the x265 8bits but have no problems with the 10 bits.

The container seems to bypass the files that are already encoded in x265.

So how can I make it convert it anyway?

I tried to search for ffmpeg command to do that, but only found the contrary, 10bits to 8 bits. And I don't understand shit.

EDIT: some 8 bits works... I'll need to investigate that REALLY weird shit...

 

I need help to bypass a nuclear attack... by my wife...

 

Thank you

Edited by Steace
confused
Link to comment

Setup unmanic and it looks great, a nice way to standardise and trim the library. Issue I have is, it skips everything. 

Constant error i see in the logs 

```ERROR:Unmanic.Worker-0 - [FORMATTED] - Exception in processing job with Worker-0: - [Errno 13] Permission denied: '/mnt/user'```

 

Any idea what is it and how I fix it? I see that all my media is read only, that much the way that SABNZBD downloads it, but even manually unchecking read only doesn't seem to fix it. 

Edited by calypsoSA
Spelling
Link to comment

this looks like a great tool. 

 

just a shame you cannot set it to remove multi-language audio, subtitles, etc and set a resolution so if there is a 1080p copy, make it 720p hevc.

 

or am i missing these settings somewhere?

 

edit - i'm all for supporting by patreon but i'd like to see what the plugins are first!

 

potential issue now is that i've started the hevc process. if it does them all, will it ignore them all once i add a plugin that say, takes all audio out except english?

Edited by Jammy B
Link to comment
this looks like a great tool. 
 
just a shame you cannot set it to remove multi-language audio, subtitles, etc and set a resolution so if there is a 1080p copy, make it 720p hevc.
 
or am i missing these settings somewhere?
 
edit - i'm all for supporting by patreon but i'd like to see what the plugins are first!
 
potential issue now is that i've started the hevc process. if it does them all, will it ignore them all once i add a plugin that say, takes all audio out except english?
First of all thanks for taking Unmanic for a spin.

Patreon does not charge until the first of the month. So you hav til then to decide if the additional stuff is worth it for you.
That being said, the next major release will enable a limited number of plugins for everyone to use without signing into anything.

Also, it is worth saying that the way that I have written the plugins, they will reprocess files that have already been converted. Eg, if you convert them to hevc now and in a few months time you want to run the plugin that reorders subtitles, it will do that to your library without reencoding the video streams. So you absolutely can process your library while waiting for a plugin to fix that other need of yours.
  • Thanks 1
Link to comment
7 hours ago, Josh.5 said:

First of all thanks for taking Unmanic for a spin.

Patreon does not charge until the first of the month. So you hav til then to decide if the additional stuff is worth it for you.
That being said, the next major release will enable a limited number of plugins for everyone to use without signing into anything.

Also, it is worth saying that the way that I have written the plugins, they will reprocess files that have already been converted. Eg, if you convert them to hevc now and in a few months time you want to run the plugin that reorders subtitles, it will do that to your library without reencoding the video streams. So you absolutely can process your library while waiting for a plugin to fix that other need of yours.


Thanks for taking the time to give a detailed reply. I will leave it running to do it’s thing and then look at the plugin aspect. 
 

my aim (besides gaining space by converting to HEVC) is to lose audio/subtitles not required and make 95% of the library 720p from the 1080p some of it currently is. 
 

It seems to be cracking on with it. 
 

some have failed and I see the logs, I will do a bit of further reading before asking what the errors mean. 
 

 

Link to comment
11 hours ago, Josh.5 said:

First of all thanks for taking Unmanic for a spin.

Patreon does not charge until the first of the month. So you hav til then to decide if the additional stuff is worth it for you.
That being said, the next major release will enable a limited number of plugins for everyone to use without signing into anything.

Also, it is worth saying that the way that I have written the plugins, they will reprocess files that have already been converted. Eg, if you convert them to hevc now and in a few months time you want to run the plugin that reorders subtitles, it will do that to your library without reencoding the video streams. So you absolutely can process your library while waiting for a plugin to fix that other need of yours.

Its good to hear some of the plugins are going free, but I also second the removal of multi-language audio and subs. And to a lesser degree seeing the plugins first, but as you explained, that's not as big an issue.

 

The re-order plugin seems to do its job just fine, but so far, for me anyway, emby still seems to pick a random language unless I set one on each individual player.

 

That's fine and well for me, but for the rest of my family that doesn't really understand how it all works, it would be great to remove the extra steps for them once and never have to worry about it again.

 

On the positive side though, your folder watch seems to work better than most of the other programs I've used, and the ease of use goes far beyond anything else so kudos there! 

Link to comment
15 minutes ago, lukeoslavia said:

Its good to hear some of the plugins are going free, but I also second the removal of multi-language audio and subs. And to a lesser degree seeing the plugins first, but as you explained, that's not as big an issue.

 

The re-order plugin seems to do its job just fine, but so far, for me anyway, emby still seems to pick a random language unless I set one on each individual player.

 

That's fine and well for me, but for the rest of my family that doesn't really understand how it all works, it would be great to remove the extra steps for them once and never have to worry about it again.

 

On the positive side though, your folder watch seems to work better than most of the other programs I've used, and the ease of use goes far beyond anything else so kudos there! 

I'm currently undertaking a complete rewrite of the frontend which includes a small overhaul of how plugins work and the opening of them up in limited quantities to everyone. This development still has around 20-30 hours remaining. The first phase is already in staging and I will be pushing the past 3 weeks work into staging again sometime this week.

 

None of these changes will make their way into a new release until they are completed. This may be a few weeks away yet.

 

Once that release comes, I will be doing nothing but writing plugins like the one you have just mentioned. The whole point of this re-write is to make Unmanic more modular to better suite everyone's unique use-cases.

 

I would suggest that over the next few weeks, if people are wanting to try out the updated plugin system, they switch over to the staging branch and provide me with some feedback on the development progression.

 

A big note of change...
Once the next release does become available, you will likely notice that all FFmpeg related settings will be completely removed from Unmanic. Unmanic is migrating away from an "FFmpeg" tool and will have new capabilities for managing libraries of any file type. That being said, you will still be able to use FFmpeg plugins on your files very similarly to how you can now with Unmanic's settings.

 

I believe I will be setting up Unmanic to limit only a single plugin enabled at a time for people who do not wish to sponsor the project (which is absolutely fine!). That still gives everyone the ability to use the plugins on their library, and the ability to try out plugins before deciding if they want to sponsor the project at all. It also gives me the ability to monetise the development effort that I put into the project over the next few years - something that will greatly help me to keep the project going.

I would be curious to know if anyone believes the sponsorship requirements that I setup through Patreon are too high? No offence taken here if you speak your mind. At the end of the day, Unmanic is free for anyone to use how they want. I have not restricted the source in any way and have given this to you freely without wanting anything in return. I have no intention of ever making the Unmanic project anything other than open source - so no proprietary v2 re-writes on the horizon. All I am trying to do now is provide my creation as a service to those who are asking for me to continue to develop new features for them (like a plugin to remove various audio streams) and requesting support for that continued development in the form of optional monthly donations. This is absolutely a hobby project of mine and I do not get offended when people use it for free and message me with feedback or ask for assistance. I'm absolutely here to help everyone in this community as much as I can.

  • Like 1
Link to comment

I can't get this to add a stereo track to my videos while keeping the original track(s) intact so I can enjoy the enhanced sound tracks at home and they can also be played by Plex for my sons.

 

I've enabled 'Enable Audio Encoding'; disabled 'Enable Audio Stream Transcoding'; and enabled 'Create a stereo clone of all audio streams' but nothing gets scanned as needing re-encoding.

 

It only works if I enable 'Enable Audio Stream Transcoding' as well which then destroys the original track(s).

Edited by Hitcher
Link to comment
I can't get this to add a stereo track to my videos while keeping the original track(s) intact so I can enjoy the enhanced sound tracks at home and they can also be played by Plex for my sons.
 
I've enabled 'Enable Audio Encoding'; disabled 'Enable Audio Stream Transcoding'; and enabled 'Create a stereo clone of all audio streams' but nothing gets scanned as needing re-encoding.
 
It only works if I enable 'Enable Audio Stream Transcoding' as well which then destroys the original track(s).
Thanks for bringing this up. I'll work on an improvement so that this is possible in a future release.
If you open a GitHub issue for this, I will close it once a solution is available and you will be notified.
Link to comment

Hello,

 

not sure why but when the audio encoding is enabled, the voices in the media seem to only be mono, I can only hear talking on the right side, not both, i've turned this off for now, i've had a bit of a look for a solution but have not found one yet,

 

any ideas?

Link to comment

Hi,

 

First of all thanks for this awesome projet!

Unfortunately, I have a problem with the final copy of the transcoded file (from the cache folder to the final destination). Here's the error in my log:

 

2021-08-24T10:40:29:INFO:Unmanic.Worker-1 - [FORMATTED] - Successfully ran worker process 'unmanic_default_stage' on file '/library/movies/Cruella (2021)/Cruella (2021).mkv'
2021-08-24T10:40:29:INFO:Unmanic.Worker-1 - [FORMATTED] - Successfully converted file '/library/movies/Cruella (2021)/Cruella (2021).mkv'
2021-08-24T10:40:29:INFO:Unmanic.Worker-1 - [FORMATTED] - Moving final cache file from '/tmp/unmanic/unmanic_file_conversion-1629820855.3718686/Cruella (2021)-1629820855.3718746-WORKING-1.mkv' to '/tmp/unmanic/unmanic_file_conversion-1629820855.3718686/Cruella (2021)-1629820855.3718746.mkv'
2021-08-24T10:40:31:INFO:Unmanic.Worker-1 - [FORMATTED] - Worker-1 finished job - /library/movies/Cruella (2021)/Cruella (2021).mkv
2021-08-24T10:40:34:INFO:Unmanic.PostProcessor - [FORMATTED] - Post-processing task - /library/movies/Cruella (2021)/Cruella (2021).mkv
2021-08-24T10:40:34:INFO:Unmanic.PostProcessor - [FORMATTED] - Copying file /tmp/unmanic/unmanic_file_conversion-1629820855.3718686/Cruella (2021)-1629820855.3718746.mkv --> /library/movies/Cruella (2021)/Cruella (2021).mkv
2021-08-24T10:40:34:ERROR:Unmanic.PostProcessor - [FORMATTED] - Exception while copying file /tmp/unmanic/unmanic_file_conversion-1629820855.3718686/Cruella (2021)-1629820855.3718746.mkv to /library/movies/Cruella (2021)/Cruella (2021).mkv:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/unmanic/libs/postprocessor.py", line 195, in post_process_file
    before_checksum = hashlib.md5(open(data.get('file_in'), 'rb').read()).hexdigest()
MemoryError
2021-08-24T10:40:34:ERROR:Unmanic.PostProcessor - [FORMATTED] - Error while running postprocessor file movement on file '/tmp/unmanic/unmanic_file_conversion-1629820855.3718686/Cruella (2021)-1629820855.3718746.mkv'. Not all postprocessor file movement functions completed.
2021-08-24T10:40:34:INFO:Unmanic.PostProcessor - [FORMATTED] - Removing task cache directory file '/tmp/unmanic/unmanic_file_conversion-1629820855.3718686/Cruella (2021)-1629820855.3718746.mkv'
2021-08-24T10:40:35:INFO:Unmanic.PostProcessor - [FORMATTED] - Removing task cache directory '/tmp/unmanic/unmanic_file_conversion-1629820855.3718686'

 

I did check if I have proper file permissions and if the docker routing is correct. I tried to run it a few times but I always end up with this error.

 

Did anyone else have had this issue?

Thanks!

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.