[Support] Josh5 - Unmanic - Library Optimiser


Recommended Posts

Is it possible to reencode audio only? My library isn't picked up when i deselect "video encoding" and only enable "audio enconding". It does however work when i select both. My goal is to remove TrueHD audio since that enables transcoding for me.

Link to comment

@Josh.5 love your work! i've had the previous build dunning almost 24/7 with no issues on CPU.

As soon as i saw you updated to add hardware encoding i installed straight away. 

 

There is a huge drop in times using 1050TI my only question is its still fairly CPU intensive.

 

I've noticed the more cores I pin the faster it encodes - its not a huge deal just with plex i barley see a CPU spike an will have 3 tanscodes running on the GPU.

Link to comment
2 hours ago, Ezzy91 said:

@Josh.5 love your work! i've had the previous build dunning almost 24/7 with no issues on CPU.

As soon as i saw you updated to add hardware encoding i installed straight away. 

 

There is a huge drop in times using 1050TI my only question is its still fairly CPU intensive.

 

I've noticed the more cores I pin the faster it encodes - its not a huge deal just with plex i barley see a CPU spike an will have 3 tanscodes running on the GPU.

Are you encoding multiple files at once? Lowering mine to a single worker dropped my CPU usage significantly. Also, I found there was absolutely no speed boost in encoding speed. 3 workers took 3x as long, so same exact speed in the end. Just higher wattage from the wall according to my UPS.

Link to comment
4 hours ago, DaClownie said:

Are you encoding multiple files at once? Lowering mine to a single worker dropped my CPU usage significantly. Also, I found there was absolutely no speed boost in encoding speed. 3 workers took 3x as long, so same exact speed in the end. Just higher wattage from the wall according to my UPS.

Stupid question but did you test it with only 2? Dont the commerical cards have a 2 stream limit unless you use a workaround? Wonder if going over the limit would slow down the entire process on the 2 left on the gpu

Link to comment
17 hours ago, chiefo said:

Stupid question but did you test it with only 2? Dont the commerical cards have a 2 stream limit unless you use a workaround? Wonder if going over the limit would slow down the entire process on the 2 left on the gpu

I just got errors when it attempted to go over the stream limit (2). The first two workers would process a file but the 3rd just kept showing errors for file after file. Based on others input I have since moved it down to 1 worker as mutliple wasn't given me any faster times. 

Edited by jonathanm
Removed discussion of workaround
  • Like 1
Link to comment
On 5/6/2020 at 6:37 PM, DaClownie said:

OK, so here are the full results of the NVidia powered encoding vs. libx265 encoding:

 

Original File: 1.5GB. .mkv container. h264. 1080p, 42 minutes long.

 

Size after conversion with nvenc_hvec and hvec_nvenc:

639.4MB. Total time to convert the media: 6 minutes and 17 seconds.

 

Size after conversion with libx265:

598.9MB. Total time to convert the media: 2 hours, 13 minutes, 58 seconds.

 

I'm not getting this kind of result with a GTX 1060.

Original 720p h264 file is just over 1GB.

hevc_nvenc on the 1060: 3 minutes and 35 seconds to encode 670MB.

libx265: 22 minutes 18 seconds to encode 280MB.

 

I'm wondering why there's such a variation in results.  Do you think different GPUs work better or worse at encoding?

Link to comment
18 hours ago, DaClownie said:

Are you encoding multiple files at once? Lowering mine to a single worker dropped my CPU usage significantly. Also, I found there was absolutely no speed boost in encoding speed. 3 workers took 3x as long, so same exact speed in the end. Just higher wattage from the wall according to my UPS.

Just the one worker, i had it pinned to 2 cores and the file took about 30 mins, pinned 6 to and it took 10 mins (file was about 3.2GB) end result was ~600mb with GPU and about 400mb with CPU (about 2.5 Hrs) 

 

in saying that i noticed my power draw is down and the overall usage on cores is down to about 60% on 6 cores (cpu only gets 80-100%) 

Link to comment
7 hours ago, jmmille said:

I'm not getting this kind of result with a GTX 1060.

Original 720p h264 file is just over 1GB.

hevc_nvenc on the 1060: 3 minutes and 35 seconds to encode 670MB.

libx265: 22 minutes 18 seconds to encode 280MB.

 

I'm wondering why there's such a variation in results.  Do you think different GPUs work better or worse at encoding?

As I've been going through my library, I've had some mixed results. Some files shrink 70%. Some shrink 25%. The overall goal of me reencoding my library was to save some space.

 

For me, saving all that time and some space is much more valuable than the additional space saved vs. the heat/power/time requirement of the libx265.

 

I'm still going to stick with the nvenc. I reencoded one library from 336GB to 206GB. If I can do that to most of my library, I'll save a couple terabytes, which is perfect. Especially with how easily the gpu transcodes h265 content.

  • Like 1
Link to comment

Multiple Watch Libraries:

 

Previously I had been able to configure multiple watch libraries/folders (ex Movies/TV Shows) based on your instructions within the configuration of Unmanic.

 

On the most recent build/version I still have it set the same with multiple watch libraries/folders as before but it will only watch the library entered in the WebGui which only has a spot for 1 and not multiple. Therefore I can no longer have it watch both my movies & tv shows. 

 

Am I maybe missing something in my setup configuration? 

 

For the time being I am running 2 docker containers with one pointed to my TV Shows and the other pointed to my Movies. Which is working out just fine. 

 

**Also I can't thank you enough for this program! I have converted I'm guessing somewhere between 5-10TBs worth of my media files so far. Now with the GPU features added its made it significantly faster and easier on my server. Keep up the great work and know its appreciated!**

Link to comment
12 hours ago, DaClownie said:

For me, saving all that time and some space is much more valuable than the additional space saved vs. the heat/power/time requirement of the libx265.

I agree with this as well.  Any space saved is welcomed.  I'm more wondering how/why your encodes with CPU and GPU were similar in size in your trials.  I'm seeing GPU about 50% as effective as the CPU in space savings.  That's what perplexed me.  What size are you seeing for your average hour long TV show (about 45 minutes) after encoding with GPU? 

Link to comment
1 hour ago, jmmille said:

I agree with this as well.  Any space saved is welcomed.  I'm more wondering how/why your encodes with CPU and GPU were similar in size in your trials.  I'm seeing GPU about 50% as effective as the CPU in space savings.  That's what perplexed me.  What size are you seeing for your average hour long TV show (about 45 minutes) after encoding with GPU? 

I think it has to do with the specific files. I'm still getting some files that are reencoding to like 70% of the original size, which was the results I saw a lot with the libx265. I'm doing a folder now and the first 30 files so far have averaged about 55% size reduction.

 

My server was consuming a ton of power, creating a ton of heat, and taking a ton of time to compress before. My goal was simply to stave off the need to start dropping hard drives into my array again. I'll save 2-3TB, and i'll set the watch for all future additions, and I'll be able to save some money in the long run.

Link to comment
7 hours ago, belliott_20 said:

Multiple Watch Libraries:

 

Previously I had been able to configure multiple watch libraries/folders (ex Movies/TV Shows) based on your instructions within the configuration of Unmanic.

 

On the most recent build/version I still have it set the same with multiple watch libraries/folders as before but it will only watch the library entered in the WebGui which only has a spot for 1 and not multiple. Therefore I can no longer have it watch both my movies & tv shows. 

 

Am I maybe missing something in my setup configuration? 

 

For the time being I am running 2 docker containers with one pointed to my TV Shows and the other pointed to my Movies. Which is working out just fine. 

 

**Also I can't thank you enough for this program! I have converted I'm guessing somewhere between 5-10TBs worth of my media files so far. Now with the GPU features added its made it significantly faster and easier on my server. Keep up the great work and know its appreciated!**

image.thumb.png.d72ace1a7f196b352946e00beab271e3.png

Set yours up like this (you don't need user0 I'm just playing around)

and within unmanic

image.png.fab041ccde2e7b698f156e45ddbecbd8.png

Edited by BomB191
Link to comment
Any word on this working with quicksync/QSV now it works with nvenc? The template seems to suggest it already does...
If you are willing to test it, I can buil a dev- tag docker image for you to download.

I had issues on my laptop getting it playing nicely.
Link to comment

Josh - many thanks for your work here. I have run this through my TV and Films and after a couple of months, gotten to where I want to be - Thanks! I have seen a really nice reduction in size.

 

Two quick questions:

  1. I have both a "TV" media folder and a "Films" media folder. Inside my TV folder, there are several shows that are broadcast daily and I use Plex to only keep the last three shows. Is there a way to tag folders so that Unmanic never looks in those folders? I am not bothered about converting shows that have a limited shelf life.
  2. For some reason, UnManic always "picks up" about 20 films / shows and adds them to the queue but does not convert them. I am assuming there is a reason they are not converted (and am fine with that) but puzzled why they are alaways added each time unManic is started up? I assume because they were not converted in the first place?

Thanks again for this awesome tool!

Link to comment

Any one notice an issue where unmanic isn't transferring completed files to the drives? I notice sometimes while the worker is constantly updating the completed task lists isn't. Not showing successes or failures. Woke up this morning to my cache drive filling up and saw unmanic had over 135gb on the cache. Restarted the docker and it cleared it all out. I'm guessing its not moving it so its not getting marked as completed or failed. This doesn't happen often, but seems to have happened twice now.

Link to comment
Any one notice an issue where unmanic isn't transferring completed files to the drives? I notice sometimes while the worker is constantly updating the completed task lists isn't. Not showing successes or failures. Woke up this morning to my cache drive filling up and saw unmanic had over 135gb on the cache. Restarted the docker and it cleared it all out. I'm guessing its not moving it so its not getting marked as completed or failed. This doesn't happen often, but seems to have happened twice now.
Yeah this has happened to me a few times. Just restart container and it goes back to normal.

Sent from my Pixel 2 XL using Tapatalk

Link to comment
5 hours ago, klipp01 said:

Yeah this has happened to me a few times. Just restart container and it goes back to normal.

Sent from my Pixel 2 XL using Tapatalk
 

Thanks for confirming I'm not crazy. It happened once more after I posted. I also noticed that I could no longer access the completed tasks or the browser tab would lock up. I landed up stopping the docker and deleting the database file. Haven't seen the issue since and my history is now working better. Everything is super snappy again.

Link to comment

Ever since updating i get this error on every file in my library

This is the version - - 0.0.1-beta7+1a6d561

 

2020-05-14T21:32:50:INFO:Unmanic.Worker - [FORMATTED] - Processing item - /library/movies/License to Drive (1988)/License to Drive (1988) h264 1080p.mp4
2020-05-14T21:32:53:INFO:Unmanic.Worker-0 - [FORMATTED] - Worker-0 picked up job - /library/movies/License to Drive (1988)/License to Drive (1988) h264 1080p.mp4
2020-05-14T21:32:53:INFO:Unmanic.Worker-0 - [FORMATTED] - Worker-0 processing job - /library/movies/License to Drive (1988)/License to Drive (1988) h264 1080p.mp4
2020-05-14T21:32:53:DEBUG:Unmanic.FFMPEGHandle - [FORMATTED] - -hide_banner -loglevel info -strict -2 -max_muxing_queue_size 512 -map 0:0 -map 0:3 -map 0:1 -map  0:1 -c:v libx265 -c:v libx265 -c:a:0 aac -c:a:1 aac -b:a:1 128k -ac 2 -metadata:s:a:1 title='Stereo'
2020-05-14T21:32:53:DEBUG:Unmanic.FFMPEGHandle - [FORMATTED] - Executing: ffmpeg -y -i /library/movies/License to Drive (1988)/License to Drive (1988) h264 1080p.mp4 -hide_banner -loglevel info -strict -2 -max_muxing_queue_size 512 -map 0:0 -map 0:3 -map 0:1 -map  0:1 -c:v libx265 -c:v libx265 -c:a:0 aac -c:a:1 aac -b:a:1 128k -ac 2 -metadata:s:a:1 title='Stereo' -y /tmp/unmanic/unmanic_file_conversion-1589506181.8727183/License to Drive (1988) h264 1080p-1589506181.8727434.mkv
2020-05-14T21:32:53:ERROR:Unmanic.Worker-0 - [FORMATTED] - Error while executing the FFMPEG command /library/movies/License to Drive (1988)/License to Drive (1988) h264 1080p.mp4. Download FFMPEG command dump from history for more information. - ['ffmpeg', '-y', '-i', '/library/movies/License to Drive (1988)/License to Drive (1988) h264 1080p.mp4', '-hide_banner', '-loglevel', 'info', '-strict', '-2', '-max_muxing_queue_size', '512', '-map', '0:0', '-map', '0:3', '-map', '0:1', '-map', ' 0:1', '-c:v', 'libx265', '-c:v', 'libx265', '-c:a:0', 'aac', '-c:a:1', 'aac', '-b:a:1', '128k', '-ac', '2', '-metadata:s:a:1', "title='Stereo'", '-y', '/tmp/unmanic/unmanic_file_conversion-1589506181.8727183/License to Drive (1988) h264 1080p-1589506181.8727434.mkv']
2020-05-14T21:32:53:WARNING:Unmanic.Worker-0 - [FORMATTED] - Failed to convert file '/library/movies/License to Drive (1988)/License to Drive (1988) h264 1080p.mp4'
2020-05-14T21:32:53:INFO:Unmanic.Worker-0 - [FORMATTED] - Worker-0 finished job - /library/movies/License to Drive (1988)/License to Drive (1988) h264 1080p.mp4
2020-05-14T21:32:55:INFO:Unmanic.PostProcessor - [FORMATTED] - Post-processing task - /library/movies/License to Drive (1988)/License to Drive (1988) h264 1080p.mp4
2020-05-14T21:32:55:DEBUG:Unmanic.PostProcessor - [FORMATTED] - Task was marked as failed.
2020-05-14T21:32:55:DEBUG:Unmanic.PostProcessor - [FORMATTED] - Removing cached file - /tmp/unmanic/unmanic_file_conversion-1589506181.8727183/License to Drive (1988) h264 1080p-1589506181.8727434.mkv
2020-05-14T21:32:55:DEBUG:Unmanic.PostProcessor - [FORMATTED] - Writing task history log.
2020-05-14T21:32:55:DEBUG:Unmanic.History - [FORMATTED] - Task dump probe data for destination file does not exist possibly due to task failure

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.