[Support] Josh5 - Unmanic - Library Optimiser


Recommended Posts

@Josh.5 I’m trying to do something a little more advanced with unmanic. I am running unmanic in a docker container on a server which is separate from my media server.
 
However when I mount my media server share via SSHFS onto the unmanic server, unmanic won’t pick up any of the files? 
 
I can transfer the files manually via SFTP and it works but that will take ages. Any ideas why unmanic won’t detect files when using SSHFS?
Not sure about sshfs, but in the past someone had an issue with a cifs mount. They needed a slightly different setup to make it work:

https://github.com/Josh5/unmanic/blob/master/docker/docker-compose-cifs.yml
Link to comment

 

 

https://github.com/Josh5/unmanic/issues/83

 

This is the feature request for the ability to remove unneeded subtitles and audio streams.

 

There would be a fair amount of work involved in making this happen. I would probably guess around 20+ hours and then a crap ton of fiddling with it because videos come in all kinds of different shapes with different names for the subtitle streams etc. This could easily sink down a rabbit hole that just is not worth it for this application.

 

I won't delete the issue on GitHub, but unless someone else wants to take on that project, it probably won't get done.

 

Link to comment

Any ETA on when this might go Non Beta?  Not rushing just want to put it on my calendar to check.  I have a bad habit of checking everyday unless I know something is a ways out.  

 

PS GREAT WORK!  It's appreciated.

Link to comment
Any ETA on when this might go Non Beta?  Not rushing just want to put it on my calendar to check.  I have a bad habit of checking everyday unless I know something is a ways out.  
 
PS GREAT WORK!  It's appreciated.
I've done a bunch of stuff this week. I'm not sure about being out of beta, but the code is starting to become more structured to a permanent standard which means we're getting closer to being out of beta. I think the app is pretty close to a point now where no more major changes will require a clean install or anything.
There is one other thing I need to do yet and that is to move all the task queues to the database. That change may or may not yet require a clean install. Once that is done, I think I'll push unmanic out if beta.

Today I'm testing Nvidia hardware encoding changes (not yet available via docker) so far so good.
  • Like 3
Link to comment
2 hours ago, Josh.5 said:

I've done a bunch of stuff this week. I'm not sure about being out of beta, but the code is starting to become more structured to a permanent standard which means we're getting closer to being out of beta. I think the app is pretty close to a point now where no more major changes will require a clean install or anything.
There is one other thing I need to do yet and that is to move all the task queues to the database. That change may or may not yet require a clean install. Once that is done, I think I'll push unmanic out if beta.

Today I'm testing Nvidia hardware encoding changes (not yet available via docker) so far so good.

All I know is that I've had this running on my server for the past couple of weeks or so with no issues whatsoever. All I can say is keep up the great work. I'm so thankful or people who do things I cannot. 

Link to comment

There is a new docker image available for anyone who is game enough to try it.

 

Edit the Unmanic docker config in Unraid and set the docker image repository to 'josh5/unmanic:dev-hw-encoding'

 

I have not tested this docker image with nvidia or intel HW accelerated encoding yet (and I don't want to as its just too much effort for minimal if any return). Testing it out running unmanic nativity works fine. In theory, if you follow the same steps used to enable plex HW encoding with the linuxserver.io Plex container, then it should work.

 

Thanks.

Link to comment
3 hours ago, Josh.5 said:

There is a new docker image available for anyone who is game enough to try it.

 

Edit the Unmanic docker config in Unraid and set the docker image repository to 'josh5/unmanic:dev-hw-encoding'

 

I have not tested this docker image with nvidia or intel HW accelerated encoding yet (and I don't want to as its just too much effort for minimal if any return). Testing it out running unmanic nativity works fine. In theory, if you follow the same steps used to enable plex HW encoding with the linuxserver.io Plex container, then it should work.

 

Thanks.

I will take a stab at this with NVENC tonight to see how it fares. If it works, I'll be using it to convert tons of libraries for some space conservation. Cartoons especially benefit a ton from compression like this, with almost no visible degredation to quality.

 

Should we see the same speed/cpu advantages that we see in Plex? I know when I transcode in Plex it handles the transcode MUCH faster than it would with normal software transcode.

Edited by DaClownie
Link to comment
I will take a stab at this with NVENC tonight to see how it fares. If it works, I'll be using it to convert tons of libraries for some space conservation. Cartoons especially benefit a ton from compression like this, with almost no visible degredation to quality.
I'd be curious to know if you do get reduced size on the outputs with nvenc. I found that while the conversion is fast, the results on all of my sample files were much bigger. I may be doing something wrong, but I think the CPU encoding is far superior.

For example, I have a 10mb h264 file. If I convert it to h265 with the lib265 encoder, it reduces to a 4.2mb file in 25 seconds. The hvec_nvenc encoder transcoding the same h264 10mb file completed the task in 7seconds, but increased the file size to 17mb.

I played around with all the buffers and stuff, and it was a lot of dicking around to get nowhere. If this was a trans code from a large 4k file to a compressed 720p, then I think it would be worth having the nvenc encoder, but for keeping the same quality, I do not think it will do what you need.
Link to comment
2 hours ago, Josh.5 said:

I'd be curious to know if you do get reduced size on the outputs with nvenc. I found that while the conversion is fast, the results on all of my sample files were much bigger. I may be doing something wrong, but I think the CPU encoding is far superior.

For example, I have a 10mb h264 file. If I convert it to h265 with the lib265 encoder, it reduces to a 4.2mb file in 25 seconds. The hvec_nvenc encoder transcoding the same h264 10mb file completed the task in 7seconds, but increased the file size to 17mb.

I played around with all the buffers and stuff, and it was a lot of dicking around to get nowhere. If this was a trans code from a large 4k file to a compressed 720p, then I think it would be worth having the nvenc encoder, but for keeping the same quality, I do not think it will do what you need.

That's unfortunate but we can play with it. I don't know much about the inner workings, so I have no idea what modifiers to settings you have to play with in the background... In testing NVENC in handbrake, The file sizes were very finicky. I still haven't got it dialed in. Converting a single episode with nvenc was increasing size from 1.6gb to 2.9gb. However, if I turned quality all the way down, it tuned it to 150mb. Ideally I'm going to get a single file, and duplicate it. I'll convert it using your normal docker with the lib265 encoder and see the size, then i'll try your nvenc docker. After that, I'll tune the handbrake file to get me the same relative size as your lib265 encoder to see the quality comparison. 

 

Are the settings with nvenc changeable on your end? or are they locked in?

Link to comment
5 minutes ago, Josh.5 said:


 

 


They are locked in. I plan on adding "advanced settings" later in the future, but for now, this is how it is.

 

Well, I'll working on getting the settings dialed into the same quality/size profile as your current encoder does, and then i'll get you access to that .json. When changing of settings is available, maybe just mirroring those settings will work well for you.

 

Appreciate the work on the utility!

  • Like 1
Link to comment
Well, I'll working on getting the settings dialed into the same quality/size profile as your current encoder does, and then i'll get you access to that .json. When changing of settings is available, maybe just mirroring those settings will work well for you.
 
Appreciate the work on the utility!
Sounds good
  • Thanks 1
Link to comment
I don't think so but is there a difference between hevc_nvenc vs. nvenc_hevc? I have both in my dropdown. Is it just a naming convention?

I moved all my movies that start with a particular letter to a separate location and I will compare before and after results. 
As far as I could tell, nvenc_hevc is an older now deprecated encoder. Use the newer hevc_nvenc one? I don't know what the difference is (if there Is any), but using the nvenc_hvec one will give you a warning at the start of your ffmpeg conversion logs
  • Thanks 1
Link to comment
6 minutes ago, rmeaux said:

I don't think so but is there a difference between hevc_nvenc vs. nvenc_hevc? I have both in my dropdown. Is it just a naming convention?

I moved all my movies that start with a particular letter to a separate location and I will compare before and after results. 

So, I converted the same file with both. Identical output, identical time frame for conversion.

 

1.5GB file, .mkv container, h264. 1080p 42ish minutes.

 

Size after conversion: 639MB, 6 minutes and 17 seconds to convert with GTX 1650 Super. (leaving audio tracks untouched)

 

Converting same file now with the libx265 now to get a size/speed comparison. Then I'll play the files side by side for quality comparison.

 

Will post results later. libx265 will probably take 4 hours since i'm only allowing it to use 2 cores/2 threads as to not bog down my server.

Edited by DaClownie
Link to comment
So, I converted the same file with both. Identical output, identical time frame for conversion.
 
1.5GB file, .mkv container, h264. 1080p 42ish minutes.
 
Size after conversion: 639MB, 6 minutes and 17 seconds to convert with GTX 1650 Super. (leaving audio tracks untouched)
 
Converting same file now with the libx265 now to get a size/speed comparison. Then I'll play the files side by side for quality comparison.
 
Will post results later. libx265 will probably take 4 hours since i'm only allowing it to use 2 cores/2 threads as to not bog down my server.
Awesome. Great to hear the container works for access to the GPU. I'll need to update the unraid template with what changes you made. Could you please send me a screenshot?
Link to comment
12 minutes ago, Josh.5 said:

Awesome. Great to hear the container works for access to the GPU. I'll need to update the unraid template with what changes you made. Could you please send me a screenshot?

1989086805_parametersandvariables.thumb.png.102c1fc6c592dfd980d326aadf5cb60e.png

 

Currently 20% into converting with libx265, I'll get the full comparison once its completed.

 

Compelling argument number one: Power usage is SIGNIFICANTLY lower. My server with the GPU conversion was pulling 110W from the wall for 6 minutes. I"m pulling 132W from the wall using all CPU. Just a matter of how long it needs to run. In theory I could convert an entire season of a show in less than 3 hours (23 episodes at 42 minutes each). I'm assuming that'll be the same amount of time as it takes per episode with lib encoder.

 

Actually, even faster. Current drivers allow for 3 simultaneous streams. I guess it'd stand to reason that it would allow for 3 simultaneous unmanic workers as well.

Edited by DaClownie
Link to comment
1989086805_parametersandvariables.thumb.png.102c1fc6c592dfd980d326aadf5cb60e.png

 

Currently 20% into converting with libx265, I'll get the full comparison once its completed.

 

Compelling argument number one: Power usage is SIGNIFICANTLY lower. My server with the GPU conversion was pulling 110W from the wall for 6 minutes. I"m pulling 132W from the wall using all CPU. Just a matter of how long it needs to run. In theory I could convert an entire season of a show in less than 3 hours (23 episodes at 42 minutes each). I'm assuming that'll be the same amount of time as it takes per episode with lib encoder.

Perfect.

 

So to get this working, you setup the Linuxserver.io Nvidia plugin on your unraid.

 

Then on the unmanic configuration you added the extra parameter:

--runtime=nvidia

 

Then added 2 new docker parameters:

 

 

NVIDIA_VISIBLE_DEVICES=GPU GUID

NVIDIA_DRIVER_CAPABILITIES=all

 

 

And that's it.

 

Is this right?

 

 

 

Link to comment
7 minutes ago, Josh.5 said:

Perfect.

 

So to get this working, you setup the Linuxserver.io Nvidia plugin on your unraid.

 

Then on the unmanic configuration you added the extra parameter:

 


--runtime=nvidia
 

 

 

Then added 2 new docker parameters:

 

 

 


NVIDIA_VISIBLE_DEVICES=GPU GUID
 

 

 


NVIDIA_DRIVER_CAPABILITIES=all
 

 

 

 

And that's it.

 

Is this right?

 

 

 

That's it. I'm on the 6.8.3 version of nvidia unraid. Not sure if it makes a difference.

Link to comment

OK, so here are the full results of the NVidia powered encoding vs. libx265 encoding:

 

Original File: 1.5GB. .mkv container. h264. 1080p, 42 minutes long.

 

Size after conversion with nvenc_hvec and hvec_nvenc:

639.4MB. Total time to convert the media: 6 minutes and 17 seconds.

 

Size after conversion with libx265:

598.9MB. Total time to convert the media: 2 hours, 13 minutes, 58 seconds.

 

Side by side comparison:

 

I literally can not see a difference. I watched about 5 minutes of the file side by side, and I can not see what is different about them. Mind you, the original isn't the best quality, but it proves the point of what we want to see.

 

Cost savings:

Factored at $0.30/kWh:

30W increase to power usage of server during encoding with nvidia. 0.003kWh per file. $0.0009 to encode.

 

60W increase to power usage of server during encoding with libx265. $0.039 to encode.

 

@Josh.5 You said we should use the hvec_nvenc or nvenc_hvec due to errors throw in the log? Just want to clarify before I change my settings to match and start converting whole folders.

 

 

side by side.png

Edited by DaClownie
Link to comment



OK, so here are the full results of the NVidia powered encoding vs. libx265 encoding:
 
Original File: 1.5GB. .mkv container. h264. 1080p, 42 minutes long.
 
Size after conversion with nvenc_hvec and hvec_nvenc:
639.4MB. Total time to convert the media: 6 minutes and 17 seconds.
 
Size after conversion with libx265:
598.9MB. Total time to convert the media: 2 hours, 13 minutes, 58 seconds.
 
Side by side comparison:
 
I literally can not see a difference. I watched about 5 minutes of the file side by side, and I can not see what is different about them. Mind you, the original isn't the best quality, but it proves the point of what we want to see.
 
Cost savings:
Factored at $0.30/kWh:
30W increase to power usage of server during encoding with nvidia. 0.003kWh per file. $0.0009 to encode.
 
60W increase to power usage of server during encoding with libx265. $0.039 to encode.
 
@Josh.5 You said we should use the hvec_nvenc or nvenc_hvec due to errors throw in the log? Just want to clarify before I change my settings to match and start converting whole folders.
 
 
732236571_sidebyside.thumb.png.05b5675223a946b4d3dbeb538cc63c46.png


That is awesome! Perhaps I will setup the Nvidia drivers on my unraid.

There isn't an error. It's just a warning message saying the nvenc_hvec encoder is depreciated.

I guess that is that anyway. Thanks for testing. I'll update the unraid template and merge all these changes.
Link to comment

So I put the first 33 movies from my movies that start with "H" in a separate folder. I just opened my Finder and that's where I landed. Turned on unManic.

unManic with HEVC-NVENC took it from 92.7GB to 31.0GB in less than 4.5 hours. I've averaged 50 to 90% reduction per transcoded file before so this falls in lime without NVENC. But it would have been days on my system via CPU only. The quality does not look any different to my eyes on my screens I use. YMMV. Also it didn't do all 33 files. It didn't like a handful of them. They were skipped. 

 

I just copied my wife's Criminal Minds folder which is 385GB to my test folder. I'll report what happens. 

 

Awesome work!!!!!

Link to comment
On 2/29/2020 at 5:32 PM, Jokerigno said:

image.png.fb85cf914cc58920e3debca42590e48c.png

 

and still the library recognition doesn't start.

Did you solve this? I have the exact same problem. My folder mappings are correct. If i "docker exec -it unmanic /bin/bash" i can see that all my content is in /library and the permissions are right as well. But it doesn't pick up my library..

Link to comment
12 hours ago, Josh.5 said:


 

 


That is awesome! Perhaps I will setup the Nvidia drivers on my unraid.

There isn't an error. It's just a warning message saying the nvenc_hvec encoder is depreciated.

I guess that is that anyway. Thanks for testing. I'll update the unraid template and merge all these changes.

 

 

Does this mean I need to change my docker back to its normal path? drop the dev-hw-encoding? Not sure when thats being rolled into the normal version.

Link to comment
 
Does this mean I need to change my docker back to its normal path? drop the dev-hw-encoding? Not sure when thats being rolled into the normal version.
Yea. The "latest" tag now has all the nvenc stuff added. You can get rid of that "dev-hw-encoding" tag now. Thanks.
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.