[Support] Josh5 - Unmanic - Library Optimiser


Recommended Posts

5 hours ago, rmeaux said:

So I tokk a copy of my appData, then changed ":0.0.1" to ":staging" and started unManic. I left HW Decoding unchecked. I also notice the scan takes a bit longer that 0.0.1 to populate a list. It does grab a file right off the bat though. 

 

This time the file got to about 15%, CPU pegged out on dashboard and I lost GUI. 

 

Let it stay active for a bit then stopped unManic. Took another copy of appData. 

 

Went back to 0.0.1 and its happy again. I will PM you nextCloud links to the appData files. Debugging made them too big to add here. 

 

Also, you had asked in a PM for the file that was failing. I restarted unManic multiple times in the past with :staging, which grabbed various files and they all did the same thing. It wasn't just one. Some of those files just went through with :0.0.1. 

 

Thanks for all your work @Josh.5, I know we'll (you) get it. This is my favorite docker. It is so satisfying to see TB's fall off. 

I'm not sure what "TB" is sorry.


I will look through your stuff today. Hopefully we find a smoking gun. I really suspect it is the audio transcoding changes in February. They are pretty big changes to how the app ran last year. Adding support for people who only want to modify the audio in their container even when the video codec is already correct.

Link to comment
46 minutes ago, Josh.5 said:

I'm not sure what "TB" is sorry.


I will look through your stuff today. Hopefully we find a smoking gun. I really suspect it is the audio transcoding changes in February. They are pretty big changes to how the app ran last year. Adding support for people who only want to modify the audio in their container even when the video codec is already correct.

 

TB = TeraBytes. I'm Cajun and you're Kiwi, it gets lost in translation. LOL

 

I had stopped unManic sometime late January early February when I first saw the issue I'm having. So nothing was being converted to HEVC. 

 

Since I cranked up unManic again since you provided the 0.0.1 branch to me, I've shaved off 2.8 TB's. 

 

No rush, enjoy the weekend. 

 

 

Link to comment

I just tried unmanic but can't get it to work. I added a test-library with a few movies and a temp-location for reencoding. But after scanning the folder they get added to the workers but nothing happens.
I can se that tmp folders/files gets created but they are only a few Mb and get larger super-slowly. After 10-15min they are still <10Mb.

I have tried both "staging" and regular repo. I have added "--cpus='.5'"" and have disabled audio-reencoding.

 

I am running an Ryzen 3900X with 64GB RAM

Link to comment
On 2/26/2021 at 12:32 PM, Josh.5 said:

Is anyone currently on staging that could provide feedback? I would love to find someone who can help test the staging branch for a week before I merge it into master.

The staging seems to be working good. Although I have not used it anymore, because I don't know how to work with the custom settings here. I will like to use quicksinc because the fact the doesn't kill my cpu or increases my energy bill

Link to comment
On 1/23/2021 at 12:22 AM, JaseNZ said:

Curious to the 3 session limit. Is this hard coded via nvidia.

I have a 1070 in the server and anything over 3 sessions (workers) will fail any encoding.

Not worried about it but was just curious.

It is dependent on your GPU and how many H.265 encoding threads it can handle. Nvidia has a break down in one of their developer sections with a grid that shows all the capabilities of the various GPUs which will show you how many streams your GPU can handle. I have a GTX 1060 that I use with Unmanic and I set it to two workers as I noticed my encodes for a large queue finished faster than with three workers believe it or not. 

Edited by Zer0Nin3r
Added the link to supported Nvidia encode/decode capabilities for GPUs
Link to comment

This is my first post here. I am really enjoying the app.  Question I have for anyone is has anyone been able to get a watch folder working for an unassigned usb drive. In an example I have I had my library already on an external usb 3.0 drive that I connected to Unraid using unassigned app. I was able to add it under the Unmanic settings as a watch folder but when I looked at the logs it showed it could not see the folder and no files were started. It did create a folder on the drive,Movies, but I had already a folder for TV in which the files I had were in. Just wondering if anyone had tested this further than I. Thank you for any information.

Edited by Knightwolf
Link to comment
14 hours ago, Knightwolf said:

This is my first post here. I am really enjoying the app.  Question I have for anyone is has anyone been able to get a watch folder working for an unassigned usb drive. In an example I have I had my library already on an external usb 3.0 drive that I connected to Unraid using unassigned app. I was able to add it under the Unmanic settings as a watch folder but when I looked at the logs it showed it could not see the folder and no files were started. It did create a folder on the drive,Movies, but I had already a folder for TV in which the files I had were in. Just wondering if anyone had tested this further than I. Thank you for any information.

@Knightwolf have you tried mapping the folder directly to the container in the template settings? adding a path to something like "/usb --> /mnt/remotes/usbstickpath" and then inside the unmanic container, change the watch folder path to "/usb/movies". that would be the next thing i would try if not. also worth noting, if your usb drive is in exFAT format, you'll need to be using Unassigned Devices Plus, in addition to the regular UD plugin

Link to comment
7 hours ago, Cpt. Chaz said:

@Knightwolf have you tried mapping the folder directly to the container in the template settings? adding a path to something like "/usb --> /mnt/remotes/usbstickpath" and then inside the unmanic container, change the watch folder path to "/usb/movies". that would be the next thing i would try if not. also worth noting, if your usb drive is in exFAT format, you'll need to be using Unassigned Devices Plus, in addition to the regular UD plugin

It seems to be working now. I think it was the .mkv file itself that was not being detected. Other files I tried worked.

Edited by Knightwolf
Link to comment

Been following the development of this for a bit and decided this morning that it seemed mature enough to give it a shot. I am currently converting a 1.5 GB mp4 720p file with unmanic pinned to only 2 cores. I know that there are better ways, but I am testing performance vs load at the moment to make sure that I can run it on this server without issue. If I overload it constantly, I may move it to the other server that doesn't run any dockers. The cores are residing at 98% fairly constantly, but the remaining 10 cores are barely moving and the rest of my processes are responding as promptly as before. I am going to test this for a bit and if all goes well, I may just turn it loose on the majority of my library. Except the 4K files I have worked so hard to make sure I get a quality I am happy with. :)

Thanks for building this, @Josh.5

  • Like 1
Link to comment

Does anyone have (or is anyone willing to write) a full, from-the-beginning guide for getting Intel QuickSync to work in Unmanic?

 

Not sure if I need one of those HDMI pretend dongle things, what I need to do to unRAID, what I need to do to the container settings, what I need to do within Unmanic's settings itself. Help would be much appreciated, it looks like it would be worth it.

 

Also how many workers do you have running, and does it use up any of the actual processing power of your CPU? Or are the QuickSync cores completely different?

Link to comment
17 hours ago, randomusername said:

Does anyone have (or is anyone willing to write) a full, from-the-beginning guide for getting Intel QuickSync to work in Unmanic?

 

Not sure if I need one of those HDMI pretend dongle things, what I need to do to unRAID, what I need to do to the container settings, what I need to do within Unmanic's settings itself. Help would be much appreciated, it looks like it would be worth it.

 

Also how many workers do you have running, and does it use up any of the actual processing power of your CPU? Or are the QuickSync cores completely different?

the quality on quicksync sucks in unmanic, but it's because it's preset to a lower quality encode. Something I like is that is quick, but I don't know how to tweak the custom script. It would be nice if we can get a slider for quality, something simple  like quantitazation parameter?

 

Link to comment
On 2/21/2021 at 9:09 PM, Josh.5 said:

I use 265 also, you just caught me while running some tests with 264. But both work.

 

Yes it does overwrite. So save a copy of your originals if you wish to compare.

 

In my tests between the defaults using intel and nvidia hardware acceleration, I've noticed that the intel hevc_vaapi encoder's defaults tends to compress a lot more resulting in poorer picture quality. I personally would recommend adding a few additional settings to the advanced options if using intel vaapi. Unless all you are going for is smaller files and are not so worried about the result video quality.

Know your busy but what additional settings in advanced options for vaapi? just built a new server with the intention of using quicksync instead of my p2000... would x265 be better than vaapi?

Link to comment
On 3/8/2021 at 10:36 AM, gacpac said:

You have to do it manually with the commands. which i haven't got the chance to learn. if you find your sweet settings let me know lol

https://trac.ffmpeg.org/wiki/Hardware/VAAPI

so it still better to use nvida hevc or software? quicksync seems good for transcoding, but h265 using quicksync still bad results?

i tried reading some guides and all i could really find was for streaming not for encode/decode...

 

edit: eh. i'll just run it and save on some space. if it degrades and its awful i can always change it and just encode again, or re-add movies. i went 4k because i had some 4k tv's but honestly 1080p should of been enough.

Edited by letrain
Link to comment
20 hours ago, letrain said:

so it still better to use nvida hevc or software? quicksync seems good for transcoding, but h265 using quicksync still bad results?

i tried reading some guides and all i could really find was for streaming not for encode/decode...

 

edit: eh. i'll just run it and save on some space. if it degrades and its awful i can always change it and just encode again, or re-add movies. i went 4k because i had some 4k tv's but honestly 1080p should of been enough.

quicksync in general is good. But this container doesn't have optimal settings so the quality is not as good as x265. If you want to do it you have to manually add the commands. Additionally, this container will overwrite your media, make sure you copy that somewhere else to then compare. 

 

 

Link to comment

HI I Need help please, unmanic just isnt working how it used to , im on latest unraid OS 6.9 and tried with 2 NVIDIA GPUs get same results. I also keep losing access to web UI, its really weird as ive not had any problems like this before

 

Deleted cleared data several times and retired installing to no avail, logs below, 

 

onerror(os.rmdir, path, sys.exc_info())
File "/usr/lib/python3.6/shutil.py", line 488, in rmtree
os.rmdir(path)
OSError: [Errno 39] Directory not empty: '/tmp/unmanic/unmanic_file_conversion-1615972102.5668645'
Running Unmanic from installed module
Starting migrations
There is nothing to migrate
UnmanicLogger - SETUP LOGGER
Clearing cache path - /tmp/unmanic/unmanic_file_conversion-1615972102.5668645
Traceback (most recent call last):
File "/usr/local/bin/unmanic", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python3.6/dist-packages/unmanic/service.py", line 403, in main
service.run()
File "/usr/local/lib/python3.6/dist-packages/unmanic/service.py", line 388, in run
self.start_threads()
File "/usr/local/lib/python3.6/dist-packages/unmanic/service.py", line 350, in start_threads
common.clean_files_in_dir(settings.CACHE_PATH)
File "/usr/local/lib/python3.6/dist-packages/unmanic/libs/common.py", line 105, in clean_files_in_dir
shutil.rmtree(root)
File "/usr/lib/python3.6/shutil.py", line 490, in rmtree
onerror(os.rmdir, path, sys.exc_info())
File "/usr/lib/python3.6/shutil.py", line 488, in rmtree
os.rmdir(path)
OSError: [Errno 39] Directory not empty: '/tmp/unmanic/unmanic_file_conversion-1615972102.5668645'
Running Unmanic from installed module
Starting migrations
There is nothing to migrate
UnmanicLogger - SETUP LOGGER
Clearing cache path - /tmp/unmanic/unmanic_file_conversion-1615972102.5668645
Traceback (most recent call last):
File "/usr/local/bin/unmanic", line 11, in <module>
sys.exit(main())
File "/usr/local/lib/python3.6/dist-packages/unmanic/service.py", line 403, in main
service.run()
File "/usr/local/lib/python3.6/dist-packages/unmanic/service.py", line 388, in run
self.start_threads()
File "/usr/local/lib/python3.6/dist-packages/unmanic/service.py", line 350, in start_threads
common.clean_files_in_dir(settings.CACHE_PATH)
File "/usr/local/lib/python3.6/dist-packages/unmanic/libs/common.py", line 105, in clean_files_in_dir
shutil.rmtree(root)
File "/usr/lib/python3.6/shutil.py", line 490, in rmtree
onerror(os.rmdir, path, sys.exc_info())
File "/usr/lib/python3.6/shutil.py", line 488, in rmtree
os.rmdir(path)
OSError: [Errno 39] Directory not empty: '/tmp/unmanic/unmanic_file_conversion-1615972102.5668645'

Unmanic.txt

Edited by mickh18
Link to comment
On 3/17/2021 at 4:31 AM, mickh18 said:

HI I Need help please, unmanic just isnt working how it used to , im on latest unraid OS 6.9 and tried with 2 NVIDIA GPUs get same results. I also keep losing access to web UI, its really weird as ive not had any problems like this before

 

Deleted cleared data several times and retired installing to no avail, logs below, 

 

Your symptoms sound close to mine. I was trying staging and my RAM and CPU would max out about 15% into the first transcode. I would lose web UI response for a bit and it would come back but be stuck on the same current transcode not doing anything. I went back to 0.0.1 and been running fine that way. Josh.5 thinks it may be related to Audio transcode. I tried disabling that while on staging but still had the same issue. I don't know anything about the backend of this so I am at his mercy. I love the work Josh is doing on unManic but I am perfectly happy on 0.0.1 for the time being. He'll get it. He's done nothing but improve it since its birth.  

Link to comment
On 3/18/2021 at 4:35 PM, rmeaux said:

 

Your symptoms sound close to mine. I was trying staging and my RAM and CPU would max out about 15% into the first transcode. I would lose web UI response for a bit and it would come back but be stuck on the same current transcode not doing anything. I went back to 0.0.1 and been running fine that way. Josh.5 thinks it may be related to Audio transcode. I tried disabling that while on staging but still had the same issue. I don't know anything about the backend of this so I am at his mercy. I love the work Josh is doing on unManic but I am perfectly happy on 0.0.1 for the time being. He'll get it. He's done nothing but improve it since its birth.  

Interesting. How can I downgrade to previous versions of Unmanic on Unraid to see if it resolved my issues?

Link to comment

Hi folks! Been using Unmanic for a few weeks. Finally finished re-encoding my library and it worked great. I have noticed that in the log files it seems to go back and try to re-encode a number of movies over and over again each time it scans. Is there a way to current it where it does not try and re-encode something it has already completed. Thanks.

Link to comment
On 3/20/2021 at 7:10 AM, bclinton said:

Hi folks! Been using Unmanic for a few weeks. Finally finished re-encoding my library and it worked great. I have noticed that in the log files it seems to go back and try to re-encode a number of movies over and over again each time it scans. Is there a way to current it where it does not try and re-encode something it has already completed. Thanks.

This feature has been requested many times over and I believe is on the roadmap.

 

1) Do you have Unmanic set to include closed captions? If so, try turning that off. There were issues with past releases wherein some of the CC embedded in a video file would throw an error in FFMPEG and then Unmanic would continue to keep trying those files.

 

But I see that this has been fixed now:

"Removes the subtitle stream from the container. This is useful if you intend to supply your own subtitles for your library WARNING: Unsupported subtitles will always be removed"

 

2) If you set Unmanic so that it is not including CCs, then there is something wrong with the video file. For this I turn to Handbrake (and there is a docker version of Handbrake that supports GPU encoding.) I have yet to run into a problematic video file that Handbrake couldn't handle. Try that.

 

Also, don't have Unmanic NVENC and Handbrake NVENC trying to access the same GPU at the same time or you're asking for trouble.

Link to comment

Hey team,

 

I'm getting ready to push another lot of changes to the staging branch.

 

I have been busy developing a new plugin system for Unmanic over the past 2 weeks. I chose not to merge the current staging changes into the master branch as there was identified some additional issues with conversion tasks and cache files not being delete. I am hoping that the changes that I push to the staging branch tonight fix those. But as always, if you update tomorrow and are still having issues, let me know and I will dive deeper.

 

I have begun writing some documentation on using Unamnic. You can find this here: https://docs.unmanic.app/

 

I want to give a shout out to @Cpt. Chaz as he has been putting some decent effort into making great video guides on setting up Unraid and he has made some great ones on Unmanic to date. He has provided me with some sweet updated graphics for Unmanic logos and he is taking it upon himself to create some more in-depth guides on using the application.

Check out is first Unmanic video tutorial here: 

 

 

With more to follow.

 

Cheers guys.

  • Like 2
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.