Jump to content
Josh.5

[Support] Josh5 - Unmanic - Library Optimiser

152 posts in this topic Last Reply

Recommended Posts

@Ashe @trekkiedj @itimpi

 

Please see update first post also for an example of how I have it setup. I have also added a temporary solution to adding multiple library locations if anyone needs that. I will add a feature to the WebUI to separate it in the future.

Thanks tho to you three for sorting the Docker configs out for everyone. I should have posted better details here initially.

 

@trekkiedj - Thanks for the reports regarding the ascii issues, I'm looking into that one asap. Possible ETA for a fix is by the weekend.

  • Like 3

Share this post


Link to post

Don't stress @Josh.5 You just managed to think of and start writing something everyone's been looking for, For a dam long time. So the "I wanted this 2 years ago" factor is extremely strong.

Share this post


Link to post
5 hours ago, BomB191 said:

Don't stress @Josh.5 You just managed to think of and start writing something everyone's been looking for, For a dam long time. So the "I wanted this 2 years ago" factor is extremely strong.

@Josh.5 What @BomB191 said ^^^. This work is brilliant and no rush. Don't worry at all.

Share this post


Link to post

Has there been any thought to automatically renaming the converted files to reflect the fact that they have been re-encoded?   For instance I was thinking of replacing h264 in the filename with either h265 or HEVC.

  • Like 1

Share this post


Link to post
39 minutes ago, itimpi said:

Has there been any thought to automatically renaming the converted files to reflect the fact that they have been re-encoded?   For instance I was thinking of replacing h264 in the filename with either h265 or HEVC.

Add an enhancement request on github :)

Share this post


Link to post

Just a quick note to say that i have done some testing on various files and I have seen no performance increase by using tmpfs over cache pool. given that some of the files could be fairly large and so as to not starve unraid, I have set up a unmanicache share using the SSD cache pool only and this works really well.

Share this post


Link to post

I'm assuming since its a docker on github we have to create all of the fields via the properties page. I attempted to install and didn't see any of the fields listed above. 

Share this post


Link to post
2 hours ago, kizer said:

I'm assuming since its a docker on github we have to create all of the fields via the properties page. I attempted to install and didn't see any of the fields listed above. 

Yep. there's some info in the OP on what to set to what.

Share this post


Link to post
14 hours ago, kizer said:

I'm assuming since its a docker on github we have to create all of the fields via the properties page. I attempted to install and didn't see any of the fields listed above. 

Or you could probably add this as a repository on the docker page

 

https://github.com/Josh5/unmanic/tree/master/docker

  • Upvote 1

Share this post


Link to post

There should be a new update available. This has quite a number of changes to how ffmpeg is executed.

It should resolve some issues with inotify and library scanning. For those people who created issues on github you will see the ones that "should" be now fixed marked as closed.

If I get time tomorrow I will make a push for adding some features. Perhaps some additional settings for file conversions.

  • Upvote 3

Share this post


Link to post

Oh good. Mines been chilling doing nothing for a few days just seems to hang with no HDD I/O at all. Shal check it out

 

Edit: scratch that the 3 workers get to around 10% and just starts chilling... I wonder if me slamming it with a TB of kids shows if giving it a hard time. ill change the folder its pulling from.

 

Edited by BomB191

Share this post


Link to post

Hi @Josh.5. The latest update is not functioning for me at all. Still cannot access the webui even if set to a single folder, and now nothing is converting at all. No disk activity or cpu activity.

This is what i see when trying to access the webui

Untitled.png

Share this post


Link to post
1 minute ago, trekkiedj said:

Hi @Josh.5. The latest update is not functioning for me at all. Still cannot access the webui even if set to a single folder, and now nothing is converting at all. No disk activity or cpu activity.

This is what i see when trying to access the webui

Untitled.png

I tested a bunch of files on this unicode error. I thought I had it right. Are you able to give me the exact characters that you are having a problem with?

Share this post


Link to post
4 hours ago, Josh.5 said:

There should be a new update available. This has quite a number of changes to how ffmpeg is executed.

It should resolve some issues with inotify and library scanning. For those people who created issues on github you will see the ones that "should" be now fixed marked as closed.

If I get time tomorrow I will make a push for adding some features. Perhaps some additional settings for file conversions.

Working well for me, thanks @Josh.5. Looking forward to seeing some new features. Thanks again for your work on this.

Share this post


Link to post
10 minutes ago, Josh.5 said:

I tested a bunch of files on this unicode error. I thought I had it right. Are you able to give me the exact characters that you are having a problem with?

I set the library folder to a single movie (8 mile) and this is whats in the log:

 

'clean_effects': 0,
'comment': 0,
'default': 0,
'dub': 0,
'forced': 0,
'hearing_impaired': 0,
'karaoke': 0,
'lyrics': 0,
'original': 0,
'timed_thumbnails': 0,
'visual_impaired': 0},
'duration': '6627.796000',
'duration_ts': 6627796,
'height': 1080,
'index': 10,
'r_frame_rate': '0/0',
'start_pts': 0,
'start_time': '0.000000',
'tags': {'BPS': '3866',
'BPS-eng': '3866',
'DURATION': '01:45:06.289000000',
'DURATION-eng': '01:45:06.289000000',
'NUMBER_OF_BYTES': '3047830',
'NUMBER_OF_BYTES-eng': '3047830',
'NUMBER_OF_FRAMES': '484',
'NUMBER_OF_FRAMES-eng': '484',
'_STATISTICS_TAGS': 'BPS DURATION NUMBER_OF_FRAMES '
'NUMBER_OF_BYTES',
'_STATISTICS_TAGS-eng': 'BPS DURATION NUMBER_OF_FRAMES '
'NUMBER_OF_BYTES',
'_STATISTICS_WRITING_APP': "mkvmerge v20.0.0 ('I Am The "
"Sun') 64-bit",
'_STATISTICS_WRITING_APP-eng': "mkvmerge v20.0.0 ('I Am "
"The Sun') 64-bit",
'_STATISTICS_WRITING_DATE_UTC': '2018-07-03 21:52:18',
'_STATISTICS_WRITING_DATE_UTC-eng': '2018-07-03 '
'21:52:18',
'language': 'ger',
'title': 'Forced'},
'time_base': '1/1000',
'width': 1920},
{'avg_frame_rate': '0/0',
'codec_long_name': 'DVD subtitles',
'codec_name': 'dvd_subtitle',
'codec_tag': '0x0000',
'codec_tag_string': '[0][0][0][0]',
'codec_time_base': '0/1',
'codec_type': 'subtitle',
'disposition': {'attached_pic': 0,
'clean_effects': 0,
'comment': 0,
'default': 0,
'dub': 0,
'forced': 0,
'hearing_impaired': 0,
'karaoke': 0,
'lyrics': 0,
'original': 0,
'timed_thumbnails': 0,
'visual_impaired': 0},
'duration': '6627.796000',
'duration_ts': 6627796,
'height': 1080,
'index': 11,
'r_frame_rate': '0/0',
'start_pts': 0,
'start_time': '0.000000',
'tags': {'BPS': '9567',
'BPS-eng': '9567',
'DURATION': '01:48:03.299000000',
'DURATION-eng': '01:48:03.299000000',
'NUMBER_OF_BYTES': '7753322',
'NUMBER_OF_BYTES-eng': '7753322',
'NUMBER_OF_FRAMES': '1481',
'NUMBER_OF_FRAMES-eng': '1481',
'_STATISTICS_TAGS': 'BPS DURATION NUMBER_OF_FRAMES '
'NUMBER_OF_BYTES',
'_STATISTICS_TAGS-eng': 'BPS DURATION NUMBER_OF_FRAMES '
'NUMBER_OF_BYTES',
'_STATISTICS_WRITING_APP': "mkvmerge v20.0.0 ('I Am The "
"Sun') 64-bit",
'_STATISTICS_WRITING_APP-eng': "mkvmerge v20.0.0 ('I Am "
"The Sun') 64-bit",
'_STATISTICS_WRITING_DATE_UTC': '2018-07-03 21:52:18',
'_STATISTICS_WRITING_DATE_UTC-eng': '2018-07-03 '
'21:52:18',
'language': 'ita'},
'time_base': '1/1000',
'width': 1920},
{'avg_frame_rate': '0/0',
'codec_long_name': 'DVD subtitles',
'codec_name': 'dvd_subtitle',
'codec_tag': '0x0000',
'codec_tag_string': '[0][0][0][0]',
'codec_time_base': '0/1',
'codec_type': 'subtitle',
'disposition': {'attached_pic': 0,
'clean_effects': 0,
'comment': 0,
'default': 0,
'dub': 0,
'forced': 0,
'hearing_impaired': 0,
'karaoke': 0,
'lyrics': 0,
'original': 0,
'timed_thumbnails': 0,
'visual_impaired': 0},
'duration': '6627.796000',
'duration_ts': 6627796,
'height': 1080,
'index': 12,
'r_frame_rate': '0/0',
'start_pts': 0,
'start_time': '0.000000',
'tags': {'BPS': '7001',
'BPS-eng': '7001',
'DURATION': '01:42:51.245000000',
'DURATION-eng': '01:42:51.245000000',
'NUMBER_OF_BYTES': '5400890',
'NUMBER_OF_BYTES-eng': '5400890',
'NUMBER_OF_FRAMES': '1444',
'NUMBER_OF_FRAMES-eng': '1444',
'_STATISTICS_TAGS': 'BPS DURATION NUMBER_OF_FRAMES '
'NUMBER_OF_BYTES',
'_STATISTICS_TAGS-eng': 'BPS DURATION NUMBER_OF_FRAMES '
'NUMBER_OF_BYTES',
'_STATISTICS_WRITING_APP': "mkvmerge v20.0.0 ('I Am The "
"Sun') 64-bit",
'_STATISTICS_WRITING_APP-eng': "mkvmerge v20.0.0 ('I Am "
"The Sun') 64-bit",
'_STATISTICS_WRITING_DATE_UTC': '2018-07-03 21:52:18',
'_STATISTICS_WRITING_DATE_UTC-eng': '2018-07-03 '
'21:52:18',
'language': 'jpn'},
'time_base': '1/1000',
'width': 1920},
{'avg_frame_rate': '0/0',
'codec_long_name': 'DVD subtitles',
'codec_name': 'dvd_subtitle',
'codec_tag': '0x0000',
'codec_tag_string': '[0][0][0][0]',
'codec_time_base': '0/1',
'codec_type': 'subtitle',
'disposition': {'attached_pic': 0,
'clean_effects': 0,
'comment': 0,
'default': 0,
'dub': 0,
'forced': 0,
'hearing_impaired': 0,
'karaoke': 0,
'lyrics': 0,
'original': 0,
'timed_thumbnails': 0,
'visual_impaired': 0},
'duration': '6627.796000',
'duration_ts': 6627796,
'height': 1080,
'index': 13,
'r_frame_rate': '0/0',
'start_pts': 0,
'start_time': '0.000000',
'tags': {'BPS': '8468',
'BPS-eng': '8468',
'DURATION': '01:48:03.299000000',
'DURATION-eng': '01:48:03.299000000',
'NUMBER_OF_BYTES': '6862870',
'NUMBER_OF_BYTES-eng': '6862870',
'NUMBER_OF_FRAMES': '1481',
'NUMBER_OF_FRAMES-eng': '1481',
'_STATISTICS_TAGS': 'BPS DURATION NUMBER_OF_FRAMES '
'NUMBER_OF_BYTES',
'_STATISTICS_TAGS-eng': 'BPS DURATION NUMBER_OF_FRAMES '
'NUMBER_OF_BYTES',
'_STATISTICS_WRITING_APP': "mkvmerge v20.0.0 ('I Am The "
"Sun') 64-bit",
'_STATISTICS_WRITING_APP-eng': "mkvmerge v20.0.0 ('I Am "
"The Sun') 64-bit",
'_STATISTICS_WRITING_DATE_UTC': '2018-07-03 21:52:18',
'_STATISTICS_WRITING_DATE_UTC-eng': '2018-07-03 '
'21:52:18',
'language': 'nob'},
'time_base': '1/1000',
'width': 1920},
{'avg_frame_rate': '0/0',
'codec_long_name': 'DVD subtitles',
'codec_name': 'dvd_subtitle',
'codec_tag': '0x0000',
'codec_tag_string': '[0][0][0][0]',
'codec_time_base': '0/1',
'codec_type': 'subtitle',
'disposition': {'attached_pic': 0,
'clean_effects': 0,
'comment': 0,
'default': 0,
'dub': 0,
'forced': 0,
'hearing_impaired': 0,
'karaoke': 0,
'lyrics': 0,
'original': 0,
'timed_thumbnails': 0,
'visual_impaired': 0},
'duration': '6627.796000',
'duration_ts': 6627796,
'height': 1080,
'index': 14,
'r_frame_rate': '0/0',
'start_pts': 0,
'start_time': '0.000000',
'tags': {'BPS': '10040',
'BPS-eng': '10040',
'DURATION': '01:48:03.299000000',
'DURATION-eng': '01:48:03.299000000',
'NUMBER_OF_BYTES': '8137322',
'NUMBER_OF_BYTES-eng': '8137322',
'NUMBER_OF_FRAMES': '1481',
'NUMBER_OF_FRAMES-eng': '1481',
'_STATISTICS_TAGS': 'BPS DURATION NUMBER_OF_FRAMES '
'NUMBER_OF_BYTES',
'_STATISTICS_TAGS-eng': 'BPS DURATION NUMBER_OF_FRAMES '
'NUMBER_OF_BYTES',
'_STATISTICS_WRITING_APP': "mkvmerge v20.0.0 ('I Am The "
"Sun') 64-bit",
'_STATISTICS_WRITING_APP-eng': "mkvmerge v20.0.0 ('I Am "
"The Sun') 64-bit",
'_STATISTICS_WRITING_DATE_UTC': '2018-07-03 21:52:18',
'_STATISTICS_WRITING_DATE_UTC-eng': '2018-07-03 '
'21:52:18',
'language': 'por'},
'time_base': '1/1000',
'width': 1920},
{'avg_frame_rate': '0/0',
'codec_long_name': 'DVD subtitles',
'codec_name': 'dvd_subtitle',
'codec_tag': '0x0000',
'codec_tag_string': '[0][0][0][0]',
'codec_time_base': '0/1',
'codec_type': 'subtitle',
'disposition': {'attached_pic': 0,
'clean_effects': 0,
'comment': 0,
'default': 0,
'dub': 0,
'forced': 0,
'hearing_impaired': 0,
'karaoke': 0,
'lyrics': 0,
'original': 0,
'timed_thumbnails': 0,
'visual_impaired': 0},
'duration': '6627.796000',
'duration_ts': 6627796,
'height': 1080,
'index': 15,
'r_frame_rate': '0/0',
'start_pts': 0,
'start_time': '0.000000',
'tags': {'BPS': '9314',
'BPS-eng': '9314',
'DURATION': '01:48:03.299000000',
'DURATION-eng': '01:48:03.299000000',
'NUMBER_OF_BYTES': '7548556',
'NUMBER_OF_BYTES-eng': '7548556',
'NUMBER_OF_FRAMES': '1481',
'NUMBER_OF_FRAMES-eng': '1481',
'_STATISTICS_TAGS': 'BPS DURATION NUMBER_OF_FRAMES '
'NUMBER_OF_BYTES',
'_STATISTICS_TAGS-eng': 'BPS DURATION NUMBER_OF_FRAMES '
'NUMBER_OF_BYTES',
'_STATISTICS_WRITING_APP': "mkvmerge v20.0.0 ('I Am The "
"Sun') 64-bit",
'_STATISTICS_WRITING_APP-eng': "mkvmerge v20.0.0 ('I Am "
"The Sun') 64-bit",
'_STATISTICS_WRITING_DATE_UTC': '2018-07-03 21:52:18',
'_STATISTICS_WRITING_DATE_UTC-eng': '2018-07-03 '
'21:52:18',
'language': 'spa'},
'time_base': '1/1000',
'width': 1920},
{'avg_frame_rate': '0/0',
'codec_long_name': 'DVD subtitles',
'codec_name': 'dvd_subtitle',
'codec_tag': '0x0000',
'codec_tag_string': '[0][0][0][0]',
'codec_time_base': '0/1',
'codec_type': 'subtitle',
'disposition': {'attached_pic': 0,
'clean_effects': 0,
'comment': 0,
'default': 0,
'dub': 0,
'forced': 0,
'hearing_impaired': 0,
'karaoke': 0,
'lyrics': 0,
'original': 0,
'timed_thumbnails': 0,
'visual_impaired': 0},
'duration': '6627.796000',
'duration_ts': 6627796,
'height': 1080,
'index': 16,
'r_frame_rate': '0/0',
'start_pts': 0,
'start_time': '0.000000',
'tags': {'BPS': '8872',
'BPS-eng': '8872',
'DURATION': '01:48:03.299000000',
'DURATION-eng': '01:48:03.299000000',
'NUMBER_OF_BYTES': '7190207',
'NUMBER_OF_BYTES-eng': '7190207',
'NUMBER_OF_FRAMES': '1481',
'NUMBER_OF_FRAMES-eng': '1481',
'_STATISTICS_TAGS': 'BPS DURATION NUMBER_OF_FRAMES '
'NUMBER_OF_BYTES',
'_STATISTICS_TAGS-eng': 'BPS DURATION NUMBER_OF_FRAMES '
'NUMBER_OF_BYTES',
'_STATISTICS_WRITING_APP': "mkvmerge v20.0.0 ('I Am The "
"Sun') 64-bit",
'_STATISTICS_WRITING_APP-eng': "mkvmerge v20.0.0 ('I Am "
"The Sun') 64-bit",
'_STATISTICS_WRITING_DATE_UTC': '2018-07-03 21:52:18',
'_STATISTICS_WRITING_DATE_UTC-eng': '2018-07-03 '
'21:52:18',
'language': 'swe'},
'time_base': '1/1000',
'width': 1920}]}
DEBUG:root:LibraryOptimiser - [FFMPEGHandle] File already hevc - /library/Movies/8 Mile (2002).mkv
INFO:root:LibraryOptimiser - [LibraryScanner] File does not need to be processed - /library/Movies/8 Mile (2002).mkv
ERROR:tornado.application:Uncaught exception GET / (192.168.1.130)

HTTPServerRequest(protocol='http', host='192.168.1.131:8888', method='GET', uri='/', version='HTTP/1.1', remote_ip='192.168.1.130')

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/tornado/web.py", line 1590, in _execute
result = method(*self.path_args, **self.path_kwargs)
File "/app/webserver/main.py", line 51, in get
self.render("main.html", time_now=time.time())
File "/usr/lib/python3.6/site-packages/tornado/web.py", line 766, in render
html = self.render_string(template_name, **kwargs)
File "/usr/lib/python3.6/site-packages/tornado/web.py", line 907, in render_string
return t.generate(**namespace)
File "/usr/lib/python3.6/site-packages/tornado/template.py", line 346, in generate
return execute()
File "main_html.generated.py", line 120, in _tt_execute
if isinstance(_tt_tmp, _tt_string_types): _tt_tmp = _tt_utf8(_tt_tmp) # main-completed-tasks.html:20 (via main.html:216, page_layout.html:123)
File "/usr/lib/python3.6/site-packages/tornado/escape.py", line 199, in utf8
return value.encode("utf-8")
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 35-37: surrogates not allowed
ERROR:tornado.access:500 GET / (192.168.1.130) 422.43ms

 

Share this post


Link to post
12 minutes ago, Josh.5 said:

I tested a bunch of files on this unicode error. I thought I had it right. Are you able to give me the exact characters that you are having a problem with?

should say also, it is now encoding. Just no WebUI

Share this post


Link to post

@trekkiedj can you please try from your unraid terminal copying the file `/mnt/user/appdata/unmanic/.unmanic/config/history.json`
(This was the default location. If you have changed your appdata folder it may be different on your machine)

From the looks of things there is something in your file history that is causing a problem. If you could give me this file it may help me figure it out. Also once you have a copy of it for me to debug, deleting it may be enough to set you right again so you can access the webUI

Share this post


Link to post
Just now, trekkiedj said:

should say also, it is now encoding. Just no WebUI

have you set up the web ui link?

Share this post


Link to post
2 minutes ago, Josh.5 said:

@trekkiedj can you please try from your unraid terminal copying the file `/mnt/user/appdata/unmanic/.unmanic/config/history.json`
(This was the default location. If you have changed your appdata folder it may be different on your machine)

From the looks of things there is something in your file history that is causing a problem. If you could give me this file it may help me figure it out. Also once you have a copy of it for me to debug, deleting it may be enough to set you right again so you can access the webUI

Here it is

history.json

Share this post


Link to post
1 minute ago, BomB191 said:

have you set up the web ui link?

The log that he is showing is from the webUI. It is reading some non unicode character from his history log file and falling over. If I can see what that character is I may be able to figure out why it cannot convert it to a string.

Share this post


Link to post
4 minutes ago, Josh.5 said:

@trekkiedj can you please try from your unraid terminal copying the file `/mnt/user/appdata/unmanic/.unmanic/config/history.json`
(This was the default location. If you have changed your appdata folder it may be different on your machine)

From the looks of things there is something in your file history that is causing a problem. If you could give me this file it may help me figure it out. Also once you have a copy of it for me to debug, deleting it may be enough to set you right again so you can access the webUI

And deleting it fixed the issue, thanks Josh!

Share this post


Link to post
12 minutes ago, trekkiedj said:

And deleting it fixed the issue, thanks Josh!

What are the characters of this file:
 

Storage Wars - S12E09 - Let's Give \udce2\udc80\udc98Em Something to Tonka About - WEBDL-720p - h264 AAC.mkv

I have a funny feeling that this is not longer a problem and that somehow the file's names managed to make it through the conversion process and written to this history in this messed up state.

The changes made on the last update should prevent this from happening again. So if it does happen again please report it to me. I will see if I can come up with a tidy way to gracefully handle this error in the webUI so it does not error out quite so hard...

Edited by Josh.5
More info

Share this post


Link to post

@Josh.5 so the update seemed to be running fine but now its stalled. no weird characters that I can see just. gave up.

image.png.29964c0ac13cde2fd74a1a661e9d9f87.png

 

image.png

settings.json

Edited by BomB191
more info

Share this post


Link to post
2 minutes ago, BomB191 said:

@Josh.5 so the update seemed to be running fine but now its stalled. no weird characters that I can see just. gave up.

image.png.29964c0ac13cde2fd74a1a661e9d9f87.png

 

image.png

settings.json

Just checked mine and it's the same. Gets 12% in and stops encoding.

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now