[Support] Josh5 - Unmanic - Library Optimiser


Recommended Posts

@Ashe @trekkiedj @itimpi

 

Please see update first post also for an example of how I have it setup. I have also added a temporary solution to adding multiple library locations if anyone needs that. I will add a feature to the WebUI to separate it in the future.

Thanks tho to you three for sorting the Docker configs out for everyone. I should have posted better details here initially.

 

@trekkiedj - Thanks for the reports regarding the ascii issues, I'm looking into that one asap. Possible ETA for a fix is by the weekend.

  • Like 4
Link to comment
39 minutes ago, itimpi said:

Has there been any thought to automatically renaming the converted files to reflect the fact that they have been re-encoded?   For instance I was thinking of replacing h264 in the filename with either h265 or HEVC.

Add an enhancement request on github :)

Link to comment

Just a quick note to say that i have done some testing on various files and I have seen no performance increase by using tmpfs over cache pool. given that some of the files could be fairly large and so as to not starve unraid, I have set up a unmanicache share using the SSD cache pool only and this works really well.

Link to comment

There should be a new update available. This has quite a number of changes to how ffmpeg is executed.

It should resolve some issues with inotify and library scanning. For those people who created issues on github you will see the ones that "should" be now fixed marked as closed.

If I get time tomorrow I will make a push for adding some features. Perhaps some additional settings for file conversions.

  • Like 1
  • Upvote 3
Link to comment

Oh good. Mines been chilling doing nothing for a few days just seems to hang with no HDD I/O at all. Shal check it out

 

Edit: scratch that the 3 workers get to around 10% and just starts chilling... I wonder if me slamming it with a TB of kids shows if giving it a hard time. ill change the folder its pulling from.

 

Edited by BomB191
Link to comment
1 minute ago, trekkiedj said:

Hi @Josh.5. The latest update is not functioning for me at all. Still cannot access the webui even if set to a single folder, and now nothing is converting at all. No disk activity or cpu activity.

This is what i see when trying to access the webui

Untitled.png

I tested a bunch of files on this unicode error. I thought I had it right. Are you able to give me the exact characters that you are having a problem with?

Link to comment
4 hours ago, Josh.5 said:

There should be a new update available. This has quite a number of changes to how ffmpeg is executed.

It should resolve some issues with inotify and library scanning. For those people who created issues on github you will see the ones that "should" be now fixed marked as closed.

If I get time tomorrow I will make a push for adding some features. Perhaps some additional settings for file conversions.

Working well for me, thanks @Josh.5. Looking forward to seeing some new features. Thanks again for your work on this.

Link to comment
10 minutes ago, Josh.5 said:

I tested a bunch of files on this unicode error. I thought I had it right. Are you able to give me the exact characters that you are having a problem with?

I set the library folder to a single movie (8 mile) and this is whats in the log:

 

'clean_effects': 0,
'comment': 0,
'default': 0,
'dub': 0,
'forced': 0,
'hearing_impaired': 0,
'karaoke': 0,
'lyrics': 0,
'original': 0,
'timed_thumbnails': 0,
'visual_impaired': 0},
'duration': '6627.796000',
'duration_ts': 6627796,
'height': 1080,
'index': 10,
'r_frame_rate': '0/0',
'start_pts': 0,
'start_time': '0.000000',
'tags': {'BPS': '3866',
'BPS-eng': '3866',
'DURATION': '01:45:06.289000000',
'DURATION-eng': '01:45:06.289000000',
'NUMBER_OF_BYTES': '3047830',
'NUMBER_OF_BYTES-eng': '3047830',
'NUMBER_OF_FRAMES': '484',
'NUMBER_OF_FRAMES-eng': '484',
'_STATISTICS_TAGS': 'BPS DURATION NUMBER_OF_FRAMES '
'NUMBER_OF_BYTES',
'_STATISTICS_TAGS-eng': 'BPS DURATION NUMBER_OF_FRAMES '
'NUMBER_OF_BYTES',
'_STATISTICS_WRITING_APP': "mkvmerge v20.0.0 ('I Am The "
"Sun') 64-bit",
'_STATISTICS_WRITING_APP-eng': "mkvmerge v20.0.0 ('I Am "
"The Sun') 64-bit",
'_STATISTICS_WRITING_DATE_UTC': '2018-07-03 21:52:18',
'_STATISTICS_WRITING_DATE_UTC-eng': '2018-07-03 '
'21:52:18',
'language': 'ger',
'title': 'Forced'},
'time_base': '1/1000',
'width': 1920},
{'avg_frame_rate': '0/0',
'codec_long_name': 'DVD subtitles',
'codec_name': 'dvd_subtitle',
'codec_tag': '0x0000',
'codec_tag_string': '[0][0][0][0]',
'codec_time_base': '0/1',
'codec_type': 'subtitle',
'disposition': {'attached_pic': 0,
'clean_effects': 0,
'comment': 0,
'default': 0,
'dub': 0,
'forced': 0,
'hearing_impaired': 0,
'karaoke': 0,
'lyrics': 0,
'original': 0,
'timed_thumbnails': 0,
'visual_impaired': 0},
'duration': '6627.796000',
'duration_ts': 6627796,
'height': 1080,
'index': 11,
'r_frame_rate': '0/0',
'start_pts': 0,
'start_time': '0.000000',
'tags': {'BPS': '9567',
'BPS-eng': '9567',
'DURATION': '01:48:03.299000000',
'DURATION-eng': '01:48:03.299000000',
'NUMBER_OF_BYTES': '7753322',
'NUMBER_OF_BYTES-eng': '7753322',
'NUMBER_OF_FRAMES': '1481',
'NUMBER_OF_FRAMES-eng': '1481',
'_STATISTICS_TAGS': 'BPS DURATION NUMBER_OF_FRAMES '
'NUMBER_OF_BYTES',
'_STATISTICS_TAGS-eng': 'BPS DURATION NUMBER_OF_FRAMES '
'NUMBER_OF_BYTES',
'_STATISTICS_WRITING_APP': "mkvmerge v20.0.0 ('I Am The "
"Sun') 64-bit",
'_STATISTICS_WRITING_APP-eng': "mkvmerge v20.0.0 ('I Am "
"The Sun') 64-bit",
'_STATISTICS_WRITING_DATE_UTC': '2018-07-03 21:52:18',
'_STATISTICS_WRITING_DATE_UTC-eng': '2018-07-03 '
'21:52:18',
'language': 'ita'},
'time_base': '1/1000',
'width': 1920},
{'avg_frame_rate': '0/0',
'codec_long_name': 'DVD subtitles',
'codec_name': 'dvd_subtitle',
'codec_tag': '0x0000',
'codec_tag_string': '[0][0][0][0]',
'codec_time_base': '0/1',
'codec_type': 'subtitle',
'disposition': {'attached_pic': 0,
'clean_effects': 0,
'comment': 0,
'default': 0,
'dub': 0,
'forced': 0,
'hearing_impaired': 0,
'karaoke': 0,
'lyrics': 0,
'original': 0,
'timed_thumbnails': 0,
'visual_impaired': 0},
'duration': '6627.796000',
'duration_ts': 6627796,
'height': 1080,
'index': 12,
'r_frame_rate': '0/0',
'start_pts': 0,
'start_time': '0.000000',
'tags': {'BPS': '7001',
'BPS-eng': '7001',
'DURATION': '01:42:51.245000000',
'DURATION-eng': '01:42:51.245000000',
'NUMBER_OF_BYTES': '5400890',
'NUMBER_OF_BYTES-eng': '5400890',
'NUMBER_OF_FRAMES': '1444',
'NUMBER_OF_FRAMES-eng': '1444',
'_STATISTICS_TAGS': 'BPS DURATION NUMBER_OF_FRAMES '
'NUMBER_OF_BYTES',
'_STATISTICS_TAGS-eng': 'BPS DURATION NUMBER_OF_FRAMES '
'NUMBER_OF_BYTES',
'_STATISTICS_WRITING_APP': "mkvmerge v20.0.0 ('I Am The "
"Sun') 64-bit",
'_STATISTICS_WRITING_APP-eng': "mkvmerge v20.0.0 ('I Am "
"The Sun') 64-bit",
'_STATISTICS_WRITING_DATE_UTC': '2018-07-03 21:52:18',
'_STATISTICS_WRITING_DATE_UTC-eng': '2018-07-03 '
'21:52:18',
'language': 'jpn'},
'time_base': '1/1000',
'width': 1920},
{'avg_frame_rate': '0/0',
'codec_long_name': 'DVD subtitles',
'codec_name': 'dvd_subtitle',
'codec_tag': '0x0000',
'codec_tag_string': '[0][0][0][0]',
'codec_time_base': '0/1',
'codec_type': 'subtitle',
'disposition': {'attached_pic': 0,
'clean_effects': 0,
'comment': 0,
'default': 0,
'dub': 0,
'forced': 0,
'hearing_impaired': 0,
'karaoke': 0,
'lyrics': 0,
'original': 0,
'timed_thumbnails': 0,
'visual_impaired': 0},
'duration': '6627.796000',
'duration_ts': 6627796,
'height': 1080,
'index': 13,
'r_frame_rate': '0/0',
'start_pts': 0,
'start_time': '0.000000',
'tags': {'BPS': '8468',
'BPS-eng': '8468',
'DURATION': '01:48:03.299000000',
'DURATION-eng': '01:48:03.299000000',
'NUMBER_OF_BYTES': '6862870',
'NUMBER_OF_BYTES-eng': '6862870',
'NUMBER_OF_FRAMES': '1481',
'NUMBER_OF_FRAMES-eng': '1481',
'_STATISTICS_TAGS': 'BPS DURATION NUMBER_OF_FRAMES '
'NUMBER_OF_BYTES',
'_STATISTICS_TAGS-eng': 'BPS DURATION NUMBER_OF_FRAMES '
'NUMBER_OF_BYTES',
'_STATISTICS_WRITING_APP': "mkvmerge v20.0.0 ('I Am The "
"Sun') 64-bit",
'_STATISTICS_WRITING_APP-eng': "mkvmerge v20.0.0 ('I Am "
"The Sun') 64-bit",
'_STATISTICS_WRITING_DATE_UTC': '2018-07-03 21:52:18',
'_STATISTICS_WRITING_DATE_UTC-eng': '2018-07-03 '
'21:52:18',
'language': 'nob'},
'time_base': '1/1000',
'width': 1920},
{'avg_frame_rate': '0/0',
'codec_long_name': 'DVD subtitles',
'codec_name': 'dvd_subtitle',
'codec_tag': '0x0000',
'codec_tag_string': '[0][0][0][0]',
'codec_time_base': '0/1',
'codec_type': 'subtitle',
'disposition': {'attached_pic': 0,
'clean_effects': 0,
'comment': 0,
'default': 0,
'dub': 0,
'forced': 0,
'hearing_impaired': 0,
'karaoke': 0,
'lyrics': 0,
'original': 0,
'timed_thumbnails': 0,
'visual_impaired': 0},
'duration': '6627.796000',
'duration_ts': 6627796,
'height': 1080,
'index': 14,
'r_frame_rate': '0/0',
'start_pts': 0,
'start_time': '0.000000',
'tags': {'BPS': '10040',
'BPS-eng': '10040',
'DURATION': '01:48:03.299000000',
'DURATION-eng': '01:48:03.299000000',
'NUMBER_OF_BYTES': '8137322',
'NUMBER_OF_BYTES-eng': '8137322',
'NUMBER_OF_FRAMES': '1481',
'NUMBER_OF_FRAMES-eng': '1481',
'_STATISTICS_TAGS': 'BPS DURATION NUMBER_OF_FRAMES '
'NUMBER_OF_BYTES',
'_STATISTICS_TAGS-eng': 'BPS DURATION NUMBER_OF_FRAMES '
'NUMBER_OF_BYTES',
'_STATISTICS_WRITING_APP': "mkvmerge v20.0.0 ('I Am The "
"Sun') 64-bit",
'_STATISTICS_WRITING_APP-eng': "mkvmerge v20.0.0 ('I Am "
"The Sun') 64-bit",
'_STATISTICS_WRITING_DATE_UTC': '2018-07-03 21:52:18',
'_STATISTICS_WRITING_DATE_UTC-eng': '2018-07-03 '
'21:52:18',
'language': 'por'},
'time_base': '1/1000',
'width': 1920},
{'avg_frame_rate': '0/0',
'codec_long_name': 'DVD subtitles',
'codec_name': 'dvd_subtitle',
'codec_tag': '0x0000',
'codec_tag_string': '[0][0][0][0]',
'codec_time_base': '0/1',
'codec_type': 'subtitle',
'disposition': {'attached_pic': 0,
'clean_effects': 0,
'comment': 0,
'default': 0,
'dub': 0,
'forced': 0,
'hearing_impaired': 0,
'karaoke': 0,
'lyrics': 0,
'original': 0,
'timed_thumbnails': 0,
'visual_impaired': 0},
'duration': '6627.796000',
'duration_ts': 6627796,
'height': 1080,
'index': 15,
'r_frame_rate': '0/0',
'start_pts': 0,
'start_time': '0.000000',
'tags': {'BPS': '9314',
'BPS-eng': '9314',
'DURATION': '01:48:03.299000000',
'DURATION-eng': '01:48:03.299000000',
'NUMBER_OF_BYTES': '7548556',
'NUMBER_OF_BYTES-eng': '7548556',
'NUMBER_OF_FRAMES': '1481',
'NUMBER_OF_FRAMES-eng': '1481',
'_STATISTICS_TAGS': 'BPS DURATION NUMBER_OF_FRAMES '
'NUMBER_OF_BYTES',
'_STATISTICS_TAGS-eng': 'BPS DURATION NUMBER_OF_FRAMES '
'NUMBER_OF_BYTES',
'_STATISTICS_WRITING_APP': "mkvmerge v20.0.0 ('I Am The "
"Sun') 64-bit",
'_STATISTICS_WRITING_APP-eng': "mkvmerge v20.0.0 ('I Am "
"The Sun') 64-bit",
'_STATISTICS_WRITING_DATE_UTC': '2018-07-03 21:52:18',
'_STATISTICS_WRITING_DATE_UTC-eng': '2018-07-03 '
'21:52:18',
'language': 'spa'},
'time_base': '1/1000',
'width': 1920},
{'avg_frame_rate': '0/0',
'codec_long_name': 'DVD subtitles',
'codec_name': 'dvd_subtitle',
'codec_tag': '0x0000',
'codec_tag_string': '[0][0][0][0]',
'codec_time_base': '0/1',
'codec_type': 'subtitle',
'disposition': {'attached_pic': 0,
'clean_effects': 0,
'comment': 0,
'default': 0,
'dub': 0,
'forced': 0,
'hearing_impaired': 0,
'karaoke': 0,
'lyrics': 0,
'original': 0,
'timed_thumbnails': 0,
'visual_impaired': 0},
'duration': '6627.796000',
'duration_ts': 6627796,
'height': 1080,
'index': 16,
'r_frame_rate': '0/0',
'start_pts': 0,
'start_time': '0.000000',
'tags': {'BPS': '8872',
'BPS-eng': '8872',
'DURATION': '01:48:03.299000000',
'DURATION-eng': '01:48:03.299000000',
'NUMBER_OF_BYTES': '7190207',
'NUMBER_OF_BYTES-eng': '7190207',
'NUMBER_OF_FRAMES': '1481',
'NUMBER_OF_FRAMES-eng': '1481',
'_STATISTICS_TAGS': 'BPS DURATION NUMBER_OF_FRAMES '
'NUMBER_OF_BYTES',
'_STATISTICS_TAGS-eng': 'BPS DURATION NUMBER_OF_FRAMES '
'NUMBER_OF_BYTES',
'_STATISTICS_WRITING_APP': "mkvmerge v20.0.0 ('I Am The "
"Sun') 64-bit",
'_STATISTICS_WRITING_APP-eng': "mkvmerge v20.0.0 ('I Am "
"The Sun') 64-bit",
'_STATISTICS_WRITING_DATE_UTC': '2018-07-03 21:52:18',
'_STATISTICS_WRITING_DATE_UTC-eng': '2018-07-03 '
'21:52:18',
'language': 'swe'},
'time_base': '1/1000',
'width': 1920}]}
DEBUG:root:LibraryOptimiser - [FFMPEGHandle] File already hevc - /library/Movies/8 Mile (2002).mkv
INFO:root:LibraryOptimiser - [LibraryScanner] File does not need to be processed - /library/Movies/8 Mile (2002).mkv
ERROR:tornado.application:Uncaught exception GET / (192.168.1.130)

HTTPServerRequest(protocol='http', host='192.168.1.131:8888', method='GET', uri='/', version='HTTP/1.1', remote_ip='192.168.1.130')

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/tornado/web.py", line 1590, in _execute
result = method(*self.path_args, **self.path_kwargs)
File "/app/webserver/main.py", line 51, in get
self.render("main.html", time_now=time.time())
File "/usr/lib/python3.6/site-packages/tornado/web.py", line 766, in render
html = self.render_string(template_name, **kwargs)
File "/usr/lib/python3.6/site-packages/tornado/web.py", line 907, in render_string
return t.generate(**namespace)
File "/usr/lib/python3.6/site-packages/tornado/template.py", line 346, in generate
return execute()
File "main_html.generated.py", line 120, in _tt_execute
if isinstance(_tt_tmp, _tt_string_types): _tt_tmp = _tt_utf8(_tt_tmp) # main-completed-tasks.html:20 (via main.html:216, page_layout.html:123)
File "/usr/lib/python3.6/site-packages/tornado/escape.py", line 199, in utf8
return value.encode("utf-8")
UnicodeEncodeError: 'utf-8' codec can't encode characters in position 35-37: surrogates not allowed
ERROR:tornado.access:500 GET / (192.168.1.130) 422.43ms

 

Link to comment

@trekkiedj can you please try from your unraid terminal copying the file `/mnt/user/appdata/unmanic/.unmanic/config/history.json`
(This was the default location. If you have changed your appdata folder it may be different on your machine)

From the looks of things there is something in your file history that is causing a problem. If you could give me this file it may help me figure it out. Also once you have a copy of it for me to debug, deleting it may be enough to set you right again so you can access the webUI

Link to comment
2 minutes ago, Josh.5 said:

@trekkiedj can you please try from your unraid terminal copying the file `/mnt/user/appdata/unmanic/.unmanic/config/history.json`
(This was the default location. If you have changed your appdata folder it may be different on your machine)

From the looks of things there is something in your file history that is causing a problem. If you could give me this file it may help me figure it out. Also once you have a copy of it for me to debug, deleting it may be enough to set you right again so you can access the webUI

Here it is

history.json

Link to comment
1 minute ago, BomB191 said:

have you set up the web ui link?

The log that he is showing is from the webUI. It is reading some non unicode character from his history log file and falling over. If I can see what that character is I may be able to figure out why it cannot convert it to a string.

Link to comment
4 minutes ago, Josh.5 said:

@trekkiedj can you please try from your unraid terminal copying the file `/mnt/user/appdata/unmanic/.unmanic/config/history.json`
(This was the default location. If you have changed your appdata folder it may be different on your machine)

From the looks of things there is something in your file history that is causing a problem. If you could give me this file it may help me figure it out. Also once you have a copy of it for me to debug, deleting it may be enough to set you right again so you can access the webUI

And deleting it fixed the issue, thanks Josh!

Link to comment
12 minutes ago, trekkiedj said:

And deleting it fixed the issue, thanks Josh!

What are the characters of this file:
 

Storage Wars - S12E09 - Let's Give \udce2\udc80\udc98Em Something to Tonka About - WEBDL-720p - h264 AAC.mkv

I have a funny feeling that this is not longer a problem and that somehow the file's names managed to make it through the conversion process and written to this history in this messed up state.

The changes made on the last update should prevent this from happening again. So if it does happen again please report it to me. I will see if I can come up with a tidy way to gracefully handle this error in the webUI so it does not error out quite so hard...

Edited by Josh.5
More info
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.