[Support] Josh5 - Unmanic - Library Optimiser


Recommended Posts

Thanks for the info on where to search:

 

These are my 2 errors:

 

 

set_mempolicy: Operation not permitted

 

and

 

Too many packets buffered for output stream 0:1.

 

I have 256GB of RAM and was doing a packet buffer of 10k. I reduced that to 4096. See if that helps alleviate the packet buffer error.

Unsure why the mempolicy is occuring.

Link to comment

So strange things are happening now; the Docker is now encoding files that it has previously encoded.  I have two TV shows in the watch folder and even though they have already been encoded, the app is encoding them again to HEVC.  This is quite puzzling.

 

Ok, I am an idiot.  I had both plugins installed and running:

  • Video Encoder H264 - h264_nvenc (NVIDIA GPU)
  • Video Encoder H265/HEVC - hevc_nvenc (NVIDIA GPU)

This is not the containers fault but, my own.  Initially upon configuring everything I was in such a rush to get it back up and running I wasn't paying attention and just installed everything with NVIDIA GPU in it.  So it seems as though I put the container into a vicious cycle, converting the same files to x264 and then x265.

 

This would also explain why my RAM filled to capacity yesterday and my Dual Xeons spiked to 100% on all cores.

 

I am leaving this post up for posterity in case anyone else has this issue.  Check the plugins! lol

Link to comment
2 minutes ago, CoZ said:

I put the container into a vicious cycle, converting the same files to x264 and then x265.

Such is the double edged sword of open sandbox plugin workflow.

 

It's possible to rewrite the code to warn when conflicting processes are set up, but IMHO it's better to be sure the plugins have clearly written descriptions and allow users to set things up as they wish. This new system is so much more capable, it's a whole different application compared to previous versions.

 

The only suggestion I would make is that each plugin type in the workflow have a highlight, possibly a unique color to allow at a glance to see that you have only what you need of

 

File selection and filtering

Video processing

Audio processing

Post Processed file management

 

That way if you have multiple video or multiple audio processors defined, it sticks out as something that may need attention.

Link to comment
3 hours ago, guest_user said:

Not sure, I just use the default with hardware x265 encoding and t has been rock solid..

 

image.thumb.png.61b528b717ac99b460b70a117de3ab6f.png


You are using Hardware encoding, I'm using CPU encoding, big difference in performance with the two models. 

I'm getting constant errors when writing MP4->MKV file conversion:
 

[quote]

Too many packets buffered for output stream 0:1.

x265 [info]: frame I: 25, Avg QP:18.95 kb/s: 5938.62

x265 [info]: frame P: 1018, Avg QP:20.10 kb/s: 4848.96

x265 [info]: frame B: 2053, Avg QP:23.89 kb/s: 1874.06

x265 [info]: Weighted P-Frames: Y:3.8% UV:3.3%

x265 [info]: consecutive B-frames: 16.8% 16.2% 25.8% 35.3% 5.9%

encoded 3096 frames in 3442.70s (0.90 fps), 2885.06 kb/s, Avg QP:22.60

Conversion failed!
[/quote]

 

I've reduced the buffer down to 2048 and still get this same issue. Why?

Link to comment
2 hours ago, CyrixDX4 said:


You are using Hardware encoding, I'm using CPU encoding, big difference in performance with the two models. 

I'm getting constant errors when writing MP4->MKV file conversion:
 

[quote]

Too many packets buffered for output stream 0:1.

x265 [info]: frame I: 25, Avg QP:18.95 kb/s: 5938.62

x265 [info]: frame P: 1018, Avg QP:20.10 kb/s: 4848.96

x265 [info]: frame B: 2053, Avg QP:23.89 kb/s: 1874.06

x265 [info]: Weighted P-Frames: Y:3.8% UV:3.3%

x265 [info]: consecutive B-frames: 16.8% 16.2% 25.8% 35.3% 5.9%

encoded 3096 frames in 3442.70s (0.90 fps), 2885.06 kb/s, Avg QP:22.60

Conversion failed!
[/quote]

 

I've reduced the buffer down to 2048 and still get this same issue. Why?

im getting this same issue(while CPU encoding). far more failed encoder jobs. but most (failed or successful) jobs have this error about 40 times each. 

 

set_mempolicy: Operation not permitted

  • Like 1
Link to comment

For people posting issues with failed FFmpeg commands, I want to point out that there is more to it than the last few lines containing the error. I would suggest posting complete logs.
Posting partial logs is your way of self diagnosis. If you intend on solving your own problems, then that's fine. But if you want help from someone else, this is counter productive.

Post full logs and you will get a faster resolution to your problem.

Link to comment


This is the log for a file in question:

  "A Chinese Ghost Story [1080p].mp4",
  "A Chinese Ghost Story [1080p].nfo"
]
2021-09-22T00:07:38:DEBUG:Unmanic.Plugin.encoder_video_hevc_libx265 - File '/library/movies/The Vikings (1958)/The Vikings [720p].mkv' does not contain streams require processing.
2021-09-22T00:07:38:DEBUG:Unmanic.LibraryScannerManager - [FORMATTED] - [
  "A Chinese Ghost Story [1080p].mp4",
  "A Chinese Ghost Story [1080p].nfo"
]
2021-09-22T00:07:39:DEBUG:Unmanic.Worker-W2 - [FORMATTED] - Exception while parsing command progress - time data '0' does not match format '%H:%M:%S.%f'
2021-09-22T00:07:39:DEBUG:Unmanic.Worker-W0 - [FORMATTED] - Exception while parsing command progress - time data '0' does not match format '%H:%M:%S.%f'
2021-09-22T00:07:39:DEBUG:Unmanic.Worker-W3 - [FORMATTED] - Exception while parsing command progress - time data '0' does not match format '%H:%M:%S.%f'
2021-09-22T00:07:39:DEBUG:Unmanic.Worker-W1 - [FORMATTED] - Exception while parsing command progress - time data '0' does not match format '%H:%M:%S.%f'
2021-09-22T00:07:39:DEBUG:Unmanic.Plugin.video_remuxer - File '/library/movies/A Chinese Ghost Story (1987)/A Chinese Ghost Story [1080p].mp4' should be added to task list. Probe found file needs to be processed.
2021-09-22T00:07:39:DEBUG:Unmanic.Plugin.video_remuxer - Unable to fetch file MIME type - '/library/movies/A Chinese Ghost Story (1987)/A Chinese Ghost Story [1080p].nfo'
2021-09-22T00:07:39:DEBUG:Unmanic.Plugin.reorder_subtitle_streams_by_language - Unable to fetch file MIME type - '/library/movies/A Chinese Ghost Story (1987)/A Chinese Ghost Story [1080p].nfo'
2021-09-22T00:07:39:DEBUG:Unmanic.Plugin.encoder_video_hevc_libx265 - Unable to fetch file MIME type - '/library/movies/A Chinese Ghost Story (1987)/A Chinese Ghost Story [1080p].nfo'
2021-09-22T00:07:39:DEBUG:Unmanic.LibraryScannerManager - [FORMATTED] - [
  "A Chinese Ghost Story II [1080p].mp4"
]
2021-09-22T00:07:39:INFO:Unmanic.TaskHandler - [FORMATTED] - Skipping file as it is already in the queue - /library/movies/A Chinese Ghost Story (1987)/A Chinese Ghost Story [1080p].mp4
2021-09-22T00:07:39:DEBUG:Unmanic.Plugin.video_remuxer - File '/library/movies/A Chinese Ghost Story (1987)/A Chinese Ghost Story [1080p].mp4' should be added to task list. Probe found file needs to be processed.
2021-09-22T00:07:39:DEBUG:Unmanic.Plugin.video_remuxer - Unable to fetch file MIME type - '/library/movies/A Chinese Ghost Story (1987)/A Chinese Ghost Story [1080p].nfo'
2021-09-22T00:07:39:DEBUG:Unmanic.Plugin.reorder_subtitle_streams_by_language - Unable to fetch file MIME type - '/library/movies/A Chinese Ghost Story (1987)/A Chinese Ghost Story [1080p].nfo'
2021-09-22T00:07:39:DEBUG:Unmanic.Plugin.encoder_video_hevc_libx265 - Unable to fetch file MIME type - '/library/movies/A Chinese Ghost Story (1987)/A Chinese Ghost Story [1080p].nfo'
2021-09-22T00:07:39:DEBUG:Unmanic.Worker-W3 - [FORMATTED] - Exception while parsing command progress - time data '0' does not match format '%H:%M:%S.%f'
2021-09-22T00:07:39:DEBUG:Unmanic.LibraryScannerManager - [FORMATTED] - [
  "A Chinese Ghost Story II [1080p].mp4"
]
2021-09-22T00:07:39:DEBUG:Unmanic.Worker-W1 - [FORMATTED] - Exception while parsing command progress - time data '0' does not match format '%H:%M:%S.%f'
2021-09-22T00:07:39:DEBUG:Unmanic.Worker-W2 - [FORMATTED] - Exception while parsing command progress - time data '0' does not match format '%H:%M:%S.%f'
2021-09-22T00:07:39:INFO:Unmanic.TaskHandler - [FORMATTED] - Skipping file as it is already in the queue - /library/movies/A Chinese Ghost Story (1987)/A Chinese Ghost Story [1080p].mp4
2021-09-22T00:07:39:DEBUG:Unmanic.Plugin.video_remuxer - File '/library/movies/A Chinese Ghost Story II (1990)/A Chinese Ghost Story II [1080p].mp4' should be added to task list. Probe found file needs to be processed.
2021-09-22T00:07:39:DEBUG:Unmanic.LibraryScannerManager - [FORMATTED] - [
  "A Chinese Ghost Story III [1080p].mp4"
]
2021-09-22T00:07:40:INFO:Unmanic.TaskHandler - [FORMATTED] - Skipping file as it is already in the queue - /library/movies/A Chinese Ghost Story II (1990)/A Chinese Ghost Story II [1080p].mp4
2021-09-22T00:07:40:DEBUG:Unmanic.Worker-W0 - [FORMATTED] - Exception while parsing command progress - time data '0' does not match format '%H:%M:%S.%f'
2021-09-22T00:07:40:DEBUG:Unmanic.Plugin.video_remuxer - File '/library/movies/
Can you post just the FFmpeg command log of the failed task
Link to comment
25 minutes ago, CyrixDX4 said:

 

sorry got confused on the logs wanted. 

ffmpg_error_chinese_story.txt 261.45 kB · 0 downloads

So the cause of the failure is the PNG video stream.

I will see if I can come up with an improvement to the video encoder plugin, but if you want a quick fix I would suggest installing the "Strip all image streams from file" Plugin and putting that first in the worker flow.

This will remove the png stream from your file.

Link to comment
35 minutes ago, Josh.5 said:

So the cause of the failure is the PNG video stream.

I will see if I can come up with an improvement to the video encoder plugin, but if you want a quick fix I would suggest installing the "Strip all image streams from file" Plugin and putting that first in the worker flow.

This will remove the png stream from your file.

 

eh that will break lots of subtitle files again and I have a mountain of movies with baked in subtitles.   Do you need me to reraise the ticket for this as you had closed it out recently.

Link to comment
 
eh that will break lots of subtitle files again and I have a mountain of movies with baked in subtitles.   Do you need me to reraise the ticket for this as you had closed it out recently.
This wouldn't touch your subtitles. This only removes the PNG thumbnail image in the video container.
Link to comment

I was playing with the mover plugin. Seems like the converted file is being copied from /tmp to both source location as to the mover location. Then both old and new file get removed from the source location.

Seems like double the I/O / transfer. 

https://pastebin.com/yjdc41dj

And the folder in which the source was located is not removed. Would be nice to have.

Edited by EdwinZelf
typo
Link to comment
1 hour ago, EdwinZelf said:

I was playing with the mover plugin. Seems like the converted file is being copied from /tmp to both source location as to the mover location. Then both old and new file get removed from the source location.

Seems like double the I/O / transfer. 

https://pastebin.com/yjdc41dj

And the folder in which the source was located is not removed. Would be nice to have.

Seems like you have configured the mover plugin to "Remove source files". Is that correct? Could you tell me how you have configured your "File Movement" plugins flow?

 

I took a look at your log. I am not sure why the file was copied twice to '/library/movies/' ... That does not seem right. It should have been copied there only once.

 

As for the removal of that file after the copy, I think you are right. I will rewrite the default Unmanic post-processor for file moments to overwrite the source such that if the file is flagged for removal, then not to bother.

 

Cheers

 

 

EDIT:

I've updated the Unmanic staging branch with the aforementioned fix for the multiple file copies when removing the source. Would you be able to switch to the staging docker tag and test it for me?
I've also updated the mover plugin. You will need to go to the plugin installer and pull that update (v0.0.3)

 

Thanks

Edited by Josh.5
Updated unmanic and plugin
Link to comment
14 minutes ago, Josh.5 said:

Seems like you have configured the mover plugin to "Remove source files". Is that correct? Could you tell me how you have configured your "File Movement" plugins flow?

 

I took a look at your log. I am not sure why the file was copied twice to '/library/movies/' ... That does not seem right. It should have been copied there only once.

I'm still scared to let unmanic loose on my complete media folder. So I'm playing around and selecting manually what to let unmake have a go at.

 

I manually move movie folder to: /mnt/user/media/unmanic/movie/ this is mounted to /library/movies.

I added an extra mount mount point in the docker to /mnt/user/media mounted to /medialib.

Last step I added because I'm happy with the results so far, and instead of manually move back converted folders, I gave 'Mover v2' a go.

 

Should the mover plug in not make sure the converted file is moved only once from /tmp to the destination location? not copying to both /medialib/Movies/ and to /library/Movies (which from the log it seems to do twice)

 

I have turned on the "Remove source files". So all the copies from /tmp to /library/Movies are unnecessary as they will be removed. 

 

2086735249_CleanShot2021-09-24at14_23_15.png.0019b2e8d953b3c5533f3caf7d73dfca.png

 

PS, just giving feedback on the plugin, when I let unmanic have access to my main media folder, I won't be using the mover plugin.

 

 

 

Edited by EdwinZelf
added PS
Link to comment
1 minute ago, EdwinZelf said:

I'm still scared to let unmanic loose on my complete media folder. So I'm playing around and selecting manually what to let unmake have a go at.

 

I manually move movie folder to: /mnt/user/media/unmanic/movie/ this is mounted to /library/movies.

I added an extra mount mount point in the docker to /mnt/user/media mounted to /medialib.

Last step I added because I'm happy with the results so far, and instead of manually move back converted folders, I gave 'Mover v2' a go.

 

Should the mover plug in not make sure the converted file is moved only once from /tmp to the destination location? not copying to both /medialib/Movies/ and to /library/Movies (which from the log it seems to do twice)

 

I have turned on the "Remove source files". So all the copies from /tmp to /library/Movies are unnecessary as they will be removed. 

 

2086735249_CleanShot2021-09-24at14_23_15.png.0019b2e8d953b3c5533f3caf7d73dfca.png

Have a go and pulling the staging branch like I mentioned in my last post and let me know how you get on.

  • Like 1
Link to comment
seems GitHub had an issue with building
staging
1284124483_CleanShot2021-09-24at14_49_03.png.b119bfb247b027ae57376ba3ed96e5e3.png
Ha! There is always something. Am I right?!

Looks like there is an issue with docker hub. I've re- triggered the build/push. If it fails again I'll try again in the morning.

Edit:
Nvm, looks like I broke my pipeline earlier this week and didn't notice.
Sorry. It's been a long day... I'll sort it out tomorrow.
  • Thanks 1
Link to comment
19 hours ago, Josh.5 said:

So the cause of the failure is the PNG video stream.

I will see if I can come up with an improvement to the video encoder plugin, but if you want a quick fix I would suggest installing the "Strip all image streams from file" Plugin and putting that first in the worker flow.

This will remove the png stream from your file.

I assumed this issue was the same for me. I implemented the strip images plugin as you advised(first in the worker queue). I had the same issue when i re-added the job that had previously failed. log attached. 

 

Please let me know if you have any troubleshooting steps that i should take. 

Failed Job 9.24.21.txt

Link to comment

Is it still possible to create a stereo audio layer like in the previous version of unManic? I was using it before to create a stereo track from the 5.1 tracks (if stereo didn't exist) and also to compress using nvenc to 265. That's the only functionality I haven't been able to find in the new 0.1.0 version, and its currently stopping me from using it. I'd really rather not go through the hassle of learning all new software with tdarr lol

 

Thanks!

 

EDIT: I'm going to assume it is, but its going to require custom FFMPEG configuration in the AAC conversion plugin. I just don't know how to properly format a FFMPEG command to do that. I want to keep the existing 5.1 audio, and create a duplicate audio layer (that is the new primary audio for the file) that is the stereo audio.

 

EDIT 2: Appears there's already an issue on Github related to this, with a plugin request in. I'll just keep unmanic turned off until this plugin exists because thats a major reason why I was using unmanic in the first place. I wanted to save space on the file itself, and then also create a stereo audio layer that allowed me better quality playback on my streaming devices, while maintaining the 5.1 audio layer for those with better setups than me.

Edited by DaClownie
Additional info
  • Like 1
Link to comment
20 hours ago, Josh.5 said:
20 hours ago, CyrixDX4 said:
 
eh that will break lots of subtitle files again and I have a mountain of movies with baked in subtitles.   Do you need me to reraise the ticket for this as you had closed it out recently.

This wouldn't touch your subtitles. This only removes the PNG thumbnail image in the video container.

wait is THAT all it does? I thought there was some images put in to specific movies that display subtitles or other images on the screen (John Wick movie when they go into talking about Keana being Baba Yaga, etc.).

 

 

Edited by CyrixDX4
Link to comment

I've been testing this out on a new series folders and wondering before i let it loose on my entire tv folder, is it possible to exclude folders ? there's a few tv series i'd prefer to leave as is for the time being

 

Also for general tv stuff, how do people feel about H265/HEVC media, slow and lossless settings? Should also mention i'm viewing on a 120" screen

Edited by reggierat
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.