[Support] Josh5 - Unmanic - Library Optimiser

[alturismo]

1 hour ago, enJOyIT said:

May you can share your settings? That would be great 🙂

im not using unmanic anymore, just tested it a little but it didnt fit my personal preferences as im also encoding interlaced (1080i) sources and use autocropping (1080 > 960, ...) and unmanic is a nice fire & forget tool, but was a bit limited for me.

 

in terms you want to use the advanced settings and play a little, here are my last nvenc settings which where quite ok to me (in the end i stayed on qsv which differ of course ...)

 

you  have to try a little where goes what in unmanic, suggestion, use some test setup to start with before you encode your whole library ...

 

i would try with medium preset and look if thats fine for you ... if not, you can try slow, not again ... manual work through

while medium is a very common setting for most users out there ...

[DJ-BrianC]

I've been playing around with this to optimize my library and I've noticed it will re-add some of the same movies that it already optimized the next time it scans the library. Any clues why?

[alturismo]

6 hours ago, DJ-BrianC said:

I've been playing around with this to optimize my library and I've noticed it will re-add some of the same movies that it already optimized the next time it scans the library. Any clues why?

im not using unmanic anymore, but may take a look at your logs, there should be some hints why ...

 

either format, ... whatever plugins you are using in unmanic to manage your library, convert to hevc, subs, audio, ... 

[DJ-BrianC]

Where do I find them? 

[casperse]

Hi All

 

I have tried installing Unmanic 10 times and with different versions and for some reason Library scan seems to be a problem?

Managed to get it working for a short time on version 0.1.4 (Looks like this is the one all the youtube videoes are made from?)

 

Anyway some questions - Have any of you mapped a drive from UAD with Read/write slave option?

In the log I can see it scans the files but the are not shown on the UI for processing?-New UnmanicLogs.zip

 

(182) Discord | "File scan not working in latest but 0.1.4 works?" | Josh.5's Applications

https://discord.com/channels/819327740279914516/1096354979628986409

 

[smartkid808]

I noticed anything done on my remote worker (CPU not GPU), using the File Size Metric Panel, that the remote file gets added, but the increase in individual file size both show the new size, not orig and new.  But if it completes on the servers worker, it looks fine.  Is there a setting I have wrong somewhere?  Also compression seems to be doing way more than expected (dont really care on quality for these movies).  They are for my mom (Halmark Movies), is this size reduction normal?

 

Thanks!

 

[zarkoff]

Hello, 

I just would like to ensure my "recipe" for re-encoding looks good for you ?

 

My goal is to reencode everything that is not HEVC to HEVC (H265), with AAC audio. I want to keep all audio tracks and all subtitles.

 

in Unraid Docker template: no restriction on CPU, RAM or GPU but I put " --device=/dev/dri " in extra parameters). 

My cpu is i7-6700 (iGPU Intel HD530)  with 16go ram

 

Steps are:

• Remux Video Files  (ouput container .mkv)
• Video Encoder H265/HEVC - hevc_vaapi (INTEL/AMD)   (Enable VAAPI HW Accelerated Decoding is "Disabled", Buffer 2048)
• Audio Encoder AAC (Buffer 2048)
• Reject File if Larger than Original
• File Size Metrics Data Panel

It works well, except for old AVI files that fails, and some MKV for which I've the following message ;

"Filtergraph 'format=nv12|vaapi,hwupload' was defined for video output stream 0:2 but codec copy was selected.

Filtering and streamcopy cannot be used together."

 

Do you know what I could change to make it works perfectly whatever the file is.

[IllMethods]

On 11/16/2021 at 11:26 AM, Josh.5 said:

Well if you can give me a detailed run-down on a collection of FFmpeg commands for transcoding to AV1 with various bitrates (VBR or CBR), etc. Then I can do the python work for you...

 

I looked at the docs for AV1 the other day and after about an hour I gave up.

What about now? Av1 is now easily done with handbrake, not sure about ffmpeg but I would assume so. It has also been not far off a couple years since you said this. I love your app but I'm currently waiting on something to have a decent Av1 encoder plugin, especially one that uses Intel quick sync. I have a A770 Intel Arc, so I'm wanting to take advantage of its Av1 hardware accelerated encoder. I know a lot of people out there want to take advantage of this, it would be a serious space saver for all of us.

[alturismo]

3 hours ago, IllMethods said:

I love your app but I'm currently waiting on something to have a decent Av1 encoder plugin

when you take a look at page 1, may follow the post and join the discord channel as he is more active there

 

and about AV1 encoding, can all your clients playback AV1 nativ ? or do you really want to encode then all streams while playback then ? the benefit is currently mayby only to gain some space advantage, but when your clients cant playback them (AV1 decode) your Server (Plex, Emby, ...) will always transcode them ... most likely then to h264 even ... you are aware of this ?

 

also, there is a github to may wish plugins

 

https://github.com/Unmanic/unmanic-plugins/issues

[NichollsGlen]

On 10/3/2019 at 2:46 PM, JPDom1 said:

Been experiencing random Kernal panics not knowing what is causing it. Finally tailed my syslog to get a capture when it crashed and got it.

I think this might be related to this container not 100% sure as i have also recently switched to a new system.

 

Crash attached below and it is the "FFMPEG" that gets my attention.

Although this crash is different to my others currently running a memtest will let it run over night.

System was up for 10 minutes before it crashed and my unmatic container is configured to scan every 5 minutes so this happened just after the scan.

 

TR4 1950X

64GB DDR4 Kingston 3000MHZ

Asus X399 Aorus Pro

P2000

Dell H310 x2 It mode

Lots of disks

 

Oct  3 21:34:04 Mufasa kernel: traps: ffmpeg[31800] general protection ip:152919c2051e sp:1529128aba60 err                                                                                                          or:0 in libx265.so.169[1529196fd000+533000]
Oct  3 21:34:04 Mufasa kernel: BUG: unable to handle kernel paging request at 0000000000003bc0
Oct  3 21:34:04 Mufasa kernel: PGD 0 P4D 0
Oct  3 21:34:04 Mufasa kernel: Oops: 0000 [#1] SMP NOPTI
Oct  3 21:34:04 Mufasa kernel: CPU: 8 PID: 31779 Comm: ffmpeg Tainted: P           O      4.18.20-unRAID #                                                                                                          1
Oct  3 21:34:04 Mufasa kernel: Hardware name: Gigabyte Technology Co., Ltd. X399 AORUS PRO/X399 AORUS PRO-                                                                                                          CF, BIOS F1 10/01/2018
Oct  3 21:34:04 Mufasa kernel: RIP: 0010:__dec_node_state+0x0/0x53
Oct  3 21:34:04 Mufasa kernel: Code: 84 e0 81 65 44 88 01 c3 48 8b 3f 48 89 f8 48 c1 ef 38 83 e7 03 48 c1                                                                                                           e8 3a 48 69 ff 80 06 00 00 48 03 3c c5 00 65 ea 81 eb 8d <48> 8b 97 c0 3b 00 00 89 f6 b0 ff 48 8d 4c 32 01                                                                                                           65 0f c0 01 65 44
Oct  3 21:34:04 Mufasa kernel: RSP: 0018:ffffc900273e7b70 EFLAGS: 00010213
Oct  3 21:34:04 Mufasa kernel: RAX: 0000000000000037 RBX: ffffea001aa1f940 RCX: 000000001aa1f940
Oct  3 21:34:04 Mufasa kernel: RDX: 22ffea001aa1f908 RSI: 000000000000000c RDI: 0000000000000000
Oct  3 21:34:04 Mufasa kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8806f6fb9380
Oct  3 21:34:04 Mufasa kernel: R10: ffff88107f2fc000 R11: ffff88103568afee R12: ffff8806f5492518
Oct  3 21:34:04 Mufasa kernel: R13: 80000006a87e5067 R14: ffffc900273e7c98 R15: ffffea001aa1f940
Oct  3 21:34:04 Mufasa kernel: FS:  0000152912b8bb20(0000) GS:ffff88103ca00000(0000) knlGS:000000000000000                                                                                                          0
Oct  3 21:34:04 Mufasa kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct  3 21:34:04 Mufasa kernel: CR2: 0000000000003bc0 CR3: 0000000001e0a000 CR4: 00000000003406e0
Oct  3 21:34:04 Mufasa kernel: Call Trace:
Oct  3 21:34:04 Mufasa kernel: page_remove_rmap+0x24d/0x29c
Oct  3 21:34:04 Mufasa kernel: unmap_page_range+0x5da/0x88a
Oct  3 21:34:04 Mufasa kernel: ? __switch_to_asm+0x34/0x70
Oct  3 21:34:04 Mufasa kernel: unmap_vmas+0x4b/0x7f
Oct  3 21:34:04 Mufasa kernel: exit_mmap+0xc8/0x16a
Oct  3 21:34:04 Mufasa kernel: mmput+0x4d/0xe5
Oct  3 21:34:04 Mufasa kernel: traps: ffmpeg[32284] general protection ip:14e28c68b73e sp:14e28575a4c0 err                                                                                                          or:0
Oct  3 21:34:04 Mufasa kernel: traps: ffmpeg[31226] trap invalid opcode ip:147bd24c0d55 sp:147bcb790428 er                                                                                                          ror:0 in libx265.so.169[147bd236d000+533000]
Oct  3 21:34:04 Mufasa kernel: do_exit+0x3b0/0x8b0
Oct  3 21:34:04 Mufasa kernel: do_group_exit+0x9a/0x9a
Oct  3 21:34:04 Mufasa kernel: get_signal+0x417/0x44c
Oct  3 21:34:04 Mufasa kernel: in libx265.so.169[14e28c637000+533000]
Oct  3 21:34:04 Mufasa kernel: ? __switch_to_asm+0x40/0x70
Oct  3 21:34:04 Mufasa kernel: do_signal+0x31/0x59d
Oct  3 21:34:04 Mufasa kernel: ? __switch_to_asm+0x40/0x70
Oct  3 21:34:04 Mufasa kernel: ? __switch_to_asm+0x34/0x70
Oct  3 21:34:04 Mufasa kernel: ? __switch_to_asm+0x40/0x70
Oct  3 21:34:04 Mufasa kernel: ? __switch_to_asm+0x40/0x70
Oct  3 21:34:04 Mufasa kernel: exit_to_usermode_loop+0x25/0x96
Oct  3 21:34:04 Mufasa kernel: prepare_exit_to_usermode+0x66/0x6d
Oct  3 21:34:04 Mufasa kernel: retint_user+0x8/0x8
Oct  3 21:34:04 Mufasa kernel: RIP: 0033:0x1529197cfa64
Oct  3 21:34:04 Mufasa kernel: Code: Bad RIP value.
Oct  3 21:34:04 Mufasa kernel: RSP: 002b:0000152912b86cc8 EFLAGS: 00000246 ORIG_RAX: ffffffffffffff02
Oct  3 21:34:04 Mufasa kernel: RAX: 0000000000000008 RBX: 0000152912b86e80 RCX: 0000000000000480
Oct  3 21:34:04 Mufasa kernel: RDX: 00001528ff35ff78 RSI: 0000000000000008 RDI: 0000152912b86e80
Oct  3 21:34:04 Mufasa kernel: RBP: 0000152912b86f10 R08: 0000152907d23f78 R09: 0000000000000480
Oct  3 21:34:04 Mufasa kernel: R10: 0000000000000020 R11: 0000000000000001 R12: 0000000000000008
Oct  3 21:34:04 Mufasa kernel: R13: 000055ac0aeb6b98 R14: 0000152912b87100 R15: 000055ac0b5a0048
Oct  3 21:34:04 Mufasa kernel: Modules linked in: macvlan nvidia_uvm(O) xt_CHECKSUM iptable_mangle ipt_REJ                                                                                                          ECT ebtable_filter ebtables ip6table_filter ip6_tables vhost_net tun vhost tap veth xt_nat ipt_MASQUERADE                                                                                                           iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 nf_nat_ipv4 iptable_filter ip_tables nf_nat xfs md_mod it87 h                                                                                                          wmon_vid bonding edac_mce_amd nvidia_drm(PO) nvidia_modeset(PO) nvidia(PO) drm_kms_helper kvm crct10dif_pc                                                                                                          lmul crc32_pclmul crc32c_intel ghash_clmulni_intel drm pcbc aesni_intel mpt3sas aes_x86_64 crypto_simd cry                                                                                                          ptd agpgart atlantic raid_class scsi_transport_sas syscopyarea nvme ahci i2c_piix4 k10temp sysfillrect glu                                                                                                          e_helper i2c_core sysimgblt ccp fb_sys_fops libahci wmi_bmof mxm_wmi nvme_core wmi button pcc_cpufreq acpi                                                                                                          _cpufreq
Oct  3 21:34:04 Mufasa kernel: CR2: 0000000000003bc0
Oct  3 21:34:04 Mufasa kernel: ---[ end trace 186181c9c6d6be3d ]---
Oct  3 21:34:04 Mufasa kernel: RIP: 0010:__dec_node_state+0x0/0x53
Oct  3 21:34:04 Mufasa kernel: Code: 84 e0 81 65 44 88 01 c3 48 8b 3f 48 89 f8 48 c1 ef 38 83 e7 03 48 c1                                                                                                           e8 3a 48 69 ff 80 06 00 00 48 03 3c c5 00 65 ea 81 eb 8d <48> 8b 97 c0 3b 00 00 89 f6 b0 ff 48 8d 4c 32 01                                                                                                           65 0f c0 01 65 44
Oct  3 21:34:04 Mufasa kernel: RSP: 0018:ffffc900273e7b70 EFLAGS: 00010213
Oct  3 21:34:04 Mufasa kernel: RAX: 0000000000000037 RBX: ffffea001aa1f940 RCX: 000000001aa1f940
Oct  3 21:34:04 Mufasa kernel: RDX: 22ffea001aa1f908 RSI: 000000000000000c RDI: 0000000000000000
Oct  3 21:34:04 Mufasa kernel: RBP: 0000000000000000 R08: 0000000000000000 R09: ffff8806f6fb9380
Oct  3 21:34:04 Mufasa kernel: R10: ffff88107f2fc000 R11: ffff88103568afee R12: ffff8806f5492518
Oct  3 21:34:04 Mufasa kernel: R13: 80000006a87e5067 R14: ffffc900273e7c98 R15: ffffea001aa1f940
Oct  3 21:34:04 Mufasa kernel: FS:  0000152912b8bb20(0000) GS:ffff88103ca00000(0000) knlGS:000000000000000                                                                                                          0
Oct  3 21:34:04 Mufasa kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Oct  3 21:34:04 Mufasa kernel: CR2: 00001529197cfa3a CR3: 0000000001e0a000 CR4: 00000000003406e0
Oct  3 21:34:04 Mufasa kernel: Fixing recursive fault but reboot is needed!

 

TIA,

@JPDom1 did you ever find a fix for this? I've been seeing very similar kernel panics every 15 or so days and not quite sure how to fix it.

[letrain]

On 9/3/2022 at 9:05 AM, Cpt. Chaz said:

From the unmanic discord faq channel:

 

**Q: I'm trying to convert my .avi files to x265, but the files don't seem to work afterwards.**
> A: The .avi file format does not support x265. To resolve this, use the remux plugin to convert .avi files specifically to .mp4 first. Once converted to .mp4, these files should be able to convert to x265 and/or .mkv format. More on this here https://discord.com/channels/819327740279914516/828703056971169822/1012350586718535710

how do i prevent remux from converting all my mkv files to mp4 first, is there a way to limit this to avi files only?

[alturismo]

2 hours ago, letrain said:

how do i prevent remux from converting all my mkv files to mp4 first, is there a way to limit this to avi files only?

im not using unmanic (anymore) but when i read this, how you come to the conclusion your mkv's need to be remuxed before transcoded ? it clearly states .avi files needs a remux 1st ...

 

and my test files always been mkv, also when i tested unmanic a while ago ...-

[blackbullitt]

Has anyone experienced where Unmanic will take a completed task and add it back to pending tasks after a library scan? I have added a few extra plugins (remove all subtitle streams, reject file if larger than original, limit library search by file extension) and ever since it will add all the media files back into pending tasks on the next days library scan that it had previously completed. When the worker processes the files, again, I see this in the scrolling log and then moves it to the completed tasks:

 

2023-08-09T11:07:34:WARNING:Unmanic.Melkree-Worker-1 - [FORMATTED] - No Plugin requested for Unmanic to run commands for this file

 

It has successfully converted my library to AAC and H265 in the past and had never added completed tasks back into pending tasks. My intentions are to strip the subtitles and use bazaar and pull external ones.  Bazaar is set to add the srt file to the existing folder the media file resides in. I don't see that being an issue, although I see when it runs a test scan it does pick up the existence of the srt files added to the same folder the media resides in.

Edited August 9, 2023 by blackbullitt

[DJ-BrianC]

I was having that problem for the longest time and nobody could give me a straight answer. What I eventually did was add the remux plugin and I've yet to see that happen unless which is fairly rare the file has some kind of corruption.I was having that problem for the longest time and nobody could give me a straight answer. What I eventually did was add the remux plugin and I've yet to see that happen unless which is fairly rare the file has some kind of corruption.

[blackbullitt]

I have had the remux plugin, so that wasn't the case for me. I systematically removed the new plugins one at a time and rescanned the library and watched each time to see if the pending tasks number went up.  It seems that the ignore file type by extension was causing me the heartache. Since all the new plugins were removed I turned on cancel all pending tasks on restart, restated Unmanic,  and rescanned the library and no pending tasks came up, which is how it was prior to this. I have added back in only the remove all subtitles plugin to the plugin flow and reran the library scan. I will have to wait and see how many pending tasks I have and take note of it and check it, again, on the next library scan to compare numbers. I'll keep everyone posted.

 

***Update***

My issues seems to be resolved. I'm leery of adding back in the ignore file types by extension.

Edited August 10, 2023 by blackbullitt

[DaMAN]

Ive noticed when encoding its really slow less then 1x (nvidia h265 defaults, 4 workers) when the content is on a standard spinning hard drive, if the content is on a ssd its much much faster, Im using a ram disc(64gb/32gb available) any ideas how to encode faster?

Edited August 15, 2023 by DaMAN
speelling

[scubanarc]

On 9/30/2021 at 10:04 AM, Valyth said:

Hi,

 

Amazing work on the latest version - I love being able to pick and choose plugins and options, and I think unmanic just went from a "nice to have" app to a "must have" app in my mind.

 

I am still seeing errors processing files quite occasionally - specifically, I see the following:

 

Filtergraph 'format=nv12|vaapi,hwupload' was defined for video output stream 0:0 but codec copy was selected.

Filtering and streamcopy cannot be used together.

 

Is there an issue with my configuration that could cause this, or something I can do to mitigate it?

 

Thanks in advance.

I had this exact same issue, and I think I found the problem and the fix. In my case, this was happening on files that had embedded movie posters as a stream. It was usually stream #3 (after subs).

 

These streams seem to force a filter command on ffmpeg, but unmanic is requesting a streamcopy, so you get the error above.

 

The simplest solution is to add the plugin called "Strip all image streams from file" in your stack, above the encode. As soon as I did that all of my encodes worked fine.

[Exciter]

On 8/9/2023 at 7:14 AM, blackbullitt said:

Has anyone experienced where Unmanic will take a completed task and add it back to pending tasks after a library scan? I have added a few extra plugins (remove all subtitle streams, reject file if larger than original, limit library search by file extension) and ever since it will add all the media files back into pending tasks on the next days library scan that it had previously completed. When the worker processes the files, again, I see this in the scrolling log and then moves it to the completed tasks:

 

2023-08-09T11:07:34:WARNING:Unmanic.Melkree-Worker-1 - [FORMATTED] - No Plugin requested for Unmanic to run commands for this file

 

It has successfully converted my library to AAC and H265 in the past and had never added completed tasks back into pending tasks. My intentions are to strip the subtitles and use bazaar and pull external ones.  Bazaar is set to add the srt file to the existing folder the media file resides in. I don't see that being an issue, although I see when it runs a test scan it does pick up the existence of the srt files added to the same folder the media resides in.

 

My problem was putting what I put in for the temp folder in the unmanic settings. Instead of putting /tmp/unmanic

[SomeoneOnLine]

So I am somewhat new to Unmatic. Coming from a somewhat new but basic understanding of Tdarr on unraid to convert my movie library to x265. I am evaluating Unmatic and even setup a patreon account to donate a little somethin something for the dev efforts.

However I am able to get it to work and its already converting some movies just fine (re-encoded about 30 or so movies already) it looks like when I click on the Datapanel (after I installed the plugin) I get an error popup. Anyone know what this means? Im running on unraid 6.12.4

 

At the end of the day I am looking to see how much actual space is saved etc.. after conversion of movies etc..

 

Unmatic Version: 0.2.4~7339f8b from Unraid plugin store

 

Edited September 19, 2023 by SomeoneOnLine
add unmatic version

[ConnerVT]

Josh doesn't visit here much, it ever, anymore.  You may be better off asking in the Unmanic Discord or Patreon.

[SomeoneOnLine]

ok. thanks. 

[GeorgeJetson20]

I know Josh is not here much anymore and he said to use filebot to rename files after converting but I am just wondering if anyone else was able to get this accomplished?  I used all the instructions on this in the unmanic document but it just did not work at all   I waited for the processing of startup.sh (about 20 mins) but when just doing a simple command to take the output and add like his example, it didn't do anything.  It did make my docker grow from 13-15 though.

 

Here is the log message but I have no idea what to do or what is wrong

 

2023-09-25T20:53:53:ERROR:Unmanic.PluginExecutor - [FORMATTED] - Exception while carrying out 'postprocessor.task_result' plugin runner 'postprocessor_script'
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/unmanic/libs/unplugins/executor.py", line 256, in execute_plugin_runner
    runner(data)
  File "/config/.unmanic/plugins/postprocessor_script/plugin.py", line 378, in on_postprocessor_task_results
    exec_subprocess(cmd, args)
  File "/config/.unmanic/plugins/postprocessor_script/plugin.py", line 181, in exec_subprocess
    raise Exception("Failed to execute command: '{}'".format(full_command))
Exception: Failed to execute command: 'filebot --conflict skip -non-strict -rename "/library/videos/testfile.mp4" --format "{source} {vf}-{vc}"  '
2023-09-25T20:53:55:INFO:Unmanic.PostProcessor - [FORMATTED] - Removing task cache directory '/tmp/unmanic/unmanic_file_conversion-onznv-1695689488'

 

 

If anyone else here got this to work, can you please post some notes on how you got it and if there was anything special needed outside of the document.

 

The goal is just append to the filename the following "HEVC" so i know it was converted.  Once I get this part working, i can play around more but I can't get this to work.

Edited September 26, 2023 by GeorgeJetson20

[blackbullitt]

Anybody having an issue with Unmanic taking a full movie that plays the full length before transcoding and then the transcoding chops it down to 20 minutes or less? TV shows seem to be fine, it's only movies.

 

***Edit***

I think I found the cause of the issue. As the file is being moved to the array, unmanic detects it and starts transcoding before the file has completely moved over to the array.

Edited October 6, 2023 by blackbullitt

[TheDirtyR]

On 9/10/2021 at 11:18 PM, guest_user said:

If you are using 'josh5/unmanic:staging' add the 'File Size Metrics Data Panel' plugin and it will do just that

Mine is just blank am I missing a hidden setting.

 

[Lignage]

Hi, sorry for my English google translated. Maybe someone can help me, I can't get back from the directory. What do I have to do to get back? Another question, I have Unmanic running on Unraid and installed it on my Synology. Can I set the Synology films to be processed on the Unraid?

Edited December 8, 2023 by Lignage