[Support] Josh5 - Unmanic - Library Optimiser


Recommended Posts

Greetings. I see that Unmanic has a new version so I decided to give it a look. Pretty slick! I have a quick question. I have added "--device=/dev/dri" to use my intel CPU in the extra parameter option. I would also like to limit CPU usage with "--cpus='.5" can I add both and if so how do I separate them?

 

Thanks!

Link to comment
12 hours ago, bclinton said:

Greetings. I see that Unmanic has a new version so I decided to give it a look. Pretty slick! I have a quick question. I have added "--device=/dev/dri" to use my intel CPU in the extra parameter option. I would also like to limit CPU usage with "--cpus='.5" can I add both and if so how do I separate them?

 

Thanks!

Mine is just separated with spaces, and seems to work fine.

Screenshot 2022-01-24 at 13.33.42.png

Link to comment

Has anyone settled on different settings for different file sizes? I currently have two unmanic containers, one that uses quicksync on files 1GB+ in size, and one that uses cpu transcoding on files 500MB-1GB.

 

It's a shame there's no option to filter by bitrate, as plenty of 720p+ content that (as I understand it) is suitable for quicksync transcoding, will be missed because the file happens to be <1GB. But I want to be a bit tentative, as the smaller files tend to be older, more difficult to find media.

Link to comment
14 hours ago, TX_Pilot said:

I keep getting "Unable to connect to Unmanic backend. Please check that it is running." Anybody knows what causes this error. My guess is maybe it is looking online for something I have pi-holed.

 

Exact same issue here but not sure if it is pi-hole. I am running unmanic on a Mac with no issues, but on separate hardware with Ubuntu I get the backend warning, and my workers keep resetting to 0. I do not have a firewall enabled on that machine so not sure if that is the issue. Have tried in both docker and python but same issue.

Link to comment

Well i'm really trying to user this plugin however it's not working for me.

 

On my gen 10 i5 the image that comes out its simply garbage, all pixelated and nothing usable, if i try to user the nvidia card pm the system i can't download the correct plug ins:

An error was encountered while attempting to install the requested Plugin.

I't been like this for more then 15 days any thoughts?

Edited by jpm
misspell
Link to comment

I'm probably doing something dumb but about 10% of the tv shows are actually increasing in file size - which also means that 90% of the time there's a significant decrease in file size, so I'm not complaining, I'd have just assumed it would automatically not process a file if the end result was to make it bigger- or revert it - but then of course I don't really understand the process. I'm just 90% very grateful and 10% puzzled :)  Any ideas ?

Ah Ok I found the reject file if larger than original plugin - which I hadn't added as I thought there was a five plugin limit, which seems to have been changed since I first set things up? 

 

1159241757_Unmanic2022-02-01at23_46_26.png.e293daf42b627794b4a2695b46f7c0b7.png

Edited by awediohead
typo
Link to comment
  • 2 weeks later...

Hey Josh,

 

using the "Video Encoder H265/HEVC - hevc_nvenc (NVIDIA GPU)" plugin and going over the options described at https://developer.nvidia.com/blog/nvidia-ffmpeg-transcoding-guide/ I was keen to try out the hwaccel_output_format option to see how much of a difference it would make. However I was met with an error from ffmpeg telling me:

 

Quote

Option hwaccel_output_format (select output format used with HW accelerated decoding) cannot be applied to output url /tmp/unmanic/unmanic_file_conversion-1644489070.2793753/file-1644489070.2793622-WORKING-2.mkv -- you are trying to apply an input option to an output file or vice versa. Move this option before the file it belongs to.

 

Looking at the command generated I can see that the main options are placed behind the input file:

 

Quote

ffmpeg -hide_banner -loglevel info -hwaccel cuda -hwaccel_device 0 -i /path/to/file.mkv -threads 2 -hwaccel_output_format cuda -strict -2 -max_muxing_queue_size 2048 -map 0:v:0 ...

 

Looking at the plugin's config description this should be in front of it right?

Link to comment

Hey Josh,

 

I seem to be having some trouble with transcoding to hevc using the "Video Encoder H265/HEVC - hevc_qsv (Intel)" plugin . It transcodes the files to hevc perfectly but  when playing them back using either vlc over the network or plex using the media player . there is significant visual corruption present. My cpu is a i5-11400 with 16 GB of ram and i am running the latest stable unraid version with the adde doption of  options i915 force_probe=4c8b in the i915.conf file . when setting the transcode plugin to cpu  the entire pluginflow works flawlessly .so i'm curious as to what the problem is. because i couldn't find anything about this reported anywhere.

 

thanks for the input

Sages

corruption.jpeg

Link to comment
On 1/12/2022 at 7:52 AM, Spritzup said:

I was experiencing the same issue, but it seems to be resolved now.

Related though, in the previous version of unmanic, you could set it to create a stereo track from an existing multi-channel track (while keeping the original).  That said, I can't find an option to do that anywhere... does anyone have any ideas?

 

Thanks!

 

~Spritz

Search the topic for "stereo" you will find the custom repo from last year. Add it and search for clone. I just found it. Might have to restart the docker, or refresh page for it come up. 

Link to comment

Just installed Unmanic for the first time just now and I'm trying to get everything setup.  For reasons I don't know there are no plugins available and "Refresh Repositories" displays a message that it completed the refresh but still nothing.  I've tried wiping appdata and deleting and reinstalling the docker container but no change.  Any advice on how to get the plugins to show up?

Link to comment
On 2/15/2022 at 1:21 AM, phil1c said:

Just installed Unmanic for the first time just now and I'm trying to get everything setup.  For reasons I don't know there are no plugins available and "Refresh Repositories" displays a message that it completed the refresh but still nothing.  I've tried wiping appdata and deleting and reinstalling the docker container but no change.  Any advice on how to get the plugins to show up?

Same issue here. Also, when I go to "Help & Support", I get an error "Failed to fetch Unmanic's system configuration."

 

EDIT: Looks like my config error is linked to the NVidia parameters. Starting without that extra param works fine as far as the config. Still no plugins though.

 

EDIT2: Turning off the nvidia extra param and refreshing the plugin store seems to do the trick. Yet to see whether I can get my GPU working, but at least the plugins and config are working.

Edited by JPDVM2014
Link to comment
On 2/12/2022 at 2:15 AM, Spritzup said:

So weird issue.  When I use the nvidia h265 plugin and select "lossless" my output file grows nearly 13x, from 2.8gb to 36gb.  If I select "slow", the file shrinks down to about 717mb.  Same file both times (as I'm testing my settings).  Everything else is default.

 

Thoughts?

Hasn't this always been the case for transcoding lossy to lossless? I mean, if you used a fancy DSLR to take a photo of an old black and white TV image, the image of the TV produced by the DSLR wouldn't be any more crisp than the real life TV image, but it would have the down side of being a much bigger file size because of the extraneous data in the image. Isn't that sort of how it works?

Link to comment

I am trying to use the Video Encoder H265/HEVC - hevc_vaapi (INTEL/AMD) plugin with interlaced video - want to either keep them interlaced (preferred) or apply de-interlacing). Per ffmpeg, the first option is "-ilme" and second option is probably "-yadif".

 

I am adding these commands under "write your own video options" but I did try putting in any of the 3 places. Always get an error "command not recognized". What am I doing wrong?

EDIT: I tried with "vf yadif" in the main options and it completed without error but no de interlacing was applied.
EDIT2: After spending 3 hours on this issue today, I am giving up. I believe x265 interlaced is tricky so de-interlacing is what I was trying to get going. Every guide I read about FFMPEG lists the -yadif/bwdif command should be inserted at the input section but this plugin doesn't allow me to insert custom commands there. Please help.

Edited by OrdinaryButt
Link to comment

I am looking for a "smart" way to stop unmanic docker after tasks have been completed or no tasks to execute shortly after start.

I think I can use the "External script" plugin and simply send "kill pid 1" which when tested thru unManic's terminal, does shut down the container. This should work BUT I am woried:

A) either will cause a start stop loop if there are no pending tasks *EDIT* - I can do "sleep 60 && kill pid 1" which would give me enough time to change config should I need to.

B) not execute(e.g. not shut down) if there are no tasks


The whole reason I am doing this is as I want it to start daily at certain time with "do one off scan on startup" enabled, which means it needs to be stop to execute again next day. Perhaps I could schedule a "docker restart" command but would be more ellegant to start/stop it.

Edited by OrdinaryButt
Link to comment
1 hour ago, OrdinaryButt said:

I am looking for a "smart" way to stop unmanic docker after tasks have been completed or no tasks to execute shortly after start.

I think I can use the "External script" plugin and simply send "kill pid 1" which when tested thru unManic's terminal, does shut down the container. This should work BUT I am woried:

A) either will cause a start stop loop if there are no pending tasks *EDIT* - I can do "sleep 60 && kill pid 1" which would give me enough time to change config should I need to.

B) not execute(e.g. not shut down) if there are no tasks


The whole reason I am doing this is as I want it to start daily at certain time with "do one off scan on startup" enabled, which means it needs to be stop to execute again next day. Perhaps I could schedule a "docker restart" command but would be more ellegant to start/stop it.

Is this in an attempt to save on resources? or to free up the port?
When idle, Unmanic should have a very small footprint. Less than 100MB RAM. You can schedule the library scanner to run daily. And you can schedule the workers to start or stop throughout the day. Would these features not be the better option over starting and stopping the container?


To answer your questions a bit... Unmanic has an API that can be queried to determine the state including the number of items in the pending tasks queue or what is currently being processed by the workers.

 

It is completely safe to kill the Unmanic process. This would be best done by running 'docker stop unmanic' (if unmanic is the name of the docker container)

Link to comment
1 hour ago, Josh.5 said:

Is this in an attempt to save on resources? or to free up the port?

Neither, just want to execute at idle times, once a day.

Quote

You can schedule the library scanner to run daily

How? The closest I am seeing is "Enable periodic library scans" which has an option to run every so many minutes. This will not guarantee  running at exact time of day or off hours and restarting unRAID will screw it up. My idea was to enable just the "run a one off scan on startup" and start the docker at midnight. But I need to stop the docker so it can run again next day. I can schedule docker stop, but will have to guess the time. Hence I thought would be smarter to have unmanic kill itself on on pending tasks complete, which I think I can do. However, if there are no new files a given day, it will never stop/start again (I think).

 

The context of all this is live TV recordings, where original files are overwritten once converted. So it can't be done when there is a potential recording. I discovered that the library file monitor will trigger it self the moment the recording starts which is not good (I know there is a plugin start delay but again, the recording could be 30 minutes or 4 hours).

Edited by OrdinaryButt
typos
Link to comment
1 hour ago, OrdinaryButt said:

Neither, just want to execute at idle times, once a day.

How? The closest I am seeing is "Enable periodic library scans" which has an option to run every so many minutes. This will not guarantee  running at exact time of day or off hours and restarting unRAID will screw it up. My idea was to enable just the "run a one off scan on startup" and start the docker at midnight. But I need to stop the docker so it can run again next day. I can schedule docker stop, but will have to guess the time. Hence I thought would be smarter to have unmanic kill itself on on pending tasks complete, which I think I can do. However, if there are no new files a given day, it will never stop/start again (I think).

 

The context of all this is live TV recordings, where original files are overwritten once converted. So it can't be done when there is a potential recording. I discovered that the library file monitor will trigger it self the moment the recording starts which is not good (I know there is a plugin start delay but again, the recording could be 30 minutes or 4 hours).

If you schedule the worker count, then that will do all of this for you. There is an old API endpoint which is not well documented for triggering the library scan, you could curl or wget that on a cron job. Or you could just let the library scan run whenever, but scale the worker count on a schedule like this:
image.thumb.png.14b21b9eb96ca3fac3405a78baa0ba33.png

This will set 3 workers to start at 9AM and then stop them when their next task completes at 11AM by setting that value back to 0.

Link to comment
18 hours ago, Josh.5 said:

If you schedule the worker count, then that will do all of this for you. There is an old API endpoint which is not well documented for triggering the library scan, you could curl or wget that on a cron job. Or you could just let the library scan run whenever, but scale the worker count on a schedule like this:
image.thumb.png.14b21b9eb96ca3fac3405a78baa0ba33.png

This will set 3 workers to start at 9AM and then stop them when their next task completes at 11AM by setting that value back to 0.

Sorry, I did miss this part but I have to admit, the wording is a bit unusual. I think it will do the trick though.

 

I would be nice to know what I did wrong with the kill PID method. I tried simply executing a script with bash command and it didn't run (same commands works fine in CLI). Here is the log:
 

2022-03-02T05:46:25:INFO:Unmanic.Plugin.postprocessor_script - Execute command 'bash /config/kill.sh'.
2022-03-02T05:47:25:ERROR:Unmanic.PluginExecutor - [FORMATTED] - Exception while carrying out 'postprocessor.task_result' plugin runner 'postprocessor_script'
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/unmanic/libs/unplugins/executor.py", line 272, in execute_plugin_runner
    runner(data)
  File "/config/.unmanic/plugins/postprocessor_script/plugin.py", line 161, in on_postprocessor_task_results
    exec_subprocess(cmd, args)
  File "/config/.unmanic/plugins/postprocessor_script/plugin.py", line 87, in exec_subprocess
    raise Exception("Failed to execute command: '{}'".format(full_command))
Exception: Failed to execute command: 'bash /config/kill.sh'

 

image.png.b193a78af77bc8738849e95eab1c0ad9.png

Link to comment
1 hour ago, OrdinaryButt said:

Sorry, I did miss this part but I have to admit, the wording is a bit unusual. I think it will do the trick though.

 

I would be nice to know what I did wrong with the kill PID method. I tried simply executing a script with bash command and it didn't run (same commands works fine in CLI). Here is the log:
 

2022-03-02T05:46:25:INFO:Unmanic.Plugin.postprocessor_script - Execute command 'bash /config/kill.sh'.
2022-03-02T05:47:25:ERROR:Unmanic.PluginExecutor - [FORMATTED] - Exception while carrying out 'postprocessor.task_result' plugin runner 'postprocessor_script'
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/dist-packages/unmanic/libs/unplugins/executor.py", line 272, in execute_plugin_runner
    runner(data)
  File "/config/.unmanic/plugins/postprocessor_script/plugin.py", line 161, in on_postprocessor_task_results
    exec_subprocess(cmd, args)
  File "/config/.unmanic/plugins/postprocessor_script/plugin.py", line 87, in exec_subprocess
    raise Exception("Failed to execute command: '{}'".format(full_command))
Exception: Failed to execute command: 'bash /config/kill.sh'

 

image.png.b193a78af77bc8738849e95eab1c0ad9.png

What is the contents of the kill.sh script?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.