[Support] randomninjaatk - sabnzbd-extended

[randomninjaatk]

Docker: SABnzbd Stable with audio/video post processing scripts script included!

 

Scripts:

sickbeard_mp4_automator (SMA) :: https://github.com/mdhiggins/sickbeard_mp4_automator

 

Documentation can be found here: https://github.com/RandomNinjaAtk/docker-sabnzbd-extended

[pererikk]

Hi!

Getting the following error with the default docker settings, after every download.

 

Script exit code is 1

 

 

 

 

[Sandspike]

I have the same issue. Not sure why.


Sent from my iPhone using Tapatalk

[Sandspike]

chmod 777 the scripts and problem goes away.


Sent from my iPhone using Tapatalk

[randomninjaatk]

Interesting, the scripts should have the correct permissions on startup...

[Sandspike]

Now the issue is that it isn’t detecting English and deleting files. I’m wondering if I even need the scripts or perhaps I should alter some options.


Sent from my iPhone using Tapatalk

[randomninjaatk]

It depends on what options your using. By default if you don't have SMA or MKV Cleaner enabled, if it does not detect the specified language track for either subtitle or audio, it says its a bad release.

 

However in practice, sometimes I find what would be considered english is not marked as english in the original file (Typically UND or Null language). MKV Cleaner fixes that by ignoring that check and fixing the file, but I'm not sure the same workaround is in place when SMA enabled to run in that same scenario.

 

Enabling MKV Cleaner would resolve that issue, but know that you'll be effectively processing the file twice, once with MKV Cleaner, a script I wrote, and then with SMA if your using that. 

[Sandspike]

So I use SMA in the docker container for Radarr and Sonarr, but it isn’t downloading or encoding subtitles, so that is another issue. I did put the same options in the SAB container, but it doesn’t appear to be executing anything. I actually had to add the deobfuscate script to the script folders as well. Maybe a recommended add? MKV Cleaner, where would I enable that...I don’t remember seeing it. How much extra processing are we talking?

[randomninjaatk]

So quick update that should fix it, I've been primarily using MKV Cleaner only, so this hasn't been an issue for me.

 

Fixed for SMA with: https://github.com/RandomNinjaAtk/sabnzbd-scripts/commit/adcbf92b926d9f13e394ce250c3e3b0d04894125

 

To get the update, you need to have the "Auto Update SABnzbd Extended Scripts:" set to: TRUE and then reboot the container.

 

 

8 minutes ago, Sandspike said:

So I use SMA in the docker container for Radarr and Sonarr, but it isn’t downloading or encoding subtitles, so that is another issue. I did put the same options in the SAB container, but it doesn’t appear to be executing anything. I actually had to add the deobfuscate script to the script folders as well. Maybe a recommended add? MKV Cleaner, where would I enable that...I don’t remember seeing it. How much extra processing are we talking?

 

MKV Cleaner uses MKVToolsNix to repackage the files into mkv based on your language preference. Its primary function is to mark UND/NULL audio streams to the specified language, and to remove unwanted audio/subtitle streams. Example: If you have a file with 10 audio streams, 1 is english, and your setting is english, all the other audio streams willl be removed via mkvmerge application. This also works the same for subtitles. If no modifications are needed, the script simply checks the file and moves on.

 

The sonarr-sma and radarr-sma dockers I have need some updates probably. The main MP4 automator script was updated a bunch, and I'm not sure I've incorporated the config option changes. Those changes were added to this container. I may move the configuration to outside of the container settings, because its a lot to maintain and track when that script is updated. Also doesn't help that I'm not actively using it, and just using MKV Cleaner right now.

 

 

 

Edited April 14, 2020 by randomninjaatk

[Sandspike]

So I turned it on in the docker config, rebooted. Still deleted a known English file. Any other settings to watch/enable? Is that a different post process script that needs to be enabled?

On the subtitles, yeah, on my windows box and in the SMA instructions, you have to download subliminal and other files, I’d love to figure how to do this in a docker. MP4, 720p with subs added to the file is my ideal state.

I normally execute SMA in Radarr and Sonarr, should I disable the settings in SAB? What do you recommend? If MKV Cleaner is lean enough, I would run it in SAB, then pass to Sonarr/Radarr and convert to MP4 there.

Thanks for all of this, it’s been super helpful.

[randomninjaatk]

MKV Cleaner is strictly a repackager, so it will not convert or anything else, but it will clean up the file. So it would be safe to enable it, and then have radarr/sonarr use SMA to further convert and process the file.

 

To turn on MKV CLeaner, set "Video Post Processor: Process with MKV Cleaner:" to TRUE

 

When I was using SMA, my preferred setup was:

 

Sabnzbd with SMA conversion -> Radarr/Sonarr and SMA Tagging only.

 

So the the file was converted before sonarr/radarr imported it and all radarr/sonarr was doing at that point was adding tags to the file when using MP4 as the format.

 

I stopped using SMA because I was having trouble with forced subs (disposition) when using MKV. And it's easier to use sonarr/radarr to just grab your target quality and I simply want to cleanup the file (re-tag unknown audio to correct language and remove unwanted streams), not re-encode the video/audio. 

 

 

 

Edited April 14, 2020 by randomninjaatk

[Sandspike]

Thanks I had to shut off the post processing script, as it keeps deleting files, even after cleaner appears to run.

 

Video Post Processor: Process with MKV Cleaner: True

 

2020-04-14 21:36:03,634::INFO::[postproc:388] Finished unpack_magic on XXXXX
2020-04-14 21:36:03,637::INFO::[postproc:558] Cleaning up XXXXX (keep_basic=False)

 

I'm not sure it is running actually, what should the log to say when running the cleaner?

 

On the subtitles issues, we would need a linux loadout like this (this is what I did on windows, the sub pre-reqs are missing in all sma containers):

 

pip install requests

pip install requests[security]

pip install requests-cache

pip install babelfish

pip install "guessit<2"

pip install "subliminal<2"

pip install stevedore==1.19.1

pip install python-dateutil

pip install deluge-client

pip install qtfaststart

 

 

[randomninjaatk]

9 hours ago, Sandspike said:

Thanks I had to shut off the post processing script, as it keeps deleting files, even after cleaner appears to run.

 

Video Post Processor: Process with MKV Cleaner: True

 

You need to set that to: TRUE

Just like the description says to enable it. Capitalization matters...

 

I was seeing some of the same issues today with the script, and made a bugfix change that I think resolves it.

 

The pre-reqs are installed via SMA's installation file:

https://github.com/mdhiggins/sickbeard_mp4_automator/blob/master/setup/requirements.txt

 

EDIT: Sample output

 

Example output using MKV Cleaner with no changes needed:

====================================================
Begin processing 1 of 1: File Name.mkv
Checking for audio/subtitle tracks
1 video track found!
1 audio tracks found!
2 subtitle tracks found!
Checking for "eng" video/audio/subtitle tracks
1 audio tracks found!
2 subtitle tracks found!
INFO: Video passed all checks, no processing needed
Post Processing Complete!

Example output using MKV Cleaner when changes are needed:

====================================================
Begin processing 1 of 1: File Name.mkv
Checking for audio/subtitle tracks
1 video track found!
1 audio tracks found!
1 subtitle tracks found!
Checking for "eng" video/audio/subtitle tracks
1 "unknown" audio tracks found, re-tagging as "eng"
1 subtitle tracks found!
1 "unknown" video language track found, re-tagging as "eng"
mkvmerge v19.0.0 ('Brave Captain') 64-bit
'/mnt/disks/SSD01/downloads/sabnzbd/complete/sonarr/Folder/File Name.mkv': Using the demultiplexer for the format 'Matroska'.
'/mnt/disks/SSD01/downloads/sabnzbd/complete/sonarr/Folder/File Name.mkv' track 0: Using the output module for the format 'AAC'.
'/mnt/disks/SSD01/downloads/sabnzbd/complete/sonarr/Folder/File Name.mkv' track 1: Using the output module for the format 'AVC/h.264'.
'/mnt/disks/SSD01/downloads/sabnzbd/complete/sonarr/Folder/File Name.mkv' track 2: Using the output module for the format 'text subtitles'.
The file '/mnt/disks/SSD01/downloads/sabnzbd/complete/sonarr/Folder/File Name.merged.mkv' has been opened for writing.
Progress: 0%
Progress: 34%
Progress: 67%
Progress: 82%
Progress: 99%
Progress: 100%
The cue entries (the index) are being written...
Multiplexing took 2 seconds.
SUCCESS: mkvmerge complete
INFO: Options used: -d 1 --language 1:eng -a 0 --language 0:eng
INFO: Renamed source file
INFO: Renamed temp file
INFO: Deleted source file
====================================================
Post Processing Complete!

 

Edited April 16, 2020 by randomninjaatk

[randomninjaatk]

@Sandspike

 

So after some more debugging, it does not look like a bug at all. Especially with the latest changes. It seems that the incoming files are likely poorly marked with the wrong language and because of that, the script is smart enough to detect it and mark it as bad. 

 

In one instance, the language of a file was actually English audio after testing the file, but was marked bad because the actual audio track was tagged with a different language.

 

Basically my script is working as designed, but can't fix crappy uploaders.... In any case, I'd rather it personally mark these as bad, because they are not correct to begin with... So this is not a bug and everything is working as intended...

 

If you reboot your container now, you should get the debugging version of the script. It will mark the download as failed if you have that enabled in SABnzbd, but it will no longer delete the file. 

 

I can probably work on including some more descriptive error messaging so it's less confusing going forward. But everything appears to be working as intended...

 

[randomninjaatk]

@Sandspike

 

TLDR:

Your likely downloading files that are tagged poorly. They should have been marked as the correct language from the uploader, but they are not. So the script is detecting that and marking the files bad as a result... Everything is working as intended. I've added some better error messaging if you have updates enabled and reboot your container to get the latest script updates....

 

 

Detailed information:

 

So pushed a couple more updates, to give better error messaging, you should see something like this now:

 

Begin processing 1 of 1: File Name.mkv
Checking for audio/subtitle tracks
1 video track found!
1 audio tracks found!
1 subtitle tracks found!
Checking for "eng" video/audio/subtitle tracks
INFO: "cos" audio track found!
INFO: "und" subtitle track found!
ERROR: no "eng" audio/subtitle tracks found!

 

As you can see now, it gives a better understanding to why it failed. (In my case: eng is the set language).

 

So I manually checked this file, after the recent changes will no longer delete the file, just mark it as bad, and this is what I found.... So this file's audio is actually in English, and the subtitle is also in English, but because the uploader hasn't tagged anything properly, the script cannot fix that and marks it as bad... So everything is working as intended in this case, the script can detect poorly tagged files, but cannot fix them.... in this case....

 

Some more background on the logic behind passing language verification requirements.

 

The basic logic for passing the language requirement test is this:

 

1. If a file has Foreign Audio (IE: not matching the set language), then a subtitle track must be included matching the set language language. If the file has foreign audio only and no matching subtitle language track, then the download is marked as failed. This is set that way because I personally wouldn't want a file with possibly JAP audio with JAP subtitles only, because it would be useless in my case because my primary language is English.

 

2. If a file has UND/NULL audio language track, this is assumed to be the primary set language based on your settings. MKV Cleaner will re-tag those tracks to the correct language. This will not cause the script to remove/mark these files as bad. SMA will also still be able to process these files even with MKV Cleaner disabled, and SMA also has the ability to re-tag these tracks. However, if you do not have MKV Cleaner or SMA enabled using the "TRUE" setting. These downloads will be marked as failed with a script exit 1 based on your language setting in the container.

 

MKV Cleaner will not re-tag or assume UND/NULL language subtitle tracks are the same language as the primary set language. I could make a change, so that it could re-tag those, just like it does for audio/video tracks. However, since they are text based tracks, I'd rather not incorrectly tag these as a different language. So that is why UND/NULL audio language tracks are considered bad and would not be re-tagged. These tracks are removed from the file if using MKV Cleaner, and SMA will handle them however you have it configured.

 

Hope this gives you a better understanding of what's actually happening. I think the added error messaging makes it clearer. 

 

 

Github Commit fix: https://github.com/RandomNinjaAtk/sabnzbd-scripts/commit/8982babfd173047297936ca5e5bd80ab6e0dfd21#diff-3d12cc7ce365dada5f5ca61464ce06c8

Edited April 16, 2020 by randomninjaatk

[Sandspike]

15 hours ago, randomninjaatk said:

 

You need to set that to: TRUE

Just like the description says to enable it. Capitalization matters...

 

I was seeing some of the same issues today with the script, and made a bugfix change that I think resolves it.

 

The pre-reqs are installed via SMA's installation file:

https://github.com/mdhiggins/sickbeard_mp4_automator/blob/master/setup/requirements.txt

 

EDIT: Sample output

 

Example output using MKV Cleaner with no changes needed:

====================================================
Begin processing 1 of 1: File Name.mkv
Checking for audio/subtitle tracks
1 video track found!
1 audio tracks found!
2 subtitle tracks found!
Checking for "eng" video/audio/subtitle tracks
1 audio tracks found!
2 subtitle tracks found!
INFO: Video passed all checks, no processing needed
Post Processing Complete!

Example output using MKV Cleaner when changes are needed:

====================================================
Begin processing 1 of 1: File Name.mkv
Checking for audio/subtitle tracks
1 video track found!
1 audio tracks found!
1 subtitle tracks found!
Checking for "eng" video/audio/subtitle tracks
1 "unknown" audio tracks found, re-tagging as "eng"
1 subtitle tracks found!
1 "unknown" video language track found, re-tagging as "eng"
mkvmerge v19.0.0 ('Brave Captain') 64-bit
'/mnt/disks/SSD01/downloads/sabnzbd/complete/sonarr/Folder/File Name.mkv': Using the demultiplexer for the format 'Matroska'.
'/mnt/disks/SSD01/downloads/sabnzbd/complete/sonarr/Folder/File Name.mkv' track 0: Using the output module for the format 'AAC'.
'/mnt/disks/SSD01/downloads/sabnzbd/complete/sonarr/Folder/File Name.mkv' track 1: Using the output module for the format 'AVC/h.264'.
'/mnt/disks/SSD01/downloads/sabnzbd/complete/sonarr/Folder/File Name.mkv' track 2: Using the output module for the format 'text subtitles'.
The file '/mnt/disks/SSD01/downloads/sabnzbd/complete/sonarr/Folder/File Name.merged.mkv' has been opened for writing.
Progress: 0%
Progress: 34%
Progress: 67%
Progress: 82%
Progress: 99%
Progress: 100%
The cue entries (the index) are being written...
Multiplexing took 2 seconds.
SUCCESS: mkvmerge complete
INFO: Options used: -d 1 --language 1:eng -a 0 --language 0:eng
INFO: Renamed source file
INFO: Renamed temp file
INFO: Deleted source file
====================================================
Post Processing Complete!

 

Totally my fault for not being clear, I appreciate the error check, I am using TRUE. 

 

However, is that the same in Sonarr-SMA?  I see True and False used by default. 

When I run manual.py on my windows machine, there is verbose logging that shows the show matching, subtitle downloading. 

 

On Sonarr-SMA

So if all those pre-reqs are installed, I'm confused why it would not even try, and nothing in the log, when subs is setup identical to my windows box via your options in docker.

 

Reading your subsequent post, I appreciate you checking and updating the scripts.  The more verbose output is hugely helpful.

I assume just have update on, restart the container.

 

I would like to assume the file is good rather than bad, yeah these uploaders are not tagging stuff, but 99.9% of the recent stuff is correct for me.  I was thinking MKV cleaner would tag vid/auto/txt as default language if marked as UND/null, thanks for the clarification.  But I don't want it deleted for sure.

So perhaps an opportunity to add some options.

 

Because Subs often don't match the screen, I usually manually download the SRT, then manually encode them into the file

Example:

ffmpeg -i "movie.mp4" -i "movie.srt" -c:s mov_text -c:v copy -c:a copy "movie-s.mp4"

 

Adding the subs to the MP4, takes about 30 seconds.  So it's fast, but I would love to automate.

It seems like Plex is building this feature in these days, you can hit subs, more, and search for an SRT online.

So maybe I should just worry about aud/vid.

 

Or maybe I should drop everything into a sweep folder, like spaceinvader one on youtube recommends.

Then setup a container/vm with SMA installed, and run the manual conversion.

The problem is, I was able to follow the Reddit post to get mine working, I'm just not Linux proficient enough to do the same in Linux.

But I'm learning alot in this process...thanks for all the effort.

[randomninjaatk]

So everything were discussing in here, only applies to the SAB container. (However, SMA is installed using the exact same method for all the containers I have, so that does apply to both...)

 

The radarr/sonarr-sma containers that I put out, don't do this. They strictly use SMA if you enable it as a sonarr/radarr post processing script.

 

Whereas this SAB container runs up to 4 functions for video files:

1. Deobfuscate SAB Script
2. Video File audio/subtitle language check
   1. MKV Cleaner (If enabled)
   2. SMA (MP4 automator, if enabled)

I could look into adding an option to skip the language checks, I'd personally never use it because I like to have standards, and if a release doesn't meet those requirements then I don't trust it and I'm okay with just grabbing an alternative release until one passes...

Edited April 16, 2020 by randomninjaatk

[randomninjaatk]

@Sandspike

 

I don't know if you have tried my latest updates to these containers. But I decided to give downloading subs a try in my Sonarr-SMA container, and it did successfully download them. 

 

So it should be working in all containers....

 

If you want to chat more about this, feel free to PM me and we can possibly chat on Discord...

Edited April 23, 2020 by randomninjaatk

[Sandspike]

Woot...Woot!

I’ll give it a try and report back.

[Sandspike]

Was it broken to begin with or did I mess something up and it was always working?

Care to share your docker options?

[randomninjaatk]

So a big change, mostly to make my life easier. Is all the settings are now in a config file in: /config/sma (sonarr-sma/radarr-sma) containers, or /config/scripts/configs  for sabnzbd-extended container.

 

It removes some of the ease of use that was previously setup. however its a lot to maintain when the script can change at anytime, requiring lots of re-tweaking. This should hopefully mean updates can be seamless and I don't have to actively maintain it.

[Sandspike]

That makes good sense, a static config file, with less variability that should help.

Can you share your file config options, so I can hunt for errors in mine?

[randomninjaatk]

So this was discussed and resolved via discord. All containers are working as expected, some of it was just miss configuration issues and using outdated scripts...

[Sandspike]

Thanks to @randomninjaatk worked through the issues I was having (several due to misconfiguration, some me trying to juryrig things), adding the features I wanted, and we discovered some new issues with added features (which were fixed within hours).  Just amazing, this container works perfectly for me now.