[Support] HaveAGitGat - Tdarr: Audio/Video Library Analytics & Transcode Automation


Recommended Posts

Hey all. 
 

I posted about this on the subreddit for tdarr but figured I would post here as well. At this point I’m not running tdarr because of the problem I am having. 
 

Every few days or so I run into an issue where tdarr spawns a ton (20-30) ffmpeg processes and my server comes to a crawl. 
 

When this happens I have to shutdown my server hard as I can’t seem to get it back when it’s pegged at 100%. When I get tdarr back running there’s a bunch of new “cancelled by user” items in the cancelled column. Like those were the ones transcoding and they never finish. 
 

I have tdarr set to do two transcode threads and two health check. 
 

Here is a screen cap to the process list when one of these situations occurred.

 

https://imgur.com/a/qeQ80Rq

 

 

Link to comment
9 hours ago, swells said:

Hey all. 
 

I posted about this on the subreddit for tdarr but figured I would post here as well. At this point I’m not running tdarr because of the problem I am having. 
 

Every few days or so I run into an issue where tdarr spawns a ton (20-30) ffmpeg processes and my server comes to a crawl. 
 

When this happens I have to shutdown my server hard as I can’t seem to get it back when it’s pegged at 100%. When I get tdarr back running there’s a bunch of new “cancelled by user” items in the cancelled column. Like those were the ones transcoding and they never finish. 
 

I have tdarr set to do two transcode threads and two health check. 
 

Here is a screen cap to the process list when one of these situations occurred.

 

https://imgur.com/a/qeQ80Rq

 

 

Which version are you running Tdarr or Tdarr_aio? what plugin are you using? can i get a screen shot of your docker config and the data from the info tab for one of these glitched transcodes plz also in the options tab do you have 

Auto-cancel stalled workers: ON

 

Edited by nicksphone
Link to comment
59 minutes ago, nicksphone said:

Which version are you running Tdarr or Tdarr_aio? what plugin are you using? can i get a screen shot of your docker config and the data from the info tab for one of these glitched transcodes plz also in the options tab do you have 

Auto-cancel stalled workers: ON

 

 

I'm using tdarr_aio. I am running two plugins, Migz-Order Streams and DOOM Tiered H265 MKV. Attached a screen grab of the docker config. I will wait and see if/when this happens again and send the info tab while its happening if possible. Otherwise, when I recover all they ever say is " Item was cancelled by user". I have both Auto-cancel stalled workers and Linux FFmpeg NVENC binary (3.4.5 for unRAID compatibility) set to ON. 

 

Last night I completely uninstalled/removed the tdarr docker and reinstalled everything starting fresh. So far it is running without issue, we'll see if it lasts. 

tdarr.PNG

Link to comment
5 minutes ago, swells said:

 

I'm using tdarr_aio. I am running two plugins, Migz-Order Streams and DOOM Tiered H265 MKV. Attached a screen grab of the docker config. I will wait and see if/when this happens again and send the info tab while its happening if possible. Otherwise, when I recover all they ever say is "imageproxy.php?img=&key=e5eec7c5c933ca16 Item was cancelled by user". I have both Auto-cancel stalled workers and Linux FFmpeg NVENC binary (3.4.5 for unRAID compatibility) set to ON. 

 

Last night I completely uninstalled/removed the tdarr docker and reinstalled everything starting fresh. So far it is running without issue, we'll see if it lasts. 

tdarr.PNG

everything seems fine with what you sent are any of these files .ts it hangs on?

Edited by nicksphone
Link to comment
6 hours ago, nicksphone said:

everything seems fine with what you sent are any of these files .ts it hangs on?

Ok it started happening again this today and I was able to catch it early. 

 

I  have two threads enabled for transcode and two health check. There are currently 4 processes running, two of them are movies that are now in the Transcode: Error/Cancelled column. Both of them say "Item was cancelled by user."

 

I'm attaching some screenshots from the time I noticed this. 

Here are the processes:

 

17078  100  1.3 5729580 459948 ?      Rl   06:38 355:59 /home/Tdarr/Tdarr/bundle/programs/server/assets/app/ffmpeg/ffmpeg345/ffmpeg -c:v h264_cuvid -i /home/Tdarr/Media/Movies - HD/XX (2017)/XX (2017) [Bluray-1080p].mkv -map 0 -map -0:d -c:v hevc_nvenc -rc:v vbr_hq -qmin 0 -cq:v 31 -b:v 2500k -maxrate:v 5000k -preset medium -rc-lookahead 32 -spatial_aq:v 1 -aq-strength:v 8 -c:a copy -c:s copy -max_muxing_queue_size 4096 /home/Tdarr/cache/XX (2017) [Bluray-1080p]-TdarrCacheFile-u9gUbCn-z.mkv

 

4826  100  1.2 5712284 422596 ?      Rl   05:11 444:04 /home/Tdarr/Tdarr/bundle/programs/server/assets/app/ffmpeg/ffmpeg42/ffmpeg -c:v h264_cuvid -i /home/Tdarr/Media/Movies - HD/Young Ones (2014)/Young Ones (2014) [Bluray-1080p].mkv -map 0 -map -0:d -c:v hevc_nvenc -rc:v vbr_hq -qmin 0 -cq:v 31 -b:v 2500k -maxrate:v 5000k -preset medium -rc-lookahead 32 -spatial_aq:v 1 -aq-strength:v 8 -c:a copy -c:s copy -max_muxing_queue_size 4096 /home/Tdarr/cache/Young Ones (2014) [Bluray-1080p]-TdarrCacheFile-_olUQbBMe.mkv

 

16754 21.9  1.4 5812736 492792 ?      Sl   12:33   1:13 /home/Tdarr/Tdarr/bundle/programs/server/assets/app/ffmpeg/ffmpeg345/ffmpeg -c:v h264_cuvid -i /home/Tdarr/Media/Movies - HD/Wolves (2016)/Wolves (2016) [WEBDL-1080p].mkv -map 0 -map -0:d -c:v hevc_nvenc -rc:v vbr_hq -qmin 0 -cq:v 31 -b:v 2500k -maxrate:v 5000k -preset medium -rc-lookahead 32 -spatial_aq:v 1 -aq-strength:v 8 -c:a copy -map -0:a:0 -c:s copy -map -0:s:0 -max_muxing_queue_size 4096 /home/Tdarr/cache/Wolves (2016) [WEBDL-1080p]-TdarrCacheFile-6mulRpdAg.mkv

 

11000 13.7  1.5 5821572 494684 ?      Sl   12:29   1:18 /home/Tdarr/Tdarr/bundle/programs/server/assets/app/ffmpeg/ffmpeg345/ffmpeg -c:v h264_cuvid -i /home/Tdarr/Media/Movies - HD/Woman Walks Ahead (2017)/Woman Walks Ahead (2017) [Bluray-1080p].mkv -map 0 -map -0:d -c:v hevc_nvenc -rc:v vbr_hq -qmin 0 -cq:v 31 -b:v 2500k -maxrate:v 5000k -preset medium -rc-lookahead 32 -spatial_aq:v 1 -aq-strength:v 8 -c:a copy -c:s copy -max_muxing_queue_size 4096 /home/Tdarr/cache/Woman Walks Ahead (2017) [Bluray-1080p]-TdarrCacheFile-dN0kys-vG.mkv

top.PNG

cancelled.PNG

queue.PNG

smi.PNG

Link to comment
16 hours ago, swells said:

Ok it started happening again this today and I was able to catch it early. 

 

I  have two threads enabled for transcode and two health check. There are currently 4 processes running, two of them are movies that are now in the Transcode: Error/Cancelled column. Both of them say "Item was cancelled by user."

 

I'm attaching some screenshots from the time I noticed this. 
 

ok first thing its looking like your kill stalled is not working correctly now i know the software is not great with ts files so if you delete .ts and .m2ts from the scanned for files in your lib containers your likely to get rid of this issue it solved it for me. For .ts and .m2ts files use the recording converter app its made for ts files will put them in mp4 files for you. it will still fail on some but its alot better than this at the moment. maybe when they are out of beta it will work better with .ts files. 

Edited by nicksphone
Link to comment
4 hours ago, nicksphone said:

ok first thing its looking like your kill stalled is not working correctly now i know the software is not great with ts files so if you delete .ts and .m2ts from the scanned for files in your lib containers your likely to get rid of this issue it solved it for me. For .ts and .m2ts files use the recording converter app its made for ts files will put them in mp4 files for you. it will still fail on some but its alot better than this at the moment. maybe when they are out of beta it will work better with .ts files. 

Yeah, I was just coming here to say I think I have tracked the issue to the "Auto-cancel stalled workers" option being on. I turned it off last  night and have not had an issue since. I have had a few fail, most are .ts or .m2ts.

 

Thanks for the help!

Link to comment
  • 2 weeks later...

Hi,

I'm getting errors since the latest update, i tried clearing the DB and reinstalling from scratch as well but files which are definitely videos keep on being not recognised (everyone of them).

Info are all similar:

Tdarr_Plugin_lmg1_Reorder_Streams - Pre-processing

File is not video

(...)

Same issue with every plugin, filemedium seems not to be correctly passed on, i suspect an issue with FFMpeg install in the container.

Link to comment
5 hours ago, butch2k said:

Hi,

I'm getting errors since the latest update, i tried clearing the DB and reinstalling from scratch as well but files which are definitely videos keep on being not recognised (everyone of them).

Info are all similar:

Tdarr_Plugin_lmg1_Reorder_Streams - Pre-processing

File is not video

(...)

Same issue with every plugin, filemedium seems not to be correctly passed on, i suspect an issue with FFMpeg install in the container.

If anyone else is having this issue make sure to update to 1.2068 

Link to comment

Hi all, I released a pretty big update yesterday and have added some small bug fixes since. Make sure to use 1.2068 (latest on all containers). There are still issues with the output folder option (would recommend turning that off for now unless it's working for your setup).

 

Beta v1.2068 release [19th September 2020]:

Fix for 'File is not a video' issue

Beta v1.2067 release [18th September 2020]:

Removed 'General' workers

Beta v1.2066 release [18th September 2020]:

Hide 'Low CPU Priority' toggle if running in Docker (prevent bug "pgrep cannot allocate 4611686018427387903 bytes" - https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=955697)

Reduce worker slider limits automatically when shutting down worker/s

Some UI changes

Fixed worker launch bug

Fixed FFmpeg worker percentages not showing in some cases

Fixed CCExtractor dependency error

Beta v1.206 release [17th September 2020]:
--------IMPORTANT----------------

New lean database to reduce RAM usage.

Files are only scanned with FFprobe (and optionally CC Checker) during a library scan - the initial scan time is now much faster

This scan data is then stripped down to contain only the necessary data which Tdarr needs

Files are then scanned more thoroughly before plugins are executed so that full FFprobe and ExifTool data is available

Revert queue building being in separate process (reduce RAM issues)

Fixed bug whereby plugin validator wasn't running when toggling between "Community" and "Local" checkbox

Now no need to refresh plugin stack page after adding a new Local/Community plugin to the plugins folder.

Now no need to refresh page when saving an edited plugin on the "Plugins tab"

Now no need to refresh page after pressing "Copy to Local" on a Community plugin

Now no need to refresh page after creating new plugin in Plugin Creator

Some styling changes to pages to make more use of space (For example the library nav bar has been moved above the settings to give more width for the plugin stack and schedule)

Schedule now highlights current schedule block to make scheduling easier

Added warning if library source folder or transcode cache are empty
---------v1.2061

Click on file name on any of the tabs on the Tdarr tab to show full file details (on-demand file scan is carried out)
---------v1.2062

Fixed bug with worker percentages during FFmpeg health check
---------v1.2063

Fixed 'Scan on start' file scanner bug

Fixed RAM displaying incorrectly
---------v1.2064

Fixed slowness bug when 'Library prioritisation' enabled

MediaInfo scan re-added

Beta v1.2051 release [6th September 2020]:

Add 2 second delay before deleting/renaming/moving files to prevent 'lock' permission errors after transcode

Add worker spawn delay

Prevent transcode workers from requesting item while queue is being built

Beta v1.205 release [20th August 2020]:

Clean-up/reduce size of files' log and history on start-up

Add resource stats to dashboard for process and OS used/free memory as well as OS CPU%

Beta v1.204 release [10th August 2020]:

(NVDEC) FFmpeg thorough health checks now show percentages and ETA

Quicker chown when first starting Docker container

Fix file scanner not exiting when finished (due to new cancel scan button)

Fix workers temporarily running out of items when doing quick health checks

Beta v1.203 release [9th August 2020]:

FFmpeg thorough health checks now show percentages and ETA

Export txt file of files which have failed health checks

Button added to cancel a scan

Fixed schedule bug (Tdarr not processing files when library is in schedule)

Fixed FFmpeg and HandBrake processes not exiting when shutting down a worker

Fixed worker percentage bar sometimes disappearing/reappearing

Some UI changes (nav bar collapases earlier when compressing screen)

Beta v1.202 release [8th August 2020]:

Sort queue by codec

Fixed queue sort UI bug where selected sort doesn't show correctly

Removed old FFmpeg binary (3.4.5) and removed switch on Options tab to enable it - can no longer use due to FFmpeg dependency update for 4.2

Updated FFmpeg and HandBrake binary paths - can now set own binary paths using env variables (FFmpegPath and HandBrakePath)

Some UI changes (Logout button, release notes, plugins, workers)

Health check option with nvdec

Beta v1.201 release [5th August 2020]:

New authentication system (can now log in with Discord if you have the 'Patreon: Supporter' role)

New container (thanks to Rox) - testing this week

Container features:

Much smaller: 1.9GB to 0.61GB (Compressed)

Can set PUID/PGID

FFmpeg supports vaapi, quicksync, nvenc, npp and cuda
Beta v1.200 release [30th July 2020]:

Queue builder moved to own thread (now 4-5x faster)

More robust stat and queue building process

Worker UI (Key elements highlighted in worker details + tidier process reasons)

MediaInfo added to file scanner (best option for bitrates, accessed at file.MediaInfo)

Priority Tdarr container for Patreon supporters

Fixed bug with Tdarr tab tables

5 video tutorials

Link to comment
On 9/10/2020 at 4:36 PM, swells said:

Yeah, I was just coming here to say I think I have tracked the issue to the "Auto-cancel stalled workers" option being on. I turned it off last  night and have not had an issue since. I have had a few fail, most are .ts or .m2ts.

 

Thanks for the help!

Hi sorry I have updated that message for the next version so it will show "Item was cancelled automatically by the worker stall detector as no progress was detected for 5 minutes. You can disable this on the Options tab"

Link to comment
On 9/20/2020 at 2:35 AM, dapiedude said:

@HaveAGitGat - I am running the UnRAID Tdarr_aio docker and using the haveagitgat/tdarr_aio repo. How would I make sure that I get the 1.2068 beta? Is it a different repo?

No it’s on the same one (I update both tdarr and tdarr_aio even though they’re the same). Just make sure to pull the latest haveagitgat/tdarr_aio container (unraid should do that automatically when you check for updates)

Edited by HaveAGitGat
  • Thanks 1
Link to comment

I noticed that Intel QSV seems to have stopped working recently.  I have over 173 failed transcodes (I don't look often) and they all have the same error.

[20:28:04] Intel Quick Sync Video support: no

 

Haven't made any changes to my system and QSV still works for Plex HW transcoding.  I remember we had a lot of trouble getting this to work.  Is it broken again?

 

Thanks

Link to comment

Anyone else no longer see metadata from files after Beta 1.207? My process relies on a metadata comment to not convert items already processed but since update tdarr no longer is seeing any file metadata during health check scans so everything is processing over and over :|
 

edit: was previously on Beta 1.1093

Edited by jsmontague
updated with prior version
Link to comment
  • 4 weeks later...

Alright, I'm confused. I'm running some test transcodes on a directory I made with a few movies so I don't break my whole collection. Once a movie finishes transcoding, it starts it over again. It keeps doing that endlessly. Not sure where my problem is. My goal is to transcode what is already inside my collection, placing the file in the same folder the original was.

 

Transcode argument

-Z "H.265 MKV 2160p60" -e x265 --all-subtitles --all-audio

 

Library and output  folder

/home/Tdarr/Media/Test

Link to comment

Hey guys, how can I use Tdarr to transcode my library of audio files down from way too high quality rips? I have a ton of flac and mp3-320 files that I would like to cut down in size by converting them to mp3-192 (more than good enough for my ears and equipment). I guess I'm supposed to use something ffmpeg related, but I'm at a loss how to handle non-video files using which plugins...

 

Any help greatly appreciated!

Link to comment
On 10/22/2020 at 2:31 PM, Trenta27 said:

Alright, I'm confused. I'm running some test transcodes on a directory I made with a few movies so I don't break my whole collection. Once a movie finishes transcoding, it starts it over again. It keeps doing that endlessly. Not sure where my problem is. My goal is to transcode what is already inside my collection, placing the file in the same folder the original was.

 

Transcode argument

-Z "H.265 MKV 2160p60" -e x265 --all-subtitles --all-audio

 

Library and output  folder

/home/Tdarr/Media/Test

your having it loop as its doing everything regardless of format you need to have something that says if it has this skip.

 

Link to comment
10 hours ago, tmchow said:

I just became a patreon supporter. How do I install the pro_latest version in unraid when using the tdarr_aio version?  

change the repo to haveagitgat/tdarr:pro_latest  it will want you to login after it updates use patreon login as discord is broke atm. you will also most likely have to fix permissions as well http://serverip/Tools/DockerSafeNewPerms as the free version used a different permissions and you will see lots of fail transcodes if this is the case 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.