I can't find clear troubleshooting steps. I've tried a number of things but just realized with Unraid 7, transcoding stopped working.

JellyFin error logs indicate:

Stream mapping:
  Stream #0:0 (vc1) -> setparams:default (graph 0)
  Stream #0:2 (pgssub) -> scale:default (graph 0)
  overlay_cuda:default (graph 0) -> Stream #0:0 (h264_nvenc)
  Stream #0:1 -> #0:1 (dts (dca) -> aac (native))
Press [q] to stop, [?] for help
[vc1 @ 0x560bb6551ec0] Cannot load libnvcuvid.so.1
[vc1 @ 0x560bb6551ec0] Failed loading nvcuvid.
[vc1 @ 0x560bb6551ec0] Failed setup for format cuda: hwaccel initialisation returned error.
Impossible to convert between the formats supported by the filter 'Parsed_setparams_4' and the filter 'auto_scale_0'
[fc#0 @ 0x560ba9284c00] Error reinitializing filters!
[fc#0 @ 0x560ba9284c00] Task finished with error code: -38 (Function not implemented)
[fc#0 @ 0x560ba9284c00] Terminating thread with return code -38 (Function not implemented)
[vost#0:0/h264_nvenc @ 0x560bae660800] Could not open encoder before EOF
[vost#0:0/h264_nvenc @ 0x560bae660800] Task finished with error code: -22 (Invalid argument)
[vost#0:0/h264_nvenc @ 0x560bae660800] Terminating thread with return code -22 (Invalid argument)
[out#0/hls @ 0x560ba9ae8700] Nothing was written into output file, because at least one of its streams received no packets.
frame=    0 fps=0.0 q=0.0 Lsize=       0KiB time=N/A bitrate=N/A speed=N/A    
[aac @ 0x560ba9468440] Qavg: 2085.431
Conversion failed

I'm using binhex-jellyfin and have tried a number of things:

1. Setting NVIDIA_VISIBLE_DEVICES = ALL

2. Setting NVIDIA_VISIBLE_DEVICES=GPU-19252273-3b56-cc88-d9e1-cbcdb08ab925

3. Setting extra parameters --runtime=nvidia --gpus=all (also tried without --gpus=all)

4. Saw a reddit thread about if you had nerd tools installed at some point that reinstalling it even though its deprecated in 7 could make it work, so tried that. So far nothing.

5. ChannelsDVR is also failing to transcode.

JellyFin Log (above log is from ffmpeg within jellyfin):

[2025-08-06 19:14:54.399 -05:00] [INF] [256] MediaBrowser.MediaEncoding.Attachments.AttachmentExtractor: "/usr/lib/jellyfin-ffmpeg/ffmpeg" "-dump_attachment:t \"\" -y  -i file:\"/media/Movies/Caddyshack (1980)/Caddyshack (1980) Remux-1080p.mkv\" -t 0 -f null null"
[2025-08-06 19:14:54.618 -05:00] [INF] [256] MediaBrowser.MediaEncoding.Attachments.AttachmentExtractor: ffmpeg attachment extraction completed for "file:\"/media/Movies/Caddyshack (1980)/Caddyshack (1980) Remux-1080p.mkv\"" to "/config/cache/attachments/6f0d6d4e5f3ad50773c18fcd4d5589d5"
[2025-08-06 19:14:54.618 -05:00] [INF] [256] MediaBrowser.MediaEncoding.Transcoding.TranscodeManager: "/usr/lib/jellyfin-ffmpeg/ffmpeg" "-analyzeduration 200M -probesize 1G -f matroska -init_hw_device cuda=cu:0 -filter_hw_device cu -hwaccel cuda -hwaccel_output_format cuda -noautorotate -hwaccel_flags +unsafe_output -threads 1 -canvas_size 1920x1080 -i file:\"/media/Movies/Caddyshack (1980)/Caddyshack (1980) Remux-1080p.mkv\" -noautoscale -map_metadata -1 -map_chapters -1 -threads 0 -map 0:0 -map 0:1 -map -0:0 -codec:v:0 h264_nvenc -preset p1 -b:v 27103521 -maxrate 27103521 -bufsize 54207042 -profile:v:0 high -g:v:0 72 -keyint_min:v:0 72 -filter_complex \"[0:2]scale,scale=1920:1080:fast_bilinear,format=yuva420p,hwupload=derive_device=cuda[sub];[0:0]setparams=color_primaries=bt709:color_trc=bt709:colorspace=bt709,scale_cuda=format=yuv420p[main];[main][sub]overlay_cuda=eof_action=pass:repeatlast=0\" -start_at_zero -codec:a:0 aac -ac 6 -ab 128000 -copyts -avoid_negative_ts disabled -max_muxing_queue_size 2048 -f hls -max_delay 5000000 -hls_time 3 -hls_segment_type mpegts -start_number 0 -hls_segment_filename \"/config/cache/transcodes/f1d62ab0bb6d39b630e98bcb966ba726%d.ts\" -hls_playlist_type vod -hls_list_size 0 -y \"/config/cache/transcodes/f1d62ab0bb6d39b630e98bcb966ba726.m3u8\""
[2025-08-06 19:14:54.644 -05:00] [ERR] [242] MediaBrowser.MediaEncoding.Transcoding.TranscodeManager: FFmpeg exited with code 255
[2025-08-06 19:14:54.741 -05:00] [ERR] [242] Jellyfin.Api.Middleware.ExceptionMiddleware: Error processing request. URL "GET" "/videos/6f0d6d4e-5f3a-d507-73c1-8fcd4d5589d5/hls1/main/0.ts".
MediaBrowser.Common.FfmpegException: FFmpeg exited with code 255

3 hours ago, austin said:

I'm using binhex-jellyfin and have tried a number of things:

and you also checked the jelly settings ?

GitHub

documentation/docker/faq/jellyfin.md at master · binhex/d...

Documentation for software located in the binhex repositories. - binhex/documentation

Yea I’ve done all this. Even tried Linux server and Jellyfin images. Fact that it’s also stopped in channelsdvr makes me think there is something to the not working with unraid 7 posts I’m seeing on Reddit.

4 hours ago, austin said:

Fact that it’s also stopped in channelsdvr makes me think there is something to the not working with unraid 7 posts I’m seeing on Reddit.

well, i can say my Nvidia is working nice on v7 here, but im not using it for plex (intel igpu) and i dont use channelsdvr ...

what i can say, if the NV cards would have issues overall, the proper NV plugin Thread would be flooded ... which it aint is and there are thousands of users which are using it ;)

so may rather check your diagnostics and may try some other docker to clear out issues ... sample, plex official instead.

very true - i definetly think its specific to something i've done pre unraid-7. Maybe nerd tools or something like referenced on this reddit thread.

https://www.reddit.com/r/unRAID/comments/1ipmobi/unraid_7_nerdtools_nvidia_driver_bug_dockers/

I'm troubleshooting agian right now. I tried reinstalling nerd tools, disabling any enabled features (python, etc), and then deleted nerd tools all together.

I posted the jellyfin ffmpeg logs above, here are the ChannelsDVR ones, just in case someone stumbles upon this thread through similar errors:

[Parsed_testsrc_0 @ 0x2d6e100] size:1280x720 rate:60/1 duration:-1.000000 sar:1/1
Input #0, lavfi, from 'testsrc=size=1280x720:rate=60':
  Duration: N/A, start: 0.000000, bitrate: N/A
  Stream #0:0: Video: wrapped_avframe, 1 reference frame, rgb24, 1280x720 [SAR 1:1 DAR 16:9], 60 fps, 60 tbr, 60 tbn
Stream mapping:
  Stream #0:0 -> #0:0 (wrapped_avframe (native) -> h264 (h264_nvenc))
Press [q] to stop, [?] for help
[graph 0 input from stream 0:0 @ 0x2d91980] w:1280 h:720 pixfmt:rgb24 tb:1/60 fr:60/1 sar:1/1
[auto_scale_0 @ 0x2d93b80] w:iw h:ih flags:'' interl:0
[format @ 0x2d92e80] auto-inserting filter 'auto_scale_0' between the filter 'Parsed__0' and the filter 'format'
[auto_scale_0 @ 0x2d93b80] w:1280 h:720 fmt:rgb24 sar:1/1 -> w:1280 h:720 fmt:gbrp sar:1/1 flags:0x00000004
[h264_nvenc @ 0x2d73100] Cannot load libcuda.so.1
[h264_nvenc @ 0x2d73100] Nvenc unloaded
[vost#0:0/h264_nvenc @ 0x2d72e00] Error initializing output stream: Error while opening encoder for output stream #0:0 - maybe incorrect parameters such as bit_rate, rate, width or height
Terminating demuxer thread 0
Conversion failed!

Currently moving from latest nvidia driver down to production to see if that does anything. Will report back if I find issue.

unraid-diagnostics-20250808-1558.zip