5 hours ago, Corvus said:

This problem has been going for MONTHS and it's driving me insane because just when I think I've fixed it, it happens again.

First of all, why:

Dec 17 14:55:33 NAS emhttpd: /usr/local/emhttp/plugins/user.scripts/backgroundScript.sh "/tmp/user.scripts/tmpScripts/Nvidia Patch/script" >/dev/null 2>&1

Please read the first post and especially the red text, you are modifying the driver and I can't give support for that and in reality how much concurrent streams do you have which need to be transcoded?

This is the real issue:

Dec 24 00:06:15 NAS kernel: NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x62:0x51:2584)

Dec 24 00:06:15 NAS kernel: NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0

Dec 24 00:06:17 NAS kernel: NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x62:0x51:2584)

Dec 24 00:06:17 NAS kernel: NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0

Dec 24 00:06:19 NAS kernel: NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x62:0x51:2584)

Dec 24 00:06:19 NAS kernel: NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0

Dec 24 00:06:21 NAS kernel: NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x62:0x51:2584)

Dec 24 00:06:21 NAS kernel: NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0

I can't tell you why that is caused (maybe because of your patch script) however I would recommend that you do the following:

1. Check if your BIOS is up to date

2. Check if you have Above 4G Decoding enabled

3. Check if you have Resizable BAR support enabled

4. Use that as your startup script instead of patching the driver:
   nvidia-persistenced

However I really don't see why you are using a Nvidia for transcoding, you have a Intel iGPU in your system which is probably well enough for you to transcode (Intel has no hard limit of simultaneous transcodes BTW).

20 hours ago, ich777 said:

First of all, why:

Dec 17 14:55:33 NAS emhttpd: /usr/local/emhttp/plugins/user.scripts/backgroundScript.sh "/tmp/user.scripts/tmpScripts/Nvidia Patch/script" >/dev/null 2>&1

Please read the first post and especially the red text, you are modifying the driver and I can't give support for that and in reality how much concurrent streams do you have which need to be transcoded?

This is the real issue:

Dec 24 00:06:15 NAS kernel: NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x62:0x51:2584)

Dec 24 00:06:15 NAS kernel: NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0

Dec 24 00:06:17 NAS kernel: NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x62:0x51:2584)

Dec 24 00:06:17 NAS kernel: NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0

Dec 24 00:06:19 NAS kernel: NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x62:0x51:2584)

Dec 24 00:06:19 NAS kernel: NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0

Dec 24 00:06:21 NAS kernel: NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x62:0x51:2584)

Dec 24 00:06:21 NAS kernel: NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0

I can't tell you why that is caused (maybe because of your patch script) however I would recommend that you do the following:

1. Check if your BIOS is up to date

2. Check if you have Above 4G Decoding enabled

3. Check if you have Resizable BAR support enabled

4. Use that as your startup script instead of patching the driver:
   nvidia-persistenced

However I really don't see why you are using a Nvidia for transcoding, you have a Intel iGPU in your system which is probably well enough for you to transcode (Intel has no hard limit of simultaneous transcodes BTW).

1. Why what? Sorry I'm a newbie at this. I'm assuming you mean that I modified the nvidia plugin somehow? That's impossible as I wouldn't even know how to do that. Unless I did so unwittingly by following someone else's advice. How would I undo this?
   
   2. I have a large number of users. Sometimes up to 8-10 streams needing transcoding at once.

3. BIOS is up to date. 'Above 4G decoding' is enabled in BIOS (unless I need to enable it elsewhere?). What is 'resizable BAR support'? Use what as my startup script? What does typing 'nvidia-persistenced' in terminal do? Sorry, again I'm very unfamiliar with Linux so I'll need much more detailed instructions please.
   
   4. I run a couple of VMs 24/7 (No GPUs passed through) and I don't want to eat into the performance of my CPU, so I'm using the GPU for transcoding as I had it lying around. Additionally, I also use it for running my own 24/7 IPTV channels in another docker container.

What should I be doing to fix this?

Edited December 27, 2025Dec 27 by Corvus

6 hours ago, Corvus said:

Why what? Sorry I'm a newbie at this. I'm assuming you mean that I modified the nvidia plugin somehow? That's impossible as I wouldn't even know how to do that. Unless I did so unwittingly by following someone else's advice. How would I undo this?

I don't know, you have a patch script in place which I don't support and which could be the culprit from your issue. Maybe a user script?

6 hours ago, Corvus said:

2. I have a large number of users. Sometimes up to 8-10 streams needing transcoding at once.

The iGPU from your CPU should be more than be capable of such a task.

6 hours ago, Corvus said:

BIOS is up to date. 'Above 4G decoding' is enabled in BIOS (unless I need to enable it elsewhere?). What is 'resizable BAR support'? Use what as my startup script? What does typing 'nvidia-persistenced' in terminal do? Sorry, again I'm very unfamiliar with Linux so I'll need much more detailed instructions please.

Above 4G decoding needs to be enabled in your BIOS including Resizable BAR Support <- this is also a setting in your BIOS.

You can define a User Script which runs on boot with the contents:
nvidia-persistenced

This will ensure that your card is constantly in a low power state (at least when it's not in use) and hopefully not fail to initialize.

6 hours ago, Corvus said:

4. I run a couple of VMs 24/7 (No GPUs passed through) and I don't want to eat into the performance of my CPU, so I'm using the GPU for transcoding as I had it lying around. Additionally, I also use it for running my own 24/7 IPTV channels in another docker container.

Your iGPU on the CPU is dedicated from the CPU and the CPU will be not impacted or better speaking will not be affected by the load from transcoding if you use the iGPU.

Oh, please also check your Power Supply, they degrade over time and if a power spike happens because a HDD spins up and transcoding starts it is possible that they fail to deliver "enough" power and such issues can happen.

Hi,

Santa brought me a little 5060ti to help me craate useless IA images but I'm facing issues.

I installed opensource nvidia drivers to replace my old official nvidia drivers used for my previous 1650. But I can't do anything because the card is flip-flapping with theses messages :

Dec 27 13:31:31 Tower kernel: NVRM: RmInitAdapter: osInitNvMapping failed, bailing out of RmInitAdapter

Dec 27 13:31:31 Tower kernel: NVRM: GPU 0000:02:00.0: RmInitAdapter failed! (0x22:0x51:763)

Dec 27 13:31:31 Tower kernel: NVRM: GPU 0000:02:00.0: rm_init_adapter failed, device minor number 0

Dec 27 13:31:41 Tower kernel: NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from memdescAllocInternal(pMemDesc) @ memdesc.c:1359

Dec 27 13:32:07 Tower kernel: NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from memdescAllocInternal(pMemDesc) @ memdesc.c:1359

Dec 27 13:32:07 Tower kernel: NVRM: nvAssertOkFailedNoLog: Assertion failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from nvStatus @ message_queue_cpu.c:242

Dec 27 13:32:07 Tower kernel: NVRM: _kgspInitRpcInfrastructure: GspMsgQueueInit failed

Dec 27 13:32:07 Tower kernel: NVRM: kgspConstructEngine_IMPL: init RPC infrastructure failed

I read this forum for a solution but nothing seems to work. If some kind guy could help me or give me some hints, I would be very grateful.

tower-diagnostics-20251227-1321.zip

4 minutes ago, Astrayel said:

I read this forum for a solution but nothing seems to work. If some kind guy could help me or give me some hints, I would be very grateful.

A bit more in detail please, what did you try yet?
Did you also try to add:

nvidia-persistenced to your go file or as a startup script?

Also what is this in your go file:
sysctl vm.overcommit_memory=1

I mean it won't change much but it certainly won't help your issue here.

Please make sure to enable Above 4G Decoding and also to enable Resizable BAR Support.

Make also sure that you are on the latest BIOS version.

40 minutes ago, ich777 said:

A bit more in detail please, what did you try yet?
Did you also try to add:

nvidia-persistenced to your go file or as a startup script?

Also what is this in your go file:
sysctl vm.overcommit_memory=1

I mean it won't change much but it certainly won't help your issue here.

Please make sure to enable Above 4G Decoding and also to enable Resizable BAR Support.

Make also sure that you are on the latest BIOS version.

I checked BIOS and Above 4G decoding does not exist anymore on my Z890. But Resizable BAR was already enabled.

I removed the sysctl. It's been here for a long time but I can't remember why I put it.

I added the nvidia-persistenced and I'm waiting but it looks like it did the trick.

Thanks for all. I'll come back if errors come back.

33 minutes ago, Astrayel said:

I checked BIOS and Above 4G decoding does not exist anymore on my Z890

This could also be something like: Huge address table, Support for lage address or something like that.

2 hours ago, ich777 said:

I don't know, you have a patch script in place which I don't support and which could be the culprit from your issue. Maybe a user script?

I have no idea. How do I remove it?

2 hours ago, ich777 said:

The iGPU from your CPU should be more than be capable of such a task.

Since I use the iGPU for my VMs, I still don't think this is a good idea. I also have a few AI object recognition dockers which utilize nvidia CUDA cores. Unless there's an alternative I'm missing?

2 hours ago, ich777 said:

You can define a User Script which runs on boot with the contents:
nvidia-persistenced

This will ensure that your card is constantly in a low power state (at least when it's not in use) and hopefully not fail to initialize.

So I just type that command into terminal, and that's it?

Does this remove the patch script you're seeing above?

2 hours ago, ich777 said:

Your iGPU on the CPU is dedicated from the CPU and the CPU will be not impacted or better speaking will not be affected by the load from transcoding if you use the iGPU.

Since I need the dedicated GPU for other dockers (not used simultaneously), I would still like to solve this problem.

2 hours ago, ich777 said:

Oh, please also check your Power Supply

How would I go about checking for an intermittent fault such as this?

I think I should rule out the easy fixes first, starting with this custom script you mentioned. Could you please tell me how to remove it?

1 hour ago, Corvus said:

I have no idea. How do I remove it?

Sorry, can't help with that, I don't know how you've set that up, but most definitely in User Scripts I assume.

1 hour ago, Corvus said:

Since I use the iGPU for my VMs, I still don't think this is a good idea. I also have a few AI object recognition dockers which utilize nvidia CUDA cores. Unless there's an alternative I'm missing?

Always depends, if there is a OpenVINO variant from the container available probably.

1 hour ago, Corvus said:

So I just type that command into terminal, and that's it?

Does this remove the patch script you're seeing above?

Nope, you have to actively remove your old script, I really can't tell you how or where but most certainly in User Scripts.

You also have to set up a new user script that runs once at startup with the contents:

#!/bin/bash
nvidia-persistenced

1 hour ago, Corvus said:

How would I go about checking for an intermittent fault such as this?

How old is your power supply, what power supply do you have, is the power supply sufficient in it's base configuration for the hardware that you are using?

1 hour ago, Corvus said:

I think I should rule out the easy fixes first, starting with this custom script you mentioned. Could you please tell me how to remove it?

Please ask in the User Script support thread or just look on the server for this plugin.

However most questions are not related to the Nvidia Driver Plugin and I can only give support for the driver plugin itself if something isn't working, especially if you are using the patch script, again, please read the first post in this thread on top the red text.

10 hours ago, ich777 said:

Sorry, can't help with that, I don't know how you've set that up, but most definitely in User Scripts I assume.

Ok I just found what it is. Apparently it was recommended by SpaceInvader to remove the nvidia-imposed limit on the number of simultaneous transcoding streams. Is this something that is no longer required for me to get unlimited simultaneous transcodes? Happy to remove this if so.

10 hours ago, ich777 said:

if there is a OpenVINO variant from the container available probably.

There isn't, which is why I rely on using the GPU for dockers.

10 hours ago, ich777 said:

How old is your power supply, what power supply do you have

It's a Corsair CXM 750 W 80+. More than enough for my config + drives, according to pcpartpicker.

Hi, I’m joining the club! :)
I’m seeing the same NVRM errors with an NVIDIA RTX 5060 Ti 16GB on an ASRock Z790 Pro RS/D4.

Hardware / SW:

- Unraid: 7.2.3

- NVIDIA driver: 580.95.05 (open kernel modules)

- Motherboard: ASRock Z790 Pro RS/D4

- CPU: i5-14600K

- RAM: 96GB DDR4

- GPU: RTX 5060 Ti 16GB

Symptoms:

- GPU sometimes works (nvidia-smi OK, can run compute like llama.cpp),

but then syslog starts spamming NVRM errors and the driver fails to attach/init the GPU.

Example log:

Dec 28 14:46:06 kernel: NVRM: nvAssertOkFailedNoLog: Assertion failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from nvStatus @ message_queue_cpu.c:242

Dec 28 14:46:06 kernel: NVRM: _kgspInitRpcInfrastructure: GspMsgQueueInit failed

Dec 28 14:46:06 kernel: NVRM: kgspConstructEngine_IMPL: init RPC infrastructure failed

Dec 28 14:46:06 kernel: NVRM: osInitNvMapping: *** Cannot attach gpu

Dec 28 14:46:06 kernel: NVRM: RmInitAdapter: osInitNvMapping failed, bailing out of RmInitAdapter

Dec 28 14:46:06 kernel: NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x22:0x51:763)

BIOS / platform notes:

- Above 4G Decoding: Enabled

- Resizable BAR (C.A.M.): Enabled

- VT-d: Enabled

- PCIe ASPM: Disabled

- Slot is stable.

I’ve tried the common suggested toggles/settings for days and nothing has reliably fixed it. At this point I’m not too bothered, but if anyone has a new angle or a known workaround, I’m all ears. :)

6 minutes ago, Flashman007 said:

At this point I’m not too bothered, but if anyone has a new angle or a known workaround, I’m all ears. :)

Did you also add nvidia-persistenced to a startup script and are you sure that you are booting in UEFI instead of Legacy?

Thanks for the reply. Yes to both. Also, I’m on the latest BIOS.

1 hour ago, Flashman007 said:

Thanks for the reply. Yes to both. Also, I’m on the latest BIOS.

Sorry, then I can't help, you can create a post on the Nvidia Developer forums about the issue since there are many users on there which having the same issue, but as said, I already suffered from the same issue on my Debian Desktop machine (Intel 12th gen) and once I reset the BIOS to defaults everything was starting to work again, not sure which setting causes this.

20 hours ago, Corvus said:

Ok I just found what it is. Apparently it was recommended by SpaceInvader to remove the nvidia-imposed limit on the number of simultaneous transcoding streams. Is this something that is no longer required for me to get unlimited simultaneous transcodes? Happy to remove this if so.

There isn't, which is why I rely on using the GPU for dockers.

It's a Corsair CXM 750 W 80+. More than enough for my config + drives, according to pcpartpicker.

Just a follow-up on this to see what I need to do with this user script -- or if it's even necessary anymore to get unlimited transcodes.

1 hour ago, Corvus said:

Just a follow-up on this to see what I need to do with this user script -- or if it's even necessary anymore to get unlimited transcodes.

I believe that the GTX1660 can now do 8 concurrent transcodes. Back when SpaceInvader One posted that video, NVidia drivers had it throttled to only 2 transcodes at a time.

You put ich777 in a bad spot. If you read the opening posts in this thread, he clearly states that he cannot assist anyone using this patch. It violates the Terms of Agreement from Nvidia, whom take their ToA very seriously. Which could actually put both him and Limetech at legal and financial risk.

Regarding power supplies - Your supply may be more than enough for your system if it is functioning correctly. They do fail over time, and not only catastrophically. Most of the time, they start being flaky (or to be more exact, the system they power has intermittent issues). I have seen several systems do this - many hours of troubleshooting, only to be resolved by replacing the power supply. Failing capacitors, noisy voltages cause strange things to happen, especially as computer semiconductors get physically smaller and power efficient.

Other possible issues could be the power you plug your PC into. Noise on the A/C power, improper grounding, floating neutral wire. Or it could be an issue with the NVidia driver code itself. ich777 cannot do anything about that. He generously compiles a package (and the mechanism and GUI to utilize it in Unraid) and offers his knowledge and support for this and literally hundreds of other plugins and Docker containers. Not for money or fame, but because he is genuinely a nice guy (as far as I have observed over the past few years).

Edited December 29, 2025Dec 29 by ConnerVT

1 hour ago, ConnerVT said:

I believe that the GTX1660 can now do 8 concurrent transcodes. Back when SpaceInvader One posted that video, NVidia drivers had it throttled to only 2 transcodes at a time.

Thanks, that should be more than enough for 95% of the time. At the very least, Plex can fall back to the iGPU.

1 hour ago, ConnerVT said:

You put ich777 in a bad spot. If you read the opening posts in this thread, he clearly states that he cannot assist anyone using this patch. It violates the Terms of Agreement from Nvidia, whom take their ToA very seriously. Which could actually put both him and Limetech at legal and financial risk.

I apologize and that's not my intention. If anything, I'm asking on advice in removing the patch. Now that I know the information above, I can do this confidently.

1 hour ago, ConnerVT said:

Regarding power supplies - Your supply may be more than enough for your system if it is functioning correctly. They do fail over time

I understand this, although since replacing a PSU is a major job (especially in the cramped case my system is in), I want to rule out software fixes first beforehand.

My only two cents to throw my hand into some of this....

Look into system Linux grub commands that turn off and affect ASPM.

Things like:

nvidia.NVreg_ResmanDebugLevel=255 modprobe.blacklist=nouveau

pci=noaer pcie_aspm=off

As usual find it to be ASPM power related due to sleep states...

More from eol device, new Linux kernel hardening and v550 Nvidia LTS 16 set reaching EOL....

So I'm not sure 10000 series cards will be available past kernel 6.14 with EOL or v500 coming up...

https://www.google.com/url?sa=t&source=web&rct=j&opi=89978449&url=https://www.nvidia.com/en-us/networking/products/lts-releases/&ved=2ahUKEwi-jbGdheORAxXK1fACHSWHPdcQFnoECBgQAQ&usg=AOvVaw2CEpbS-yawfaoAt8LsFcB6

The NVIDIA GTX 16-series GPUs have been completely discontinued from production as of early 2024, and remaining market inventory is being exhausted. However, official driver support is ongoing.

Driver and Software End of Life (EOL)

GeForce GTX 16-series (Consumer GPUs): These cards are still receiving regular Game Ready driver updates. NVIDIA has shifted its focus to the RTX brand, but the 16-series (Turing architecture without RT/Tensor cores) is not yet in a legacy support phase like the older GTX 10-series.

NVIDIA vGPU Software 16 Releases (Enterprise Software): The specific vGPU Software 16 Long-Term Support (LTS) branch is supported until July 2026.

Key Dates for Related Products

Product/Software Branch Status/Action End of Life (EOL) Date

GTX 16-series GPUs Production discontinued Early 2024

vGPU Software 16 LTS Branch Supported branch July 2026

GTX 10-series (Pascal), 900-series (Maxwell) GPUs Transitioning to legacy/quarterly security updates Regular driver support ends October 2025

NVIDIA Data Center GPU Drivers 550 family EOL reached June 2025

For the latest official information on specific software lifecycles, you can refer to the NVIDIA Long-Term Support Releases page or the NVIDIA Driver End of Life tracker.

NVIDIA Virtual GPU (vGPU) Software

Aug 4, 2025 — NVIDIA vGPU Software 16 Releases. Branch status: Long-Term Support Branch supported until July 2026.

Sources in NVIDIA Docs...

NVIDIA to End GeForce Regular Driver Support for Maxwell, Volta, and Pascal ...

Jul 30, 2025 — NVIDIA is planning to retire regular driver updates for its older, pre-RTX GPU generations after October 2025, the company announced. These include ...

Sources in TechPowerUp

Per following Nvidia I would expect 10 series and 20 series to end drivers support in future kernel past 6.14 come July 2026 with v16 LTS leaving the Nvidia Enterprise space... (Similar thing happens wo the gyx 780 series. Nothing wrong with hardware software to new and doesn't complie and latest kerenl

1 hour ago, bmartino1 said:

Look into system Linux grub commands that turn off and affect ASPM.

Please can you not give wrong advice, most of the users have disabled ASPM if you look through the last diagnostics.

1 hour ago, bmartino1 said:

nvidia.NVreg_ResmanDebugLevel=255

Why would you set this to 255? This parameter usually allows an integer between 0-3 and IIRC -1 for complete verbosity.

1 hour ago, bmartino1 said:

modprobe.blacklist=nouveau

This should be done via a modprobe file, however nouveau should by default be blacklisted on Unraid.

1 hour ago, bmartino1 said:

pci=noaer

Please only disable this if you have issues with Advance Error Reporting.

1 hour ago, bmartino1 said:

https://www.google.com/url?sa=t&source=web&rct=j&opi=89978449&url=https://www.nvidia.com/en-us/networking/products/lts-releases/&ved=2ahUKEwi-jbGdheORAxXK1fACHSWHPdcQFnoECBgQAQ&usg=AOvVaw2CEpbS-yawfaoAt8LsFcB6

Sorry but is this AI generated? This talks about Networking LTS support from Nvidia, not Graphics.

1 hour ago, bmartino1 said:

The NVIDIA GTX 16-series GPUs have been completely discontinued from production as of early 2024, and remaining market inventory is being exhausted. However, official driver support is ongoing.

Driver and Software End of Life (EOL)

GeForce GTX 16-series (Consumer GPUs): These cards are still receiving regular Game Ready driver updates. NVIDIA has shifted its focus to the RTX brand, but the 16-series (Turing architecture without RT/Tensor cores) is not yet in a legacy support phase like the older GTX 10-series.

NVIDIA vGPU Software 16 Releases (Enterprise Software): The specific vGPU Software 16 Long-Term Support (LTS) branch is supported until July 2026.

Key Dates for Related Products

Product/Software Branch Status/Action End of Life (EOL) Date

GTX 16-series GPUs Production discontinued Early 2024

vGPU Software 16 LTS Branch Supported branch July 2026

GTX 10-series (Pascal), 900-series (Maxwell) GPUs Transitioning to legacy/quarterly security updates Regular driver support ends October 2025

NVIDIA Data Center GPU Drivers 550 family EOL reached June 2025

For the latest official information on specific software lifecycles, you can refer to the NVIDIA Long-Term Support Releases page or the NVIDIA Driver End of Life tracker.

NVIDIA Virtual GPU (vGPU) Software

Aug 4, 2025 — NVIDIA vGPU Software 16 Releases. Branch status: Long-Term Support Branch supported until July 2026.

It would be way easier to write:
GTX16 series and up will still receive regular driver updates

GTX10 and GTX900 series will be converted to legacy branch and will be supported for at least a year or two (the last legacy driver, 470 driver series, supported Kepler and Maxwell for about 4 years)

You always have to take into consideration if you write EOL, most users would think they have to immediately have to buy a new GPU which is most of the times not true.

1 hour ago, bmartino1 said:

Jul 30, 2025 — NVIDIA is planning to retire regular driver updates for its older, pre-RTX GPU generations after October 2025, the company announced. These include ...

The key word is regular driver updates, they will still receive driver updates (security patches and compatibility updates) in the legacy branch.

Again, you can't know what Nvidia ever does, first they said they will not support GTX10 series in 580 driver but now it seems that Nvidia makes the 580 driver series to the new legacy branch for these cards.

Time will tell.

I've been struggling with this for literal days now, and I'm losing my mind trying to get nvidia support in docker. It worked fine for several weeks, and then I rebooted after (I believe) it automatically downloaded the "latest" 590 updated driver, which broke compatibility with my GTX 1050 Ti. I would prefer to keep using this card for transcoding and similar...

Now whenever I try and run any container that requires the GPU, I get the following error(s):

# docker run --rm --runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=GPU-c96d3b5a-bdb2-4b41-b817-22755b00d28d -e NVIDIA_DRIVER_CAPABILITIES=all ubuntu nvidia-smi
docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running prestart hook #1: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
nvidia-container-cli: initialization error: load library failed: libnvidia-ml.so.1: cannot open shared object file: permission denied: unknown.

# nvidia-container-cli -d /dev/tty info
-- WARNING, the following logs are for debugging purposes only --
I1230 04:18:39.723995 107649 nvc.c:396] initializing library context (version=1.17.9~1.17.9, build=e78456a41c71d73eabf7a85d1e531a108789349b)
I1230 04:18:39.724072 107649 nvc.c:367] using root /
I1230 04:18:39.724080 107649 nvc.c:368] using ldcache /etc/ld.so.cache
I1230 04:18:39.724098 107649 nvc.c:369] using unprivileged user 65534:65534
I1230 04:18:39.724165 107649 nvc.c:413] attempting to load dxcore to see if we are running under Windows Subsystem for Linux (WSL)
I1230 04:18:39.724260 107649 nvc.c:415] dxcore initialization failed, continuing assuming a non-WSL environment
I1230 04:18:39.724480 107650 rpc.c:71] starting driver rpc service
I1230 04:18:39.724951 107649 rpc.c:135] driver rpc service terminated with signal 15
nvidia-container-cli: initialization error: load library failed: libnvidia-ml.so.1: cannot open shared object file: permission denied
I1230 04:18:39.724997 107649 nvc.c:455] shutting down library context

However nvidia-smi works fine on the unraid host, and the GPU is detected and loaded by the driver. GPU stats shows the card on the dashboard, and as far as I can tell, everything should be working.

/usr/lib/libnvidia-ml.so.1 exists and has 0755 permissions

/usr/lib64/libnvidia-ml.so.1 exists and has 0755 permissions

But as soon as I try and run it in a container... load library failed: libnvidia-ml.so.1: permission denied: unknown

Things I've tried:

• Downgrading the driver to all 580.* versions and the single listed 575.* version

• Downgrading Unraid to 7.2.0 (where it did seem to be working for a while)

• Removing the plugin entirely, rebooting, reinstalling the plugin and driver

• Adjusting various settings in /etc/nvidia-container-toolkit/config.toml (including no-cgroups=true/false), debug logging, ldstore/ldconfig settings, etc.

• Running containers with various combinations of --runtime=nvidia and --gpus=all flags, as well as visible devices and driver capabilities environment variables

• Various permissions on the libraries (0777/0644, etc)

• Various users:groups for nvidia-container-runtime (65564:65564)

• nvidia-ctk runtime configure

• Adding nvidia-persistenced to startup scripts

• Full path/implicit path to the runtime in /etc/docker/daemon.json

• Rebooting, rebooting, rebooting

• restarting the docker service over and over and over

I would appreciate any insight into the issue that anyone can provide. I'm about to give up entirely.

vault-diagnostics-20251229-2006.zip

2 hours ago, t3rminus said:

I've been struggling with this for literal days now, and I'm losing my mind trying to get nvidia support in docker. It worked fine for several weeks

First of all in which containers do you try to run your GPU?

2 hours ago, t3rminus said:

then I rebooted after (I believe) it automatically downloaded the "latest" 590 updated driver, which broke compatibility with my GTX 1050 Ti

Exactly, Pascal, so to speak GTX 10 series isn't supported anymore with the 590 driver series, please use the 580 driver version.

2 hours ago, t3rminus said:

/usr/lib/libnvidia-ml.so.1 exists and has 0755 permissions

/usr/lib64/libnvidia-ml.so.1 exists and has 0755 permissions

Please don't change that.

However from what I see, one container is definitely working since you get this in your logs:

Dec 29 19:59:30 Vault kernel: nvidia_uvm: module uses symbols nvUvmInterfaceDisableAccessCntr from proprietary module nvidia, inheriting taint.

Are you really sure that the ubuntu container that you are trying to use supports Nvidia at all?

Not all containers work with Nvidia or at least can cause trouble.

Please also use a more recent driver than 575.x for testing.

I would also recommend that you do the following:

1. Remove the Nvidia Driver plugin

2. Reboot

3. Install the Nvidia Driver plugin

4. Got to the plugin page and select Driver version 590.x

5. Click "Update & Download"

6. Reboot

7. Test it in a container which is known to work like Jellyfin or Plex

2 hours ago, t3rminus said:

1. Adjusting various settings in /etc/nvidia-container-toolkit/config.toml (including no-cgroups=true/false), debug logging, ldstore/ldconfig settings, etc.

2. Running containers with various combinations of --runtime=nvidia and --gpus=all flags, as well as visible devices and driver capabilities environment variables

3. Various permissions on the libraries (0777/0644, etc)

4. Various users:groups for nvidia-container-runtime (65564:65564)

5. nvidia-ctk runtime configure

6. Adding nvidia-persistenced to startup scripts

7. Full path/implicit path to the runtime in /etc/docker/daemon.json

8. Rebooting, rebooting, rebooting

9. restarting the docker service over and over and over

To all of that, please don't do anything manually, the plugin does everything for you.

The plugin is working for at least 10k users if not more, I don't have the exact numbers in my head.

Just as a precaution, please also make sure that Resizable BAR support and Above 4G Decoding is enabled in your BIOS.

19 minutes ago, ich777 said:

First of all in which containers do you try to run your GPU?

Ultimately, I will be using Jellyfin, and beszel-agent-nvidia to monitor the system. Both were confirmed working prior to this.

19 minutes ago, ich777 said:

Exactly, Pascal, so to speak GTX 10 series isn't supported anymore with the 590 driver series, please use the 580 driver version.

Noted. I will use the v580

19 minutes ago, ich777 said:

Please don't change that.

I wouldn't, other than as a sanity check. I'm mostly trying to demonstrate it's not actually a permissions issue, despite the logs saying "Permission denied"

19 minutes ago, ich777 said:

However from what I see, one container is definitely working since you get this in your logs:

Dec 29 19:59:30 Vault kernel: nvidia_uvm: module uses symbols nvUvmInterfaceDisableAccessCntr from proprietary module nvidia, inheriting taint.

Are you really sure that the ubuntu container that you are trying to use supports Nvidia at all?

Not all containers work with Nvidia or at least can cause trouble.

The ubuntu docker container is the documented way to test nvidia-container-toolkit, per NVidia's documentation:
https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/1.17.6/sample-workload.html

19 minutes ago, ich777 said:

Please also use a more recent driver than 575.x for testing.

I would also recommend that you do the following:

1. Remove the Nvidia Driver plugin

2. Reboot

3. Install the Nvidia Driver plugin

4. Got to the plugin page and select Driver version 590.x

5. Click "Update & Download"

6. Reboot

7. Test it in a container which is known to work like Jellyfin or Plex

I assume you mean v580.x considering my card wouldn't be supported on v590.x?

I have completed the following steps:

1. Remove the Nvidia Driver plugin

2. Reboot

3. Install the Nvidia Driver plugin

4. Go to the plugin page and select Driver version v580.119.02

5. Click "Update & Download"

6. Reboot

7. Configure Jellyfin per the first post:

   docker run
     -d
     --name='Jellyfin'
     --net='bridge'
     --pids-limit 2048
     -e TZ="America/Los_Angeles"
     -e HOST_OS="Unraid"
     -e HOST_HOSTNAME="Vault"
     -e HOST_CONTAINERNAME="Jellyfin"
     -e 'NVIDIA_VISIBLE_DEVICES'='GPU-c96d3b5a-bdb2-4b41-b817-22755b00d28d'
     -e 'NVIDIA_DRIVER_CAPABILITIES'='all'
     -e 'PUID'='99'
     -e 'PGID'='100'
     -e 'UMASK'='022'
     -l net.unraid.docker.managed=dockerman
     -l net.unraid.docker.webui='http://[IP]:[PORT:8096]'
     -l net.unraid.docker.icon='https://raw.githubusercontent.com/linuxserver/docker-templates/master/linuxserver.io/img/jellyfin-logo.png'
     -p '8096:8096/tcp'
     -p '7359:7359/udp'
     -v '/mnt/user/TV/':'/data/tvshows':'rw'
     -v '/mnt/user/Movies/':'/data/movies':'rw'
     -v '/mnt/cache/appdata/Jellyfin/cache':'/cache':'rw'
     -v '/mnt/user/appdata/Jellyfin':'/config':'rw'
     --runtime=nvidia 'jellyfin/jellyfin:latest'
   
   7fba60687a82316ca2dcb78868ea64ac4f3f6a16a8c8f41306278774d3df934c
   docker: Error response from daemon: failed to create task for container: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error running prestart hook #1: exit status 1, stdout: , stderr: Auto-detected mode as 'legacy'
   nvidia-container-cli: initialization error: load library failed: libnvidia-ml.so.1: cannot open shared object file: permission denied: unknown.
   
   The command failed.

19 minutes ago, ich777 said:

To all of that, please don't do anything manually, the plugin does everything for you.

The plugin is working for at least 10k users if not more, I don't have the exact numbers in my head.

Just as a precaution, please also make sure that Resizable BAR support and Above 4G Decoding is enabled in your BIOS.

I updated my BIOS/Firmware in the process of troubleshooting, entirely to add options for Resizable BAR and Above 4G Decoding. I can confirm they are both enabled.

1 hour ago, t3rminus said:

I assume you mean v580.x considering my card wouldn't be supported on v590.x?

Sorry my bad.

1 hour ago, t3rminus said:

I updated my BIOS/Firmware in the process of troubleshooting, entirely to add options for Resizable BAR and Above 4G Decoding. I can confirm they are both enabled.

Please post your Diagnostics again so that I can see what's going on and what the Nvidia driver does.

Here are my current diagnostics.

vault-diagnostics-20251230-0912.zip

54 minutes ago, t3rminus said:

Here are my current diagnostics.

Please try to disable you iGPU from your Ryzen CPU.

The Nvidia Driver, at least it seems from the Diagnostics, works just fine.

The error message is very strange since the library is basically there but why it fails is unknown, this is a really strange issue.

I really don't know what causes this on your system.