3 hours ago, shremi said:

I captured this yesterday does this work ?

I'm really not sure what's going on since this shouldn't happen at all:

Dec 16 00:03:10 Tower kernel: NVRM: API mismatch: the client 'nvidia-smi' (pid 12017)

Dec 16 00:03:10 Tower kernel: NVRM: has the version 580.119.02, but this kernel module has

Dec 16 00:03:10 Tower kernel: NVRM: the version 580.95.05. Please make sure that this

Dec 16 00:03:10 Tower kernel: NVRM: kernel module and all NVIDIA driver components

Dec 16 00:03:10 Tower kernel: NVRM: have the same version.

Dec 16 00:03:10 Tower kernel: NVRM: API mismatch: the client 'nvidia-smi' (pid 12025)

Dec 16 00:03:10 Tower kernel: NVRM: has the version 580.119.02, but this kernel module has

Dec 16 00:03:10 Tower kernel: NVRM: the version 580.95.05. Please make sure that this

Dec 16 00:03:10 Tower kernel: NVRM: kernel module and all NVIDIA driver components

Dec 16 00:03:10 Tower kernel: NVRM: have the same version.

Do you have any custom scripts in place which replacing nvidia-smi? Or do you maybe have an AI/LXC container which is possibly using another version from nvidia-smi.

It's very strange since it seems you are the only one who has this issue so I assume this has something to do with your machine. Even the post two above yours says that the driver 580.119.02 works just fine.

Hi all!

I am trying to add a 5060ti to my server, next to my 1660super.

After not seeing the card in Unraid, I tried switching to the opensource drivers, but still only the 1660 super shows up:

root@Fenrir:~# nvidia-smi -L

GPU 0: NVIDIA GeForce GTX 1660 SUPER (UUID: GPU-d3aa70ab-7012-fa6d-a9c0-d793b052fba6)

However:

root@Fenrir:~# lspci | grep -i nvidia

01:00.0 VGA compatible controller: NVIDIA Corporation GB206 [GeForce RTX 5060 Ti] (rev a1)

01:00.1 Audio device: NVIDIA Corporation GB206 High Definition Audio Controller (rev a1)

81:00.0 VGA compatible controller: NVIDIA Corporation TU116 [GeForce GTX 1660 SUPER] (rev a1)

81:00.1 Audio device: NVIDIA Corporation TU116 High Definition Audio Controller (rev a1)

81:00.2 USB controller: NVIDIA Corporation TU116 USB 3.1 Host Controller (rev a1)

81:00.3 Serial bus controller: NVIDIA Corporation TU116 USB Type-C UCSI Controller (rev a1)

...not sure what nvidia-persistenced should do, but I tried adding it to my go file, but to no avail.

The 1660super still works, but hopefully someone can help Thanks!

Edited December 18, 2025Dec 18 by MaTi

19 hours ago, ich777 said:

I'm really not sure what's going on since this shouldn't happen at all:

Dec 16 00:03:10 Tower kernel: NVRM: API mismatch: the client 'nvidia-smi' (pid 12017)

Dec 16 00:03:10 Tower kernel: NVRM: has the version 580.119.02, but this kernel module has

Dec 16 00:03:10 Tower kernel: NVRM: the version 580.95.05. Please make sure that this

Dec 16 00:03:10 Tower kernel: NVRM: kernel module and all NVIDIA driver components

Dec 16 00:03:10 Tower kernel: NVRM: have the same version.

Dec 16 00:03:10 Tower kernel: NVRM: API mismatch: the client 'nvidia-smi' (pid 12025)

Dec 16 00:03:10 Tower kernel: NVRM: has the version 580.119.02, but this kernel module has

Dec 16 00:03:10 Tower kernel: NVRM: the version 580.95.05. Please make sure that this

Dec 16 00:03:10 Tower kernel: NVRM: kernel module and all NVIDIA driver components

Dec 16 00:03:10 Tower kernel: NVRM: have the same version.

Do you have any custom scripts in place which replacing nvidia-smi? Or do you maybe have an AI/LXC container which is possibly using another version from nvidia-smi.

It's very strange since it seems you are the only one who has this issue so I assume this has something to do with your machine. Even the post two above yours says that the driver 580.119.02 works just fine.

Thanks for getting back is there a command line i can use to check for this ???

My scripts are only related to backups and file management nothing that touches the GPU.

As for docker containers i only use nvidia for Jellyfin and Frigate , everything else is not gpu bound.

12 hours ago, MaTi said:

I am trying to add a 5060ti to my server, next to my 1660super.

Are you sure that you have Resizable BAR Support, Above 4G Decoding enabled and are on the latest BIOS version?

Dec 17 19:04:19 Fenrir kernel: NVRM: This PCI I/O region assigned to your NVIDIA device is invalid:

Dec 17 19:04:19 Fenrir kernel: NVRM: BAR1 is 0M @ 0x0 (PCI:0000:01:00.0)

Dec 17 19:04:19 Fenrir kernel: nvidia 0000:01:00.0: probe with driver nvidia failed with error -1

Dec 17 19:04:19 Fenrir kernel: NVRM: The NVIDIA probe routine failed for 1 device(s).

Dec 17 19:04:19 Fenrir kernel: NVRM: loading NVIDIA UNIX Open Kernel Module for x86_64 580.95.05 Release Build (dvs-builder@U22-I3-B17-02-5) Tue Sep 23 09:55:41 UTC 2025

24 minutes ago, shremi said:

Thanks for getting back is there a command line i can use to check for this ???

What happens when you issue nvidia-smi from a Unraid terminal?

I just can assume that something is replacing nvidia-smi with an older version since nobody else is having this issue so far.

18 minutes ago, ich777 said:

What happens when you issue nvidia-smi from a Unraid terminal?

I just can assume that something is replacing nvidia-smi with an older version since nobody else is having this issue so far.

right now it is fixed since i moved to a rtx 2070 supper but i plan on using the 1060 again :

yestertay showed like this :

Failed to initialize NVML: Driver/library version mismatch

NVML library version: 590.44

3 minutes ago, shremi said:

Failed to initialize NVML: Driver/library version mismatch

NVML library version: 590.44

That's really strange to me since I always compile from the official driver package and all files should be the same driver version because it's the package from Nvidia and TBH others would have the same issue as you, so my assumption was something is replacing the nvidia-smi library or other libraries on your system.

35 minutes ago, ich777 said:

That's really strange to me since I always compile from the official driver package and all files should be the same driver version because it's the package from Nvidia and TBH others would have the same issue as you, so my assumption was something is replacing the nvidia-smi library or other libraries on your system.

I can reproduce the issue and it’s a kernel/userspace version split, not scripts.
After selecting Production 580.119.02, nvidia-smi and nvidia-container are 580.119.02, but /proc/driver/nvidia/version and modinfo nvidia show the Open Kernel Module 580.95.05 (Dual MIT/GPL) still loaded.
dmesg shows: “API mismatch… client has 580.119.02 but kernel module has 580.95.05”.
Also both NVML libs exist on disk: libnvidia-ml.so.580.95.05 and libnvidia-ml.so.580.119.02 in /usr/lib and /usr/lib64.
So the plugin updates userspace but does not replace/load the matching kernel module when switching from Open Source driver to Production branch.

13 minutes ago, shremi said:

So the plugin updates userspace but does not replace/load the matching kernel module when switching from Open Source driver to Production branch.

Two question, do you reboot when switching versions?

Do you install the package manually?

Sorry but I really can't tell what's wrong since nobody else seem to have that issue and the drivers for Unraid 7.2.2 are downloaded a lot, here are the numbers from GitHub:

name: 6.12.54-Unraid,

name: nvidia-575.64.05-6.12.54-Unraid-1.txz,

download_count: 795,

name: nvidia-580.105.08-6.12.54-Unraid-1.txz,

download_count: 17521,

name: nvidia-580.119.02-6.12.54-Unraid-1.txz,

download_count: 424,

name: nvidia-580.65.06-6.12.54-Unraid-1.txz,

download_count: 124,

name: nvidia-580.95.05-6.12.54-Unraid-1.txz,

download_count: 9395,

name: nvidia-590.44.01-6.12.54-Unraid-1.txz,

download_count: 10315,

name: nvos-580.95.05-6.12.54-Unraid-1.txz,

download_count: 1671,

17 minutes ago, shremi said:

After selecting Production 580.119.02, nvidia-smi and nvidia-container are 580.119.02, but /proc/driver/nvidia/version and modinfo nvidia show the Open Kernel Module 580.95.05 (Dual MIT/GPL) still loaded.
dmesg shows: “API mismatch… client has 580.119.02 but kernel module has 580.95.05”.
Also both NVML libs exist on disk: libnvidia-ml.so.580.95.05 and libnvidia-ml.so.580.119.02 in /usr/lib and /usr/lib64.

I just downloaded the open source driver package for version 580.95.05 and every library is consistned and has only 580.95.05 in them.

I also downloaded the driver package for version 580.119.02 and there are only the libraries for driver 580.119.02 in it.

Then I did the same for the driver package for version 590.44.01 and the libraries match the driver version.

So to speak, I really don't know what the issue on your system is, everything is in the correct spot.

You'll find the packages here if you want to take a look at them yourself:

https://github.com/unraid/unraid-nvidia-driver/releases/tag/6.12.54-Unraid

I really don't know what's going on on your system.

6 minutes ago, ich777 said:

Two question, do you reboot when switching versions?

Do you install the package manually?

Sorry but I really can't tell what's wrong since nobody else seem to have that issue and the driver 580.95.05 für unraid 7.2.2 alread has about 1700 downloads.

I just downloaded the open source driver package for version 580.95.05 and every library is consistned and has only 580.95.05 in them.

I also downloaded the driver package for version 580.119.02 and there are only the libraries for driver 580.119.02 in it.

Then I did the same for the driver package for version 590.44.01 and the libraries match the driver version.

So to speak, I really don't know what the issue on your system is, everything is in the correct spot.

You'll find the packages here if you want to take a look at them yourself:

https://github.com/unraid/unraid-nvidia-driver/releases/tag/6.12.54-Unraid

I really don't know what's going on on your system.

mmmmm is there a safe way to delete everything and reinstall which commands should i use ???

Just now, shremi said:

mmmmm is there a safe way to delete everything and reinstall which commands should i use ???

Just reboot as the driver says.

You haven't answered a single question, you install the packages manually correct?

After inspecting your Diagnostics from a previous post, you have both packages installed:

...
-rw-r--r-- 1 root root 12777 Dec 16 00:01 nvidia-580.119.02-6.12.54-Unraid-1

...

-rw-r--r-- 1 root root 12710 Dec 16 00:01 nvos-580.95.05-6.12.54-Unraid-1
...

which is something the plugin won't even attempt to do.

The plugin will only install one driver and even when changing the driver version the only driver that will be installed is the one which was installed and the new driver will be installed on next reboot, the plugin even tells you to reboot after you changed driver versions, except if you change the driver version to the current installed one.

You don't have to delete anything, just switch to the driver version you want to use in the plugin, wait for the download to finish and reboot after that, as the plugin tells you to.

39 minutes ago, ich777 said:

Just reboot as the driver says.

You haven't answered a single question, you install the packages manually correct?

After inspecting your Diagnostics from a previous post, you have both packages installed:

...
-rw-r--r-- 1 root root 12777 Dec 16 00:01 nvidia-580.119.02-6.12.54-Unraid-1

...

-rw-r--r-- 1 root root 12710 Dec 16 00:01 nvos-580.95.05-6.12.54-Unraid-1
...

which is something the plugin won't even attempt to do.

The plugin will only install one driver and even when changing the driver version the only driver that will be installed is the one which was installed and the new driver will be installed on next reboot, the plugin even tells you to reboot after you changed driver versions, except if you change the driver version to the current installed one.

You don't have to delete anything, just switch to the driver version you want to use in the plugin, wait for the download to finish and reboot after that, as the plugin tells you to.

According to chat gpt :

rm -rf /boot/config/plugins/nvidia-driver/packages/*

rm -rf /boot/config/plugins/nvidia-driver/NVIDIA-OpenKernel*

rm -rf /boot/config/plugins/nvidia-driver/NVIDIA-Driver-Open*

Reinstall nd reboot , this fixed the issue and now can properly download lets see if it survives a reboot

Edited December 18, 2025Dec 18 by shremi

4 minutes ago, shremi said:

According to chat gpt :

rm -rf /boot/config/plugins/nvidia-driver/packages/*

rm -rf /boot/config/plugins/nvidia-driver/NVIDIA-OpenKernel*

rm -rf /boot/config/plugins/nvidia-driver/NVIDIA-Driver-Open*

I really don't know why you did that, that is not necessary, the plugin manages all on it's own, just select the driver you want to have, wait for the download to finish and reboot.

...and as always the AI is mostly wrong with that answer anyways, I really don't understand why you trust a AI agent more than the plugin author. :D

You don't need to do any custom commands or anything, the plugin is designed to be easy to use without any user intervention.

7 minutes ago, shremi said:

Reinstall nd reboot , this fixed the issue and now can properly download lets see if it survives a reboot

This will survive a reboot, I really don't understand why it shouldn't?

Please don't do anything with custom commands or anything, the plugin manages everything on it's own.

15 minutes ago, ich777 said:

I really don't know why you did that, that is not necessary, the plugin manages all on it's own, just select the driver you want to have, wait for the download to finish and reboot.

...and as always the AI is mostly wrong with that answer anyways, I really don't understand why you trust a AI agent more than the plugin author. :D

You don't need to do any custom commands or anything, the plugin is designed to be easy to use without any user intervention.

This will survive a reboot, I really don't understand why it shouldn't?

Please don't do anything with custom commands or anything, the plugin manages everything on it's own.

No no no , why wouldnt i trust you ????/

As i stated , i tried multiple time and even proved with the screenshots selecting the version and waited , then rebooted this didnt worked for me at all i provided screenshots yesterday .

That command i didnt know what it did but it worked i can reboot and have the version i selected

4 hours ago, ich777 said:

Are you sure that you have Resizable BAR Support, Above 4G Decoding enabled and are on the latest BIOS version?

I missed that! Thanks! Now it is working; thanks for your help. Highly appreciated!

For anyone trying this with an Asrock TRX50 WS motherboard:

I had to update my v10.x bios to the latest to date (13.05 at time of writing) and that solved it.

I spent a good 10mins looking for 'BAR support' or '>4G decoding' options, but couldn't find it.

Then I gave up and decided to ask chatgpt.. She told me Asrock had deliberately removed these settings from the BIOS and just enabled them for all ports by default.

I was like "Suuuuure :) ", but decided to try anyway and sure enough:

PS: Do spare yourself a small heart attack when you see your arrays not coming back online:

Ofc the bios update will overwrite settings such as bifurcation on PCIe slots (my 4x nvme adapter) and reset the MCIO ports to the default PCIE, instead of SATA.

All up and running now. Thanks again for helping out. I can now start playing with some LLMs on 16GB VRAM :).

On 12/11/2025 at 6:44 AM, ich777 said:

As said that has nothing to do with System Memory, that refers to Kernel Memory, so to speak reserved memory for the Kernel in System Memory and the message is actually produced by the Nvidia Driver, so nothing that I could do about.

Try it.

As another test, please post nvidia-persistenced in your go file and execute it once (so that you don't have to reboot).

Would a userscript running at boot;

nvidia-smi --persistence-mode=ENABLED

Be the same thing as;

nvidia-persistenced

?

7 hours ago, AnnabellaRenee87 said:

Would a userscript running at boot;

nvidia-smi --persistence-mode=ENABLED

Be the same thing as;

nvidia-persistenced

?

Similar yes, but the old way of doing that and will be eventually be deprecated by Nvidia, please use nvidia-persistenced <- you can also put that in a User Script that runs on boot.

Hello,

I am planning to place a 5060ti 16GB into my unraid build to

- run NVR (agent dvr or frigate) including AI object detection

- use gpu for machine learning tasks in immich

- test some llms locally on the unraid server

I found several posts here and in other places that the 50xx series cards do have some issues with unraid, specifically I have the following questions:

- does the opensource driver work out of the box with the 5060 ti and flawlessly work for my intended uses?

- does the opensource driver have any drawbacks compared to the proprietary one (performance, power states)?

- are future versions of the proprietary driver expected to support the 5060ti on unraid?

I hope this is the right place to place my questions.

9 minutes ago, jakobU02 said:

- does the opensource driver work out of the box with the 5060 ti and flawlessly work for my intended uses?

Maybe, depends on your hardware and you BIOS settings.

Most issues are caused because pairing outdated hardware with 50 series cards, outdated BIOS versions, wrong settings in the BIOS.

9 minutes ago, jakobU02 said:

- does the opensource driver have any drawbacks compared to the proprietary one (performance, power states)?

Irrelevant since 50 series only supports the Open Source driver.

9 minutes ago, jakobU02 said:

- are future versions of the proprietary driver expected to support the 5060ti on unraid?

The Proprietary driver doesn't support 50 series and that has nothing to do with Unraid since this is a limitation from Nvidia.

20 minutes ago, ich777 said:

31 minutes ago, jakobU02 said:

- does the opensource driver have any drawbacks compared to the proprietary one (performance, power states)?

Irrelevant since 50 series only supports the Open Source driver.

Thank you for the quick response. I see. For me it is relevant, since I could still go for a different non-50-series card. Would you recommend to stick to the 5060 + opensource, or go for an older card with proprietary driver support? Is there a documentation site to see what's actually different between proprietary and open source version?

6 hours ago, jakobU02 said:

For me it is relevant, since I could still go for a different non-50-series card.

Why would you do that? If you can get it for cheap or at least for a reasonable price I would always stick to the newer cards.

6 hours ago, jakobU02 said:

Would you recommend to stick to the 5060 + opensource

If you are choosing a 5060 you have no other choice to go with the Open Source driver, but as said above, that always depends on your hardware and you didn't mention yet what hardware that you have nor uploaded Diagnostics, but even then I really can't tell if it works or not.

But I would see it the other way around, the open source driver for 7.2.2 (and 7.2.3) where already downloaded about 1.800 times and only a few people postet here that it won't work and to add to that, most of the issues where solved (mentioned above the issues and solutions), here are the exact downlaod numbers:

name: 6.12.54-Unraid,

name: nvidia-575.64.05-6.12.54-Unraid-1.txz,

download_count: 829,

name: nvidia-580.105.08-6.12.54-Unraid-1.txz,

download_count: 17538,

name: nvidia-580.119.02-6.12.54-Unraid-1.txz,

download_count: 1064,

name: nvidia-580.65.06-6.12.54-Unraid-1.txz,

download_count: 132,

name: nvidia-580.95.05-6.12.54-Unraid-1.txz,

download_count: 9402,

name: nvidia-590.44.01-6.12.54-Unraid-1.txz,

download_count: 10457,

name: nvidia-590.48.01-6.12.54-Unraid-1.txz,

download_count: 3420,

name: nvos-580.95.05-6.12.54-Unraid-1.txz,

download_count: 1776,

6 hours ago, jakobU02 said:

or go for an older card with proprietary driver support?

As said, I wouldn't do it.

6 hours ago, jakobU02 said:

Is there a documentation site to see what's actually different between proprietary and open source version?

No, there is none and the reason is simple, the proprietary driver is closed source, so no one knows what's exactly different except Nvidia, however Nvidia recommends to use the Open Source driver anyways and there you can see how it works since the Kernel module is Open Source, the libraries are still closed source:

GitHub

GitHub - NVIDIA/open-gpu-kernel-modules: NVIDIA Linux ope...

NVIDIA Linux open GPU kernel module source. Contribute to NVIDIA/open-gpu-kernel-modules development by creating an account on GitHub.

I see. So, provided the mainboard/bios (Asus proart creator B650) and cpu (AMD EPYC 4345P) are compatible, chances are good to get cuda and hardware encoding running in the dockers when using the open source driver. Is this statement correct?

Hello!

I keep getting this code reappearing in the syslog. It makes my nvidia card disappear and reappear every so often in my Unraid. Not sure what the cause is. This has persisted across different drivers and even different gpus (a 5050 with the open source driver). Not sure what the issue is or what the fix is:

ec 19 19:39:44 Dingleberry kernel: NVRM: GPU 0000:2b:00.0: RmInitAdapter failed! (0x22:0x51:884)
Dec 19 19:39:44 Dingleberry kernel: NVRM: GPU 0000:2b:00.0: rm_init_adapter failed, device minor number 0

I have attached the syslog and diag as well. Please let me know any thoughts you guys have. Thank you!

dingleberry-syslog-20251220-1909.zip dingleberry-diagnostics-20251220-1410.zip

On 12/14/2025 at 9:57 PM, ich777 said:

You'll also see that in the syslog, no dmesg needed, you have a Xid 119 error, please see here.

I'm not sure, that could be also be a driver issue. Have you yet tried to reseat the card, please also check your Power Supply if it's up to the task, don't forget that power supplies also degrade over time.

Maybe also try to set the PCIe Generation to something like Gen 4 or Gen 3 and see if that helps.

I tried everything but the forced GSP was not fixable...
I know opt for a new GPU and found the Quadro RTX 2000 Ada (not the A2000), couldn't find many reports of how stable this GPU is under Unraid, is there any site i could check that?

I worry about this GPU not being able to work with my MB / CPU on unraid too

Hello - after a recent update to 7.2.3 (from 7.2.2) i started getting this error. Diagnostics attached.

Can anyone help resolve this please?

Nvidia Info:

Nvidia Driver Version: 590.48.01

Open Source Kernel Module: No

Installed GPU(s):
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

GPU does show up still in the system devices, but I do get a notification saying there was an error and asking me to install the driver manually.

Nvidia Driver

Dec 21, 09:01 AM

Notification

Found new Nvidia Driver v590.48.01 but a download error occurred! Please try to download the driver manually!

OMMU group 1:				[10de:1c31] b3:00.0 VGA compatible controller: NVIDIA Corporation GP106GL [Quadro P2200] (rev a1)
				[10de:10f1] b3:00.1 Audio device: NVIDIA Corporation GP106 High Definition Audio Controller (rev a1)

tower-diagnostics-20251221-1902.zip