Howdy

I'm seeing NVRM Out of Memory lines in my dmesg using a 5070ti + latest open source driver:

[Fri Jan 23 21:21:59 2026] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc$
 @ mem_desc.c:1359                                                                                                                                              
[Fri Jan 23 21:22:00 2026] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc)
 @ mem_desc.c:1359                                                                                                                                              
[Fri Jan 23 21:22:02 2026] NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from _memdescAllocInternal(pMemDesc)
 @ mem_desc.c:1359  

The GPU is currently passed through to a Windows 11 VM, but it is not bound to VFIO (as I sometimes shut the VM down and use the GPU for transcoding using unmanic.

Seems odd to me Unraid would complain about this with the GPU being passed through.

If this is expected, that's fine, or if I need to attach a diag i can. The fix common problems plugin complains almost daily that I have oom errors, but the only oom in the entire log is this.

Cheers,

52 minutes ago, Stokkes said:

The GPU is currently passed through to a Windows 11 VM, but it is not bound to VFIO (as I sometimes shut the VM down and use the GPU for transcoding using unmanic.

First of all, this is not a recommended setup.

However if you are passing it through to a VM you shouldn't have any issues at all.

Please make sure that you are on the latest BIOS version, have 4G Decoding Enabled and Resizable BAR Support Enabled.

This is usually a driver/hardware issue but I can't do much about that since this only means that the card can't assign Hardware resources.

54 minutes ago, Stokkes said:

If this is expected, that's fine, or if I need to attach a diag i can. The fix common problems plugin complains almost daily that I have oom errors, but the only oom in the entire log is this.

I assume that's because of this error, maybe @Squid can make an exception for this kind of OOM error since this is not a real OOM error.

8 hours ago, ich777 said:

I don't think that the Nvidia driver is the culprit since it seems that lsof is the cause of the issue.

Did you maybe install some other Docker/LXC/VM that could be the culprit?

The main reasons I think it's the gpu is due to changing pcie to 3.0 reduced the frequency of the random crashes and when I hit the gpu with a load it crashes.

I might just have a hardware issue though as I haven't changed anything between 2/04/25 - 1/16/26. no new Docker/LXC/VM.

I plan to run an extended mem test to see if anything shows up there.

anyway thanks for the help!

Having trouble updating to v580.126.09 production branch.

I have removed tried removing the app, rebooting then reinstalling. I have tried multiple different versions of the driver, none of them let me get past the error below.

Failed to initialize NVML: Driver/library version mismatch

NVML library version: 590.48

Card is a Quadro P620.

Any insights appreciated.

EDIT: I was often getting checksum errors when trying to re-download the same drivers. This does not always happen. Behind a pihole, but switching it off made no difference to the checksum error.

hotfuzz-diagnostics-20260126-1808.zip

Edited January 26Jan 26 by rscole

2 hours ago, rscole said:

Having trouble updating to v580.126.09 production branch.

It seems a bit strange what's going on on your system.

I see that Unraid installs the 580.x driver and then the plugin installs the 590.x driver, not sure how this is even possible since first this is done:

Jan 26 18:05:03 HOTFUZZ kernel: NVRM: loading NVIDIA UNIX x86_64 Kernel Module 580.126.09 Wed Jan 7 22:59:56 UTC 2026

Jan 26 18:05:03 HOTFUZZ kernel: nvidia-modeset: Loading NVIDIA Kernel Mode Setting Driver for UNIX platforms 580.126.09 Wed Jan 7 22:32:52 UTC 2026

Jan 26 18:05:03 HOTFUZZ kernel: [drm] [nvidia-drm] [GPU ID 0x00000400] Loading driver

Jan 26 18:05:03 HOTFUZZ kernel: [drm] Initialized nvidia-drm 0.0.0 for 0000:04:00.0 on minor 1

and then the plugin does:

Jan 26 18:05:59 HOTFUZZ rc.local: --------------------Nvidia driver v590.48.01 found locally---------------------

Jan 26 18:05:59 HOTFUZZ rc.local: --------------Installation of Nvidia driver v590.48.01 successful--------------

however I only see this package on your boot drive:

/boot/config/plugins/nvidia-driver/packages/6.12.54/:

total 410040

-rw------- 1 root root 419869688 Dec 19 05:18 nvidia-590.48.01-6.12.54-Unraid-1.txz

-rw------- 1 root root 33 Dec 19 05:18 nvidia-590.48.01-6.12.54-Unraid-1.txz.md5

I also see this in your installed packages:

-rw-r--r-- 1 root root 12777 Jan 26 18:04 nvidia-580.126.09-6.12.54-Unraid-1

-rw-r--r-- 1 root root 12878 Jan 26 18:04 nvidia-590.48.01-6.12.54-Unraid-1

Which I can't explain since this shouldn't be possible at all.

Are you sure that your USB Boot drive is known good?

Do you have any php errors on your Nvidia Driver Settings page, if yes do that:
https://forums.unraid.net/topic/98978-plugin-nvidia-driver/page/178/#findComment-1487747

If you have none of these issues also make sure to run a memtest.

No PHP errors at all, but i have removed and reinstalled the app, and still have the same problem. (That was one of my first attempts to troubleshoot)

I've been using the same USB for about 3 years.

I'll do a memtest and report back.

Greetings.

I'm having problems with the Nvidia-Driver plugin. I initially uninstalled it because I was having erratic behavior on my Tesla P4, with the intent to reinstall to see if it fixed the problem. However, now I cannot reinstall the plugin. I get the following failure:

--------------Can't download Nvidia Driver Package v580.82.09-----------------
Removing package: nvidia-driver-2026.01.16
plugin: run failed: '/bin/bash' returned 1
Executing hook script: CA_postHook
Clearing Community Applications plugin cache
Executing hook script: post_plugin_checks
Plugin installed

I'm currently running Unraid 7.1.4, and just replaced my Boot USB thinking that might be the problem.

Any ideas?

3 hours ago, rscole said:

No PHP errors at all, but i have removed and reinstalled the app, and still have the same problem. (That was one of my first attempts to troubleshoot)

Did you do that like:

1. Remove the plugin

2. Reboot

3. Install the plugin (don't yet set the new driver)

4. Reboot

5. Go to your syslog and search for "NVRM" and see if the driver issue still persists

6. Change the driver in the plugin to a 580.x version

7. Reboot

8. Go to your syslog and search for "NVRM" and see if the driver issue still persists

3 hours ago, Moose_Flunky said:

Any ideas?

Diagnostics?

Did you do that like:

1. Remove the plugin

2. Reboot

3. Install the plugin (don't yet set the new driver)

4. Reboot

5. Go to your syslog and search for "NVRM" and see if the driver issue still persists

6. Change the driver in the plugin to a 580.x version

7. Reboot

8. Go to your syslog and search for "NVRM" and see if the driver issue still persists

It's been a few days since I did it, but I will give it another go. I think it automatically installed the 590.x when I installed the app (which you're suggesting it should not have done).

I couldn't do a memtest, but using the live memory tester app didn't give any errors. I did 10GB out of 32GB for the test (ECC ram.)

56 minutes ago, rscole said:

I think it automatically installed the 590.x when I installed the app (which you're suggesting it should not have done).

On plugin installation the plugin will always automatically install driver version 590.x

57 minutes ago, rscole said:

I did 10GB out of 32GB for the test (ECC ram.)

This is not good test, since you have to run a full memory test not only partially.

However, please do the steps as recommended above, exactly like that, and the plugin will always default to the latest driver which is the 590.x one.

Maybe also pull diagnostics after each reboot so that I can see what's going on.

On 1/8/2026 at 2:45 AM, ich777 said:

That's not easy achievable since you would have to find a driver which includes CUDA 12.8.

I assume you have a container that not supports 13.1 correct? Is there maybe a updated container available which supports the new CUDA version?

Unfortunately not, pytorch doesn't support 13.1 yet :(

I recently installed a Gigabyte 1080 Ti and am struggling to get the driver to recognize the card. I have tried the following:

1. I tried the production branch install, open-source install, and other listed versions. All gave the same issue:

   1. "No devices were found"

I noticed some weird behavior as well. When uninstalling the driver, I saw that my GPU statistics app was able to see part of the GPU's identity but when I installed the driver again, GPU statistics now showed "N/A" in all fields:

2. I have also disabled Secure Boot and the integrated CPU graphics within my BIOS settings.

3. I made sure to disable and enable Docker upon first install.

4. I have removed and cleaned the PCiE slot on my motherboard, and reseated it.

The card's fans come on, my BIOS can see the PCIE slot is in use. I'm not sure what to do. I feel like I'm going crazy trying to get this driver to work.

Thanks in advance.

18 hours ago, ich777 said:

Diagnostics?

Sorry. I was able to get the driver installed. For some reason, it appears that the download would time out while pulling the latest Nvidia driver. I was able to follow the download and it eventually pulled and installed correctly.

Again, sorry to have bothered you and thank you for the support

7 hours ago, mburlia said:

Unfortunately not, pytorch doesn't support 13.1 yet :(

So which driver do you need for which Unraid version?

6 hours ago, salem said:

The card's fans come on, my BIOS can see the PCIE slot is in use. I'm not sure what to do. I feel like I'm going crazy trying to get this driver to work.

Please post your Diagnostics because I can't see what's going on without them.

On plugin installation the plugin will always automatically install driver version 590.x

This is not good test, since you have to run a full memory test not only partially.

However, please do the steps as recommended above, exactly like that, and the plugin will always default to the latest driver which is the 590.x one.

Maybe also pull diagnostics after each reboot so that I can see what's going on.

Chose the production 580.x, still not working. Three sets of diagnostics attached for you.

hotfuzz-diagnostics-20260128-2104.zip hotfuzz-diagnostics-20260128-2126.zip hotfuzz-diagnostics-20260128-2150.zip

41 minutes ago, rscole said:

Chose the production 580.x, still not working. Three sets of diagnostics attached for you.

Thank you for that, in the first Diagnostics I see that the driver is not installed, that's good, in the second I see that the plugin got installed but I don't see that the driver version was changed and of course that the driver is complaining about the GPU being not supported because of Pascal.

However in the third Diagnostics I also see the driver complaining of not being compatible because of Pascal.

Are you sure you set the the driver version in the plugin settings to version 580.x because I don't see that it got changed in the 2nd Diagnostics before rebooting, please also do keep in mind to let the dialog open until the dialog says the driver was fully downloaded and you get a notification to reboot.

I also noticed that you are on Unraid 7.2.0, is it possible for you to upgrade to the latest version since this could also solve some issues.

15 hours ago, ich777 said:

Please post your Diagnostics because I can't see what's going on without them.

Sorry about that. Here they are!

diagnostics-20260128-1726.zip

Thank you for that, in the first Diagnostics I see that the driver is not installed, that's good, in the second I see that the plugin got installed but I don't see that the driver version was changed and of course that the driver is complaining about the GPU being not supported because of Pascal.

However in the third Diagnostics I also see the driver complaining of not being compatible because of Pascal.

Are you sure you set the the driver version in the plugin settings to version 580.x because I don't see that it got changed in the 2nd Diagnostics before rebooting, please also do keep in mind to let the dialog open until the dialog says the driver was fully downloaded and you get a notification to reboot.

I also noticed that you are on Unraid 7.2.0, is it possible for you to upgrade to the latest version since this could also solve some issues.

It has installed the Production branch now: 580.126.09

I would note that I had set it to Production and enabled the auto download and push notification. So whether I waited long enough (I always did) or not, then one of the two methods should have downloaded it correctly.

When I updated to 7.2.3, it appears to have applied the nvidia driver change at the same time. I can now see the GPU ID.

So its solved for me, but slightly annoying it needed a minor OS update in the process.

Thanks for looking into it @ich777. If it is of any value, I have added the final diagnostics after the OS update, reboot and Nvidia driver being correctly applied.

hotfuzz-diagnostics-20260129-1526.zip

I recently installed a Gigabyte 1080 Ti and am struggling to get the driver to recognize the card. I have tried the following:

1. I tried the production branch install, open-source install, and other listed versions. All gave the same issue:

   1. "No devices were found"

I noticed some weird behavior as well. When uninstalling the driver, I saw that my GPU statistics app was able to see part of the GPU's identity but when I installed the driver again, GPU statistics now showed "N/A" in all fields:

2. I have also disabled Secure Boot and the integrated CPU graphics within my BIOS settings.

3. I made sure to disable and enable Docker upon first install.

4. I have removed and cleaned the PCiE slot on my motherboard, and reseated it.

The card's fans come on, my BIOS can see the PCIE slot is in use. I'm not sure what to do. I feel like I'm going crazy trying to get this driver to work.

Thanks in advance.

I did not update the plugin due to me having issues the last time when they discontinued support with the latest driver for older cards. Still working with selected driver: Nvidia Driver Version: 580.119.02

On 1/28/2026 at 1:21 AM, salem said:

Thanks in advance.

You have a Xid error in your syslog, to be precise 62:

Jan 27 16:08:32 Mochi kernel: NVRM: Xid (PCI:0000:01:00): 62, 0001b0d6 0001ae80 0002ab00 0002aab2 0001af25 0001af2d 00000011 00000000

Jan 27 16:09:32 Mochi kernel: NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x25:0x65:1623)

Jan 27 16:09:32 Mochi kernel: NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0

You can see what this means here: https://docs.nvidia.com/deploy/xid-errors/analyzing-xid-catalog.html

However I would recommend that you do the following things:
Upgrade your BIOS

Make sure to enable Above 4G Decoding and Resizable BAR Support in your BIOS

Upgrade to the latest Unraid version

Probably this is caused because the BIOS version is outdated since it seems that your GPU fails to reset.

2 hours ago, Axelfony said:

I did not update the plugin due to me having issues the last time when they discontinued support with the latest driver for older cards. Still working with selected driver: Nvidia Driver Version: 580.119.02

This has nothing to do with updating the plugin, the plugin updates and the driver updates are separated from each other and the plugin will always check if a new driver is needed when upgrading to a newer Unraid version.

However I would strongly recommend to upgrade the plugin in your case since I made a change that the driver version 580.x will be downloaded if driver version 580.x is selected, please do note this will not work when you set it to Production, New Feature or Latest driver version.

To clarify that, plugin updates won't change the installed driver.

Hello reaching out with a quick question; as been meaning to upgrade the unraid OS to the latest. (current 7.1.4)

Currently running: Nvidia Driver Version: 580.82.09

At this moment plugins has detected an update for the NVidia Driver, when looking at the details of the driver it states....
Select preferred driver version:

Latest Versions:

latest: v580.82.09

Production Branch: Not found for this Unraid version!

Was curious if it is OK to proceed with upgrading the OS before the NVidia driver?

There was a point in time in December a member of the community was 7.2.0 and 7.2.2 OS and was using version "latest" 590.48.01 which crashed some of the docker applications that leveraged the GPU, to which they mentioned moving down to the "production" 580.119.02 to resolve the concern.

Seeking advise regarding the proper ordered procedure:

1. Upgrade OS > Upgrade Nvidia Driver (to alleviate the "Production Branch: Not found for this Unraid version!")

2. Upgrade Nvidia Driver > Upgrade OS (not sure if that will 'break' anything performing this order?)

Use to run upgrade assistant and realized it is no longer within the tools menu, perhaps it is now deprecated?

Appreciate the feedback whenever someone has a moment to follow-up.

Thank you.

If you are still using the Quadro P2000 you will want to stay on 580.xx.xx. Nvidia has obsoleted the Pascal based cards and is removing support for them from 590.xx.xx.

Best to manually select a 580.xx.xx version (latest one is fine).