Skip to content
View in the app

A better way to browse. Learn more.

Unraid

A full-screen app on your home screen with push notifications, badges and more.

To install this app on iOS and iPadOS
  1. Tap the Share icon in Safari
  2. Scroll the menu and tap Add to Home Screen.
  3. Tap Add in the top-right corner.
To install this app on Android
  1. Tap the 3-dot menu (⋮) in the top-right corner of the browser.
  2. Tap Add to Home screen or Install app.
  3. Confirm by tapping Install.

[Plugin] Nvidia-Driver

Featured Replies

Hi all,

 

Gfx card: Nvidia GTX 1650

Driver: v560.28.03

Unraid: 7.0.0 beta 2

 

Hoping to get some help with a driver issues and im at my wits end with it. After a random amount of time (could be a few hours or days) I get warnings in my syslog about GPU errors starring with:

 

NVRM: Xid (PCI:0000:02:00): 119, pid=1185, name=nv_open_q, Timeout after 6s of waiting for RPC response from GPU0 GSP! Expected function 103 (GSP_RM_ALLOC) (0x80 0x38).

 

and

 

NVRM: Rate limiting GSP RPC error prints for GPU at PCI:0000:02:00 (printing 1 of every 30).  The GPU likely needs to be reset.

 

This eventually results in a total system crash requiring a system reboot. I have had this card for a year or so and everything has been fine up until now.

 

Appreciate any help anyone is able to provide.

 

  • Replies 5.9k
  • Views 1.2m
  • Created
  • Last Reply

Top Posters In This Topic

Most Popular Posts

  • To utilize your Nvidia graphics card in your Docker container(s) the basic steps are:   Add '--runtime=nvidia' in your Docker template in 'Extra Parameters' (you have to enable 'Advanced

  • Recompiled the drivers and they are now just working fine (to get it working scroll down):   Please do the following (this is only necessary if you upgraded before I recompiled the dri

  • I'm currently spinning up my build VM and compiling the drivers again, currently drivers for 6.11.0 stable are not available...

Posted Images

Apologies,

 

Please find attached log for todays crash and reboot.

 

Thanks again.

logs.txt

  • Author
4 hours ago, Thundermonk said:

Sorry, hopefully thats everything.

It seems that you BIOS has issues assigning address space for the GPU:

Jul 29 09:32:49 NAS kernel: resource: resource sanity check: requesting [mem 0x00000000000e0000-0x00000000000fffff], which spans more than PCI Bus 0000:00 [mem 0x000d0000-0x000e7fff window]
Jul 29 09:32:49 NAS kernel: caller _nv043843rm+0x35/0x70 [nvidia] mapping multiple BARs

 

However since this is a GTX 1650 I would strongly recommend that you switch from Legacy boot to UEFI boot, please also make sure that you've enabled above 4G Decoding and if available Resizable BAR support, but I don't think that your BIOS has the last option in it.

 

You can also take a look here what the XID error means (in your case 119).

 

Did this happen with older drivers or did you just install the card into your server?

Please also check if your system memory is okay and maybe do a memtest.

4 hours ago, ich777 said:

It seems that you BIOS has issues assigning address space for the GPU:

Jul 29 09:32:49 NAS kernel: resource: resource sanity check: requesting [mem 0x00000000000e0000-0x00000000000fffff], which spans more than PCI Bus 0000:00 [mem 0x000d0000-0x000e7fff window]
Jul 29 09:32:49 NAS kernel: caller _nv043843rm+0x35/0x70 [nvidia] mapping multiple BARs

 

However since this is a GTX 1650 I would strongly recommend that you switch from Legacy boot to UEFI boot, please also make sure that you've enabled above 4G Decoding and if available Resizable BAR support, but I don't think that your BIOS has the last option in it.

 

You can also take a look here what the XID error means (in your case 119).

 

Did this happen with older drivers or did you just install the card into your server?

Please also check if your system memory is okay and maybe do a memtest.

 

 

Thanks again for looking in to this for me.

 

From what I can tell it was using UEFI boot. 4G decoding was enabled but no resizeable BAR support (as you hinted).

 

Memtest was fine, passed with no errors.

 

With regards to the Drivers, I am unsure what version of the drivers I was using when everything was working fine. It was not something I even looked at until I started having problems. It had been working for fine for a year or so until now with no changes to the system or hardware. I could try rolling back through the drivers to see if that helps.

 

Although it sounds like it might be time to retire the motherboard (after I confirm the gfx card works in another machine ok), she has served me well over the years :)

 

Thanks again for your help, very much appreciated.

  • Author
16 minutes ago, Thundermonk said:

From what I can tell it was using UEFI boot.

You are using legacy:
grafik.png.26220e4f73e4597fdf0b5ccf7e3e9069.png

otherwise this folder would be named EFI (without - at the end).

 

Maybe try to switch to UEFI and see if that helps (click on the blue text Flash on the Main page, at the bottom click Permit UEFI boot and click Apply).

After that you should be able to boot with UEFI.

1 hour ago, ich777 said:

You are using legacy:
grafik.png.26220e4f73e4597fdf0b5ccf7e3e9069.png

otherwise this folder would be named EFI (without - at the end).

 

Maybe try to switch to UEFI and see if that helps (click on the blue text Flash on the Main page, at the bottom click Permit UEFI boot and click Apply).

After that you should be able to boot with UEFI.

Not sure if you are looking at a different motherboard bios?

 

The only refrence I could find to UEFI was in the images attached, changing it to "UEFI only" causes it to boot in to the bios constantly. Could that folder be a handover from a previous system I used to run? The seccond image to me suggests its booting in UEFI?

bios.JPG

BIOS2.JPG

  • Author
33 minutes ago, Thundermonk said:

Not sure if you are looking at a different motherboard bios?

The above screenshot is your USB boot device... You have to click on the Flash drive on the Main page as mentioned above in the Unraid WebUI and follow the steps from above to allow UEFI boot.

 

When the folder is named EFI- it is not possible for your BIOS to boot into UEFI mode AFAIK, this is what the Permit UEFI boot actually does in the Unraid WebUI, it changes the name from the EFI- folder to EFI

 

33 minutes ago, Thundermonk said:

 

bios.JPG

Maybe this is why it's working, because you are allowing Legacy too and it eventually falls back to Legacy.

33 minutes ago, ich777 said:

The above screenshot is your USB boot device... You have to click on the Flash drive on the Main page as mentioned above in the Unraid WebUI and follow the steps from above to allow UEFI boot.

 

When the folder is named EFI- it is not possible for your BIOS to boot into UEFI mode AFAIK, this is what the Permit UEFI boot actually does in the Unraid WebUI, it changes the name from the EFI- folder to EFI

 

Maybe this is why it's working, because you are allowing Legacy too and it eventually falls back to Legacy.

 

Apologies, totally misunderstood what you meant. I have changed it to permit UEFI boot mode and the folder is now just EFI so looks like that worked. Fingers crossed this works!

 

Once again, thank you for taking the time to help.

boot.JPG

is there any way to install a newer driver? I see that the plugin installed 520.56.06 for GeForce GTX 1050 Ti but apperently frigate needs 530 or highter

  • Author
3 minutes ago, Richard de Weerd said:

is there any way to install a newer driver? I see that the plugin installed 520.56.06 for GeForce GTX 1050 Ti but apperently frigate needs 530 or highter

On what Unraid version are you? Diagnostics?

  • Author
Just now, Richard de Weerd said:

6.11.1

Do you have Diagnostics?

You are a bit behind... Unraid 6.12.11 is the latest version.

18 hours ago, ich777 said:

Do you have Diagnostics?

You are a bit behind... Unraid 6.12.11 is the latest version.

Updating helped, thanks for pointing me, I should have found that myself!

@ich777 Hi! I would like to know if you are using https://github.com/NVIDIA/nvidia-container-toolkit for this plugin? If yes, is this plugin up to date with the nvidia releases in the link shared? If yes, why is the nvidia container toolkit version not mentioned in the driver? 

I am planning to install pytorch and other python packages in a docker and wondering how reliable this plugin is or is it possible to install the driver directly on top unraid host?

Thank you! 

  • Author
On 7/31/2024 at 3:44 PM, mans_ said:

@ich777 Hi! I would like to know if you are using https://github.com/NVIDIA/nvidia-container-toolkit for this plugin? If yes, is this plugin up to date with the nvidia releases in the link shared? If yes, why is the nvidia container toolkit version not mentioned in the driver? 

Yes otherwise noting would work for Unraid.

 

It's not mentioned in the plugin itself because these specific Unraid driver packages are made specifically with Docker support in mind otherwise it won't make much sense to release just a driver package for Unraid.

 

If you go through the source you can see that I add both libnvidia-container and nvidia-container-toolkit (also in my repositories <- including how they are compiled) :

https://github.com/ich777/unraid-nvidia-driver/blob/master/source/compile.sh#L45-L60

The slack-desc includes that:

nvidia-driver Package contents:

Nvidia-Driver v560.28.03
libnvidia-container v1.15.0
nvidia-container-toolkit v1.16.1



Custom nvidia-driver for Unraid Kernel v6.1.99 by ich777

but that is not visible to the user because the installation is done in the background.

 

Hope that helps.

 

  

EDIT: Sorry I completely missed that part:

On 7/31/2024 at 3:44 PM, mans_ said:

I am planning to install pytorch and other python packages in a docker and wondering how reliable this plugin is or is it possible to install the driver directly on top unraid host?

I don't know what you mean by how stable that plugin is, it is basically the Nvidia driver.

Maybe someone else can answer but I most of the times get requests here because of hardware incompatibilities, because of cards dying or some other issues but not about the driver(s) not working.

 

The latest driver version 560.28.03 has about 4500 downloads on it for Unraid version 6.12.11

So I assume everything is working.

 

Sure you can of course install the driver on top of Unraid but this is a really tedious process because you have to do that every time you reboot, basically this is why this plugin exists, I produce a driver package for every single Unraid version.

I compile the Production/New Feature/Beta and Legacy Branches in terms of driver packages and of course the latest driver version for each stable Unraid version.

 

As long as you stay on the latest version from Unraid you should always be or better speaking have always the option to upgrade to the latest Nvidia driver version.

2 hours ago, ich777 said:

Yes otherwise noting would work for Unraid.

 

It's not mentioned in the plugin itself because these specific Unraid driver packages are made specificity with Docker support in mind otherwise it won't make much sense to release just a driver package.

 

If you go through the source you can see that I add both libnvidia-container and nvidia-container-toolkit (also in my repositories <- including how they are compiled) :

https://github.com/ich777/unraid-nvidia-driver/blob/master/source/compile.sh#L45-L60

The slack-desc includes that:

nvidia-driver Package contents:

Nvidia-Driver v560.28.03
libnvidia-container v1.15.0
nvidia-container-toolkit v1.16.1



Custom nvidia-driver for Unraid Kernel v6.1.99 by ich777

but that is not visible to the user because the installation is done in the background.

 

Hope that helps.

 

  

EDIT: Sorry I completely missed that part:

I don't know what you mean by how stable that plugin is, it is basically the Nvidia driver.

Maybe someone else can answer but I most of the times get requests here because of hardware incompatibilities, because of cards dying or some other issues but not about the driver(s) not working.

 

The latest driver version 560.28.03 has about 4500 downloads on it for Unraid version 6.12.11

So I assume everything is working.

 

Sure you can of course install the driver on top of Unraid but this is a really tedious process because you have to do that every time you reboot, basically this is why this plugin exists, I produce a driver package for every single Unraid version.

I compile the Production/New Feature/Beta and Legacy Branches in terms of driver packages and of course the latest driver version for each stable Unraid version.

 

As long as you stay on the latest version from Unraid you should always be or better speaking have always the option to upgrade to the latest Nvidia driver version.

 

Thank you for your work and reply. I just have a few more questions.

Will a 4090 work? (I see no reason why it won't but you say something about hardware incompatibilities, so just confirming)

If yes, is possible to pass through dual (2x4090) GPU to the SAME container with this plugin driver?

Edited by mans_

  • Author
1 hour ago, mans_ said:

Will a 4090 work?

Yes, if the driver is new enough to support it, but the answer is basically yes, drivers should support a 4090

 

1 hour ago, mans_ said:

If yes, is possible to pass through dual (2x4090) GPU to the SAME container with this plugin driver?

Sure why not...? But keep in mind that depends on the container but I think most AI containers are capable of using multiple GPUs.

On 7/31/2024 at 10:08 AM, ich777 said:

Hope that helps.

I love reading how the sausage is made.  Thanks for sharing, and for your outstanding work as well as your continued dedication.

 

(Sometime a guy needs to hear that what they do is appreciated...)

  • 2 weeks later...

I have an Nvidia P1000 due to height limitations within my case (CS381)

 

Up until v545.29.06 my usual test of transcoding a source 4k x265 to an x264 1080p 10Mbps has worked wonders and it barely bothers the GPU

 

If I upgrade however to Unraid 7 beta 2, this Nvidia version is stripped from the plugin list and gives me a newer version automatically. But this causes Plex crashing

 

What are my options? Stay on Unraid 6? Buy a newer 1/2 height GPU? Can we keep v545.29.06 an option?

Edited by shabash

  • Author
1 hour ago, shabash said:

If I upgrade however to Unraid 7 beta 2, this Nvidia version is stripped from the plugin list and gives me a newer version automatically. But this causes Plex crashing

Do you have Diagnostics? Troubleshooting is really hard without them.

The version is not stripped, it is most likely not available because you where on a really old version maybe...?

What version did you use before? What version do you use now?

 

A bit more information would be also nice, does it instantly crash? What kind of source material do you try to transcode AVC, HEVC, HDR,...?

 

1 hour ago, shabash said:

Can we keep v545.29.06 an option?

No, why?

As you can see here your GPU is still supported and my GPU (Nvidia T400) is still working just fine with Jellyfin and Unraid 7.0

 

What container are you using? From which maintainer?

 

I just checked and I see that there are about 750 downloads on the latest Nvidia Driver so I assume the driver is working for other users too.

Hi guys. First my apologies if this was asked before. I just could not find this specific problem and is asking for advice and help  please. I am running an unraid server (v6.12.11) on an ASUS ROG STRIX B560-A GAMING motherboard and Intel i7-11700K. It runs fine with Plex and a couple of other docker applications. I also had a Gigabyte GeForce RTX 3060 GPU to transcode plex media, etc. I took this out as the Intel processor's internal GPU works just as fine and besides, I just now build a new server on an AMD processor (ASRock X570 Phantom Gaming 4) with an AMD Ryzen 9 5900X processor. The Ryzen does not have an onboard GPU, so I had to plug in the GTX 3060.

The new server runs fine and will be replacing the older machine, why copyig all the data (Some 65Tb's worth). IT is still busy copying, etc...

The problem is, the new server absolutely refuses to install the NVIDIA driver from the APPs. I installed the GPU Stats plugin, even the Intel GPU Top, but then removed it again as that was just to get the NVTop plugin to install, etc... After all of this, the driver refuses to install and gives me the following message:

 

Any advice or help will be much appreciated. Many thanks.

image.png.7c736e277fc9b89e9d6ec4512d8b4f63.png

  • Author
6 minutes ago, nielu said:

Any advice or help will be much appreciated. Many thanks.

Please post a screenshot from the full plugin page.

 

Are you actually sure that your server has full and unrestricted access to the Internet?

The plugin uses the GitHub API which is known to cause issues with Unifi hardware and AdBlockers.

 

Please make also sure that you have at least 300MB of free space on your USB Boot device for the driver.

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

Account

Navigation

Search

Search

Configure browser push notifications

Chrome (Android)
  1. Tap the lock icon next to the address bar.
  2. Tap Permissions → Notifications.
  3. Adjust your preference.
Chrome (Desktop)
  1. Click the padlock icon in the address bar.
  2. Select Site settings.
  3. Find Notifications and adjust your preference.