[Plugin] Nvidia-Driver

[thelostnorthman]

hi,

just upgraded Unraid to 6.12 and nvidia driver to 535.54.03 however it seems to have stopped using the gpu.

the nvidia app still shows my graphics card and so does nvidi-smi command.

The card is used for plex transcoding only. Tested nvidia-smi inside and outside the docker container terminal and it shows the card, however trying to play media it needs to transcode it's showing 0% GPU usage and 0 processes.

Any pointers would be great, many thanks.

 

 

 

Mon Jun 19 15:14:51 2023       
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 535.54.03              Driver Version: 535.54.03    CUDA Version: 12.2     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  Quadro P400                    Off | 00000000:07:00.0 Off |                  N/A |
| 31%   44C    P0              N/A /  N/A |      0MiB /  2048MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
                                                                                         
+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

 

Nvidia Info:

Nvidia Driver Version:535.54.03

Open Source Kernel Module:No

Installed GPU(s):0:
Quadro P400
07:00.0
GPU-1eead256-a395-17c4-331a-109bdf1313bf
 

 

System Info

Unraid Version:6.12.0

Kernel:6.1.33-Unraid

Edited June 19, 2023 by thelostnorthman

[ich777]

59 minutes ago, Omen said:

Found at: ./var/lib/pkgtools/packages/nvidia-driver-2023.05.16

Deleted it, the error is almost the same:

Because that is not the actual issue that you are facing here…

 

What‘s most likely the case here is that the plugin couldn‘t communicate with GitHub for some reason (PiHole, AdGuard, Unifi switches, to many GitHub API requests,…)

 

1 hour ago, Omen said:

Seems like deletion on the last step helped, but after reboot. Solved.

After the second reboot it seems that the driver was able to communicate with GitHub and the package was downloaded successful.

 

Another reason could be that you didn‘t wait until the Plugin Update Helper wasn‘t finished and told you that it is safe now to reboot.

[ich777]

23 minutes ago, thelostnorthman said:

I've now lost the home page tile.

Can you be a bit more specific please?

What do you mean with the home page title?

Please post your Diagnostics.

[thelostnorthman]

4 minutes ago, ich777 said:

Can you be a bit more specific please?

What do you mean with the home page title?

Please post your Diagnostics.

Apologies updated my post, ignore the home page tile thing. What I'm finding is that the gpu isn't being used by the plex container anymore. 

Attached my diagnostics thanks.

forge-diagnostics-20230619-1554.zip

[ich777]

Just now, thelostnorthman said:

What I'm finding is that the gpu isn't being used by the plex container anymore. 

But this then seems more like a Plex issue than a driver issue.

 

If the card is recognized on the plugin page it should be working.

 

What input material are you trying to transcode? Do you use the Plex webclient to force a transcode, it is know that there is an issue and transcodig isn‘t working there.

Also see this post and a few above:

 

[thelostnorthman]

7 minutes ago, ich777 said:

But this then seems more like a Plex issue than a driver issue.

 

If the card is recognized on the plugin page it should be working.

 

What input material are you trying to transcode? Do you use the Plex webclient to force a transcode, it is know that there is an issue and transcodig isn‘t working there.

Also see this post and a few above:

 

I've been trying to transcode 4k to 1080 on the android app, the samsung tv app and web client. Even straight 4k original on my home network it would pass that to the GPU to deal with but all bottlenecked at CPU now, no GPU usage.

[ich777]

26 minutes ago, thelostnorthman said:

no GPU usage.

What does the Plex transcoding log say, what variables habe you passed through in the Plex template? Did you do it like mentioned in the second post or did you use another variable, Extra Parameter?

[frakman1]

I have the Nvidia Quadro K4000 graphics card.

When I try to install the latest from the plugin, I get this error:

 

 

The Nvidia drivers page points me to this driver for Linux 64bit which is fairly recent (March 2023)

 

However, when I use the plugin, the only 470.xxx one that it provides is the one at the bottom: v470.141.03

I want to be able to use this card with Docker for apps that use the "--runtime=nvidia" parameter like stable-diffusion etc. that detect the NVIDIA GPU and ask for its GPUID e.g. GPU-xxxxxxx-xxxx-xxxx-xxxx-xxxxxxx.

 

Is this something that can be supported?

 

 

 

 

 

Edited June 20, 2023 by frakman1

[ich777]

1 hour ago, frakman1 said:

When I try to install the latest from the plugin, I get this error:

Without the Diagnostics I can't say anything. It is most likely the case that there is something else preventing the card from showing up.

 

1 hour ago, frakman1 said:

However, when I use the plugin, the only 470.xxx one that it provides is the one at the bottom: v470.141.03

This is still the same driver...

[frakman1]

3 hours ago, ich777 said:

This is still the same driver...

Thanks. I will try and get some logs but it's good to know which driver to use now.

 

[ich777]

1 hour ago, frakman1 said:

Thanks. I will try and get some logs but it's good to know which driver to use now.

 

Please post your Diagnostics so that I can tell what‘s going on, you can also send them via PM if you feel more comfortable that way.

[enJOyIT]

Are there any issues with unraid 6.12.1?

[ich777]

5 minutes ago, enJOyIT said:

Are there any issues with unraid 6.12.1?

Not that I‘m aware of, what issues are you talking about?

If you are ecperiencing issues, please always include your Diagnostics.

 

6.12.1 is in terms of the Nvidia Driver plugin the same as 6.12.0

[enJOyIT]

I just posted here: 

 

 

At bootup it's telling me that the nvidia package seems to be corrupt. How can I redownload the driver?

 

Ah, I think I have to wait until the redownload is finished 🙂 I didn't read the whole message after Checksum error 🙂

 

Should there be a "porgress bar" while downloading? --> No, it's finished now...

 

Rebooted and it's working again.

 

Thank you 😄

Edited June 21, 2023 by enJOyIT

[ich777]

55 minutes ago, enJOyIT said:

I just posted here: 

This is a plugin and you should always use the support thread because strictly speaking Limetech has nothing to do with it.

 

55 minutes ago, enJOyIT said:

At bootup it's telling me that the nvidia package seems to be corrupt. How can I redownload the driver?

Did you waited with the reboot until the Plugin-Update-Helper told you it is safe to reboot?

 

55 minutes ago, enJOyIT said:

Ah, I think I have to wait until the redownload is finished 🙂 I didn't read the whole message after Checksum error 🙂

After the download finished it should have told you just to reboot.

 

55 minutes ago, enJOyIT said:

Should there be a "porgress bar" while downloading? --> No, it's finished now...

No, it tells you to wait until the Done button appears and that you should not close this window.

 

 

Glad that it is now working again, please always read the dialogues.

[Olitrolli]

Hi there!

 

After the latest update of the plugin including an update of Unraid to 6.12.1 and the latest nvidia driver, it seems my graphics card no longer shows up. 

 

Keeps displaying the driver as being corrupt on the boot screen, even after multiple re-installs, re-downloads of various driver versions and reboots - without any success.

 

I don't have any Adguard or the like on my home network.

 

Worth noting I have also reinstalled Unraid on my USB flash drive, only keeping the config folder.

 

Hope someone can help me narrow down my problem 

tower-diagnostics-20230621-1834.zip

[ich777]

5 minutes ago, Olitrolli said:

Keeps displaying the driver as being corrupt

Where does it say this, do you have a screenshot please?

The driver size seems correct to me and I don't think the checksum is wrong.

 

Have you yet tried just to reboot after verifying the checksums? From what I can tell from your syslog it wasn't downloaded and installed later on in the boot process.

[Olitrolli]

9 minutes ago, ich777 said:

Where does it say this, do you have a screenshot please?

The driver size seems correct to me and I don't think the checksum is wrong.

 

Have you yet tried just to reboot after verifying the checksums? From what I can tell from your syslog it wasn't downloaded and installed later on in the boot process.

This is the response I get:

FATAL: Module nvidia not found in directory /lib/modules.....

 

I deleted the plugin. Went to download it again and now it won't let me download it.

 

Just rebooted the system as well and same response :-((

 

[Olitrolli]

Okay. So through the power of rebooting again again I managed to download the plugin and get confirmation that the driver had been installed. I pressed the "done" button and went for another reboot to finalize the install.

 

This is what the plugin displays once booted into Unraid.

 

 

[ich777]

40 minutes ago, Olitrolli said:

I deleted the plugin. Went to download it again and now it won't let me download it.

 

Just rebooted the system as well and same response :-((

The usual way is that you remove the plugin, then reboot and then reinstall the plugin again and reboot again.

 

30 minutes ago, Olitrolli said:

This is what the plugin displays once booted into Unraid.

Please post your Diagnostics, this is completely different from before.

[Olitrolli]

So the Unraid UI froze and crashed before I was able to download the diagnostics and I was forced to reboot once again.

 

 

I managed to take a picture of the boot which might point us in the direction of the problem.

 

The "NVIDIA-SMI..." no longer shows up in the plugin after having logged back in.

 

Again, really appreciate the help here

 

tower-diagnostics-20230621-2056.zip

[ich777]

1 hour ago, Olitrolli said:

Again, really appreciate the help here

These Diagnostics are the same as before, the driver did not install for whatever reason on your system even if it tells it in the syslog.

The message on boot just tells you that the driver package did not install

 

I need the Diagnostics when the plugin reports that "nvidia-smi is not able..."

 

Do the following:

1. Uninstall the plugin
2. Reboot
3. Pull a fresh copy from the plugin from the CA App
4. Reboot
5. The plugin should report that nvidia-smi failed -> pull the Diagnostics imitatively and post them here

 

Did you change anything in the BIOS, did you maybe do a BIOS update or anything similar, was it always the case that you've booted into UEFI instead of Legacy mode?

Please check also if you have C-States in the BIOS disabled -> this causes mostly issues with Nvidia cards.

[Olitrolli]

36 minutes ago, ich777 said:

These Diagnostics are the same as before, the driver did not install for whatever reason on your system even if it tells it in the syslog.

The message on boot just tells you that the driver package did not install

 

I need the Diagnostics when the plugin reports that "nvidia-smi is not able..."

 

Do the following:

1. Uninstall the plugin
2. Reboot
3. Pull a fresh copy from the plugin from the CA App
4. Reboot
5. The plugin should report that nvidia-smi failed -> pull the Diagnostics imitatively and post them here

 

Did you change anything in the BIOS, did you maybe do a BIOS update or anything similar, was it always the case that you've booted into UEFI instead of Legacy mode?

Please check also if you have C-States in the BIOS disabled -> this causes mostly issues with Nvidia cards.

 

Okay! So I replicated the nvidia-smi fail scenario and attached the diagnostics below.

 

No I haven't changed any BIOS settings recently and have always booted into UEFI. Would you recommend booting in Legacy mode instead?

 

Will check C-states in BIOS when I get home but haven't touched or had any problems with it before :-/

tower-diagnostics-20230621-2259.zip

[ich777]

8 hours ago, Olitrolli said:

Okay! So I replicated the nvidia-smi fail scenario and attached the diagnostics below.

I have to look into that issue when I got home from work.

 

It is really strange to me since another user on 6.12.1 is utilizing the driver just fine:

 

8 hours ago, Olitrolli said:

No I haven't changed any BIOS settings recently and have always booted into UEFI. Would you recommend booting in Legacy mode instead?

With the Nvidia Driver yes, however if you haven't experienced any issues so far then it should work on 6.12.1 also just fine.

 

8 hours ago, Olitrolli said:

Will check C-states in BIOS when I get home but haven't touched or had any problems with it before 😕

Most AMD users have to disable C-States so that the card can work properly.

 

I think something else is going on on your system but I can't tell for sure what.

 

BTW just because I noticed it, don't switch between various driver version since this won't magically solve your issue, stick to the latest one by switching it makes diagnosing the issue much harder for me.

 

What is the device 0000:07:00.0 in your system and why did you bind it to VFIO? If all of that doesn't help can you try to downgrade to 6.12.0 and see if that fixes the issue?

 

I've also now took a quick look how often driver version 535.54.03 was downloaded and as of time of writing it was downloaded about 2000 times, so I'm assuming it is working fine since otherwise I would have many reports that the driver isn't working.

[Olitrolli]

13 hours ago, ich777 said:

I have to look into that issue when I got home from work.

 

It is really strange to me since another user on 6.12.1 is utilizing the driver just fine:

 

With the Nvidia Driver yes, however if you haven't experienced any issues so far then it should work on 6.12.1 also just fine.

 

Most AMD users have to disable C-States so that the card can work properly.

 

I think something else is going on on your system but I can't tell for sure what.

 

BTW just because I noticed it, don't switch between various driver version since this won't magically solve your issue, stick to the latest one by switching it makes diagnosing the issue much harder for me.

 

What is the device 0000:07:00.0 in your system and why did you bind it to VFIO? If all of that doesn't help can you try to downgrade to 6.12.0 and see if that fixes the issue?

 

I've also now took a quick look how often driver version 535.54.03 was downloaded and as of time of writing it was downloaded about 2000 times, so I'm assuming it is working fine since otherwise I would have many reports that the driver isn't working.

 

So I re-seated the graphics card and disabled C-states in the BIOS and boom! No more "corrupted" driver.

 

Before that I had even tried to reinstall Unraid on an entirely new and formatted USB flash drive with same results.

 

I have never touched the C-state BIOS option prior - weird if that had turned into a problem all of a sudden.

 

Thank you so much @ich777!

 

I really want to support you for the help you do for this community. Is sponsoring you on Github the right place to go?