Skip to content
View in the app

A better way to browse. Learn more.

Unraid

A full-screen app on your home screen with push notifications, badges and more.

To install this app on iOS and iPadOS
  1. Tap the Share icon in Safari
  2. Scroll the menu and tap Add to Home Screen.
  3. Tap Add in the top-right corner.
To install this app on Android
  1. Tap the 3-dot menu (⋮) in the top-right corner of the browser.
  2. Tap Add to Home screen or Install app.
  3. Confirm by tapping Install.

[Plugin] Nvidia-Driver

Featured Replies

Hi 

 

Thank you for making this driver possible. 

 

I recently purchased a 3090FE and it was pretty much plug and play with the help of this driver. I have enabled persistence mode using a user script (First array start) and the GPU was sitting GPU IDLE (P8) at around 11w. There didnt seem to be an issue with the iGPU that i am using for plex transcodes / immich transcodes. 

 

2 days on, i noticed it is now sitting idle in throttled SW POWER CAP ACTIVE, and at a higher wattage 24w. memory is sitting at 405 Mhz constant,  Occasionally, it switches to  GPU IDLE, but then back again to SW POWER CAP.  

 

I tried disabling dockers, restarting the server, but this hasnt helped.  Any Ideas?

 

nvidia-smi -q

Edited by kenzo86

  • Replies 5.9k
  • Views 1.2m
  • Created
  • Last Reply

Top Posters In This Topic

Most Popular Posts

  • To utilize your Nvidia graphics card in your Docker container(s) the basic steps are:   Add '--runtime=nvidia' in your Docker template in 'Extra Parameters' (you have to enable 'Advanced

  • Recompiled the drivers and they are now just working fine (to get it working scroll down):   Please do the following (this is only necessary if you upgraded before I recompiled the dri

  • I'm currently spinning up my build VM and compiling the drivers again, currently drivers for 6.11.0 stable are not available...

Posted Images

  • Author
1 hour ago, kenzo86 said:

2 days on, i noticed it is now sitting idle in throttled SW POWER CAP ACTIVE, and at a higher wattage 24w. memory is sitting at 405 Mhz constant,  Occasionally, it switches to  GPU IDLE, but then back again to SW POWER CAP.  

Without Diagnostics it is really hard to troubleshoot, please post your Diagnostics because otherwise I can't say anything without them.

 

1 hour ago, kenzo86 said:

I have enabled persistence mode using a user script (First array start) and the GPU was sitting idle (P8) at around 11w.

How did you do that? Do you have the script or a link how you did that?

 

Usually it is enough to put the line:

nvidia-persistenced

at the end in the go file (/boot/config/go).

 

No complicated scripts or anything needed nowadays.

 

May I ask for what do you use the GPU?

On 10/17/2024 at 2:30 PM, ich777 said:

Without Diagnostics it is really hard to troubleshoot, please post your Diagnostics because otherwise I can't say anything without them.

 

How did you do that? Do you have the script or a link how you did that?

 

Usually it is enough to put the line:

nvidia-persistenced

at the end in the go file (/boot/config/go).

 

No complicated scripts or anything needed nowadays.

 

May I ask for what do you use the GPU?

 

Thanks for replying.

 

1) I have attached diagnostics.

 

2) User scripts in settings --> create a script which runs the 'nvidia-persistenced' line on array startup.

 

''#!/bin/bash
nvidia-persistenced''

 

3) I use the GPU for Invoke-AI and hopefully for steam-headless later on.

 

The throttled sw power cap status at idle resolves after a server reboot (gpu idle). I've spoken to another user (Pri) on the official discord who has had the same issue unresolved for a while.

 

At 100% load, the 3090 is in P0 (throttled: sw power cap) - see attached screenshot of GPU statistics. I have also attached a screenshot of the GPU statistics at idle with sw power cap and gigher power draw (should be 14w idle)

 

The gpu is connected via a type 5 cable (12pin to 8+8) to psu. I did not change anything in bios after installing the dGPU.

tower-diagnostics-20241017-1442.zip

IMG_20241017_154535.jpg

IMG_20241017_171627.jpg

Edited by kenzo86

On 10/15/2024 at 3:19 PM, JokerzWild said:

Are you sure that no BIOS settings changed and so on?

You can also try to re-seat the card in the slot and check if the external power cables are plugged in correctly.

@ich777

 

I really appreciate all the help you provided...not to mention the integration between UNRAID & NVIDIA!

 

I did get a chance to re-seat the card in the slot.  I double checked the power connections and gave them an extra push to make sure they were in tightly.  Slightly off-topic but I fixed the antenna for the onboard wireless NIC as well.

 

After re-seating the card, I was able to make it visible within the settings->nvidia tab, without crashing the server.

 

Again, thank you for your help and the integration!

  • Author
On 10/17/2024 at 3:49 PM, kenzo86 said:

Thanks for replying.

So the power doesn't go down after a few days, correct?

 

I really can't help with that since this could be also be caused by a firmware but.

You could try to kill nvidia-persistenced and run it again, maybe that will help.

On 10/15/2024 at 12:29 AM, ich777 said:

The driver package and the plugin are two separate things, so to speak, I suspect the plugin package on your system is broken but not the driver.

So to speak if you see error messages on the plugin page you have to remove the plugin, reboot, install the plugin and preferably reboot again.

 

No worries, take your time.

I forgot to report back on Friday. I did the recommended uninstall, reboot, reinstall and now everything appears to work as expected. Thanks for the help.

Hi,

last night i noticed the fans on my GTX 1070 running (while usually it's not the case).

Today, looking at the GPU stats in the dashboard i noticed this:

image.png.9ced8b8d889eb3a5bf7626c6b24a4b20.png

Beside having fan spinning, and some load on memory, the power state says "P5" and the the card being throttling.
As you can see, nothing is using it.

If i run nvtop everything goes back as it should be:

image.png.1040d694bd2af5d38b0591f74543b6e3.png

BUT, still says GPU is throttling.

This is what nvtop shows:
image.png.c2c2ad217dd5bf81abf0de5eaa45323a.png

As i close nvtop, the card goes back to the previous state (as in 1st image).

I tried to uninstall drivers, reboot, reinstall drivers, reboot - but nothing changes.
I'm quite puzzled, everything was perfect until last night.

 

All images (and diagnostics) taken with docker and VSs disabled.


I attach my diagnostics to the post.incubus-diagnostics-20241022-1513.zip
Any help would be greatly appreciated.

thanks.

 

Edited by dhstsw

  • Author
23 minutes ago, dhstsw said:

I tried to uninstall drivers, reboot, reinstall drivers, reboot - but nothing changes.

I assume this card was working before without any issues correct?

 

Please issue:

nvidia-persistenced

from the command line, wait a few minutes and see if it drops down to P8

 

Have you yet tried installing a driver lower than version 560?

Quote

I assume this card was working before without any issues correct?

Yes, perfectly.

 

Quote

Please issue:

nvidia-persistenced
from the command line, wait a few minutes and see if it drops down to P8


With the command everything goes as it should be.
How can i have it issued (with a script i guess) but, also, why this now? :-/ Any idea?

 

Quote

Have you yet tried a driver lower than version 560 to install?


Not yet.

  • Author
5 minutes ago, dhstsw said:

How can i have it issued (with a script i guess) but, also, why this now? 😕 Any idea?

This should have been always be the case.

You‘ll find many threads here on the forums about Nvidia cards not going to P8, this always depends on the implementation on the GPU BIOS and most of the times it is enough to put some load on it after boot and then they go automatically into P8 after a while.

 

Just put it in your go file /boot/config/go at the end, thats it.

Quote

Just put it in your go file /boot/config/go at the end, thats it.

Will do, thanks.

But has there been any driver update in the last days? Otherwise i can't understand the change in behaviours.

  • Author
Just now, dhstsw said:

But has there been any driver update in the last days? Otherwise i can't understand the change in behaviours.

Nope, at least not when you haven‘t upgraded.

  • 2 weeks later...

Hardware transcodes not releasing .so files

I am currently running plex container on Unraid 6.12.13. I have an Nvidia Quadro P4000 currently using Nvidia driver version 565.57.01. I noticed this behavior about 2 weeks ago when I went to reboot my Unraid server and noticed that the Array would not shut down properly. I would receive a message stating Retry unmounting disk shares. This was preventing a clean shut down of the server. Digging in more I discovered that about two dozen various decoder.so files using Process shfs with Process ID 6454were hung and would not shut down cleanly and this was causing the issue. I dug in deeper and tested. It looks as if when an item is transcoded in plex it causes the decoder.so files not to release once it is done. In turn the Nvidia driver settings page returns - Unable to determine the device handle for GPU0000:65:00.0: Unknown Error, under the Installed GPU(s): section.  Once this occurs the GPU no longer transcodes and only the CPU does at that point, pinning the CPU.
 

Plex container log:
Error while decoding stream #0:0: Operation not permitted
[wmav2 @ 0x151bfa885600] block_len_bits 5 out of range
Error while decoding stream #0:0: Operation not permitted
[wmav2 @ 0x151bfa885600] prev_block_len_bits 6 out of range
Error while decoding stream #0:0: Operation not permitted
[wmav2 @ 0x151bfa885600] block_len_bits 5 out of range
Error while decoding stream #0:0: Operation not permitted
[wmav2 @ 0x151bfa885600] block_len_bits 5 out of range
Error while decoding stream #0:0: Operation not permitted
[wmav2 @ 0x151bfa885600] overflow (129 > 116) in spectral RLE, ignoring
Error while decoding stream #0:0: Operation not permitted
[wmav2 @ 0x151bfa885600] block_len_bits 6 out of range
Error while decoding stream #0:0: Operation not permitted
[wmav2 @ 0x151bfa885600] prev_block_len_bits 6 out of range
Error while decoding stream #0:0: Operation not permitted
size=       7kB time=00:05:39.19 bitrate=   0.2kbits/s speed=4.41e+03x
video:0kB audio:7kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 2.368775%
Conversion failed!
 

6.1.106-Unraid x86_64
Client:
 Version:           24.0.9
 API version:       1.43
 Go version:        go1.20.13
 Git commit:        2936816
 Built:             Thu Feb  1 00:47:46 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.9
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.13
  Git commit:       fca702d
  Built:            Thu Feb  1 00:49:16 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.13
  GitCommit:        7c3aca7a610df76212171d200ca3811ff6096eb8
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
 

NAME=Slackware
VERSION="15.0"
ID=slackware
VERSION_ID=15.0
PRETTY_NAME="Slackware 15.0 x86_64 (post 15.0 -current)"
ANSI_COLOR="0;34"
CPE_NAME="cpe:/o:slackware:slackware_linux:15.0"
HOME_URL="http://slackware.com/"
SUPPORT_URL="http://www.linuxquestions.org/questions/slackware-14/"
BUG_REPORT_URL="http://www.linuxquestions.org/questions/slackware-14/"
VERSION_CODENAME=current

 

What could be causing the decoder.so files to prevent being released?

 

Screenshot 2024-10-30 102229.png

Screenshot 2024-10-30 102250.png

Screenshot 2024-10-30 102310.png

Screenshot 2024-10-30 102332.png

Edited by eranwolf
Adding images

  • Author
51 minutes ago, eranwolf said:

What could be causing the decoder.so files to prevent being released?

Most likely Plex but that is just an assumption.

 

I don't suspect a driver issue since the driver on it's own is not responsible for releasing a library, the transcoder is.

You could also try an older driver version but I don't suspect that this is the cause of the issue.

I also thought this might be the case at first. However, I used Emby as a test, and the same thing happened.

 

Also, when I use the Stop Array command, the docker completely shuts down. However, if I restart just the Plex app, I get the Server Execution Error because of the decoder issue. 

Edited by eranwolf

  • Author
17 minutes ago, eranwolf said:

Also, when I use the Stop Array command, the docker completely shuts down. However, if I restart just the Plex app, I get the Server Execution Error because of the decoder issue. 

Did you yet try another driver version, slightly older?

 

17 minutes ago, eranwolf said:

I also thought this might be the case at first. However, I used Emby as a test, and the same thing happened.

Does the server shut down with Emby?

 

Are you also sure that no SSH session or NFS session is open and blocking the unmount from the disks?

Do you maybe also have Diagnostics?

 

I still don't think that the driver is the cause of the issue because more people would report this most definitely.

Just a fun fact, I see that roughly 5800 users have installed this exact driver package you are using (at least roughly 5800 downloads are on that exact package).

  • Author

@eranwolf oh I see you've edited your first post with a screenshot from the files, however I still don't think that the driver does cause this since these files are part from the Plex container AFAIK and not the libraries from the Nvidia driver.

 

Are you sure that your Plex container is not running some kind of maintenance/updates in the background and this is the cause of the issue.

 

It would be also good to know which container (repository) are you using.

I am using linxserver.io repository:

 

Linuxserver.io version: 1.41.1.9057-af5eaea7a-ls241
Build-date: 2024-10-24T17:32:03+00:00
───────────────────────────────────────

Setting permissions on /transcode
 Server already claimed 
 permissions for /dev/dri/renderD128 are good 
 permissions for /dev/dri/card0 are good 
Docker is used for versioning skip update check
[custom-init] No custom files found, skipping...
Starting Plex Media Server. . . (you can ignore the libusb_init error)
[ls.io-init] done.

I have checked and no maintenance or tasks are running when this occurs. I am going to pull the diagnostic logs as soon as it happens again.

  • Author
20 minutes ago, eranwolf said:

I have checked and no maintenance or tasks are running when this occurs. I am going to pull the diagnostic logs as soon as it happens again.

I still don't think that's a Nvidia Driver issue since as said above, these files are shipped with the container and as said, may more people would have the same issue.

 

Take a look into /boot/logs, every time you have a unclean shutdown Unraid will automatically create Diagnostics.

OK got these sections from the Diag log. This occurred after the transcode stopped.

Oct 28 20:47:24 Tower kernel: NVRM: GPU at PCI:0000:65:00: GPU-xxxxxx-xxxxxx-xxxxx-xxxxx
Oct 28 20:47:24 Tower kernel: NVRM: GPU Board Serial Number: 1323019024289
Oct 28 20:47:24 Tower kernel: NVRM: Xid (PCI:0000:65:00): 79, pid='<unknown>', name=<unknown>, GPU has fallen off the bus.
Oct 28 20:47:24 Tower kernel: NVRM: GPU 0000:65:00.0: GPU has fallen off the bus.
Oct 28 20:47:24 Tower kernel: NVRM: GPU 0000:65:00.0: GPU serial number is 1323019024289.
Oct 28 20:47:24 Tower kernel: NVRM: A GPU crash dump has been created. If possible, please run
Oct 28 20:47:24 Tower kernel: NVRM: nvidia-bug-report.sh as root to collect this data before
Oct 28 20:47:24 Tower kernel: NVRM: the NVIDIA kernel module is unloaded.
Oct 28 20:47:24 Tower kernel: NVRM: Xid (PCI:0000:65:00): 154, pid='<unknown>', name=<unknown>, GPU recovery action changed from 0x0 (None) to 0x1 (GPU Reset Required)

This is when I hit stop array:

Oct 29 10:20:01 Tower root: Error response from daemon: cannot stop container: plex: tried to kill container, but did not receive an exit event
Oct 29 10:20:01 Tower root: Error response from daemon: cannot stop container: EmbyServer: tried to kill container, but did not receive an exit event
Oct 29 10:20:13 Tower root: stopping dockerd ...
Oct 29 10:20:14 Tower root: ... Waiting to die.
### [PREVIOUS LINE REPEATED 14 TIMES] ###
Oct 29 10:20:28 Tower root: Error: process will not die!
Oct 29 10:20:28 Tower root: ... Killed.

AND

Oct 29 10:20:49 Tower emhttpd: shcmd (153): umount /mnt/cache
Oct 29 10:20:49 Tower root: umount: /mnt/cache: target is busy.
Oct 29 10:20:49 Tower emhttpd: shcmd (153): exit status: 32
Oct 29 10:20:49 Tower emhttpd: Retry unmounting disk share(s)...
Oct 29 10:20:54 Tower emhttpd: Unmounting disks...
Oct 29 10:20:54 Tower emhttpd: shcmd (154): umount /mnt/cache
Oct 29 10:20:54 Tower root: umount: /mnt/cache: target is busy.
Oct 29 10:20:54 Tower emhttpd: shcmd (154): exit status: 32
Oct 29 10:20:54 Tower emhttpd: Retry unmounting disk share(s)...
Oct 29 10:20:59 Tower emhttpd: Unmounting disks...
Oct 29 10:20:59 Tower emhttpd: shcmd (155): umount /mnt/cache
Oct 29 10:20:59 Tower root: umount: /mnt/cache: target is busy.
Oct 29 10:20:59 Tower emhttpd: shcmd (155): exit status: 32
Oct 29 10:20:59 Tower emhttpd: Retry unmounting disk share(s)...
Oct 29 10:21:04 Tower emhttpd: Unmounting disks...
Oct 29 10:21:04 Tower emhttpd: shcmd (156): umount /mnt/cache

So it looks as if the cache drive is unable to be unmounted.

Also, just for reference, this is what I am getting in the Nvidia Driver plugin after the transcoding issue occurs.

Screenshot 2024-10-30 172416.png

Edited by eranwolf

  • Author
7 minutes ago, eranwolf said:

OK got these sections from the Diag log. This occurred after the transcode stopped.

Still no Diagnostics…

5 hours ago, eranwolf said:

Attached

 

 

i would say here is your issue ... the card "crashed" and "fallen of the bus"

 

Oct 30 16:38:37 Tower kernel: NVRM: GPU at PCI:0000:65:00: GPU-3bd5904d-113e-2444-5620-ae26423c8255
Oct 30 16:38:37 Tower kernel: NVRM: GPU Board Serial Number: 1323019024289
Oct 30 16:38:37 Tower kernel: NVRM: Xid (PCI:0000:65:00): 79, pid='<unknown>', name=<unknown>, GPU has fallen off the bus.
Oct 30 16:38:37 Tower kernel: NVRM: GPU 0000:65:00.0: GPU has fallen off the bus.
Oct 30 16:38:37 Tower kernel: NVRM: GPU 0000:65:00.0: GPU serial number is 1323019024289.
Oct 30 16:38:37 Tower kernel: NVRM: A GPU crash dump has been created. If possible, please run
Oct 30 16:38:37 Tower kernel: NVRM: nvidia-bug-report.sh as root to collect this data before
Oct 30 16:38:37 Tower kernel: NVRM: the NVIDIA kernel module is unloaded.
Oct 30 16:38:37 Tower kernel: NVRM: Xid (PCI:0000:65:00): 154, pid='<unknown>', name=<unknown>, GPU recovery action changed from 0x0 (None) to 0x1 (GPU Reset Required)

 

can be many causes and may check

 

/ power supply still good enough

/ card sits properly in the slot

/ card is "on the edge" and "dying"

/ rbar and above 4g enabled in BIOS

 

may side test the card on another system if its running there solid "in use" ...

 

most likely hardware issue ... may @ich777 has some more infos

  • Author
9 hours ago, eranwolf said:

Attached

You have a Xid 79 error which means the GPU has fallen from the BUS, this is pretty hard to debug because this can be caused by nearly anything.

(You can learn more about XID error messages here)

 

You said the card was working fine before correct or is this a new card or better speaking did you change anything in terms of hardware?

 

Please check if your power supply is up to the task and has not degraded to a point where it can not cover spikes in power usage, try to re-seat the card in the PCIe slot, check your BIOS settings (especially check if Resizable BAR support and Above 4G Decoding is enabled).

 

If all of that won't help, if it is possible for you, take the card from your server into a PC, install the Nvidia Driver and put a 3D load on it for at least an hour (something like FurMark will do the job just fine) just to check if the card is okay and not defective.

This wouldn't be the first Quadro Card that is reported to be (semi-)broken.

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

Account

Navigation

Search

Search

Configure browser push notifications

Chrome (Android)
  1. Tap the lock icon next to the address bar.
  2. Tap Permissions → Notifications.
  3. Adjust your preference.
Chrome (Desktop)
  1. Click the padlock icon in the address bar.
  2. Select Site settings.
  3. Find Notifications and adjust your preference.