5 minutes ago, dhstsw said:

How can i have it issued (with a script i guess) but, also, why this now? 😕 Any idea?

This should have been always be the case.

You‘ll find many threads here on the forums about Nvidia cards not going to P8, this always depends on the implementation on the GPU BIOS and most of the times it is enough to put some load on it after boot and then they go automatically into P8 after a while.

 

Just put it in your go file /boot/config/go at the end, thats it.

Quote

Just put it in your go file /boot/config/go at the end, thats it.

Will do, thanks.

But has there been any driver update in the last days? Otherwise i can't understand the change in behaviours.

Just now, dhstsw said:

But has there been any driver update in the last days? Otherwise i can't understand the change in behaviours.

Nope, at least not when you haven‘t upgraded.

Hardware transcodes not releasing .so files

I am currently running plex container on Unraid 6.12.13. I have an Nvidia Quadro P4000 currently using Nvidia driver version 565.57.01. I noticed this behavior about 2 weeks ago when I went to reboot my Unraid server and noticed that the Array would not shut down properly. I would receive a message stating Retry unmounting disk shares. This was preventing a clean shut down of the server. Digging in more I discovered that about two dozen various decoder.so files using Process shfs with Process ID 6454were hung and would not shut down cleanly and this was causing the issue. I dug in deeper and tested. It looks as if when an item is transcoded in plex it causes the decoder.so files not to release once it is done. In turn the Nvidia driver settings page returns - Unable to determine the device handle for GPU0000:65:00.0: Unknown Error, under the Installed GPU(s): section.  Once this occurs the GPU no longer transcodes and only the CPU does at that point, pinning the CPU.
 

Plex container log:
Error while decoding stream #0:0: Operation not permitted
[wmav2 @ 0x151bfa885600] block_len_bits 5 out of range
Error while decoding stream #0:0: Operation not permitted
[wmav2 @ 0x151bfa885600] prev_block_len_bits 6 out of range
Error while decoding stream #0:0: Operation not permitted
[wmav2 @ 0x151bfa885600] block_len_bits 5 out of range
Error while decoding stream #0:0: Operation not permitted
[wmav2 @ 0x151bfa885600] block_len_bits 5 out of range
Error while decoding stream #0:0: Operation not permitted
[wmav2 @ 0x151bfa885600] overflow (129 > 116) in spectral RLE, ignoring
Error while decoding stream #0:0: Operation not permitted
[wmav2 @ 0x151bfa885600] block_len_bits 6 out of range
Error while decoding stream #0:0: Operation not permitted
[wmav2 @ 0x151bfa885600] prev_block_len_bits 6 out of range
Error while decoding stream #0:0: Operation not permitted
size=       7kB time=00:05:39.19 bitrate=   0.2kbits/s speed=4.41e+03x
video:0kB audio:7kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 2.368775%
Conversion failed!
 

6.1.106-Unraid x86_64
Client:
 Version:           24.0.9
 API version:       1.43
 Go version:        go1.20.13
 Git commit:        2936816
 Built:             Thu Feb  1 00:47:46 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          24.0.9
  API version:      1.43 (minimum version 1.12)
  Go version:       go1.20.13
  Git commit:       fca702d
  Built:            Thu Feb  1 00:49:16 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.7.13
  GitCommit:        7c3aca7a610df76212171d200ca3811ff6096eb8
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0
 

NAME=Slackware
VERSION="15.0"
ID=slackware
VERSION_ID=15.0
PRETTY_NAME="Slackware 15.0 x86_64 (post 15.0 -current)"
ANSI_COLOR="0;34"
CPE_NAME="cpe:/o:slackware:slackware_linux:15.0"
HOME_URL="http://slackware.com/"
SUPPORT_URL="http://www.linuxquestions.org/questions/slackware-14/"
BUG_REPORT_URL="http://www.linuxquestions.org/questions/slackware-14/"
VERSION_CODENAME=current

 

What could be causing the decoder.so files to prevent being released?

 

Edited October 30, 20241 yr by eranwolf
Adding images

51 minutes ago, eranwolf said:

What could be causing the decoder.so files to prevent being released?

Most likely Plex but that is just an assumption.

 

I don't suspect a driver issue since the driver on it's own is not responsible for releasing a library, the transcoder is.

You could also try an older driver version but I don't suspect that this is the cause of the issue.

I also thought this might be the case at first. However, I used Emby as a test, and the same thing happened.

 

Also, when I use the Stop Array command, the docker completely shuts down. However, if I restart just the Plex app, I get the Server Execution Error because of the decoder issue. 

Edited October 30, 20241 yr by eranwolf

17 minutes ago, eranwolf said:

Also, when I use the Stop Array command, the docker completely shuts down. However, if I restart just the Plex app, I get the Server Execution Error because of the decoder issue. 

Did you yet try another driver version, slightly older?

 

17 minutes ago, eranwolf said:

I also thought this might be the case at first. However, I used Emby as a test, and the same thing happened.

Does the server shut down with Emby?

 

Are you also sure that no SSH session or NFS session is open and blocking the unmount from the disks?

Do you maybe also have Diagnostics?

 

I still don't think that the driver is the cause of the issue because more people would report this most definitely.

Just a fun fact, I see that roughly 5800 users have installed this exact driver package you are using (at least roughly 5800 downloads are on that exact package).

@eranwolf oh I see you've edited your first post with a screenshot from the files, however I still don't think that the driver does cause this since these files are part from the Plex container AFAIK and not the libraries from the Nvidia driver.

 

Are you sure that your Plex container is not running some kind of maintenance/updates in the background and this is the cause of the issue.

 

It would be also good to know which container (repository) are you using.

I am using linxserver.io repository:

 

Linuxserver.io version: 1.41.1.9057-af5eaea7a-ls241
Build-date: 2024-10-24T17:32:03+00:00
───────────────────────────────────────

Setting permissions on /transcode
 Server already claimed 
 permissions for /dev/dri/renderD128 are good 
 permissions for /dev/dri/card0 are good 
Docker is used for versioning skip update check
[custom-init] No custom files found, skipping...
Starting Plex Media Server. . . (you can ignore the libusb_init error)
[ls.io-init] done.

I have checked and no maintenance or tasks are running when this occurs. I am going to pull the diagnostic logs as soon as it happens again.

20 minutes ago, eranwolf said:

I have checked and no maintenance or tasks are running when this occurs. I am going to pull the diagnostic logs as soon as it happens again.

I still don't think that's a Nvidia Driver issue since as said above, these files are shipped with the container and as said, may more people would have the same issue.

 

Take a look into /boot/logs, every time you have a unclean shutdown Unraid will automatically create Diagnostics.

OK got these sections from the Diag log. This occurred after the transcode stopped.

Oct 28 20:47:24 Tower kernel: NVRM: GPU at PCI:0000:65:00: GPU-xxxxxx-xxxxxx-xxxxx-xxxxx
Oct 28 20:47:24 Tower kernel: NVRM: GPU Board Serial Number: 1323019024289
Oct 28 20:47:24 Tower kernel: NVRM: Xid (PCI:0000:65:00): 79, pid='<unknown>', name=<unknown>, GPU has fallen off the bus.
Oct 28 20:47:24 Tower kernel: NVRM: GPU 0000:65:00.0: GPU has fallen off the bus.
Oct 28 20:47:24 Tower kernel: NVRM: GPU 0000:65:00.0: GPU serial number is 1323019024289.
Oct 28 20:47:24 Tower kernel: NVRM: A GPU crash dump has been created. If possible, please run
Oct 28 20:47:24 Tower kernel: NVRM: nvidia-bug-report.sh as root to collect this data before
Oct 28 20:47:24 Tower kernel: NVRM: the NVIDIA kernel module is unloaded.
Oct 28 20:47:24 Tower kernel: NVRM: Xid (PCI:0000:65:00): 154, pid='<unknown>', name=<unknown>, GPU recovery action changed from 0x0 (None) to 0x1 (GPU Reset Required)

This is when I hit stop array:

Oct 29 10:20:01 Tower root: Error response from daemon: cannot stop container: plex: tried to kill container, but did not receive an exit event
Oct 29 10:20:01 Tower root: Error response from daemon: cannot stop container: EmbyServer: tried to kill container, but did not receive an exit event
Oct 29 10:20:13 Tower root: stopping dockerd ...
Oct 29 10:20:14 Tower root: ... Waiting to die.
### [PREVIOUS LINE REPEATED 14 TIMES] ###
Oct 29 10:20:28 Tower root: Error: process will not die!
Oct 29 10:20:28 Tower root: ... Killed.

AND

Oct 29 10:20:49 Tower emhttpd: shcmd (153): umount /mnt/cache
Oct 29 10:20:49 Tower root: umount: /mnt/cache: target is busy.
Oct 29 10:20:49 Tower emhttpd: shcmd (153): exit status: 32
Oct 29 10:20:49 Tower emhttpd: Retry unmounting disk share(s)...
Oct 29 10:20:54 Tower emhttpd: Unmounting disks...
Oct 29 10:20:54 Tower emhttpd: shcmd (154): umount /mnt/cache
Oct 29 10:20:54 Tower root: umount: /mnt/cache: target is busy.
Oct 29 10:20:54 Tower emhttpd: shcmd (154): exit status: 32
Oct 29 10:20:54 Tower emhttpd: Retry unmounting disk share(s)...
Oct 29 10:20:59 Tower emhttpd: Unmounting disks...
Oct 29 10:20:59 Tower emhttpd: shcmd (155): umount /mnt/cache
Oct 29 10:20:59 Tower root: umount: /mnt/cache: target is busy.
Oct 29 10:20:59 Tower emhttpd: shcmd (155): exit status: 32
Oct 29 10:20:59 Tower emhttpd: Retry unmounting disk share(s)...
Oct 29 10:21:04 Tower emhttpd: Unmounting disks...
Oct 29 10:21:04 Tower emhttpd: shcmd (156): umount /mnt/cache

So it looks as if the cache drive is unable to be unmounted.

Also, just for reference, this is what I am getting in the Nvidia Driver plugin after the transcoding issue occurs.

Edited October 30, 20241 yr by eranwolf

7 minutes ago, eranwolf said:

OK got these sections from the Diag log. This occurred after the transcode stopped.

Still no Diagnostics…

Attached

tower-diagnostics-20241030-1729.zip

5 hours ago, eranwolf said:

Attached

 

 

i would say here is your issue ... the card "crashed" and "fallen of the bus"

 

Oct 30 16:38:37 Tower kernel: NVRM: GPU at PCI:0000:65:00: GPU-3bd5904d-113e-2444-5620-ae26423c8255
Oct 30 16:38:37 Tower kernel: NVRM: GPU Board Serial Number: 1323019024289
Oct 30 16:38:37 Tower kernel: NVRM: Xid (PCI:0000:65:00): 79, pid='<unknown>', name=<unknown>, GPU has fallen off the bus.
Oct 30 16:38:37 Tower kernel: NVRM: GPU 0000:65:00.0: GPU has fallen off the bus.
Oct 30 16:38:37 Tower kernel: NVRM: GPU 0000:65:00.0: GPU serial number is 1323019024289.
Oct 30 16:38:37 Tower kernel: NVRM: A GPU crash dump has been created. If possible, please run
Oct 30 16:38:37 Tower kernel: NVRM: nvidia-bug-report.sh as root to collect this data before
Oct 30 16:38:37 Tower kernel: NVRM: the NVIDIA kernel module is unloaded.
Oct 30 16:38:37 Tower kernel: NVRM: Xid (PCI:0000:65:00): 154, pid='<unknown>', name=<unknown>, GPU recovery action changed from 0x0 (None) to 0x1 (GPU Reset Required)

 

can be many causes and may check

 

/ power supply still good enough

/ card sits properly in the slot

/ card is "on the edge" and "dying"

/ rbar and above 4g enabled in BIOS

 

may side test the card on another system if its running there solid "in use" ...

 

most likely hardware issue ... may @ich777 has some more infos

9 hours ago, eranwolf said:

Attached

You have a Xid 79 error which means the GPU has fallen from the BUS, this is pretty hard to debug because this can be caused by nearly anything.

(You can learn more about XID error messages here)

 

You said the card was working fine before correct or is this a new card or better speaking did you change anything in terms of hardware?

 

Please check if your power supply is up to the task and has not degraded to a point where it can not cover spikes in power usage, try to re-seat the card in the PCIe slot, check your BIOS settings (especially check if Resizable BAR support and Above 4G Decoding is enabled).

 

If all of that won't help, if it is possible for you, take the card from your server into a PC, install the Nvidia Driver and put a 3D load on it for at least an hour (something like FurMark will do the job just fine) just to check if the card is okay and not defective.

This wouldn't be the first Quadro Card that is reported to be (semi-)broken.

hi @ich777i'm wondering if you can shed some light on how I can locate / fix this. 

I am using proxmox and unraid as a vm. I make use of vGPU unlock on proxmox, and I have passed a vGPU to unraid. 
we can see that here 

 

root@alexandria:~# nvidia-smi
Thu Oct 31 08:17:45 2024
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 550.127.05             Driver Version: 550.127.05     CUDA Version: 12.4     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  GRID V40-2B4                   Off |   00000000:0B:00.0 Off |                  N/A |
| N/A   N/A    P8             N/A /  N/A  |       1MiB /   2048MiB |      0%   Prohibited |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+

+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI        PID   Type   Process name                              GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|  No running processes found                                                             |
+-----------------------------------------------------------------------------------------+

now when I do

root@alexandria:~# nvidia-smi -q | grep "License"
    vGPU Software Licensed Product
        License Status                    : Unlicensed

I wish to auth this, and i'm making use of https://git.collinwebdesigns.de/oscar.krause/fastapi-dls#unraid-guest

which entails running 

https://git.collinwebdesigns.de/oscar.krause/fastapi-dls/-/commit/1fc5ac83789e8f27bb8f41ac445250b36ecb08b4

 

upon running that script im stuck at 

[>] Setting up gridd.conf
cp: cannot stat '/etc/nvidia/gridd.conf.template': No such file or directory
[!] Error configuring gridd.conf, did you install the drivers correctly?

where would this file be located? or do you have any insight into what im doing wrongly?
 

31 minutes ago, mrpops2ko said:

I make use of vGPU unlock on proxmox, and I have passed a vGPU to unraid. 

im pretty sure thats not supported as of EULA Nvidia ... i wouldnt expect support therefore here.

48 minutes ago, mrpops2ko said:

I make use of vGPU unlock on proxmox, and I have passed a vGPU to unraid. 

Please read the first post in the thread (big red text).

No support from my side.

ah sorry about that, missed it whilst scrolling

OK thank you for the assist. No nothing has changed at all. This is the same system and config that has been running for two years now. Haven't swapped out a single part since it was built. I do have another Quadro card so I will swap it out and test and report back.

 

Edited October 31, 20241 yr by eranwolf

There was a recent security bulleting pushed out about driver vulnerabilities... I'm unsure what "driver branch" is in use with this plugin and how would you manually update?



This is what I see available to me on 7 beta 2


4060 Ti is the card

Thanks!

Edited October 31, 20241 yr by melmurp
added card

3 minutes ago, melmurp said:

This is what I see available to me on 7 beta 2

It might be a good idea to consider upgrading to the latest beta.4 version, as it includes the latest driver versions, which could enhance your system's security.

 

I certainly don't mean to be overly critical. It's just important to ensure that when discussing CVEs, you're operating on the most current version of the OS to maintain optimal security.

 

Hope that helps.

1 hour ago, eranwolf said:

Haven't swapped out a single part since it was built.

Please also take the Power Supply into consideration and not only the graphics card.

3 hours ago, ich777 said:

It might be a good idea to consider upgrading to the latest beta.4 version, as it includes the latest driver versions, which could enhance your system's security.

 

I certainly don't mean to be overly critical. It's just important to ensure that when discussing CVEs, you're operating on the most current version of the OS to maintain optimal security.

 

Hope that helps.

That was it.. updated to 4 and now got latest

 

Appreciate the tip