Hey guys, I made a post here with the full details, but in short, I believe my system is completely locking up due to my GPU and this plugin and I'm trying to find some help to validate the 3090 TI is supported. I can't easily replicate the failure, and when it does fail, the system is completely locked, so I have to power cycle it.

 

I can't post a screen shot as I've physically pulled the pulled the power cables from my 3090 so I can verify the system is stable (runs for more than 24 hours) and also let the parity check complete. I can verify the GPU was found and properly working (I've got ollama setup to use it) so I know the nvidia-smi command was properly finding the GPU.

 

I also saw something about how using this in a VM and docker-container can cause a hard freeze, but I'm not doing that. I had experimented with both, but I've unbound things from the VFIO page, and I also don't have any VMs currently setup.

 

Since it's a hard freeze, getting any logs is problematic. I did setup a syslog server, but the only message from around the time of the freeze was "monitor_nchan: Stop running nchan processes"

 

I also had similar issues when I was running a dedicated build of Ubuntu on this server (before installing Unraid 7.0.1) and I was able to resolve it by eventually finding the right driver version (I think I took several steps backwards/downgrading) but I haven't had that same luck here.

 

If I have to plug the GPU back in and run any additional diagnostics I'm happy to, I just need to know what commands I can do to start figuring out exactly what might be going wrong.

9 minutes ago, ChadDa3mon said:

I'm trying to find some help to validate the 3090 TI is supported

Yes it is.
 

10 minutes ago, ChadDa3mon said:

I can't easily replicate the failure, and when it does fail, the system is completely locked, so I have to power cycle it.

The Diagnostics that you've posted in the other thread are not really helpful since no Nvidia card is visible, most certainly because you've disconnected the external power, however it should display a message that the external power plugs are not plugged in.

 

14 minutes ago, ChadDa3mon said:

Since it's a hard freeze, getting any logs is problematic.

It would be helpful if you would include which containers are using the GPU.

Please note that a few people already reported here that AI containers cause some times a Call Trace which ultimately results in the Server crashing <- but not in all cases the whole server crashes.

 

22 minutes ago, ChadDa3mon said:

I also had similar issues when I was running a dedicated build of Ubuntu on this server (before installing Unraid 7.0.1) and I was able to resolve it by eventually finding the right driver version (I think I took several steps backwards/downgrading) but I haven't had that same luck here.

Have you yet considered that a container causes this and yet tried enable container by container until it happens again, as said above, a few people already reported the GPU not working after some time when using AI containers.

Maybe try to disable all your AI containers and only enable Plex as a starting point, when that works you can enable the next container in addition to Plex.

 

25 minutes ago, ChadDa3mon said:

If I have to plug the GPU back in and run any additional diagnostics I'm happy to, I just need to know what commands I can do to start figuring out exactly what might be going wrong.

Please try what I recommended above. If your Server crashes just with Plex enabled then possibly your GPU is at fault but this is only a assumption, please make also sure that your Power Supply is up to the task, so to speak can it deliver enough power and is it not too old since power supplies degrade over time and if a power spike happens it can take down the whole server.

 

Hope that helps, the driver is indeed working and your GPU is supported by the driver (regardless from which version).

17 hours ago, ich777 said:

The issue is this line here in your go file:

# Copy Docker daemon JSON
mkdir -p /etc/docker
cp /boot/config/daemon.json /etc/docker/daemon.json &

 

What contents do you have in that file (boot/config/daemon.json) and why do you even change the daemon.json?

I have my docker containers log to loki/promtail.

I was wondering if this would be the case. Interesting it was working prior to the upgrade.

 

What do I need to add to it to make this work?

 

Looks like the daemon.json should be the following:

{
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}

So maybe a better way would be to use sed? I'm a complete novice to sed, but modifying what you gave a previous user, the below should work?

 

sed -i '0,/{/a\    "debug": false,\n    "live-restore": true,\n    "log-driver": "loki",\n    "log-opts": {\n        "loki-batch-size": "400",\n        "loki-max-backoff": "30s", \n        "loki-url": "https://loki.darkxshadows.net/loki/api/v1/push",\n        "max-buffer-size": "5m",\n        "max-file": "5",\n        "max-size": "5m",\n        "mode": "non-blocking"\n    },' /etc/docker/daemon.json

 

Edited March 11, 20251 yr by TheEyeTGuy
Added more info.

9 hours ago, TheEyeTGuy said:

Interesting it was working prior to the upgrade.

I'm not too sure why it was working since it shouldn't have.

 

9 hours ago, TheEyeTGuy said:

So maybe a better way would be to use sed? I'm a complete novice to sed, but modifying what you gave a previous user, the below should work?

I would like to recommend that you look at the default daemon.json:

{
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}

and then add this to your /boot/config/daemon.json with that you can still use your copy method.

On 3/4/2025 at 8:05 AM, ich777 said:

Are the containers starting?

 

I just tried it on my test server and everything seems to work fine.

 

From your Diagnostics I see nothing obvious, did you change the hardware or anything similar?

How do you check if the transcoding/machine learning is not working?

Hi ITch long story short, have to un-install the plugins & Drivers and then reboot. Then reinstall and put in the community driver & Reboot then finally put in the Nvidia driver and reboot and that seemed to of worked. 

Edited March 11, 20251 yr by Rkpaxam

16 hours ago, ich777 said:

I'm not too sure why it was working since it shouldn't have.

 

I would like to recommend that you look at the default daemon.json:

{
    "runtimes": {
        "nvidia": {
            "path": "/usr/bin/nvidia-container-runtime",
            "runtimeArgs": []
        }
    }
}

and then add this to your /boot/config/daemon.json with that you can still use your copy method.

 

Perfect. I'll stick with what I know, haha. Thank you for the quick responses!

On 3/10/2025 at 9:41 AM, ich777 said:

Yes it is.
 

The Diagnostics that you've posted in the other thread are not really helpful since no Nvidia card is visible, most certainly because you've disconnected the external power, however it should display a message that the external power plugs are not plugged in.

 

It would be helpful if you would include which containers are using the GPU.

Please note that a few people already reported here that AI containers cause some times a Call Trace which ultimately results in the Server crashing <- but not in all cases the whole server crashes.

 

Have you yet considered that a container causes this and yet tried enable container by container until it happens again, as said above, a few people already reported the GPU not working after some time when using AI containers.

Maybe try to disable all your AI containers and only enable Plex as a starting point, when that works you can enable the next container in addition to Plex.

 

Please try what I recommended above. If your Server crashes just with Plex enabled then possibly your GPU is at fault but this is only a assumption, please make also sure that your Power Supply is up to the task, so to speak can it deliver enough power and is it not too old since power supplies degrade over time and if a power spike happens it can take down the whole server.

 

Hope that helps, the driver is indeed working and your GPU is supported by the driver (regardless from which version).

Hi,

I've been experiencing repeated hard crashes on my Unraid 7.0.0 server. The system becomes completely unresponsive — requiring a power cycle — and I’ve been unable to reproduce the issue on demand. It seems to occur once every day or every other day.

To help isolate the cause, I’ve enabled the syslog server with logs mirrored to the flash drive. After the most recent crash, I reviewed the logs and found what appears to be a kernel-level hang involving the NVIDIA driver.

I'm currently using:

• NVIDIA RTX 2080 Ti
• Driver Version: 2025.02.10
  Unraid Version: 7.0.0

rcu: INFO: rcu_preempt detected stalls on CPUs/tasks
Sending NMI from CPU 3 to CPUs 9:
NMI backtrace for cpu 9
CPU: 9 PID: 846208 Comm: nvidia-smi Tainted: P      D W  O       6.6.68-Unraid
os_acquire_spinlock [...] [nvidia]
native_queued_spin_lock_slowpath [...]
rm_perform_version_check [...] [nvidia]
nvidia_unlocked_ioctl [...] [nvidia]

 

The above output from syslog-previous suggests that the system hangs due to a deadlock/spinlock issue within the NVIDIA kernel module, triggered during a call to nvidia-smi. I’m not running any VMs, and I haven’t bound any devices to VFIO. However, I do have Docker containers using the GPU — including Immich (AI/photo analysis) and others that transcode media.

I’ve attached my syslog-previous file taken right after the crash. It includes the full trace and related kernel output.

 

Any feedback or driver version recommendations would be hugely appreciated. Let me know if there are additional diagnostics I should run or if you'd like a full system diagnostic zip.

syslog-previous

23 minutes ago, cinema-plug said:

Any feedback or driver version recommendations would be hugely appreciated. Let me know if there are additional diagnostics I should run or if you'd like a full system diagnostic zip.

Please post your Diagnostics.

 

24 minutes ago, cinema-plug said:

triggered during a call to nvidia-smi

Do you have the Dashboard open somewhere when that happened?

 

25 minutes ago, cinema-plug said:

I do have Docker containers using the GPU

Do you have other containers related to AI also running, lately I got many reports about hard locks with Nvidia cards from users who are running LLMs and other AI containers, I suspect that the containers are responsible for the crash.

 

Are you also sure that your Power Supply is up to the task since Power Supplies also degrade over time and if there is a spike in power draw such things can also happen.

 

 

And since you have this line in your syslog:

Mar 10 08:05:31 Server-Name emhttpd: /usr/local/emhttp/plugins/user.scripts/backgroundScript.sh "/tmp/user.scripts/tmpScripts/Unlock NVIDIA/script" >/dev/null 2>&1/usr/local/emhttp/plugins/user.scripts/backgroundScript.sh "/tmp/user.scripts/tmpScripts/Install qBittorrent API/script" >/dev/null 2>&1

 

No further support until you removed that and evaluated that crashes are happening without that script, please read the first post of this thread, right at the top, red text.

14 minutes ago, ich777 said:

Please post your Diagnostics.

 

Do you have the Dashboard open somewhere when that happened?

 

Do you have other containers related to AI also running, lately I got many reports about hard locks with Nvidia cards from users who are running LLMs and other AI containers, I suspect that the containers are responsible for the crash.

 

Are you also sure that your Power Supply is up to the task since Power Supplies also degrade over time and if there is a spike in power draw such things can also happen.

 

 

And since you have this line in your syslog:

Mar 10 08:05:31 Server-Name emhttpd: /usr/local/emhttp/plugins/user.scripts/backgroundScript.sh "/tmp/user.scripts/tmpScripts/Unlock NVIDIA/script" >/dev/null 2>&1/usr/local/emhttp/plugins/user.scripts/backgroundScript.sh "/tmp/user.scripts/tmpScripts/Install qBittorrent API/script" >/dev/null 2>&1

 

No further support until you removed that and evaluated that crashes are happening without that script, please read the first post of this thread, right at the top, red text.

Thanks for the response.

 

I've attached my full diagnostics as requested (cinema-plug-diagnostics-20250313-0844.zip).

 

To address your questions:

• Dashboard Access: Yes, I typically have the Dashboard tab open on a separate PC for quick monitoring. I’ll keep that closed during further testing to see if it impacts stability.
• AI Containers: The only AI-related container I run is Immich, and I’ve experienced crashes even with Immich stopped. No other LLMs or AI-intensive containers are running.
• Power Supply: I’m confident the PSU isn’t the issue. It’s relatively new, from a reputable brand, and the server has been running stable for months. The crashes began only after upgrading to the latest NVIDIA driver (2025.02.10) and/or Unraid 7.0.0, so I suspect the issue is related to one of those.
• Unlocker Script: I saw the log entry for the unlocker script, this was leftover from an older configuration and hadn’t been removed. I’ve now removed it entirely. I don’t believe it was the cause of the crash, but I’ll continue testing without it and monitor the system for further issues.

Thanks again for the guidance—I'll continue logging and report back with any new findings.

cinema-plug-diagnostics-20250313-0844.zip

13 minutes ago, cinema-plug said:

Yes

Please do keep in mind that GPU Statistics calls nvidia-smi every N seconds that you've configured in the GPU Statistics settings, you could try to remove it for now and see if it still crashes.

 

I won't recommend leaving open a Dashboard somewhere because it is a wast of resources and also energy in my opinion, no matter how cheap electricity is... Please do keep in mind that this is only my opinion and I'm not here to judge, just saying.

 

However if it's caused by nvidia-smi this is most certainly the cause of the issue, either try to remove GPU Statistics or increase the polling time to 10 or even 20 seconds if that would be suitable for you.

 

16 minutes ago, cinema-plug said:

Immich

Shouldn't cause a crahs, AFAIK only LLMs and other AI containers are problematic.

 

18 minutes ago, cinema-plug said:

I suspect the issue is related to one of those.

Please do keep in mind that containers are also updated and that containers can also cause a Kernel crash because Docker or LXC containers share the same Kernel as the host in a isolated namespace but it's the same namespace.

 

11 minutes ago, cinema-plug said:

I don’t believe it was the cause of the crash

You'll never know since you are messing with the libraries and as pointed out in the first post no support until it is clearly visible that it is removed (it's still in the Diagnostics).

 

13 minutes ago, ich777 said:

Please do keep in mind that GPU Statistics calls nvidia-smi every N seconds that you've configured in the GPU Statistics settings, you could try to remove it for now and see if it still crashes.

 

I won't recommend leaving open a Dashboard somewhere because it is a wast of resources and also energy in my opinion, no matter how cheap electricity is... Please do keep in mind that this is only my opinion and I'm not here to judge, just saying.

 

However if it's caused by nvidia-smi this is most certainly the cause of the issue, either try to remove GPU Statistics or increase the polling time to 10 or even 20 seconds if that would be suitable for you.

 

Shouldn't cause a crahs, AFAIK only LLMs and other AI containers are problematic.

 

Please do keep in mind that containers are also updated and that containers can also cause a Kernel crash because Docker or LXC containers share the same Kernel as the host in a isolated namespace but it's the same namespace.

 

You'll never know since you are messing with the libraries and as pointed out in the first post no support until it is clearly visible that it is removed (it's still in the Diagnostics).

 

 

Thanks for the follow-up.

 

You're right — I had the GPU Statistics plugin set to poll every 2 seconds and all options were enabled (including NVIDIA-specific metrics). I've now increased the polling interval to 20000 ms, and if the issue continues, I’ll remove the plugin entirely for further testing.

 

Also, I completely understand the dashboard note — I’ve closed it and won’t keep it open moving forward.

 

As mentioned earlier, the unlocker script has been fully removed. It's likely still in diagnostics as I haven't forced a system restart, yet. 

 

I’ll continue monitoring now with those changes in place. Will report back once I’ve had a few days to test stability under these conditions.

Good morning everyone, I'm somewhat newer to the on rate scene and I have installed two Nvidia RTX Quadros 6000. The issue I'm running into is I can install the new video plugins no problem but I believe they are installing the consumer versions of the driver when I need the data center server version of the drivers. Reason this is an issue is I am seeing Significantly poor performance off of the consumer drivers and during testing I uninstalled the Nvidia plugin and the Gpus ran amazing, is there any way to install those drivers I've done a ton of research and I've tried as many different things to know success any help is greatly appreciated.

 

 

1 hour ago, InfrastructureAdmin6969 said:

Reason this is an issue is I am seeing Significantly poor performance off of the consumer drivers and during testing I uninstalled the Nvidia plugin and the Gpus ran amazing, is there any way to install those drivers I've done a ton of research and I've tried as many different things to know success any help is greatly appreciated.

What driver did you use specifically? Did you try the open source variant too?

 

The workstation driver is not part of this plugin, I have to look into if it would be even possible to create precompiled packages like it is the case for the existing plugin.

I have tried installing v570.124.04, v565.77, v565.57.01, v550.144.03, as well as the latest, production branch, and new feature branch. I have not however installed the Open Source Driver.

Driver I am trying to install is the following: "Data Center Driver for Linux x64 570.124.06 | Linux 64-bit"

 

44 minutes ago, InfrastructureAdmin6969 said:

Driver I am trying to install is the following: "Data Center Driver for Linux x64 570.124.06 | Linux 64-bit"

I know what you are trying to do but you need a complete development environment since you have to compile the driver, then create a package and after that you have to install that on Unraid.

 

Please keep in mind that if you reboot the installation will be gone if you don't do it like that.

 

What Unraid version are you on? I should be able to compile a test package for you.

6.12.15, but wiling to upgrade if it makes your life easier.

47 minutes ago, ich777 said:

I know what you are trying to do but you need a complete development environment since you have to compile the driver, then create a package and after that you have to install that on Unraid.

 

Please keep in mind that if you reboot the installation will be gone if you don't do it like that.

 

What Unraid version are you on? I should be able to compile a test package for you.

 

It isn't said enough: This man is the best.  👏

1 hour ago, InfrastructureAdmin6969 said:

6.12.15, but wiling to upgrade if it makes your life easier.

Please upgrade if possible and let me know to what Unraid version you upgrading to.

now getting 

Vendor command returned unparseable data. on top plugin change to .7 1.3 but only seems to happen when transcoding? 

15 minutes ago, Rkpaxam said:

now getting 

Vendor command returned unparseable data. on top plugin change to .7 1.3 but only seems to happen when transcoding? 

Please report that in the correct support thread which is the GPU Statistics Plugin support thread.

I am updating to 7.0.1 now.

I have an Nvidia Quadro P2000 card I've been using for a few years with unraid 6.12.x.  

 

I'm trying to update to 7.0.1 and got this warning after the install (but before rebooting):

 

 

 

On the Nvidia settings tool, i see this:

 

Edited March 18, 20251 yr by tmchow

3 minutes ago, tmchow said:

On the Nvidia settings tool, i see this:

Your plugin is broken (you'll see that on the two Deprecated lines, these should be not there and is a clear indication that something is not working correctly), please do the following:

1. Uninstall the plugin
2. Go to Settings -> Disk Settings -> set "Enable auto start" to "No" -> click Apply
3. Reboot
4. Install the Nvidia Driver plugin again (don't start the Array)
5. Go to Settings -> Disk Settings -> set "Enable auto start" to "Yes" -> click Apply
6. Reboot

 

After that everything should be back to normal.

46 minutes ago, InfrastructureAdmin6969 said:

I am updating to 7.0.1 now.

This will take me a bit (probably until tomorrow), I'll get back to you with a Private Message.

Sounds good, thank you very much!