Hi

 

I've been having a strange issue the last 2 weeks, trying to get my RTX 2080 working in my server. Until 2 months ago, it was used for my vm. Then took the card out for egpu purpose, and now regret it.

 

I can se my card in system devices, but nvidia-smi shows no devices.

 

 

 

 

The card is quite hot and one of the two fans are spinning at maximum.

unraid-diagnostics-20250403-2157.zip

12 minutes ago, eviln said:

I've been having a strange issue the last 2 weeks

First of all, please remove those lines from your go file, these are not needed anymore:

# force iptable mangle module to load (required for *vpn dockers)
/sbin/modprobe iptable_mangle
modprobe i915
chmod -R 777 /dev/dri

(at least the last two lines are safe to remove).

 

You've also seen this (your syslog is full of those messages) :

Apr  3 16:00:01 unraid shfs: share cache full
Apr  3 16:00:01 unraid shfs: assign_disk: /appdata/tdarr/logs/Tdarr_Server_Log.txt (28) No space left on device

 

Because of those messages I can't extract any viable information from your syslog because as said, it's full of those messages, however from your previous boot I can see this message which indicates an issue with your card:

Apr  1 19:54:30 unraid kernel: NVRM: Xid (PCI:0000:01:00): 140, An uncorrectable ECC error detected (possible firmware handling failure) DRAM:-1840691974, LTC:0, MMU:0, PCIE:0

 

You can learn more about XID errors here, however Error code 140 usually means something is wrong with the hardware.

@ich777 Thank you for the quick response.

 

I'm an idiot.. Set the GPU to vfio at boot and made a vm with both virtuel and the gpu.

 

At boot the VM showed "Connect the PCIe power cables for all graphic cards". Then i remembered my PCIe extender cables were not connected in the other end...

 

Now its working again.

I have a GT730 for basic video display in windows 11 and MacOS using OCLP.

Is there a way to install the legacy nvidia driver 470.x in UNRAID so I can monitor the stats of the card in gpu statistics? 

30 minutes ago, mikeyosm said:

Is there a way to install the legacy nvidia driver 470.x in UNRAID so I can monitor the stats of the card in gpu statistics? 

A bit more information please, on what Unraid version are you? Diagnostics would always help, even for such a basic question.

 

However, if you are on Unraid 7.1.0-beta series then the answer is no since the legacy driver is not compatible with Kernel 6.12+

 

You are also mentioning that you using the card for usage in VM, you know that you shouldn't install this plugin if you are using the card in VMs, first post in this thread:

1 hour ago, ich777 said:

A bit more information please, on what Unraid version are you? Diagnostics would always help, even for such a basic question.

 

However, if you are on Unraid 7.1.0-beta series then the answer is no since the legacy driver is not compatible with Kernel 6.12+

 

You are also mentioning that you using the card for usage in VM, you know that you shouldn't install this plugin if you are using the card in VMs, first post in this thread:

OK, that explains it. Thanks for the explanation.

On 3/30/2025 at 2:00 PM, msalad said:

Hey @ich777,

 

I did the steps you indicated. Plex booted with my nvidia gpu passed through without issue. I installed the gpu statistics plugin and it worked as expected.

 

I also have an intel A310 gpu in my server and was able to install intel_gpu_top, which worked as expected as well. Interestingly, in my original server, the gpu statistics plugin only works with my nvidia gpu, not my intel gpu. Not sure if this is relevant but just wanted to give you as much info as I can. This reminded me that I also installed intel_gpu_top while still on unraid 6.12.14, immediately before upgrading to unraid 7.0.0. Maybe that caused a conflict?

 

It seems like something is broken with my original server and is not related to my hardware. I think it's time to format my original usb, install fresh unraid 7.0.1, and rebuild my server piece by piece. Would you agree?

Hi @ich777,

 

Final update

 

I was able to fix the issue 🙂

 

I used a new flash drive and fresh Unraid 6.12.15 install to rebuild my server from scratch.

 

When I booted up the array for the first time, all of my data and dockers came back. I then did a fresh install of the Nvidia driver plugin, and to my amazement I still had the nvidia-container-cli permissions issue when starting Plex w/ my gpu passed through.

 

To fix, I found that an specific order of operations key:

 

My first attempt: 1) delete docker.img, 2) reinstall Plex docker and configure gpu passthrough, 3) install Nvidia driver, 4) disable + enable docker, 5) restart.

 

This failed - I encountered the nvidia-container-cli permissions issue again on my next boot.

 

Attempt two: 1) disable docker, delete docker.img, enable docker, 2) install Nvidia driver, 3) disable + enable docker, 4) install Plex docker and configure gpu passthrough, 5) restart.

 

WORKS!!

 

For some reason, it was crucial that I a) delete my docker.img and then b) install the Nvidia driver first before any of my old dockers.

 

I do wonder if deleting my docker.img and the other steps I took would have fixed the issue in my original Unraid install but I'm not going to get hung up on that. 

 

This saga is finally at an end. Thanks for all of your help!

Hi All,

 

I'm currently having issues getting my 5070 ti not being detected. Currently running on 7.1.0-rc1 due to battlemage support.

Plex docker detects it with the app, and VMs can see it but the NVIDIA plugin is returning no devices.

Plugin



Plex


VM



I do see interesting logs but from the driver plugin i can't use the open source ones

diagnostics-20250412-1042.zip

3 hours ago, lien said:

I'm currently having issues getting my 5070 ti not being detected. Currently running on 7.1.0-rc1 due to battlemage support.

Please go to the plugin page again and select the Open Source driver (blue one), click Update & Download, wait for the download to finish and reboot your server.

What version driver should i be using if i have pascal (p4) and RTX4000?

10 hours ago, kerbys said:

What version driver should i be using if i have pascal (p4) and RTX4000?

Any of the drivers should work.

Depending of course on what Unraid version you on.

 

Please post Diagnostics if your card is not recognized.

Heyo,
Have an issue that I've implemented a temporary fix on. Recently updated from Unraid 6 to Unraid 7.0.1 which caused the Nvidia Driver to disappear upon every reboot. Installing the Nvidia plugin brand new works until another reboot which is the concistent pattern. Looking inside the `/boot/extra` folder revealed that nothing was placed in there. AFAIK, an nvidia driver *.txz file must be present for unraid to install the driver on bootup. So I copy/pasta'd like so due to not being able to make a symlink: 

cp /boot/config/plugins/nvidia-driver/packages/6.6.78/nvidia-570.133.07-6.6.78-Unraid-1.txz /boot/extra/

This works, the driver is permanently present regardless of reboot and all my docker containers use my graphics card correctly. However, I imagine any update to the driver will not actually be done as it would have to change this file in the `/boot/extra` directory which it wasn't doing in the first place. Any ideas how to fix this?

NOTE: I did have a few packages installed by Nerd Tools which is no longer supported in Unraid 7+ AFAIK. I think I've removed everything having to do with this, but there could be something left over. 

diagnostics-20250413-1858.zip

Edited April 14, 20251 yr by pa77y

3 hours ago, pa77y said:

This works, the driver is permanently present regardless of reboot and all my docker containers use my graphics card correctly. However, I imagine any update to the driver will not actually be done as it would have to change this file in the `/boot/extra` directory which it wasn't doing in the first place. Any ideas how to fix this?

regarding the logs, the driver is now installed 2x ... from /boot/extra and from the plugin.

 

regardless, it should work without your /boot/extra "sideload"

 

what i see, you run a script called "nvidia gpu calm", may tried without that ? depending now what is in there may ...

4 hours ago, pa77y said:

Looking inside the `/boot/extra` folder revealed that nothing was placed in there. AFAIK, an nvidia driver *.txz file must be present for unraid to install the driver on bootup.

Where do you have that information from?

 

As @alturismo already pointed out, your driver is now installed twice, one time here:

Apr 13 18:25:56 Boogaloo --: Installing: nvidia-570.133.07-6.6.78-Unraid-1:

and the second time here (which then also activates the driver) :

Apr 13 18:26:01 Boogaloo plugin-manager: checking: /boot/config/plugins/nvidia-driver/nvidia-driver-2025.03.25.txz - MD5

 

However what is this script:

/tmp/user.scripts/tmpScripts/calm nvidia gpu/script

 

However what is really strange from what I see in your previous syslog is that the driver isn't even installed to begin with however I really can't tell what's going on since I would need a Diagnostics file where it doesn't work.

 

In your previous syslog I see that the Nvidia driver isn't even installed and you install it way later after Unraid is already started:

Apr 13 17:29:12 Boogaloo plugin-manager: creating: /boot/config/plugins/nvidia-driver/nvidia-driver-2025.03.25.txz - downloading from URL https://github.com/unraid/unraid-nvidia-driver/raw/master/packages/nvidia-driver-2025.03.25.txz

then it got installed and loaded:

Apr 13 17:33:58 Boogaloo kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 239

 

Then you've installed the latest Open Source version, may I ask why you did that, after that you installed the latest driver:

Apr 13 18:01:40 Boogaloo emhttpd: cmd: /usr/local/emhttp/plugins/nvidia-driver/include/exec.sh update_version latest

(did you wait here until it says the download finished and it is safe to close the window)

 

and then you've rebooted:

Apr 13 18:21:56 Boogaloo shutdown[375947]: shutting down for system reboot

 

After the reboot something is definitely wrong since your machine won't even try to install the driver early in the boot process, again, as said, I would really need some Diagnostics where it doesn't work.

@alturiso
Indeed, copying the driver to have it installed twice is suboptimal. As for the `calm gpu` script. It just runs an `nvidia-smi` command. Disabling has no effect:

Quote

#!/bin/bash
#set persistence mode
nvidia-smi -pm 1

@ich777
Ah, apologies. Here's the Diagnostics where the nvidia issue occurs. Nothing pops out at me nvidia wise. All the logs concerning the open source driver was me trying different things. Obviously, none worked. As for where I heard about the `/boot/extra` file, it was via a LLM that referenced a post from here somewhere. It was good enough for me to give it a shot.

NOTE: The driver is not seen at all. The following result in the terminal:
 

nvidia-smi

-bash: nvidia-smi: command not found

EDIT: @ich777, I always waited till the graphics download and install completed before exiting the plugin install prompt. Everything worked as expected at these points. Its just the loss of driver persistence after reboot thats the issue.

EDIT2: Noticed that the only time there's an issue in the logs is on the monitor plugged into the unraid server pc upon bootup. I can't find where these logs are in unraid easily so I'll paste the interesting stuff here. I've included a few lines above and below that don't look related:

...
update_services: delay = 45s
rc.local: Installing /boot/extra packages
xz: (stdin): Compressed data is corrupt
cat: write error: Broken pipe
modprobe: FATAL: Module nvidia not found in directory /lib/modules/6.6.78-Unraid
rc.local: Installing language packs
...

I don't know what the corrupt data log is about. The /boot/extra folder is empty. Same with the Broken pipe. The /lib/modules/* is populated when I ssh into the machine. It looks properly setup. Could it be that its not in place when this bit of logic runs? This is all occurring before the initial TTY is setup.
 

boogaloo-diagnostics-20250414-0144.zip

Edited April 14, 20251 yr by pa77y

2 hours ago, pa77y said:

I don't know what the corrupt data log is about. The /boot/extra folder is empty. Same with the Broken pipe. The /lib/modules/* is populated when I ssh into the machine. It looks properly setup. Could it be that its not in place when this bit of logic runs? This is all occurring before the initial TTY is setup.

This is an indication that the driver package fails to extract/install on your system for whatever reason.

I have nouw double checked the filesize from the package which is exactly the same over here for the version that you are using.

 

2 hours ago, pa77y said:

xz: (stdin): Compressed data is corrupt

I'm not 100% sure why this is not working through the plugin nor the integrated method that Unraid uses much earlier in the boot process on your system.

 

However I would recommend that you go into the BIOS and disable all overclocking features both on the CPU and RAM and retest.

 

2 hours ago, pa77y said:

As for where I heard about the `/boot/extra` file, it was via a LLM that referenced a post from here somewhere. It was good enough for me to give it a shot.

Sorry but don't do it like that, this is a really dirty workaround and shouldn't be done this way since Unraid has it's own built in mechanisms to install drivers and I don't support the way through /boot/extra.

I’m getting error notifications from the nvidia plugin

 

Quote

unRAID Server: Notification
 

Link: https://Brunnhilde.dahoney.me/Settings/nvidia-driver

Event: Nvidia Driver
Subject: Notification
Description: Found new Nvidia Driver v570.133.07 but a download error occurred! Please try to download the driver manually!
Importance: alert
 

But when I go to the plugins tab and try to manually download I get a popup showing that the updated plugin has already been downloaded.

 

1 hour ago, wgstarks said:

I’m getting error notifications from the nvidia plugin

Please try to remove the plugin entirely (you containers will still work as expected) and then install the Nvidia Driver plugin from the CA App again and see if you get the same error?

fyi on my problem. `modprobe: FATAL: Module nvidia not found in directory /lib/modules/6.6.78-Unraid`

I ended up installing a fresh unraid while keeping all my disks, shares, docker containers, etc. The problem magically disappeared. I updated my bios to most recent before doing this, but it had no effect. Everything works as expected now.

@ich777, @alturismo, thanks for the diagnoses aid. Its always nice to have a forum to bounce around ideas in.

EDIT: Oh yeah, there was no overclocking set on the motherboard. It was all on "auto". So, as compatible/stable as I can see it being. That said, it is an 

ASRock motherboard. Not the most stable brand AFAIK

Edited April 15, 20251 yr by pa77y

49 minutes ago, pa77y said:

I ended up installing a fresh unraid while keeping all my disks, shares, docker containers, etc. The problem magically disappeared. I updated my bios to most recent before doing this, but it had no effect. Everything works as expected now.

This is really strange since I had another user with a similar issue not long ago and I couldn't explain why that was happening but it's good to know that this solved your issue.

Something must be interfering with the installation or whatever, I really don't know what could cause this.

 

However it's great that you've let me know and that your issues is solved!

Thanks for the reply.

So, I had the issue where my GPU would come alive for a moment then go back to sleep. (Again)

The RTX 4060 would start to transcode for a moment then go back to sleep then everything would fall back to the CPU.

 

I never got around to testing the P2000 but that cards dedicated to Tdarr for converting older formats to h265.

 

I found that if i commented out "nvidia-persistenced" from my Go file then everything started working perfectly....... on the latest driver too.

The 4060 is idling like it's suppoesd to but the P2000 keeps waking up and going back to sleep.

It states it's only idling at 20 watts but I think it can do better? Is there a way to send "nvidia-persistenced" to just the P2000?

 

EDIT, I commented out "nvidia-persistenced" because it's the only thing i could think of that was still poking at the driver.

Edited April 18, 20251 yr by AnnabellaRenee87

4 hours ago, AnnabellaRenee87 said:

Is there a way to send "nvidia-persistenced" to just the P2000?

yes, but be aware, nvidia is considering to drop this feature (at least thats calles the last years )

 

as sample, im using it to en / disable on VM Startup / Shutdown.

 

depending now on card from nvidia-smi output as sample for 2 cards

 

nvidia-smi -i 0 -pm 1

nvidia-smi -i 1 -pm 0

 

card 0 (1st card) pers mode enable

card 1 (2nd card) pers mode disable

 

so, -i (0,1,2,3,...) to specify which card, -pm to en/disable pers mode (1 enable, 0 disable)

7 hours ago, AnnabellaRenee87 said:

So, I had the issue where my GPU would come alive for a moment then go back to sleep. (Again)

May I ask first which container are you testing transcoding with?

 

The only time that I hear from such issues is in combination with Plex and trying transcoding in the WebPlayer.

It's a known issue that the Plex WebPlayer shows just a quick transcoding process on the GPU and then either showing a black screen or falling back to the CPU transcoder when switching the quality and trying to force a transcode.

 

For proper testing you should use a Native Plex Client like the Android or iOS App and forcing a transcode there or try it with Jellyfin.

 

The transcoder log should/could also give more information about that <- but it's not always that obvious what the problem is.

 

2 hours ago, alturismo said:

yes, but be aware, nvidia is considering to drop this feature (at least thats calles the last years )

It has been finally deprecated with the newer drivers, I think starting with version 570+

I was testing with Plex on my phone while on 5G/LTE.

I was also testing with Emby and JellyFin (Yes i have all 3 runing because I'm a nerd)

Don't suppose you can keep the v565.77 drivers as legacy for the immedeate future? I ask because where Pascal is apparently going to be dropped at some point. I also need to find another single slot card (that I can afford) that can handle doing transcodes of a show from a totally legal place that may be in MP4 or AVI lol to H265.

5 hours ago, AnnabellaRenee87 said:

Don't suppose you can keep the v565.77 drivers as legacy for the immedeate future?

Sorry but no, since these are really not legacy drivers, at least not now. 😅

 

6 hours ago, AnnabellaRenee87 said:

I ask because where Pascal is apparently going to be dropped at some point.

That‘s true but first they will be legacy drivers, I can‘t imagine that they drop these card immedeately but I also have to say these cards are now really old.

 

6 hours ago, AnnabellaRenee87 said:

I also need to find another single slot card (that I can afford) that can handle doing transcodes of a show from a totally legal place that may be in MP4 or AVI lol to H265.

Look at something like the Nvidia A400 or A1000

 

However the issue that you are experiencing is really weird since no one else reported such an issue and many more people have enabled nvidia-persistenced, do you already habe new hardware? Maybe it‘s related to the old hardware that you are running.