1 minute ago, BoKKeR said:

I am posting diagnostic s before and after update to rc,  the update seems to have not fixed the issue. 

That's why Diagnostics are always important, it's not working because your Nvidia GPU is bound to VFIO.

Apr 25 10:56:32 unraid kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 240
Apr 25 10:56:32 unraid kernel: NVRM: GPU 0000:07:00.0 is already bound to vfio-pci.
Apr 25 10:56:32 unraid kernel: NVRM: The NVIDIA probe routine was not called for 1 device(s).
Apr 25 10:56:32 unraid kernel: NVRM: This can occur when another driver was loaded and 
Apr 25 10:56:32 unraid kernel: NVRM: obtained ownership of the NVIDIA device(s).
Apr 25 10:56:32 unraid kernel: NVRM: Try unloading the conflicting kernel module (and/or
Apr 25 10:56:32 unraid kernel: NVRM: reconfigure your kernel without the conflicting
Apr 25 10:56:32 unraid kernel: NVRM: driver(s)), then try loading the NVIDIA kernel module
Apr 25 10:56:32 unraid kernel: NVRM: again.
Apr 25 10:56:32 unraid kernel: NVRM: No NVIDIA devices probed.
Apr 25 10:56:32 unraid kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 240

 

And your VFIO config:

Processing 0000:07:00.0 10de:2482
Vendor:Device 10de:2482 found at 0000:07:00.0

IOMMU group members (sans bridges):
/sys/bus/pci/devices/0000:07:00.0/iommu_group/devices/0000:07:00.0
/sys/bus/pci/devices/0000:07:00.0/iommu_group/devices/0000:07:00.1

Binding...

success...

 

And your PCI device:

07:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA104 [GeForce RTX 3070 Ti] [10de:2482] (rev a1)
	Subsystem: ASUSTeK Computer Inc. Device [1043:8813]
	Kernel driver in use: vfio-pci
	Kernel modules: nouveau, nvidia_drm, nvidia
07:00.1 Audio device [0403]: NVIDIA Corporation GA104 High Definition Audio Controller [10de:228b] (rev a1)
	Subsystem: ASUSTeK Computer Inc. Device [1043:8813]
	Kernel driver in use: vfio-pci

 

 

Unbind it from VFIO and it will work again.

Thanks that solved it, works also on 7.0.1 seems like.

Hello,

 

I recently upgraded my unraid from 7.0 to 7.1 latest version. after upgrade my NVIDIA GPU having issues and all docker containers using this NVIDIA GPU did not start and getting runtime error. Then i looked at the driver utility and i saw "ERROR: No driver found!". I also tried to revert back to 7.0 version and now i am still getting same GPU and app install issue in that version as well. I also tried to reinstall NVIDIA Driver utility manually but still not resolved. I also tried all possible solution available online including this tread but no luck. Note: I do see NVIDIA driver showing available in the system devices so ruled out the physical issue with NVIDIA GTX 1660 SUPER card. I am stuck now and all my key docker containers are down due to GPU not being available. 

2 hours ago, oldmunk said:

Then i looked at the driver utility and i saw "ERROR: No driver found!".

Please post your Diagnostics, without them troubleshooting is not possible for hardware devices.

I have an issue with my GTX 1060

The devices have been detected on System Devices, and on lspci

However, the latest and production drivers didn't detect it. When I put the open source driver, I got the error "unraid NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running."

 

Attached are the screenshots, and the diagnostics file,
Thank you

zenithlab1.0-diagnostics-20250505-1810.zip

50 minutes ago, zenaufa said:

I have an issue with my GTX 1060

The devices have been detected on System Devices, and on lspci

This is odd since it says that the driver is in use:

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] [10de:1c03] (rev a1)
	Subsystem: ZOTAC International (MCO) Ltd. Device [19da:1438]
	Kernel driver in use: nvidia
	Kernel modules: nvidia_drm, nvidia
01:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)
	Subsystem: ZOTAC International (MCO) Ltd. Device [19da:1438]

 

Did you make any changes lately to your system (BIOS Update, Add/Remove hardware,...)?
I see that the initialization fails:

May  5 18:09:58 ZenithLab1 kernel: NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x24:0x72:1589)
May  5 18:09:58 ZenithLab1 kernel: NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0
May  5 18:09:59 ZenithLab1 kernel: NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x24:0x72:1589)
May  5 18:09:59 ZenithLab1 kernel: NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0

 

Please also make sure that you are on the latest BIOS version and please check to enable Above 4G Decoding and Resizable BAR support in your BIOS.

 

I assume the card was working before in this Server or did you just install the card?

This is a new card i got it second hand from the marketplace, I was using P400 before, and it's working smoothly.

 

I have enabled the Above 4G Decoding and Resizeable BAR in my BIOS
I also plugin the GPU into a monitor via HDMI, and can see the computer booting and unraid console there.

Edited May 5, 20251 yr by zenaufa

42 minutes ago, zenaufa said:

This is a new card i got it second hand from the marketplace, I was using P400 before, and it's working smoothly.

May I ask why the switch to a GTX1060, Pascal will be Soon be "legacy" and not be supported by the latest driver version. Something like a Nvidia T400, T600 or even T1000 should be way enough for a few simultaneous transcodes.

 

I just saw that you have a Ryzen 1st gen CPU, I think you have to disable C-States or was it P-States IIRC so that all Nvidia cards work properly.

 

45 minutes ago, zenaufa said:

I also plugin the GPU into a monitor via HDMI, and can see the computer booting and unraid console there.

That doesn't mean that the card works properly, do you have another computer where you could test the 1060 and install the Nvidia driver?

The card want's to initialize but it refuses to initialize, this could be due to a Firmware (BIOS) issue or the card being defective.

I'm having some issues with fan control, starting with unraid 7.0.x I've added 2 Dell A4000 GPU's to my Dell R740xd server. Running 2 cards, the fan control state = 0 seems to work, fans spin up with the GPU core temperature. I've now added 2 more Dell A4000 GPU's to my server, with a total of 4 cards. This seems to break fan control, both automatic and manual. Same issue with unraid 7.1 released yesterday. I've tried multiple drivers, latest: v575.51.02, Open Source Driver: v570.144 and Production Branch: v570.144. It does not make a difference. 

GPU0 and GPU1 seems to have basic fan control, but they are always spinning at around 30%, no matter of temperature. GPU2 and GPU3, the fans never spin at all. This leads to the cards quickly hitting 103c and throttling down to more or less 0 power. 

I've tried to manually control the fans using:

nvidia-settings -c :0 -a [gpu:0]/GPUFanControlState=1 -a [fan:0]/GPUTargetFanSpeed=80 
nvidia-settings -c :0 -a [gpu:1]/GPUFanControlState=1 -a [fan:1]/GPUTargetFanSpeed=80 
nvidia-settings -c :0 -a [gpu:2]/GPUFanControlState=1 -a [fan:2]/GPUTargetFanSpeed=80 
nvidia-settings -c :0 -a [gpu:3]/GPUFanControlState=1 -a [fan:3]/GPUTargetFanSpeed=80 

nvidia accepts the settings, but only gpu0 and GPU1 are set to fan speed 80, gpu 2-3 are still set at 0%. Changing to:

nvidia-settings -c :0 -a "[gpu:0]/GPUFanControlState=0"
nvidia-settings -c :0 -a "[gpu:1]/GPUFanControlState=0"
nvidia-settings -c :0 -a "[gpu:2]/GPUFanControlState=0"
nvidia-settings -c :0 -a "[gpu:3]/GPUFanControlState=0"

 only affects GPU0 and GPU1, but they do not go to 0% when cool, they stay at around 30%

It seems like vBIOS is not in control of the fan curve but the driver/system is always in control of the fan curve.

 

I only use the GPUS for ollama, ollama is working correctly spreading the models across the 4 GPU's

 

Edited May 6, 20251 yr by Output
added open source driver

1 minute ago, Output said:

I'm having some issues with fan control

First of all, was fan control working for you since it shouldn't work on Unraid.

IIRC no one was ever able to control the fans on Unraid.

You would need to boot in GUI mode to be even able so that nvidia-settings would properly accept the commands and even then it won't work properly AFAIK.

 

2 minutes ago, Output said:

It seems like vBIOS is not in control of the fan curve but the driver/system is always in control of the fan curve

Unless you are doing something to control the fan speed, the GPU or better speaking the vBIOS is in charge of fan control.

Are you sure that nothing in the vBIOS from the two cards are preventing manual fan adjustment?

I only noticed this because of throttling when running the models, I have not tried to control the fans before finding out that the fans do not run. The cards are all the same with the same firmware on all 4.

I tried to control fanspeed to see if I could fix the issue

 

I know it works with 2 cards, since when testing ollama with 2 cards i did monitor fan speed and temps

Edited May 6, 20251 yr by Output
added info

30 minutes ago, Output said:

I only noticed this because of throttling when running the models, I have not tried to control the fans before finding out that the fans do not run. The cards are all the same with the same firmware on all 4.

I tried to control fanspeed to see if I could fix the issue

Usually the fans or better speaking the fan curve is stored in the vBIOS and controlled by the card itself, the driver on it's own by default does nothing to the curve nor to fan control.

 

When you issue the commands that you wrote above you get no output but usually it also won't change anything because these commands do not work on Unraid.

 

30 minutes ago, Output said:

I know it works with 2 cards, since when testing ollama with 2 cards i did monitor fan speed and temps

Are those probably branded HP or Dell, it might be possible that their management software does in fact is responsible for the fan curve and even if the Firmware seems the same it must not be the same if they are branded differently.

All 4 cards are dell cards, only serial numbers differ:

In the bios, I cant find a way to control the fan curve, but as I said, 2 cards work fine.

 

Running the commands, to set fan speed control to automatic actually seems to do something:

 

Attribute 'GPUFanControlState' ([gpu:0]) assigned value 0.


Attribute 'GPUFanControlState' ([gpu:1]) assigned value 0.


Attribute 'GPUFanControlState' ([gpu:2]) assigned value 0.


Attribute 'GPUFanControlState' ([gpu:3]) assigned value 0.

 

But it does not change the actual fan values, GPU1 & 2 are fan 41% and GPU2-3 are fan 0%

 

Running commands:

nvidia-settings -c :0 -a [gpu:0]/GPUFanControlState=1 -a [fan:0]/GPUTargetFanSpeed=50  
nvidia-settings -c :0 -a [gpu:1]/GPUFanControlState=1 -a [fan:1]/GPUTargetFanSpeed=50 
nvidia-settings -c :0 -a [gpu:2]/GPUFanControlState=1 -a [fan:2]/GPUTargetFanSpeed=50 
nvidia-settings -c :0 -a [gpu:3]/GPUFanControlState=1 -a [fan:3]/GPUTargetFanSpeed=50 

output in terminal:
Attribute 'GPUFanControlState' ([gpu:0]) assigned value 1.

Attribute 'GPUTargetFanSpeed' ([fan:0]) assigned value 50.


Attribute 'GPUFanControlState' ([gpu:1]) assigned value 1.

Attribute 'GPUTargetFanSpeed' ([fan:1]) assigned value 50.


Attribute 'GPUFanControlState' ([gpu:2]) assigned value 1.

Attribute 'GPUTargetFanSpeed' ([fan:2]) assigned value 50.


Attribute 'GPUFanControlState' ([gpu:3]) assigned value 1.

Attribute 'GPUTargetFanSpeed' ([fan:3]) assigned value 50.

 

This results in GPU0 & 1 the fans are running at 50% and GPU2 & 3 fans still at 0%

 

So, the commands do work, but still only for the first 2 GPU's, not for 3 and 4.

I'm only running the commands to test why I have this issue, not that I wish to control the fans of the GPUs

 

Edited May 6, 20251 yr by Output

2 hours ago, Output said:

I'm only running the commands to test why I have this issue, not that I wish to control the fans of the GPUs

I really can't help with that sorry, I was never able to control the fans.

Is there a way to install older driver via plugin? I need v370.xx for Nvidia Quadro 2000 on HP Z800. If manualy run, driver needs gcc. Thanks!

Edited May 6, 20251 yr by kurjak

1 hour ago, kurjak said:

Is there a way to install older driver via plugin? I need v370.xx for Nvidia Quadro 2000 on HP Z800. If manualy run, driver needs gcc. Thanks!

May I first ask why do you need this driver or what is your use case for this driver, better speaking what should the card do?

Thanks for answering. I wanted to try to use some llm-s which refuse to start if nvidia driver isnt loaded. And new driver does not work. See the pic.

54 minutes ago, kurjak said:

Thanks for answering. I wanted to try to use some llm-s which refuse to start if nvidia driver isnt loaded. And new driver does not work. See the pic.

Have you seen the first recommended post on top of this thread?

 

Issue is that these old card don‘t support the container runtime and in addition to that LLMs won‘t run on them because they got too less VRAM and also they are missing feature sets.

After the latest Unraid version (7.1.0) I am unable to use my Quadro K420, because the last driver that supports this card is "470.256.02". How can I install this version of the NVIDIA driver?

 

4 hours ago, ich777 said:

Have you seen the first recommended post on top of this thread?

 

Issue is that these old card don‘t support the container runtime and in addition to that LLMs won‘t run on them because they got too less VRAM and also they are missing feature sets.

Sorry, I'm new. I will look for some supported card. Thanks!

I'm dealing with the same problem like Sander0542.

On 5/6/2025 at 8:13 PM, Sander0542 said:

After the latest Unraid version (7.1.0) I am unable to use my Quadro K420, because the last driver that supports this card is "470.256.02". How can I install this version of the NVIDIA driver?

Sorry but the "Legacy" driver isn't available any more for Kernel version 6.12.x and doesn't compile against these new Kernels, if Nvidia releases a new "Legacy" driver I'm happy to compile it but for now it seems that Pascal becomes the new "Legacy" cards.

 

Please also see this conversation on GitHub for more information: Click

28 minutes ago, kurjak said:

Sorry, I'm new. I will look for some supported card. Thanks!

No worries, I'm here to help.

 

Maybe try to look for something like a Nvidia T400, T600 or even T1000 but please be aware that those cards nothing that you wan't for LLMs and Machine Learning in general since you need huge amounts of VRAM.

@ich777 do you think it's possible to build the driver on kernel 6.12 using this kind of patch ?

https://gist.github.com/joanbm/a6d3f7f873a60dec0aa4a734c0f1d64e

Edited May 6, 20251 yr by Goudale
add more details

24 minutes ago, Goudale said:

@ich777 do you think it's possible to build the driver on kernel 6.12 using this kind of patch ?

I've already tried that and that crashed my machine twice (maybe it was because I tested it with a relatively "new" card Nvidia T400)... and I really want to ship an unstable driver.

 

Please also note that this patch will probably work for this version but when Unraid moves on to another Kernel version another patch will be needed and this is now only a matter of time...