3 hours ago, ich777 said:

9 hours ago, AnnabellaRenee87 said:

I ask because where Pascal is apparently going to be dropped at some point.

That‘s true but first they will be legacy drivers, I can‘t imagine that they drop these card immedeately but I also have to say these cards are now really old.

 

HEY!  I'm really old, but I still get the job done.  😜

Hi, I recently got the Gigabyte 5060ti Windforce OC. I am able to boot into my unraid system and am able to get video output from the card. However, my nvidia drivers is showing me: "NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running." I'm running a Ryzen 7 3700x with a B450 Tomahawk Max Mobo. I'm not sure if the half PCIE slot from the 5060ti has anything to do with it or maybe I'm setting something in my bios wrong. I've attached how the GPU looks like for reference:

 

 

 

I searched around and it says to put it to PCIE gen 4, but my mobo doesn't have that option only up to gen 3. I also saw a reddit post to put it in gen 2 but that didn't work either.

Edited April 23, 20251 yr by jayryuki

42 minutes ago, jayryuki said:

I searched around and it says to put it to PCIE gen 4, but my mobo doesn't have that option only up to gen 3. I also saw a reddit post to put it in gen 2 but that didn't work either.

Diagnostics would help a lot for such specific questions and to diagnose the issue properly.

 

However 5000 Series cards only work with the Open Source drivers, please select the Open Source driver, wait for the Download to finish and reboot your server.

1 hour ago, ich777 said:

Diagnostics would help a lot for such specific questions and to diagnose the issue properly.

 

However 5000 Series cards only work with the Open Source drivers, please select the Open Source driver, wait for the Download to finish and reboot your server.

Thank you so much, I must have missed the post about that. Thank you very much, it is working now! 

16 minutes ago, jayryuki said:

Thank you so much, I must have missed the post about that. Thank you very much, it is working now! 

Glad that everything is working now, please include Diagnostics next time you got issues with a Driver plugin.

On 4/15/2025 at 4:10 PM, pa77y said:

fyi on my problem. `modprobe: FATAL: Module nvidia not found in directory /lib/modules/6.6.78-Unraid`

I ended up installing a fresh unraid while keeping all my disks, shares, docker containers, etc. The problem magically disappeared. I updated my bios to most recent before doing this, but it had no effect. Everything works as expected now.

@ich777, @alturismo,thanks for the diagnosdiagnosests always nice to have a forum to bounce around ideas in.

EDIT: Oh yeah, there was re was clocking set on the motherboard. It was all on "auto". So, as compatible/stable as I cas I c it bee it bein saTha it is an 

ASRock motherboard. Not the most stable brand AFAIK

I'm currently having the exact same issue as you (also on an asrock mobo) , all overclocking is disabled as I was fiddling with bios settings to try to hit deep idle cstates. Unfortunately for me this coincided with an update to rc1 so I'm not sure if it was the bios or the software update that caused it. Id really like to avoid doing a fresh unraid install.

servbot-diagnostics-20250423-2318.zip

Edited April 24, 20251 yr by sage2050

4 hours ago, sage2050 said:

Id really like to avoid doing a fresh unraid install.

What is all of that in your go file:

cp /boot/custom/docker-shell /usr/local/bin

#copy mount scripts for each virtofs share
#Games
cp /boot/virtiofsd/games.sh /usr/libexec/games.sh
chmod +x /usr/libexec/games.sh

sed -i 's/$options\[\] = "--fd=".$argoptions\['"'"'fd'"'"'\];/if (isset($argoptions\['"'"'fd'"'"'\])) {\n    $options[] = "--fd=".$argoptions\['"'"'fd'"'"'\];\n}/' /usr/local/emhttp/plugins/dynamix.vm.manager/scripts/virtiofsd.php 

 

Are you also really sure you need all of that:

BOOT_IMAGE=/bzimage pcie_acs_override=downstream,multifunction modprobe.blacklist=i2c_i801,i2c_smbus vfio_iommu_type1.allow_unsafe_interrupts=1 initrd=/bzroot

 

Please note that Unraid is not designed to be general purpose server, it's an app server so I'm also questioning these packages:

-rw------- 1 root root  448876 Jan  6 11:37 btop-1.2.13-x86_64-1_SBo_UES.txz
-rw------- 1 root root   60236 Dec 12 16:23 iotop-0.6-x86_64-14cf.txz
-rw------- 1 root root   14776 Jan  6 11:37 mktorrent-1.1_de7d011-x86_64-1_SBo_UES.txz
-rw------- 1 root root 1797248 Jul 17  2024 openssl11-1.1.1za-x86_64-1.txz
-rw------- 1 root root  207368 May  3  2024 powertop-2.15-x86_64-2.txz
-rw------- 1 root root  267728 Jan  6 11:37 unrar-7.1.2-x86_64-1cf.txz

 

Did you run at some point a custom Kernel?

 

 

Yes, it's exactly the same issue since the Nvidia driver package is found on boot by the plugin:

Apr 23 22:52:45 Servbot rc.local: --------------------Nvidia driver v575.51.02 found locally---------------------
Apr 23 22:52:45 Servbot rc.local: -----------------Installing Nvidia Driver Package v575.51.02-------------------
Apr 23 22:52:45 Servbot rc.local: --------------Installation of Nvidia driver v575.51.02 successful--------------

however this is basically not correct because the package should be installed way earlier in the boot process which is not done on your system which is exactly the same issue as for the previous user who reported that issue.

 

I really don't know by what this is caused, I can only assume something is messing with rc.modules (which is responsible for installing the 3rd party drivers early in the boot process).

i honestly dont remember what the docker-shell line is for, i will look into that,

edit: its from an ancient spaceinvader one video about bashing into a running docker container. it's probably no longer necessary.

 

the second script is a vm virtiofs trick thats now deprecated, the bootimage looks like old iommu separation that's now native to unraid, that can definitely be cleaned up.

 

those pkgs are tools from the now deprecated nerd pack that i use frequently (except openssl, i dont even remember what i needed that for) so I installed them myself, not sure why they would cause any issues.

 

ive never run a custom kernel, just betas and release candidates.'

 

the successful driver install you're seeing is possibly me removing the plugin and reinstalling it, as that's the only way to get it working after a reboot, like in pa77ys case.

Edited April 24, 20251 yr by sage2050

1 hour ago, sage2050 said:

the successful driver install you're seeing is possibly me removing the plugin and reinstalling it

No, this is on plugin installation at boot and yes, I also saw that you reinstalled it later again.

 

1 hour ago, sage2050 said:

as that's the only way to get it working after a reboot, like in pa77ys case.

Sorry I really can't help, it seem like you are using a highly customized Unraid installation, not in terms of Containers or VMs since you have a lot of stuff which is now deprecated or simply not working anymore as you've pointed out above?

 

Are you also sure you don't run any User Scripts or something like that and also that your USB Flash device is working properly? Issue is that you have a lot of stuff on there which I really don't know why it's there:

/boot
total 1228160
-rw-------  1 root root   6220854 Apr  3 13:49 250403174920.BMP
-rw-------  1 root root   1440054 Apr 23 08:06 250423120618.BMP
-rw-------  1 root root   1440054 Apr 23 08:06 250423120627.BMP
-rw-------  1 root root   1440054 Apr 23 08:06 250423120628.BMP
-rw-------  1 root root   1440054 Apr 23 08:06 250423120629.BMP
-rw-------  1 root root   6220854 Apr 23 08:07 250423120730.BMP

you even have BMP files on your flash drive and other stuff which I really wouldn't put on there.

 

I really can't tell what the issue is, maybe try the following:

1. Grab a USB Key (if you have one laying around)
2. Install Unraid on there
3. Boot Unraid from this newly created USB Falsh device on your Server
4. Don't start the Array!!!
5. Install CA App by clicking on Apps
6. Download the Nvidia Driver plugin (yes this is working too if you don't start the Array)
7. Wait for the Download to finish
8. Reboot
9. Log in again
10. See if the driver was installed properly

After that you can shutdown the server and boot from your existing USB device as usual, the above is just for testing if a new installation would work and install the driver just fine.

well i updated my motherboard bios this morning and the problem went away 🤷‍♂️

1 hour ago, sage2050 said:

well i updated my motherboard bios this morning and the problem went away 🤷‍♂️

Well… that‘s strange…

 

Glad to hear that it‘s resolved.

--------Nothing to do, Nvidia Driver v575.51.02 already downloaded!---------

------------------------------Verifying CHECKSUM!------------------------------

----------------------------------CHECKSUM OK!---------------------------------

root@unraid:~# nvidia-smi
NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running.

root@unraid:~# sudo modprobe nvidia
modprobe: ERROR: could not insert 'nvidia': No such devic

running with a RTX3070. This started happening after upgrade to 7.0.0, any ideas? 

9 minutes ago, BoKKeR said:

running with a RTX3070. This started happening after upgrade to 7.0.0, any ideas? 

If you are willing to, please try to upgrade to Unraid 7.1.0-rc.1, this should hopefully solve the issue.

 

Diagnostics would also help.

Thanks thats good to know, will just wait it out

2 minutes ago, BoKKeR said:

Thanks thats good to know, will just wait it out

Do you have Diagnostics since these would help me to understand what's going on?

I have now multiple reports about that issue and I'm not sure what the cause of the issue is.

 

BTW it would help a lot if you could upgrade and see if the issue is solved on 7.1.0-rc.1, you can also downgrade afterwards if you don't want to stay on 7.1.0-rc.1 <- this build is stable as always.

I am posting diagnostic s before and after update to rc,  the update seems to have not fixed the issue. 

unraid-diagnostics-20250425-1039.zip unraid-diagnostics-20250425-1105.zip

1 minute ago, BoKKeR said:

I am posting diagnostic s before and after update to rc,  the update seems to have not fixed the issue. 

That's why Diagnostics are always important, it's not working because your Nvidia GPU is bound to VFIO.

Apr 25 10:56:32 unraid kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 240
Apr 25 10:56:32 unraid kernel: NVRM: GPU 0000:07:00.0 is already bound to vfio-pci.
Apr 25 10:56:32 unraid kernel: NVRM: The NVIDIA probe routine was not called for 1 device(s).
Apr 25 10:56:32 unraid kernel: NVRM: This can occur when another driver was loaded and 
Apr 25 10:56:32 unraid kernel: NVRM: obtained ownership of the NVIDIA device(s).
Apr 25 10:56:32 unraid kernel: NVRM: Try unloading the conflicting kernel module (and/or
Apr 25 10:56:32 unraid kernel: NVRM: reconfigure your kernel without the conflicting
Apr 25 10:56:32 unraid kernel: NVRM: driver(s)), then try loading the NVIDIA kernel module
Apr 25 10:56:32 unraid kernel: NVRM: again.
Apr 25 10:56:32 unraid kernel: NVRM: No NVIDIA devices probed.
Apr 25 10:56:32 unraid kernel: nvidia-nvlink: Unregistered Nvlink Core, major device number 240

 

And your VFIO config:

Processing 0000:07:00.0 10de:2482
Vendor:Device 10de:2482 found at 0000:07:00.0

IOMMU group members (sans bridges):
/sys/bus/pci/devices/0000:07:00.0/iommu_group/devices/0000:07:00.0
/sys/bus/pci/devices/0000:07:00.0/iommu_group/devices/0000:07:00.1

Binding...

success...

 

And your PCI device:

07:00.0 VGA compatible controller [0300]: NVIDIA Corporation GA104 [GeForce RTX 3070 Ti] [10de:2482] (rev a1)
	Subsystem: ASUSTeK Computer Inc. Device [1043:8813]
	Kernel driver in use: vfio-pci
	Kernel modules: nouveau, nvidia_drm, nvidia
07:00.1 Audio device [0403]: NVIDIA Corporation GA104 High Definition Audio Controller [10de:228b] (rev a1)
	Subsystem: ASUSTeK Computer Inc. Device [1043:8813]
	Kernel driver in use: vfio-pci

 

 

Unbind it from VFIO and it will work again.

Thanks that solved it, works also on 7.0.1 seems like.

Hello,

 

I recently upgraded my unraid from 7.0 to 7.1 latest version. after upgrade my NVIDIA GPU having issues and all docker containers using this NVIDIA GPU did not start and getting runtime error. Then i looked at the driver utility and i saw "ERROR: No driver found!". I also tried to revert back to 7.0 version and now i am still getting same GPU and app install issue in that version as well. I also tried to reinstall NVIDIA Driver utility manually but still not resolved. I also tried all possible solution available online including this tread but no luck. Note: I do see NVIDIA driver showing available in the system devices so ruled out the physical issue with NVIDIA GTX 1660 SUPER card. I am stuck now and all my key docker containers are down due to GPU not being available. 

2 hours ago, oldmunk said:

Then i looked at the driver utility and i saw "ERROR: No driver found!".

Please post your Diagnostics, without them troubleshooting is not possible for hardware devices.

I have an issue with my GTX 1060

The devices have been detected on System Devices, and on lspci

However, the latest and production drivers didn't detect it. When I put the open source driver, I got the error "unraid NVIDIA-SMI has failed because it couldn't communicate with the NVIDIA driver. Make sure that the latest NVIDIA driver is installed and running."

 

Attached are the screenshots, and the diagnostics file,
Thank you

zenithlab1.0-diagnostics-20250505-1810.zip

50 minutes ago, zenaufa said:

I have an issue with my GTX 1060

The devices have been detected on System Devices, and on lspci

This is odd since it says that the driver is in use:

01:00.0 VGA compatible controller [0300]: NVIDIA Corporation GP106 [GeForce GTX 1060 6GB] [10de:1c03] (rev a1)
	Subsystem: ZOTAC International (MCO) Ltd. Device [19da:1438]
	Kernel driver in use: nvidia
	Kernel modules: nvidia_drm, nvidia
01:00.1 Audio device [0403]: NVIDIA Corporation GP106 High Definition Audio Controller [10de:10f1] (rev a1)
	Subsystem: ZOTAC International (MCO) Ltd. Device [19da:1438]

 

Did you make any changes lately to your system (BIOS Update, Add/Remove hardware,...)?
I see that the initialization fails:

May  5 18:09:58 ZenithLab1 kernel: NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x24:0x72:1589)
May  5 18:09:58 ZenithLab1 kernel: NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0
May  5 18:09:59 ZenithLab1 kernel: NVRM: GPU 0000:01:00.0: RmInitAdapter failed! (0x24:0x72:1589)
May  5 18:09:59 ZenithLab1 kernel: NVRM: GPU 0000:01:00.0: rm_init_adapter failed, device minor number 0

 

Please also make sure that you are on the latest BIOS version and please check to enable Above 4G Decoding and Resizable BAR support in your BIOS.

 

I assume the card was working before in this Server or did you just install the card?

This is a new card i got it second hand from the marketplace, I was using P400 before, and it's working smoothly.

 

I have enabled the Above 4G Decoding and Resizeable BAR in my BIOS
I also plugin the GPU into a monitor via HDMI, and can see the computer booting and unraid console there.

Edited May 5, 20251 yr by zenaufa

42 minutes ago, zenaufa said:

This is a new card i got it second hand from the marketplace, I was using P400 before, and it's working smoothly.

May I ask why the switch to a GTX1060, Pascal will be Soon be "legacy" and not be supported by the latest driver version. Something like a Nvidia T400, T600 or even T1000 should be way enough for a few simultaneous transcodes.

 

I just saw that you have a Ryzen 1st gen CPU, I think you have to disable C-States or was it P-States IIRC so that all Nvidia cards work properly.

 

45 minutes ago, zenaufa said:

I also plugin the GPU into a monitor via HDMI, and can see the computer booting and unraid console there.

That doesn't mean that the card works properly, do you have another computer where you could test the 1060 and install the Nvidia driver?

The card want's to initialize but it refuses to initialize, this could be due to a Firmware (BIOS) issue or the card being defective.

I'm having some issues with fan control, starting with unraid 7.0.x I've added 2 Dell A4000 GPU's to my Dell R740xd server. Running 2 cards, the fan control state = 0 seems to work, fans spin up with the GPU core temperature. I've now added 2 more Dell A4000 GPU's to my server, with a total of 4 cards. This seems to break fan control, both automatic and manual. Same issue with unraid 7.1 released yesterday. I've tried multiple drivers, latest: v575.51.02, Open Source Driver: v570.144 and Production Branch: v570.144. It does not make a difference. 

GPU0 and GPU1 seems to have basic fan control, but they are always spinning at around 30%, no matter of temperature. GPU2 and GPU3, the fans never spin at all. This leads to the cards quickly hitting 103c and throttling down to more or less 0 power. 

I've tried to manually control the fans using:

nvidia-settings -c :0 -a [gpu:0]/GPUFanControlState=1 -a [fan:0]/GPUTargetFanSpeed=80 
nvidia-settings -c :0 -a [gpu:1]/GPUFanControlState=1 -a [fan:1]/GPUTargetFanSpeed=80 
nvidia-settings -c :0 -a [gpu:2]/GPUFanControlState=1 -a [fan:2]/GPUTargetFanSpeed=80 
nvidia-settings -c :0 -a [gpu:3]/GPUFanControlState=1 -a [fan:3]/GPUTargetFanSpeed=80 

nvidia accepts the settings, but only gpu0 and GPU1 are set to fan speed 80, gpu 2-3 are still set at 0%. Changing to:

nvidia-settings -c :0 -a "[gpu:0]/GPUFanControlState=0"
nvidia-settings -c :0 -a "[gpu:1]/GPUFanControlState=0"
nvidia-settings -c :0 -a "[gpu:2]/GPUFanControlState=0"
nvidia-settings -c :0 -a "[gpu:3]/GPUFanControlState=0"

 only affects GPU0 and GPU1, but they do not go to 0% when cool, they stay at around 30%

It seems like vBIOS is not in control of the fan curve but the driver/system is always in control of the fan curve.

 

I only use the GPUS for ollama, ollama is working correctly spreading the models across the 4 GPU's

 

Edited May 6, 20251 yr by Output
added open source driver