questions
will this plugin support vgpu instances?

Both a virtulized gpu in say a proxmox vm and/or a say grid k2 example...
I ask because i finally got my hands on a quadro a6000

but the production, lattest and open source driver doesn't seem to detect my card

Example
lspci
04:00.0 VGA compatible controller: NVIDIA Corporation TU102GL [Quadro RTX 6000/8000] (rev a1)

 

curious with 

as the open source driver is getting vgpu support?
https://www.phoronix.com/news/NVIDIA-Open-GPU-Virtualization

Edited October 5, 20241 yr by bmartino1

47 minutes ago, bmartino1 said:

will this plugin support vgpu instances?

I don‘t think so since I assume you want to pass theough a vGPU to Unraid and then use this driver to utilize the vGPU.

 

Please do keep in mind that I can‘t give support for virtualized GPUs or passed through GPUs since there are too many variables involved.

Multiple people who virtualize Unraid on Proxmox or VMWare have reported issue with that since they forgot to set a flag while passing through the GPU.

 

If you pass through a whole card to a VM you don‘t need the plugin.

 

47 minutes ago, bmartino1 said:

as the open source driver is getting vgpu support?

Yeah but I don‘t think that‘s already implemented correct?

However if the OpenSource driver get‘s this feature the driver should support it.

 

Please note that these driver plugins that I provide are just downloading precompiled driver for the Unraid version you are running.

When the driver is installed you can do whatever you like with the driver so to speak, if the driver supports it then yes you should be able to create vGPUs and use it in VMs but this is will be not part of this plugin if additional steps are necessary.

I just upgraded from Unraid 7 beta 2 to Unraid 7 beta 3 and received this warning:

Quote

Nvidia Driver v470.239.06 download failed, please go to the support thread for this plugin and make a post with a screenshot from this error!

I don't know why it's saying to add a screenshot without giving any additional useful information. Anyway, why is it trying to download such an old driver version? The current "production" version is v550.120 and the current installed version is v550.107.02. Is this error legit or am I safe to reboot into the new version?

26 minutes ago, FlexibleToast said:

Is this error legit or am I safe to reboot into the new version?

may reboot and see what happens, looks more like a plugin issue while upgrading.

 

are your plugins "up to date" ?

 

if it fails on reboot, may remove / reinstall the plugin and reboot once more

The Nvidia plugin is up to date. I have the auto update plugin and set it to update every Friday any update that is over 3 days old. I travel for work every week, so I'm not going to attempt the reboot until Friday. I'll report back if I have any issues.

2 hours ago, FlexibleToast said:

Anyway, why is it trying to download such an old driver version?

I suspect a corrupt plugin, can you please post a screenshot from the full plugin page?

 

To fix a faulty plugin, uninstall the plugin, reboot, install the plugin again, reboot again (you could also disable and enable Docker once but I would recommend rebooting) and everything should work as before.

I had this running great, following these instructions.  I was trying to upgrade my RAM and ran into a cascade of problems, one of which is now any attempt to reinstall the NVIDIA driver causes my entire system to freeze, requiring a hard reset.  I had to remove & replace the GPU to get to the RAM slots.  UNRAID sees the device, the install works fine, but when I go to look at the NVIDIA driver information under settings, the entire system freezes within 20 seconds, and I never get any the NVIDIA driver page to come up.

 

Any help would be appreciated.

 

PLEASE NOTE:  All equipment in my device is the same as it was before, when the GPU was humming along.  The only difference is I had to replace my parity drive.  Parity has been re-written, and parity check has run a few times.  I had a massive Docker issue at the same time I was wrestling with the GPU, which required me to reinstall the docker.img file

hwProf.xml tower-diagnostics-20241014-1006.zip

16 hours ago, ich777 said:

I suspect a corrupt plugin, can you please post a screenshot from the full plugin page?

 

To fix a faulty plugin, uninstall the plugin, reboot, install the plugin again, reboot again (you could also disable and enable Docker once but I would recommend rebooting) and everything should work as before.

To be clear, the running state it is in right now works. I followed the upgrade messages that said not to reboot until I receive a notification saying it was okay to. I can get the screenshot of the plugin page, but like I said I travel for work and I don't want to reboot a working system while I'm away. I can do that part of the troubleshooting Friday.

8 hours ago, FlexibleToast said:

To be clear, the running state it is in right now works.

The driver package and the plugin are two separate things, so to speak, I suspect the plugin package on your system is broken but not the driver.

So to speak if you see error messages on the plugin page you have to remove the plugin, reboot, install the plugin and preferably reboot again.

 

8 hours ago, FlexibleToast said:

but like I said I travel for work and I don't want to reboot a working system while I'm away. I can do that part of the troubleshooting Friday.

No worries, take your time.

8 hours ago, JokerzWild said:

NVIDIA driver

Please try to install the Nvidia driver, then pull the Diagnostics (which might can take a bit longer than usual) and post them here.

 

Are you sure that no BIOS settings changed and so on?

You can also try to re-seat the card in the slot and check if the external power cables are plugged in correctly.

Hi,

 

Im currently having an issue with my nvidia card with the system crashing randomly every couple of hours, Ive removed the plugin as per recommendation from another mod however i still need my gpu for transcoding etc, Ive managed to get everything working however im not able to get around the crashing.

Please see attached diagnostics as well as syslog at the time of the crash

syslogs.txt imteounraid-diagnostics-20241013-1504.zip

52 minutes ago, IVIazor said:

Please see attached diagnostics as well as syslog at the time of the crash

First of all please remove those two lines from your go file since they aren't needed:

modprobe i915
chmod -R 777 /dev/dri

 

 

From what I can see in your logs where the crash happened you have a XID 119 error:

Oct 13 21:37:45	192.168.0.100	ImTeoUnraid	kern	warning	kernel	NVRM: GPU at PCI:0000:01:00: GPU-e895e0b5-fa25-4886-be12-f2baf798032f
Oct 13 21:37:45	192.168.0.100	ImTeoUnraid	kern	warning	kernel	NVRM: Xid (PCI:0000:01:00): 119, pid=1208, name=nv_open_q, Timeout after 6s of waiting for RPC response from GPU0 GSP! Expected function 103 (GSP_RM_ALLOC) (0x80 0x38).
Oct 13 21:37:45	192.168.0.100	ImTeoUnraid	kern	warning	kernel	NVRM: GPU0 GSP RPC buffer contains function 103 (GSP_RM_ALLOC) and data 0x0000000000000080 0x0000000000000038.

You can read more about and what maybe causes that here.

 

However this XID error is pretty much useless in terms of what could be the cause of the issue.

 

I would recommend doing the following:

1. Upgrade to Unraid 6.12.13
2. Upgrade your BIOS
3. Make sure in the BIOS Above 4G Decoding is enabled
4. Make sure in the BIOS Resizable BAR is enabled

 

What Power Supply are you using?

Are you sure it is still up to the task?

May I ask for what do you use the dGPU?

Was the GPU working before or did you just put it in the system?

13 hours ago, ich777 said:

Please try to install the Nvidia driver, then pull the Diagnostics (which might can take a bit longer than usual) and post them here.

 

Are you sure that no BIOS settings changed and so on?

You can also try to re-seat the card in the slot and check if the external power cables are plugged in correctly.

Thanks for the reply @ich777 !

 

I will have to find a time window to reinstall the Nvidia driver, i.e. a moment where I won't get yelled at for bringing down Plex, lol. 

 

Here's my parade of problems, leading up to this:

 

1) Attempted to upgrade RAM from 24 GB to 64 GB.  Had to remove GPU to get access to half of the clips.

2) Broke flimsy antenna, the one connected to my WiFi card (NBD, not planning on using it).

3) Broke off the plastic tab off my parity Hard Disk, taking some connection needles with it.  Had to replace the hard drive and SATA cable on that one.

4) RAM upgrade prevented the system from booting.  No POST errors, not loading to BIOS, no video output, nothing.  Had to roll back to my old RAM.

5) Rebuilt Parity

6) Rebuilt Docker stack.

7)  After crashing UNRAID three times from NVIDIA driver installs, gave up.

\8) I did realize the power cables to the GPU were not the way they were supposed to be, so I switched them back (same power plugs though) - still crashed after driver install. 

9) I am getting video output from GPU.  Physical connection (DVI) works fine.

10) Previous attempts to run diag with the Nvidia driver installed causes unraid to freeze (froze on nvidia-smi), so I'm a bit hesitant to try this again, though I will once I have a good window.  Previous attempts included having the GPU with reversed power plugs, so that could be part of the problem.

11) BIOS settings having changed since I enabled virtualization about a month ago.

 

I will try the diag dump with the driver installed.  Hopefully switching the power cables back resolve that issue.

Edited October 15, 20241 yr by JokerzWild

12 hours ago, JokerzWild said:

9) I am getting video output from GPU.  Physical connection (DVI) works fine.

Are you also sure that the GPU is working properly, just because you get a output without the driver installed doesn't mean that the GPU is working.

If possible try to put the GPU in another system, install the drivers and put some 3D load on it like FurMark for an hour or so to check if the card is in fact working.

 

12 hours ago, JokerzWild said:

7)  After crashing UNRAID three times from NVIDIA driver installs, gave up.

Just to make your life easier, Stop the Array, go to the CA App and then install the driver plugin since this will ensure that you don't have to run a parity check after the reboot and even if a parity check is starting after the crash it is safe to abort it since the Array wasn't started. Hope that makes sense to you.

You can also disable Autostart from the Array in the Disk Settings since this will safe you a lot of time and headache.

6 hours ago, ich777 said:

Are you also sure that the GPU is working properly, just because you get a output without the driver installed doesn't mean that the GPU is working.

If possible try to put the GPU in another system, install the drivers and put some 3D load on it like FurMark for an hour or so to check if the card is in fact working.

 

 

Thanks again @ich777!

 

GPU for sure isn't working properly.  My comment about the video output was more to the effect of "it's not 100% fried" than the GPU being OK.  It was OK before I started messing with it ( sigh ).  Sadly, I don't have another system to test it on as my old NAS platform died.

 

Thanks for the reminder on the Array management - I did start halting the array about half-way through my repeated tests.  I did totally forget about disabling Autostart though...Thanks!

 

I will likely get a chance today to do some more troubleshooting.  Thanks again!

I had a chance to bring the array down and try the nvidia install again, which worked fine.  Like last time the diagnostics froze on 'nvidia-smi --query 2' command.  I ran it manually within the console and it also hung up.  Two CPU cores were pegged at 100% during both of these.  

 

And because I like poking my tongue on a painful tooth, I also tried going to Settings->Nvidia, which froze UNRAID again.  I hard reset it, removed the nvidia driver, and restarted the array without issue.  I'm beginning to think that I damaged the GPU during my work on the chassis last week.

 

I tried going to the root of the flash to find the temporary diagnostic folder the diag operation makes, but it got deleted with the reboot.  Other diags still exist on boot/logs.

 

 

Hi 

 

Thank you for making this driver possible. 

 

I recently purchased a 3090FE and it was pretty much plug and play with the help of this driver. I have enabled persistence mode using a user script (First array start) and the GPU was sitting GPU IDLE (P8) at around 11w. There didnt seem to be an issue with the iGPU that i am using for plex transcodes / immich transcodes. 

 

2 days on, i noticed it is now sitting idle in throttled SW POWER CAP ACTIVE, and at a higher wattage 24w. memory is sitting at 405 Mhz constant,  Occasionally, it switches to  GPU IDLE, but then back again to SW POWER CAP.  

 

I tried disabling dockers, restarting the server, but this hasnt helped.  Any Ideas?

 

nvidia-smi -q

Edited October 17, 20241 yr by kenzo86

1 hour ago, kenzo86 said:

2 days on, i noticed it is now sitting idle in throttled SW POWER CAP ACTIVE, and at a higher wattage 24w. memory is sitting at 405 Mhz constant,  Occasionally, it switches to  GPU IDLE, but then back again to SW POWER CAP.  

Without Diagnostics it is really hard to troubleshoot, please post your Diagnostics because otherwise I can't say anything without them.

 

1 hour ago, kenzo86 said:

I have enabled persistence mode using a user script (First array start) and the GPU was sitting idle (P8) at around 11w.

How did you do that? Do you have the script or a link how you did that?

 

Usually it is enough to put the line:

nvidia-persistenced

at the end in the go file (/boot/config/go).

 

No complicated scripts or anything needed nowadays.

 

May I ask for what do you use the GPU?

On 10/17/2024 at 2:30 PM, ich777 said:

Without Diagnostics it is really hard to troubleshoot, please post your Diagnostics because otherwise I can't say anything without them.

 

How did you do that? Do you have the script or a link how you did that?

 

Usually it is enough to put the line:

nvidia-persistenced

at the end in the go file (/boot/config/go).

 

No complicated scripts or anything needed nowadays.

 

May I ask for what do you use the GPU?

 

Thanks for replying.

 

1) I have attached diagnostics.

 

2) User scripts in settings --> create a script which runs the 'nvidia-persistenced' line on array startup.

 

''#!/bin/bash
nvidia-persistenced''

 

3) I use the GPU for Invoke-AI and hopefully for steam-headless later on.

 

The throttled sw power cap status at idle resolves after a server reboot (gpu idle). I've spoken to another user (Pri) on the official discord who has had the same issue unresolved for a while.

 

At 100% load, the 3090 is in P0 (throttled: sw power cap) - see attached screenshot of GPU statistics. I have also attached a screenshot of the GPU statistics at idle with sw power cap and gigher power draw (should be 14w idle)

 

The gpu is connected via a type 5 cable (12pin to 8+8) to psu. I did not change anything in bios after installing the dGPU.

tower-diagnostics-20241017-1442.zip

Edited October 18, 20241 yr by kenzo86

On 10/15/2024 at 3:19 PM, JokerzWild said:

Are you sure that no BIOS settings changed and so on?

You can also try to re-seat the card in the slot and check if the external power cables are plugged in correctly.

@ich777

 

I really appreciate all the help you provided...not to mention the integration between UNRAID & NVIDIA!

 

I did get a chance to re-seat the card in the slot.  I double checked the power connections and gave them an extra push to make sure they were in tightly.  Slightly off-topic but I fixed the antenna for the onboard wireless NIC as well.

 

After re-seating the card, I was able to make it visible within the settings->nvidia tab, without crashing the server.

 

Again, thank you for your help and the integration!

On 10/17/2024 at 3:49 PM, kenzo86 said:

Thanks for replying.

So the power doesn't go down after a few days, correct?

 

I really can't help with that since this could be also be caused by a firmware but.

You could try to kill nvidia-persistenced and run it again, maybe that will help.

On 10/15/2024 at 12:29 AM, ich777 said:

The driver package and the plugin are two separate things, so to speak, I suspect the plugin package on your system is broken but not the driver.

So to speak if you see error messages on the plugin page you have to remove the plugin, reboot, install the plugin and preferably reboot again.

 

No worries, take your time.

I forgot to report back on Friday. I did the recommended uninstall, reboot, reinstall and now everything appears to work as expected. Thanks for the help.

Hi,

last night i noticed the fans on my GTX 1070 running (while usually it's not the case).

Today, looking at the GPU stats in the dashboard i noticed this:



Beside having fan spinning, and some load on memory, the power state says "P5" and the the card being throttling.
As you can see, nothing is using it.

If i run nvtop everything goes back as it should be:



BUT, still says GPU is throttling.

This is what nvtop shows:


As i close nvtop, the card goes back to the previous state (as in 1st image).

I tried to uninstall drivers, reboot, reinstall drivers, reboot - but nothing changes.
I'm quite puzzled, everything was perfect until last night.

 

All images (and diagnostics) taken with docker and VSs disabled.

I attach my diagnostics to the post.incubus-diagnostics-20241022-1513.zip
Any help would be greatly appreciated.

thanks.

 

Edited October 22, 20241 yr by dhstsw

23 minutes ago, dhstsw said:

I tried to uninstall drivers, reboot, reinstall drivers, reboot - but nothing changes.

I assume this card was working before without any issues correct?

 

Please issue:

nvidia-persistenced

from the command line, wait a few minutes and see if it drops down to P8

 

Have you yet tried installing a driver lower than version 560?

Quote

I assume this card was working before without any issues correct?

Yes, perfectly.

 

Quote

Please issue:

nvidia-persistenced
from the command line, wait a few minutes and see if it drops down to P8

With the command everything goes as it should be.
How can i have it issued (with a script i guess) but, also, why this now? :-/ Any idea?

 

Quote

Have you yet tried a driver lower than version 560 to install?

Not yet.