Hi Guys,

 

quick question, can someone please look at this log file and give me some advice, does this look like a configuration error or a dead GPU? (GPU is a 3060, but was used for mining for about 3-4 years)

 

When the drives crash (I think that is what is happening) 1 or 2 things happen

 

1) 

The system is unable to be accessed via GUI, this can be fixed by running the command:

/etc/rc.d/rc.php-fpm reload

 

2)

The system hangs, well its still works, but 100% CPU usage and the GUI takes a good 5-10 minutes to look until eventually the system completely hangs, unable to use console or login via SSH, only way to fix this is to hard power off the machine and power on.

 

I have attached the syslog log file, if someone could give me some suggestions that would be great!

 

Cheers,

Az

syslog.log

diagnostics-20250108-1001.zip

Edited January 7, 20251 yr by Azrelix

1 hour ago, Azrelix said:

quick question, can someone please look at this log file and give me some advice, does this look like a configuration error or a dead GPU? (GPU is a 3060, but was used for mining for about 3-4 years)

Please post your full Diagnostics.

 

You can also pull Diagnostics by issuing:

diagnostics

from a SSH session, you'll then find them in /boot/logs

19 minutes ago, ich777 said:

Please post your full Diagnostics.

 

You can also pull Diagnostics by issuing:

diagnostics

from a SSH session, you'll then find them in /boot/logs

Apologies I have added this to my original post!

7 hours ago, Azrelix said:

Apologies I have added this to my original post!

Sorry but I don't see that in the syslog from your Diagnostics.

 

Does this only happen when you put some load on the GPU?

To be completely honest, I've never such an issue...

 

It might be the case that the GPU is dead however if it is possible for you to test the GPU in a Windows machine with the drivers installed and some load on it (Furmark for an hour) that would be optimal.

1 hour ago, ich777 said:

Sorry but I don't see that in the syslog from your Diagnostics.

 

Does this only happen when you put some load on the GPU?

To be completely honest, I've never such an issue...

 

It might be the case that the GPU is dead however if it is possible for you to test the GPU in a Windows machine with the drivers installed and some load on it (Furmark for an hour) that would be optimal.

see what sorry? the syslog would have potentially been deleted as I had to force reset, here is the full syslog that I have, I setup syslog server for a copy of this.

 

The server auto transcodes files, so it may be whilst JF or Tdarr requests something but I can run Tdarr for 1-2 days sometimes without issues and then it crashes, its very strange.

 

I have attached a copy of the syslog that was exported.

 

I will try the GPU in a windows machine as well to see how that goes.

syslog-192.168.2.3.log

28 minutes ago, Azrelix said:

Tdarr

There where some issues in the past with Tdarr which crashed the server occasional but I assumed those where resolved.

Maybe it's an issue with Tdarr but I really can't tell for sure.

 

What I find a bit strange that this seems to be caused by nvidia-smi, maybe Tdarr is calling nvidia-smi or do you maybe had the Dashboard somewhere open since this also calls nvidia-smi frequently.

1 minute ago, ich777 said:

There where some issues in the past with Tdarr which crashed the server occasional but I assumed those where resolved.

Maybe it's an issue with Tdarr but I really can't tell for sure.

 

What I find a bit strange that this seems to be caused by nvidia-smi, maybe Tdarr is calling nvidia-smi or do you maybe had the Dashboard somewhere open since this also calls nvidia-smi frequently.

I nearly always have the unraid dashboard open in chrome somewhere, thanks for the help though, I have stopped tdarr for now and ill close the dashboard and will do some stress tests on the GPU as well!

I was going mad trying to fix it, glad it wasn't just me that was stumped! haha

 

On 1/7/2025 at 1:52 AM, ich777 said:

Can you maybe share that list so that I can also take a look at that?

 

I had closed the webpage with the list, finally found it.  https://hub.docker.com/r/codeproject/ai-server/tags

This is essentially what created the whole problem to start with. I was trying to get the latest version of code project ai so that it would have the latest models included. Best I can tell, I have the latest version now that utilizes gpu, but it still doesnt have the models i was looking for. I know that part is unrelated to unraid/nvidia driver.

So prior to all the upgrades, the GPU stats window on the unraid dashboard used to show an icon for plex and code project ai. This gave me a very quick visual to verify that the dockers were using the gpu. The plex icon still shows up under "active apps" but the code project ai is not showing up. I do get the python3.8 under processes when i look at nvidia-smi, but no icon on the unraid dashboard. Any ideas?

 

 

On 1/7/2025 at 1:52 AM, ich777 said:

 

 

1 minute ago, Itsaulgoodman said:

I do get the python3.8 under processes when i look at nvidia-smi, but no icon on the unraid dashboard. Any ideas?

Sorry but the icons are not part of the Nvidia Driver, you have to report that in the GPU Statistics plugin support thread.

 

Does the Code Project AI container now work for you?

2 hours ago, ich777 said:

Sorry but the icons are not part of the Nvidia Driver, you have to report that in the GPU Statistics plugin support thread.

 

Does the Code Project AI container now work for you?

It seems to be working correctly, just missing the icon now. Thank you for your help!!

9 minutes ago, Itsaulgoodman said:

It seems to be working correctly, just missing the icon now. Thank you for your help!!

What was the fix? Or does this message always appear?

Pardon my ignorance— I have never upgraded Unraid since I began using Nvidia card in server.

 

The Unraid upgrade instructions for V7 mention making sure this plugin is updated BUT if I look at it right this plugin already supports latest Unraid kernel v6.6.68 which is what is used in Unraid 7?

 

Anything else I need to be thinking of before I upgrade server in relation to Nvidia plugin?

 

 

5 hours ago, blaine07 said:

The Unraid upgrade instructions for V7 mention making sure this plugin is updated BUT if I look at it right this plugin already supports latest Unraid kernel v6.6.68 which is what is used in Unraid 7?

Just start the update process, right at the end it will say that 3rd party driver downloads are starting and you should watch your Unraid notifications until it is safe to reboot and only reboot after that message is displayed (you can watch the whole process at the link below - about 1 minute since it shows the whole process) :

https://github.com/ich777/unraid-plugin_update_helper?tab=readme-ov-file#update-notifications

Hello All, after reading through this guide and following the steps, plex still doesnt show a GPU in the settings. I am using a GTX750TI and it shows to be compatible, but i am wondering if it matters what driver I select for this GPU?

 

Any pointers would be greatly appreciated!

 

EDIT: started checking older driver versions. after installing v470.256.02, it instructs you to reboot. worked afterwards. either the reboot or the driver change solved my issue. for others facing the same issue.

Edited January 10, 20251 yr by Aerodb
update to solved

7 hours ago, Aerodb said:

GTX750TI

This graphics card is not capable of encoding/decoding most of the common formats nowadays because it is pretty old by todays standards.

 

You can take a look here what the card is actually capable of:

https://developer.nvidia.com/video-encode-and-decode-gpu-support-matrix-new

 

I would recommend that you look into something new or even used because you can get something like a Nvidia T400 for pretty cheap nowadays.

8 minutes ago, AT123454321 said:

Where are you lacated in the world?

Do you have Diagnostics?

Romania, I just rebooted. Fingers crossed. The driver is for a Tesla P40 used for Jellyfin decoding.

22 minutes ago, AT123454321 said:

Romania, I just rebooted. Fingers crossed. The driver is for a Tesla P40 used for Jellyfin decoding.

Keep in mind that your server will try to download the driver now on boot and if it's like for me over here in Europe the downloads from GitHub are really slow currently about 1MB/s this can take some time since the driver package is about 250MB.

 

If it is not working, just do that:

1. Uninstall the Nvidia Driver plugin (please also check if it's in the Plugin Error tab within the Plugin section in Unraid and remove it)
2. Reboot
3. Install the Nvidia Driver plugin
4. Reboot

 

After that it should work.

Drives are not loading afther upgrade from 6.12.14 to 7.0.0

 

 

Iahve unstalled and installed the drivers. Rebooting between avery stage. 

 

nest2-diagnostics-20250112-0515.zip

11 minutes ago, mikeofoslo said:

Drives are not loading afther upgrade from 6.12.14 to 7.0.0

 

 

Iahve unstalled and installed the drivers. Rebooting between avery stage. 

 

nest2-diagnostics-20250112-0515.zip 155.85 kB · 0 downloads

The Nvidia drivers seem to install fine. 

On the otherhand, it appears, you dont even have a Nvidia GPU in your system. Its not detected at all

Edited January 12, 20251 yr by Mainfrezzer
rephasing and typo

1 hour ago, mikeofoslo said:

Iahve unstalled and installed the drivers. Rebooting between avery stage. 

As @Mainfrezzer said:

Jan 12 05:04:57 NEST2 kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 241
Jan 12 05:04:57 NEST2 kernel: NVRM: No NVIDIA GPU found.

 

5 hours ago, Mainfrezzer said:

The Nvidia drivers seem to install fine. 

On the otherhand, it appears, you dont even have a Nvidia GPU in your system. Its not detected at all

The Nvidia GPU is still inside the server. It was working fine under 6.12.14. 

4 hours ago, ich777 said:

As @Mainfrezzer said:

Jan 12 05:04:57 NEST2 kernel: nvidia-nvlink: Nvlink Core is being initialized, major device number 241
Jan 12 05:04:57 NEST2 kernel: NVRM: No NVIDIA GPU found.

 

The Nvidia GPU is still inside the server. It was working fine under 6.12.14.  This is my backupserver and was the first to go for an upgrade. The main server is practically identical when hardware is concerned. 

 

I think I'll have to ty to roll back for now. 

54 minutes ago, mikeofoslo said:

The Nvidia GPU is still inside the server. It was working fine under 6.12.14. 

51 minutes ago, mikeofoslo said:

I think I'll have to ty to roll back for now. 

Please think again about it...

 

The card is not even recognized by your system, this has nothing to do with the Upgrade or Unraid itself.

The card should at least show up in lspci which it doesn't do...

Maybe try to shutdown your server entirely, pull the cord from the wall, wait for a few minutes and turn it back on after that, maybe the card just hangs or something like that.