46 minutes ago, alturismo said:

 

may delete the vfio bind, looks like this was from another device you had a vfio bind active (possible another GPU from earlier times ?) for VM passthrough ?

 

vfio bind should only be activated for devices which you want to passthrough a VM ...

I did have an AMD gpu i did some testing with, but never anything more then plug it in really

How would i delete the bind?

12 minutes ago, Tithonius said:

How would i delete the bind?

remove the entry in here (open the file and clean it)

 

 

and may take another look here if you have a manual syslinux entry ...

 

 

if there is something here ...

Okay, did that, but would that really be the reason for the 500 internal server error?

 

Also, just to clarify, I have uninstalled the app, rebooted, reinstalled the app, and rebooted again and i haven't had the error since, so maybe it was just a weird thing. but i was just curious if there was anything in the diagnostic that was alarming.

Edited March 8, 20233 yr by Tithonius

11 minutes ago, Tithonius said:

but i was just curious if there was anything in the diagnostic that was alarming.

I don't see anything in the last Diagnostics what's concerning, the card is properly recognized and should be ready for use.

If this happens again please issue:

diagnostics

from the terminal and post the Diagnostics afterwards here (the Diagnostics will be saved in the logs folder on your USB Boot Device).

 

I could be also be the case that something else was crashing your server but it seems rather strange to me that you where still able to connect through SSH, usually everything crashes and the server needs a hard reboot.

The next thing could also be that the card dropped from the bus, but these are all assumptions.

Hello,

I'm having trouble having the plugin recognize my GTX960 .

I think there might be a kernel conflict according to the logs, but I can't find out how to get rid of the conflicting kernel. What can I do?

 

I think the existing kernel is vfio, since I'm vm-ing unraid from proxmox.

 

39 minutes ago, alex01763 said:

What can I do?

Please post your Diagnostics.

3 hours ago, ich777 said:

Please post your Diagnostics.

Here are my diagnostics. also, I got this alert too:

 

unraid-001-diagnostics-20230310-1927.zip

5 minutes ago, alex01763 said:

Here are my diagnostics.

From what I know you have to maybe do some configuration on Proxmox if you are virtualizing Unraid but I can't help with that, however you have bound your card to VFIO for whatever reason in your syslinux.conf:

BOOT_IMAGE=/bzimage initrd=/bzroot intel_iommu=on iommu=pt vfio-pci.ids=10de:1401,10de:0fba video=efifb:off

Please remove the part:

vfio-pci.ids=10de:1401,10de:0fba video=efifb:off

(I would recommend binding a a PCI device through the System Device menu in Unraid and not through the syslinux.conf)

 

Please note that if you bind it to VFIO the card won't be recognized by the Plugin anyways... If you want to use it with the plugin please unbind the card from VFIO.

 

10 minutes ago, alex01763 said:

also, I got this alert too:

Ignore this for now, don't know why it alerted you about that because from what I see the latest driver was downloaded.

14 minutes ago, ich777 said:

Please remove the part:

 

Thanks, this worked!

I selected driver vision 470.xx and this fixed my issue

THANKS!

On 2/24/2023 at 10:41 AM, Doogs said:

Hey ich - Following up with you.  I've removed the card and it (unraid) been stable.  The nVidia M60 is powered by an 8-pin CPU cable and I don't think the stock 8-pin CPU cable will suffice since the official adapter takes 2 PCI-E cables and 'must carry 200+W....'.  

TL;DR - I don't think it's a plugin issue.  I'll follow up again once I get the correct adapter.  I can only surmise the card was working previously with the stock 8-pin CPU cable because my previous build was 7 drives and my latest build has double that.  As always, thank you for your continued support and prompt responses!

 

Cheers!

Hey ich,

 

Installed the card with the correct power cable and I'm still having issues.  The last thing I can do is try another card - I have access to 4 more of these.  Any suggestions on what else I could do?

 

EDIT - I'll pop the card back in, wait for the issue to happen and grab diag

Edited March 13, 20233 yr by Doogs

So my problem has progressed now. As a recap, I have a 1050ti installed with the latest updates to everything in my server. All drivers, os updates and such. I have a problem where randomly the server will work fine for days at a time, then just crash. I used to just have the gui crash, but now am getting entire server lockups. I enabled syslog logging earlier, but when the problem happened again I didn't see anything in the syslog that was concerning, and the helpful peeps over on the unRAID discord didn't see anything either. The diagnostic doesn't look any different then it did last time... I can't get a diagnostic while it's crashing anymore due to the server being hard locked. When I reboot and do diagnostics everything goes back to being normal. This has only been happening ever since I added the GPU to the system so it has to be related to this somehow. Is there another way for me to get different logs or something that will help me narrow down the issue? 

The GPU was pulled from a working gaming rig, and has never been abused, and doesn't seem to be really failing. It's transcodes all look great, and I'm having no other issues with it while it is working.

 

The card is also only showing as being on a gen 3 by 2x connection in GPU statistics even though it is installed in a gen 3 16x slot.. could that be pointing to a potential problem? The only other card in the system in my lsi hba, so it can't be a lanes issue with the CPU, there should be plenty available. I may try reseating or moving the GPU to a different slot, but I think that may be the only slot on that board it will fit in. 

 

I guess if it passes the memtest that is running atm then I'll try reseating the GPU to see if that's a problem. I already replaced all the memory in the system, and upgraded to 64GB so it's not a memory space issue, and I replaced all the ram after these problems started occuring as a troubleshooting step. I am running a memtest just to be sure, but early signs are showing that the memory is fine. 

 

I'm kinda at my wits end, and really am having trouble as I have never had problems with this machine until I added the gpu. It's super discouraging...

 

When I get home from work hopefully the memtest will be done and I'll boot the system again and do another diagnostic as well as post my syslog just for y'all to look over. I'm also going to make sure that some idle power stuff in bios isn't the problem as I see that can be an issue for others here. 

 

idk tbh I'm kinda just guessing here...

Edited March 13, 20233 yr by Tithonius

1 hour ago, Tithonius said:

So my problem has progressed now. As a recap, I have a 1050ti installed with the latest updates to everything in my server. All drivers, os updates and such. I have a problem where randomly the server will work fine for days at a time, then just crash. I used to just have the gui crash, but now am getting entire server lockups. I enabled syslog logging earlier, but when the problem happened again I didn't see anything in the syslog that was concerning, and the helpful peeps over on the unRAID discord didn't see anything either. The diagnostic doesn't look any different then it did last time... I can't get a diagnostic while it's crashing anymore due to the server being hard locked. When I reboot and do diagnostics everything goes back to being normal. This has only been happening ever since I added the GPU to the system so it has to be related to this somehow. Is there another way for me to get different logs or something that will help me narrow down the issue? 

Does your card disappear from nvidia-settings as well?  My card does and if I leave it that way, my server will entirely lock up as well.  Trying to see if there is a pattern here.

7 hours ago, Doogs said:

The last thing I can do is try another card - I have access to 4 more of these.  Any suggestions on what else I could do?

Not really, I can only think of a hardware compatibility issue... Other than that if you made sure that you are on the latest BIOS version, enabled Above 4G Decoding and Resizable BAR support I don't have any more recommendations besides booting in legacy mode (CSM).

You can try to set the PCIe Gen to Gen2 or even Gen3 in the BIOS manually if that makes a difference (that won't affect performance in the case for this cards, at least not noticeable).

3 hours ago, Tithonius said:

I didn't see anything in the syslog that was concerning, and the helpful peeps over on the unRAID discord didn't see anything either. The diagnostic doesn't look any different then it did last time... I can't get a diagnostic while it's crashing anymore due to the server being hard locked. When I reboot and do diagnostics everything goes back to being normal.

This is caused because your server is crashing really hard and even the syslog server won't work anymore when the crash happens, have you yet tried to connect a monitor tho the GPU to see the console output. The only thing you can really do to capture whats happening is to connect a screen and wait for a crash to happen, after that take a picture from the output.

 

Maybe it's related to MACVLAN, do you have Docker containers in br0 and Host Access enabled? Maybe try to switch to IPVLAN in the Docker settings (with Advanced View enabled).

 

3 hours ago, Tithonius said:

The card is also only showing as being on a gen 3 by 2x connection in GPU statistics even though it is installed in a gen 3 16x slot.. could that be pointing to a potential problem?

Usually that should be no issue.

 

3 hours ago, Tithonius said:

I'm kinda at my wits end, and really am having trouble as I have never had problems with this machine until I added the gpu. It's super discouraging...

Yes I know, many people have such issues but mostly because of hardware incompatibility issues or a bad implementation in the BIOS.

 

Maybe try to set the PCIe Gen to a fixed value instead of leaving it at Auto if you have the option for that.

3 hours ago, ich777 said:

This is caused because your server is crashing really hard and even the syslog server won't work anymore when the crash happens, have you yet tried to connect a monitor tho the GPU to see the console output. The only thing you can really do to capture whats happening is to connect a screen and wait for a crash to happen, after that take a picture from the output.

 

 

 

Okay, so i have a screen attached now, i can see the login prompt, i assume that I just login to the non gui on the screen then just leave it? Or do I have to have it show me console output somehow?

7 hours ago, Tithonius said:

Okay, so i have a screen attached now, i can see the login prompt

I completely forgot that you have to execute this command so that the screen doesn't goes blank or better speaking to sleep:

setterm --blank 0

 

7 hours ago, Tithonius said:

i assume that I just login to the non gui on the screen then just leave it?

You don't have to be logged in, just log in once, execute the command and maybe log out again (but you don't have to log out strictly speaking).

11 hours ago, ich777 said:

I completely forgot that you have to execute this command so that the screen doesn't goes blank or better speaking to sleep:

setterm --blank 0

 

You don't have to be logged in, just log in once, execute the command and maybe log out again (but you don't have to log out strictly speaking).

Okay, so last night I set the pcie gen to 3 instead of auto, reseated the GPU in the slot, and reseated all the memory. It passed a 13 hour memory test with 0 errors, and seems to be okay. When I woke up again this morning I saw that it had hard crashed again... The screen went to sleep so when I get home from work today I'll set the screen to not sleep like you said and see if the console can show me what's going on. I assume the command I want to run is the "tail -f /var/log/syslog" command yeah? But that would seem to just show me what the syslog shows, and the syslog didn't have anything last time..

 

Or is there a command that will show me a more verbose output? I wouldn't be opposed to having the most info possible if that's a thing. 

 

What command should I be running to see the console output?

 

I also set the docker network to IPVLAN and restarted docker, but didn't reboot after that change. Should I have rebooted?

 

On the plus side the GPU is showing at a x16 lane now so that's progress I guess... Still crashing though.

Edited March 14, 20233 yr by Tithonius

How possible is it that it's the GPU statistics plugin causing this crash?

That's the only thing that has been new since these crashes have been happening...

 

Also, is it possible that Tdarr is causing the crash? 

Edited March 14, 20233 yr by Tithonius

1 hour ago, Tithonius said:

I assume the command I want to run is the "tail -f /var/log/syslog" command yeah

No, you don't need to run anything, the Kernel Panic is displayed automatically.

 

1 hour ago, Tithonius said:

Or is there a command that will show me a more verbose output? I wouldn't be opposed to having the most info possible if that's a thing. 

No, what do you want to see, you are seeing the entire syslog...

 

1 hour ago, Tithonius said:

What command should I be running to see the console output?

Leave it as it is and it will display the Kernel Panic if one occurs.

 

52 minutes ago, Tithonius said:

How possible is it that it's the GPU statistics plugin causing this crash?

If you are not on the Dashboard very unlikely...

 

52 minutes ago, Tithonius said:

Also, is it possible that Tdarr is causing the crash? 

Could be also the cause of the issue, back in the days Tdarr was notorious for crashing servers, that's why I recommend to use Unmanic.

On 3/2/2023 at 4:08 AM, ich777 said:

Then the crash is most certainly not related to the Nvidia Driver.

 

Have you yet changed MACVLAN to IPVLAN in your Docker settings? MACVLAN is notorious to crash servers with similar Kernel panics.

 

May I ask why? The open source module has no real benefit...

Had another crash again, this time it was when I know for sure there was lots of transcoding happening with TDARR. Perhaps then it does only crash when the GPU is being utilized?

 

Yes I changed it to MACVLAN and noticed no difference and the crash still occurred.

 

No reason, was just interested in it.

 

Is there anything else I can try, can you link me to some general troubleshooting tips? Sorry for late reply, I've been busy.

6 hours ago, LimesKey said:

Is there anything else I can try,

syslog mirror to flash to may get better infos what is happening while crashing

attach a monitor to see on the screen what is happening while crashing

 

would be my approach now ... and common when you read a little about troubleshooting.

 

and may just asked, your PSU is powerful enough ? if yes, forget the question  i just remembered my 2070S liked to crash my unraid Server while Gaming ... but i knew its been "on the edge"  new PSU, then its been gone.

On 3/17/2023 at 12:56 AM, alturismo said:

syslog mirror to flash to may get better infos what is happening while crashing

attach a monitor to see on the screen what is happening while crashing

 

would be my approach now ... and common when you read a little about troubleshooting.

 

and may just asked, your PSU is powerful enough ? if yes, forget the question  i just remembered my 2070S liked to crash my unraid Server while Gaming ... but i knew its been "on the edge"  new PSU, then its been gone.

I have syslog mirror to Unraid not flash, will that work? Actually the last time it crashed I put the GPU on another 500W PSU, so I have 2 x 500W PSUs, 1 is for the CPU and 2 HDDs, the other is for the GPU + another 2 HDDs. In a bit I will get a real server PSU with 1000w+ that would be a lot more efficient. It's a little janky but the GPU barely uses more than 70W while transcoding and overall it should not exceed the 500W of my PSU even if everything was on 1 PSU. Both PSUs are pretty-good quality corsair I believe and they are on a UPS.