39 minutes ago, ich777 said:

Did you yet try to to lower the PCIe Gen to something like Gen3.

Since you have a second gen Ryzen CPU please also disable C-States for your CPU entirely and see if that helps.

However from looking through your syslog I see probably a few errors:

NVRM: kbifInitLtr_GB202: LTR is disabled in the hierarchy

This is an error that I never saw before to be completely honest.

And then you get this error which I've seen a lot recently but in your case it's a bit different and it seems it's related to the CPU but I really can't tell, however what you can do is that you report that on the Nvidia Developer Linux Forums subforums:

Aug 31 05:19:39 Tower kernel: NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from memdescAllocInternal(pMemDesc) @ memdesc.c:1363

Aug 31 05:19:40 Tower kernel: NVRM: kbifInitLtr_GB202: LTR is disabled in the hierarchy

Aug 31 05:19:40 Tower kernel: eth0: renamed from veth6aa9d85

Aug 31 05:19:40 Tower kernel: br-6deb7f169b14: port 20(veth7d89dc4) entered blocking state

Aug 31 05:19:40 Tower kernel: br-6deb7f169b14: port 20(veth7d89dc4) entered forwarding state

Aug 31 05:19:41 Tower kernel: NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from memdescAllocInternal(pMemDesc) @ memdesc.c:1363

Aug 31 05:19:42 Tower kernel: NVRM: kbifInitLtr_GB202: LTR is disabled in the hierarchy

Aug 31 05:19:42 Tower kernel: NVRM: nvCheckOkFailedNoLog: Check failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from memdescAllocInternal(pMemDesc) @ memdesc.c:1363

Aug 31 05:19:42 Tower kernel: NVRM: nvAssertOkFailedNoLog: Assertion failed: Out of memory [NV_ERR_NO_MEMORY] (0x00000051) returned from nvStatus @ message_queue_cpu.c:242

Aug 31 05:19:42 Tower kernel: NVRM: _kgspInitRpcInfrastructure: GspMsgQueueInit failed

Aug 31 05:19:42 Tower kernel: NVRM: kgspConstructEngine_IMPL: init RPC infrastructure failed

Aug 31 05:19:42 Tower kernel: NVRM: osInitNvMapping: *** Cannot attach gpu

Aug 31 05:19:42 Tower kernel: NVRM: RmInitAdapter: osInitNvMapping failed, bailing out of RmInitAdapter

Aug 31 05:19:42 Tower kernel: NVRM: GPU 0000:09:00.0: RmInitAdapter failed! (0x22:0x51:763)

Aug 31 05:19:42 Tower kernel: NVRM: GPU 0000:09:00.0: rm_init_adapter failed, device minor number 0

Aug 31 05:19:48 Tower kernel: NVRM: nvAssertFailedNoLog: Assertion failed: rmapiLockIsOwner() && rmGpuLockIsOwner() @ conf_compute_api.c:78

This is unrelated to the Nvidia Driver but you have lots of XFC Checksum errors:

Aug 29 06:12:57 Tower kernel: XFS (md1p1): Metadata CRC error detected at xfs_dir3_data_read_verify+0x7e/0x100, xfs_dir3_data block 0xbdd8fbe8

Aug 29 06:12:57 Tower kernel: XFS (md1p1): Unmount and run xfs_repair

Aug 29 06:12:57 Tower kernel: XFS (md1p1): First 128 bytes of corrupted metadata buffer:

Aug 29 06:12:57 Tower kernel: 00000000: 58 44 44 33 fa 13 dc ae 00 00 00 00 bd d8 fb e8 XDD3............

Aug 29 06:12:57 Tower kernel: 00000010: 00 00 00 01 00 04 b3 99 d7 b2 3a 12 2b 94 46 cb ..........:.+.F.

Aug 29 06:12:57 Tower kernel: 00000020: a0 27 c6 a5 40 66 31 b1 00 00 00 00 cf 30 95 6e .'[email protected]

Aug 29 06:12:57 Tower kernel: 00000030: 07 80 08 80 00 00 00 00 00 00 00 00 ff 7f 00 00 ................

Aug 29 06:12:57 Tower kernel: 00000040: 00 00 00 00 cf 30 96 15 16 53 6c 75 6d 20 4c 6f .....0...Slum Lo

Aug 29 06:12:57 Tower kernel: 00000050: 72 64 20 28 48 75 6d 61 6e 29 2e 63 68 75 6d 01 rd (Human).chum.

Aug 29 06:12:57 Tower kernel: 00000060: 00 00 00 00 00 00 00 40 00 00 00 00 cf 30 96 16 [email protected]..

Aug 29 06:12:57 Tower kernel: 00000070: 14 53 6c 75 6d 20 4c 6f 72 64 20 28 4f 72 6b 29 .Slum Lord (Ork)

I would recommend that you report that alongside with your Diagnostics in the General Support subforums here on the Forum.

I'll take it one step at a time then and start by disabling C-States and see how that improves things. I'm not sure what you mean by lower the PCIe Gen to Gen3. I'll double check, but I think that's the max speed of the Mobo's slots. Yeah, the x16 slots appear to be 3.0 slots. Is there a setting in Unraid I'm unfamiliar with?

14 hours ago, mrguymiah said:

I'm not sure what you mean by lower the PCIe Gen to Gen3. I'll double check, but I think that's the max speed of the Mobo's slots. Yeah, the x16 slots appear to be 3.0 slots. Is there a setting in Unraid I'm unfamiliar with?

That's a setting in your BIOS usually but only available when you have a Motherboard that supports newer PCIe Generations than 3.0

I have a gtx 650 that can't be detected by the plugin. How would I go about installing the legacy drivers? I have the correct one downloaded, but have never attempted this before. I'm struggling to find any info on it at the moment.

1 hour ago, mrkrabs said:

I have a gtx 650 that can't be detected by the plugin. How would I go about installing the legacy drivers? I have the correct one downloaded, but have never attempted this before. I'm struggling to find any info on it at the moment.

afaik the gtx650 is outdated and not supported anymore.

also, there is nothing really you can do with this card anyhow ? no nvenc, no ...

2 hours ago, mrkrabs said:

How would I go about installing the legacy drivers?

In addition to @alturismo‘s post:

26 minutes ago, alturismo said:

afaik the gtx650 is outdated and not supported anymore.

also, there is nothing really you can do with this card anyhow ? no nvenc, no ...

tragic, was hoping to get some kind of use out of it after finding it in an old parts bin. Guess it goes on a shelf as a museum piece. Thanks for the response!

3 minutes ago, ich777 said:

In addition to @alturismo‘s post:

All good, thank you!

@ich777 - Thanks for your awesome work on this plugin. Quick question - I am using P2000 for Plex and Immich dockers. What branch should I remain on - it is very confusing and after reading a few comments above I am more confused by legacy and proprietary drivers.

Cheers!

3 hours ago, Unraid-arr said:

@ich777 - Thanks for your awesome work on this plugin. Quick question - I am using P2000 for Plex and Immich dockers. What branch should I remain on - it is very confusing and after reading a few comments above I am more confused by legacy and proprietary drivers.

For now on the latest, we will see what the future or better speaking Nvidia is bringing to the table.

On 8/31/2025 at 7:59 AM, sausuke said:

Anyone how to solve this? I'm on 7.1.4 and using Open Source Driver: v575.57.08 with nvidia 5070ti. When I stopped using VM with GPU passthrough, sometimes nvidia driver cannot detect the 5070ti anymore? I need to restart the server again for it to appear?

Here's the log

Finally solved this problem, it's on this thread https://forums.developer.nvidia.com/t/575-64-nvrm-out-of-memory-error-causes-dgpu-to-not-be-usable-after-some-time/336877/11

I upgraded to the latest beta 7.2.0 beta and the same problem exist with latest driver v580.76.05. Used chat gpt to convert it to userscript. That Out of memory [NV_ERR_NO_MEMORY] (0x00000051) occurs on when using VM and playing a game or benchmark related like heaven benchmark.

How to create that error:
Open VM>Play a game or heaven benchmark>Shutdown VM> and when I go to Nvidia driver in settings keep refreshing and you'll see that Check failed: Out of memory [NV_ERR_NO_MEMORY] error and GPU will disappear later.

No error when just booting Windows VM and shutdown.

I haven't tried the latest driver in windows if that fixes this error. Will update later

here's the userscript and use it after VM releases my 5070ti to Unraid OS

#!/bin/bash

ERR_TEXT="NV_ERR"

LOOKBACK_MIN=5

ERR_LINE=$(dmesg | grep "$ERR_TEXT" | tail -n 1)

echo "$ERR_LINE"

if [ -n "$ERR_LINE" ]; then

ERR_TS=$(echo "$ERR_LINE" | awk -F'[][]' '{print $2}')

CURR_TS=$(cat /proc/uptime | awk '{print $1}')

ERR_TS_INT=$(printf "%.0f" "$ERR_TS")

CURR_TS_INT=$(printf "%.0f" "$CURR_TS")

TS_DIFF=$(((CURR_TS_INT - ERR_TS_INT)/60))

echo " $ERR_TEXT found, occurred $TS_DIFF minutes ago"

if [ "$TS_DIFF" -le "$LOOKBACK_MIN" ]; then

echo " syncing and dropping caches"

sync

sleep 5

if echo 3 > /proc/sys/vm/drop_caches; then

echo " caches successfully dropped"

else

echo " error dropping caches"

fi

else

echo " $ERR_TEXT is not recent; try re-running your VM"

fi

else

echo " $ERR_TEXT not found in recent dmesg buffer"

fi

logs after that error using that script

Edited September 7, 20251 yr by sausuke

Hello, will the next Unraid version support the proprietary driver for the 5000 - series graphics card? Currently, when using Steam Headless, acceleration features such as DLSS cannot be called.

4 hours ago, 王若凡 said:

Hello, will the next Unraid version support the proprietary driver for the 5000 - series graphics card? Currently, when using Steam Headless, acceleration features such as DLSS cannot be called.

AFAIK the Proprietary driver doesn't support 5000 series anymore.

However for certain games you have to pass over custom start arguments to be able to use some Nvidia features, at least this was the case on my Debian Linux Desktop computer for some games but most games nowadays support it right OOB. I'm not familiar with Steam Headless much but maybe a package in the container is missing since you also need some base OS packages but that is just a guess.

hey so I have an issue and havent seen a definitive fix so far. My Nvidia GPU wont update drivers. its a RTX 2070. the nivida tool gives this error.

I tried manually updating with terminal and no luck there either. i do have an error about something being depreciated but google gives nothing about the error or a fix. Im hoping someone here can help. I am currently here are the settings.

1 hour ago, DemonWarriorTech said:

hey so I have an issue and havent seen a definitive fix so far. My Nvidia GPU wont update drivers. its a RTX 2070. the nivida tool gives this error.

There is a Recommended Post on top of this thread, which is sadly not easily visible anymore since the new design, however here is the link to the post (this applies to your issue):

On 9/7/2025 at 9:27 AM, sausuke said:

Finally solved this problem, it's on this thread https://forums.developer.nvidia.com/t/575-64-nvrm-out-of-memory-error-causes-dgpu-to-not-be-usable-after-some-time/336877/11

I upgraded to the latest beta 7.2.0 beta and the same problem exist with latest driver v580.76.05. Used chat gpt to convert it to userscript. That Out of memory [NV_ERR_NO_MEMORY] (0x00000051) occurs on when using VM and playing a game or benchmark related like heaven benchmark.

How to create that error:
Open VM>Play a game or heaven benchmark>Shutdown VM> and when I go to Nvidia driver in settings keep refreshing and you'll see that Check failed: Out of memory [NV_ERR_NO_MEMORY] error and GPU will disappear later.

No error when just booting Windows VM and shutdown.

I haven't tried the latest driver in windows if that fixes this error. Will update later

here's the userscript and use it after VM releases my 5070ti to Unraid OS

#!/bin/bash

ERR_TEXT="NV_ERR"

LOOKBACK_MIN=5

ERR_LINE=$(dmesg | grep "$ERR_TEXT" | tail -n 1)

echo "$ERR_LINE"

if [ -n "$ERR_LINE" ]; then

ERR_TS=$(echo "$ERR_LINE" | awk -F'[][]' '{print $2}')

CURR_TS=$(cat /proc/uptime | awk '{print $1}')

ERR_TS_INT=$(printf "%.0f" "$ERR_TS")

CURR_TS_INT=$(printf "%.0f" "$CURR_TS")

TS_DIFF=$(((CURR_TS_INT - ERR_TS_INT)/60))

echo " $ERR_TEXT found, occurred $TS_DIFF minutes ago"

if [ "$TS_DIFF" -le "$LOOKBACK_MIN" ]; then

echo " syncing and dropping caches"

sync

sleep 5

if echo 3 > /proc/sys/vm/drop_caches; then

echo " caches successfully dropped"

else

echo " error dropping caches"

fi

else

echo " $ERR_TEXT is not recent; try re-running your VM"

fi

else

echo " $ERR_TEXT not found in recent dmesg buffer"

fi

logs after that error using that script

Update to this, got the same error NV_ERR_NO_MEMORY when using latest driver in Windows VM after shutdown, but I used the 5070ti in Invoke Ai with no problems with Unraid apps unlike lower version of Nvidia Driver in plugin. I guess I will just use the userscript and test the later version of Nvidia drivers for Unraid

On 9/9/2025 at 3:32 PM, ich777 said:

There is a Recommended Post on top of this thread, which is sadly not easily visible anymore since the new design, however here is the link to the post (this applies to your issue):

thank you! that seemed to work! it was driving me nuts

On 8/26/2025 at 5:30 AM, ich777 said:

Please post your Diagnostics, I can't say anything without them.

Okay, so sorry for the delay. Been troubleshooting some bad RAM, had a drive die, and been moving. After all these other issues have been fixed. I'm still having an issue with the driver. Attached are diags.

I saw that you suggested updating to the beta. I actually have a ticket open with Lime support about long restart times, and it's apparently related to the NVIDIA driver. (at least according to them) I'd like to prevent updating to the beta until they advise me to.

Thoughts based on what you're seeing?

unraid-diagnostics-20250912-0036 ANON.zip

2 hours ago, lukeb873 said:

Thoughts based on what you're seeing?

Not really, where does it fail to transcode? Plex?

Please also make sure to lower the PCIe Gen for the slot that the card is in to PCIe Gen 3

I really can't tell what's going on since this seems like some hardware incompatibility issue, maybe a newer driver will fix this issue if Nvidia releases one.

JellyFin. Typically, I see it disappear from my GPU stats plugin, and that's how I know it's now messed up and won't be able to do any sort of hardware transcoding.

Could you say more about why I need to lower this?

Gotcha. In the other post, you've recommended moving to the beta. Is this still something you might recommend? I'm still waiting to hear back from Lime support to see what they have to say.

2 hours ago, lukeb873 said:

Could you say more about why I need to lower this?

Something seems to be messed up with 5000 series and the PCIe 5.0 specification, however for your usecase that shouldn affect performance.

2 hours ago, lukeb873 said:

Is this still something you might recommend?

You can try that, since it helped other people, I recommended that because of a newer driver and of course also newer Kernel.

2 hours ago, lukeb873 said:

I'm still waiting to hear back from Lime support to see what they have to say.

I don't think that they can say much because this is basically a 3rd party driver and ot would be probably better to ask on the Nvodia Developer Forums.

However it seems to me that this is a general issue with 5000 series cards and some Motherboards (especially AMD based) since you are not allone with that issue, but most of the time lowering PCIe version is enough to avoid that issue.

So this is what Lime Support said: I can see the exact same issue with the Nvidia driver still occurring quite often. I can see repeated cycles of out of memory followed by RmInitAdapter failed The drive is still crashing and trying to reset the GPU. Hopefully the Nvidia forums might be able to help you with this. But I'd say this is almost certainly the cause of your parity checks pausing because when the driver crashes, it can freeze the system and so interrupt the I/O operations like the parity check. The system will recover from the driver crash, but the parity process isn't always able to actually resume on its own. Looking at the logs, this is what I would suspect. Now, as you've already tried just reseating the cables, I'm assuming you've got a sufficient power supply because your RTX 5070 does use quite a lot of wattage, and an unstable power supply can cause this kind of issue. But that said, if it's happening when the GPU isn't really under load, it makes this probably less likely. 

I do have a Corsair RM1000X so the PSU isn't the issue. As he stated, I've been having issues with the parity check randomly pausing which he's saying seems to be related to the driver crashing.

On 9/13/2025 at 11:36 PM, ich777 said:

Something seems to be messed up with 5000 series and the PCIe 5.0 specification, however for your usecase that shouldn affect performance.

So when the GPU diagnostics plugin does work, it says PCIe Gen (MAX): 4 (4), which I would think means it's not even running at Gen5 in the first place. Would you still recommend going down to gen3?

On 9/13/2025 at 11:36 PM, ich777 said:

You can try that, since it helped other people, I recommended that because of a newer driver and of course also newer Kernel.

I want to see if I can get my Immich container fixed first prior to going to a beta, but after that, I'd be open to moving to the beta.

34 minutes ago, lukeb873 said:

Would you still recommend going down to gen3?

Yes.

34 minutes ago, lukeb873 said:

I do have a Corsair RM1000X so the PSU isn't the issue.

How old is the power supply? You know that PSUs degrade over time and this wouldn't be the first time a user installing a new power supply after 8 years and the issue is gone.

However I still think it's related to hardware incompatibilty or the driver not being fully bug free.

Hello,

I’ve been running into an issue for several weeks where my Nvidia GPU keeps disappearing.

• I replaced the GPU

• I tested different Nvidia driver versions

• I changed the PCIe slot

• I uninstalled the GPU Statistics plugin

• I uninstalled the Nvidia plugin, rebooted the server, and reinstalled it

• I disabled pcie_aspm

The only thing that temporarily fixes it is rebooting the server — but after some time, the GPU disappears again.

I have another Unraid server with the same motherboard and two GPUs, and I don’t have any issues on that system. However, on that machine I don’t use the Nvidia plugin, since both GPUs are used only for VM passthrough.

At this point I don’t know what else to try… It’s quite problematic because the GPU is used by Jellyfin, and as soon as the GPU disappears I get playback errors until I reboot.

Do you have any ideas that could help me move forward with troubleshooting this issue?

Here is the GPU specification when everything is working :

Thanks in advance.

unraid-diagnostics-20250925-0848.zip

Edited September 25, 20251 yr by thymon

2 hours ago, thymon said:

Do you have any ideas that could help me move forward with troubleshooting this issue?

Please update your BIOS since it seems to be way out of date.

However this seems like a common issue with newer drivers, I can only tell you to go to the Nvidia Developer forums and describe your issue there since it seems a lot of users, mostly with AMD Systems experiencing such issues.

However I would recommend that you update your BIOS first.

BTW, you have a lot of and I mean a lot of SSH messages in your syslog, is everything good there?

2 hours ago, ich777 said:

BTW, you have a lot of and I mean a lot of SSH messages in your syslog, is everything good there?

Yes, I saw that too. But I think it's either my rsync scripts or the LuckyBackup docker. I need to look into that. Thanks for your reply, I'll start by updating the BIOS.