NFS shares stop working beyond 6.12.9

[Draic]

I have been stuck on 6.12.9 because I keep having issues with versions after that point.

on all latest releases NFS would always stop working after an hour of uptime for the server. There is no error message that I can see on either ends.

 

I am not technical enough to know how to troubleshoot that without help ><

prospero-diagnostics-20240901-1503.zip

[klamath]

I am having issues with 6.12.13, Never had any NFS issues in recent memory, upgraded from 6.12.9 to 6.12.13 on Friday and on Sunday NFS packed up on the unraid server.  Required a reboot of the server to restore connectivity.  Nothing in syslog stood out as the problem.

[armageddon421]

I had a similar thing just happen. Upgraded from 6.12.9 to 6.12.13 last week, used NFS only sporadically. Today after a few hours of continuous use, it suddenly became completely unresponsive, even breaking ongoing file transfers. rc.d/rc.nfsd restart did not help at all, there were a handful of [nfsd] processes listed in ps/htop and they did not change at all when issuing rc.nfsd stop/start. Resorted to a full reboot, that fixed it (for now).

 

Everything else seemed tottally fine with unraid (SMB/docker/VMs).

[coolspot]

On 9/9/2024 at 1:46 PM, klamath said:

I am having issues with 6.12.13, Never had any NFS issues in recent memory, upgraded from 6.12.9 to 6.12.13 on Friday and on Sunday NFS packed up on the unraid server.  Required a reboot of the server to restore connectivity.  Nothing in syslog stood out as the problem.

I just found this thread, my NFS shares stopped working and I suspected it was due to a recent unRAID upgrade. I am also on 6.12.13.

[armageddon421]

It happened again this night, from logs I can see it must have been just 1.5h after I rebooted unraid. On the client machine, all processes accessing the NFS share are stuck in CPU_IOWAIT state. Even "ls" on a mount is just stuck for minutes already now.

 

The only other thing I changed to make NFS work reliably in the first place was to disable hard links. There are only two shares exported, they are set to private and can only be accessed from a certain VLAN/subnet.

[armageddon421]

Just downloaded the diagnostics logs, there is absolutely nothing around the time it must have died, and very little in general.

[armageddon421]

Correction: I was on 6.12.8 before.

 

I'm reverting back to 6.12.8 now to see if that fixes the issue. I hope this will be resolved in the current versions soon.

[coolspot]

Downgrading to 6.12.8 fixed NFS for me - so something was broken between 6.12.8 and the newest release.

[armageddon421]

Same for me, no problem since the downgrade.

[klamath]

I ended up reverting to 6.12.11, everything is stable again, NFS is very broken in 6.12.13.

[Jayman6014]

It is broken in the 7.0 Beta 3 as well.

If I uploaded a lot of stuff to my Plex Docker on another server, it seems that the high load on the NFS when Plex is making thumbnails etc.. as it processes all the new media causes Unraid to lock up.  It also seems to be IP based, my docker server cannot access the NFS shares but I have a 2nd plex server, non-docker I am working on and NFS on that still works without issues.  Like everyone else I have to do a full reboot to fix it.

 

[dlandon]

21 minutes ago, Jayman6014 said:

It is broken in the 7.0 Beta 3 as well.

If I uploaded a lot of stuff to my Plex Docker on another server, it seems that the high load on the NFS when Plex is making thumbnails etc.. as it processes all the new media causes Unraid to lock up.  It also seems to be IP based, my docker server cannot access the NFS shares but I have a 2nd plex server, non-docker I am working on and NFS on that still works without issues.  Like everyone else I have to do a full reboot to fix it.

 

Post diagnostics from 7.0 Beta 3.

[PANiCnz]

Same issue here after a clean install to 6.12.13. Interestingly had previously been running 6.12.11 fine, yet when I manually downgraded to 6.12.11 the NFS issue remained. Have manually downgraded to 6.12.9 to get NFS working again. 

Edited October 24, 2024 by PANiCnz

[EcN]

I think I updated to 6.12.13 recently and the timing of it makes it seems like it was because of this version as well. I've downgraded to 6.12.11 just now and will see how it goes. Getting the same symptoms as everyone describes here. 

ec-unraid-diagnostics-20241024-1513.zip

Edited October 25, 2024 by EcN

[EcN]

Downgrading to x.x.11 didn't help. 

For those of you having the issue, happen to be using proxmox to virtualize unraid? Want to rule out the VM angle. The services I use for NFS is plex, arr services, and frigate. Could def be not the OS's fault. 

ec-unraid-diagnostics-20241025-0747.zip

[armageddon421]

Nope, running bare metal.

[PANiCnz]

Bare metal too. 

[warpspeed]

Yep, had this too, reported it. Doesn't seem to have gotten anywhere. I'm sitting on 6.12.10. On 6.12.10 I get occasional issues with some files, but it mostly works fine. 6.12.11 it won't even connect.

 

Here's some existing threads:

 

Edited October 28, 2024 by warpspeed

[warpspeed]

6.12.13 came with Kernel 6.1.106. 6.1 is now up to 6.1.114. There have been a bunch of NFS changes in 6.1.107, 110, 111, and 113.

 

It'd be great to see another point release for 6.12, with (in the least) a kernel update to see if it resolves the NFS issues.

 

Here's all the NFS changes since 6.12.13 (6.1.106).

 

6.1.107

  Jeff Layton (1):
      nfsd: drop the nfsd_put helper

  NeilBrown (6):
      NFS: avoid infinite loop in pnfs_update_layout.
      nfsd: Simplify code around svc_exit_thread() call in nfsd()
      nfsd: separate nfsd_last_thread() from nfsd_put()
      NFSD: simplify error paths in nfsd_svc()
      nfsd: call nfsd_last_thread() before final nfsd_put()
      nfsd: don't call locks_release_private() twice concurrently


6.1.110

  Trond Myklebust (1):
      NFSv4: Add missing rescheduling points in nfs_client_return_marked_delegations


6.1.111

  Trond Myklebust (2):
      NFSv4: Fix clearing of layout segments in layoutreturn
      NFS: Avoid unnecessary rescanning of the per-server delegation list


6.1.113

  Chuck Lever (3):
      fs: Create a generic is_dot_dotdot() utility
      NFSD: Fix NFSv4's PUTPUBFH operation
      NFSD: Mark filecache "down" if init fails

  Guoqing Jiang (2):
      nfsd: call cache_put if xdr_reserve_space returns NULL

  Jeff Layton (2):
      nfsd: remove unneeded EEXIST error check in nfsd_do_file_acquire
      nfsd: fix refcount leak when file is unhashed after being found

  Li Lingfeng (3):
      nfsd: return -EINVAL when namelen is 0
      nfs: fix memory leak in error path of nfs4_do_reclaim
      nfsd: map the EBADMSG to nfserr_io to avoid warning

  NeilBrown (1):
      nfsd: fix delegation_blocked() to block correctly for at least 30 seconds

  Yanjun Zhang (1):
      NFSv4: Prevent NULL-pointer dereference in nfs42_complete_copies()

 

Edited October 28, 2024 by warpspeed

[bmartino1]

this is noted in bug reports...

the only fix i have found is set your nfs share and reboot unraid. then nfs works as it should sometimes a user script to stop rpc, then nfs then start rpc first and then nfs can fix this. The unriad Dev are aware and working on a fix...
 

 

[coolspot]

On 10/25/2024 at 10:52 AM, EcN said:

Downgrading to x.x.11 didn't help. 

For those of you having the issue, happen to be using proxmox to virtualize unraid? Want to rule out the VM angle. The services I use for NFS is plex, arr services, and frigate. Could def be not the OS's fault. 

I do not have any containers or proxmox running, only file sharing services. I do have a bunch of plug-ins (most which I don't really use) but I can't see anything that would affect networking except for TailScale?

[dlandon]

31 minutes ago, coolspot said:

I do not have any containers or proxmox running, only file sharing services. I do have a bunch of plug-ins (most which I don't really use) but I can't see anything that would affect networking except for TailScale?

Tailscale restarts services and this will cause NFS to stop working.  It should be fixed in the next release of 6.12.

[dlandon]

Please retest this with 6.12.14.

[JRusty15]

On 11/28/2024 at 8:44 AM, dlandon said:

Please retest this with 6.12.14.

I'm having the same issue between two unraid servers. Both on 6.12.14. Can't get NFS shares to connect, just tells me server is offline and can't mount it.

[dlandon]

2 hours ago, JRusty15 said:

I'm having the same issue between two unraid servers. Both on 6.12.14. Can't get NFS shares to connect, just tells me server is offline and can't mount it.

It looks like you are talking about UD.  That is not this problem.  Go to  Settings->Unassigned Devices and click on the 'Remote Servers' tab and enter your remote server name or IP address and then click 'Test'.  UD will run a check and show what is not working about your remote share mounts.

 

If you need further help, show the test results here.