Skip to content
View in the app

A better way to browse. Learn more.

Unraid

A full-screen app on your home screen with push notifications, badges and more.

To install this app on iOS and iPadOS
  1. Tap the Share icon in Safari
  2. Scroll the menu and tap Add to Home Screen.
  3. Tap Add in the top-right corner.
To install this app on Android
  1. Tap the 3-dot menu (⋮) in the top-right corner of the browser.
  2. Tap Add to Home screen or Install app.
  3. Confirm by tapping Install.

Repeated problem with NFS

Featured Replies

Got an unRAID box recently moved to production. I have a problem after a couple of days that's driving me nuts and is consistent. At some random point a single NFSD daemon suddenly spikes to 100% CPU and hangs there. Once this happens, the SYSTEM memory in the dashboard starts rising and rising eventually exhausting all the memory in the system. There aren't a ton of NFS clients but this is the main use case for this server. Storage backend is a ZFS pool.

This is not new per se; the array itself was only recently built but has been problematic since the start. Basically runs for a few days then stops responding with a crashed NFSD (or once, more than one) process eating up a CPU core of utilization and memory growing out of control.

NFS services are generally unaffected except for this constant memory growth. All the clients are operating completely normally and none of them seem particularly busy either.

Checked DMESG and SYSLOG for errors but nothing seems amiss. Since this a Dell T440 server I checked iDRAC to see if it had snagged any hardware failures but nothing comes up there either.

Diagnostics attached. Server's still running and I'm monitoring the situation but the memory use is still growing. Not sure what'll happen when the process eventually eats all the RAM.

ixios-diagnostics-20251127-1133.zip

Edited by Sinister Crayon

  • Community Expert

Nothing obvious that I can see in the diags, if you are using the default 8 threads, increase that to 32 (Settings - NFS)

You can also install strace by copying the attachment below to the /extra folder on the flash drive, then reboot to install, if it happens again, get the NFS PID from top, and then post the output from

strace -p PID

strace-6.16-x86_64-1.txz

  • Author

Will do. I'll start on a reboot now and will post an update when it happens again.

For the record I had already increased the NFS threads to 32 )

Edited by Sinister Crayon

  • 2 weeks later...
  • Author

Alright... problem has occurred again. I tried doing "strace -p PID" but got "Operation not permitted."

  • Community Expert

Just to confirm, you used the actual PID number, right?

  • Author

Yes. In this case it was "18171". It was one of my NFSD processes, again. I was able to stop the array and restart it and the problem is resolved again. I did not reboot the server, just restarted the array. I did try to just restart NFS first but it wouldn't stop the NFS services, just hung until I stopped the entire array.

The SYSTEM memory taken up by the rogue NFSD process is still taken up; it got up to 24G of RAM and my machine's still showing a "SYSTEM" memory load of 24G while under normal operations it usually sits around 6-8GB of SYSTEM memory.

image.png

Of note; I have some machines connected to the array via NFS. One of them at the time the problem occurred showed an error that the NFS server was not responding in its DMESG logs. However, everything appeared to be working just fine as in I could enter and browse the NFS mounted filesystems without an issue and all the apps running on it were running fine. I did reboot this box as part of my troubleshooting and during boot it refused to reconnect to the NFS server (the unRAID machine). It was only after I restarted the array that the NFS mounts came back without an issue. The mount line used to mount those NFS mounts is;

"10.10.10.202:/mnt/ixios/sandata /sandata nfs _netdev,noatime,nodiratime,fsc,nconnect=8,actimeo=1,timeo=600 0 0"

I may reboot this system today to try to free that memory up unless you have any specific commands you'd like me to run to gather data. I have attached an updated diagnostic dump since I did not reboot the server, just restarted the array. The problem is not occurring right now since I restarted the array except for the "lost" memory. If it starts again I can try to run the strace again but I just tried strace again on an existing NFSD process but it doesn't appear I can attach to any of them. strace against another process running on the system (Pi-Hole) returns data as expected. NFSD processes all return "Operation not permitted"

ixios-diagnostics-20251208-0720.zip

Edited by Sinister Crayon

  • Community Expert
1 hour ago, Sinister Crayon said:

unless you have any specific commands you'd like me to run to gather data.

Sorry, not really, my NFS knowledge is very limited, if that doesn't command doesn't work, don't really have any other suggestions.

  • Author

Very well. Submitted a ticket and referenced this thread.

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

Account

Navigation

Search

Search

Configure browser push notifications

Chrome (Android)
  1. Tap the lock icon next to the address bar.
  2. Tap Permissions → Notifications.
  3. Adjust your preference.
Chrome (Desktop)
  1. Click the padlock icon in the address bar.
  2. Select Site settings.
  3. Find Notifications and adjust your preference.