November 27, 2025Nov 27 Got an unRAID box recently moved to production. I have a problem after a couple of days that's driving me nuts and is consistent. At some random point a single NFSD daemon suddenly spikes to 100% CPU and hangs there. Once this happens, the SYSTEM memory in the dashboard starts rising and rising eventually exhausting all the memory in the system. There aren't a ton of NFS clients but this is the main use case for this server. Storage backend is a ZFS pool. This is not new per se; the array itself was only recently built but has been problematic since the start. Basically runs for a few days then stops responding with a crashed NFSD (or once, more than one) process eating up a CPU core of utilization and memory growing out of control.NFS services are generally unaffected except for this constant memory growth. All the clients are operating completely normally and none of them seem particularly busy either.Checked DMESG and SYSLOG for errors but nothing seems amiss. Since this a Dell T440 server I checked iDRAC to see if it had snagged any hardware failures but nothing comes up there either.Diagnostics attached. Server's still running and I'm monitoring the situation but the memory use is still growing. Not sure what'll happen when the process eventually eats all the RAM.ixios-diagnostics-20251127-1133.zip Edited November 27, 2025Nov 27 by Sinister Crayon
November 27, 2025Nov 27 Community Expert Nothing obvious that I can see in the diags, if you are using the default 8 threads, increase that to 32 (Settings - NFS)You can also install strace by copying the attachment below to the /extra folder on the flash drive, then reboot to install, if it happens again, get the NFS PID from top, and then post the output fromstrace -p PID strace-6.16-x86_64-1.txz
November 27, 2025Nov 27 Author Will do. I'll start on a reboot now and will post an update when it happens again.For the record I had already increased the NFS threads to 32 Edited November 27, 2025Nov 27 by Sinister Crayon
December 7, 2025Dec 7 Author Alright... problem has occurred again. I tried doing "strace -p PID" but got "Operation not permitted."
December 8, 2025Dec 8 Author Yes. In this case it was "18171". It was one of my NFSD processes, again. I was able to stop the array and restart it and the problem is resolved again. I did not reboot the server, just restarted the array. I did try to just restart NFS first but it wouldn't stop the NFS services, just hung until I stopped the entire array.The SYSTEM memory taken up by the rogue NFSD process is still taken up; it got up to 24G of RAM and my machine's still showing a "SYSTEM" memory load of 24G while under normal operations it usually sits around 6-8GB of SYSTEM memory.Of note; I have some machines connected to the array via NFS. One of them at the time the problem occurred showed an error that the NFS server was not responding in its DMESG logs. However, everything appeared to be working just fine as in I could enter and browse the NFS mounted filesystems without an issue and all the apps running on it were running fine. I did reboot this box as part of my troubleshooting and during boot it refused to reconnect to the NFS server (the unRAID machine). It was only after I restarted the array that the NFS mounts came back without an issue. The mount line used to mount those NFS mounts is;"10.10.10.202:/mnt/ixios/sandata /sandata nfs _netdev,noatime,nodiratime,fsc,nconnect=8,actimeo=1,timeo=600 0 0"I may reboot this system today to try to free that memory up unless you have any specific commands you'd like me to run to gather data. I have attached an updated diagnostic dump since I did not reboot the server, just restarted the array. The problem is not occurring right now since I restarted the array except for the "lost" memory. If it starts again I can try to run the strace again but I just tried strace again on an existing NFSD process but it doesn't appear I can attach to any of them. strace against another process running on the system (Pi-Hole) returns data as expected. NFSD processes all return "Operation not permitted"ixios-diagnostics-20251208-0720.zip Edited December 8, 2025Dec 8 by Sinister Crayon
December 8, 2025Dec 8 Community Expert 1 hour ago, Sinister Crayon said:unless you have any specific commands you'd like me to run to gather data.Sorry, not really, my NFS knowledge is very limited, if that doesn't command doesn't work, don't really have any other suggestions.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.