NFS shares disappear about 24 hours after a reboot, seems to be something with exporting XFS. How am I supposed to finish a rebuild if the box stops sharing data 24 hours in?

Please post the diagnostics after that happens.

I just reboot it, as soon as it happens again I'll post, but dmesg is full of lines like this

 

[Dec17 19:46] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000092b286ac xid e4e1344c
[Dec17 23:04] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000005a3c61d6 xid 5c1dc68f
[  +5.962039] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000005a3c61d6 xid 5d1dc68f
[ +27.335711] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000005a3c61d6 xid 601dc68f
[Dec17 23:10] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000005a3c61d6 xid 761dc68f
[Dec17 23:11] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000005a3c61d6 xid 7d1dc68f
[Dec17 23:12] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000005a3c61d6 xid 811dc68f
[ +22.925024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000005a3c61d6 xid 831dc68f
[Dec17 23:14] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000005a3c61d6 xid 8c1dc68f

I've disabled DOcker, all third party apps, regardless of what I do, NFS shares drop out when I get up every morning to check on the progress of rebuilding. Several days into only being able to rebuild around 8% then I have to reboot because I cannot access my data over the network.

I thought Unraid was reliable.

Please post the complete diagnostics zip.

I did post it to support, before reading through it, this is what they responded:
 

Quote

We have been working on a nasty NFS issue starting in the later 6.12 releases from a Linux Kernel update and continuing into the 7.0 beta and rc releases.  That issue is that the NFS daemon does not stop properly from a stop/start or a restart.  We believe it is now fixed in what will end up being 7.0.0-rc2. 

Unraid has been knowingly pushing out updates with broken NFS implementation since at least 6.12.10

Here is the diagnostics.

tnas-diagnostics-20241218-1042.zip

I recently ran into a similar issue when adding an NFS share between unraid and my proxmox server.  I'd lose shares after about a day.  Looks like my issue was caused by unraid hitting the maximum open file limit, only set to 40k by default.

I'm using this script to increase the file limit, haven't had any issues since:
https://github.com/samssausages/unraid_scripts_and_fixes/blob/630629b0c40309f93626518f9f78472aa36fcc2c/unraid_increase_file_lock_limit.sh

We have identified and corrected the issue with NFS Daemon not stopping properly.  This fix will be in the 7.0 rc2 release and will be backported to 6.12.  Please retest with this release and see if the issue is corrected.

 

2 hours ago, samsausages said:

I recently ran into a similar issue when adding an NFS share between unraid and my proxmox server.  I'd lose shares after about a day.  Looks like my issue was caused by unraid hitting the maximum open file limit, only set to 40k by default.

This is interesting.  I will look into this further.

5 hours ago, samsausages said:

I recently ran into a similar issue when adding an NFS share between unraid and my proxmox server.  I'd lose shares after about a day.  Looks like my issue was caused by unraid hitting the maximum open file limit, only set to 40k by default.

I'm using this script to increase the file limit, haven't had any issues since:
https://github.com/samssausages/unraid_scripts_and_fixes/blob/630629b0c40309f93626518f9f78472aa36fcc2c/unraid_increase_file_lock_limit.sh

 

How do you know you are running out of file descriptors, perhaps seeing "too many open files" errors?  Do you only see this with NFS?

 

Note the system max is set in /proc/sys/fs/file-max which is 813,384 for Unraid, but this can be adjusted too.

 

Curious: what is your workload which is requiring so many open file descriptors?

23 hours ago, limetech said:

How do you know you are running out of file descriptors

After losing shares about every 24 hours (did seem to depend on workload, usually when I ingested large amounts of data), I started searching for causes.  There wasn't a specific error in the log so I had to go through trial and error.
But my search lead me to people talking about the file limit, so I gave it a shot and I stopped dropping shares.

My workload is mainly media, archiving and data hoarding, my stash is very large as I download entire youtube channels and entire websites, with hundreds of thousands of videos and well over 300TB on unraid.

I have another server that runs Plex and connects to unraid using NFS.  I also use a container that generates plex thumbnails in parallel, so it's faster, and a bunch of other services that process media.  Running all those jobs often resulted in the array getting dropped.  It's a 3rd gen Eypc server with 40tb in NVMe, and I do tax the system quite a bit when I have other systems accessing it to help process media. 

I did see it, rarely, without NFS.  But it was so rare that I didn't worry about it, maybe once in 6 months.  But when I moved my Plex to a 2nd server and added NFS for the file share, it got a lot more frequent, and when I started using containers to help re-build the plex thumbnails, it became a daily occourance.

Edited December 20, 20241 yr by samsausages

26 minutes ago, samsausages said:

After losing shares about every 24 hours

Thank you for the reply and very sorry for what's happening.

What do you mean by "losing shares"?  Are shares still present in webGUI Share List?  What is your recovery for when you get into this situation, eg, do you have to reboot the Unraid server?

 

29 minutes ago, samsausages said:

Running all those jobs often resulted in the array getting dropped. 

Do you mean the unRAID array itself?

@limetech No worries!  With losing shares I mean that on the "shares" tab all the shares would be gone.  Couldn't browse the shares even in the CLI and all containers/vm's that use storage crash.  Doesn't matter if it's unraid array or cache.

Disks would still show up on the "main" tab.  I don't remember if I could still browse disks under /main/diskx, unfortunately.  So I can't confirm if it was the fuse layer or the disk mount itself.

A reboot would resolve the issue.

Do you happen to have a set of diagnostics captured when Unraid server is in this failed state?

There is a config variable in v7.0.0-rc.2 which can be used to set the max number of open file descriptors for the 'shfs' processes.  At time of this post, there is no webGUI control for setting this.  Reason being we're not sure if this is the best solution for this issue.

 

Edit the file 'config/share.cfg' and look for this variable:

shareNOFILE=""

 

If the variable is not present you can just add it.  The value of the variable is the numeric value to set for Max Open Files.  For example, to set to 64K you would set to:

 

shareNOFILE="65536"

 

Setting to "" or "0" will cause the Max Open Files to be set to default for all processes as defined in /etc/initscript which is "40960".

 

After changing this variable it is necessary to Stop/Start array for it to take effect.  You should see a syslog message similar to:

Dec 20 11:32:45 Tower shfs: set_nofile: setting rlim_max to 65536

 

@limetech I do have a syslog server setup, but I don't have a good log aggregator setup, so digging through them is a bit of a pain right now.  I did find the logs from last time I crashed, before I put the script in place.  Attached are the relevant logs from right before and after that reboot.  Looks like it crashed around 5am and I rebooted at about 5:35am

If you need logs from another modules let me know, but pretty much everything else is just complaining about losing storage access.

Nice to see a an attempt to fix it!  I'm reluctant to try it right now because I can't easily roll back my zfs pools, so I'll probably just hang on unless that script stops working.

unraid.log

NFS on RC2 died, 20 hours after reboot. Cannot restart NFS service from CLI, ti just hangs. Adding the shareNOFILE="65536" to share.cfg and rebooting

No errors I can see in the log other than this, but it does not show issues with NFS

 

[Dec19 21:13] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid 783fc85a
[Dec19 21:47] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid 6340c85a
[Dec20 03:01] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b0609b42 xid fad92703
[ +27.326179] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid 6b47c85a
[  +3.119699] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid 6c47c85a
[Dec20 03:02] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid 6e47c85a
[Dec20 03:06] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b0609b42 xid 17da2703
[  +1.849612] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b0609b42 xid 18da2703
[  +7.975743] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid 8247c85a
[  +7.896596] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid 8347c85a
[Dec20 03:07] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid 8547c85a
[ +10.003578] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid 8647c85a
[  +3.202704] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid 8747c85a
[ +30.375831] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid 8847c85a
[Dec20 03:08] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid 8947c85a
[ +31.033258] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid 8b47c85a
[Dec20 03:09] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid 8c47c85a
[ +13.820106] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid 8d47c85a
[  +0.000217] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid 8e47c85a
[Dec20 03:10] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid 8f47c85a
[Dec20 03:11] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid 9047c85a
[Dec20 03:12] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid 9147c85a
[  +8.195476] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid 9247c85a
[Dec20 03:16] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid 9747c85a
[ +11.514576] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid 9847c85a
[Dec20 03:17] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000b0609b42 xid 32da2703
[  +3.337581] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid 9a47c85a
[ +11.704560] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid 9b47c85a
[Dec20 03:18] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid 9e47c85a
[  +5.081014] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid 9f47c85a
[ +39.729223] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid a047c85a
[Dec20 03:19] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid a147c85a
[ +28.455490] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid a247c85a
[Dec20 03:20] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid a347c85a
[ +40.411037] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid a547c85a
[Dec20 03:21] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid a947c85a
[Dec20 03:24] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid ae47c85a
[Dec20 03:25] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid b547c85a
[Dec20 03:26] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid b747c85a
[ +25.864910] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid ba47c85a
[Dec20 03:27] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid bb47c85a
[ +31.432609] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid be47c85a
[Dec20 03:29] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid c247c85a
[Dec20 05:33] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000d5831e90 xid 64ca9260
[Dec20 05:37] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000d5831e90 xid 93ca9260
[Dec20 05:39] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000d5831e90 xid a3ca9260
[Dec20 05:42] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000d5831e90 xid c5ca9260
[Dec20 05:43] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000d5831e90 xid d4ca9260
[Dec20 05:45] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000d5831e90 xid e3ca9260

 And array refuses to stop

 

26 minutes ago, badmark said:

And array refuses to stop

Post diagnostics when the array won't stop.

Diagtnas-diagnostics-202412120-1039.zip

1 hour ago, badmark said:

Diagtnas-diagnostics-202412120-1039.zip

I thought you were running 7.0.0-rc2?  Please run rc2 with the addition to the share.cfg file.  You can edit this file, update to rc2, then reboot and it will take effect.

Only the System logs (dmesg) is from the new one, the server was hard locked, all I had was a terminal window running dmesg -wH until it was also unusable - this was on RC2.

25 hours in after adding the shareNOFILE="65536" line to config. Rebuild @ 5..8% with 12 days to go.

Finger crossed.

4 minutes ago, badmark said:

25 hours in after adding the shareNOFILE="65536" line to config. Rebuild @ 5..8% with 12 days to go.

Finger crossed.

Does the log show the:

[Dec19 21:13] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 00000000c25205b7 xid 783fc85a

messages.

 

This is on 7.0.0 rc2?

Yes, still on 7.0.0 RC2 with the script still enabled - Made it 2625 hours in before NFS shares stopped responding, no error attached, just stopped. Paused the rebuild, then stopped, or tried to, waited almost ~30 minutes then sent a reboot command. I don't have the time to backup and rebuild the USB drive with 6.12.9 until after the holidays, we are three families under one roof and it's non stop; "Fun" or least that what they say.

I did take a full diagnostics before I sent a reboot.

Happy Holidays everyone that is helping!

tnas-diagnostics-20241221-2043.zip

On 12/21/2024 at 4:11 PM, dlandon said:

shareNOFILE="65536"

Please try a much higher number, eg, "655360".