I'm really confused here.  You said:

On 12/21/2024 at 6:06 PM, badmark said:

25 hours in after adding the shareNOFILE="65536" line to config. Rebuild @ 5..8% with 12 days to go.

and the same day you said:

On 12/21/2024 at 8:00 PM, badmark said:

Yes, still on 7.0.0 RC2 with the script still enabled - Made it 2625 hours in before NFS shares stopped responding, no error attached, just stopped.

How can that be?

 

And stop using the script.  Make this change as suggested:

4 hours ago, limetech said:

Please try a much higher number, eg, "655360".

We need to have answers that we can implement in Unraid.  The script is not the answer to address this issue.

I have not had time to rollback, but I stopped the array rebuild and the server has been up for two days now without issue. I still need to rebuild the array, but am reporting that NFS is still working because I do not have array rebuild running, could these two be associated?

Happy Holidays!

Rolled back to 6.12.9 Rebuild in progress. Fingers crossed NFS remains active until the rebuild is finished, Estimated finish: 25 days, 14 hours, 41 minutes

1 minute ago, badmark said:

Estimated finish: 25 days, 14 hours, 41 minutes

That sounds far too long.   You might want to post diagnostics to see if we can see a reason for it to take that long.

On 12/18/2024 at 9:53 AM, badmark said:

I just reboot it, as soon as it happens again I'll post, but dmesg is full of lines like this

 

[Dec17 19:46] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 0000000092b286ac xid e4e1344c
[Dec17 23:04] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000005a3c61d6 xid 5c1dc68f
[  +5.962039] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000005a3c61d6 xid 5d1dc68f
[ +27.335711] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000005a3c61d6 xid 601dc68f
[Dec17 23:10] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000005a3c61d6 xid 761dc68f
[Dec17 23:11] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000005a3c61d6 xid 7d1dc68f
[Dec17 23:12] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000005a3c61d6 xid 811dc68f
[ +22.925024] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000005a3c61d6 xid 831dc68f
[Dec17 23:14] receive_cb_reply: Got unrecognized reply: calldir 0x1 xpt_bc_xprt 000000005a3c61d6 xid 8c1dc68f

I've disabled DOcker, all third party apps, regardless of what I do, NFS shares drop out when I get up every morning to check on the progress of rebuilding. Several days into only being able to rebuild around 8% then I have to reboot because I cannot access my data over the network.

I thought Unraid was reliable.

I am having a similar issue. Googling the dmesg entries brought me here, as my syslog had similar messages. This all started when I upgraded from 6.12.8 to 6.12.14. After upgrading, I could not mount any unraid exported NFS shares no matter what I tried. I downgraded to 6.12.9 and NFS seemed to work fine for about half a day then I started getting those syslog messages and NFS became unresponsive. I have now downgraded back to 6.12.8 and NFS is working fine for now.

Edited December 26, 20241 yr by lespaul

Nope NFS still dying when rebuilding array on 6.12.9!

Which version does not have NFS issues?

Downgraded to 6.11.5 rebuilding array now, GUI elements are out of place, but everything seems to be in working order. Estimated finish: 20 days, 4 hours, 57 minutes

Finger crossed this finally does it as my only other option is to build an entirely new box with drives to transfer it all over!

 

4 hours ago, lespaul said:

I am having a similar issue. Googling the dmesg entries brought me here, as my syslog had similar messages. This all started when I upgraded from 6.12.8 to 6.12.14. After upgrading, I could not mount any unraid exported NFS shares no matter what I tried. I downgraded to 6.12.9 and NFS seemed to work fine for about half a day then I started getting those syslog messages and NFS became unresponsive. I have now downgraded back to 6.12.8 and NFS is working fine for now.

Unraid 6.12.9 and beyond all have NFS issues.  We believe we have addressed the last of the NFS issues with 7.0.0-rc2.  All Unraid versions from 6.12.9 to 6.12.14 will have one issue or another with NFS.  Either stay on 6.12.8, or upgrade to 7.0.0-rc2

39 minutes ago, badmark said:

Downgraded to 6.11.5 rebuilding array now, GUI elements are out of place, but everything seems to be in working order. Estimated finish: 20 days, 4 hours, 57 minutes

Finger crossed this finally does it as my only other option is to build an entirely new box with drives to transfer it all over!

 

Finish your array rebuild and then come back here for more debugging.

Not even 3 hours into rebuild and NFS shares are gone.

What now?

 

tnas-diagnostics-20241226-1540.zip

Do the following:

• Upgrade to 7.0.0-rc2.  This is the best version to use with NFS issues addressed.

• Set the shareNOFILE="655360".  A much larger number.

• Do not use the sript mentioned above.  Be sure to stop/start or reboot the server to get the setting to take effect.

Upgraded back to 7.0.0 RC2, running array rebuild and NFS shares are online. shareNOFILE setting has been changed to "655360", previous script disabled.

Finger crossed,  Estimated finish: 13 days, 11 minutes
 

I just found this in your latest syslog.

Dec 26 12:51:50 TNAS kernel: r8169 0000:08:00.0 eth0: jumbo features [frames: 9194 bytes, tx checksumming: ko]
Dec 26 12:51:50 TNAS kernel: r8169 0000:09:00.0 eth1: jumbo features [frames: 9194 bytes, tx checksumming: ko]

There are several issues here.  First the Realtek NICs are notorious for having issues with Linux because the drivers are not well maintained, and second Jumbo Frames are not recommended because it requires a lot of network configuration to get it right and honestly it is not worth the issues caused by misconfiguring Jumnbo Frames.

 

What is confusing is that eth0 is showing in ifconfig with an mtu=1500:

eth0: flags=4419<UP,BROADCAST,RUNNING,PROMISC,MULTICAST>  mtu 1500
        ether 38:d5:47:78:ac:53  txqueuelen 1000  (Ethernet)
        RX packets 9658345  bytes 13781359482 (12.8 GiB)
        RX errors 0  dropped 347  overruns 0  frame 0
        TX packets 1068484  bytes 413838386 (394.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0
        device memory 0xfe400000-fe41ffff

 

Please check your configuration and verify that Jumbo Frames are not being used.  This can cause a lot of problems with NFS and SMB shares.

As I stated earlier, I was trying o upgrade my ethernet, but that multiple port card is not on the network, only the igb (Intel) onboard 1 Gbps ethernet (eth0) is the only one connected.

The MAC points to Asus, the brand of the motherboard: https://macaddress.io/mac-address-lookup/Ekndd7MO2Z

I haven't made any changes to Jumbo Frames.

Server just stopped rebooted. 

This was dmesg output prior:

 

3 hours ago, badmark said:

Not even 3 hours into rebuild and NFS shares are gone.

Do you lose just the NFS share access or are all your shares gone.  i.e. Do you see shares in the UI on the 'Shares' tab, or is it empty?

Is there any possibility you can boot the server in safe mode and run the rebuild to see if there is something going on with your hardware or something else?  You have too many things going on for us to find what issue is causing your problems.  You should get the emulated disk rebuilt anyway.

Server just continues to reboot now, this was never an issue before.

If I don't have access to my NAS, then my entire network is pointless. I've replaced 4 drives while running Unraid, I've never encountered this issue. I kept the same number of services (Proxmox, Docker Swarm and Plex) running while everything rebuilt. This time the server is just flat rebooting and not even throwing an error?!

Restarted in Safe mode, rebuilding the array now. I guess I'll end the year with my entire family hating me...

19 hours ago, badmark said:

Restarted in Safe mode, rebuilding the array now. I guess I'll end the year with my entire family hating me...

When that is complete, I will have a plan on what we want to do next.  In order to develop that plan, I will need some information.  Please answer these questions:

• When you say shares disappear, what shares are you referring to?  Unraid NFS shares, UD NFS Shares, or Unraid user shares?  Look at the 'Shares' tab of 'Main' and let me know if user shares show.  If none show, that's an issue with shfs.  If shares show it will be an NFS issue.  When the shares drop, can you access the user shares with a Windows computer?
• I see you are mounting some Drobo shares with UD.  Are these the troublesome shares, or the Unraid shares mounted by another server using NFS?
• What packages are you loading with NerdTools and Un-get?
• What comprises your NFS traffic?  Media sharing, downloaders, backup?
• Can you disable all your NIC cards with the exception of the Intel NIC?

There is a Linux bug report here that may be related to the issue you are having,

 Elapsed time: 1 day, 22 hours, 25 minutes - I had as minimal as possible talking over NFS, I thought I was going to make it, but 13% in and poofd, they're gone!

Diagnostics attached.

 

tnas-diagnostics-20241228-1914.zip

I was hopoing you would run the rebuild without the NFS sharing going on.

 

I'm seeing 15 NFS clients attaching and detaching indicating a network or workload issue.  Add this to your /flash/config/go file and reboot:

# Increase the number of NFS threads.
sed -i 's#/usr/sbin/rpc.nfsd#/usr/sbin/rpc.nfsd 32#' /etc/rc.d/rc.nfsd
/etc/rc.d/rc.nfsd restart

Try again.

 

You said that you lightened the NFS load, but it still looks pretty heavy.  If your rebuild is that slow, you are probably hammering the server.

 

Also, be sure to set the file descriptors value to the large number.

 

I'm still looking for some answers to my questions in the previous post so I can help you further.  Without more information, I may not be able to offer much more help.

I'm sorry, I'm all over the place, what questions have I not yet answered?