6.12.1 | NFS Not Coming Online

Please provide diagnostics

 

diagnostics cannot be provided as VM names contain sensitive information. What information is missing from my post found within the diagnostics?

Without diagnostics we can not troubleshoot, it holds a lot more information than your one liner.

 

I could modify the diag package but I do not know where else sensitive information is stored and not obfuscated as ./xml/ contains cleartext filenames.

 

Then my ask is to tell me what information is required and I will obtain it.

Please PM diagnostics to me, so I can troubleshoot.

 

That's a normal message that doesn't mean anything:

Jun 20 16:24:30 BackupServer rpc.statd[6882]: Failed to read /var/lib/nfs/state: Success

That's from my system and NFS is working.

That's not the problem.  That's why the full diagnostics are necessary.

 

Nothing with NFS has changed.  If you're doing a custom exports file, remove your custom settings and see if NFS will start.

Researching some more it appears something is tripped up on the initial loading of nfsd

 

root@Bakery:~# showmount -e
clnt_create: RPC: Program not registered

root@Bakery:~# cat /var/log/syslog | grep nfs
Jun 21 07:38:34 Bakery rpc.nfsd[13045]: unable to bind AF_INET TCP socket: errno 98 (Address already in use)
Jun 21 07:38:34 Bakery rpc.nfsd[13042]: unable to bind AF_INET TCP socket: errno 98 (Address already in use)
Jun 21 07:38:34 Bakery rpc.nfsd[13045]: unable to bind AF_INET TCP socket: errno 98 (Address already in use)
Jun 21 07:38:34 Bakery rpc.nfsd[13044]: unable to bind AF_INET TCP socket: errno 98 (Address already in use)
Jun 21 07:38:34 Bakery rpc.nfsd[13043]: unable to bind AF_INET TCP socket: errno 98 (Address already in use)
Jun 21 07:38:34 Bakery rpc.nfsd[13042]: unable to bind AF_INET TCP socket: errno 98 (Address already in use)
Jun 21 07:38:34 Bakery rpc.nfsd[13044]: unable to bind AF_INET TCP socket: errno 98 (Address already in use)
Jun 21 07:38:34 Bakery rpc.nfsd[13043]: unable to bind AF_INET6 TCP socket: errno 98 (Address already in use)
Jun 21 07:38:34 Bakery rpc.nfsd[13042]: unable to bind AF_INET TCP socket: errno 98 (Address already in use)
Jun 21 07:38:34 Bakery rpc.nfsd[13045]: unable to bind AF_INET TCP socket: errno 98 (Address already in use)
Jun 21 07:38:34 Bakery rpc.nfsd[13044]: unable to bind AF_INET TCP socket: errno 98 (Address already in use)
Jun 21 07:38:34 Bakery rpc.nfsd[13045]: unable to bind AF_INET6 TCP socket: errno 98 (Address already in use)
Jun 21 07:38:34 Bakery rpc.nfsd[13042]: unable to bind AF_INET6 TCP socket: errno 98 (Address already in use)
Jun 21 07:38:34 Bakery rpc.nfsd[13044]: unable to bind AF_INET6 TCP socket: errno 98 (Address already in use)
Jun 21 07:38:34 Bakery rpc.nfsd[13042]: unable to bind AF_INET TCP socket: errno 98 (Address already in use)
Jun 21 07:38:34 Bakery rpc.nfsd[13042]: unable to bind AF_INET6 TCP socket: errno 98 (Address already in use)
Jun 21 07:38:34 Bakery rpc.nfsd[13042]: unable to set any sockets for nfsd
Jun 21 07:38:34 Bakery rpc.nfsd[13044]: unable to bind AF_INET TCP socket: errno 98 (Address already in use)
Jun 21 07:38:34 Bakery rpc.nfsd[13044]: unable to bind AF_INET6 TCP socket: errno 98 (Address already in use)
Jun 21 07:38:34 Bakery rpc.nfsd[13044]: unable to set any sockets for nfsd

root@Bakery:/etc/rc.d# ./rc.nfsd restart
Starting RPC NSM (Network Status Monitor):  /sbin/rpc.statd 
Starting NFS server daemons:
  /usr/sbin/exportfs -r
  /usr/sbin/rpc.nfsd -u -s -H 10.0.10.200 -H 10.253.0.1 -H 10.253.1.1 -H 2603:6000:8b01:433a::7c -H 127.0.0.1 -H ::1 8
  /usr/sbin/rpc.mountd

 

Restarting the daemon seems to bring the service up which tells me it's not a custom exports issue. My hunch is that when rpc.nfsd is being called up initially the IP arguments are not being passed correctly.

 

What are you looking for in the diag package? A quick glance yesterday at even the anonymized version shows I cannot share that file.

 

Edited June 21, 2023 by DiscoverIt

I also had an issue with this. Upgraded to 6.12.1 and my NFS mounts were no longer available. This is the diagnostics file I grabbed before rolling back. I do not have any custom NFS settings, other than running `exportfs -ra` on a schedule, due to stale file issues when the mover runs.

 

smurf-raid-diagnostics-20230621-0816.zip

Edited June 21, 2023 by LilDrunkenSmurf

I can reproduce the issue, will make a fix.

Thx

 

44 minutes ago, bonienl said:

I can reproduce the issue, will make a fix.

Thx

 

What is the cause if nothing related to NFS was changed?

The release notes were missing a NFS related change.

 

What was the missed release note?

Edited June 21, 2023 by LilDrunkenSmurf

1 hour ago, bonienl said:

The release notes were missing a NFS related change.

 

Was it improperly binding interfaces on boot?

 

It’s just a bit irritating as my first question was if a change to NFS occurred and the only concern was diagnostics and not a single mention of this supposed change. 

Edited June 21, 2023 by DiscoverIt

1 hour ago, DiscoverIt said:

Was it improperly binding interfaces on boot?

 

It’s just a bit irritating as my first question was if a change to NFS occurred and the only concern was diagnostics and not a single mention of this supposed change. 

And yet someone else came in here, stated they had the issue, provided diagnostics and unraid was magically able to find and fix the issue...It's almost as if the diagnostics tell unraid what they need to find and fix the problem...SO WEIRD!

4 hours ago, sittingmongoose said:

And yet someone else came in here, stated they had the issue, provided diagnostics and unraid was magically able to find and fix the issue...It's almost as if the diagnostics tell unraid what they need to find and fix the problem...SO WEIRD!

Going to assume this is a troll post as it adds zero productiveness. Regardless if diag was provided the question was posed if anything changed to NFS or the underlying dependencies and the answer was no until the answer became "fixed, there was a change to NFS in 6.12.2".

Edited June 22, 2023 by DiscoverIt

23 hours ago, DiscoverIt said:

Regardless if diag was provided the question was posed if anything changed to NFS or the underlying dependencies and the answer was no until the answer became "fixed, there was a change to NFS in 6.12.2".

 

Nothing nefarious here, just folks trying to help. One person was going off the release notes and said "nothing changed".  Another person realized a change they made did affect NFS but they forgot to list it in the release notes.

 

On 6/21/2023 at 11:32 AM, DiscoverIt said:

It’s just a bit irritating as my first question was if a change to NFS occurred and the only concern was diagnostics and not a single mention of this supposed change. 

 

Diagnostics are a critical part of the troubleshooting process. We ask for them because it helps understand what the problem is so we can reproduce it. We don't have time to play 20 questions to get the right details, if someone refuses to provide diagnostics then it often means their issue can't be resolved until someone else has the same issue and provides them, looks like that is what happened here.

 

BTW, much of Unraid is available in our public repo:

https://github.com/unraid/webgui

You can match up commit dates with the 6.12.1 release to see the exact changes if you are interested.

7 minutes ago, ljm42 said:

 

Nothing nefarious here, just folks trying to help. One person was going off the release notes and said "nothing changed".  Another person realized a change they made did affect NFS but they forgot to list it in the release notes.

 

 

Diagnostics are a critical part of the troubleshooting process. We ask for them because it helps understand what the problem is so we can reproduce it. We don't have time to play 20 questions to get the right details, if someone refuses to provide diagnostics then it often means their issue can't be resolved until someone else has the same issue and provides them, looks like that is what happened here.

 

BTW, much of Unraid is available in our public repo:

https://github.com/unraid/webgui

You can match up commit dates with the 6.12.1 release to see the exact changes if you are interested.

 

Thanks, will make a feature request to further anonymize data to aid in future discussions.

So what was the NFS change that wasn't in the release notes?

8 hours ago, LilDrunkenSmurf said:

So what was the NFS change that wasn't in the release notes?

 

It would probably be something similar to the other items starting with "network" here:

https://docs.unraid.net/unraid-os/release-notes/6.12.1

So any ETA for a work around or a patch? 

 

I am suffering on this one. Possibly unrelated, I seem to be experiencing system instability with system crashes after I run `/etc/rc.d/rc.nfsd start` manually. 

 

disregard I see Version 6.12.2 2023-06-29 came out yesterday. 

Edited June 30, 2023 by flinte

Changed Status to Retest

I have tested 6.12.2, and it is not working in my server.

I tried to add unraid nfs public remote folder in Synology and it failed. It used to be working in 6.11.5. The setting is the same. I'm using virbr0 with 192.168.122.1 to connect Unraid nfs folder. 

I can ping 192.168.122.1 but nfs is not working.

 

9 hours ago, jinlife said:

I have tested 6.12.2, and it is not working in my server.

I tried to add unraid nfs public remote folder in Synology and it failed. It used to be working in 6.11.5. The setting is the same. I'm using virbr0 with 192.168.122.1 to connect Unraid nfs folder. 

I can ping 192.168.122.1 but nfs is not working.

 

Post diagnostics.