NFS stale file handle

[scubanarc]

I get stale file handles all the time on my user shares. Especially when I've done a long operation, or walked away for a long time. I come back, do an ls, and get the following:

 

ls: cannot open directory '.': Stale file handle

 

The easiest fix is to "cd .." and then cd back into the directory and all is well.

 

This problem only exists on my unraid servers. None of my other servers do this. I know that this was a problem back in 2012 and that it was fixed. It appears to be a problem again now.

[dlandon]

Post diagnostics.

[scubanarc]

attached

hal-diagnostics-20220826-0840.zip

[dlandon]

You are running Unraid 6.9.  That version of Unraid uses NFSv3 that is know to have stale file handle issues.  Upgrade to 6.10 that uses NFSv4 that does not have the stale file handle issues.

[scubanarc]

Awesome, great to know. Any major things to be concerned about with changing from 6.9 to 6.10, especially considering the NFS version change? I'd hate to lose all my mappings.

[dlandon]

 

[scubanarc]

Interestingly that doesn't mention that change from NFSv3 to NFSv4. That's such a major change that you'd think it would be included in the release notes. All my web searches about NFSv4 are not productive. Any other information you can share? NFS is a major part of my workflow, so I want to read everything I can before taking the plunge.

[Heikki Heer]

On 8/27/2022 at 5:57 AM, scubanarc said:

Interestingly that doesn't mention that change from NFSv3 to NFSv4. That's such a major change that you'd think it would be included in the release notes. All my web searches about NFSv4 are not productive. Any other information you can share? NFS is a major part of my workflow, so I want to read everything I can before taking the plunge.

Same with me. In UNRAID 6.9.2 I fight with NFS stale file handles especially in connection with mysql und influxdb. So I researched and was happy to see that NFS v4 is finally coming to UNRAID. But after an hassle-free upgrade process to 6.10.3 some of my NFS shares were totally messed up. So much so that I was not able to change the ownership or permissions as root (su). So I downgraded - again hassle-free - to 9.6.2 and all ownership and permissions were back to normal.

 

Why? And why is there so little to no information about that issue that obviously so many of UNRAIDs paying clients have?

[Heikki Heer]

On 8/26/2022 at 10:27 PM, scubanarc said:

Awesome, great to know. Any major things to be concerned about with changing from 6.9 to 6.10, especially considering the NFS version change? I'd hate to lose all my mappings.

As I just wrote: there might be sewere nfs ownership and permission issues when changing 6.9 to 6.10. I tried to analyze a bit what happend in my case. Simple file shares were not affected. It seemed that only shares were affected where databases are stored. But thats just a rough guess as I needed the server to work quickly again and had little time to dig into the problem.

[scubanarc]

NFS4 uses a new pseudofilesystem concept. It is not a straightforward migration path from NFS3 to NFS4. You most likely need to create a "root" nfs folder and share that as the pseudofilesystem root, then "bind mount" all of your desired exports into that folder.

 

I have not made the leap because I'm afraid of the down-time. Has anyone else made the upgrade from NFS3 to NFS4 on UnRaid? If so, how did it go?

[scubanarc]

Or does UnRaid take care of the pseudofilesystem for you, and you just use the GUI to say "Export" and that is all?

 

We really need some more documentation on this before I jump in.

[scubanarc]

Still wondering if UnRaid takes care of the pseudofilesystem or if we have to manage it by hand? How exactly does UnRaid support NFSv4?

[Heikki Heer]

19 minutes ago, scubanarc said:

Still wondering if UnRaid takes care of the pseudofilesystem or if we have to manage it by hand? How exactly does UnRaid support NFSv4?

I have no clue. NFS is very lightly documented. I did not find anywhere any useful information. Would be great if Unraid would document better. For the time being I would be very, very careful with upgrading to 6.10.x as the upgrade might mess up ownership and permissions for some folder and files (my rough guess is that only database related folder and files are affected). I run a kubernetes node on a vm on the the unraid server and after upgrading some of the container related, mounted nfs folders and files got messed up. Some of them were database related.

I paid for the product. A little help from UNRAID would be nice

[scubanarc]

1 hour ago, Heikki Heer said:

I paid for the product. A little help from UNRAID would be nice

I'm in the same boat. I have multiple UnRaid licenses and the only reply that I've gotten from them is "Upgrade to 6.10.x to fix stale file handles", but when I ask for details about how they fall silent.

 

Hey Limetech, can you please document how to convert from NFSv3 over to NFSv4? At least address the issue of whether Unraid handles the pseudofilesystem in exports or not. If Unraid does not handle the pseudofilesystem, then what exactly do I need to do?

[dlandon]

I'll try the best I can to begin to answer some of the issues raised here.  I'm not an expert on NFS, but I'll do what I can to get answers.

 

I think there is some confusion here about the NFSv4 volatile file handles.  I don't understand the comments about pseudofilesystem.  The stale handles in NFSv3 occurred on Unraid mainly when files were moved by the mover.  Moving the files to another location (that's what the mover does) would make the old file handle reference to the file 'stale'.  There were many hacks tried to overcome the file handles - none of the attempts were all that succesfull.  The volatile file handles feature of NFSv4 takes care the stale file handles much better.

 

29 minutes ago, scubanarc said:

Hey Limetech, can you please document how to convert from NFSv3 over to NFSv4?

As far as I know there is no conversion required.  NFSv4 works the same as NFSv3 on Unraid.

 

Those of you with permissions issues, please post diagnostics when you have those issues and we can see if we can find the answers you need.

[scubanarc]

2 hours ago, dlandon said:

As far as I know there is no conversion required.  NFSv4 works the same as NFSv3 on Unraid.

NFSv4 uses a very different concept of exporting paths. You have to export a root filesystem and then export paths relative to that root filesystem. This article explains it well: https://ertw.com/blog/2007/06/24/migrating-from-nfs3-to-nfs4/

 

How does Unraid handle this? Does Unraid export all of /mnt/users as the NFS4 root and then export shares from there? If so, then my clients will have to map to just "/sharename" instead of "/mnt/user/sharename".

 

This really needs to be addressed in the 6.10 release notes.

Edited September 6, 2022 by scubanarc

[dlandon]

27 minutes ago, scubanarc said:

How does Unraid handle this? Does Unraid export all of /mnt/users as the NFS4 root and then export shares from there? If so, then my clients will have to map to just "/sharename" instead of "/mnt/user/sharename".

 

Unraid does not use fstab.  This is the exports file from my test server:

root@BackupServer:~# cat /etc/exports
# See exports(5) for a description.
# This file contains a list of all directories exported to other computers.
# It is used by rpc.nfsd and rpc.mountd.

"/mnt/user/AV Media" -fsid=106,async,no_subtree_check *(rw,sec=sys,insecure,anongid=100,anonuid=99,all_squash)
"/mnt/user/Computer Backups" -fsid=103,async,no_subtree_check *(rw,sec=sys,insecure,anongid=100,anonuid=99,all_squash)
"/mnt/user/Public" -fsid=107,async,no_subtree_check *(rw,sec=sys,insecure,anongid=100,anonuid=99,all_squash)
"/mnt/user/isos" -fsid=105,async,no_subtree_check *(rw,sec=sys,insecure,anongid=100,anonuid=99,all_squash)
root@BackupServer:~# 

Looks to me to be /mnt/user/sharename.

 

This is UD mounting a NFS share on my Unraid test machine:

Sep  6 15:55:42 BackupServer unassigned.devices: Mounting Remote Share 'BACKUPSERVER:/mnt/user/Public'...
Sep  6 15:55:42 BackupServer unassigned.devices: Mount NFS command: /sbin/mount -t 'nfs4' -o rw,noacl 'BACKUPSERVER:/mnt/user/Public' '/mnt/remotes/BACKUPSERVER_Public'
Sep  6 15:55:42 BackupServer kernel: Key type dns_resolver registered
Sep  6 15:55:42 BackupServer kernel: NFS: Registering the id_resolver key type
Sep  6 15:55:42 BackupServer kernel: Key type id_resolver registered
Sep  6 15:55:42 BackupServer kernel: Key type id_legacy registered
Sep  6 15:55:42 BackupServer  rpc.mountd[5560]: v4.0 client attached: 0x89ca572963161e48 from "127.0.0.1:856"
Sep  6 15:55:42 BackupServer  nfsrahead[12889]: setting /mnt/remotes/BACKUPSERVER_Public readahead to 128
Sep  6 15:55:43 BackupServer unassigned.devices: Successfully mounted 'BACKUPSERVER:/mnt/user/Public' on '/mnt/remotes/BACKUPSERVER_Public'.
Sep  6 15:55:43 BackupServer unassigned.devices: Adding SMB share 'BACKUPSERVER_Public'.
Sep  6 15:56:42 BackupServer unassigned.devices: Removing Remote SMB/NFS share 'BACKUPSERVER:/mnt/user/Public'...
Sep  6 15:56:42 BackupServer unassigned.devices: Unmounting Remote SMB/NFS Share 'BACKUPSERVER:/mnt/user/Public'...
Sep  6 15:56:42 BackupServer unassigned.devices: Synching file system on '/mnt/remotes/BACKUPSERVER_Public'.
Sep  6 15:56:42 BackupServer unassigned.devices: Unmount cmd: /sbin/umount -l 'BACKUPSERVER:/mnt/user/Public' 2>&1
Sep  6 15:56:42 BackupServer unassigned.devices: Successfully unmounted 'Public'
Sep  6 15:56:42 BackupServer unassigned.devices: Removing SMB share 'BACKUPSERVER_Public'

 

[scubanarc]

10 minutes ago, dlandon said:

Unraid does not use fstab. 

 

So this means no bind mounts, correct?

 

Also, it looks like Unraid is not using any sort of root filesystem, either, because I do not see fsid=0 anywhere.

 

I think that answers my questions. Thanks.

[dlandon]

Just now, scubanarc said:

So this means no bind mounts, correct?

Correct.

[scubanarc]

I went ahead and upgraded from 6.9 to 6.10 yesterday and crossed my fingers that the NFS mounts would not break.

 

Great news... nothing broke! Literally nothing NFS related broke at all. It was wonderful.

 

If anyone else is in this predicament and holding back upgrading to 6.10 because of the change from NFSv3 to NFSv4, I just wanted to share that for me it was a seamless experience.

 

It appears that Unraid does not export a root filesystem, so there is no pseudofilesystem to think about. I feel like this is fine in an Unraid environment where your exports are all already stored under /mnt/user/ anyways, and you're really not supposed to manage your /etc/exports file manually.

 

Thank you @dlandon for your helpful replies.

[AshranPewter]

Getting stale file handles through NFS, on 6.11.5

 

Attached are the diagnostics. If there's an old setting I need to mess with glad to change anything. Thought I was done with this...

tianding-diagnostics-20221207-0852.zip

 

Edit: Quick note, I did turn on hard links in global share settings recently to see if it would fix, the issue presented itself before and after the setting was changed.

Edited December 7, 2022 by AshranPewter

[scubanarc]

Are you sure that your client is using NFSv4 and not v3?

[dlandon]

Where are you seeing stale file handles?  There aren't any logged in the syslog.

[AshranPewter]

17 hours ago, scubanarc said:

Are you sure that your client is using NFSv4 and not v3?

These are the flags for all my mounts on my client system (ubuntu server) using nfsstat -m

Flags: rw,relatime,vers=4.2

 

17 hours ago, dlandon said:

Where are you seeing stale file handles?  There aren't any logged in the syslog.

 

It's actually really weird 

I know about it because in my plex container I get the error:

Please check that the file exists and the necessary drive is mounted

 

And within the plex files it shows as:

 

Dec 06, 2022 04:46:25.428 [0x7fba5a989b38] ERROR - [Req#47f3/Transcode/Req#4806] Couldn't check for the existence of file "/data/media/music/2CELLOS/Dedicated (2021)/2CELLOS - Dedicated - 07 - I Don't Care.mp3": boost::filesystem::status: Stale file handle: "/data/media/music/2CELLOS/Dedicated (2021)/2CELLOS - Dedicated - 07 - I Don't Care.mp3"

 

I have these mounted on all my other containers just fine... so maybe something to do with.

 

 

Also just now after I've written the above I had a stale file handle error, the docker containers were able to access the media just fine. But in a ssh session with my server it would consistently show stale file handle when I ls in the Media mount. And it would show ?? ?? ? for all the permissions instead of drwxrwxrwx, 99 users.

 

I just did a reboot and it seemed to fix it but still annoying. Don't see anything in the logs for unraid but I'll upload the latest.

tianding-diagnostics-20221208-1151.zip

 

Edit: The weirdest thing is that on my client system, I can access the media files just fine through my docker apps now. Plex even works fine, but through the commandline using 'ls -la' it shows as stale file handles and I can't view it at all.

 

I only run a single docker container on this and have minimal configuration so I'm ok with just resetting it all and start fresh with 6.11.5. But if there's a setting I can change that would fix it, it would definitely save some time.

Edited December 8, 2022 by AshranPewter
some extras here

[AshranPewter]

Bump, maybe someone has some insights or something I can run to fix it!

 

edit: (without removing the mover), if I don't have cache while moving these large files it affects the entire system and if the transfer take too long, can't play any media with my media server. So if possible a solution without that would be awesome...

 

I'm using the mover plugin, not sure if plugin.old is causing this? it's showing up in my logs.

Edited December 10, 2022 by AshranPewter