HawkZ

Wanted to update here. I've been too tied up with work to dig into this too much but over the past (many) months have tinkered with various fruit settings to no avail, not able to find the combination that would both keep fast browsing from Macs yet also not have my backup from my PC (Crashplan) have me routinely hit the open file limit on my Unraid box, requiring me to kill my smb session to regain access to the file shares. It's been quite a painful run, but my attention was elsewhere. Fast forward to this past week, when I upgraded from 6.11.5 to 6.12.4-rc19 on a whim before I dug back into this again. To my amazement, post upgrade I have yet to be locked out for open files, even with Crashplan running regular backups. I don't have notes showing the old fruit settings to compare to but I am guessing something changed there, and/or this was a samba bug that an upgraded version fixed? I am not seeing anything else in the changelogs to really indicate anything that would affect this. Does anyone have any ideas as to what was the culprit/fix?

@JorgeB You are exactly right. I was holding off on making any determinations pending further watching, but was just coming here to reply when I saw your response. So far, have not had the runaway open files issue recur. Have initiated several manual backup runs since the change was implemented to try to trigger it and still no instances of issue. This is really looking like the culprit. Thank you!!! Will keep monitoring but expecting this to be resolved. I do have Mac machines in the house, and we have one machine (which I probably need to reload) that has seemed to have some network share connection issues/delays since the change, but my wife (primary Mac user) has not reported issues on her system yet. It appears from the description this setting may be necessary for being a Time Machine repository (not currently used) as well as more responsive Finder browsing. I may have to dig into some of these vfs_fruit settings if leaving this whole setting is not an option.

This was enabled. I flipped this off and will monitor. Thank you!

CrashPlan backs up files when changed. Runs continuously throughout the day. For local drives on a system it is supposed to trigger a backup on change. For backing up remote volumes it scans it periodically throughout the day. When it does that it inspects each file and does a size/date comparison to the last backed up version to determine if it has changed and needs to be backed up for a change. I don't manually initiate a backup, but after it decides to scan it again, the SMB open files start building up until we hit the limit. I do have over 65535 files, but have no idea why the OpenFiles plugin shows 2 open entries per file. Watching it closer it does appear that files I open through other means only show 1 entry open and clear after I close the file. Items opened through CrashPlan stay open, despite me killing the CrashPlan program/service. I did another test and if I reach the max SMB file limit and reboot the client computer, it does clear the SMB max open files condition (expectedly - but wanted to confirm that data point). Having narrowed this to files touched by CrashPlan I'm a bit stumped as to why this was never a problem with my Drobo but is with Unraid. No response from CrashPlan support yet but given this worked before I doubt they will have much to say about a different result on Unraid which they don't officially support. Really appreciate your assistance here and any further ideas you may have, but other than the knowledge that this worked fine with the Drobo (which was perhaps masking the issue somehow), everything here now has me leaning towards this being a CrashPlan application issue not releasing files properly.

Back to hitting the limit again unfortunately. Here are updated diags. tc-diagnostics-20221206-2314.zip

Right, I saw this in the logs as well. This was being generated from a VPN docker container I was tinkering with but wasn't working. I have regenerated the network.cfg file and switched docker to ipvlan instead of macvlan. In addition, I've turned off all Docker containers other than MariaDB which is slated to be in use soon. I am glad to wipe this one too as a troubleshooting measure though if you feel it is related. Just fired up CrashPlan again and will post back another diagnostic if/when it hits the limit unless you have additional input in the meantime. Thank you!

Here you go. tc-diagnostics-20221206-1026.zip

Well I am approaching it from both sides here and have a ticket open with CrashPlan as well in case something changed recently with an update, but I don't suspect it did (nor can I find any config settings surrounding closing open files). It runs a periodic check for if a file has changed and backs it up in a continual fashion. This is the first time I've had any issues with running into the limit of SMB open files or had visibility into what was being left open, but the configuration of the backup has not changed from when I did this against a Drobo NAS. To the PC/backup software it just sees the drive as mapped drive letters. The difference here is just that the shares are now sitting on Unraid vs the Drobo which is why my primary suspicion is that this is on the Unraid side. I am open to it being being either component here, just trying to narrow this down. Also looking at anything on the Windows pc side that has to do with SMB session timeouts. FWIW, I killed the CrashPlan process on the PC and the SMB open file count did not clear until I killed the smbd process in Open Files. Probably let it sit for about 30 mins as a test.

@dlandon Why are multiple open files being created per-touch/read and never being released? That seems to be the root of the issue:

@dlandon For whatever reason, this issue has returned. Have upped the open file limit to 65535 and am hitting this as well. Attempts to push the limit higher than this have not been successful. Since my prior NAS (Drobo) did not have this issue I am back to my original premise that Unraid is not closing opened SMB files properly. The OpenFiles plugin is showing 2 instances of each file touched held "open" and the list sits there and does not shrink over time, just gets bigger. What are the next steps in troubleshooting here?

@dlandon Thank you very much for your assistance here. I installed the User Scripts plugin and set this to run at startup of the array. Since doing this I have not been able to reproduce the issue, so I suspect it is resolved at this point. I have no settings to adjust on CrashPlan to control the number of files it touches at once or it's methods so either the initial backup of the re-introduced files on the new Unraid server (really should not be "new" backup since everything was in the same folder structure/drive letter - however file ACLs were undoubtedly slightly different) was generating a lot of file touches and the multiple stops/restarts when it hit the maximum have let it finally finish, or else the new maximum limit is raised to the point that it allows it to do what it needed to do. Not knowing if I have a unique case here or not, or if there are negative performance aspects to this setting, I'd certainly consider the increased soft limit to be in consideration for adding in as a default in a future update to avoid this issue for other users. Thanks again!

Thanks for the reply! It looks like I probably need to add the following to the /etc/security/limits.conf * soft nofile 40960 However when adding that it does not seem to take effect after restarting samba. After rebooting the entire Unraid server, the setting in limits.conf does not persist. How can I apply this soft limit on all users either to test or to make persistent?

Ok - finding a bit more. While the above limits would seem where they are set, these are the limits on the running smbd process that are maxing out. Note the "Max open files" of 16464 which matches where it is capping out. However, you'll notice the smbd process is not running under root, which is where I gathered all the current user limit data before, so this process running under user level is getting a DIFFERENT set of limits to it: So, where are these set? Also, of note, found this, but while this fixed the issue, "kernel change notify = no" seems like this would leave some gaps (potentially requiring a restart of samba) to let anyone see any file changes not initiated through samba access: https://www.truenas.com/community/threads/kernel-change-notify-yes.45379/

Per some steps in this article: http://woshub.com/too-many-open-files-error-linux/ it does not appear that the limit at 16424 is from any of these...

This thread is over 4.5 years old and issue still looks like it has not been addressed. Maybe someone listening can add additional instruction to the wiki or the downloader page for the USB Creator, considering the wiki now tells windows users to just use that. I just created a new USB drive as well and had this issue. Had to perform manual steps as others in this video: