So I noticed my syslog had spanned to a 2nd file and a little investigation showed that it had a few (~14) thousand lines of shfs_open Too many open files interspersed with shfs_flush bad file descriptors.  This happened when using TerraCopy to copy some directories with large numbers of small files.  After the copy there were ~14000 files that didn't pass the CRC test and Terracopy reported that it could not open the dest file for testing.

 

Some digging here turned up several posts regarding the open file limit of Linux and the open file limit of Samba.

 

When I told TerraCopy to recopy and skip files that were already in the dest it started doing the CRC again and I noticed that some of the files were getting the same error.

 

I opened a telnet session and watched the syslog and sure enough it was reporting the errors there.  Trying to time it while the errors were popping up in the syslog I did a lsof|wc -l and the most I ever got was ~3000 which is lower than what cat /proc/sys/fs/file-max reports (201835).

 

I added an smb-extra.conf file and put max open files = 20000 in the global section and reloaded the smb config using smbcontrol while Terracopy was paused.  I let Terracopy continue and still got the errors.  I let it go through until the end and about half the CRC checks had failed so I

added ulimit -n 20000 to the ident.cfg, took the array offline, and rebooted the unRAID box.

 

Prior to the reboot, testparm -s squawked about rlimit_max being set to 1024, after the reboot there was no more warning.

 

I had Terracopy retry the failed files after the reboot and it completed the CRC check on about ~6k files without error.

 

I'm thinking that the problem is solved and just want someone that's been down this road to go over the data and be sure I have fixed the correct thing

 

Also because the ident.cfg is volatile, I created a script to fix it, but running the script out of /boot/config/go executes too late in the startup process to be effective.  If the file gets clobbered by unRAID then the script fixes it on the next boot, but doesn't actually take affect until the 2nd reboot.  Does unRAID have any cleanup scripts that run on shutdown where the fix script could get called, or does unRAID just use the standard rc scripts on shutdown?

You can run "ulimit -n 200000" directly from the go script.

You can run "ulimit -n 200000" directly from the go script.

You can, but I'm not sure it will affect the samba daemon process.