Same problem here.

 

Maybe related to this?

https://bugs.launchpad.net/ubuntu/+source/samba/+bug/462172

 

We need a Unraid update with this version of samba.

UnRAID was updated to that version of samba.

The bug you are pointing to, and the fix to it, were discussed in the first page of this thread. See Joe's post:

http://lime-technology.com/forum/index.php?topic=5004.msg46112#msg46112

 

Tanks for pointing me that Unraid was already updated with that version of Samba.

 

I tried all the fixes on this thread and I'm still having the problem.

 

I replicate it easily with Beyond Compare. When I compare directories with a large number of small files the problem appears.

 

Just noticed this thread -- have had this issue for several months, ever since building my UnRAID server.

 

It is (a) repeatable;  (b) an issue with every version of UnRAID I've tried (thru 4.5.4); and © never happens if I copy the same folder locally or across my network to another Windows machine.

 

... i.e. it only happens when copying to/from the UnRAID server.

 

If I copy TO the UnRAID server with Windows Explorer the copy works fine -- but when I copy FROM the UnRAID server I get the "Too many files ..." message (repeated pressing of the "Retry" button will eventually let the copy work.

 

If I use TeraCopy the copy works; but the Test always fails.  However ... the copy is actually good.  If I copy the folder back to my Windows box (using the aforementioned repeated "Retry" clicks) and then compare to the original, it always matches.

 

Has anyone found a solution to this?

As a follow-up,  I decided to try Joe's suggestion to add "ulimit -n 20000" to the /boot/config/ident.cfg file.

 

That SEEMS to have resolved this -- at least one folder on my UnRAID that would reliably exhibit the issue if I tried to copy it to any Windows machine will now copy with no problem.

 

To avoid this issue, I've been Zip'ing folders that have large number of small files and just storing the .Zip file on the server for my backups -- but I'll try a couple of those that would reliably have issues sometime in the next few days.

 

But the "ulimit" line clearly helps -- and hopefully simply makes the issue go away !!

 

I had thought the large number of small files issue was some kind of limitation in the Reiser file system ... good to know that's not the case.

 

Finally spent a few hours testing this some more.    The "ulimit -n 20000"  line in /boot/config/ident.cfg  definitely seems to have resolved this.    I copied two folders with thousands of small files each to the server and they both copied perfectly -- I had previously had to .zip them into a single file to get these files stored on the server.

 

I then copied a large backup folder (over 100GB with ~ 120,000 files) and it also copied fine.

 

I'm convinced this has, for all intents and purposes, resolved the issue.

 

I too had this problem when backing up many small files from my unRaid server to an external USB drive.  I entered the "ulimit -n 20000" line in the ident.cfg file and re-booted unRaid.  Worked like a charm.  The entry in the ident.cfg file does not survive when changes are made though that re-write the file, so it will have to be entered again if that happens.

 

I have been using unRaid for several months and while it is a bit techy, it works like a dream.  Many years ago I worked with a Unix like product so I'm not totally in the dark, but I'm still challenged by Linux and unRaid.  I'm glad to see that there are those out there that offer help in the unRaid forums.

This is an interesting issue.  For those who can make this happen "at will", and Joe's fix of putting 'ulimit' command in ident.cfg seems to work, please try this... Instead of putting 'ulimit' command in ident.cfg, put at top of 'go script', ie,

 

ulimit -n 16404

 

Why 16404?  This is 16384 + 20, and 16384 is the default 'max open files' for Samba, and 40 is a "fudge factor" added to this value within the samba code (you can look in source3/lib/util.c of the samba source).  It looks to me that samba is trying to set this limit, but for some reason it's not working.

 

I think by putting it in the 'go' script, emhttp will inherit it, as will samba process that eventually gets launched by emhttp (via rc.samba).

 

Note that if from a telnet session or console, if you type 'samba restart' then the problem will be back,  unless you first type 'ulimit -n 16404' then 'samba restart'.

 

Anyway I'd like the know the results of this test if anyone cares to try...

I'm setting up a test that I can run today.  I'll let you know when I get the results.

Ok, I added the "ulimit -n 16404" to the go script at the top. and re-booted unRaid.

 

I ran two tests on a Windows 7 workstation that would produce the "Too many open files" problem.

 

Both tests ran fine and I got no "Too many open files" errors.

 

I am coinfused about the math though.  You said that the number should be 16384 + 20 = 16404, but then you said the fudge factor was 40.  Is 16404 the correct number for ulimit?

Ok, I added the "ulimit -n 16404" to the go script at the top. and re-booted unRaid.

 

I ran two tests on a Windows 7 workstation that would produce the "Too many open files" problem.

 

Both tests ran fine and I got no "Too many open files" errors.

 

I am coinfused about the math though.  You said that the number should be 16384 + 20 = 16404, but then you said the fudge factor was 40.  Is 16404 the correct number for ulimit?

 

Thank you for running those tests.  Looks like that's the correct workaround for now.  Yes the 'fudge factor' is 20, I don't know where I got 40 from  

 

Edit: to be clear, the workaround I'm suggesting is to put 'ulimit -n 16404' at the top of your 'go' script; but realize if you type 'samba stop/start/restart' from a command line, you should first type the 'ulimit -n 16404' command.  This is because executing the 'ulimit' command sets limits for the currently running process (a bash shell process), and any sub-process created will inherit the resource limits of the parent - this is why putting it in the 'go' script works.  But the real fix is to determine why the Samba code which tries to do this doesn't work - I'll investigate this in the 5.0-beta code branch.

 

Edit2: For those who might use the 'testparm' command, you will find that this also eliminates the following message:

 

"rlimit_max: rlimit_max (1024) below minimum Windows limit (16384)"

 

But the real fix is to determine why the Samba code which tries to do this doesn't work - I'll investigate this in the 5.0-beta code branch.

 

After looking through the code it appears Samba is doing the right thing.  You can verify this by typing this command:

 

ps -ef

 

and find the process id (PID) of an smbd processes (will be the first number after the word 'root' in the list).  Next type this:

 

cat /proc/<pid>/limits     <-- use PID number above for <pid>

 

You will see that "Max open files" is set to 16404 (or 16424), which is correct.

 

So.. question to those who can make this happen....

 

Does this happen only when transferring to/from User Shares?  Put another way, without having the 'ulimit' fix in place, can you get "too many open files" problem when transferring to/from disk shares?

What I am doing when this happens is I'm backing up files from unRaid shares to a USB drive when running the program from a Windows 7 computer.  The backup program I use compresses files while it is backing up.  It is running on a Windows 7 computer.  I have seen several thousand files open at one time on the unRaid server.  IMO the 16404 limit of Samba is not what it is running into.  I've never seen the open files anywhere near that high.

 

I host some buisiness FTP online backups and the FTP share I have on the unRaid server has many small files.  I believe the largest is about 300 MB.  Total space used is about 50GB to 55GB.

 

When I first set up my unRaid server several months ago, I got this same issue when I did a copy and paste from my WHS shares to the unRaid shares from a Windows 7 computer.  I found that using robocopy to copy the files from the WHS to unRaid did not produce the issue.

 

Attached is a snipet of a log file that shows the file open errors in unRaid.  I had to edit out some personal information from the log.  I hope I did not goof it up.

 

I can run additional testing for you, but I'm not sure what it is you're asking from your last post.  It sounds like you want me to try to do this same thing but rather than using \\tower\FTP you want me to use \\tower\diskx\FTP?  I think I got that right.  Anyway my concern is that the files may be spread across all 7 of my drives and might not cause the problem if I limit the copy to one disk.

 

BTW, I am also one of those that gave up on WHS when Microsoft decided to pull DE from Vail.  There are a lot of things to like about WHS, but I do not like the idea of commiting to a technology that gets yanked on a moment's notice, and the key feature of that technology is removed.  I'm really liking the direction of unRaid with what I'm seeing in version 5.

syslog-20101226-181849.txt

I can run additional testing for you, but I'm not sure what it is you're asking from your last post.  It sounds like you want me to try to do this same thing but rather than using \\tower\FTP you want me to use \\tower\diskx\FTP?  I think I got that right.  Anyway my concern is that the files may be spread across all 7 of my drives and might not cause the problem if I limit the copy to one disk.

 

Yes. That is what Lime Tech wants tested, essentially using '/mnt/disk#/' instead of '/mnt/user/'. Likely suspect is the 'shfs' process not having a high enough resource limit or a resource leak instead of the samba process.

I can run additional testing for you, but I'm not sure what it is you're asking from your last post.  It sounds like you want me to try to do this same thing but rather than using \\tower\FTP you want me to use \\tower\diskx\FTP?  I think I got that right.  Anyway my concern is that the files may be spread across all 7 of my drives and might not cause the problem if I limit the copy to one disk.

Right, try a copy to \\tower\diskx\FTP.  Also remember to remove the 'ulimit -n' line from your 'go' script or 'ident.cfg' file.  If you wouldn't mind doing this test, it would save me some time but I think I know what the problem is...

 

Likely suspect is the 'shfs' process not having a high enough resource limit

 

Yup.  This will require a 4.6.1 release  :'(

I can test this, but it will take me a while to set the stage.  My server is pretty much running 24/7 and I need to find a "hole" in the outside FTP activity so I can re-boot and not affect any users.  I'll let you know the results when it is completed.  I should be able to get this done today.

 

Glad to help.  I also develop software and I know that 90% of the solution is reproducing the problem consistently and reliably so it can be shot down.

Ok.  I removed the 'ulimit' from the go script and had no other configuration changes.  I re-booted unRaid.  I set up the test to back up from \\tower\disk1\FTP.  The testing completed without any errors.  I ran it twice with one test just copying files, and another test with compression on just to be sure.  The compression method of backing up would produce the problem for sure when using \\tower\FTP.

 

Hope this helps.

Ok.  I removed the 'ulimit' from the go script and had no other configuration changes.  I re-booted unRaid.  I set up the test to back up from \\tower\disk1\FTP.  The testing completed without any errors.  I ran it twice with one test just copying files, and another test with compression on just to be sure.  The compression method of backing up would produce the problem for sure when using \\tower\FTP.

 

Hope this helps.

 

Helps a great deal, thank you.

Tom,

 

Does this mean the "ulimit" command isn't necessary in the 5.0 release?

 

... what about 4.6 ??

 

Tom,

 

Does this mean the "ulimit" command isn't necessary in the 5.0 release?

 

... what about 4.6 ??

 

I think Tom is narrowing down the issue.  It does not appear to occur when file transfers occur directly to the "disk"' shares, and that seems to eliminate the samba process itself.  From what I've been able to understand, it does occur when going through a user-share, so it points to an issue with the "ulimit" being too low in the shared-file-system process. 

 

It has not yet been fixed, so 4.6 has the same issue.  It is why in one post Tom mentioned a 4.6.1 release will be needed to impliment the fix in the 4.6 series.

 

The exact same problem does exist in all the 5.0beta releases so far, so 5.0b2 has the same problem. Since 5.0 is not yet released, it will probably be fixed before that occurs, now that it has been narrowed down. It may be in the 5.0beta3 release if he fixes it before that release is deployed.

 

Joe L.

Interesting.    I hadn't noticed that you could do the writes directly to a disk and not have the problem.

 

A couple things I've determined that may/may not be useful:

 

I have a single folder with 243 subfolders and 11,454 files that I've used to test this.    It consistently causes the problem, so it's a good test.  In trying it with various releases, I've found the following when using UnRAID from a Windows 7 box ...

 

=>  If I "Copy" the folder to an UnRAID share I get no error messages.

=>  If I "TeraCopy" the folder to an UnRAID share the validation ALWAYS fails

=>  If I do a folder comparison (I use FolderMatch) it will fail due to the "too many files" error

=>  HOWEVER ... if I "Copy" the folder back from the UnRAID server and then do a comparison with the original folder, it matches.

 

... in other words, the writes are working okay;  but apparently when you read the folder back Windows 7 caches files (or perhaps just directory info) quickly enough that it hits the "too many files" limit on the reads.    If I set the "Ulimit -n 20000" parameter everything works perfectly with no errors.  (clearly it doesn't have to be 20000 ... next time I modify the config I'll change it to the 16404 Tom suggests).

 

Turns out this is about a 5 line fix in the code.  The 'ulimit -n' command in the 'go' script is a viable workaround, but I will probably release 4.6.1 with this fix in it.

FYI, Samba Version 3.5.6 disregards the ulimit setting,

and sets its own Max open files to 16520.

 

 

FYI, Samba Version 3.5.6 disregards the ulimit setting,

and sets its own Max open files to 16520.

That only works if it is running as "root" (only root can raise the open file limit) but will not solve the issue of the user-share file-system process having a lower limit.  That will take the 4.6.1 release and Tom's 5 line fix.

FYI, Samba Version 3.5.6 disregards the ulimit setting,

and sets its own Max open files to 16520.

That only works if it is running as "root" (only root can raise the open file limit) but will not solve the issue of the user-share file-system process having a lower limit.  That will take the 4.6.1 release and Tom's 5 line fix.

 

The problem is in shfs (the user share file system), not Samba.  The reason it works to put the 'ulimit -n' command either in 'go' script is because the shell process executing 'go' invokes emhttp, which inherits the resource limit, and in turn invokes 'shfs' which also inherits the resource limit.

 

The reason it works to put in 'ident.cfg' is because as Joe pointed out before, this file is source'ed by rc_samba (the script to start samba) and thus executed at that time.  This script is invoked by emhttp, and it's setting the resource limit for the emhttp process.  Later emhttp starts up shfs and again shfs inherits the resource limit.

 

What's interesting, if you set 'ulimit -n 20000' in ident.cfg, you do NOT see this value for max open files limit on the smb daemons because, as purko points out, samba sets it's own limit (not sure where you get 16520 though, different samba version perhaps?).

 

So either way, the resource limit gets set correctly for shfs - entirely by accident.  I think I'm going to have to produce a 4.6.1 that corrects this problem, I'm just trying to decide whether to put the 4K-sector fix in as well 

FYI, Samba Version 3.5.6 disregards the ulimit setting,

and sets its own Max open files to 16520.

 

not sure where you get 16520 though, different samba version perhaps?

 

 

Yes, different versions.  I was talking about Samba 3.5.6.

That's what I've been running on top of stock unRAID 4.5.6.