The bigger the directory the (disproportionally) slower the access


NLS

Recommended Posts

I have a seemingly huge issue with my unRAID.
Seems the larger the folder in number of files, the slower the file access inside it.
And I mean disproportionally. I don't mean that double the files the double the file access.
And I am not just referring to listing the contents.
I am talking about actual file access, edits etc.

A tiny file with a small change that in a small unRAID folder takes non-measurable time after saving, may take... 1 minute (!!!) for the same tiny file if it resides in a huge folder (containing let say 100k files).

I have read about case sensitivity and SAMBA issues with it, but I cannot experiment with it (yet) as I need to take down the array (else smbd cannot restart).
But I DID try mounting a share using NFS, both case sensitive and non case sensitive...
Supposedly NFS doesn't have such issues.
Results? Even worse!
Actually listing the folder of 100k files in windows explorer didn't even give me any contents after 5 minutes (then I dismounted the share).
Both with case sensitive and non.
I tried listing the folder in command line (where "dir" is more... erm... direct) and indeed started listing files but after around "c" letter it started having hiccups!

With SAMBA at least, listing the folder (in windows) does work, but takes almost 2 minutes.

There is definitely an issue. But large folders is common for me (as I handle large emulation collections) and this is major.
For example I started a batch edit of simple text files in 7200 folders... normally it should take less than a second per file (a single file per folder).
It took more than 2 hours for the first 400 files in folders!!! (so will take almost 2 days for a simple task!)

 

Note that "normal" share access (of smaller folders, file copies etc.) is "ok-ish", not fantastic (40-50MB/s?) but given my unRAID setup it is ok.

 

Any ideas???

 

Link to comment

In Windows Explorer, right click on the Share and select   Properties  and look for the   Customize   tab.   Now look for the "Optimize this folder for:"  and set it to 'General items".   Click the check box to "Also apply this template to all subfolders". 

 

I am not surprised at the sort times for a folder with a 100,000 items in it.  I don't know what sort routine was used by the Windows developers but I would bet they never tested it to see how it handled a 100,000 items!   (The simple bubble sort routine is very fast for a few items but the time increases by the square as the number of items increases.  And this does not even consider the time required to fetch directory data for these 100,000 items across a network.)

 

@JorgeB's question about the filesystem is a very valid one.  The old Reiserfs has real significant problems under certain conditions!

 

Edited by Frank1940
Link to comment

I think you are missing the point.

My problem is not if folder browsing is slow. It is yes, ok, but not a major problem.

 

Again a "demo" of the problem:

 

Editing a pack of let say 400 x 100kb files within a 400 files folder = 150 seconds.

Editing a pack of same 400 x 100kb files within a 100000 files folder = 2 hours.

(numbers are not real, but they are close to scale)

 

This is the problem.

 

I am using latest non-beta release for unRAID and using the default fs it installs on data disks.

Client is Windows 10 x64 with all patches.

 

Edited by NLS
Link to comment
  • 1 year later...

I wake up my own old topic... Because I never fixed that problem, neither did someone actually give me an idea of what could be wrong.

Help?

No I cannot really break the folders to smaller ones (they are part of a specific collection that needs the files in place).

 

This is crazy. I am trying to modify (through some program) around 100000 small files (much less than a mb each) and looks like it will take DAYS!

This is insufferable.

 

Edited by NLS
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.