Large number of files in a folder causing SMB to fail


Recommended Posts

I have just move my database backup to my UNRAID server, on to the disk array.

The database has pictures that are linked to the database and stored in a folder next to the database files.

Over time the picture collection has grown to 29,900 images in one folder.

 

Viewing the server and folder via the Web browser I can view the 29,900 files, it takes a few moments for the window to show all the images and Krusader happily moves the folder from cache pool to the array.

 

The problem starts when I try and view the folder from my Mac running OS 12.4 (This also happens with OS 12.3.1) via a SMB share.

 

When opening the folder it hangs with a loading symbol but never completes to show the 29,900 images, I can close the window but the finder than crashes and I have to restart my computer.

 

I am currently running UNRAID Version: 6.10.2 but this also happened under Version: 6.10.1.

I have only just moved this large folder to UNRAID so unsure about previous versions.

 

Also I moved the folder to a ssd cache pool and made no difference.

 

All the other shares on the cache pool or array work at speed but do not have the quantity of files that the database contains,

Probably a 1,000 files at the most.

 

Is there a maximum number of files you can have in a folder with a SMB share?

 

I have already added the following code to the SMB Extras under setting to get the search, copy to and from working.

 

#unassigned_devices_start
#Unassigned devices share includes
   include = /tmp/unassigned.devices/smb-settings.conf
#unassigned_devices_end

[global]
   vfs objects = catia fruit streams_xattr
   fruit:nfs_aces = no
   fruit:zero_file_id = yes
   fruit:metadata = stream
   fruit:encoding = native
   spotlight backend = tracker

 

[share name]
   path = /mnt/user/share name
   veto files = /._*/.DS_Store/
   delete veto files = yes
   spotlight = yes

 

Even in  Version: 6.10.1. I need the SMB Extras code to get the search working.

Screenshot 2022-05-30 at 17.22.44.png

Edited by wow001
add image showing the total number of files
Link to comment
Just now, tucansam said:

I just downloaded it as well.  Gonna just go with defaults?

NO. Only enable it on often used problem shares.

 

Enabling it on everything will likely result in worse performance, and drives that never spin down. It works by reading the directory listing at frequent intervals, so that when a listing is asked for, it will likely be in RAM. If you ask it to read too many items, they won't all stay in RAM, and reading them again will keep the drive spun up as it's read over and over again.

Link to comment
8 minutes ago, tucansam said:

I just downloaded it as well.  Gonna just go with defaults?

Only enable it on the shares that need to be cached.  It has a bad habit of caching a lot more that you want, like appdata and system shares that are normally on an SSD and is a waste of resources to cache.

Link to comment

Thank you for the information I have now "include" just the one share that contains the 29,000 images, will check back in the morning and see how it goes.

 

As you say it only this share that is a problem, all my other shares work at speed.

The folder with 29,000 images is not at the top level of the share and is 4 folders deep but it effects the whole share.

Link to comment
On 5/30/2022 at 8:12 PM, wow001 said:

When opening the folder it hangs with a loading symbol but never completes to show the 29,900 images

 

Are you trying to display 29,000 thumbnails?

 

A complete aside:  Why do you have 29,000 photos in a single folder/directory?  How do you ever find anything in it?  

Link to comment

One thing you might also want to consider is changing the SMB setting for the share to be case sensitive (rather than the usual default of case insensitive).  On my system this resulted in a significant performance boost when accessing files on the SMB shares.

 

However you can obviously only do that if you access the files with pathnames of the correct case;  in your case if the pathnames in the database have been stored with the correct case.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.