• 6.8.0 SMB Ver 4.11.3 significant performance decrease when opening files in folders with 1000+ files in them.


    je82
    • Annoyance

    Hello,

     

    I noticed today after upgrading that my SMB performance had decreased by a lot, but not in the way you would expect. The write/read speeds are fine.

     

    The problem seems to only occur whenever you open a media file in a share where there is 1000+ media files stored in the same root folder.

     

    If you take 50 of the media files, create a folder and move them into a folder so there's only 50 files in the folder, then playback will be normal, new smb is doing something which adds a ton of delay.

     

    Doing the exact same thing on 6.7.2 is much more rapid.

     

    You can re-create the issue by having 1000+ media files in a folder and browse next between them. The more files you have in the folder the slower the performance.

     

    Thanks.

     

     

     

     




    User Feedback

    Recommended Comments



    Are you sure that this is related to large folders only?

     

    The whole day I was fighting with a similar effect. All my apps are portable and stored remote. Opening a file e.g. media/executable took minutes.

     

    I deactivated AV Scanner, Windows Search, did Reboot several times. It was poor pain. It was not possible to work. If it's the same tomorrow, I will go back to 6.7.2 AND reinstall my Windows Client.

     

    BTW, in addition Array Performance dropped by ~10% here.

     

    Edited by hawihoney
    Link to comment

    Yes, as you can see in the demo, i take the very first files that are slow when playing them from a folder where there are 2777 files and put them in their own folder, and the play them, and suddenly everything is nearly instant.

     

    This does not occur in 6.7.2 as also demonstrated in the video.

     

    I found this in the RC bug reports which is most likely very related to my issue

     

     

    Link to comment

    I've had similar experiences but in pure windows environments.

    In general, having large numbers of files in a given folder will cause major speed issues when it comes to dealing with folder display.  I'm kind of surprised 6.7.2 worked faster, though.  You should see the performance impact when you have 10,000+ files in a folder (hint... don't do that.)  Mind you, i'm digging through memories that are 15+ years old here, but it has something to do with SMB and it's presentation methods.  Local disk browsing or even direct file access had no speed issues, just folder browsing of very large folders.

     

    If you have that many files, there's got to be a logical way to organize them into subfolders.

     

    Link to comment

    This is not to excuse the performance regression but more to discourage anyone thinking this is a sane way of organizing files.  😏

     

    As commented elsewhere, this is not sustainable on pure WinOS environments either.  It was a real PITA to cleanup 250,000+ files stored in a single directory.

     

    The solution we eventually employed was to use tiers of subdirectories. For bulk storage of scanned in files we have a database table with file metadata, a GUID PK, and then the absolute file path. The file path used a pattern something like: \\server\share\content\YYYYMM\DD\<first 3 characters of the guid>\<entire guid>\files .

     

    Naturally this doesn't work for user consumption directly from the filesystem for media files, but it's really why no one should ever store all their files in just one directory. At least use subdirectories with an event name or date if it's personal videos and pictures.

     

    Link to comment

    this is not about having 100k files, this is simply a huge performance decrease, at least 250% less speed with a folder of 2777 files in it, where it was zero issue in 6.7.2 so to ignore this issue, sorry to say, i'm back to 6.7.2 because concurrent performance is less of an issue because i only let cache write at nights anyway.

     

    i get that you cannot optimize for 100 k files in a folder but when you see a over 200% decrease in performance when you have 2777 files in a folder its pretty obvious that there's a culprit, the issue is obviously not huge but if unattended it will most likely grow into something bad.

     

    2 hours ago, BRiT said:

    At least use subdirectories with an event name or date if it's personal videos and pictures.

    Sorry to say this this data is categorized :) Youtube has a lot of content and this is all the tourmaterial from this show, its 2777 videos. How more would you want me to categorize it, by outfit or by date/time? It seems stupid to even ask to do it, 2777 files is NOT a problem on a modern filesystem in year 2019!

     

    Link to comment

    FWIW I tried accessing a share on a remote server which has 3088 items in the top-level and it populated windows explorer near instantaneously.  This was via WireGuard connection where the remote server has crappy DSL internet access with mere 4Mbits upload.  These were all music directories and files and playing them via VLC worked ok, there was a slight pause to read the files but I attribute this to the aforementioned crappy DSL link.  Clearly this is not exhibiting the issue being mentioned.

    Link to comment
    16 hours ago, limetech said:

    FWIW I tried accessing a share on a remote server which has 3088 items in the top-level and it populated windows explorer near instantaneously.  This was via WireGuard connection where the remote server has crappy DSL internet access with mere 4Mbits upload.  These were all music directories and files and playing them via VLC worked ok, there was a slight pause to read the files but I attribute this to the aforementioned crappy DSL link.  Clearly this is not exhibiting the issue being mentioned.

    I guess it's the definition of near instantaneously. In my testing over many thousands of calls, I was averaging 2.5s vs 0.7s (for 6.7.2.) for 3k items. When 2 programs are accessing SMB simultaenously, that becomes 5s vs. 1.4s. For 10 programs, 25s vs 7s. I think it's common for services to access SMB shares on a server simultaenously.

    Link to comment

    Other things worth testing, try it on a disk share instead of user share, and if anyone is in a position to do it also try a different protocol like NFS or AFP.

    • Thanks 1
    Link to comment

    I don't have a test server for unRAID so can only try out these suggestions on a weekend when I don't need my production environment up and running. For now, I'm going back to 6.6.7 to avoid slow SMB and the concurrent disk problem in 6.7.2.

     

    Also, I think there was something else going on in addition to the 3-4x slower directory listings. Some of my apps would lag for 20 minutes compared to 5 seconds, so I think there were additional SMB performance regressions. I detailed some other slow behavior in the Prerelease thread, but those were just the regressions I happened to notice from debugging the code in a couple of my apps one weekend, so I may have missed others.

    Link to comment

    @limetech

    What happens when you compare a recursive listing of folders with hundreds of subfolders with each hundreds of files between 6.8.0 and 6.7.2.

    In my case, I am listing backup directories with thousands of directories and several hundred thousands of files and this is significantly slower by factors than on 6.7.2.

    Right now, while I was sending new backups to my Unraid and wanted to browse a folder, everything hung up. Transfer stopped for minutes, Unraid had mover running during transfer and browsing via explorer did timeout. Shouldn't the new multi-stream IO handle such cases? Might the new multi-stream IO be the culprit? Can we disable it and test the legacy behaviour with current Unraid to compare?

    It is difficult to tell you exactly what is going wrong here because I have no direct comparison, but something is definitely not right with the performance like never before...

    Link to comment

    Similar issues that I am observing with UnRAID 6.8.0 vs. 6.6.7 - Was not using 6.7.2 due to concurrent write issues observed.

     

    As I don't have a video to demonstrate 6.6.7 without downgrading my system from 6.8.0, I am showing you a similar example of the same media content but served from a Synology 1815+ over the same network (Gigabit) path back to my Windows 10 Pro PC and you can see the performance difference that I am currently observing with 6.8.0 and SMB shares.

     

    Synology performance that was equivalent to what I was used to seeing with UnRAID 6.6.7:

     

     

    Now with UnRAID 6.8.0, watch in the bottom left corner of the File Explorer window where you can observe it is loading and summarizing the count of the number of folders. You can see the folder sort order is the same yet it takes long enough to refresh the main folder to show it in alphabetical order first then finally revert back to the Date modified order.

     

     

    I am also seeing a significant slow down when processing a directory listing in PHP script that is responsible for renaming folders from a VM running within UnRIAD 6.8.0. This VM is running CentOS 8.x and is mapping the SMB share via CIFS mount within the VM back into the file share of UnRAID. This script ran very quickly on 6.6.7 but on 6.8.0, I can see it is taking upwards of 20-30 seconds while the server is completely idle.

     

    Server itself is a SuperMicro 4U with X9DRI-LN4+ motherboard and dual Xeon E5-2670v2 CPUs.

    Edited by Xed
    Fix version previously used.
    Link to comment

    I am seeing this exact same thing, i went from 6.7.2 to 6.8.0 and the performance dip in large media folder is horrendous. It went from maybe 5 seconds to load a huge directory to now 30+ seconds, it's so bad that I'm considering downgrading. I experienced something similar to this in the past and had to force SMB1 to be used, but this is absolutely happening because of something in Unraid 6.8.0 and I hope it's addressed soon.

    Link to comment

    I saw something similar, but noticed that if I reference my server with the ".local" (ie tower.local) performance is much better.

     

    Link to comment

     

    I have also been struggling with this. I'm tempted to try 6.8.1RC1 since that updates Samba to 4.11.4

    On 1/8/2020 at 7:27 AM, stanger89 said:

    I saw something similar, but noticed that if I reference my server with the ".local" (ie tower.local) performance is much better.

     

    Interesting, what's the performance like when you use IP address?

     

    Link to comment
    On 1/9/2020 at 10:10 AM, smashingtool said:

     

    I have also been struggling with this. I'm tempted to try 6.8.1RC1 since that updates Samba to 4.11.4

    Interesting, what's the performance like when you use IP address?

     

    I am seeing no difference if I use IP to access the share.

     

    I have noticed something else though. When I am connecting to the share via WIFI (Unifi AP), it does not take several seconds to refresh and resort the directory within Windows 10. I noticed this when I tested with my work laptop to the same share (on the same home network). I then plugged in an ethernet jack to the network and now I see the same behaviour with the laptop as well as my desktop PC. 

     

    So there could be something in Windows or a network driver that is not queuing the data properly or out of order or maybe MTU mismatch, I don't know!

    Link to comment

    anyone checked if 6.8.1RC1 resolves this issue? I'm still on 6.7.2 due to the terrible smb performance in 6.8.0.

    Link to comment
    23 hours ago, je82 said:

    Looks like 6.8.1 just went stable, anyone taken the plunge and tried the smb performance?

    I installed it, no difference. SMB is updated to 4.11 but I don't know if Windows 10 even supports that or if it would make a difference.

    • Thanks 1
    Link to comment
    9 hours ago, severanced said:

    I installed it, no difference. SMB is updated to 4.11 but I don't know if Windows 10 even supports that or if it would make a difference.

    Too bad.  I'm looking to upgrade but won't pull the trigger if this problem still exists.

    • Like 1
    Link to comment

    I used to have horrid network share under win10 years ago, under smb or even when navigating shares on other windows machines.

    Similar symptoms to that described by someone above, +30 seconds of waiting.

    Trick was to untick the 2 checkbox (privacy) settings under explorer > folder options > general

    That might help in this instance, if you'd like to try.

     

    For what it's worth I have multiple folders with thousands of folders in each and never see drastic slowdowns,.

    Note that although I have cache directories installed, it's been disabled for months.

     

    win10.folderoptions.jpg

    note: pic is one from net, but note check boxes under Privacy should be unticked.

    Edited by tjb_altf4
    Link to comment

    I'm noticing something very similar myself.  This is an entirely new server using 6.8.1. Systems are having trouble seeing the actual SMB share.  But once it shows up performance is fine.  

    Link to comment



    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.