• Slow SMB performance


    ptr727
    • Minor


    User Feedback

    Recommended Comments



    9 minutes ago, StevenD said:

    You

    You really should use a much larger file for testing.  At least bigger than your RAM size.  I typically use 50-100GB files for testing.

    Except that the speed issues I was experience only presented themselves with really small files, like ~4kb and the speed would drop to abysmally slow. As the packet captures showed, it was having lots of errors for each file on the server side.

     

    This issue is still present in the 6.9 beta.

    Link to comment
    Share on other sites
    49 minutes ago, TexasUnraid said:

    Except that the speed issues I was experience only presented themselves with really small files, like ~4kb and the speed would drop to abysmally slow. As the packet captures showed, it was having lots of errors for each file on the server side.

     

    This issue is still present in the 6.9 beta.

    I don't believe I quoted you, or addressed my comment to you.

    Link to comment
    Share on other sites
    2 minutes ago, StevenD said:

    I don't believe I quoted you, or addressed my comment to you.

    No, I was just pointing out that testing with larger files is not always better and sometimes will give false positive results.

    Link to comment
    Share on other sites
    2 hours ago, StevenD said:

    You really should use a much larger file for testing.  At least bigger than your RAM size.  I typically use 50-100GB files for testing.

    I noticed CloudMounter uses a local cache. Hence the blazing 1300Mbps...

    Link to comment
    Share on other sites
    35 minutes ago, Womabre said:

    noticed CloudMounter uses a local cache.

    SMB should usually use a cache as well, but does not happen with my Unraid server on downloading the same file multiple times (32GB RAM installed, but never maxes out 10G).

    Link to comment
    Share on other sites

    Regarding slow Samba, may I make a suggestion?

     

    If you are not using ACL, and chances are you are not, then disable that feature in Samba config.  That made a HUGE difference on my server, and for the first time ever I was able to saturate my network adapter. 

     

    Try it out:  Add this line in the "Samba extra configuration", see Settings -> SMB -> SMB Extras

    nt acl support = No

    (Or just edit  /boot/config/smb-extra.conf  and add that line in the [global] section.)

    You can also run a testparm on the command line to verify that the setting is in effect.

    Restart Samba, and re-run your speed tests.  Let us know how it goes.

     

    Note:  I ran my tests ram disk to ram disk, to eliminate everything else but Samba.  I do not know how much this will help with user-shares, as I don't use them.  Chances are, user-shares may have separate problems, all of which contribute to the overall network slowness.

    Edited by Pourko
    Link to comment
    Share on other sites

    Interesting, I will try the ACL disable.

     

    What is ACL exactly? I pretty much just use windows for SMB. It seems to be some sort of windows permissions interface?

     

    What real world features does this have?

    Link to comment
    Share on other sites
    On 9/2/2020 at 6:59 AM, TexasUnraid said:

    Interesting, I will try the ACL disable.

     

    What is ACL exactly? I pretty much just use windows for SMB. It seems to be some sort of windows permissions interface?

     

    What real world features does this have?

    ACL is access control list.  It's a way of controlling permissions.  On a Windows machine you have NTFS filesystem permissions with an ACL and you also have the share level permissions.  Typically you control permissions using the filesystem since it offers more control.

     

    On *nix with Samba, it will try to map the filesystem permissions to NT filesystem permissions.  Since unRAID really only relies on share level permissions to control access this may eliminate some extra overhead.  I'm going to test it myself to see if I notice any improvement.  If you're not interested in having a public share that you can restrict access to individual folders under the share, you don't really need this enabled.

    • Thanks 1
    Link to comment
    Share on other sites
    14 minutes ago, _whatever said:

    ACL is access control list.  It's a way of controlling permissions.  On a Windows machine you have NTFS filesystem permissions with an ACL and you also have the share level permissions.  Typically you control permissions using the filesystem since it offers more control.

     

    On *nix with Samba, it will try to map the filesystem permissions to NT filesystem permissions.  Since unRAID really only relies on share level permissions to control access this may eliminate some extra overhead.  I'm going to test it myself to see if I notice any improvement.  If you're not interested in having a public share that you can restrict access to individual folders under the share, you don't really need this enabled.

    Cool, thanks for the explanation. I don't use public shares at all so sounds like I will be fine.

    Link to comment
    Share on other sites
    2 minutes ago, TexasUnraid said:

    Cool, thanks for the explanation. I don't use public shares at all so sounds like I will be fine.

    Well, that isn't the only use case.  If you have another share that is private and multiple users have access, you can use the filesystem permissions to stop certain users from being able to traverse certain folders within the share.  If you're not doing this, you'll be okay.

    Link to comment
    Share on other sites
    1 minute ago, subivoodoo said:

    Any performance gains with disabled ACL?

    I won't really know until I do my backup next week, that is where I see the biggest slowdowns, taking almost 10x as long as when I was using windows.

    Edited by TexasUnraid
    Link to comment
    Share on other sites
    1 hour ago, subivoodoo said:

    Any performance gains with disabled ACL?

    I have not had a chance to test but I plan to later today while copying a lot of 4KB files.

    Link to comment
    Share on other sites

    I tested this using about 2500 4KB files and I saw no discernible difference with nt acl support = no and nt acl support = yes.  Transfer rates bounced between 95Kbps and 108Kbps with either value used.

    Link to comment
    Share on other sites

    @_whatever

    Please check how many parallel SMB connections are open while the backup is running. Open PowerShell and enter this command:

    Get-smbconnection

    Regarding my testing Unraid has a really bad scalability of SMB sessions ("NumOpens"). This causes that a running backup completely kills the transfer speed for the SMB connection. The interesting part is that a parallel running second SMB connection is able to transfer with full speed (as long it does not open many sessions, too).

    Link to comment
    Share on other sites
    6 hours ago, mgutt said:

    @_whatever

    Please check how many parallel SMB connections are open while the backup is running. Open PowerShell and enter this command:

    
    Get-smbconnection

     

    I'll check and see.  I'm just using file explorer to copy the files from a local folder to a cached user share on unRAID.  The throughput seems to be consistent with what I get when copying the same files from a Windows 10 machine to another Windows 10 machine both on 1GbE connections.  When my scheduled image backups run using Macrium Reflect, I get pretty much full gigabit transfer rates for the duration of the backup.

     

    I'm also going to copy using some multi-threaded programs, like robocopy, unstoppable copier, etc.  It seems to me like the throughput I get using Explorer is about what I should expect to get with many small files, but to your point, the issue people experience may have something to do with multiple concurrent threads/connections.

    Link to comment
    Share on other sites

    @ptr727 (and all other users)

    I have a theory regarding this problem. I guess its the same thing that causes different transfers speeds for different unraid users with 10G connections. Unraid uses a "layer" for all user shares with the process "shfs". This process is single-threaded and by that limited through the performance of a single cpu core/thread.

     

    You are using the E3-1270 v3 and it reaches 7131 passmark points. As your CPU uses hyperthreading I'm not sure if the shfs process is able to use the maximum of a single core or splits the load. If its using 8 different thread, one has only 891 passmark points. Since a few days I'm using the i3-8100 and with its 6152 passmark points its weaker than yours, but as it has only 4 cores/threads its core performance is guaranteed at 1538 points. Since I have this CPU I'm able to max out 10G and have much better SMB performance.

     

    I would be nice if we could test my theory. At first you need to create many random files on your servers cache (replace the two "Music" with one of your share names):

    # create random files on unraid server
    mkdir /mnt/cache/Music/randomfiles
    for n in {1..10000}; do
        dd status=none if=/dev/urandom of=/mnt/cache/Music/randomfiles/$( printf %03d "$n" ).bin bs=1 count=$(( RANDOM + 1024 ))
    done

    Then you create a single 10GB file on the cache (again replace "Music"):

    dd if=/dev/urandom iflag=fullblock of=/mnt/cache/Music/20GB.bin bs=1GiB count=20

    Now you use your Windows Client to download the file "20GB.bin" through the Windows Explorer.

     

    While the download is running you open in Windows the command line (cmd) and execute the following (replace "tower" against your servers name and "Music" against your sharename):

    start robocopy \\tower\Music\randomfiles\ C:\randomfiles\ /E /ZB /R:10 /W:5 /TBD /IS /IT /NP /MT:128

    This happened as long I had the Atom C3758 (≈584 core passmark points😞

    2020-09-11_14-03-01_02-00_463_294_1240270_1.jpg.127af1b8ba6c9cbc611af69ab3c7b35d.jpg

     

    And this with my new i3-8100:

    444850375_2020-09-2109_22_28.png.b94225dd8b6c9db8f28a20b2f0c75acd.png

     

    Finally you could retry this test after you disabled Hyperthreading in the BIOS. By that we could be sure how relevant it is or not.

    • Thanks 2
    Link to comment
    Share on other sites

    I am on beta 30 and the smb performance is still abysmal for small files or just dealing with large amounts of files.

     

    Except for transferring large files, everything takes many times longer with unraid then windows. Backups are a royal pain, takes half a day to do what used to take an hour.

     

    I now go out of my way to not do anything over SMB if I can possibly help it. I log into the server and use krusader on the server itself to mess with files anytime I possibly can.

    • Like 1
    Link to comment
    Share on other sites

    I assumed this slowness was specific to my Mac environment - painfully slow Time Machine backups. Looks like it's unRAID, huh.

    Link to comment
    Share on other sites
    42 minutes ago, CS01-HS said:

    I assumed this slowness was specific to my Mac environment - painfully slow Time Machine backups. Looks like it's unRAID, huh.

    Yeah, unraid for sure.

    37 minutes ago, rhard said:

    Could it be this has something to do with old SSD write issue? Did you already reformat your NVMs if any?

    https://unraid.net/blog/unraid-6-9-beta29

    No, that was a separate issue. I dealt with that by creating a seprate XFS formatted pool for docker etc with regular backups to the array and using my BTRFS main cache pool just for data.

     

    There is still some excessive writes compared to what I think it should be given the workload but not enough to worry about now vs the 100x writes I was seeing before.

     

    This SMB issue has been present all along. Anytime I deal with a large number of files, everything slows to a crawl. Even just scanning my media collection takes forever now.

     

    I did some packet captures awhile back and there was a lot of failures and issues in the packets the explained the slowness but no explanation on what caused them.

    Link to comment
    Share on other sites

    I recently found this thread too.. my write speeds from win10 vm to unraid shares is capped at 2MB/s per transfer... very slow :(

    Link to comment
    Share on other sites



    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.