• [6.7RC] Whole RC got slow disk performance issue


    Vr2Io
    • Solved Annoyance

    This is to report whole 6.7RC got slow disk performance, I have setup as below

     

    - 3 disk connect to LSI HBA

    - 1 SSD connect to onboard SATA

    - each disks have only one 24GB file for testing

    - same issue reproduce on 3 different PC platform

     

    Test by cat /mnt/disk(x)/xxxx > /dev/null

     

    In 6.6.7, even 3 disk read at same time, throughput was normal

    61.PNG.2eae59087a1edb6c194672a4cac2fa5b.PNG    63.PNG.9433b027f8c49de528e3776cd0ca8a4b.PNG

     

    But after 6.7RC, even onlyone disk access, throughput already drop.

    71.PNG.9086bc38787b23f01f322f5551702060.PNG    73.PNG.66b7b58a7d88bc3966c35a8d1853e2a1.PNG

     

    I also make test on SSD which connect to onboard SATA to eliminate the issue cause by HBA.

    7S.PNG.6f879f10d8df24d17b5645264f1196a7.PNG

     

     

     

     




    User Feedback

    Recommended Comments

    Please try a command like this:

     

    time cat /mnt/disk(x)/xxxx > /dev/null

     

    Hopefully 'xxxx' is a single large file.  Then calculate the transfer rate and see if it agrees with the dashboard display - just wondering if this is true transfer rate issue or reporting issue.

    Link to comment

    Sure all disks have a single large 24GB file only and not dashboard display issue.

     

    Below are the test make, 6.7rc then restore back to 6.6.7

     

    time cat /mnt/disk1/D/1 > /dev/null   ( Disk connect to LSI )
    time cat /mnt/disk4/D/4 > /dev/null   ( SSD connect to onboard SATA )

     

    7.png.1ccba4445cfb2118d2e07cac39339aed.png

    Edited by Benson
    Link to comment

    Can't reproduce this on my test server with a 20GB file, can you gives us more detail on the hardware used, mainly board/CPU? Just in case I have something similar where I can try.


     

    v6.6.7
    
    root@Test:~# time cat /mnt/disk1/1.mkv > /dev/null
    
    real    1m19.655s
    user    0m0.128s
    sys     0m22.403s
    
    v6.7rc8
    
    root@Test:~# time cat /mnt/disk1/1.mkv > /dev/null
    
    real    1m19.453s
    user    0m0.112s
    sys     0m24.293s

     

    Edited by johnnie.black
    Link to comment
    57 minutes ago, johnnie.black said:

    Can't reproduce this on my test server with a 20GB file, can you gives us more detail on the hardware used, mainly board/CPU? Just in case I have something similar where I can try.

    Note with thanks, I make some change of disk 1 (disk) and disk 4 (SSD) from BTRFS to XFS, then XFS disk problem gone. But BTRFS disk still slow.

    1.PNG.c71266639f10ce0e3c898869c5c066d6.PNG

     

    Next, I try adjust all tunable stripes from DEFAULT by 10x ( also try 2X have improve ), and problem seems completely gone.

    2.PNG.6c3457cb54775522f051f319852cecfc.PNG

     

    So, for all BTRFS disk system, should change all DEFAULT tunable stripes setting ?

    How LT would handle, which haven't problem on 6.6.7 ?

     

     

     

     

    Edited by Benson
    Link to comment
    29 minutes ago, Benson said:

    Next, I try adjust all tunable stripes from DEFAULT by 10x ( also try 2X have improve ), and problem seems completely gone.

    I was also using btrfs, but I use tunable settings that are considerably higher than defaults, that would explain why I wasn't seeing the problem.

    Link to comment
    15 minutes ago, johnnie.black said:

    I was also using btrfs, but I use tunable settings that are considerably higher than defaults, that would explain why I wasn't seeing the problem.

    Yes, thanks again for your testing.

    Link to comment

    Thank you for the analysis, we'll study this further to see if we can create an auto-tuned setting for that.

    Link to comment
    7 hours ago, limetech said:

    Thank you for the analysis, we'll study this further to see if we can create an auto-tuned setting for that.

    afbff9b9e5b22328dfe2fdbd843dee79dd03456f

    Link to comment


    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.