• [6.7.x] Very slow array concurrent performance


    JorgeB
    • Solved Urgent

    Since I can remember Unraid has never been great at simultaneous array disk performance, but it was pretty acceptable, since v6.7 there have been various users complaining for example of very poor performance when running the mover and trying to stream a movie.

     

    I noticed this myself yesterday when I couldn't even start watching an SD video using Kodi just because there were writes going on to a different array disk, and this server doesn't even have a parity drive, so did a quick test on my test server and the problem is easily reproducible and started with the first v6.7 release candidate, rc1.

     

    How to reproduce:

     

    -Server just needs 2 assigned array data devices (no parity needed, but same happens with parity) and one cache device, no encryption, all devices are btrfs formatted

    -Used cp to copy a few video files from cache to disk2

    -While cp is going on tried to stream a movie from disk1, took a long time to start and would keep stalling/buffering

     

    Tried to copy one file from disk1 (still while cp is going one on disk2), with V6.6.7:

     

    2083897607_Screenshot2019-08-0511_58_06.png.520373133cc121c80a361538a5fcc99b.png

     

    with v6.7rc1:

     

    856181720_Screenshot2019-08-0511_54_15.png.310bce8dbd6ed80d11d97727de55ac14.png

     

    A few times transfer will go higher for a couple of seconds but most times it's at a few KB/s or completely stalled.

     

    Also tried with all unencrypted xfs formatted devices and it was the same:

     

    1954593604_Screenshot2019-08-0512_21_37.png.6fb39b088e6cc77d99e45b37ea3184d8.png

     

    Server where problem was detected and test server have no hardware in common, one is based on X11 Supermicro board, test server is X9 series, server using HDDs, test server using SSDs so very unlikely to be hardware related.

    • Like 1
    • Upvote 22



    User Feedback

    Recommended Comments



    10 minutes ago, itimpi said:

    I am reasonably certain that the issue HAS been acknowledged by Limetech!   We have definitely been told it is on the list of issues that are hoping to be cleared for the 6.8 rc1 release. (which is hopefully going to be made available soon).

     

    I suspect that any solution is non-trivial and is somewhere at the Linux driver level which means it can be difficult to get resolved in a way that has no side-effects elsewhere.

    If you can find evidence of that, I'll believe you.  And it would make the remainder of the people on this list happy.  But to my eyes, no it hasn't, only through a third party.  Quite poor form in my opinion.

    Link to comment
    5 hours ago, Marshalleq said:

    If you can find evidence of that, I'll believe you.  And it would make the remainder of the people on this list happy.  But to my eyes, no it hasn't, only through a third party.  Quite poor form in my opinion.

    For what it's worth. I can confirm it is on the todo list of Limetech and actively looking for a solution.

    image.png

    Link to comment

    I've also run into these issues on my system. Glad to see it is being worked on. My performance on 6.7.x is much, much worse than on 6.6.x

    Link to comment
    5 hours ago, bonienl said:

    For what it's worth. I can confirm it is on the todo list of Limetech and actively looking for a solution.

    image.png

    Thanks.  Seems like it was only added three weeks ago though - that's disappointing.

    Edited by Marshalleq
    Link to comment
    21 minutes ago, Kaveh said:

    Has anyone tried the 6.7.3 release candidates?  It looks like a new kernel and md/unraid version in rc3.

    It’s certainly not fixed in rc1 or rc2. 

    Link to comment

    rc3 just came out.  Changelog below.  It does mention a new version of md/unraid  and a kernel as per notes below.  I've installed it - no idea if this is addressing this fix though.

     

    Version 6.7.3-rc3 2019-09-17

    (Changes vs. 6.7.2)

    Linux kernel:

    version: 4.19.73

    md/unraid: version 2.9.8

     

    Version 6.7.2 2019-06-25

    Base distro:

    kernel-firmware: version 20190620_7ae3a09

    php version: 7.2.19 (restore sqlite support)

    sqlite version: 3.28.0

    Linux kernel:

    version: 4.19.56

    Link to comment
    On 9/12/2019 at 2:56 PM, yendi said:

    Is there any ETA for next version? It's getting out of control, my server is simply unusable.

    This is simply by running the mover or copying a file. PLEASE RELEASE AN EMERGENCY FIX.

    And no, I will not downgrade and risk other side effects, i'm in a production env.

    image.png.860dad5e85517c4fde898e707ce77729.png

    Just curious - what is this graph from?

    Link to comment

    RC4?  I only just saw RC3.  There's no RC4 showing up for me....

     

    I assume you mean rc3 - it does seem a lot snappier to me.  Don't know about the disk issues yet - but I am hopeful! :D  Will have to do some testing.

    Edited by Marshalleq
    Link to comment
    24 minutes ago, Marshalleq said:

    RC4?  I only just saw RC3.  There's no RC4 showing up for me....

     

    I assume you mean rc3 - it does seem a lot snappier to me.  Don't know about the disk issues yet - but I am hopeful! :D  Will have to do some testing.

    I am only seeing rc3 as well when checking for updates, but Tom mentioned in another thread that rc3 did not fix the issue but rc4 might. 

    Link to comment

    Do you know if Limetech have been able to replicate the slow array issue? Seems that they’ve had real trouble replicating the SQL issue, which makes it tough for them to fix it.  If they can replicate the slow array then there’s hope that they’ll be able to fix it. Seems like it’s easier to reproduce and more folks are seeing it which hopefully makes it easier to track down. If not then we are stuck on 6.6.7 for the foreseeable future. 

    Edited by Lignumaqua
    Link to comment
    6 minutes ago, Lignumaqua said:

    Do you know if Limetech have been able to replicate the slow array issue?

    Not directly but would think so since this issue is very easy to reproduce, unlike the SQL corruption problem.

    Link to comment
    28 minutes ago, Lignumaqua said:

    Do you know if Limetech have been able to replicate the slow array issue?

    Yes.  -rc4 doesn't fix the issue, during testing was fooled by caching.  Thought I accounted for that but it was late at night.  The Linux block layer has undergone significant changes in last few releases, and I've had to do a lot of re-learnin'.

    • Like 1
    Link to comment
    2 hours ago, GHunter said:

    May you don't see RC4 unless you have RC3 installed?? I have RC3 installed and RC4 is showing as available.

    I had rc3 installed and couldn’t see it. Maybe there’s a schedule when it refreshes available

    updates. I thought rc3 was new, that showed up for me only yesterday. 

     

    Edit - It's shown up for me this morning - so something must introduce a delay.

    Edited by Marshalleq
    Link to comment

    I want to add my hat into the list of people experiencing this issue with 6.7.x. I have run down what seems like every rabbit hole trying to fix the problem thinking it was related to a CPU pinning or isolation issue. I wish I had found this post sooner. There should be a list of known ongoing problems linked to the Unraid dashboard.

     

    I know Unraid runs very barebones as a company, but this seems like an urgent issue. Indeed, Unraid is not that expensive, but I have come to rely on it more and more. I think it should be priced higher and even have an ongoing monthly fee to fund a higher level of development work.

     

    Therefore, also mark me down as someone willing to pay for a higher level of support and development and premium features. Unraid was born out of a group of people who wanted something better and now is going more mainstream to less technically savvy people. This issue has been extremely troublesome to me, and I have spent so much time on it that I will donate $100 to whoever proves they find the fix.

    Link to comment
    1 hour ago, Michael Woodson said:

    I will donate $100 to whoever proves they find the fix.

     

    I found it! It's in 6.8. 🤡

    9 minutes ago, limetech said:

    This is fixed in upcoming 6.8 release.

     

    • Like 2
    • Haha 3
    Link to comment



    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.