Jump to content
  • [6.7.x] Very slow array concurrent performance


    johnnie.black
    • Solved Urgent

    Since I can remember Unraid has never been great at simultaneous array disk performance, but it was pretty acceptable, since v6.7 there have been various users complaining for example of very poor performance when running the mover and trying to stream a movie.

     

    I noticed this myself yesterday when I couldn't even start watching an SD video using Kodi just because there were writes going on to a different array disk, and this server doesn't even have a parity drive, so did a quick test on my test server and the problem is easily reproducible and started with the first v6.7 release candidate, rc1.

     

    How to reproduce:

     

    -Server just needs 2 assigned array data devices (no parity needed, but same happens with parity) and one cache device, no encryption, all devices are btrfs formatted

    -Used cp to copy a few video files from cache to disk2

    -While cp is going on tried to stream a movie from disk1, took a long time to start and would keep stalling/buffering

     

    Tried to copy one file from disk1 (still while cp is going one on disk2), with V6.6.7:

     

    2083897607_Screenshot2019-08-0511_58_06.png.520373133cc121c80a361538a5fcc99b.png

     

    with v6.7rc1:

     

    856181720_Screenshot2019-08-0511_54_15.png.310bce8dbd6ed80d11d97727de55ac14.png

     

    A few times transfer will go higher for a couple of seconds but most times it's at a few KB/s or completely stalled.

     

    Also tried with all unencrypted xfs formatted devices and it was the same:

     

    1954593604_Screenshot2019-08-0512_21_37.png.6fb39b088e6cc77d99e45b37ea3184d8.png

     

    Server where problem was detected and test server have no hardware in common, one is based on X11 Supermicro board, test server is X9 series, server using HDDs, test server using SSDs so very unlikely to be hardware related.

    • Like 1


    User Feedback

    Recommended Comments



    16 minutes ago, Benson said:

    Unraid 6.8?

    Unraid 6.8 is a kernel version?

    -just kidding, as always expect news soon(tm)

    • Like 1

    Share this comment


    Link to comment
    Share on other sites

    Another confirmation here that I see this behaviour on 6.7.X. Downgrade back to 6.6.7 and all my speed issues are resolved.

    Edited by Outcasst

    Share this comment


    Link to comment
    Share on other sites
    5 hours ago, bonienl said:

    I guess hardware plays a role too.

    I have pretty fast drives and with reconstruct write on, I do get satisfying results.

    Below a 12GB file copied from my PC (nvme disk) directly to the array over a 10G connection.

    It starts of at 1 GBps (10 Gbps) until the RAM caching is full and continues writing at around 140 MBps (1.12 Gbps)

    Just to highlight, because you're staff/limetech/closely associated, what you mention aboove ISN'T what this thread is about, its a side discussion.

     

    The issue here is that under 6.7, if I am streaming a movie and on another PC initiate a large file copy, the whole system comes to a grinding halt.  Video stops, copy slows, it's unworkable.

     

    From my tests when I reach the 2.5 - 3Gb mark of a copy/transfer, the freeze kicks in.

     

    It would be handy if LT could acknowledge the thread and issue.

    Share this comment


    Link to comment
    Share on other sites

    Nowhere did I ignore or dismiss the issue reported by johnnie.black.

    I reacted to the statement that Unraid is ALWAYS slow, which I disagree and hence my example.

     

    Just to confirm the issue is still present. When a cache to array transfer kicks in, it brings an existing stream down to a craw (see picture below where speed plummets to a few KB).

     

    image.png.608d594536e506363ae0b425702da50d.png

     

    Pretty sure LT is aware and looking for solutions, please see this statement.

     

    Share this comment


    Link to comment
    Share on other sites
    22 minutes ago, bonienl said:

    Nowhere did I ignore or dismiss the issue reported by johnnie.black.

    I reacted to the statement that Unraid is ALWAYS slow, which I disagree and hence my example.

     

    Pretty sure LT is aware and looking for solutions, please see this statement.

    All good.  I wasn't accusing you of ignoring, just wanted to make sure the core issue  is understood.

    Yeah, I read that and understand the philosophy, but I wonder if just a simple "Thanks, we've seen the issue, we'll come back when we have something or need more info" would help qwell the fears.  Or even a thread label like "Investigating" or ""Accepted", then they can stay away from posting.

     

    Anyway, that's all for another thread.

    Edited by dalben

    Share this comment


    Link to comment
    Share on other sites

    I am also experiencing alot of these issues.

     

    How easy and safe is it downgrade?

    Share this comment


    Link to comment
    Share on other sites

    Just chiming in to say I am experiencing these issues as well.  Hopefully there is a fix soon.  Not being able to use mover is making things really difficult.

    Share this comment


    Link to comment
    Share on other sites
    On 8/26/2019 at 2:01 AM, dalben said:

    All good.  I wasn't accusing you of ignoring, just wanted to make sure the core issue  is understood.

    Yeah, I read that and understand the philosophy, but I wonder if just a simple "Thanks, we've seen the issue, we'll come back when we have something or need more info" would help qwell the fears.  Or even a thread label like "Investigating" or ""Accepted", then they can stay away from posting.

     

    Anyway, that's all for another thread.

    That's a fair point:  a secondary tag of "accepted" and then "closed" (when appropriate) would alleviate a lot of anxiety.

     

    Share this comment


    Link to comment
    Share on other sites

    I believe I'm experiencing this too. While on 6.7.2, when the mover runs, I had some big performance issues - notably with streaming things from an array drive on my LAN with Plex, but I also saw slowdowns with things like rsync. I am not sure what version this behavior started, because my mover generally only runs at 2am, however I transferred a bunch of data to my cache drive during the day recently, and manually started the mover and I noticed the issue (and so did my wife - "what's wrong with Plex?!").

     

    I've not done any formal troubleshooting, but I did revert back to 6.6.7 and haven't experienced the problem again. I might have some time this week to experiment a bit more formally.

    Edited by ClunkClunk

    Share this comment


    Link to comment
    Share on other sites
    5 hours ago, modsiw said:

    Also having this issue on 6.7.2 .

     

    I'm also having an oddity with busybox iostat double reporting a device. Logs: https://pastebin.com/raw/eBateMPw

     

    Diagnostics attached.

    unraid-diagnostics-20190827-1141.zip 161.96 kB · 0 downloads

    Downgraded to 6.6.7 . As expected this fixed the iowait %.

     

    After the downgrade, iostat is now reporting devices correctly. Perhaps that's a clue for root cause analysis.

    Share this comment


    Link to comment
    Share on other sites

    So no news or indication this is being looked at by LT yet.  Does anyone know if it's being looked at or addressed in 6.8?

     

    This really is a show stopper.  My server's #1 purpose for most of the family is to stream media.  If that streaming gets interrupted because radarr or medusa has found a new file we wanted then that's isn't good

    Share this comment


    Link to comment
    Share on other sites

    There is an "internal" list of items/issues which need to be addressed for Unraid 6.8. This is one of them.

     

    Share this comment


    Link to comment
    Share on other sites

    Thanks guys, it's reassuring to know it's being looked at.

    Share this comment


    Link to comment
    Share on other sites

    I just updated a docker on my server and it ground to a halt and took forever to install the update.  After that, my iowait remained very high even after the docker was done installing.   I had to reboot my server to get it to come back down and become usable again.  This is really not good.  The strange thing is that it's fine for a while after a reboot.

     

    craigr

    Share this comment


    Link to comment
    Share on other sites

     You can add me to the list of people incapable of using the latest version of unraid. Due to extremely low simultaneous read bandwidth from the array. Multiple users trying to access data are get halting, And extremely low read and write rates (were talking low KB/s).

    Share this comment


    Link to comment
    Share on other sites

    Think this issue affects me as well. Still trying out Unraid with a trial license, and have been disappointed with the performance. If I'm migrating files from my old NAS to the Unraid, everything else just slows down immensely.

    Share this comment


    Link to comment
    Share on other sites

    Does anyone have a sense whether 6.8 will fix this? I am considering downgrading to 6.6.7 as everything freezes during mover (I moved a bunch of mission critical services to an older PC for now), but I started off on 6.7.x, so I would need to downgrade manually and am afraid I'll break something eg plugins/settings.

    Share this comment


    Link to comment
    Share on other sites

    I'm having this same issue as well. While moving anything to the disk array (IE with the mover) trying to read from the array is a futile effort as it slows down to an unusable degree. Reverted to 6.6.7 and things seem to work much better, with the array usable again while the mover is running.

    Share this comment


    Link to comment
    Share on other sites
    11 hours ago, golli53 said:

    , so I would need to downgrade manually and am afraid I'll break something eg plugins/settings.

    Should be fine, and you can always upgrade back to v6.7 if needed, LT is trying to fix this issue for v6.8rc1, which will hopefully be released soon will be released soon™

     

     

    Edited by johnnie.black

    Share this comment


    Link to comment
    Share on other sites
    12 hours ago, Ancan said:

    Think this issue affects me as well. Still trying out Unraid with a trial license, and have been disappointed with the performance. If I'm migrating files from my old NAS to the Unraid, everything else just slows down immensely.

    Downgraded to 6.6.7 and things are still slow, but I think I might be just hitting the limit for Unraid write-speeds at around 40MBps. Too bad everything else slows down also. Would be nice having some counters about disk troughput, queue length and such in the GUI.

     

     

    Share this comment


    Link to comment
    Share on other sites
    3 minutes ago, Ancan said:

    but I think I might be just hitting the limit for Unraid write-speeds at around 40MBps.

    That's about normal for default write mode, you can try turbo write.

    Share this comment


    Link to comment
    Share on other sites
    1 hour ago, Ancan said:

    Would be nice having some counters about disk troughput

    In the Main page, click on the blue icon at top right to toggle between display of disk counters or disk throughput

    Share this comment


    Link to comment
    Share on other sites
    22 minutes ago, johnnie.black said:

    That's about normal for default write mode, you can try turbo write.

    Thanks, that improved a bit. But everything else still slows down drastically whenever I'm transferring files. Often can't reach Plex web interface, listing dockers in Unraid just gives me the bouncy wave animation, sometimes the CPU load bars just sit at 0% all of them. On a Ryzen 2600 with 32GB RAM and still far from saturating 1Gb network.

    Share this comment


    Link to comment
    Share on other sites



    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.