• Copying speeds with certain hardware


    opentoe
    • Minor

    I do believe there is an inherent bug somewhere that possibly does not like my hardware. I've been using unraid since the beginning of V5 or even before that. When V6 first came out things were good. After a couple of revisions is when these copying issues started. I always would get 100MB/sec on copy speeds using a Windows workstation on the network and even faster using the terminal and midnight commander. My server box is a simple norco 4224 box. I have a 1000 watt power supply which has been replaced. The cables that go from the backplane to the controller cards have been replaced with new ones. I've changed the memory out temporarily with a friend's memory. I've disabled the on-board NIC and purchased an Intel add on card. I've re-seated the processor and ran extensive tests on the processor itself. The mainboard is an Asus X79 that has tried different bioses. I've tried different controllers. Dell PERC 310's and/or the SuperMicro SAS2LP-MV8. I have like 4 of each of these cards. Tried them together and in different combinations. If I did not have so much data, I'd try to go back to V5 and see what performance I would get. Since the copying of files issues happens via the terminal this eliminates pretty much anything outside the server box. I've even taken the mainboard out of the server box and installed it in a small desktop case and threw Windows 10 on it. Problem free. And while I had Windows 10 on it, I ran several heavy duty burn-in test programs just for the heck of it. Flawless. Reinstalled the board back into the 4224 box, booted up just fine. I've also moved my pro key to a new USB flash stick that was vanilla. No plugins installed, plain vanilla. Tried changing some bios settings, but really not that many. I basically turned off all VM related stuff and tried different settings with that. 

     

    In a nutshell when copying files the system would start to copy just fine around 80-100MB/sec. Then eventually slowly drop down to 0MB/sec and the copy would literally stop completely for about 30-40 seconds.  So if I'm copying a large amount of data it could take days to do this. ( Yes, the parity drive has been replaced ) . Takes about 2.5 minutes to copy a 6GB file when it would take seconds before. I don't even attempt to copy amounts of data anymore and I also think this issue is affecting my system as a whole. Specially when watching a movie. During the entire movie it would freeze ( which is where it stops ) then picks up again and the movie catches up again with the player. This happens with my Dune players AND my Roku players. 

     

    Another oddity I did a parity sync which took 2 hours ( great! ) but the average speed indicates 1.1GB/sec.  Is that even possible? Attached a screen shot of that and diagnostics. I'm pretty much lost from here. I did attach a terminal copy that looks very weird, but you'll get the gist of what is happening once you see it. It is just one 6GB file being copied.

     

    Any suggestions or recommendations is highly appreciated. 

    Thanks

     

    parity.jpg

    sun-diagnostics-20190609-0516.zip




    User Feedback

    Recommended Comments

    The first thing you cat try is to go into your Disk Setting and change "Tunable (md_write_method)" to "reconstruct write".

     

    My personal experience and observation is that the array file system drivers are not very well threaded or maybe not that advanced in their caching abilities. They seem to synchronously alternate reads and writes in order to fill and empty the cache. I've not ever observed this type of behavior in other operating systems where a parity of reads/writes is achieved to balance asynchronous read/write performance. Again, this is strictly my personal observation. I don't have any technical knowledge with how the array cache actually functions.

    Link to comment
    22 hours ago, Taddeusz said:

    The first thing you cat try is to go into your Disk Setting and change "Tunable (md_write_method)" to "reconstruct write".

     

    My personal experience and observation is that the array file system drivers are not very well threaded or maybe not that advanced in their caching abilities. They seem to synchronously alternate reads and writes in order to fill and empty the cache. I've not ever observed this type of behavior in other operating systems where a parity of reads/writes is achieved to balance asynchronous read/write performance. Again, this is strictly my personal observation. I don't have any technical knowledge with how the array cache actually functions.

    Prior to having that option in the disk settings I use to have it in my "go" file. I believe it was md_write_method 1. Yea, I've went back and forth with trying AUTO/READ-WRITE-MODIFY/RECONSTRUCT-WRITE with no improvement.

     

    I've kept it on RECONSTRUCT.

    Link to comment
    8 hours ago, opentoe said:

    I've kept it on RECONSTRUCT.

    Yes, set it in disk setting rather in go file.

     

    Your setting "Tunable (nr_requests):" was set to 8 only. Default was 128, I simple set all tunables by 10x.

    If you want got smooth transfer curve, pls try disable "Tunable (enable Direct IO):" at "Global Share Settings page".

     

     

     

    Some comment on your test result

     

    - You have 32GB RAM, so data usuallly cache a lot and queuing write to array. When you stop file transfer, it still in queuing and will affect the observe result, you need waiting for while.

    - You have many disk, so you also need confirm System-HBA-Disk have enough bandwidth, and the test on other Windows machine haven't much meaning, because those access was single disk, not reconstruct write with all disk.

    Edited by Benson
    Link to comment
    On 6/11/2019 at 9:15 AM, Benson said:

    Yes, set it in disk setting rather in go file.

     

    Your setting "Tunable (nr_requests):" was set to 8 only. Default was 128, I simple set all tunables by 10x.

    If you want got smooth transfer curve, pls try disable "Tunable (enable Direct IO):" at "Global Share Settings page".

     

     

     

    Some comment on your test result

     

    - You have 32GB RAM, so data usuallly cache a lot and queuing write to array. When you stop file transfer, it still in queuing and will affect the observe result, you need waiting for while.

    - You have many disk, so you also need confirm System-HBA-Disk have enough bandwidth, and the test on other Windows machine haven't much meaning, because those access was single disk, not reconstruct write with all disk.

    When I performed the terminal copy test using midnight commander I used disk shares and made sure the source disk was not on the destination controller. So there shouldn't have been any issues with one 6GB file. Like I said previously I never had any issues like this. I actually had more disks in the array and three controllers. Thinking it would help by removing a couple of drives and also removing a third controller didn't have any affect at all. I use to get 112MB/sec on large copies, weather or not the files were small or large. I could be copying/moving 2-3 terabytes and consistently get 112MB/sec. I remember this because there was a bug push a while ago when a lot of us were changing the file system from btrfs to xfs and that caused A LOT of file copying. Some copies lasted hours and hours and would never dip down below 100MB/sec during the copy. I have a lot of memory ( 32GB ) for creating some VM's and running dockers. Because of this issue and I'm not sure where it is going to take me I removed all my dockers and removed all my VM's. You'd thik when copying a 6GB file there wouldn't be any caching going on since it could fit the whole file in memory. I don't know if any other users notice issues like this, but when you have been running unraid for such a long time you notice every little thing. I thought I would have this issue settled in a few days or less but since it is going on for several months not much I can do. And I do get fantastic speeds during a parity sync. No dipping in speeds at all, as one time I almost watched the entire check just to make sure, all because I really wanted to get to the bottom of this. It is pretty sad. I deleted weeks and weeks worth of DVD's and movies that I have ripped knowing I can rip them back but not touching one disk until all problems are fixed and box runs like it use to. 

    Link to comment


    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.
    Note: Your post will require moderator approval before it will be visible.

    Guest
    Add a comment...

    ×   Pasted as rich text.   Restore formatting

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.


  • Status Definitions

     

    Open = Under consideration.

     

    Solved = The issue has been resolved.

     

    Solved version = The issue has been resolved in the indicated release version.

     

    Closed = Feedback or opinion better posted on our forum for discussion. Also for reports we cannot reproduce or need more information. In this case just add a comment and we will review it again.

     

    Retest = Please retest in latest release.


    Priority Definitions

     

    Minor = Something not working correctly.

     

    Urgent = Server crash, data loss, or other showstopper.

     

    Annoyance = Doesn't affect functionality but should be fixed.

     

    Other = Announcement or other non-issue.