unraid-tunables-tester.sh - A New Utility to Optimize unRAID md_* Tunables


Recommended Posts

In the past have found my server to be nicely responsive when I'm using something like this:

md_num_stripes=3072

md_write_limit=1024

md_sync_window=1024

Interestingly enough, even at 50% bigger your read allocation is worse than stock unRAID.

I don't see why you'd call it "worse", when it is in fact giving me better performance.  See, sync_window and write_limit in this example are exactly what your tuning test would suggest.  Only on top of that I have the num_stripes value a little larger.  The result is having both fastest possible parity check speeds and better overall responsiveness when reading/writing to the server while a parity check is running.

 

testing of all three tunables would be meaningful, but the methodology for such a test has not been defined.  Since one of the tunables involves writing, the only way to test all three would be to run a parity check, read files, and write files, all simultaneously.  Oh, and you would want to do that while excluding any outside variables like LAN connectivity and desktop performance.  If you want to go ahead and code it

That's an interesting challenge.  I think I'll modify my script to do a few iterations for num_stripes, say at 11%, 15%, and 20% at each pass, while at the same time a controlled and throttled read from the array is taking place.

 

Six pages later we come full circle to post #4 and the link in post #7.  You have an interesting and useful script, but it is only optimizing for maximum parity throughput with no regard for simultaneous array reads or writes (the basic media server functions).  Thus, the script has the potential to cause problems for people wanting to use their media server while parity sync is active after using this script.  That will lead those to run it in the off-hours which for most is already done that way.

 

All said, it is a useful tool especially for those with large arrays where parity sync can take painful amounts of time.  However, I would think to paint a clearer picture for people about the purpose of this script a note or disclaimer should be put in the first post to inform people that this script tries to tweak the the tune-ables while only quantifying one of the three (md_sync_window) and empirically assigning a value for the other two (md_num_stripes being some percent higher than md_sync_window and md_write_limit basically being carried).  I can see some getting an increase in performance all around and not just with parity throughput because they have more RAM than processor.  Testing with simultaneous writes and checking IO waits would definitely make this an all-encompassing tool to provide the best numbers for the tune-ables.  With writes comes the issue of other people's data which is most likely the reason Tom went conservative in the first place (along with the 512MB RAM min requirement).

Link to comment

Here you go Pauven, some data for your hard work.

 

 

Tunables Report from  unRAID Tunables Tester v2.0 by Pauven

 

NOTE: Use the smallest set of values that produce good results. Larger values

      increase server memory use, and may cause stability issues with unRAID,

      especially if you have any add-ons or plug-ins installed.

 

Test | num_stripes | write_limit | sync_window |  Speed

--- FULLY AUTOMATIC TEST PASS 1 (Rough - 20 Sample Points @ 3min Duration)---

  1  |    1408    |    768    |    512    |  84.4 MB/s

  2  |    1536    |    768    |    640    |  82.4 MB/s

  3  |    1664    |    768    |    768    |  84.7 MB/s

  4  |    1920    |    896    |    896    |  84.9 MB/s

  5  |    2176    |    1024    |    1024    |  85.0 MB/s

  6  |    2560    |    1152    |    1152    |  84.8 MB/s

  7  |    2816    |    1280    |    1280    |  84.9 MB/s

  8  |    3072    |    1408    |    1408    |  85.0 MB/s

  9  |    3328    |    1536    |    1536    |  84.9 MB/s

  10  |    3584    |    1664    |    1664    |  85.0 MB/s

  11  |    3968    |    1792    |    1792    |  85.0 MB/s

  12  |    4224    |    1920    |    1920    |  84.4 MB/s

  13  |    4480    |    2048    |    2048    |  84.9 MB/s

  14  |    4736    |    2176    |    2176    |  85.0 MB/s

  15  |    5120    |    2304    |    2304    |  84.8 MB/s

  16  |    5376    |    2432    |    2432    |  85.0 MB/s

  17  |    5632    |    2560    |    2560    |  84.9 MB/s

  18  |    5888    |    2688    |    2688    |  85.0 MB/s

  19  |    6144    |    2816    |    2816    |  85.0 MB/s

  20  |    6528    |    2944    |    2944    |  85.0 MB/s

--- Targeting Fastest Result of md_sync_window 1024 bytes for Medium Pass ---

--- FULLY AUTOMATIC TEST PASS 2 (Final - 16 Sample Points @ 4min Duration)---

  21  |    2008    |    904    |    904    |  84.8 MB/s

  22  |    2024    |    912    |    912    |  84.8 MB/s

  23  |    2040    |    920    |    920    |  84.7 MB/s

  24  |    2056    |    928    |    928    |  84.7 MB/s

  25  |    2080    |    936    |    936    |  82.9 MB/s

  26  |    2096    |    944    |    944    |  84.7 MB/s

  27  |    2112    |    952    |    952    |  84.8 MB/s

  28  |    2128    |    960    |    960    |  84.3 MB/s

  29  |    2144    |    968    |    968    |  84.4 MB/s

  30  |    2168    |    976    |    976    |  84.8 MB/s

  31  |    2184    |    984    |    984    |  84.2 MB/s

  32  |    2200    |    992    |    992    |  84.9 MB/s

  33  |    2216    |    1000    |    1000    |  84.8 MB/s

  34  |    2240    |    1008    |    1008    |  84.7 MB/s

  35  |    2256    |    1016    |    1016    |  84.7 MB/s

  36  |    2272    |    1024    |    1024    |  73.7 MB/s

 

Completed: 2 Hrs 8 Min 12 Sec.

 

Best Bang for the Buck: Test 1 with a speed of 84.4 MB/s

 

    Tunable (md_num_stripes): 1408

    Tunable (md_write_limit): 768

    Tunable (md_sync_window): 512

 

These settings will consume 99MB of RAM on your hardware.

 

 

Unthrottled values for your server came from Test 32 with a speed of 84.9 MB/s

 

    Tunable (md_num_stripes): 2200

    Tunable (md_write_limit): 992

    Tunable (md_sync_window): 992

 

These settings will consume 154MB of RAM on your hardware.

This is 64MB more than your current utilization of 90MB.

NOTE: Adding additional drives will increase memory consumption.

 

In unRAID, go to Settings > Disk Settings to set your chosen parameter values.

 

Link to comment

Wow, I think I need to make some hardware modifications.  I'm not even coming close to the throughput others are reporting.  I ran in full auto mode and the best I could get was 18.7 MB/s.  I've got a quad-core i5 processor with 8GB ram.  My issue must be the HD controller.

 

Best Bang for the Buck: Test 3 with a speed of 18.5 MB/s

 

    Tunable (md_num_stripes): 1664

    Tunable (md_write_limit): 768

    Tunable (md_sync_window): 768

 

Link to comment

Wow, I think I need to make some hardware modifications.  I'm not even coming close to the throughput others are reporting.  I ran in full auto mode and the best I could get was 18.7 MB/s.  I've got a quad-core i5 processor with 8GB ram.  My issue must be the HD controller.

 

Best Bang for the Buck: Test 3 with a speed of 18.5 MB/s

 

    Tunable (md_num_stripes): 1664

    Tunable (md_write_limit): 768

    Tunable (md_sync_window): 768

 

Something clearly needs adjustment -- that's even slower than I'd expect with PCI controllers.    Detail your hardware configuration and what disk drives you're using.    That kind of speed sounds like IDE drives with 40-wire cables !!    But that's VERY unlikely on a system with an i5.

 

Another thought:  Are you running RC16c?  There was an issue with > 4GB of RAM in an earlier release candidate that caused very slow speeds.    If you're not running the latest RC, upgrade to it and run the test again.

 

Link to comment

Ran normal test:

 

Tunables Report from  unRAID Tunables Tester v2.0 by Pauven

 

NOTE: Use the smallest set of values that produce good results. Larger values

      increase server memory use, and may cause stability issues with unRAID,

      especially if you have any add-ons or plug-ins installed.

 

Test | num_stripes | write_limit | sync_window |  Speed

---------------------------------------------------------------------------

  1  |    1280    |    768    |    384    | 108.5 MB/s

  2  |    1408    |    768    |    512    | 118.9 MB/s

  3  |    1536    |    768    |    640    | 125.8 MB/s

  4  |    1664    |    768    |    768    | 127.0 MB/s

  5  |    1920    |    896    |    896    | 127.1 MB/s

  6  |    2176    |    1024    |    1024    | 122.5 MB/s

  7  |    2560    |    1152    |    1152    | 127.6 MB/s

  8  |    2816    |    1280    |    1280    | 127.7 MB/s

  9  |    3072    |    1408    |    1408    | 126.5 MB/s

  10  |    3328    |    1536    |    1536    | 127.5 MB/s

  11  |    3584    |    1664    |    1664    | 127.5 MB/s

  12  |    3968    |    1792    |    1792    | 127.5 MB/s

  13  |    4224    |    1920    |    1920    | 127.5 MB/s

  14  |    4480    |    2048    |    2048    | 127.5 MB/s

 

Completed: 0 Hrs 28 Min 48 Sec.

 

Best Bang for the Buck: Test 4 with a speed of 127.0 MB/s

 

    Tunable (md_num_stripes): 1664

    Tunable (md_write_limit): 768

    Tunable (md_sync_window): 768

 

These settings will consume 149MB of RAM on your hardware.

 

 

Unthrottled values for your server came from Test 8 with a speed of 127.7 MB/s

 

    Tunable (md_num_stripes): 2816

    Tunable (md_write_limit): 1280

    Tunable (md_sync_window): 1280

 

These settings will consume 253MB of RAM on your hardware.

This is 23MB more than your current utilization of 230MB.

NOTE: Adding additional drives will increase memory consumption.

 

In unRAID, go to Settings > Disk Settings to set your chosen parameter values.

Link to comment

Are you running RC16c?  There was an issue with > 4GB of RAM in an earlier release candidate that caused very slow speeds.    If you're not running the latest RC, upgrade to it

That issue was never fixed.  Only a workaround was put in syslinux.cfg that lets you boot in 4GB if your physical RAM is more than that.

 

The 4K boot option was removed after RC14.  The release notes for RC15 noted that a newer kernel was used, and that the 4k option had been removed.    The option hasn't been in syslinux.cfg since, and the issue isn't listed in the Release 5 issue list.    It certainly won't hurt to try adding a mem=4095 limit to your config (or simply remove 4GB) ... but I thought this had been resolved.    If you're running RC14 or earlier, I'd try simply upgrading to RC16c first.

 

Link to comment

The 4K boot option was removed after RC14.  The release notes for RC15 noted that a newer kernel was used, and that the 4k option had been removed.    The option hasn't been in syslinux.cfg since, and the issue isn't listed in the Release 5 issue list.  ... but I thought this had been resolved.

The 4GB option was removed not because the issue was fixed, but because it was a halfassed workaround and it causes a crash if you have less than 4GB physical RAM and you tell the kernel to boot in 4GB RAM.  The issue for the affected systems is still exactly the same with the latest v5rc16c.  Using that boot option is still "the cure" on such machine if it has 4GB-or-more physical RAM in it.

 

 

Link to comment

Using that boot option is still "the cure" on such machine if it has 4GB-or-more physical RAM in it.

 

That boot option is no longer presented -- it was removed in RC15.    Clearly you could still add a mem=4095 parameter line manually, but at least according to limetech this issue was resolved long ago (RC15).

 

This is solved in -rc15 without having to use any kernel parameters.

 

... and the issue is marked as "Solved" in the OS 5.x issue list:  http://lime-technology.com/forum/index.php?topic=27788.0

 

Link to comment

Using that boot option is still "the cure" on such machine if it has 4GB-or-more physical RAM in it.

 

That boot option is no longer presented -- it was removed in RC15.    Clearly you could still add a mem=4095 parameter line manually, but at least according to limetech this issue was resolved long ago (RC15).

 

This is solved in -rc15 without having to use any kernel parameters.

 

... and the issue is marked as "Solved" in the OS 5.x issue list:  http://lime-technology.com/forum/index.php?topic=27788.0

You don't know what you're talking about.  What you're calling solved is when Tom deleted half the posts in that thread, including Barzija's whole account, and called the issue "solved".  Nice way of solving things!  I have two such Supermicro boards, and I can see for myself whether the issue it's solved or not.

 

 

Link to comment

Question - at the end of the test, it states that you should stop & start the array. Would it be wise to request the user to do it prior to testing as well?

 

I ran full-auto twice and received results that jumped all over the place. Only one test scored the same both times, the others had differences as much as 6 mb/sec. Is the later typical?

 

Currently running the Extreme test (10 min) with default selections to see if it gives a bell curve that I expect the results should reflect.

 

Tunables Report from  unRAID Tunables Tester v2.0 by Pauven

NOTE: Use the smallest set of values that produce good results. Larger values
      increase server memory use, and may cause stability issues with unRAID,
      especially if you have any add-ons or plug-ins installed.

Test | num_stripes | write_limit | sync_window |   Speed 
--- FULLY AUTOMATIC TEST PASS 1 (Rough - 20 Sample Points @ 3min Duration)---
   1  |    1408     |     768     |     512     |  71.1 MB/s 
   2  |    1536     |     768     |     640     |  63.8 MB/s 
   3  |    1664     |     768     |     768     |  60.0 MB/s 
   4  |    1920     |     896     |     896     |  61.7 MB/s 
   5  |    2176     |    1024     |    1024     |  56.7 MB/s 
   6  |    2560     |    1152     |    1152     |  58.4 MB/s 
   7  |    2816     |    1280     |    1280     |  58.5 MB/s 
   8  |    3072     |    1408     |    1408     |  56.6 MB/s 
   9  |    3328     |    1536     |    1536     |  54.0 MB/s 
  10  |    3584     |    1664     |    1664     |  55.5 MB/s 
  11  |    3968     |    1792     |    1792     |  56.8 MB/s 
  12  |    4224     |    1920     |    1920     |  51.4 MB/s 
  13  |    4480     |    2048     |    2048     |  56.4 MB/s 
  14  |    4736     |    2176     |    2176     |  54.4 MB/s 
  15  |    5120     |    2304     |    2304     |  54.1 MB/s 
  16  |    5376     |    2432     |    2432     |  53.7 MB/s 
  17  |    5632     |    2560     |    2560     |  49.8 MB/s 
  18  |    5888     |    2688     |    2688     |  54.7 MB/s 
  19  |    6144     |    2816     |    2816     |  52.9 MB/s 
  20  |    6528     |    2944     |    2944     |  51.1 MB/s 
--- Targeting Fastest Result of md_sync_window 512 bytes for Medium Pass ---
--- FULLY AUTOMATIC TEST PASS 2 (Final - 16 Sample Points @ 4min Duration)---
  21  |    1288     |     768     |     392     |  80.0 MB/s 
  22  |    1296     |     768     |     400     |  77.5 MB/s 
  23  |    1304     |     768     |     408     |  75.8 MB/s 
  24  |    1312     |     768     |     416     |  77.9 MB/s 
  25  |    1320     |     768     |     424     |  76.2 MB/s 
  26  |    1328     |     768     |     432     |  74.7 MB/s 
  27  |    1336     |     768     |     440     |  74.4 MB/s 
  28  |    1344     |     768     |     448     |  74.4 MB/s 
  29  |    1360     |     768     |     456     |  74.3 MB/s 
  30  |    1368     |     768     |     464     |  78.3 MB/s 
  31  |    1376     |     768     |     472     |  77.4 MB/s 
  32  |    1384     |     768     |     480     |  74.6 MB/s 
  33  |    1392     |     768     |     488     |  72.7 MB/s 
  34  |    1400     |     768     |     496     |  74.4 MB/s 
  35  |    1408     |     768     |     504     |  74.5 MB/s 
  36  |    1416     |     768     |     512     |  73.8 MB/s 

Completed: 2 Hrs 9 Min 8 Sec.

Best Bang for the Buck: Test 1 with a speed of 71.1 MB/s

     Tunable (md_num_stripes): 1408
     Tunable (md_write_limit): 768
     Tunable (md_sync_window): 512

These settings will consume 55MB of RAM on your hardware.


Unthrottled values for your server came from Test 21 with a speed of 80.0 MB/s

     Tunable (md_num_stripes): 1288
     Tunable (md_write_limit): 768
     Tunable (md_sync_window): 392

These settings will consume 50MB of RAM on your hardware.
This is -20MB less than your current utilization of 70MB.
NOTE: Adding additional drives will increase memory consumption.

In unRAID, go to Settings > Disk Settings to set your chosen parameter values.

Link to comment

Question - at the end of the test, it states that you should stop & start the array. Would it be wise to request the user to do it prior to testing as well?

 

I ran full-auto twice and received results that jumped all over the place. Only one test scored the same both times, the others had differences as much as 6 mb/sec. Is the later typical?

 

Hey jbartlett,

 

It certainly doesn't hurt to do a stop&start before the test.  In all honesty, I haven't seen the rampant memory use I expected from running just these tests, so my recommendation may be overkill.  In some of my earlier manual testing, when I was purposely trying to crash my server by running pre-clears and reads and parity checks simultaneously, my memory use would skyrocket and the server would get slow - a simple stop&start restored normal memory utilization without needing a reboot, so that is why I threw that recommendation in there.

 

Maybe it is just me, but I think I see a bell curve on your hardware, it is just that the peak of your curve is at 384... or perhaps even lower!  I think it would be interesting to run a test below 384 on your hardware - but the current version of the utility won't let you.  If you're interested, I'll make a new version with a lower limit and post it today.  Maybe you can go faster by going lower...

 

-Paul

Link to comment

Ran normal test:

  1  |    1280    |    768    |    384    | 108.5 MB/s

...

  4  |    1664    |    768    |    768    | 127.0 MB/s

...

Best Bang for the Buck: Test 4 with a speed of 127.0 MB/s

 

    Tunable (md_num_stripes): 1664

    Tunable (md_write_limit): 768

    Tunable (md_sync_window): 768

 

These settings will consume 149MB of RAM on your hardware.

 

Hey unevent, those are good looking results.  I wouldn't expect much of an improvement on your server since you are already tuned, but by dropping the values to the Best Bang values, you can free up some memory while keeping performance relatively the same.

 

I know you are concerned about the recommended values being md_sync_window centric, with no regard for the other two metrics, but there actually is regard for the other parameters, just not a method to directly test them in this utility.

 

What is happening behind the scenes is that we are finding how many stripes is needed to allow unthrottled access to your array, and this number is found by running a parity check while tuning the md_sync_window.

 

This same number of stripes, by extension, is also a good value for the md_write_limit, as it should represent the point at which writes are unthrottled.  At this point, admittedly this is just a theory, but it is a very rational theory that simply needs to be proven, or disproven as the case may be.

 

The md_num_stripes represents two things:  the total impact to server memory, as well as any additional overage, beyond syncs and writes, that remain for reads.  I have maintained that same percentage allocation (10%) that stock unRAID allocated.  Is this the best allocation?  Probably not, but if it isn't then neither is the unRAID stock allocation, since they are one and the same. 

 

If anything, I theorize that reads (i.e. watching a Blu-Ray) while running a parity check may be slightly worse on a tuned system for one simple fact:  Since the hard drives are now being accessed at their maximum potential by any one task (be it reads, writes or syncs), they have no spare processing capacity to handle multiple tasks.  It is at this point that the balancing of reads vs writes vs syncs, handled by the mdcmd driver, ultimate impacts your viewing enjoyment of a Blu-Ray while running a parity check.

 

It seems that your idea of tuning for playback enjoyment is to starve the parity check, leaving excess processing capacity on the hard drives so that reads have no competition.  While that is a valid proposition, I don't think purposely de-tuning one process to get performance on another process is the greatest of solutions.  I think your assertion that Tom de-tuned parity checks on purpose is conjecture, as we can see in many of these test results that optimum performance comes from unRAID stock values.  If anything, Tom chose nice default values that work great on many servers, and work decent on all servers, while being sized for the smallest of servers.

 

I would be interested to know how Tom prioritizes reads vs. writes vs. syncs:  is this something hard coded into mdcmd, or is it controlled by the allocation % of these three tunable parameters.  Perhaps someone would care to peruse the mdcmd.c source file and see if they can see a methodology. 

 

If it turns out that prioritization is controlled by these three parameters, since I never allocate more than 45% to syncs or writes, if you are running a parity check while playing a Blu-Ray, the Blu-Ray is getting 55% of the throughput, while the parity check gets 45%.  And if you have a server that can deliver 100 MB/s, then in theory the Blu-Ray should be getting more than enough throughput at 55 MB/s theoretical maximum, since Blu-Rays only need about 5 MB/s.  Of course, having hard drives thrashing about trying to do two things at once is never optimal, so theoretical maximum would never be reached.

 

Obviously we've [all?] experienced streaming issues while running a parity check.  As my 'back of the napkin' calculations show above, the cause isn't readily apparent, and we need more insight into the inner workings, and/or more tests to evaluate more combinations of factors.  You've proposed some interesting sounding tests, but I'm not sure how you would go about building those tests.  Primarily, I've never even see the number of IO's exposed in unRAID.  How do you test something you can't even see?

 

We also shouldn't be lulled into thinking that these three tunables are the only factors at play - these are just what Tom gave us to play with.  NCQ may affect playback quality while running parity checks.  So may cache memory parameters, available memory, driver revisions, etc.

 

Any additional tests will probably have to be coded by someone else.  This was primarily a utility for myself, and a nice little programming challenge to fill up some spare time.  Since I don't really have any more spare time, nor a desire for anything more, I think it is time for someone else to take up the candle.  And many unRAID users, like Patilan, make me look like a newb when it comes to Bash programming... and rightly so.  They would be better choices moving forward.

 

As far as making a statement in the initial post that this utility has no regard for the md_num_stripes and md_write_limit parameters, I will make no such statement, since I have clearly put consideration into how all three values are set.  I freely described my methodology, and everyone is free to use or ignore this utility as they feel want to do.

 

Link to comment

Wow, I think I need to make some hardware modifications.  I'm not even coming close to the throughput others are reporting.  I ran in full auto mode and the best I could get was 18.7 MB/s.  I've got a quad-core i5 processor with 8GB ram.  My issue must be the HD controller.

 

Best Bang for the Buck: Test 3 with a speed of 18.5 MB/s

 

    Tunable (md_num_stripes): 1664

    Tunable (md_write_limit): 768

    Tunable (md_sync_window): 768

 

Something clearly needs adjustment -- that's even slower than I'd expect with PCI controllers.    Detail your hardware configuration and what disk drives you're using.    That kind of speed sounds like IDE drives with 40-wire cables !!    But that's VERY unlikely on a system with an i5.

 

Another thought:  Are you running RC16c?  There was an issue with > 4GB of RAM in an earlier release candidate that caused very slow speeds.    If you're not running the latest RC, upgrade to it and run the test again.

 

Sorry, I forgot to mention that I am running RC16c.  I'm guessing that the PCI SATA controller card is my issue  Full Details of my hardware config include:

 

SUPERMICRO AOC-SAT2-MV8 64-bit PCI-X133MHz SATA II (3.0Gb/s) Controller Card

http://www.newegg.com/Product/Product.aspx?Item=N82E16815121009

 

MSI Z87-G43 LGA 1150 Intel Z87 HDMI SATA 6Gb/s USB 3.0 ATX High Performance CF Gaming Intel Motherboard

http://www.newegg.com/Product/Product.aspx?Item=N82E16813130694

 

Intel Core i5-4670K Haswell 3.4GHz LGA 1150 84W Quad-Core Desktop Processor Intel HD Graphics BX80646I54670K

http://www.newegg.com/Product/Product.aspx?Item=N82E16819116899

 

Patriot Signature 8GB (2 x 4GB) 240-Pin DDR3 SDRAM DDR3 1600 (PC3 12800) Desktop Memory with heatshield Model PSD38G1600KH

http://www.newegg.com/Product/Product.aspx?Item=N82E16820220570

 

Other than that, I have an 11-drive 15.5TB system supported with a 4TB Parity drive.  The drives include the following models from Hitachi, Seagate & WD:

 

HDS5C4040ALE630

ST3000DM001

WD10EACS

WD10EAVS

WD10EADS

WD15EARS

WD20EARS

WD30EZRX

 

I've got 6 drives cabled to the MB and 5 drives cabled to the SUPERMICRO controller.

Link to comment

Other than that, I have an 11-drive 15.5TB system supported with a 4TB Parity drive.  The drives include the following models from Hitachi, Seagate & WD:

 

HDS5C4040ALE630

ST3000DM001

WD10EACS

WD10EAVS

WD10EADS

WD15EARS

WD20EARS

WD30EZRX

 

I've got 6 drives cabled to the MB and 5 drives cabled to the SUPERMICRO controller.

 

JarDo, could you be more specific on the hard drives:  how many of each, what size & RPM they are, and which controller each one is connected to?

Link to comment

I had an array filled with these WD10EACS.  Slowest 1TB drives I've owned.

They were fine for storing and reading movies, but anything else was painfully slow.

 

Couple these with the PCI card and that's a bottleneck(PCI) and a half(slower drives).

 

The maximum speed you can achieve with that PCI card is 133MB/s.  266MB/s if on PCI 2.0.

You would benefit greatly from a PCIe controller.

 

Link to comment

Yikes!  JarDo, if I'm reading your build correctly, you've installed a PCI-X hd controller card into a motherboard that doesn't have any PCI-X slots.  That means you've installed it into a regular PCI slot, with dramatically limited bandwidth!

 

The theoretical maximum amount of data exchanged between the processor and peripherals with PCI-X is 1.06 GB/s, compared to 133 MB/s with standard PCI

 

Since you have 5 drives on the controller card, which is being limited to 133 MB/s by the PCI slot it is installed in, you are getting 133/5=26.6 MB/s, maximum!  After accounting for overhead and such, your 18.5 MB/s number makes perfect sense!

 

Replace that PCI-X card with a proper PCIe (PCI-Express) card, which your motherboard supports!

 

-Paul

Link to comment
Maybe it is just me, but I think I see a bell curve on your hardware, it is just that the peak of your curve is at 384... or perhaps even lower!  I think it would be interesting to run a test below 384 on your hardware - but the current version of the utility won't let you.  If you're interested, I'll make a new version with a lower limit and post it today.  Maybe you can go faster by going lower...

 

That would be great if you could add support for lower ranges. I have a cheap 32GB SSD in my array that I use for caching purposes and while it's sustained read rating shouldn't have any affect, it clearly does. Running the 10 minute test which took my array to 1% on the parity check and well beyond the 32GB range, I saw much smoother results. I ran another test continuing the high range and it hovered around a .7 MB span.

 

-John

 

Tunables Report from  unRAID Tunables Tester v2.0 by Pauven

NOTE: Use the smallest set of values that produce good results. Larger values
      increase server memory use, and may cause stability issues with unRAID,
      especially if you have any add-ons or plug-ins installed.

Test | num_stripes | write_limit | sync_window |   Speed 
---------------------------------------------------------------------------
   1  |    1280     |     768     |     384     |  80.2 MB/s 
   2  |    1408     |     768     |     512     |  72.4 MB/s 
   3  |    1536     |     768     |     640     |  65.6 MB/s 
   4  |    1664     |     768     |     768     |  60.6 MB/s 
   5  |    1920     |     896     |     896     |  59.1 MB/s 
   6  |    2176     |    1024     |    1024     |  56.3 MB/s 
   7  |    2560     |    1152     |    1152     |  55.0 MB/s 
   8  |    2816     |    1280     |    1280     |  56.7 MB/s 
   9  |    3072     |    1408     |    1408     |  54.6 MB/s 
  10  |    3328     |    1536     |    1536     |  58.0 MB/s 
  11  |    3584     |    1664     |    1664     |  54.0 MB/s 
  12  |    3968     |    1792     |    1792     |  54.5 MB/s 
  13  |    4224     |    1920     |    1920     |  54.5 MB/s 
  14  |    4480     |    2048     |    2048     |  51.4 MB/s 

Completed: 2 Hrs 26 Min 23 Sec.

Best Bang for the Buck: Test 1 with a speed of 80.2 MB/s

     Tunable (md_num_stripes): 1280
     Tunable (md_write_limit): 768
     Tunable (md_sync_window): 384

These settings will consume 50MB of RAM on your hardware.


Unthrottled values for your server came from Test 1 with a speed of 80.2 MB/s

     Tunable (md_num_stripes): 1280
     Tunable (md_write_limit): 768
     Tunable (md_sync_window): 384

These settings will consume 50MB of RAM on your hardware.
This is -20MB less than your current utilization of 70MB.
NOTE: Adding additional drives will increase memory consumption.

In unRAID, go to Settings > Disk Settings to set your chosen parameter values.

Link to comment

That would be great if you could add support for lower ranges. I have a cheap 32GB SSD in my array that I use for caching purposes and while it's sustained read rating shouldn't have any affect, it clearly does. Running the 10 minute test which took my array to 1% on the parity check and well beyond the 32GB range, I saw much smoother results. I ran another test continuing the high range and it hovered around a .7 MB span.

 

-John

 

Hey John,

 

I could be wrong, but I don't think the Cache Drive has any impact on Parity Checks, so it wouldn't influence these test results.  Longer tests just naturally provide smoother results.

 

I just posted v2.1 with support down to 128 bytes.  You'll have to use the (M)anual option to access it.

 

Here's my results for comparison:

 

Test | num_stripes | write_limit | sync_window |   Speed 
---------------------------------------------------------------------------
   1  |    992     |     768     |     128     |  20.1 MB/s 
   2  |    1024     |     768     |     160     |  21.7 MB/s 
   3  |    1056     |     768     |     192     |  24.8 MB/s 
   4  |    1088     |     768     |     224     |  32.3 MB/s 
   5  |    1120     |     768     |     256     |  36.5 MB/s 
   6  |    1152     |     768     |     288     |  45.4 MB/s 
   7  |    1184     |     768     |     320     |  55.5 MB/s 
   8  |    1216     |     768     |     352     |  61.4 MB/s 
   9  |    1280     |     768     |     384     |  74.3 MB/s 

Link to comment

I have updated the utility to version 2.1 (see the main post).  The changes are very minor, so feel free to skip this version unless you need one of the two changes.

 

New Features in v2.1:

  • Added support for starting position byte values down to 128 bytes (previously 384).  You will have to select (M)anual on the Starting Position Override screen, then you can manually enter the starting value.  You can also select down to 128 on the Ending Position Override screen too.
  • Fixed a typo on the FULLAUTO option on the Test Type screen - it now correctly indicates the test will take 2.1 hours.

 

You can download the file from post #1:  http://lime-technology.com/forum/index.php?topic=29009.msg259087#msg259087

 

-Paul

 

 

Link to comment

Other than that, I have an 11-drive 15.5TB system supported with a 4TB Parity drive.  The drives include the following models from Hitachi, Seagate & WD:

 

HDS5C4040ALE630

ST3000DM001

WD10EACS

WD10EAVS

WD10EADS

WD15EARS

WD20EARS

WD30EZRX

 

I've got 6 drives cabled to the MB and 5 drives cabled to the SUPERMICRO controller.

 

JarDo, could you be more specific on the hard drives:  how many of each, what size & RPM they are, and which controller each one is connected to?

 

Here are the details of my drives.  Slot 8 is unassigned.

 

AssignmentMfg DateFirmwareModelCapacityRPMCache (MB)Interface

Parity?/?/2012MPAOA3B0Hitachi HDS5C4040ALE6304TB5,40032MB SATA1

Disk112/29/200980.00A80WDC_WD20EARS-00S8B12TB5,40064MB SATA2

Disk22/21/200801.01B01WDC_WD10EACS-00ZJB01TB5,40016MB SATA3

Disk34/16/200801.01B01WDC_WD10EACS-00ZJB01TB5,40016SAT2-MV8_0

Disk41/13/2012CC4CST3000DM001-9YN1663TB7,20064MB SATA4

Disk51/23/200901.01A01WDC_WD10EAVS-00D7B11TB5,4008SATA2-MV8_1

Disk612/1/200801.01B01WDC_WD10EACS-14ZJB01TB5,40016SATA2-MV8_2

Disk71/19/200801.01A01WDC_WD10EAVS-00D7B11TB5,4008SATA2-MV8_3

Disk912/24/200980.00A80WDC_WD15EARS-00Z5B11.5TB5,40064MB SATA5

Disk1012/7/201280.00A80WDC_WD30EZRX-00DC0B03TB5,40064MB SATA6

Disk111/6/200901.01A01WDC_WD10EADS-00L5B11TB5,40032SATA2-MV8_5

 

 

Link to comment

 

Replace that PCI-X card with a proper PCIe (PCI-Express) card, which your motherboard supports!

 

-Paul

 

Makes good sense.  Thank you very much for the complete explanation.  Now, I have to find a 8-port PCIe card to replace my 8-port PCI card.  Does anyone listening to this thread have a suggestion they can offer?

Link to comment

I could be wrong, but I don't think the Cache Drive has any impact on Parity Checks, so it wouldn't influence these test results.  Longer tests just naturally provide smoother results.

The drive itself isn't the unraid cache drive, it just holds cache data for my xbmc clients so there's no spin-up delay. Off to run 2.1, even before upgrading to 5.0 final! ;)

Link to comment

I'm guessing that the PCI SATA controller card is my issue

 

Absolutely !!    You've got 5 drives connected to a PCI interface card !!    Note that although your card supports a much higher bandwidth (since it's PCI-X), you're using it in a PCI slot, so it's throttled back to PCI speed ... and you're sharing it with 5 drives !!

 

And since parity check speed is limited by the slowest drive, those drives on the PCI controller are killing your speed.

 

Replace your AOC-SAT2-MV8 with the PCIe version (SAS2LP-MV8) and you'll see a BIG improvement !!

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.