unraid-tunables-tester.sh - A New Utility to Optimize unRAID md_* Tunables

[Pauven]

Hey Squid, thanks for the info.  That makes sense.  It also makes it seem unwise for me to attempt testing nr_requests, especially on multi-drive setups.

 

However, I had already coded and tested it before you posted your reply, so take a gander at these results:

 

Test 1 - nr_req=128 sync_thresh=2815 - 18.109 GB in 239.968 secs =  77.3 MB/s
Test 2 - nr_req=128 sync_thresh=1408 - 18.571 GB in 239.959 secs =  79.3 MB/s
Test 3 - nr_req=  8 sync_thresh=2815 - 29.960 GB in 240.030 secs = 127.8 MB/s
Test 4 - nr_req=  8 sync_thresh=1408 - 28.717 GB in 240.006 secs = 122.5 MB/s

 

Here's what is interesting about those results:  With all WD Red 3TB drives, on my HighPoint 2760A controller, I was anticipating that nr_request at a stock value of 128, and md_sync_thresh at a 50% value (1408 in this case) was going to be the fastest.

 

Not only was the opposite true, 127.8 is easily the fastest time I have seen in 2 days of testing.  This is doubly (is that a word?) interesting since the md_sync_window was set to 2816 during this test, which is my old optimized setting under 5.0, not the new 760 value that gives me 127.2 on 6.2 (with nr_requests=128 and md_sync_thresh=192 stock values).

 

Now the flaw in my earlier tests was that I was leaving md_sync_thresh at the stock value, 192, regardless of what I set md_sync_window to for each test.

 

I'll have to modify my code and test again, setting md_sync_window to a stock 384 before evaluating the md_sync_thresh at 383 and 192, to figure out if 50% or 99% is the right solution for md_sync_thresh, then use that ratio for all of the md_sync_window tests.

 

-Paul

[BRiT]

I'll have to modify my code and test again, setting md_sync_window to a stock 384 before evaluating the md_sync_thresh at 383 and 192, to figure out if 50% or 99% is the right solution for md_sync_thresh, then use that ratio for all of the md_sync_window tests.

 

Those settings will be different for each user based on their sata controller drives and drives. I hope you mean you will need to have the tuner do that series of tests before it narrows things down before further tuning the other settings, and not that you will hard-code the tuner to use those settings.

[Pauven]

Hey BRiT,

 

I'm not sure what I mean... yet.  I was thinking that, in order to consistently determine if md_sync_thresh should be at -1 or 50%, and nr_requests at 8 or 128, that I should set the server to default unRAID values for the purpose of testing those two parameters.  This would be temporary.

 

But I just ran a test this way, and I got very different results than my previous test.  Now, keep in mind that md_sync_window at 384 is not optimized for my server, so the numbers should be lower than my post above.  That said, the results of this test didn't match my previous test, with md_sync_window at 2816.

 

Test 1 - nr_req=128 sync_thresh=383 - 12.255 GB in 240.802 secs =  52.1 MB/s
Test 2 - nr_req=128 sync_thresh=192 - 26.474 GB in 241.087 secs = 112.4 MB/s
Test 3 - nr_req=  8 sync_thresh=383 - 27.300 GB in 241.066 secs = 116.0 MB/s
Test 4 - nr_req=  8 sync_thresh=192 - 27.665 GB in 241.016 secs = 117.5 MB/s

 

Previously, at md_sync_window=2816, Test 3 was my fastest.  Now, at 384, Test 4 is my fastest.

 

I'm now thinking I need to test each md_sync_window value in combination with both md_sync_thresh values - doubling the # of tests!!!

 

-Paul

[Squid]

I'm now thinking I need to test each md_sync_window value in combination with both md_sync_thresh values - doubling the # of tests!!!

It only takes time.  And if you come up with the best combination, then I'm sure nobody will mind.

[BRiT]

Hey BRiT,

 

I'm not sure what I mean... yet.  I was thinking that, in order to consistently determine if md_sync_thresh should be at -1 or 50%, and nr_requests at 8 or 128, that I should set the server to default unRAID values for the purpose of testing those two parameters.  This would be temporary.

 

But I just ran a test this way, and I got very different results than my previous test.  Now, keep in mind that md_sync_window at 384 is not optimized for my server, so the numbers should be lower than my post above.  That said, the results of this test didn't match my previous test, with md_sync_window at 2816.

 

Test 1 - nr_req=128 sync_thresh=383 - 12.255 GB in 240.802 secs =  52.1 MB/s
Test 2 - nr_req=128 sync_thresh=192 - 26.474 GB in 241.087 secs = 112.4 MB/s
Test 3 - nr_req=  8 sync_thresh=383 - 27.300 GB in 241.066 secs = 116.0 MB/s
Test 4 - nr_req=  8 sync_thresh=192 - 27.665 GB in 241.016 secs = 117.5 MB/s

 

Previously, at md_sync_window=2816, Test 3 was my fastest.  Now, at 384, Test 4 is my fastest.

 

I'm now thinking I need to test each md_sync_window value in combination with both md_sync_thresh values - doubling the # of tests!!!

 

-Paul

 

Maybe things have to be done in stages?

Stage 1 is determining what nr_req is set to 8 or 128. For this maybe only 4 tests need to be done, at Max-1, and 50% Max.

 

Is that relatively consistent in that test 3 and 4 always give faster results than test 1 and 2, regardless of the other settings?

 

Once nr_req is dialed in then testing on the other variables can take place?

[Pauven]

It only takes time.  And if you come up with the best combination, then I'm sure nobody will mind.

 

True.  It's probably me that minds the most.  This is a very frustrating project to work on, because each test takes so much time.  I've gone from just over 2 hours for a FullAuto test on v2.2 of this utility, to now 8-16 hours for the new version, with new parameters being tested for v6.2, more test points, and longer individual tests to improve accuracy.  I make a 5-second code change, and it takes 30+ minutes to test, if I'm lucky.  Makes progress very slow.

 

I just ran a test again on my server, this time using an md_sync_window of 760 (which my full test picked out as the fastest when I ran it last night).  Test 3 edged back into the lead again.

 

md_sync_window=760:

Test 1 - nr_req=128 sync_thresh=759 - 20.356 GB in 241.028 secs =  86.5 MB/s
Test 2 - nr_req=128 sync_thresh=380 - 27.237 GB in 240.932 secs = 115.8 MB/s
Test 3 - nr_req=  8 sync_thresh=759 - 28.881 GB in 241.016 secs = 122.7 MB/s
Test 4 - nr_req=  8 sync_thresh=380 - 28.777 GB in 240.943 secs = 122.3 MB/s

Beginning md_sync_window testing:
Test   1 - md_sync_window= 760 - 29.109 GB in 241.045 seconds = 123.7 MB/s

 

There's also the run-to-run variance.  Test 3 above used the exact same settings as that md_sync_window Test 1 in the same results, run 4 minutes apart from each other.  Over a 4 minute test cycle there was a 1MB/s variance.  Yes, I suppose that's less than 1% variance, but it is also bigger than the variance between Test 3 vs 4.  Not really enough accuracy to pick a winning combination.

 

Is that relatively consistent in that test 3 and 4 always give faster results than test 1 and 2, regardless of the other settings?

 

Kinda consistent, in a really inconsistent way.  I just reran this test at md_sync_window=2816 (the first one I posted several posts back)...

 

md_sync_window=2816 (first test):

Test 1 - nr_req=128 sync_thresh=2815 - 18.109 GB in 239.968 secs =  77.3 MB/s
Test 2 - nr_req=128 sync_thresh=1408 - 18.571 GB in 239.959 secs =  79.3 MB/s
Test 3 - nr_req=  8 sync_thresh=2815 - 29.960 GB in 240.030 secs = 127.8 MB/s
Test 4 - nr_req=  8 sync_thresh=1408 - 28.717 GB in 240.006 secs = 122.5 MB/s

 

And this time I got dramatically different results:

 

md_sync_window=2816 (second test):

Test 1 - nr_req=128 sync_thresh=2815 - 28.737 GB in 241.066 secs = 122.1 MB/s
Test 2 - nr_req=128 sync_thresh=1408 - 29.903 GB in 241.083 secs = 127.0 MB/s
Test 3 - nr_req=  8 sync_thresh=2815 - 30.556 GB in 241.038 secs = 129.8 MB/s
Test 4 - nr_req=  8 sync_thresh=1408 - 30.484 GB in 241.025 secs = 129.5 MB/s

 

This time Test 2 was high enough to beat Test 4 from the previous run.  What changed between runs?  Nothing, except the results.  Technically, Test 3 still won, and nr_requests still worked better at 8 than 128, but on this last run it's almost too close to call.

 

This is getting back to the point where I got some flak years ago because the Fastest and Best Bang results didn't always come out right.  Trying to automate this testing process is hard enough on my own server, but to come up with a methodology that reliably works on other servers.... sheez.  Can't believe I'm actually doing this again.

 

On the flip side, the new tests are helping.  Previously, I was getting < 110MB/s with md_sync_thresh and nr_requests both at stock values and md_sync_window at 2816 - and this was turning in a decent parity check time.  Now by tweaking those new settings, I'm squeaking out an additional 20 MB/s... nice.

 

I'm also finding that the same high md_sync_window #'s that worked so well on 5.0 are again working well on 6.2, but only in combination with tweaks to md_sync_thresh and nr_requests.

 

Oh, and BRiT, I agree with you, I think I can just use this preliminary test to get the nr_requests value, and just test md_sync_thresh twice with each md_sync_window.

 

-Paul

 

[JorgeB]

From my tests, and yours appear to agree,  nr_request=8 never performs worse than default, same or faster, so you could always set it to 8 for the tests, in the end you give the best results and it's up to the user to take all values, or change all except nr_requests.

 

This would cut the number of combinations to test significantly.

[Squid]

From my tests, and yours appear to agree,  nr_request=8 never performs worse than default, same or faster, so you could always set it to 8 for the tests, in the end you give the best results and it's up to the user to take all values, or change all except nr_requests.

 

This would cut the number of combinations to test significantly.

Unless you're one of the users that had the parity check slowdowns on the SAS2LP's.  Then nr_requests has a significant effect.

[JorgeB]

From my tests, and yours appear to agree,  nr_request=8 never performs worse than default, same or faster, so you could always set it to 8 for the tests, in the end you give the best results and it's up to the user to take all values, or change all except nr_requests.

 

This would cut the number of combinations to test significantly.

Unless you're one of the users that had the parity check slowdowns on the SAS2LP's.  Then nr_requests has a significant effect.

 

Yes, that's why I said to always set it to 8, it benefits SAS2LP users (also a little SASLP users) and doesn't make any difference for LSI and other controllers, all my servers are set to 8, most don't have SAS2LP.

[Squid]

From my tests, and yours appear to agree,  nr_request=8 never performs worse than default, same or faster, so you could always set it to 8 for the tests, in the end you give the best results and it's up to the user to take all values, or change all except nr_requests.

 

This would cut the number of combinations to test significantly.

Unless you're one of the users that had the parity check slowdowns on the SAS2LP's.  Then nr_requests has a significant effect.

 

Yes, that's why I said to always set it to 8, it benefits SAS2LP users (also a little SASLP users) and doesn't make any difference for LSI and other controllers, all my servers are set to 8, most don't have SAS2LP.

Had to reread the nr_requests thread.  I thought that since unRaid has a default now of 128 that was the fix.  But looks like 8 was the fix.  128 is probably just for safety sake.

[JorgeB]

Had to reread the nr_requests thread.  I thought that since unRaid has a default now of 128 that was the fix.  But looks like 8 was the fix.  128 is probably just for safety sake.

 

There was talk at the time that maybe setting it to 8 could have negative effects on other operations, likes normal writes, etc, as far as i know no one ever reported any issues.

[Pauven]

...nr_request=8 never performs worse than default, same or faster, so you could always set it to 8 for the tests...

 

I'm not convinced.

 

I increased the fullauto test time from 4 minutes up to 5 minutes, and started up a new run (expecting it to take about 14 hours).  Checked in on it 30 minutes later to see the results of the four nr_requests & md_sync_thresh tests, and was surprised to see that Test 2 had won (nr_request=128) by a smidge over Test 3 (nr_requests=.  Granted, that's only happened once in 5+ tests, but it happened all the same.  Based upon this, I think it best to leave in the test.  For all I know, there's some more bizaro servers out there like JBartlett's that behave the exact opposite of a typical server.

 

Anyway, frustrated with these results, I cancelled the test.  I really wanted to see a fullauto test result with nr_request=8 on my server.

 

I again tweaked the routine, doubling the amount of time spent on the four nr_requests & md_sync_thresh tests, from 5 to 10 minutes each, thinking an extra 20 minutes is well worth the increase in accuracy to determine the right nr_requests value for the rest of the test.  I also doubled the md_num_stripes and md_sync_window to twice stock unRAID values while running these four tests (I figured that put it more in the middle of the overall test range, giving more balanced results).  Keep in mind, I'm just playing around here, trying different strategies to find a way to make this tool even work, so don't be concerned that I'm hard coding values just yet.

 

I started the fullauto test again, and this time while Test 3 won out by 4MB/s, Test 2 beat Test 4. 

 

Test 1 - nr_req=128 sync_thresh=767 - 53.604 GB in 598.443 secs =  91.7 MB/s
Test 2 - nr_req=128 sync_thresh=384 - 71.401 GB in 598.327 secs = 122.2 MB/s
Test 3 - nr_req=  8 sync_thresh=767 - 73.899 GB in 598.487 secs = 126.4 MB/s
Test 4 - nr_req=  8 sync_thresh=384 - 71.205 GB in 598.409 secs = 121.8 MB/s

 

There's some really weird interplay between these two values.  With nr_requests=128, then md_sync_thresh=Max-1 becomes the slowest method, but with nr_requests=8, Max-1 becomes the fastest method.

 

With md_sync_thresh=50% of Max, the nr_requests value has little impact on my server, as both results were neck and neck, easily within the margin of error.  Both results were very good, but still 3%-4% slower the fastest result.

 

The fullauto test is continuing nicely, this time with nr_requests=8, and the highest result I have so far is 130.2 MB/s at md_sync_window=1280 and md_sync_thresh=1279, almost 6 MB/s faster than the value I got yesterday at md_sync_window=1280 but with unRAID stock nr_requests and md_sync_thresh values.  Even better, in general most of the results at every test point are faster than the test I ran last night, and most are close to maxing out the server, so even if I can't find the absolute best value, I sure can find some really good ones...

 

a Tests are md_sync_thresh=md_sync_window-1, b Tests =50% md_sync_window

Test   1a - md_sync_window= 384 - 34.399 GB in 299.196 seconds = 117.7 MB/s
Test   1b - md_sync_window= 384 - 33.674 GB in 299.224 seconds = 115.2 MB/s
Test   2a - md_sync_window= 448 - 35.336 GB in 299.335 seconds = 120.9 MB/s
Test   2b - md_sync_window= 448 - 34.476 GB in 299.180 seconds = 118.0 MB/s
Test   3a - md_sync_window= 512 - 35.415 GB in 299.260 seconds = 121.2 MB/s
Test   3b - md_sync_window= 512 - 37.088 GB in 299.155 seconds = 126.9 MB/s
Test   4a - md_sync_window= 576 - 35.046 GB in 299.219 seconds = 119.9 MB/s
Test   4b - md_sync_window= 576 - 36.070 GB in 299.209 seconds = 123.4 MB/s
Test   5a - md_sync_window= 640 - 35.232 GB in 299.214 seconds = 120.6 MB/s
Test   5b - md_sync_window= 640 - 36.328 GB in 299.193 seconds = 124.3 MB/s
Test   6a - md_sync_window= 704 - 36.257 GB in 299.219 seconds = 124.1 MB/s
Test   6b - md_sync_window= 704 - 35.934 GB in 299.182 seconds = 123.0 MB/s
Test   7a - md_sync_window= 768 - 36.430 GB in 299.277 seconds = 124.6 MB/s
Test   7b - md_sync_window= 768 - 35.909 GB in 299.183 seconds = 122.9 MB/s
Test   8a - md_sync_window= 832 - 37.932 GB in 299.366 seconds = 129.7 MB/s
Test   8b - md_sync_window= 832 - 36.401 GB in 299.242 seconds = 124.6 MB/s
Test   9a - md_sync_window= 896 - 37.873 GB in 299.292 seconds = 129.6 MB/s
Test   9b - md_sync_window= 896 - 36.511 GB in 299.247 seconds = 124.9 MB/s
Test  10a - md_sync_window= 960 - 37.297 GB in 299.335 seconds = 127.6 MB/s
Test  10b - md_sync_window= 960 - 36.797 GB in 299.216 seconds = 125.9 MB/s
Test  11a - md_sync_window=1024 - 37.634 GB in 299.349 seconds = 128.7 MB/s
Test  11b - md_sync_window=1024 - 37.048 GB in 299.287 seconds = 126.8 MB/s
Test  12a - md_sync_window=1088 - 37.747 GB in 299.280 seconds = 129.2 MB/s
Test  12b - md_sync_window=1088 - 37.078 GB in 299.239 seconds = 126.9 MB/s
Test  13a - md_sync_window=1152 - 37.818 GB in 299.275 seconds = 129.4 MB/s
Test  13b - md_sync_window=1152 - 37.084 GB in 299.269 seconds = 126.9 MB/s
Test  14a - md_sync_window=1216 - 37.968 GB in 299.337 seconds = 129.9 MB/s
Test  14b - md_sync_window=1216 - 37.400 GB in 299.277 seconds = 128.0 MB/s
Test  15a - md_sync_window=1280 - 38.054 GB in 299.330 seconds = 130.2 MB/s
Test  15b - md_sync_window=1280 - Test Range Entered - Time Remaining: 242s

[Pauven]

One of the reasons I'd like to get this converted to a GUI plug-in is so we can add graphical elements like line charts, showing the results.

 

It looks like my server settles down nicely md_sync_window values over 1000, and using Max-1 for md_sync_thresh offers a consistent increase of 50% Max.

 

Test isn't even at the halfway point yet...

 

 

[JorgeB]

Looking good 

[Pauven]

The test took 15 hours to complete, but I'm pretty happy with the result.  On unRAID v5, my server was able to hit 139 MB/s in this test.  Upgrading to v6.2 plus adding a second parity dropped the speeds to about 110 MB/s.  I figured dual parity would never run as fast as single parity, and while that may be true, the results below show that I've gotten really close at 131.5 MB/s.

 

Before tweaking the md_sync_thresh and nr_requests values, I found the data to be somewhat chaotic, so I added a lot of additional test points.  The test data below is the result of 164 parity check start/stops.  The 8 tests for md_sync_thresh/nr_requests ran for 10 minutes each, while the other 156 md_sync_window tests ran for 5 minutes each.  That's 14 hours and 20 minutes of pure test data.  The actual test took an extra 40 minutes, primarily because I don't start each test at the beginning of each parity check, but rather let it run a little bit to let it get up to speed before I start measuring, on average about 14 seconds for each one.

 

Tunables Report from unRAID Tunables Tester v4.0 by Pauven (for unRAID v6.2)

Automatic Parity Sync Test run on Sat Aug 20 16:01:24 EDT 2016
--- FULLY AUTOMATIC TEST PASS 1 (Rough - 45 Sample Points @ 5min Duration)---

NOTE: Use the smallest set of values that produce good results. Larger values
      increase server memory use, and may cause stability issues with unRAID,
      especially if you have any add-ons or plug-ins installed.

Current Values:  md_num_stripes=6256, md_sync_window=2816, md_sync_thresh=192

Test | num_stripes | sync_window | nr_requests | sync_thresh |   Speed 
  1   |     1536    |     768     |     128     |     767     |  91.7 MB/s 
  2   |     1536    |     768     |     128     |     384     | 122.2 MB/s 
  3   |     1536    |     768     |       8     |     767     | 126.4 MB/s 
  4   |     1536    |     768     |       8     |     384     | 121.8 MB/s 

Fastest vals were nr_reqs=8 and sync_thresh=99% of sync_window at 126.4 MB/s
These values will be used for the remainder of the test


Beginning md_sync_window testing:

Test | num_stripes | sync_window |   Speed 
   1a |    6400     |     384     | 117.7 MB/s 
   1b |    6400     |     384     | 115.2 MB/s 
   2a |    6400     |     448     | 120.9 MB/s 
   2b |    6400     |     448     | 118.0 MB/s 
   3a |    6400     |     512     | 121.2 MB/s 
   3b |    6400     |     512     | 126.9 MB/s 
   4a |    6400     |     576     | 119.9 MB/s 
   4b |    6400     |     576     | 123.4 MB/s 
   5a |    6400     |     640     | 120.6 MB/s 
   5b |    6400     |     640     | 124.3 MB/s 
   6a |    6400     |     704     | 124.1 MB/s 
   6b |    6400     |     704     | 123.0 MB/s 
   7a |    6400     |     768     | 124.6 MB/s 
   7b |    6400     |     768     | 122.9 MB/s 
   8a |    6400     |     832     | 129.7 MB/s 
   8b |    6400     |     832     | 124.6 MB/s 
   9a |    6400     |     896     | 129.6 MB/s 
   9b |    6400     |     896     | 124.9 MB/s 
  10a |    6400     |     960     | 127.6 MB/s 
  10b |    6400     |     960     | 125.9 MB/s 
  11a |    6400     |    1024     | 128.7 MB/s 
  11b |    6400     |    1024     | 126.8 MB/s 
  12a |    6400     |    1088     | 129.2 MB/s 
  12b |    6400     |    1088     | 126.9 MB/s 
  13a |    6400     |    1152     | 129.4 MB/s 
  13b |    6400     |    1152     | 126.9 MB/s 
  14a |    6400     |    1216     | 129.9 MB/s 
  14b |    6400     |    1216     | 128.0 MB/s 
  15a |    6400     |    1280     | 130.2 MB/s 
  15b |    6400     |    1280     | 128.3 MB/s 
  16a |    6400     |    1344     | 130.2 MB/s 
  16b |    6400     |    1344     | 129.0 MB/s 
  17a |    6400     |    1408     | 130.5 MB/s 
  17b |    6400     |    1408     | 128.9 MB/s 
  18a |    6400     |    1472     | 130.6 MB/s 
  18b |    6400     |    1472     | 129.1 MB/s 
  19a |    6400     |    1536     | 130.7 MB/s 
  19b |    6400     |    1536     | 129.5 MB/s 
  20a |    6400     |    1600     | 130.5 MB/s 
  20b |    6400     |    1600     | 129.8 MB/s 
  21a |    6400     |    1664     | 130.8 MB/s 
  21b |    6400     |    1664     | 130.2 MB/s 
  22a |    6400     |    1728     | 130.8 MB/s 
  22b |    6400     |    1728     | 130.4 MB/s 
  23a |    6400     |    1792     | 130.8 MB/s 
  23b |    6400     |    1792     | 130.4 MB/s 
  24a |    6400     |    1856     | 130.8 MB/s 
  24b |    6400     |    1856     | 130.6 MB/s 
  25a |    6400     |    1920     | 130.7 MB/s 
  25b |    6400     |    1920     | 130.6 MB/s 
  26a |    6400     |    1984     | 131.0 MB/s 
  26b |    6400     |    1984     | 130.6 MB/s 
  27a |    6400     |    2048     | 131.1 MB/s 
  27b |    6400     |    2048     | 130.4 MB/s 
  28a |    6400     |    2112     | 130.9 MB/s 
  28b |    6400     |    2112     | 130.6 MB/s 
  29a |    6400     |    2176     | 131.0 MB/s 
  29b |    6400     |    2176     | 130.6 MB/s 
  30a |    6400     |    2240     | 130.8 MB/s 
  30b |    6400     |    2240     | 130.8 MB/s 
  31a |    6400     |    2304     | 131.0 MB/s 
  31b |    6400     |    2304     | 130.8 MB/s 
  32a |    6400     |    2368     | 131.3 MB/s 
  32b |    6400     |    2368     | 130.8 MB/s 
  33a |    6400     |    2432     | 130.7 MB/s 
  33b |    6400     |    2432     | 130.7 MB/s 
  34a |    6400     |    2496     | 131.0 MB/s 
  34b |    6400     |    2496     | 130.8 MB/s 
  35a |    6400     |    2560     | 131.1 MB/s 
  35b |    6400     |    2560     | 130.8 MB/s 
  36a |    6400     |    2624     | 130.8 MB/s 
  36b |    6400     |    2624     | 130.8 MB/s 
  37a |    6400     |    2688     | 130.9 MB/s 
  37b |    6400     |    2688     | 130.6 MB/s 
  38a |    6400     |    2752     | 131.1 MB/s 
  38b |    6400     |    2752     | 130.9 MB/s 
  39a |    6400     |    2816     | 130.7 MB/s 
  39b |    6400     |    2816     | 131.0 MB/s 
  40a |    6400     |    2880     | 130.7 MB/s 
  40b |    6400     |    2880     | 131.0 MB/s 
  41a |    6400     |    2944     | 131.1 MB/s 
  41b |    6400     |    2944     | 131.0 MB/s 
  42a |    6400     |    3008     | 131.1 MB/s 
  42b |    6400     |    3008     | 131.2 MB/s 
  43a |    6400     |    3072     | 131.1 MB/s 
  43b |    6400     |    3072     | 130.9 MB/s 
  44a |    6400     |    3136     | 130.9 MB/s 
  44b |    6400     |    3136     | 130.8 MB/s 
  45a |    6400     |    3200     | 131.0 MB/s 
  45b |    6400     |    3200     | 131.0 MB/s 

The Fastest Sync Speed tested was md_sync_window=2368 at 131.3 MB/s
This will consume 200 MB just for the Syncs (additional for Writes and Reads)

The Thriftiest Sync Speed tested was md_sync_window=832 at 129.7 MB/s
This will consume 70 MB just for the Syncs (additional for Writes and Reads)

--- Targeting Fastest Result of md_sync_window 2368 bytes for Final Pass ---
--- FULLY AUTOMATIC TEST PASS 2 (Fine - 33 Sample Points @ 5min Duration)---

NOTE: Use the smallest set of values that produce good results. Larger values
      increase server memory use, and may cause stability issues with unRAID,
      especially if you have any add-ons or plug-ins installed.

Current Values:  md_num_stripes=6256, md_sync_window=2816, md_sync_thresh=192

Test | num_stripes | sync_window | nr_requests | sync_thresh |   Speed 
  1   |     1536    |     768     |     128     |     767     |  91.5 MB/s 
  2   |     1536    |     768     |     128     |     384     | 123.3 MB/s 
  3   |     1536    |     768     |       8     |     767     | 124.8 MB/s 
  4   |     1536    |     768     |       8     |     384     | 123.0 MB/s 

Fastest vals were nr_reqs=8 and sync_thresh=99% of sync_window at 124.8 MB/s
These values will be used for the remainder of the test


Beginning md_sync_window testing:

Test | num_stripes | sync_window |   Speed 
   1a |    4992     |    2240     | 130.9 MB/s 
   1b |    4992     |    2240     | 130.8 MB/s 
   2a |    4992     |    2248     | 130.9 MB/s 
   2b |    4992     |    2248     | 130.7 MB/s 
   3a |    4992     |    2256     | 131.2 MB/s 
   3b |    4992     |    2256     | 131.2 MB/s 
   4a |    4992     |    2264     | 131.1 MB/s 
   4b |    4992     |    2264     | 130.9 MB/s 
   5a |    4992     |    2272     | 131.0 MB/s 
   5b |    4992     |    2272     | 130.7 MB/s 
   6a |    4992     |    2280     | 131.0 MB/s 
   6b |    4992     |    2280     | 130.9 MB/s 
   7a |    4992     |    2288     | 130.8 MB/s 
   7b |    4992     |    2288     | 131.0 MB/s 
   8a |    4992     |    2296     | 130.8 MB/s 
   8b |    4992     |    2296     | 130.9 MB/s 
   9a |    4992     |    2304     | 130.8 MB/s 
   9b |    4992     |    2304     | 130.7 MB/s 
  10a |    4992     |    2312     | 130.9 MB/s 
  10b |    4992     |    2312     | 131.1 MB/s 
  11a |    4992     |    2320     | 131.1 MB/s 
  11b |    4992     |    2320     | 131.0 MB/s 
  12a |    4992     |    2328     | 131.0 MB/s 
  12b |    4992     |    2328     | 131.1 MB/s 
  13a |    4992     |    2336     | 131.0 MB/s 
  13b |    4992     |    2336     | 130.6 MB/s 
  14a |    4992     |    2344     | 131.0 MB/s 
  14b |    4992     |    2344     | 130.9 MB/s 
  15a |    4992     |    2352     | 131.0 MB/s 
  15b |    4992     |    2352     | 130.9 MB/s 
  16a |    4992     |    2360     | 131.0 MB/s 
  16b |    4992     |    2360     | 130.8 MB/s 
  17a |    4992     |    2368     | 131.1 MB/s 
  17b |    4992     |    2368     | 130.9 MB/s 
  18a |    4992     |    2376     | 131.0 MB/s 
  18b |    4992     |    2376     | 130.8 MB/s 
  19a |    4992     |    2384     | 131.0 MB/s 
  19b |    4992     |    2384     | 130.8 MB/s 
  20a |    4992     |    2392     | 130.9 MB/s 
  20b |    4992     |    2392     | 130.8 MB/s 
  21a |    4992     |    2400     | 131.0 MB/s 
  21b |    4992     |    2400     | 130.9 MB/s 
  22a |    4992     |    2408     | 130.9 MB/s 
  22b |    4992     |    2408     | 130.6 MB/s 
  23a |    4992     |    2416     | 130.9 MB/s 
  23b |    4992     |    2416     | 130.8 MB/s 
  24a |    4992     |    2424     | 131.5 MB/s 
  24b |    4992     |    2424     | 130.7 MB/s 
  25a |    4992     |    2432     | 131.3 MB/s 
  25b |    4992     |    2432     | 131.0 MB/s 
  26a |    4992     |    2440     | 131.0 MB/s 
  26b |    4992     |    2440     | 130.7 MB/s 
  27a |    4992     |    2448     | 130.9 MB/s 
  27b |    4992     |    2448     | 130.9 MB/s 
  28a |    4992     |    2456     | 131.1 MB/s 
  28b |    4992     |    2456     | 130.9 MB/s 
  29a |    4992     |    2464     | 131.0 MB/s 
  29b |    4992     |    2464     | 131.1 MB/s 
  30a |    4992     |    2472     | 131.0 MB/s 
  30b |    4992     |    2472     | 130.8 MB/s 
  31a |    4992     |    2480     | 130.7 MB/s 
  31b |    4992     |    2480     | 131.0 MB/s 
  32a |    4992     |    2488     | 131.3 MB/s 
  32b |    4992     |    2488     | 130.9 MB/s 
  33a |    4992     |    2496     | 131.0 MB/s 
  33b |    4992     |    2496     | 131.0 MB/s 

The Fastest Sync Speed tested was md_sync_window=2424 at 131.5 MB/s
This will consume 204 MB just for the Syncs (additional for Writes and Reads)

The Thriftiest Sync Speed tested was md_sync_window=2240 at 130.9 MB/s
This will consume 189 MB just for the Syncs (additional for Writes and Reads)


Completed: 15 Hrs 01 Min 36 Sec.

 

I've still got plenty of tweaking to do.  Now that I've got a full set of quality data, I'll evaluate if I can reduce the # of test points to cut down on running time.  My first thought is no, because if I go back to sampling every 128 stripes, instead of the 64 stripes of this test, I would have missed the peak at 2368, and instead found the second highest peak at 3008, centering Pass 2 there.  While the final speed would likely have been comparable, it would consume about 25% more memory for the additional stripes.  I think I prefer the additional sample points, even if the test takes significantly longer.  It's not like this is a daily or even monthly job, after all.

 

I'm also running the md_sync_thresh/nr_requests test twice, once before Pass 1, and again before Pass 2, but both times it runs with the same parameters.  I'm thinking the second run should be using the fastest md_sync_window from Pass 1.

 

Going to the charts (made in Excel, not from the utility), you can see a lot of volatility in the results at low md_sync_window values.  Above 1000, the server starts to settle down and you can see some nice curves forming. 

 

The md_sync_thresh at 50% Max line gradually catches up with the Max-1 line, and finally passes it at 2816 - the exact same value that produced the fastest results on my server on unRAID v5.  Coincidence?  I think not.

 

 

The fastest result was 131.3 MB/s at md_sync_window=2368 with md_sync_thresh=2367 (Max-1).

 

Also noteworthy is a 131.2 MB/s at md_sync_window=3008 with md_sync_thresh=1504 (50% Max).  Basically an identical speed, but takes more stripes/memory to get there.

 

While I talked about getting rid of the Best Bang for the Buck routine altogether on unRAID v6.2, I already had my new routine coded 3 years ago, so I just left it in.  It's now called Thriftiest, and it correctly identifies the lowest value that gives 95% of peak speed achieved during the test.  The new routine no longer gets confused like the old routine, as I'm using an array to store all the values, so I can sort and analyze them in the right order.

 

The Thriftiest value it found was at md_sync_window=832, and it achieved 98.8% of peak speed at significantly lower stripes/memory.  There's always a possibility that this spike is just a non-repeatable fluke, so if I was going to use it, I would run a manual test targeting that range with more detail and longer run times to ensure the result is real (or, hey, just run a parity check with that value and see if it's good).

 

The next chart is Pass 2.  It uses the fastest result from Pass 1, and does a more thorough scan of the range around it.  That's right, I now scan both below and above the fastest value from Pass 1.  In v2.2 of this utility I only scanned below, something that got questioned.  And rightfully so, as the fastest result of 131.5 MB/s came from above the fastest md_sync_window of Pass 1.

 

The chart makes the 131.5 MB/s look a lot faster than it really is.  From slowest point (130.6 MB/s) to fastest point is only 0.9 MB/s, within the margin of error, and all are good speeds.

 

 

I just kicked off a Correcting Parity Check using the fastest values from above.  Since upgrading to v6.2 and adding a 2nd parity drive, my checks have been taking 8.5 hours.  I think the new settings will bring me down below 8 hours, but probably not to the 7.5 hours I got on v5.0.

 

-Paul

[Frank1940]

In another thread  (  http://lime-technology.com/forum/index.php?topic=51036.45  ), Squid asked me to post up my result from using his modified script to see if the script would change the performance with dual parity.  The system used for this test was my Test Bed server (spec's below) running ver6 rc4.  Here are the results of the parity check after setting the tunables to the Best-Band suggestions. 

 

Event: unRAID Parity check
Subject: Notice [ROSE] - Parity check finished (0 errors)
Description: Duration: 12 hours, 59 minutes, 20 seconds. Average speed: 64.2 MB/s

 

Important disclosures:  (1) md_sync_thresh setting was set to one-half of md_sync_window  setting.  (2) The md_write_limit was left at the default because there is no place to set it in the GUI.

 

Here are the result of the Parity test with the default setting for ver 6 rc4

 

Event: unRAID Parity check
Subject: Notice [ROSE] - Parity check finished (0 errors)
Description: Duration: 12 hours, 58 minutes, 22 seconds. Average speed: 64.2 MB/s

 

The output of the script is attached. 

 

EDIT:  md-write_Limit was mislabeled.

 

EDIT 2:  Parity check speed for single parity is about 104MB/s ver 6 regardless of the release.

TunablesReport.txt

[Pauven]

Hey Frank,

 

Welcome to the thead, and thanks for sharing your output.

 

You mentioned v6 RC4 twice, do you mean v6.2 RC4? 

 

You said there's no place to set md_sync_window in the GUI - yes there is, right above the setting for md_sync_thresh. 

 

You can ignore the md_write_limit values in the results, they no longer apply to 6.2.

 

You said you left md_sync_window at stock (assuming 384) and md_sync_thresh at 50% (assuming 192). From reading your test results, I think this has a big speed penalty.

 

In your test results, at md_sync_window=512 you're at 51.2 MB/s, increasing to 57+ at 640.  Assuming this is a curve, I would expect 384 to be significantly less than 51.2, probably closer to 40 MB/s.  You could verify this by running a (M)anual test using the tool, with a start and end point at 384, any interval, for a duration of 3 minutes or more.

 

I think you're running this in your testbed, right, the one with these drives?:  1TB Hitachi DeskStar HD(Parity 2),  3 X 1TB Hitachi DeskStar HD

 

If so, those 1TB drives are holding back ultimate speeds, but perhaps you already know that.

 

My new script should do a much better job of evaluating settings on a 6.2 server, but it isn't quite ready for public consumption.  Maybe in a week or so you can be among my first testers of the new script.

 

Your results look similar to mine, peaking low and then significantly dropping off, before I added in the new special tests for nr_requests and md_sync_thresh.  With my new tests, the drop off disappeared and the highest speeds came from a much higher number of stripes.  v2.2 of the tool, that you're running, isn't capable of finding the best values in unRAID v6.2.

 

-Paul

 

 

[TheDragon]

Hey Frank,

 

Welcome to the thead, and thanks for sharing your output.

 

You mentioned v6 RC4 twice, do you mean v6.2 RC4? 

 

You said there's no place to set md_sync_window in the GUI - yes there is, right above the setting for md_sync_thresh. 

 

You can ignore the md_write_limit values in the results, they no longer apply to 6.2.

 

You said you left md_sync_window at stock (assuming 384) and md_sync_thresh at 50% (assuming 192). From reading your test results, I think this has a big speed penalty.

 

In your test results, at md_sync_window=512 you're at 51.2 MB/s, increasing to 57+ at 640.  Assuming this is a curve, I would expect 384 to be significantly less than 51.2, probably closer to 40 MB/s.  You could verify this by running a (M)anual test using the tool, with a start and end point at 384, any interval, for a duration of 3 minutes or more.

 

I think you're running this in your testbed, right, the one with these drives?:  1TB Hitachi DeskStar HD(Parity 2),  3 X 1TB Hitachi DeskStar HD

 

If so, those 1TB drives are holding back ultimate speeds, but perhaps you already know that.

 

My new script should do a much better job of evaluating settings on a 6.2 server, but it isn't quite ready for public consumption.  Maybe in a week or so you can be among my first testers of the new script.

 

Your results look similar to mine, peaking low and then significantly dropping off, before I added in the new special tests for nr_requests and md_sync_thresh.  With my new tests, the drop off disappeared and the highest speeds came from a much higher number of stripes.  v2.2 of the tool, that you're running, isn't capable of finding the best values in unRAID v6.2.

 

-Paul

 

Hi Paul

 

I'm using the latest 6.2 RC and would be more than happy to help you test your revised script and report back results. ?

[Pauven]

I'm using the latest 6.2 RC and would be more than happy to help you test your revised script and report back results. ?

 

Thanks jack0w!  Just watch this thread during this week, I expect to have it available soon.

 

-Paul

[Pauven]

Reaching out for help from some users running 6.2 with Dual Parity and NOT a PRO license.  It might also be interesting to see those with a PRO license, but the Array drive count configured to less than the default 24.

 

I need the output of this command:

 

mdcmd status | grep "rdevStatus."

 

Here is what I see on my 6.2 server with Dual Parity and a PRO license:

rdevStatus.0=DISK_OK
rdevStatus.1=DISK_OK
rdevStatus.2=DISK_OK
rdevStatus.3=DISK_OK
rdevStatus.4=DISK_OK
rdevStatus.5=DISK_OK
rdevStatus.6=DISK_OK
rdevStatus.7=DISK_OK
rdevStatus.8=DISK_OK
rdevStatus.9=DISK_OK
rdevStatus.10=DISK_OK
rdevStatus.11=DISK_OK
rdevStatus.12=DISK_OK
rdevStatus.13=DISK_NP
rdevStatus.14=DISK_NP
rdevStatus.15=DISK_NP
rdevStatus.16=DISK_NP
rdevStatus.17=DISK_OK
rdevStatus.18=DISK_OK
rdevStatus.19=DISK_NP
rdevStatus.20=DISK_NP
rdevStatus.21=DISK_NP
rdevStatus.22=DISK_NP
rdevStatus.23=DISK_NP
rdevStatus.24=DISK_NP
rdevStatus.25=DISK_NP
rdevStatus.26=DISK_NP
rdevStatus.27=DISK_NP
rdevStatus.28=DISK_NP
rdevStatus.29=DISK_OK

 

I'm trying to determine if the second parity drive always shows up as drive 29, or if this moves around based upon license or configuration.

 

Thanks,

Paul

[Squid]

You should be able to parse /var/local/emhttp/disks.ini

 

In the [parity2] section it has an IDX label  (which on my system spec in sig) comes up with 29

[JorgeB]

Parity2 is always disk29.

[Pauven]

Thanks guys.

 

Does the "/boot/config/extra.cfg" configuration file still work with 6.2, or is that obsolete or replaced/changed?

[Squid]

It still was there (at least in RC2 for experimental hardlink support on user shares), but that's about the only reference I can find for it anywhere on the forum with 6.2

[Pauven]

Okay, thanks.  I have some logic in the script to look at it if it exists, wasn't sure if it was still needed.  I'll keep it.