Pauven

Awesome, I love it, thanks for sharing! At first glance, once of the things I noticed is that the individual disk speeds are lower than the parity check speed, and more closely match what I've been reporting in my tool. For example, the parity check speed is reporting 148.6 MB/s, but the individual drives are fluctuating from low 130's to a high of 147 MB/s. Hah! For a second there I saw disk speeds of 285 MB/s!!! Must have caught a momentary burst at just the right moment. bonienl, any concern that monitoring the speeds will impact performance? What is the timeframe over which the speed is calculated? Can it be configured to average over a longer time span? Thanks! -Paul

For both johnnie.black and myself, nr_requests=128 produced the fastest result, but only when properly tuned with the other values. Looking at my results, using nr_requests=128 or 16 would have produced nearly identical results, and based upon 128 providing better ultimate performance I would probably stick with 128. Going as low as nr_requests=8 might have come up short in finding the best md_sync_window range, though, as it seems to both peak and flatline a bit early, but with ultimately lower performance than 128. This is easier to see if you look at nr_requests=4, which peaks early at sync_window=768 and stays flat at higher values, never exceeding 96% of peak performance. nr_requests=8 peaked at early 1536 and stayed flat from there, never exceeding 97.5% of peak performance. Both nr_requests=16 and 128 peaked at 3072, with 16 providing 99% of peak performance, and 128 providing 100%. Actually, looks like 32, 64 and 128 all peaked the same, and followed very similar curves. I guess I could be convinced that 32 might be a nice middle of the road approach that is known to mimic 128, but may provide better tracking on some systems for which 128 never works well.

The problem with that approach is that, on my server at optimum md_sync_window: nr_requests=128 worked best at sync_thresh=-64 nr_requests=16 worked best at sync_thresh=-8 nr_requests=8 worked best at both sync_thresh=-1 and -64 If I only test for one, I might not find the right combination of sync_thresh vs nr_requests. I have no way of accurately predicting which nr_requests is going to be fastest, nor which sync_thresh is going to be fastest. I think I have to test all the combos to see which one wins. And on my server at optimum md_sync_window, from the best to the worst sync_thresh&nr_requests combo there was a 6% variance. Hypothetically, if Test 1 found that sync_thresh=-8 was best, and that was used for Test 2, and then again for Test 3 to find the best nr_requests value, it would have come back as nr_requests=16. Then in Test 4 I would have tested all values for nr_requests=16 and found that sync_thresh=-8 is best. The problem is that nr_requests=128 with sync_thresh=-64 was actually best, but got excluded from the last round of testing because nr_requests=128 didn't test well at sync_thresh=-8.

Fleshing out the testing strategy, on my server Test 1 would look like this: Test | num_stripes | sync_window | nr_requests | sync_thresh | Speed --------------------------------------------------------------------------- 1 | 768 | 384 | 128 | 376 | 53.1 MB/s 2 | 768 | 384 | 128 | 320 | 129.9 MB/s 3 | 768 | 384 | 128 | 192 | 117.6 MB/s 4 | 1536 | 768 | 128 | 760 | 93.5 MB/s 5 | 1536 | 768 | 128 | 704 | 132.1 MB/s 6 | 1536 | 768 | 128 | 384 | 124.9 MB/s 7 | 3072 | 1536 | 128 | 1528 | 127.6 MB/s 8 | 3072 | 1536 | 128 | 1472 | 133.3 MB/s 9 | 3072 | 1536 | 128 | 768 | 130.2 MB/s 10 | 6144 | 3072 | 128 | 3064 | 132.3 MB/s 11 | 6144 | 3072 | 128 | 3008 | 136.3 MB/s 12 | 6144 | 3072 | 128 | 1536 | 130.9 MB/s Fastest vals were sync_window=3072 sync_thresh=(sync_window - 64) at 136.3 MB/s Because we're only testing nr_requests=128, the number of test points is dramatically reduced. With 10-minute length tests, this completes in 2 hours. On my server, Test 2 then runs in the very high range from sync_window 1536 to 3456 in increments of 64, using fastest sync_thresh method from test 1. This test is also run at nr_requests=128. That's 30 test points, for a running time of 5 hours. Hypothetically, Test 2 finds that sync_window=3072 is the leading edge of the peak. Finally, Test 3 runs the another nr_requests test, this time at sync_window=3072, and testing nr_requests=128/16/8, and sync_thresh=-1, -4 to -64 in increments of 4, and /2. This still feels a little excessive, at 9 hours for test 3. 16 hours would be the total running time for tests 1-3. Test | num_stripes | sync_window | nr_requests | sync_thresh | Speed --------------------------------------------------------------------------- 1 | 6144 | 3072 | 128 | 3071 | 130.4 MB/s 2 | 6144 | 3072 | 128 | 3068 | 130.4 MB/s 3 | 6144 | 3072 | 128 | 3064 | 132.3 MB/s 4 | 6144 | 3072 | 128 | 3060 | 132.3 MB/s 5 | 6144 | 3072 | 128 | 3056 | 132.3 MB/s 6 | 6144 | 3072 | 128 | 3052 | 132.3 MB/s 7 | 6144 | 3072 | 128 | 3048 | 132.3 MB/s 8 | 6144 | 3072 | 128 | 3044 | 132.3 MB/s 9 | 6144 | 3072 | 128 | 3040 | 132.3 MB/s 10 | 6144 | 3072 | 128 | 3036 | 132.3 MB/s 11 | 6144 | 3072 | 128 | 3032 | 132.3 MB/s 12 | 6144 | 3072 | 128 | 3028 | 132.3 MB/s 13 | 6144 | 3072 | 128 | 3024 | 132.3 MB/s 14 | 6144 | 3072 | 128 | 3020 | 132.3 MB/s 15 | 6144 | 3072 | 128 | 3016 | 132.3 MB/s 16 | 6144 | 3072 | 128 | 3012 | 132.3 MB/s 17 | 6144 | 3072 | 128 | 3008 | 136.3 MB/s 18 | 6144 | 3072 | 128 | 1536 | 130.9 MB/s --------------------------------------------------------------------------- 19 | 6144 | 3072 | 16 | 3071 | 130.4 MB/s 20 | 6144 | 3072 | 16 | 3068 | 130.4 MB/s 21 | 6144 | 3072 | 16 | 3064 | 132.3 MB/s 22 | 6144 | 3072 | 16 | 3060 | 132.3 MB/s 23 | 6144 | 3072 | 16 | 3056 | 132.3 MB/s 24 | 6144 | 3072 | 16 | 3052 | 132.3 MB/s 25 | 6144 | 3072 | 16 | 3048 | 132.3 MB/s 26 | 6144 | 3072 | 16 | 3044 | 132.3 MB/s 27 | 6144 | 3072 | 16 | 3040 | 132.3 MB/s 28 | 6144 | 3072 | 16 | 3036 | 132.3 MB/s 29 | 6144 | 3072 | 16 | 3032 | 132.3 MB/s 30 | 6144 | 3072 | 16 | 3028 | 132.3 MB/s 31 | 6144 | 3072 | 16 | 3024 | 132.3 MB/s 32 | 6144 | 3072 | 16 | 3020 | 132.3 MB/s 33 | 6144 | 3072 | 16 | 3016 | 132.3 MB/s 34 | 6144 | 3072 | 16 | 3012 | 132.3 MB/s 35 | 6144 | 3072 | 16 | 3008 | 136.3 MB/s 36 | 6144 | 3072 | 16 | 1536 | 130.9 MB/s --------------------------------------------------------------------------- 37 | 6144 | 3072 | 8 | 3071 | 130.4 MB/s 38 | 6144 | 3072 | 8 | 3068 | 130.4 MB/s 39 | 6144 | 3072 | 8 | 3064 | 132.3 MB/s 40 | 6144 | 3072 | 8 | 3060 | 132.3 MB/s 41 | 6144 | 3072 | 8 | 3056 | 132.3 MB/s 42 | 6144 | 3072 | 8 | 3052 | 132.3 MB/s 43 | 6144 | 3072 | 8 | 3048 | 132.3 MB/s 44 | 6144 | 3072 | 8 | 3044 | 132.3 MB/s 45 | 6144 | 3072 | 8 | 3040 | 132.3 MB/s 46 | 6144 | 3072 | 8 | 3036 | 132.3 MB/s 47 | 6144 | 3072 | 8 | 3032 | 132.3 MB/s 48 | 6144 | 3072 | 8 | 3028 | 132.3 MB/s 49 | 6144 | 3072 | 8 | 3024 | 132.3 MB/s 50 | 6144 | 3072 | 8 | 3020 | 132.3 MB/s 51 | 6144 | 3072 | 8 | 3016 | 132.3 MB/s 52 | 6144 | 3072 | 8 | 3012 | 132.3 MB/s 53 | 6144 | 3072 | 8 | 3008 | 136.3 MB/s 54 | 6144 | 3072 | 8 | 1536 | 130.9 MB/s Fastest vals were nr_reqs=128 and sync_thresh=(sync_window - 64) at 136.3 MB/s

How about using the first test to find only sync_window and sync_thresh? Looks to me like with nr_requests set at default there's a better chance of finding the optimal sync_thresh, also looks like the best sync_thresh is the same (or in the case of your last test, practically the same) with the various nr_request values, so after finding the optimal window and thresh values you could do a test on those changing only nr_requests, I believe this would be faster and provide better results than trying to find optimal values for all 3 settings at the same time. You're absolutely right. I like it.

Well, that's disappointing, in that the optimum sync_thresh assignment changes with controllers/drives. Means more exhaustive testing is required. Perhaps what would be better is a multi-staged approach. 1st) Run the nr_requests test at various md_sync_windows to determine if nr_requests should be 128, 16 or 8 for the next tests, and determine if sync_thresh should be -8, -64, or /2. This test would also identify whether md_sync_window testing should be in the low, medium or high range. This test would take about 6 hours. 2nd) Run the md_sync_window test using the nr_requests & sync_thresh values identified in the first test. This test would identify the best md_sync_window. Not sure at the moment, but this test should take 3-6 hours. 3rd) Run an enhanced nr_requests/sync_thresh test using the md_sync_window from the second test. Because it's only testing a single md_sync_window, we can run a lot more tests at various nr_requests & sync_thresh values, to hone in on what's best. This test could take 6-10 hours, depending up how thorough we make it. Based upon your results, if I had to take a guess the above strategy would play out as follows: 1) The first test: nr_requests=8 and sync_thresh=sync_window-8 would probably come out on top, and with md_sync_window values in the high range. While obviously not the fastest, these should be good enough to find the best md_sync_window in test 2. 2) The second test would focus on md_sync_windows in the high range (say 1024-3072), and would find something in the neighborhood of 2048 works best. 3) The third test would focus on md_sync_window=2048, and try all the potential sync_thresh (-1, -4 to -64 in increments of 4, and /2) and nr_requests (128, 16, values, finding that sync_thresh=-20 and whatever nr_requests works best. Thoughts?

And here's the long results. Longer than expected, at 20 hours!!! Each individual test point lasted 10 minutes. NOTE: I hand calculated the percentages compared to the fastest speed, which came from the last test group below. sync_window=384, which is not ideal on my server: Test | num_stripes | sync_window | nr_requests | sync_thresh | Speed --------------------------------------------------------------------------- 1 | 768 | 384 | 128 | 383 | 51.3 MB/s - 37% 2 | 768 | 384 | 128 | 376 | 53.1 MB/s - 39% 3 | 768 | 384 | 128 | 320 | 129.9 MB/s - 95.3% * 4 | 768 | 384 | 128 | 192 | 117.6 MB/s - 86% --------------------------------------------------------------------------- 5 | 768 | 384 | 64 | 383 | 51.8 MB/s - 37% 6 | 768 | 384 | 64 | 376 | 53.2 MB/s - 39% 7 | 768 | 384 | 64 | 320 | 129.9 MB/s - 95.3% * 8 | 768 | 384 | 64 | 192 | 121.2 MB/s - 89% --------------------------------------------------------------------------- 9 | 768 | 384 | 32 | 383 | 51.4 MB/s - 37% 10 | 768 | 384 | 32 | 376 | 55.4 MB/s - 40% 11 | 768 | 384 | 32 | 320 | 131.4 MB/s - 96.4% * 12 | 768 | 384 | 32 | 192 | 119.4 MB/s - 87% --------------------------------------------------------------------------- 13 | 768 | 384 | 16 | 383 | 88.6 MB/s - 65% 14 | 768 | 384 | 16 | 376 | 92.5 MB/s - 68% 15 | 768 | 384 | 16 | 320 | 129.9 MB/s - 95.3% * 16 | 768 | 384 | 16 | 192 | 116.9 MB/s - 85% --------------------------------------------------------------------------- 17 | 768 | 384 | 8 | 383 | 117.5 MB/s - 86% 18 | 768 | 384 | 8 | 376 | 120.1 MB/s - 88% 19 | 768 | 384 | 8 | 320 | 131.5 MB/s - 96.5% *** 20 | 768 | 384 | 8 | 192 | 119.6 MB/s - 88% --------------------------------------------------------------------------- 21 | 768 | 384 | 4 | 383 | 124.2 MB/s - 91.1% 22 | 768 | 384 | 4 | 376 | 124.7 MB/s - 91.5% 23 | 768 | 384 | 4 | 320 | 129.5 MB/s * 95.0% 24 | 768 | 384 | 4 | 192 | 121.4 MB/s - 89% --------------------------------------------------------------------------- 25 | 768 | 384 | 1 (act 4) | 383 | 124.3 MB/s - 91.2% 26 | 768 | 384 | 1 (act 4) | 376 | 125.0 MB/s - 91.7% 27 | 768 | 384 | 1 (act 4) | 320 | 129.3 MB/s - 94.7% * 28 | 768 | 384 | 1 (act 4) | 192 | 121.0 MB/s - 89% Fastest vals were nr_reqs=8 and sync_thresh=(sync_window - 64) at 131.5 MB/s These results seem to illustrate how nr_requests=8 gained such popularity, not because of how good it is, but rather how bad 128 can be when combined with commonly used sync_window and sync_thresh values, delivering <40% of maximum speeds, wow!!! But changing nr_requests alone wasn't enough, it needs to be set in conjunction with sync_window and sync_thresh. Noteworthy here is that using sync_thresh=(sync_window-64) produces good results, averaging around 95% of maximum tested speed, regardless of nr_requests. In this test group, it appears that sync_window-1 is equivalent to sync_window-8. Also noteworthy is that nr_requests=4 (and 1, which is really 4 again) produce decent results with all sync_thresh values. sync_window=768, which is a little better but still not optimized on my server: Test | num_stripes | sync_window | nr_requests | sync_thresh | Speed --------------------------------------------------------------------------- 1 | 1536 | 768 | 128 | 767 | 93.7 MB/s - 69% 2 | 1536 | 768 | 128 | 760 | 93.5 MB/s - 69% 3 | 1536 | 768 | 128 | 704 | 132.1 MB/s - 96.9% * 4 | 1536 | 768 | 128 | 384 | 124.9 MB/s - 91.6% --------------------------------------------------------------------------- 5 | 1536 | 768 | 64 | 767 | 93.9 MB/s - 69% 6 | 1536 | 768 | 64 | 760 | 93.3 MB/s - 69% 7 | 1536 | 768 | 64 | 704 | 132.2 MB/s - 97.0% *** 8 | 1536 | 768 | 64 | 384 | 125.1 MB/s - 91.8% --------------------------------------------------------------------------- 9 | 1536 | 768 | 32 | 767 | 89.9 MB/s - 66% 10 | 1536 | 768 | 32 | 760 | 94.4 MB/s - 69% 11 | 1536 | 768 | 32 | 704 | 132.0 MB/s - 96.8% * 12 | 1536 | 768 | 32 | 384 | 125.4 MB/s - 92.0% --------------------------------------------------------------------------- 13 | 1536 | 768 | 16 | 767 | 125.8 MB/s - 92.3% 14 | 1536 | 768 | 16 | 760 | 125.8 MB/s - 92.3% 15 | 1536 | 768 | 16 | 704 | 132.1 MB/s - 96.9% * 16 | 1536 | 768 | 16 | 384 | 123.9 MB/s - 90.9% --------------------------------------------------------------------------- 17 | 1536 | 768 | 8 | 767 | 128.2 MB/s - 94.1% 18 | 1536 | 768 | 8 | 760 | 129.2 MB/s - 94.8% 19 | 1536 | 768 | 8 | 704 | 132.0 MB/s - 96.8% * 20 | 1536 | 768 | 8 | 384 | 123.8 MB/s - 90.8% --------------------------------------------------------------------------- 21 | 1536 | 768 | 4 | 767 | 131.1 MB/s - 96.2% * 22 | 1536 | 768 | 4 | 760 | 130.1 MB/s - 95.5% 23 | 1536 | 768 | 4 | 704 | 128.3 MB/s - 94.1% 24 | 1536 | 768 | 4 | 384 | 128.8 MB/s - 94.5% --------------------------------------------------------------------------- 25 | 1536 | 768 | 1 (act 4) | 767 | 129.9 MB/s - 95.3% 26 | 1536 | 768 | 1 (act 4) | 760 | 130.5 MB/s - 95.7% * 27 | 1536 | 768 | 1 (act 4) | 704 | 128.1 MB/s - 94.0% 28 | 1536 | 768 | 1 (act 4) | 384 | 128.9 MB/s - 94.6% Fastest vals were nr_reqs=64 and sync_thresh=(sync_window - 64) at 132.2 MB/s In general, all results are getting faster as the md_sync_window approaches optimization. Again, using sync_thresh=(sync_window-64) produces good results, averaging around 96% of maximum tested speed, regardless of nr_requests. Still appears that sync_window-1 is equivalent to sync_window-8. Also noteworthy is that nr_requests=4 (and 1, which is really 4 again) produce decent results with all sync_thresh values. sync_window=1536, on the fringe of my server's sweet zone: Test | num_stripes | sync_window | nr_requests | sync_thresh | Speed --------------------------------------------------------------------------- 1 | 3072 | 1536 | 128 | 1535 | 124.5 MB/s - 91.3% 2 | 3072 | 1536 | 128 | 1528 | 127.6 MB/s - 93.6% 3 | 3072 | 1536 | 128 | 1472 | 133.3 MB/s - 97.8% * 4 | 3072 | 1536 | 128 | 768 | 130.2 MB/s - 95.5% --------------------------------------------------------------------------- 5 | 3072 | 1536 | 64 | 1535 | 124.9 MB/s - 91.6% 6 | 3072 | 1536 | 64 | 1528 | 127.9 MB/s - 93.8% 7 | 3072 | 1536 | 64 | 1472 | 133.6 MB/s - 98.0% *** 8 | 3072 | 1536 | 64 | 768 | 128.2 MB/s - 94.1% --------------------------------------------------------------------------- 9 | 3072 | 1536 | 32 | 1535 | 124.8 MB/s - 91.6% 10 | 3072 | 1536 | 32 | 1528 | 129.8 MB/s - 95.2% 11 | 3072 | 1536 | 32 | 1472 | 133.6 MB/s - 98.0% *** 12 | 3072 | 1536 | 32 | 768 | 130.3 MB/s - 95.6% --------------------------------------------------------------------------- 13 | 3072 | 1536 | 16 | 1535 | 132.2 MB/s - 97.0% 14 | 3072 | 1536 | 16 | 1528 | 132.4 MB/s - 97.1% 15 | 3072 | 1536 | 16 | 1472 | 133.5 MB/s - 97.9% * 16 | 3072 | 1536 | 16 | 768 | 127.9 MB/s - 93.8% --------------------------------------------------------------------------- 17 | 3072 | 1536 | 8 | 1535 | 132.6 MB/s - 97.3% 18 | 3072 | 1536 | 8 | 1528 | 132.8 MB/s - 97.4% * 19 | 3072 | 1536 | 8 | 1472 | 132.8 MB/s - 97.4% * 20 | 3072 | 1536 | 8 | 768 | 130.2 MB/s - 95.5% --------------------------------------------------------------------------- 21 | 3072 | 1536 | 4 | 1535 | 129.7 MB/s - 95.2% 22 | 3072 | 1536 | 4 | 1528 | 129.8 MB/s - 95.2% * 23 | 3072 | 1536 | 4 | 1472 | 129.6 MB/s - 95.1% 24 | 3072 | 1536 | 4 | 768 | 129.2 MB/s - 94.8% --------------------------------------------------------------------------- 25 | 3072 | 1536 | 1 (act 4) | 1535 | 129.4 MB/s - 94.9% 26 | 3072 | 1536 | 1 (act 4) | 1528 | 130.8 MB/s - 96.0% * 27 | 3072 | 1536 | 1 (act 4) | 1472 | 129.6 MB/s - 95.1% 28 | 3072 | 1536 | 1 (act 4) | 768 | 129.9 MB/s - 95.3% Fastest vals were nr_reqs=64 and sync_thresh=(sync_window - 64) at 133.6 MB/s All results are still getting faster as the md_sync_window approaches optimization. Again, using sync_thresh=(sync_window-64) produces good results, averaging around 96% of maximum tested speed, regardless of nr_requests. Still appears that sync_window-1 is mostly equivalent to sync_window-8, though in a few cases sync_window-8 appears a little better. And again, nr_requests=4 (and 1, which is really 4 again) produced good results with all sync_thresh values, but this time so did 8 and 16, and on average both did better than 4. sync_window=3072, finally into my server's sweet zone, and the results prove it: Test | num_stripes | sync_window | nr_requests | sync_thresh | Speed --------------------------------------------------------------------------- 1 | 6144 | 3072 | 128 | 3071 | 130.4 MB/s - 95.7% 2 | 6144 | 3072 | 128 | 3064 | 132.3 MB/s - 97.1% 3 | 6144 | 3072 | 128 | 3008 | 136.3 MB/s - 100% ***FASTEST*** 4 | 6144 | 3072 | 128 | 1536 | 130.9 MB/s - 96.0% --------------------------------------------------------------------------- 5 | 6144 | 3072 | 64 | 3071 | 129.7 MB/s - 95.2% 6 | 6144 | 3072 | 64 | 3064 | 132.1 MB/s - 96.9% 7 | 6144 | 3072 | 64 | 3008 | 136.2 MB/s - 99.9% * 8 | 6144 | 3072 | 64 | 1536 | 130.6 MB/s - 95.8% --------------------------------------------------------------------------- 9 | 6144 | 3072 | 32 | 3071 | 130.2 MB/s - 95.5% 10 | 6144 | 3072 | 32 | 3064 | 133.7 MB/s - 98.1% 11 | 6144 | 3072 | 32 | 3008 | 136.2 MB/s - 99.9% * 12 | 6144 | 3072 | 32 | 1536 | 131.0 MB/s - 96.1% --------------------------------------------------------------------------- 13 | 6144 | 3072 | 16 | 3071 | 134.9 MB/s - 99.0% * 14 | 6144 | 3072 | 16 | 3064 | 135.0 MB/s - 99.0% * 15 | 6144 | 3072 | 16 | 3008 | 134.8 MB/s - 98.9% 16 | 6144 | 3072 | 16 | 1536 | 134.4 MB/s - 98.6% --------------------------------------------------------------------------- 17 | 6144 | 3072 | 8 | 3071 | 132.9 MB/s - 97.5% * 18 | 6144 | 3072 | 8 | 3064 | 132.8 MB/s - 97.4% 19 | 6144 | 3072 | 8 | 3008 | 132.9 MB/s - 97.5% * 20 | 6144 | 3072 | 8 | 1536 | 132.8 MB/s - 97.4% --------------------------------------------------------------------------- 21 | 6144 | 3072 | 4 | 3071 | 130.7 MB/s - 95.9% * 22 | 6144 | 3072 | 4 | 3064 | 129.8 MB/s - 95.2% 23 | 6144 | 3072 | 4 | 3008 | 129.6 MB/s - 95.1% 24 | 6144 | 3072 | 4 | 1536 | 128.3 MB/s - 94.1% --------------------------------------------------------------------------- 25 | 6144 | 3072 | 1 (act 4) | 3071 | 129.7 MB/s - 95.2% * 26 | 6144 | 3072 | 1 (act 4) | 3064 | 129.5 MB/s - 95.0% 27 | 6144 | 3072 | 1 (act 4) | 3008 | 129.5 MB/s - 95.0% 28 | 6144 | 3072 | 1 (act 4) | 1536 | 129.5 MB/s - 95.0% Fastest vals were nr_reqs=128 and sync_thresh=(sync_window - 64) at 136.3 MB/s We see a jump of 3 MB/s now that the md_sync_window is in the optimized region for my server. Pretty much undefeated, using sync_thresh=(sync_window-64) produces great results, averaging about 98% of maximum tested speed, regardless of nr_requests value, plus it produced the 3 fastest measured speeds during the entire test. Still appears that sync_window-1 is mostly equivalent to sync_window-8, and even turned in a great result at nr_requests=16. Actually, nr_requests=16 made every sync_thresh value look great here. 8 and 4 did okay. 8 was interesting in that almost every result was identical. Accuracy: Since nr_requests=1 was really just a retest of nr_requests=4, you can get a feel for how accurate these results are by comparing them. Most of the time they are within a few points of each other, worst case was a 1.2 MB/s variance. That's a variance of less than 1%, so these are pretty accurate results at 10 minutes. This test was too long, and has too much fat in it. Obvious fat to cut is nr_requests=1, which unRAID overrides to 4. I also don't see much value in nr_requests=4, 16, or 64. I think 8, 32 and 128 are more than representative. Or perhaps 8, 16 and 128, since 16 did really well once we got into a sync_window optimized region. I also don't see any value in sync_thresh=sync_window-1. sync_window-8 and sync_window/2 are questionable in light of sync_window-64's excellence, but this might just be on my server, so for now I think I will keep them. If I cut the fat above, that would bring this overall test down to 36 test points, compared to the 112 I just ran. It would complete in about 6 hours.

In this same vein, it may be productive to offer the option to take down the ethernet interface for the duration of the test. That way if it suits the user, they could remove the possibility of external influence during testing. Trick would be to only allow it if there was local console access to get back from a bad state. Can you detect in your script if it was called from a local console? That feature is already available, has been there since day one. To turn it on, simply unplug your ethernet cable.

Low accuracy due to short test lengths. The exact same test run multiple times with the same parameters will return different results each time when the test length is short. In the 10 minute tests I just completed, the nr_requests=1 is virtually identical to ne_requests=4.

So you're thinking nr_requests is somehow related to NCQ? I don't have NCQ enabled on my server, and nr_requests are having a huge impact in performance. Don't know if that means anything.

Now I wish I had bothered to look at this before running the current long test. I had pulled nr_requests=1 out of my test routine after your previous post, but then decided I would test it at least once and put it back in - adding 2.5 hours to the overall test length. My test routine just hit nr_requests=1, and here's what the system is showing: root@Tower:~# cat /sys/block/sdj/queue/nr_requests 4 I'm really surprised, as I set the nr_requests value directly by echoing the value I want to /sys/block/sdX/queue/nr_requests, bypassing the GUI. I didn't anticipate there would be some process to come behind and "fix" the value after I assigned it. Live and learn. Good news is we can eliminate nr_requests=1 from the tests. Might as well ask, is 128 truly the upper limit? I can't look now, the test is running.

The following are all 1 minute tests, so pretty low accuracy. Consider this a rough preview. Here's the first round of nr_request and sync_thresh tests, all with sync_window=384: Test | num_stripes | sync_window | nr_requests | sync_thresh | Speed --------------------------------------------------------------------------- 1 | 768 | 384 | 128 | 383 | 47.1 MB/s 2 | 768 | 384 | 128 | 376 | 49.8 MB/s 3 | 768 | 384 | 128 | 320 | 114.1 MB/s Fastest 4 | 768 | 384 | 128 | 192 | 104.5 MB/s --------------------------------------------------------------------------- 5 | 768 | 384 | 64 | 383 | 45.5 MB/s 6 | 768 | 384 | 64 | 376 | 50.1 MB/s 7 | 768 | 384 | 64 | 320 | 112.2 MB/s Fastest 8 | 768 | 384 | 64 | 192 | 106.4 MB/s --------------------------------------------------------------------------- 9 | 768 | 384 | 32 | 383 | 46.5 MB/s 10 | 768 | 384 | 32 | 376 | 50.2 MB/s 11 | 768 | 384 | 32 | 320 | 113.6 MB/s Fastest 12 | 768 | 384 | 32 | 192 | 103.1 MB/s --------------------------------------------------------------------------- 13 | 768 | 384 | 16 | 383 | 80.6 MB/s 14 | 768 | 384 | 16 | 376 | 81.9 MB/s 15 | 768 | 384 | 16 | 320 | 112.8 MB/s Fastest 16 | 768 | 384 | 16 | 192 | 103.7 MB/s --------------------------------------------------------------------------- 17 | 768 | 384 | 8 | 383 | 101.2 MB/s 18 | 768 | 384 | 8 | 376 | 105.3 MB/s 19 | 768 | 384 | 8 | 320 | 112.7 MB/s Fastest 20 | 768 | 384 | 8 | 192 | 107.2 MB/s --------------------------------------------------------------------------- 21 | 768 | 384 | 4 | 383 | 107.1 MB/s 22 | 768 | 384 | 4 | 376 | 110.0 MB/s 23 | 768 | 384 | 4 | 320 | 111.2 MB/s Fastest 24 | 768 | 384 | 4 | 192 | 105.6 MB/s --------------------------------------------------------------------------- 25 | 768 | 384 | 1 | 383 | 109.8 MB/s 26 | 768 | 384 | 1 | 376 | 111.1 MB/s 27 | 768 | 384 | 1 | 320 | 111.9 MB/s Fastest 28 | 768 | 384 | 1 | 192 | 108.7 MB/s Fastest vals were nr_reqs=128 and sync_thresh=(sync_window - 64) at 114.1 MB/s Interestingly, regardless of nr_requests value, sync_thresh=(sync_window - 64) always produced the fastest result. Second round at sync_window=768: Test | num_stripes | sync_window | nr_requests | sync_thresh | Speed --------------------------------------------------------------------------- 1 | 1536 | 768 | 128 | 767 | 73.7 MB/s 2 | 1536 | 768 | 128 | 760 | 79.3 MB/s 3 | 1536 | 768 | 128 | 704 | 111.5 MB/s Fastest 4 | 1536 | 768 | 128 | 384 | 108.1 MB/s --------------------------------------------------------------------------- 5 | 1536 | 768 | 64 | 767 | 74.8 MB/s 6 | 1536 | 768 | 64 | 760 | 78.4 MB/s 7 | 1536 | 768 | 64 | 704 | 111.4 MB/s Fastest 8 | 1536 | 768 | 64 | 384 | 108.9 MB/s --------------------------------------------------------------------------- 9 | 1536 | 768 | 32 | 767 | 73.2 MB/s 10 | 1536 | 768 | 32 | 760 | 80.6 MB/s 11 | 1536 | 768 | 32 | 704 | 112.4 MB/s Fastest 12 | 1536 | 768 | 32 | 384 | 103.9 MB/s --------------------------------------------------------------------------- 13 | 1536 | 768 | 16 | 767 | 102.1 MB/s 14 | 1536 | 768 | 16 | 760 | 107.4 MB/s 15 | 1536 | 768 | 16 | 704 | 111.1 MB/s Fastest 16 | 1536 | 768 | 16 | 384 | 108.6 MB/s --------------------------------------------------------------------------- 17 | 1536 | 768 | 8 | 767 | 105.2 MB/s 18 | 1536 | 768 | 8 | 760 | 106.5 MB/s 19 | 1536 | 768 | 8 | 704 | 112.1 MB/s Fastest 20 | 1536 | 768 | 8 | 384 | 107.7 MB/s --------------------------------------------------------------------------- 21 | 1536 | 768 | 4 | 767 | 110.0 MB/s Fastest 22 | 1536 | 768 | 4 | 760 | 108.9 MB/s 23 | 1536 | 768 | 4 | 704 | 109.3 MB/s 24 | 1536 | 768 | 4 | 384 | 107.8 MB/s --------------------------------------------------------------------------- 25 | 1536 | 768 | 1 | 767 | 108.6 MB/s 26 | 1536 | 768 | 1 | 760 | 109.4 MB/s 27 | 1536 | 768 | 1 | 704 | 109.6 MB/s Fastest 28 | 1536 | 768 | 1 | 384 | 107.8 MB/s Fastest vals were nr_reqs=32 and sync_thresh=(sync_window - 64) at 112.4 MB/s Except for a single test point (nr_requests=4), once again sync_thresh=(sync_window - 64) was fastest. Third round at sync_window=1536: Test | num_stripes | sync_window | nr_requests | sync_thresh | Speed --------------------------------------------------------------------------- 1 | 3072 | 1536 | 128 | 1535 | 99.7 MB/s 2 | 3072 | 1536 | 128 | 1528 | 114.4 MB/s Fastest 3 | 3072 | 1536 | 128 | 1472 | 109.1 MB/s 4 | 3072 | 1536 | 128 | 768 | 110.3 MB/s --------------------------------------------------------------------------- 5 | 3072 | 1536 | 64 | 1535 | 100.9 MB/s 6 | 3072 | 1536 | 64 | 1528 | 99.2 MB/s 7 | 3072 | 1536 | 64 | 1472 | 107.9 MB/s 8 | 3072 | 1536 | 64 | 768 | 109.8 MB/s Fastest --------------------------------------------------------------------------- 9 | 3072 | 1536 | 32 | 1535 | 99.1 MB/s 10 | 3072 | 1536 | 32 | 1528 | 104.3 MB/s 11 | 3072 | 1536 | 32 | 1472 | 110.3 MB/s 12 | 3072 | 1536 | 32 | 768 | 110.4 MB/s Fastest --------------------------------------------------------------------------- 13 | 3072 | 1536 | 16 | 1535 | 104.8 MB/s 14 | 3072 | 1536 | 16 | 1528 | 103.5 MB/s 15 | 3072 | 1536 | 16 | 1472 | 107.8 MB/s 16 | 3072 | 1536 | 16 | 768 | 110.5 MB/s Fastest --------------------------------------------------------------------------- 17 | 3072 | 1536 | 8 | 1535 | 110.9 MB/s 18 | 3072 | 1536 | 8 | 1528 | 110.6 MB/s 19 | 3072 | 1536 | 8 | 1472 | 112.3 MB/s 20 | 3072 | 1536 | 8 | 768 | 113.9 MB/s Fastest --------------------------------------------------------------------------- 21 | 3072 | 1536 | 4 | 1535 | 112.9 MB/s 22 | 3072 | 1536 | 4 | 1528 | 115.7 MB/s Fastest 23 | 3072 | 1536 | 4 | 1472 | 112.9 MB/s 24 | 3072 | 1536 | 4 | 768 | 110.4 MB/s --------------------------------------------------------------------------- 25 | 3072 | 1536 | 1 | 1535 | 107.6 MB/s 26 | 3072 | 1536 | 1 | 1528 | 110.2 MB/s Fastest 27 | 3072 | 1536 | 1 | 1472 | 108.8 MB/s 28 | 3072 | 1536 | 1 | 768 | 107.5 MB/s Fastest vals were nr_reqs=4 and sync_thresh=(sync_window - at 115.7 MB/s My server likes higher sync_window values, and it's starting to get into the sweet spot with sync_window set to 1536, which changed up the results quite a bit. This time, sync_thresh=(sync_window - 64) wasn't fastest at all. This time it was (sync_window/2) battling (sync_window - 8 ). The fourth and final round at sync_window=3072: Test | num_stripes | sync_window | nr_requests | sync_thresh | Speed --------------------------------------------------------------------------- 1 | 6144 | 3072 | 128 | 3071 | 117.6 MB/s Fastest 2 | 6144 | 3072 | 128 | 3064 | 112.9 MB/s 3 | 6144 | 3072 | 128 | 3008 | 116.5 MB/s 4 | 6144 | 3072 | 128 | 1536 | 112.0 MB/s --------------------------------------------------------------------------- 5 | 6144 | 3072 | 64 | 3071 | 116.0 MB/s Fastest 6 | 6144 | 3072 | 64 | 3064 | 102.5 MB/s 7 | 6144 | 3072 | 64 | 3008 | 113.4 MB/s 8 | 6144 | 3072 | 64 | 1536 | 110.7 MB/s --------------------------------------------------------------------------- 9 | 6144 | 3072 | 32 | 3071 | 102.2 MB/s 10 | 6144 | 3072 | 32 | 3064 | 105.7 MB/s 11 | 6144 | 3072 | 32 | 3008 | 114.1 MB/s Fastest 12 | 6144 | 3072 | 32 | 1536 | 111.1 MB/s --------------------------------------------------------------------------- 13 | 6144 | 3072 | 16 | 3071 | 118.5 MB/s 14 | 6144 | 3072 | 16 | 3064 | 124.0 MB/s Fastest 15 | 6144 | 3072 | 16 | 3008 | 121.4 MB/s 16 | 6144 | 3072 | 16 | 1536 | 117.1 MB/s --------------------------------------------------------------------------- 17 | 6144 | 3072 | 8 | 3071 | 112.6 MB/s Fastest 18 | 6144 | 3072 | 8 | 3064 | 112.3 MB/s 19 | 6144 | 3072 | 8 | 3008 | 111.5 MB/s 20 | 6144 | 3072 | 8 | 1536 | 111.8 MB/s --------------------------------------------------------------------------- 21 | 6144 | 3072 | 4 | 3071 | 107.7 MB/s 22 | 6144 | 3072 | 4 | 3064 | 108.0 MB/s 23 | 6144 | 3072 | 4 | 3008 | 107.0 MB/s 24 | 6144 | 3072 | 4 | 1536 | 109.3 MB/s Fastest --------------------------------------------------------------------------- 25 | 6144 | 3072 | 1 | 3071 | 109.2 MB/s 26 | 6144 | 3072 | 1 | 3064 | 110.1 MB/s Fastest 27 | 6144 | 3072 | 1 | 3008 | 109.4 MB/s 28 | 6144 | 3072 | 1 | 1536 | 108.7 MB/s Fastest vals were nr_reqs=16 and sync_thresh=(sync_window - at 124.0 MB/s This sync_window is in my server's sweet spot, and now for the first time setting sync_thresh=(sync_window -1 ) finally eeked out a few victories. Actually, every method of setting sync_thresh got a victory here. Perhaps once a server gets closer to optimized on sync_window, sync_thresh becomes less important... Also very interesting is that nr_requests=16 produced a dramatically faster result than all other tests. Going to run a long version of this test overnight, with 10 minute tests instead of 1 minute. It will take about 18 hours. Let you know tomorrow afternoon how it went.

Thanks johnnie.black! You mean 79.8 to 80.0, correct? To my eye, I see a bell curve. Hmmm, didn't know it had a minimum value. Interesting that 4 looks better than 8, even if only very slightly. I'm running a similar test on my server now, using 7 different nr_requests values of 128, 64, 32, 16, 8, 4, and 1 (perhaps I can set at the command line...), in combination with 4 different sync_thresh values: sync_window-1, sync_window-8, sync_window-64 and sync_window/2. That's 28 test points. I'm repeating it three times, at sync_windows 384, 768 and 1536. That's 84 total test points. I should have preview results this evening, and more formal results from a longer test overnight. Early assessment is that sync_window-64 is looking nice for sync_thresh.

johnnie.black, you're freaking awesome! I can't quite tell if that is a bell curve with a peak around 1984, or a linear decline with a peak around 2040. You also skipped over 1960 which worked best on the other controller. If you can run the 4 extra test points above, that would help. That is an awesome discovery. It may help explain why my script is having a hard time finding good values. I'm still looking for the pattern on why certain values are better than others. If I can see the pattern, I can script it. In this case, if the peak is 1984, that is sync_window-64, one of the values I was interested in. But how to interpret that? Is that 50% of nr_requests? Is that -8 per drive on that controller? Is that 3.125% of sync_window? Or is that just -64 works nice on this controller for no explainable reason... This has me wondering why 128 and 8 are the only values we're testing for nr_requests. Could you complete the following? stripes | window | nr_reqs | thresh | Speed ----------------------------------------------------------- 4096 | 2048 | 128 | 1024 | 77.1MB/s 4096 | 2048 | 64 | 1024 | ? MB/s 4096 | 2048 | 32 | 1024 | ? MB/s 4096 | 2048 | 16 | 1024 | ? MB/s 4096 | 2048 | 8 | 1024 | 79.8MB/s 4096 | 2048 | 4 | 1024 | ? MB/s 4096 | 2048 | 1 | 1024 | ? MB/s Did I say you're awesome? I think I need a bigger word. Thanks so much!

I'm so disappointed in your sub-gigabyte speeds... Sure would be awesome to run a parity check on my 3TB drives in under an hour!

stripes | window | nr_reqs | thresh | Speed ----------------------------------------------------------- 4096 | 2240 | 8 | 1960 | 206.6MB/s stripes | window | nr_reqs | thresh | Speed ----------------------------------------------------------- 4096 | 2048 | 8 | 1984 | 207.9MB/s 4096 | 2048 | 8 | 1968 | 207.9MB/s 4096 | 2048 | 8 | 1952 | 209.3MB/s 4096 | 2048 | 8 | 1920 | 207.9MB/s Note that a check with the same settings sometimes is a second shorter or longer, because it's a very small array it makes a difference of a few MB/s, so when the results are very close they could be practically considered the same, e.g.: Duration: 2 minutes, 32 seconds. Average speed: 210.6 MB/s Duration: 2 minutes, 33 seconds. Average speed: 209.3 MB/s Duration: 2 minutes, 34 seconds. Average speed: 207.9 MB/s So with a sync_window=2048 a sync_thresh from ~1900 to ~1990 gives very similar results. Very interesting. I'm still seeing how you found a great combination in 2048/1960, but I can't fathom why that is such a great combination. I'm typically pretty good at finding the pattern. This one is eluding me.

You both make excellent points. On the flip side, I'm seeing a lot of consistency in the results, and not the flickering of results that would be expected when running the same test multiple times. Perhaps I missed it, but I didn't see even a single second variance when the same test point was run multiple times. To me, it appears that there is more accuracy feeding into the internal calculation, but that a rounding function near the end of the calculation is rounding to the nearest second. Hard to say whom is right, could go either way. Regardless, I'm loving Rosetta Stone. And for a parity check that runs 152 seconds, 3 seconds is 2%. Not huge, but bigger than a typical rounding error.

Giving it some more thought, not sure why 80 would be special, other than it it 10x8. 8x8 (64), 8x12 (96), and 8x16 (128) might be more interesting. sync_window=2048, sync_thresh=1984 (-64) = ? sync_window=2048, sync_thresh=1968 (-80) = ? sync_window=2048, sync_thresh=1952 (-96) = ? sync_window=2048, sync_thresh=1920 (-128) = ? I can't help but think that the best numbers are factors of 8. 2048=(8*256), in binary this is a very nice number.

Thank you, that helps. Not sure what it means yet, but it helps. I'm really surprised at how much this: stripes | window | nr_reqs | thresh | Speed ----------------------------------------------------------- 6400 | 2240 | 8 | 1960 | 206.6MB/s is slower than this: stripes | window | nr_reqs | thresh | Speed ----------------------------------------------------------- 4096 | 2048 | 8 | 1960 | 210.6MB/s Please run this: stripes | window | nr_reqs | thresh | Speed ----------------------------------------------------------- 4096 | 2240 | 8 | 1960 | ? MB/s I'm trying to understand if we cranked up num_stripes too high at 6400, so it is causing problems, or if the speed impact was due solely to the different sync_window. I've also noticed in your results that sync_window=2048 with md_sync_thresh=1960 consistently produces speeds of 210.6. But increasing sync_window to 2240 dropped it by 4 MB/s. Similarly, decreasing sync_thresh to 1945 also dropped it by 4 MB/s. It seems that the two values have to be spaced a specific distance apart - 88 in this case. You didn't try a spacing of 80 in your tests, wonder if that would work even better? i.e. sync_window=2048, sync_thresh=1968.

num_stripes=4096: 210.6 num_stripes=2096: 209.3 You're right, difference represents only one second for the total time, so we can consider the same speed. Excellent. I think that is the first proof. Next, how about testing various md_sync_windows with static md_sync_thresh. Since 1960 works as a good sync_thresh on Rosetta Stone, let's use that: Test | stripes | window | nr_reqs | thresh | Speed ----------------------------------------------------------- 1 | 6400 | 1960 | 8 | 1960 | ? MB/s <--what happens when sync_thresh = sync_window? Probably bad... 2 | 6400 | 1968 | 8 | 1960 | ? MB/s <--basically sync_window-1, but -8 to give a nice round # 3 | 6400 | 2240 | 8 | 1960 | ? MB/s <--sync_window * 7/8 4 | 6400 | 2614 | 8 | 1960 | ? MB/s <--sync_window * 3/4 5 | 6400 | 3920 | 8 | 1960 | ? MB/s <--sync_window * 2/4 6 | 6400 | 5880 | 8 | 1960 | ? MB/s <--sync_window * 1/4

I really appreciate this information. Please keep it up. Are you sure the unRAID default is md_num_stripes=1280? I really don't know, so I'm asking. I think this server should be renamed: Rosetta Stone. With the ability to complete a full parity check in only 2.5 minutes, and at such high speeds, this server can fully reveal the interactions between different values, and in a short amount of time. I'm not 100% sure of how v6.2 behaves now that md_write_limit is gone, but here is what we uncovered with v5 - hopefully this will give you some ideas for testing v6.2: md_num_stripes is the total # of stripes available for reading/writing/syncing. It must always remain the highest number. md_sync_window is the total # of stripes that parity syncs are limited to. While parity checks are underway, md_num_stripes-md_sync_window=the # of remaining stripes available to handle read/write requests. So for example, if md_num_stripes=5000, and md_sync_window=1000, then during a parity check there are 4000 stripes are remaining to handle reads/writes. md_write_limit is the total # of stripes that array writes were limited to on v5.x. While writing is underway, md_num_stripes-md_write_limit=the # of remaining stripes available to handle read/sync requests. So for example, if md_num_stripes=5000, and md_write_limit=1500, then during writing, 3500 stripes are remaining to handle reads/syncs. Using the above examples, if you were reading, writing and running a parity sync all at the same time, 5000-1000-1500=2500, so parity checks would get 1000 stripes, writes would get 1500 stripes, and reads would get the remaining 2500 stripes. Lime-Tech never revealed any priority of reads vs. writes vs syncs, but if I had to guess, on v5 syncs had priority because of how much people complained about stuttering while watching movies during parity checks. That's just a guess, though, plus I haven't been on the forums enough to know if this general behavior has changed in 6.x. We had to make some educated guesses in both the above and in interpreting how to come up with the right values. Of the three activities, reading vs. writing vs. parity checks, we figured parity checks were the most intensive activity, as it was the only one that engaged all drives simultaneously. Based upon that, one could surmise that if 1000 was enough for a parity check, then it was more than enough for reading or writing, as those were "easier" tasks. So then we deduced that this was really a balancing equation. md_num_stripes vs. md_write_limit vs. md_sync_window determined how the server was balanced to do one task over another. If you set num_stripes to 3000, write_limit to 1000, and sync_window to 1000, then during all three you get 1000 read, 1000 write, 1000 sync, or a perfectly balanced 1:1:1 ratio and more than sufficient for reads and writes. (To be clear, Lime-Tech never confirmed this deduction, so it could be wrong.) You could also re-balance the server by changing the ratio. So if 1000 was enough for parity checks, and if you set num_stripes=1500, write_limit=250, and sync_window=1000, then when the server is doing all three at the same time, you get 250 read, 250 write, and 1000 sync, or a 1:1:4 balancing ratio. If you did something crazy, like set num_stripes=1001, write_limit=1 and sync_window=1000, then if you're only reading you get 1001 stripes, only writing you get 1 stripe, reading and writing you get 1 write and 1000 read, but if reading/writing/syncing you get 1000 sync, 1 write, 0 read - completely starving the reads. Since md_write_limit is gone, it might be that Lime-Tech is now auto-balancing reads vs writes. If the default values are num_stripes=1280 and sync_window=384, then LT is leaving 896 stripes to handle reads and writes during a parity check, and giving us no control or visibility over the balancing of reads vs writes. I think it is also telling that LT is setting default values of num_stripes 1280 and sync_window 384, as that is more than 3:1. Having my script set num_stripes 2:1 vs. sync_window may be too low for a properly balance read vs. write vs. sync ratio. Getting back to your tests, for a parity check test, num_stripes should have nearly zero impact as long as A) you're not read/writing during the test, and B) num_stripes is bigger than sync_window. Sync_window is the primary factor for parity check speed. On Rosetta Stone, you could test this by running nr_requests=8, sync_window=2048, sync_thresh=1960, doing this twice once with num_stripes=4096, and again with num_stripes=2096, barely above sync_window. Just make sure nothing is reading/writing to the array during this test. From what I've read, md_sync_thresh is how aggressively unRAID works to refill md_sync_window. Like a waiter at a restaurant refilling your drink, do they refill after every sip (md_sync_thresh=md_sync_window-1), after the glass is half empty (md_sync_thresh=md_sync_window/2), or when you're sucking on ice (md_sync_thresh=. If I had to guess as to what's happening with various controller cards, some cards have a slow processor that can't multi-task, while others have a fast one. A slow processor is like the diner who pauses eating while the waiter refills their drink, so refilling after every sip makes it nearly impossible to eat. A fast processor is like the diner who keeps chowing down, ignoring the waiter refilling their glass, so refilling after every sip doesn't slow anything down. Lower nr_requests values probably work for slow and fast cards alike, but if you go too low, then eventually sync is starved of stripes waiting on the next refill. It might be that, for cards that like fewer refills, using even lower values like 25% of md_sync_window will gain even more performance, so long as we don't go so low that the glass ever runs dry. Yes, anything is possible. More test points only take more time. I could try 25%, 50%, 75% and md_sync_window-1 to paint a nice general picture. I could also do 10%, 20%, 30%, 40%, 50%, 60%, 70%, 80%, 90% and sync_window-1. The real challenge here in optimizing parity check speeds is that now instead of the single parameter we had to worry about on unRAID v5 (sync_window), we now have 3 (sync_window, sync_thresh and nr_requests), plus we see that adjusting some parameters up/down affects the behavior of other parameters. I can't just test 50 settings and call it a day, I need to test 50 x 2 x 10, or 1000 combinations of settings. This is an exponentially bigger problem on unRAID v6.2. Also challenging is that because nr_requests and sync_thresh seem to directly impact each other, I need to test both simultaneously. And because they behave differently at different sync_window values, I need to test them at various sync_windows too. I have yet to wrap my head around a good routine to testing all three values. I could easily program a routine that test every possible combination, millions of them, it would just take years to run. For setting md_num_stripes, for now we are probably okay just setting it to 2x or 3x the md_sync_window value. We'll need read/write tests in the future to figure out the best strategy.

Same here. Same here too. DOH!

Another test with poor results. Can't really make a judgement call on good values, other than lower seems a bit better. I don't recall v2.2 on unRAID 5.x having such inconsistency, but maybe I'm just forgetting. I was anticipating better results since the new test sample is almost twice the size of the v2.2 test sample. I'm thinking I need to revamp the test strategy. The first pass seems to give a good overall picture, and if the curve forms it makes it pretty easy to see what values to try. With the results I'm seeing on 6.2, the second pass seems pretty worthless. It "might" find a slightly better value, but then again it seems just as likely to give crap results and be a total waste of time. Considering the 2nd pass runs for almost 6 hours alone, I could eliminate it and put that time towards making the first pass more accurate. I can double the sample running time to 10 minutes on the first pass. It could run as short as 6 hours, even with the double accuracy. If the extra 1b/1c/1d tests get triggered, it could run for a max around 20 hours. Having a single pass, start to finish, instead of multiple overlapping passes, should also make the results easier to graph. I'm also thinking of separating out the nr_requests test, making it standalone. This new standalone test would replace the Short Auto. To me, the whole purpose of the Short Auto was just to give a quick preview to determine if the Normal Auto is worth running. The nr_requests portion of the Short Auto appears to be answering that question, and at the same time the low accuracy of the Short Auto is just causing too many questions and doubts. I could make the nr_requests be more robust too, testing more points at low, medium and high values, to give a better idea of what values are right for each system. The best result could feed into the Normal Auto test, so it could skip testing nr_requests and instead focus on md_sync_window values. I would appreciate some feedback from the beta testers. Does this sound like a better solution? Do you have any other ideas? Paul

Thanks for posting your results. I do have some concerns with what I'm seeing. First is a lack of consistency (or accuracy) at lower run times. These two nr_requests tests (10 minutes each): Test | num_stripes | sync_window | nr_requests | sync_thresh | Speed --------------------------------------------------------------------------- 3 | 1536 | 768 | 8 | 767 | 95.9 MB/s 4 | 1536 | 768 | 8 | 384 | 80.1 MB/s Are the data test points in these 2 md_sync_window tests in Test 7 (5 minutes each) Test | RAM | stripes | window | reqs | thresh | MB/s | thresh | MB/s ------------------------------------------------------------------------ 7 | 81 | 1536 | 768 | 8 | 767 | 98.3 | 384 | 69.6 The high md_sync_threst tests (at 767) were actually pretty good, but the 50% test (at 384) were different by > 10 MB/s. The first pass is looking pretty good, I'm even okay with the fastest values it found: Test | RAM | stripes | window | reqs | thresh | MB/s | thresh | MB/s ------------------------------------------------------------------------ 9 | 184 | 3456 | 1728 | 8 | 1727 | 105.8 | 864 | 101.1 But then it retested those values in the nr_requests test (Pass 2), with very different results: Test | num_stripes | sync_window | nr_requests | sync_thresh | Speed --------------------------------------------------------------------------- 1 | 3456 | 1728 | 128 | 1727 | 93.9 MB/s 2 | 3456 | 1728 | 128 | 864 | 81.0 MB/s 3 | 3456 | 1728 | 8 | 1727 | 75.2 MB/s 4 | 3456 | 1728 | 8 | 864 | 73.9 MB/s All of a sudden, the server is producing speeds > 25 MB/s slower! This is a longer test (10 minutes), so perhaps we were getting falsely high values at 5 minutes? I'm really not sure. But it is from here things really go downhill. Instead of using nr_requests=8, which it used from the first pass, it switches to 128 because that is now testing faster. But the measured speeds in Pass 2 are all lower than what we were seeing on Pass 1. This does reveal a small cosmetic bug in my code, because Pass 2 returns lower results, the fastest speed came from Pass 1. While my Fastest/Thriftiest/Recommended report correctly found this, it incorrectly reported the nr_requests value to use with those values: The Fastest Sync Speed tested was md_sync_window=1216 at 105.0 MB/s Tunable (md_num_stripes): 2432 Tunable (md_sync_window): 1216 <--This was from Pass 1, not Pass 2 (good) Tunable (md_sync_thresh): 1215 Tunable (nr_requests): 128 <--But Pass 1 was nr_requests=8, not 128 (bad) I charted the results. Red/Blue are Pass 1, Green/Purple are Pass 2 with a lot of sample points in a small range. I liked the semi-curve Pass 1 was producing, but when the nr_requests was switched to 128 for Pass 2, the results looked pretty bad: Thoughts? Paul

Hey Squid, would you mind posting the csv files for both of those? Thanks, Paul