Agree, these settings may be optimal for some servers, and while I didn't have much time this week and intend to do more testing with different controllers later, all results point to a optimal setting that is usually a little below sync_window, just not sure if there is a set value, like -60 that is optimal for everything, ideally the script would run a short test in the beginning to try and find it.
I retested a single LSI9211 with larger (and faster) SSDs in the hopes of seeing better defined results, and while they are, the optimal thresh value changed from the previous tests (previous tests where done with 2 controllers at the same time, so maybe why the difference, but I don't have 16 of largest SSDs to test with both again, and using only one controller with the smallest SSDs won't help either because results will be limited by their max speed in almost all tests)
Sync_window=2048
stripes | window | nr_reqs | thresh | Speed
-----------------------------------------------------------
4096 | 2048 | 128 | 2047 | 289.7MB/s
4096 | 2048 | 128 | 2040 | 321.7MB/s
4096 | 2048 | 128 | 2036 | 335.2MB/s
4096 | 2048 | 128 | 2032 | 337.0MB/s
4096 | 2048 | 128 | 2028 | 340.5MB/s
4096 | 2048 | 128 | 2024 | 333.5MB/s
4096 | 2048 | 128 | 2016 | 330.0MB/s
4096 | 2048 | 128 | 1984 | 330.0MB/s
4096 | 2048 | 128 | 1960 | 330.0MB/s
4096 | 2048 | 128 | 1952 | 330.0MB/s
4096 | 2048 | 128 | 1920 | 330.0MB/s
4096 | 2048 | 128 | 1856 | 325.0MB/s
4096 | 2048 | 128 | 1792 | 326.6MB/s
4096 | 2048 | 128 | 1536 | 323.3MB/s
4096 | 2048 | 128 | 1280 | 320.1MB/s
4096 | 2048 | 128 | 1024 | 314.4MB/s
Same sync_window but nr_requests=8 for the 4 fastest results (like before, looks like it doesn't make a big difference with LSI controllers)
stripes | window | nr_reqs | thresh | Speed
-----------------------------------------------------------
4096 | 2048 | 8 | 2036 | 337.0MB/s
4096 | 2048 | 8 | 2032 | 340.5MB/s
4096 | 2048 | 8 | 2028 | 340.5MB/s
4096 | 2048 | 8 | 2024 | 335.2MB/s
Sync_window=1024 and nr_request back to default:
stripes | window | nr_reqs | thresh | Speed
-----------------------------------------------------------
2048 | 1024 | 128 | 1023 | 293.7MB/s
2048 | 1024 | 128 | 1016 | 328.3MB/s
2048 | 1024 | 128 | 1012 | 331.7MB/s
2048 | 1024 | 128 | 1008 | 333.5MB/s
2048 | 1024 | 128 | 1004 | 337.0MB/s
2048 | 1024 | 128 | 1000 | 325.0MB/s
2048 | 1024 | 128 | 996 | 316.9MB/s
Sync_window=3072
stripes | window | nr_reqs | thresh | Speed
-----------------------------------------------------------
6144 | 3072 | 128 | 3071 | 295.0MB/s
6144 | 3072 | 128 | 3064 | 321.7MB/s
6144 | 3072 | 128 | 3056 | 335.2MB/s
6144 | 3072 | 128 | 3052 | 337.0MB/s
6144 | 3072 | 128 | 3048 | 333.5MB/s
6144 | 3072 | 128 | 3040 | 333.5MB/s
6144 | 3072 | 128 | 3032 | 331.7MB/s
6144 | 3072 | 128 | 3024 | 331.7MB/s
6144 | 3072 | 128 | 3016 | 326.6MB/s
Best results were always with a thresh=sync_window-20, previous tests with 2 controllers best setting for thresh was sync_window-60.