Pauven

I like this sed fix, thanks! For those that have NVMe drives (and any others who want to share, the more samples the more I can program around variances), I need to see the output from this command: lshw -quiet -short -c storage

Currently, I see zero value in running a dirty pass with UTT. There's a reason we don't take pictures of the stars during the day, as the noise from the sun overpowers the faint starlight, and all you see is blue sky and a big ball of fire. The goal of UTT is to identify the right range of value combinations that works well on a server. If you are running a dirty pass, with events randomly running, then the fastest set of values might be tested during heaviest random load, and end up looking like the slowest combo, making the data worthless. I guarantee you that if you access your array for ANY reason, that the random reads and writes to random sectors on a hard drive, which force the heads to move out of position, will affect speeds. That's just simply physics. I think BRiT's results illustrate that perfectly. How much it affects the speeds is depending upon the nature of the reads/writes (big files, small files, 1 file, 1k files, position on disk platter, etc.) and your hardware/drives and which drive you're writing to, and is nearly impossible to replicate the exact same transactions pass after pass so that all UTT tests are under the exact same conditions. The only way for UTT to provide a comparative analysis of different combinations of values is to do so without any external influence. If you want to take pictures of stars, you want to do it on a dark, moonless night, with no clouds or light pollution. Safe mode is an easy way to turn off all the extraneous noise that is polluting the performance picture, which I why I recommended it as an option for BRiT. Any user that seems random highs/lows should consider running a UTT test in Safe Mode to see if that cleans up the results. "for sure", implying 100% certainty? No. But there are some manual tests you could do to get an idea. Essentially, run a clean (not dirty) UTT test. From that, take a few sets of values: Unraid stock values Thriftiest values Recommended values Fastest values And some values that consume even more memory, beyond the Fastest, that still provide fast speeds Apply each set of values, then start a Parity Check, then perform some read/write performance tests. I'll leave it up to you on how to do those performance tests, though there are plenty of ways to accomplish it. Whatever test you do, you should try to do it in a way that you can repeat it for each combination of Tunables above. Some ideas of tests: You could create a script to generate a random content file (Linux can do this easily), then make a copy of it, then delete it. You could read files from the array, or write files to the array over the network using a network benchmarking tool. You could do the network test from multiple client PC's concurrently for an even higher server load. You could stream a few movies to different PC's while running the parity check, seeing which Tunable settings provide a seamless viewing experience. Depending upon your own performance goals, you could be looking for which set of values provides the fastest parity checks, or which provides the fastest read/write speeds, or which has the best balance of both. In my small family, it's unlikely that I would ever be streaming more than 2 movies concurrently, but maybe you have a big family and lot's of TV's, and want to stream 6 movies concurrently. A real torture test would be to stream multiple movies from the exact same disk. Get 4 streams going, then start a parity check and see how far it progresses in 10 minutes, and monitor the streams for viewing glitches. Change your tunables, reset your streams, then start a new parity check and measure again. I think you could do all this in just a few hours. I would say that this approach is better than trying to run a dirty UTT test. UTT can ONLY tell you which values are good when run clean. But you can then take those values and do some manual load testing with them, to see which ones perform well when the server is being tasked with various concurrent loads. One last idea: I had started programming some read/write tests for UTT v3 which I didn't release. Unraid 6.x broke those tests (I don't remember why, but they somehow got broken), so I removed them from the new UTT v4, but I'm more than happy to post them if someone wants to take a stab at making them work under Unraid 6.x. It's been a while since I looked at them, but I believe that these read/write tests tried to automate some of what I discussed above - running repeatable read/write tests while running parity checks with different Tunable values. Essentially, you would pick a disk in your array to write the file to, pick a size for the file, and the script would write a file of that size using random data (isolating write speed from all other factors), then read it back to test read speed. But to be honest, in testing on my own server, I never found that a server under load responded differently to the Tunables than a parity check on a completely idle server - meaning that the fastest UTT values remained fastest even under load. But that was just on my hardware, and may not be an absolute for every server. For that reason, I don't find trying to tune for heavy server load worth my time, as in my experience tuning for Parity Check speeds works for all loads, on my hardware.

That checks out. Looks like the slowdowns were about every 5 tests. Since you were running 10-minute tests, and there's on average about 30 seconds of overhead on each of your tests (varies by array size and speed), every 5 tests is about 53 minutes. CacheDirs is less of a concern, since by rapidly scanning your directories, you keeping your disks from spinning up, so no disk access actually occurs.

I'm noticing a lot of variability in your results. While the fastest result from Pass 1 (Test 4b), which was retested in Pass 2 (Test 25) both produced 181.2 MB/s (100% consistent, nice), Pass 2 Test 48 (181.8 MB/s) was retested in Pass 3 Test 1q @ 171.2 MB/s. Also, Pass 1 Test 3b @ 153.4 MB/s was retested in Pass 2 Test 1 @ 181.2 MB/s. The way the #'s are jumping around in Pass 2, up/down/up/down, rather than a bell curve, suggests you are experiencing random slowdowns that are not related to the settings. I don't know if you have any Dockers, VM's, or other processes running, or file access occurring that is causing this randomness. You may want to try running this again in safe mode, with all plugins, dockers, VM's disabled, and no file access occurring. I would also imagine that if you ran the test again, as-is with no server changes, you would see different answers for the same tests, due to the randomness observed. You can also probably err on the lower side for values. It looks like your server can hit over 180 MB/s with md_sync_window in the 768 range (Pass 1 Tests 2a/b/c), which is 99% of the fastest and in the Recommended range.

I suppose someone could add an NVMe drive to their array, no? Seems like I remember @johnnie.black having an Unraid server using just SSD's. The --no-nvme option might be a good solution to at least fix this report. As Xaero mentioned, these error messages are harmless and informational only, but unwanted all the same.

For the purposes of Parity Check speeds and the UTT tests, it doesn't really matter. Native Command Queuing (NCQ) is primarily a feature that affect lots of random reads/writes. Imagine you were give a list of 100 random books to check out of your library. If you worked the list top to bottom, you would ping-pong all over the library, running from aisle to aisle, trying to find the books in the listed order. NCQ effectively re-orders the list, so you can grab all the books on aisle 1 before going to aisle 2 and so on. Moving the drive heads around on spinning disk platters takes time, so you want to minimize movements between reads/writes for best performance, which NCQ does. So it can make random operations faster - especially if there are a lot of them. I'm not sure what you'd have to do to generate this type of load in a home environment, this is more of an enterprise feature. The Parity Check, on the other hand, is like being tasked to read all the books in the library, one after another, without skipping any, going shelf by shelf, aisle by aisle. I think it might be possible that NCQ can actually make sequential, non-random tasks slower, as there is some overhead in the NCQ processing, but this might not be the issue it was when NCQ first came on the market.

Please run lsscsi -st from the command line and paste the results here.

Just an FYI: Getlshw is depreciated on Unraid 6.x. Even back on Unraid 6.2, 3 years ago, I made a note that it didn't seem necessary. I've been thinking of removing it, but I always have this fear that as soon as I remove it someone will need it. Regardless, your declaration order should work fine. I previously added a lot of debug echo statements to the GetDisks and DiskReport, to troubleshoot the variables and arrays as they were processing. That's how I got it working on my server. Of course, those echo statements are long gone now, sorry.

No worries. I was mainly just curious if they were around 16.5 hours. A pre-clear read on a single disk should operate at 100% speed (in theory), so if your parity check completed in the same time, then it is operating at 100% maximum speed. I really have no idea if a parity check could complete in the same time as a pre-clear read pass (parity check overhead?, bandwith limitations for all drives concurrent?), but I'm fairly confident in saying that a parity check could never be faster than a pre-clear read pass. So it makes a nice benchmark to compare against.

Though I have a NVMe cache drive, lsscsi -st doesn't even show it on my server. Certainly makes development harder when you can't even simulate all the possibilities. root@Tower:/boot/utt# lsscsi -st [0:0:0:0] disk usb:1-5:1.0 /dev/sda 4.00GB [12:0:0:0] disk sas:0x0000000000000000 /dev/sdb 3.00TB [12:0:1:0] disk sas:0x0100000000000000 /dev/sdc 3.00TB [12:0:2:0] disk sas:0x0200000000000000 /dev/sdd 3.00TB [12:0:3:0] disk sas:0x0300000000000000 /dev/sde 3.00TB [12:0:4:0] disk sas:0x0400000000000000 /dev/sdf 8.00TB [12:0:5:0] disk sas:0x0700000000000000 /dev/sdg 8.00TB [13:0:0:0] disk sas:0x0000000000000000 /dev/sdh 8.00TB [13:0:1:0] disk sas:0x0100000000000000 /dev/sdi 3.00TB [13:0:2:0] disk sas:0x0200000000000000 /dev/sdj 3.00TB [13:0:3:0] disk sas:0x0300000000000000 /dev/sdk 3.00TB [13:0:4:0] disk sas:0x0400000000000000 /dev/sdl 3.00TB [13:0:5:0] disk sas:0x0500000000000000 /dev/sdm 3.00TB [13:0:6:0] disk sas:0x0600000000000000 /dev/sdn 3.00TB [13:0:7:0] disk sas:0x0700000000000000 /dev/sdo 3.00TB [14:0:0:0] disk sas:0x0000000000000000 /dev/sdp 3.00TB [14:0:1:0] disk sas:0x0100000000000000 /dev/sdq 3.00TB [14:0:2:0] disk sas:0x0200000000000000 /dev/sdr 3.00TB [14:0:3:0] disk sas:0x0300000000000000 /dev/sds 3.00TB [14:0:4:0] disk sas:0x0400000000000000 /dev/sdt 3.00TB [14:0:5:0] disk sas:0x0500000000000000 /dev/sdu 3.00TB [14:0:6:0] disk sas:0x0600000000000000 /dev/sdv 4.00TB [14:0:7:0] disk sas:0x0700000000000000 /dev/sdw 4.00TB

\\server\flash\preclear_reports or /boot/preclear_reports Thanks!

Below are my initial results from UTT v4.1. The main difference here is that Pass 2 now uses the lowest md_sync_window that reaches at least 99.8% of the fastest speed in Pass 1, and Pass 3 is using the lowest md_sync_window that reaches at least 99.8% of the fastest speed but looking at both Pass 1 and Pass 2 results to find it. I think I'm happy with using the 99.8%+ value for Pass 2. The fastest peak speed I've ever seen on my server is 139.5 MB/s, and Pass 2 still hits this a few times (and at lower values, nice!), so I think the new logic helped it find the leading edge of the absolute peak. I'm less convinced that using the 99.8%+ value for Pass 3 was the right choice, as it ends up spending quite a bit of time testing a value (2944 @ 139.3 MB/s) that wasn't the fastest (4032 @ 139.5 MB/s). Plus, the final results still show 4032 as the Fastest, so having Pass 3 test a lower value seems irrelevant in the final results, and 2944 wasn't even included in the final results, making Pass 3 seem pointless. I'm thinking the right path here is to use the 99.8%+ value for Pass 2, but the absolute Fastest value for Pass 3. Thoughts? --- INITIAL BASELINE TEST OF CURRENT VALUES (1 Sample Point @ 10min Duration)--- Tst | RAM | stri | win | req | thresh | MB/s ---------------------------------------------- 1 | 679 | 7680 | 3840 | 128 | 3800 | 139.5 --- BASELINE TEST OF UNRAID DEFAULT VALUES (1 Sample Point @ 10min Duration)--- Tst | RAM | stri | win | req | thresh | MB/s ---------------------------------------------- 1 | 113 | 1280 | 384 | 128 | 192 | 134.7 --- TEST PASS 1 (2.5 Hrs - 12 Sample Points @ 10min Duration) --- Tst | RAM | stri | win | req | thresh | MB/s | thresh | MB/s | thresh | MB/s -------------------------------------------------------------------------------- 1 | 67 | 768 | 384 | 128 | 376 | 41.5 | 320 | 135.4 | 192 | 134.6 2 | 135 | 1536 | 768 | 128 | 760 | 61.6 | 704 | 136.3 | 384 | 135.3 3 | 271 | 3072 | 1536 | 128 | 1528 | 84.9 | 1472 | 137.5 | 768 | 136.2 4 | 543 | 6144 | 3072 | 128 | 3064 | 130.2 | 3008 | 139.3 | 1536 | 137.5 --- TEST PASS 1_HIGH (40 Min - 3 Sample Points @ 10min Duration)--- Tst | RAM | stri | win | req | thresh | MB/s | thresh | MB/s | thresh | MB/s -------------------------------------------------------------------------------- 1 |1086 |12288 | 6144 | 128 | 6136 | 139.3 | 6080 | 139.4 | 3072 | 139.3 --- TEST PASS 1_VERYHIGH (40 Min - 3 Sample Points @ 10min Duration)--- Tst | RAM | stri | win | req | thresh | MB/s | thresh | MB/s | thresh | MB/s -------------------------------------------------------------------------------- 1 |1630 |18432 | 9216 | 128 | 9208 | 139.2 | 9152 | 138.0 | 4608 | 138.5 --- Using md_sync_window=3072 & md_sync_thresh=window-64 for Pass 2 --- --- TEST PASS 2 (10 Hrs - 49 Sample Points @ 10min Duration) --- Tst | RAM | stri | win | req | thresh | MB/s ---------------------------------------------- 1 | 271 | 3072 | 1536 | 128 | 1472 | 137.5 2 | 283 | 3200 | 1600 | 128 | 1536 | 137.6 3 | 294 | 3328 | 1664 | 128 | 1600 | 137.6 4 | 305 | 3456 | 1728 | 128 | 1664 | 137.9 5 | 317 | 3584 | 1792 | 128 | 1728 | 137.8 6 | 328 | 3712 | 1856 | 128 | 1792 | 137.8 7 | 339 | 3840 | 1920 | 128 | 1856 | 138.0 8 | 350 | 3968 | 1984 | 128 | 1920 | 138.2 9 | 362 | 4096 | 2048 | 128 | 1984 | 138.1 10 | 373 | 4224 | 2112 | 128 | 2048 | 138.3 11 | 384 | 4352 | 2176 | 128 | 2112 | 138.5 12 | 396 | 4480 | 2240 | 128 | 2176 | 138.4 13 | 407 | 4608 | 2304 | 128 | 2240 | 138.5 14 | 418 | 4736 | 2368 | 128 | 2304 | 138.8 15 | 430 | 4864 | 2432 | 128 | 2368 | 138.6 16 | 441 | 4992 | 2496 | 128 | 2432 | 138.4 17 | 452 | 5120 | 2560 | 128 | 2496 | 139.0 18 | 464 | 5248 | 2624 | 128 | 2560 | 138.8 19 | 475 | 5376 | 2688 | 128 | 2624 | 138.5 20 | 486 | 5504 | 2752 | 128 | 2688 | 139.2 21 | 498 | 5632 | 2816 | 128 | 2752 | 139.0 22 | 509 | 5760 | 2880 | 128 | 2816 | 139.2 23 | 520 | 5888 | 2944 | 128 | 2880 | 139.3 24 | 532 | 6016 | 3008 | 128 | 2944 | 139.2 25 | 543 | 6144 | 3072 | 128 | 3008 | 139.3 26 | 554 | 6272 | 3136 | 128 | 3072 | 139.4 27 | 566 | 6400 | 3200 | 128 | 3136 | 139.2 28 | 577 | 6528 | 3264 | 128 | 3200 | 139.4 29 | 588 | 6656 | 3328 | 128 | 3264 | 139.3 30 | 600 | 6784 | 3392 | 128 | 3328 | 139.2 31 | 611 | 6912 | 3456 | 128 | 3392 | 139.4 32 | 622 | 7040 | 3520 | 128 | 3456 | 139.4 33 | 634 | 7168 | 3584 | 128 | 3520 | 139.4 34 | 645 | 7296 | 3648 | 128 | 3584 | 139.4 35 | 656 | 7424 | 3712 | 128 | 3648 | 139.4 36 | 668 | 7552 | 3776 | 128 | 3712 | 139.3 37 | 679 | 7680 | 3840 | 128 | 3776 | 139.4 38 | 690 | 7808 | 3904 | 128 | 3840 | 139.4 39 | 701 | 7936 | 3968 | 128 | 3904 | 139.4 40 | 713 | 8064 | 4032 | 128 | 3968 | 139.5 41 | 724 | 8192 | 4096 | 128 | 4032 | 139.4 42 | 735 | 8320 | 4160 | 128 | 4096 | 139.4 43 | 747 | 8448 | 4224 | 128 | 4160 | 139.5 44 | 758 | 8576 | 4288 | 128 | 4224 | 139.3 45 | 769 | 8704 | 4352 | 128 | 4288 | 139.4 46 | 781 | 8832 | 4416 | 128 | 4352 | 139.5 47 | 792 | 8960 | 4480 | 128 | 4416 | 139.4 48 | 803 | 9088 | 4544 | 128 | 4480 | 139.5 49 | 815 | 9216 | 4608 | 128 | 4544 | 139.5 --- Using md_sync_window=2944 for Pass 3 --- --- TEST PASS 3 (4 Hrs - 18 Sample Points @ 10min Duration) --- Tst | RAM | stri | win | req | thresh | MB/s ---------------------------------------------- 1a | 520 | 5888 | 2944 | 128 | 2943 | 125.9 1b | 520 | 5888 | 2944 | 128 | 2940 | 126.6 1c | 520 | 5888 | 2944 | 128 | 2936 | 123.5 1d | 520 | 5888 | 2944 | 128 | 2932 | 123.5 1e | 520 | 5888 | 2944 | 128 | 2928 | 131.8 1f | 520 | 5888 | 2944 | 128 | 2924 | 137.3 1g | 520 | 5888 | 2944 | 128 | 2920 | 138.1 1h | 520 | 5888 | 2944 | 128 | 2916 | 139.1 1i | 520 | 5888 | 2944 | 128 | 2912 | 139.3 1j | 520 | 5888 | 2944 | 128 | 2908 | 139.1 1k | 520 | 5888 | 2944 | 128 | 2904 | 139.2 1l | 520 | 5888 | 2944 | 128 | 2900 | 139.3 1m | 520 | 5888 | 2944 | 128 | 2896 | 139.1 1n | 520 | 5888 | 2944 | 128 | 2892 | 139.3 1o | 520 | 5888 | 2944 | 128 | 2888 | 138.3 1p | 520 | 5888 | 2944 | 128 | 2884 | 138.9 1q | 520 | 5888 | 2944 | 128 | 2880 | 139.3 1r | 520 | 5888 | 2944 | 128 | 1472 | 137.6 The results below do NOT include the Baseline test of current values. The Fastest settings tested give a peak speed of 139.5 MB/s md_sync_window: 4032 md_num_stripes: 8064 md_sync_thresh: 3968 nr_requests: 128 This will consume 713 MB (34 MB more than your current utilization of 679 MB) The Thriftiest settings (95% of Fastest) give a peak speed of 135.4 MB/s md_sync_window: 384 md_num_stripes: 768 md_sync_thresh: 320 nr_requests: 128 This will consume 67 MB (612 MB less than your current utilization of 679 MB) The Recommended settings (99% of Fastest) give a peak speed of 138.2 MB/s md_sync_window: 1984 md_num_stripes: 3968 md_sync_thresh: 1920 nr_requests: 128 This will consume 350 MB (329 MB less than your current utilization of 679 MB) NOTE: Adding additional drives will increase memory consumption. In Unraid, go to Settings > Disk Settings to set your chosen parameter values. Completed: 15 Hrs 20 Min 40 Sec.

With 32GB of RAM, I'd say go with the fastest. Looks like your current values weren't too far off the mark, performance wise, so I'm not sure if you'll see a performance improvement or not. Any reason why the disk report at the end is showing you have a 512 GB Parity drive, but 3 & 6 TB data drives?

Awesome, thanks! Maybe next time you can run the Xtra-Long version, which includes an extra 36 test points evaluating the nr_requests values. Would be nice to know if there are any servers that like low nr_request values after everything else is tuned.

Hmmm, I don't know. Typo? Copy&Paste relic? I agree it looks wrong. Do you think that was the problem?

Wow, that's better than I anticipated. Congrats! Just wondering, how long does it take to do one pass of a pre-clear (i.e. the final read) for one of your disks?

I've implemented a new function to use the lowest md_sync_window that is at least 99.8% of max speed for the next test pass. Early testing is that it is working as expected. I'll test it overnight before releasing UTT v4.1. Thanks for the suggestion, which I used with a slight modification. I'm now comparing to an existing test result (if it exists), and only replace it if the new result is faster. This will be in UTT v4.1 too. No pressure, and I appreciate the help, but thought I should mention that I would like to include this fix in v4.1 if possible. My eyes start swimming when I look at this code section, and I'm having a hard time seeing what I'm doing wrong. I had a similar issue on my own server, and I reworked the code and thought it was fixed. My previous issue was that I was hard-coding the substring of the bus ID, and with my server now having 14 SCSI buses, the transition from 1-digit IDs to 2-digit IDs was throwing off the logic. So I modified the substring of the bus ID to work regardless of the string length. I don't see how that would be affecting Daniel's report, though, since all of his SCSI ID's are single digit. There was also an issue with the lsscsi -H output and the $SCSIs array variable under Unraid v6.6. Back when I first wrote the GetDisks function under Unraid v6.2, the first 3 entries were always strings I needed to skip. But under Unraid v6.6, somehow the array order got reversed, and now I have to skip the last 3 values. That's why I have that $maxentry-3 calculation in the DiskReport function. I've noticed that before I did the $maxentry fix ( eval Disks=\${$SCSI[@]:0:$maxentry} ), I was getting similar results to what Daniel is getting, so perhaps the bug in my logic is related to the $maxentry.

That sounds good. If I combine that with the two-pass logic that comes up with the Thrifty/Recommended values, this should be trivial. First I scan for the fastest speed, then I scan for the lowest value that achieves 99.5% or 99.9% of fastest measured speed. In DanielCoffey's test, that would have resulted in 6144 being used for Pass 2 (which is what I would have picked) instead of 9216. A threshold of 99.4% would have been too low, as that would have picked up the 3072 @ 161.6 MB/s result from Pass 1. I think 99.8% is the lowest I would want to go.

You're welcome! With lows of 289 MB/s, and highs of 602 MB/s, this VM server definitely shows like it will respond well to tuning. Though with only 21 GB of virtual storage (SSD?), not much point in tuning... 🤔 I don't know how closely your production box matches your VM, so you should always run the Short test on each box the first time just to see if it responds to changing values. Some servers produce the same speeds no matter what values are used, and for them there's no need to try to tune, it would just be a waste of time and electricity.

Good catch! Luckily that's only the Notification subject line, so it doesn't affect the test results. Easy fix. I briefly looked at alternatives, but I find that the giant block of echos makes it slightly easier for me to do text alignment, and to find text output statements. It didn't seem worth the effort for me to clean it up. It doesn't affect script performance or functionality, so it really only matters to the rare few people who poke their head under the hood. Like I said before, Bash is not my forte. I haven't touched Bash in 3 years, since I last worked on this script, and this might be the last time I ever touch it. If this ever gets converted into a plug-in, it would probably need to be rewritten in PHP, so one less reason to spend effort on cosmetic code enhancements.

Thanks! Bus position. I want to show which drives are connected to each controller. For example, here's mine: SCSI Host Controllers and Connected Drives -------------------------------------------------- [0] scsi0 usb-storage - [0:0:0:0] flash sda 4.00GB Patriot Memory [1] scsi1 ahci - [2] scsi2 ahci - [3] scsi3 ahci - [4] scsi4 ahci - [5] scsi5 ahci - [6] scsi6 ahci - [7] scsi7 ahci - [8] scsi8 ahci - [9] scsi9 ahci - [10] scsi10 ahci - [11] scsi11 ahci - [12] scsi12 mvsas - HighPoint Technologies, Inc. [12:0:0:0] disk17 sdb 3.00TB WDC WD30EFRX-68A [12:0:1:0] disk18 sdc 3.00TB WDC WD30EFRX-68A [12:0:2:0] disk19 sdd 3.00TB WDC WD30EFRX-68E [12:0:3:0] disk20 sde 3.00TB WDC WD30EFRX-68E [12:0:4:0] parity2 sdf 8.00TB HGST HUH728080AL [12:0:5:0] parity sdg 8.00TB HGST HUH728080AL [13] scsi13 mvsas - HighPoint Technologies, Inc. [13:0:0:0] disk1 sdh 8.00TB HGST HUH728080AL [13:0:1:0] disk2 sdi 3.00TB WDC WD30EFRX-68A [13:0:2:0] disk3 sdj 3.00TB WDC WD30EFRX-68E [13:0:3:0] disk4 sdk 3.00TB WDC WD30EFRX-68A [13:0:4:0] disk5 sdl 3.00TB WDC WD30EFRX-68A [13:0:5:0] disk6 sdm 3.00TB WDC WD30EFRX-68A [13:0:6:0] disk7 sdn 3.00TB WDC WD30EFRX-68A [13:0:7:0] disk8 sdo 3.00TB WDC WD30EFRX-68A [14] scsi14 mvsas - HighPoint Technologies, Inc. [14:0:0:0] disk9 sdp 3.00TB WDC WD30EFRX-68A [14:0:1:0] disk10 sdq 3.00TB WDC WD30EFRX-68A [14:0:2:0] disk11 sdr 3.00TB WDC WD30EFRX-68A [14:0:3:0] disk12 sds 3.00TB WDC WD30EFRX-68A [14:0:4:0] disk13 sdt 3.00TB WDC WD30EFRX-68A [14:0:5:0] disk14 sdu 3.00TB WDC WD30EFRX-68E [14:0:6:0] disk15 sdv 4.00TB ST4000VN000-1H41 [14:0:7:0] disk16 sdw 4.00TB ST4000VN000-1H41

I just noticed something else in your results. For the most part, 162.4 MB/s is the same as 162.5 MB/s, you're not going to notice the difference. But the script searches for the highest value, and so it hones in on 162.5. In Pass 2, it hit 162.5 at both md_sync_window 6400 (Test 3) and 7296 (Test 10) (plus several other even higher md_sync_window values). It used the lowest md_sync_window that hit the peak speed in Pass 3, so 6400. In Pass 3, md_sync_window 6400 was retested 16 times with different md_sync_thresh values, and the highest peak speed reached was 162.4. Since the exact same values from Pass 2 Test 3 were retested in Pass 3 Test 1r, the new 162.4 result ended up replacing 162.5 in the results table. Then, in the very final step of the interface, all of the results were scanned again to find the fastest result. Since the 6400 result was now 162.4 (Pass 3 replaced the Pass 2 value), the fastest became 7296. This is a side effect of how the results are stored and that the script does retest a few combos more than once. While I've thought about this issue, I've never figured out a good solution. Long story short, md_sync_window 6144 @ 162.4 MB/s is every bit as fast as 7296 @ 162.5 MB/s, and it has a significantly lower memory utilization. In fact, you can probably go even lower than 6144, as it looks like your server hits peak speeds between 3072 and 6144, but the script didn't test those because 9216 happened to get lucky and spit out a 162.5, so the script targeted the highest range.

Excellent! The previous UTT v2.2 script didn't test md_sync_thresh at all, it just left it at the current value (probably 192) for the entire test. UTT v4.0 for Unraid 6.x replaces the old md_write_limit tests with md_sync_thresh tests. As you can see from your results, it makes a big difference in peak speeds. I just did some napkin math, and by my estimate bumping up peak speeds from 156 to 162 MB/s might drop your parity check time by around 3 minutes. The jump from 143 MB/s stock settings is much larger, probably a 30 minute speedup. I look forward to you posting your new parity check times. I think the Thrifty settings are very interesting. Where stock Unraid settings only net you around 143 MB/s, the Thrifty settings get you over 156 MB/s with lower memory utilization than stock! The Thriftiest settings (95% of Fastest) give a peak speed of 156.4 MB/s md_sync_window: 384 md_num_stripes: 768 md_sync_thresh: 376 nr_requests: 128 This will consume 22 MB (16 MB less than your current utilization of 38 MB) I do see a bug in the report output, at the end where the drives are listed. The sdb drive (7:0:7:0) should have been listed at the very bottom, but somehow got listed at the top. And drives sdf and sdg got listed twice. None of this affects the test, it's just extra info for the report. It just bugs me that I thought I had that sorting worked out. SCSI Host Controllers and Connected Drives -------------------------------------------------- [0] scsi0 usb-storage - [7:0:7:0] parity sdb 8.00TB WDC WD80EFZX-68U [1] scsi1 ahci - [2] scsi2 ahci - [3] scsi3 ahci - [7:0:4:0] disk3 sdf 8.00TB WDC WD80EFZX-68U [4] scsi4 ahci - [7:0:5:0] disk4 sdg 8.00TB WDC WD80EFZX-68U [5] scsi5 ahci - [6] scsi6 ahci - [7] scsi7 mpt2sas - SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] [7:0:1:0] parity2 sdc 8.00TB WDC WD80EFZX-68U [7:0:2:0] disk1 sdd 8.00TB WDC WD80EFZX-68U [7:0:3:0] disk2 sde 8.00TB WDC WD80EFZX-68U [7:0:4:0] disk3 sdf 8.00TB WDC WD80EFZX-68U [7:0:5:0] disk4 sdg 8.00TB WDC WD80EFZX-68U [7:0:6:0] disk5 sdh 8.00TB WDC WD80EFZX-68U Thanks for sharing! Paul

I think that would make be a negligible improvement. Considering the test you're running is somewhere in the neighborhood of 16-28 hours, reducing run time by 10 minutes isn't going to make much of a difference. Besides, another way of looking at this is that by running the exact same test back-to-back, you get an idea of the accuracy of the tests. You've already established that you have an accuracy no greater than +/- 0.2 MB/s. There's a few places in the script that the exact same test gets rerun (i.e. fastest from Pass 1 = Test 25 from Pass 2, and the fastest from Pass 2 will be repeated once in Pass 3). There's value to being able to see how consistent the results are when running the exact same test more than once, as that's the only way you can tell what is just noise and what is a true speed difference. Otherwise you might put too much importance in a 0.2 MB/s speed increase from one combo to another.

Bingo.