unraid-tunables-tester.sh - A New Utility to Optimize unRAID md_* Tunables


Recommended Posts

I ran another set of tests, near 10 hours but I had to abort as I lost a disk during the test (no idea if related or not).  I wanted to try and recover rather than keep thrashing the array.

 

The anomaly at Test #43 is when I lost the disk I think, the timing lines up.  Files from this aborted run attached. 

 

In case it's useful, here's my CPU/RAM specs.

Motherboard:	ASUSTeK Computer INC. - P8H67-M EVO
Processor:	Intel® Core™ i5-2500 CPU @ 3.30GHz
HVM:	Enabled
IOMMU:	Enabled
Cache:	L1-Cache = 256 kB (max. capacity 256 kB)
L2-Cache = 1024 kB (max. capacity 1024 kB)
L3-Cache = 6144 kB (max. capacity 6144 kB)
Memory:	16 GB (max. installable capacity 32 GB)
ChannelA-DIMM0 = 4096 MB, 1333 MT/s
ChannelA-DIMM1 = 4096 MB, 1333 MT/s
ChannelB-DIMM0 = 4096 MB, 1333 MT/s
ChannelB-DIMM1 = 4096 MB, 1333 MT/s

 

LongSyncTestReport_2019_08_26_0710.csv LongSyncTestReport_2019_08_26_0710.txt

Link to comment
4 hours ago, dalben said:

I ran another set of tests, near 10 hours but I had to abort as I lost a disk during the test (no idea if related or not).  I wanted to try and recover rather than keep thrashing the array.

 

Array integrity comes first, you did the right thing.

 

4 hours ago, dalben said:

Files from this aborted run attached. 

 

Unfortunately, the slightly different results in this run causes the script to test a lower range in Pass 2, so it didn't retest that magical 177 MB/s test from your earlier run.  Would have been interesting to see it retested, and if it consistently performs better.

Link to comment
3 hours ago, Pauven said:

Array integrity comes first, you did the right thing.

I have yet to try this tool but as more drives have been added to my system, write speeds have definitely taken a hit. I did change a couple of the tunables from the defaults but that was when the system was running on fewer than 8 drives. I'm now at 16 drives so I realize parity calc will slightly slow down writes.

 

That said, I'd like to optimize and try this tool, but one thing I haven't seen is mention of the safety of using the tool with array drives that are all populated with data. As the tool must do writes, how much danger is there that it could cause disk corruption like @dalben had happen? I haven't been able to read the whole thread and don't want to since the latest version is a far cry from what it was when 1st released.

 

I'm sure it's very safe to use, but before I try I just wanted to query how often users have reported drive failure/corruption while running the tool? Also, should I reset the tunables to default before trying the tool? Thanks!

Link to comment

I can answer the last part... no, you don't have to set Tunables to default before using the tool. The first test is performed using "current". It then takes a peek at "default". Only then does it start probing the possible values that may affect the performance.

 

If you want to set the values back to Default at any time, simply go to the Disk Settings page and delete the value in the fields that say "User Defined" and hit Apply. It will automatically reset them to Default for you.

  • Upvote 1
Link to comment
2 hours ago, AgentXXL said:

As the tool must do writes, how much danger is there that it could cause disk corruption like @dalben had happen?

UTT does not do any writes, only reads. 

 

Specifically, it applies a combination of tunables parameters, then initiates a non-correcting (read only) parity check, let's it run for 5 or 10 minutes (depending upon the test length you chose), then aborts the parity check.  It then tries the next set of values and repeats.

 

2 hours ago, AgentXXL said:

I just wanted to query how often users have reported drive failure/corruption while running the tool?

I believe dalben's report might be the very first time a drive failure has been reported during testing.  UTT v4 works the same basic way as the previous versions, so there's years of data behind that statement.

 

In theory, the tests that UTT performs are no more strenuous then a regular parity check.  But anytime you spin up and use your hard drives, especially all of them at once generating max possible heat, you risk a drive failure - same as during a parity check. 

 

Some may feel that the stress is slightly harder than a parity check, as UTT keeps repeating the first 5/10 minutes of the parity check, for dozens of times (minimum 82 times, maximum 139 times), so it keeps all of your drives spinning at their fastest/hottest for the entire test period, unlike a true parity check that would allow smaller drives to complete and spin down as larger drives continue the check.

 

But the stress should be less than hard drive bench-marking, especially tests that do random small file reads/writes and generate lots of head movement.

Edited by Pauven
  • Upvote 2
Link to comment

Thanks @DanielCoffey and @Pauven for the quick responses. I guess I hadn't read up enough on what the tool does, but it was recommended to use since the tunables are what's been recommended to try and improve write performance. As it's only doing reads, I do feel a lot safer using it. My 'server room' is air conditioned so I might just set the air conditioner to run it cooler than normal to try and combat the heat generation of the test.

 

Looking forward to the results!

Link to comment

My monthly parity check completed in another record time for my server, dropping another 12 seconds (haha).

 

image.png.06bea17742442c07ab5fe9235e3233fb.png

 

Even though the UTT v4.1 enhancements resulted in slightly better peak numbers, my server was already well optimized so the additional performance was not impactful.

Link to comment

I did the Long test and used the recommended settings.But I see a drop in transfer speeds when I transfer files from 1 hdd to another in same share using mc (ssh).It dropped from 100 MB/s+ to ~40-50 MB/s.

Unraid 6.x Tunables Tester v4.1 by Pauven

             Tunables Report produced Wed Aug 21 18:02:37 NZST 2019

                              Run on server: 

                           Long Parity Sync Test


Current Values:  md_num_stripes=4096, md_sync_window=2048, md_sync_thresh=2000
                 Global nr_requests=128
                 Disk Specific nr_requests Values:
                    sdj=128, sdi=128, sdg=128, sdh=128, sdf=128, sdm=128, 
                    sdk=128, sde=128, sdc=128, sdd=128, 


--- INITIAL BASELINE TEST OF CURRENT VALUES (1 Sample Point @ 10min Duration)---
Tst | RAM | stri |  win | req | thresh |  MB/s
----------------------------------------------
  1 | 186 | 4096 | 2048 | 128 |  2000  | 141.7 


--- BASELINE TEST OF UNRAID DEFAULT VALUES (1 Sample Point @ 10min Duration)---
Tst | RAM | stri |  win | req | thresh |  MB/s
----------------------------------------------
  1 |  58 | 1280 |  384 | 128 |   192  | 141.7 


 --- TEST PASS 1 (2.5 Hrs - 12 Sample Points @ 10min Duration) ---
Tst | RAM | stri |  win | req | thresh |  MB/s | thresh |  MB/s | thresh |  MB/s
--------------------------------------------------------------------------------
  1 |  34 |  768 |  384 | 128 |   376  | 134.7 |   320  | 140.6 |   192  | 141.7
  2 |  69 | 1536 |  768 | 128 |   760  | 141.4 |   704  | 141.3 |   384  | 141.2
  3 | 139 | 3072 | 1536 | 128 |  1528  | 141.7 |  1472  | 141.8 |   768  | 141.4
  4 | 279 | 6144 | 3072 | 128 |  3064  | 141.7 |  3008  | 141.5 |  1536  | 141.7

 --- TEST PASS 1_LOW (3 Hrs - 15 Sample Points @ 10min Duration) ---
Tst | RAM | stri |  win | req | thresh |  MB/s | thresh |  MB/s | thresh |  MB/s
--------------------------------------------------------------------------------
  1 |   5 |  128 |   64 | 128 |    56  | 141.5 |     0  | 141.4 |    32  | 141.2
  2 |  11 |  256 |  128 | 128 |   120  | 140.6 |    64  | 141.6 |    64  | 141.5
  3 |  17 |  384 |  192 | 128 |   184  | 140.5 |   128  | 141.6 |    96  | 141.6
  4 |  23 |  512 |  256 | 128 |   248  | 140.7 |   192  | 141.7 |   128  | 141.7
  5 |  29 |  640 |  320 | 128 |   312  | 140.4 |   256  | 139.4 |   160  | 141.6

 --- TEST PASS 1_HIGH (40 Min - 3 Sample Points @ 10min Duration)---
Tst | RAM | stri |  win | req | thresh |  MB/s | thresh |  MB/s | thresh |  MB/s
--------------------------------------------------------------------------------
  1 | 558 |12288 | 6144 | 128 |  6136  | 141.4 |  6080  | 141.4 |  3072  | 141.5

 --- TEST PASS 1_VERYHIGH (40 Min - 3 Sample Points @ 10min Duration)---
Tst | RAM | stri |  win | req | thresh |  MB/s | thresh |  MB/s | thresh |  MB/s
--------------------------------------------------------------------------------
  1 | 838 |18432 | 9216 | 128 |  9208  | 141.5 |  9152  | 141.4 |  4608  | 141.4

 --- Using md_sync_window=128 & md_sync_thresh=window-64 for Pass 2 ---

 --- TEST PASS 2 (10 Hrs - 49 Sample Points @ 10min Duration) ---
Tst | RAM | stri |  win | req | thresh |  MB/s
----------------------------------------------
  1 |   0 |   16 |   8 | 128 |     -56  | 121.1
  2 |   1 |   32 |   16 | 128 |     -48  | 132.7
  3 |   2 |   48 |   24 | 128 |     -40  | 134.2
  4 |   2 |   64 |   32 | 128 |     -32  | 111.1
  5 |   3 |   80 |   40 | 128 |     -24  |  91.9
  6 |   4 |   96 |   48 | 128 |     -16  |  89.8
  7 |   5 |  112 |   56 | 128 |     -8  |  97.9
  8 |   5 |  128 |   64 | 128 |     0  | 141.4
  9 |   6 |  144 |   72 | 128 |     8  | 141.4
 10 |   7 |  160 |   80 | 128 |    16  | 141.6
 11 |   8 |  176 |   88 | 128 |    24  | 141.6
 12 |   8 |  192 |   96 | 128 |    32  | 141.5
 13 |   9 |  208 |  104 | 128 |    40  | 141.6
 14 |  10 |  224 |  112 | 128 |    48  | 141.4
 15 |  10 |  240 |  120 | 128 |    56  | 141.6
 16 |  11 |  256 |  128 | 128 |    64  | 141.6
 17 |  12 |  272 |  136 | 128 |    72  | 141.6
 18 |  13 |  288 |  144 | 128 |    80  | 141.7
 19 |  13 |  304 |  152 | 128 |    88  | 141.6
 20 |  14 |  320 |  160 | 128 |    96  | 141.7
 21 |  15 |  336 |  168 | 128 |   104  | 141.7
 22 |  16 |  352 |  176 | 128 |   112  | 141.8
 23 |  16 |  368 |  184 | 128 |   120  | 141.7
 24 |  17 |  384 |  192 | 128 |   128  | 141.8
 25 |  18 |  400 |  200 | 128 |   136  | 141.7
 26 |  18 |  416 |  208 | 128 |   144  | 141.8
 27 |  19 |  432 |  216 | 128 |   152  | 141.8
 28 |  20 |  448 |  224 | 128 |   160  | 141.7
 29 |  21 |  464 |  232 | 128 |   168  | 141.8
 30 |  21 |  480 |  240 | 128 |   176  | 141.6
 31 |  22 |  496 |  248 | 128 |   184  | 141.5
 32 |  23 |  512 |  256 | 128 |   192  | 141.8
 33 |  24 |  528 |  264 | 128 |   200  | 141.1
 34 |  24 |  544 |  272 | 128 |   208  | 141.3
 35 |  25 |  560 |  280 | 128 |   216  | 140.9
 36 |  26 |  576 |  288 | 128 |   224  | 141.3
 37 |  26 |  592 |  296 | 128 |   232  | 141.3
 38 |  27 |  608 |  304 | 128 |   240  | 140.9
 39 |  28 |  624 |  312 | 128 |   248  | 141.1
 40 |  29 |  640 |  320 | 128 |   256  | 141.0

 --- Using fastest result of md_sync_window=128 for Pass 3 ---

 --- TEST PASS 3 (4 Hrs - 18 Sample Points @ 10min Duration) ---
Tst | RAM | stri |  win | req | thresh |  MB/s
----------------------------------------------
 1a |  11 |  256 |  128 | 128 |   127  |  84.9
 1b |  11 |  256 |  128 | 128 |   124  | 115.1
 1c |  11 |  256 |  128 | 128 |   120  | 140.7
 1d |  11 |  256 |  128 | 128 |   116  | 141.4
 1e |  11 |  256 |  128 | 128 |   112  | 141.5
 1f |  11 |  256 |  128 | 128 |   108  | 141.5
 1g |  11 |  256 |  128 | 128 |   104  | 141.6
 1h |  11 |  256 |  128 | 128 |   100  | 141.7
 1i |  11 |  256 |  128 | 128 |    96  | 141.7
 1j |  11 |  256 |  128 | 128 |    92  | 141.6
 1k |  11 |  256 |  128 | 128 |    88  | 141.7
 1l |  11 |  256 |  128 | 128 |    84  | 141.4
 1m |  11 |  256 |  128 | 128 |    80  | 141.7
 1n |  11 |  256 |  128 | 128 |    76  | 141.7
 1o |  11 |  256 |  128 | 128 |    72  | 141.4
 1p |  11 |  256 |  128 | 128 |    68  | 141.2
 1q |  11 |  256 |  128 | 128 |    64  | 141.6
 1r |  11 |  256 |  128 | 128 |    64  | 141.5

The results below do NOT include the Baseline test of current values.

The Fastest settings tested give a peak speed of 141.8 MB/s
     md_sync_window: 176          md_num_stripes: 352
     md_sync_thresh: 112             nr_requests: 128
This will consume 16 MB (170 MB less than your current utilization of 186 MB)

The Thriftiest settings (95% of Fastest) give a peak speed of 141.5 MB/s
     md_sync_window: 64          md_num_stripes: 128
     md_sync_thresh: 56             nr_requests: 128
This will consume 5 MB (181 MB less than your current utilization of 186 MB)

The Recommended settings (99% of Fastest) give a peak speed of 141.5 MB/s
     md_sync_window: 64          md_num_stripes: 128
     md_sync_thresh: 56             nr_requests: 128
This will consume 5 MB (181 MB less than your current utilization of 186 MB)

NOTE: Adding additional drives will increase memory consumption.

In Unraid, go to Settings > Disk Settings to set your chosen parameter values.

Completed: 17 Hrs 30 Min 35 Sec.


NOTE: Use the smallest set of values that produce good results. Larger values
      increase server memory use, and may cause stability issues with Unraid,
      especially if you have any add-ons or plug-ins installed.


System Info:  
              Unraid version 6.7.2
                   md_num_stripes=4096
                   md_sync_window=2048
                   md_sync_thresh=2000
                   nr_requests=128 (Global Setting)
                   sbNumDisks=11
              CPU: AMD Ryzen 7 2700 Eight-Core Processor
              RAM: 32GiB System Memory

Outputting free low memory information...

              total        used        free      shared  buff/cache   available
Mem:       32930436      776632    30546456     1483776     1607348    30277472
Low:       32930436     2383980    30546456
High:             0           0           0
Swap:             0           0           0


                SCSI Host Controllers and Connected Array Drives
 ------------------------------------------------------------------------------

 [0] scsi0	usb-storage 	
 [0:0:0:0]	flash		sda	30.7GB	Ultra Fit

 [1] scsi1	ahci 	
 [1:0:0:0]	parity		sdj	10.0TB	ST10000NM0016-1T

 [2] scsi2	ahci 	

 [3] scsi3	ahci 	

 [4] scsi4	ahci 	

 [5] scsi5	ahci 	

 [6] scsi6	ahci 	
 [6:0:0:0]	disk6		sdk	8.00TB	ST8000DM004-2CX1

 [7] scsi7	ahci 	

 [8] scsi8	ahci 	

 [9] scsi9	mpt3sas 	SAS3008 PCI-Express Fusion-MPT SAS-3
 [9:0:0:0]	cache2		sdb	500GB	WDC WDS500G2B0B-
 [9:0:1:0]	disk8		sdc	10.0TB	WDC WD100EZAZ-11
 [9:0:2:0]	disk9		sdd	10.0TB	WDC WD100EZAZ-11
 [9:0:3:0]	disk7		sde	10.0TB	WDC WD100EZAZ-11
 [9:0:4:0]	disk4		sdf	8.00TB	WDC WD80EMAZ-00W
 [9:0:5:0]	disk2		sdg	8.00TB	WDC WD80EMAZ-00M
 [9:0:6:0]	disk3		sdh	8.00TB	WDC WD80EZAZ-11T
 [9:0:7:0]	disk1		sdi	8.00TB	WDC WD80EMAZ-00M

 [10] scsi10	ahci 	

 [11] scsi11	ahci 	
 [11:0:0:0]	disk5		sdm	2.00TB	WDC WD20EFRX-68E

 [12] scsi12	ahci 	

 [N0] scsiN0	nvme0 	NVMe
 [N:0:4:1]	cache		nvme0n1	500GB	Samsung SSD 970 

 [N1] scsiN1	nvme1 	NVMe
 [N:1:1:1]	cache3		nvme1n1	500GB	WDC WDS500G1B0C-


                             *** END OF REPORT ***

 

Link to comment

I forgot about it being a new month, and having a new automatic parity check.

 

The first tweak was fairly close performance wise to the last regularly scheduled run, but better than the earlier normal scheduled runs. The last tweak using the latest version with a long run has shaved a little over 3 minutes off!

 

Last Tweak: 15 hr, 10 min, 52 secs @ 146.4 MB/s. 

 

First Tweak: 15 hr, 14 min, 7 sec @ 145.9 MB/s

 

Most Recent Historicals (before tweaking):

15 hr, 14 min, 53 sec @ 145.8 MB/s

15 hr, 20 min, 12 sec @ 144.9 MB/s

15 hr, 21 min, 5 sec @ 144.8 MB/s

15 hr, 17 min, 56 sec @ 145.3 MB/s

Link to comment
3 hours ago, DanielCoffey said:

It seems to be there now but here it is anyway...

Hi @DanielCoffey, thanks for lending a helping hand.  Even though it has the same name, for some reason the file you posted has a different size than the original version.  I think it would be wise if you remove the file you posted, just in case.

 

Also, the original file is hosted on the Unraid forum, which has done a decent job of hosting files for years.  Not sure why vekselstrom had an issue downloading, though it seems to have been a temporary issue.  I think it would be best if we keep the download option centralized in the first post, which gives me control over updates.

 

Link to comment
On 9/2/2019 at 6:15 PM, mAYH3M said:

I did the Long test and used the recommended settings.But I see a drop in transfer speeds when I transfer files from 1 hdd to another in same share using mc (ssh).It dropped from 100 MB/s+ to ~40-50 MB/s.

 

I think that is a really interesting finding.  In a disk to disk transfer, you're both reading from and writing to 3 disks simultaneously (4 if you had dual parity), which is a very different workload than just reading from all disks simultaneously.   I'm guessing what happened is that you went so low in memory, that disk to disk transfers were impacted.  I'll have to do some testing on my server and see if that is something I can replicate.

 

You have a server that responds very well to low values, at least as far as parity checks go.  Actually, it seems to respond the same for almost any set of values, achieving around 141 MB/s across the board except for a few edge cases.  For that type of server, you're probably best off just running stock Unraid tunables settings.

Link to comment

 1c | 285 | 8448 | 4224 | 128 |  4216  | 323.3 :D

 

Wish it was that fast!  Any comments on the results?  The original settings were put in when I built the server in 2016.

 

I'm currently looking at moving to 8/10Gb disks, probably shucking the WD elements as the price of reds are ridiculous!

                   Unraid 6.x Tunables Tester v4.1 by Pauven

             Tunables Report produced Thu Sep 19 21:27:49 BST 2019

                              Run on server: Zeus

                          Extra-Long Parity Sync Test


Current Values:  md_num_stripes=4096, md_sync_window=2048, md_sync_thresh=2000
                 Global nr_requests=8
                 Disk Specific nr_requests Values:
                    sdc=8, sdb=8, sdh=8, sdg=8, sdf=8, sde=8, 
                    sdd=8, 


--- INITIAL BASELINE TEST OF CURRENT VALUES (1 Sample Point @ 10min Duration)---
Tst | RAM | stri |  win | req | thresh |  MB/s
----------------------------------------------
  1 | 138 | 4096 | 2048 |   8 |  2000  | 154.7 


--- BASELINE TEST OF UNRAID DEFAULT VALUES (1 Sample Point @ 10min Duration)---
Tst | RAM | stri |  win | req | thresh |  MB/s
----------------------------------------------
  1 |  43 | 1280 |  384 | 128 |   192  | 149.8 


 --- TEST PASS 1 (2.5 Hrs - 12 Sample Points @ 10min Duration) ---
Tst | RAM | stri |  win | req | thresh |  MB/s | thresh |  MB/s | thresh |  MB/s
--------------------------------------------------------------------------------
  1 |  25 |  768 |  384 | 128 |   376  | 151.2 |   320  | 151.1 |   192  | 148.6
  2 |  51 | 1536 |  768 | 128 |   760  | 153.7 |   704  | 153.6 |   384  | 152.3
  3 | 103 | 3072 | 1536 | 128 |  1528  | 155.4 |  1472  | 155.7 |   768  | 153.2
  4 | 207 | 6144 | 3072 | 128 |  3064  | 157.7 |  3008  | 157.8 |  1536  | 156.5

 --- TEST PASS 1_HIGH (40 Min - 3 Sample Points @ 10min Duration)---
Tst | RAM | stri |  win | req | thresh |  MB/s | thresh |  MB/s | thresh |  MB/s
--------------------------------------------------------------------------------
  1 | 414 |12288 | 6144 | 128 |  6136  | 157.7 |  6080  | 157.8 |  3072  | 157.9

 --- TEST PASS 1_VERYHIGH (40 Min - 3 Sample Points @ 10min Duration)---
Tst | RAM | stri |  win | req | thresh |  MB/s | thresh |  MB/s | thresh |  MB/s
--------------------------------------------------------------------------------
  1 | 622 |18432 | 9216 | 128 |  9208  | 157.8 |  9152  | 157.8 |  4608  | 157.9

 --- Using md_sync_window=3072 & md_sync_thresh=window-64 for Pass 2 ---

 --- TEST PASS 2 (10 Hrs - 49 Sample Points @ 10min Duration) ---
Tst | RAM | stri |  win | req | thresh |  MB/s
----------------------------------------------
  1 | 103 | 3072 | 1536 | 128 |  1472  | 156.3
  2 | 108 | 3200 | 1600 | 128 |  1536  | 156.2
  3 | 112 | 3328 | 1664 | 128 |  1600  | 156.4
  4 | 116 | 3456 | 1728 | 128 |  1664  | 156.7
  5 | 121 | 3584 | 1792 | 128 |  1728  | 156.6
  6 | 125 | 3712 | 1856 | 128 |  1792  | 156.6
  7 | 129 | 3840 | 1920 | 128 |  1856  | 157.1
  8 | 133 | 3968 | 1984 | 128 |  1920  | 157.0
  9 | 138 | 4096 | 2048 | 128 |  1984  | 156.9
 10 | 142 | 4224 | 2112 | 128 |  2048  | 157.1
 11 | 146 | 4352 | 2176 | 128 |  2112  | 157.4
 12 | 151 | 4480 | 2240 | 128 |  2176  | 157.2
 13 | 155 | 4608 | 2304 | 128 |  2240  | 157.2
 14 | 159 | 4736 | 2368 | 128 |  2304  | 157.5
 15 | 164 | 4864 | 2432 | 128 |  2368  | 157.5
 16 | 168 | 4992 | 2496 | 128 |  2432  | 157.3
 17 | 172 | 5120 | 2560 | 128 |  2496  | 157.5
 18 | 177 | 5248 | 2624 | 128 |  2560  | 157.7
 19 | 181 | 5376 | 2688 | 128 |  2624  | 157.5
 20 | 185 | 5504 | 2752 | 128 |  2688  | 157.6
 21 | 190 | 5632 | 2816 | 128 |  2752  | 157.7
 22 | 194 | 5760 | 2880 | 128 |  2816  | 157.6
 23 | 198 | 5888 | 2944 | 128 |  2880  | 157.7
 24 | 203 | 6016 | 3008 | 128 |  2944  | 157.6
 25 | 207 | 6144 | 3072 | 128 |  3008  | 157.9
 26 | 211 | 6272 | 3136 | 128 |  3072  | 157.8
 27 | 216 | 6400 | 3200 | 128 |  3136  | 157.6
 28 | 220 | 6528 | 3264 | 128 |  3200  | 157.7
 29 | 224 | 6656 | 3328 | 128 |  3264  | 157.9
 30 | 229 | 6784 | 3392 | 128 |  3328  | 157.7
 31 | 233 | 6912 | 3456 | 128 |  3392  | 157.7
 32 | 237 | 7040 | 3520 | 128 |  3456  | 157.9
 33 | 242 | 7168 | 3584 | 128 |  3520  | 157.8
 34 | 246 | 7296 | 3648 | 128 |  3584  | 157.7
 35 | 250 | 7424 | 3712 | 128 |  3648  | 157.7
 36 | 255 | 7552 | 3776 | 128 |  3712  | 157.9
 37 | 259 | 7680 | 3840 | 128 |  3776  | 157.8
 38 | 263 | 7808 | 3904 | 128 |  3840  | 157.8
 39 | 267 | 7936 | 3968 | 128 |  3904  | 157.9
 40 | 272 | 8064 | 4032 | 128 |  3968  | 157.7
 41 | 276 | 8192 | 4096 | 128 |  4032  | 157.7
 42 | 280 | 8320 | 4160 | 128 |  4096  | 157.8
 43 | 285 | 8448 | 4224 | 128 |  4160  | 158.0
 44 | 289 | 8576 | 4288 | 128 |  4224  | 157.8
 45 | 293 | 8704 | 4352 | 128 |  4288  | 157.8
 46 | 298 | 8832 | 4416 | 128 |  4352  | 157.9
 47 | 302 | 8960 | 4480 | 128 |  4416  | 157.7
 48 | 306 | 9088 | 4544 | 128 |  4480  | 157.8
 49 | 311 | 9216 | 4608 | 128 |  4544  | 157.8

 --- Using fastest result of md_sync_window=4224 for Pass 3 ---

 --- TEST PASS 3 (12 Hrs - 54 Sample Points @ 10min Duration) ---
Tst | RAM | stri |  win | req | thresh |  MB/s
----------------------------------------------
 1a | 285 | 8448 | 4224 | 128 |  4223  | 158.0
 1b | 285 | 8448 | 4224 | 128 |  4220  | 157.8
 1c | 285 | 8448 | 4224 | 128 |  4216  | 323.3
 1d | 285 | 8448 | 4224 | 128 |  4212  | 157.8
 1e | 285 | 8448 | 4224 | 128 |  4208  | 157.8
 1f | 285 | 8448 | 4224 | 128 |  4204  | 157.8
 1g | 285 | 8448 | 4224 | 128 |  4200  | 158.0
 1h | 285 | 8448 | 4224 | 128 |  4196  | 157.8
 1i | 285 | 8448 | 4224 | 128 |  4192  | 157.7
 1j | 285 | 8448 | 4224 | 128 |  4188  | 157.8
 1k | 285 | 8448 | 4224 | 128 |  4184  | 157.9
 1l | 285 | 8448 | 4224 | 128 |  4180  | 157.8
 1m | 285 | 8448 | 4224 | 128 |  4176  | 157.8
 1n | 285 | 8448 | 4224 | 128 |  4172  | 158.0
 1o | 285 | 8448 | 4224 | 128 |  4168  | 157.8
 1p | 285 | 8448 | 4224 | 128 |  4164  | 157.7
 1q | 285 | 8448 | 4224 | 128 |  4160  | 157.8
 1r | 285 | 8448 | 4224 | 128 |  2112  | 157.9
 2a | 285 | 8448 | 4224 |  16 |  4223  | 157.0
 2b | 285 | 8448 | 4224 |  16 |  4220  | 157.0
 2c | 285 | 8448 | 4224 |  16 |  4216  | 157.3
 2d | 285 | 8448 | 4224 |  16 |  4212  | 157.0
 2e | 285 | 8448 | 4224 |  16 |  4208  | 157.1
 2f | 285 | 8448 | 4224 |  16 |  4204  | 157.1
 2g | 285 | 8448 | 4224 |  16 |  4200  | 157.3
 2h | 285 | 8448 | 4224 |  16 |  4196  | 157.0
 2i | 285 | 8448 | 4224 |  16 |  4192  | 157.1
 2j | 285 | 8448 | 4224 |  16 |  4188  | 157.2
 2k | 285 | 8448 | 4224 |  16 |  4184  | 157.1
 2l | 285 | 8448 | 4224 |  16 |  4180  | 157.1
 2m | 285 | 8448 | 4224 |  16 |  4176  | 157.1
 2n | 285 | 8448 | 4224 |  16 |  4172  | 157.3
 2o | 285 | 8448 | 4224 |  16 |  4168  | 157.1
 2p | 285 | 8448 | 4224 |  16 |  4164  | 157.0
 2q | 285 | 8448 | 4224 |  16 |  4160  | 157.1
 2r | 285 | 8448 | 4224 |  16 |  2112  | 157.3
 3a | 285 | 8448 | 4224 |   8 |  4223  | 154.9
 3b | 285 | 8448 | 4224 |   8 |  4220  | 155.0
 3c | 285 | 8448 | 4224 |   8 |  4216  | 154.9
 3d | 285 | 8448 | 4224 |   8 |  4212  | 155.1
 3e | 285 | 8448 | 4224 |   8 |  4208  | 154.8
 3f | 285 | 8448 | 4224 |   8 |  4204  | 155.0
 3g | 285 | 8448 | 4224 |   8 |  4200  | 155.2
 3h | 285 | 8448 | 4224 |   8 |  4196  | 155.0
 3i | 285 | 8448 | 4224 |   8 |  4192  | 155.0
 3j | 285 | 8448 | 4224 |   8 |  4188  | 155.1
 3k | 285 | 8448 | 4224 |   8 |  4184  | 155.2
 3l | 285 | 8448 | 4224 |   8 |  4180  | 153.9
 3m | 285 | 8448 | 4224 |   8 |  4176  | 155.0
 3n | 285 | 8448 | 4224 |   8 |  4172  | 155.2
 3o | 285 | 8448 | 4224 |   8 |  4168  | 154.9
 3p | 285 | 8448 | 4224 |   8 |  4164  | 154.9
 3q | 285 | 8448 | 4224 |   8 |  4160  | 155.0
 3r | 285 | 8448 | 4224 |   8 |  2112  | 155.1

The results below do NOT include the Baseline test of current values.

The Fastest settings tested give a peak speed of 323.3 MB/s
     md_sync_window: 4224          md_num_stripes: 8448
     md_sync_thresh: 4216             nr_requests: 128
This will consume 285 MB (147 MB more than your current utilization of 138 MB)

The Thriftiest settings (95% of Fastest) give a peak speed of 323.3 MB/s
     md_sync_window: 4224          md_num_stripes: 8448
     md_sync_thresh: 4216             nr_requests: 128
This will consume 285 MB (147 MB more than your current utilization of 138 MB)

The Recommended settings (99% of Fastest) give a peak speed of 323.3 MB/s
     md_sync_window: 4224          md_num_stripes: 8448
     md_sync_thresh: 4216             nr_requests: 128
This will consume 285 MB (147 MB more than your current utilization of 138 MB)

NOTE: Adding additional drives will increase memory consumption.

In Unraid, go to Settings > Disk Settings to set your chosen parameter values.

Completed: 21 Hrs 13 Min 42 Sec.


NOTE: Use the smallest set of values that produce good results. Larger values
      increase server memory use, and may cause stability issues with Unraid,
      especially if you have any add-ons or plug-ins installed.


System Info:  Zeus
              Unraid version 6.7.2
                   md_num_stripes=4096
                   md_sync_window=2048
                   md_sync_thresh=2000
                   nr_requests=8 (Global Setting)
                   sbNumDisks=8
              CPU: Intel(R) Xeon(R) CPU E5-2670 0 @ 2.60GHz
              RAM: System Memory
         System Memory

Outputting free low memory information...

              total        used        free      shared  buff/cache   available
Mem:       66000560     1580196    63223520     1143264     1196844    62802068
Low:       66000560     2777040    63223520
High:             0           0           0
Swap:             0           0           0


                SCSI Host Controllers and Connected Array Drives
 ------------------------------------------------------------------------------

 [0] scsi0	usb-storage 	
 [0:0:0:0]	flash		sda	15.3GB	Cruzer Fit

 [1] scsi1	isci 	C602 chipset 4-Port SATA Storage Control Unit
 [1:0:0:0]	cache		sdi	512GB	SanDisk SD8SB8U5

 [2] scsi2	ahci 	

 [3] scsi3	ahci 	

 [4] scsi4	ahci 	

 [5] scsi5	ahci 	

 [6] scsi6	ahci 	

 [7] scsi7	ahci 	

 [8] scsi8	mpt2sas 	SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon]
 [8:0:0:0]	disk1		sdb	4.00TB	ST4000VN000-1H41
 [8:0:1:0]	parity		sdc	4.00TB	ST4000VN000-1H41
 [8:0:2:0]	disk6		sdd	4.00TB	ST4000VN000-1H41
 [8:0:3:0]	disk5		sde	4.00TB	ST4000VN000-1H41
 [8:0:4:0]	disk4		sdf	4.00TB	ST4000VN000-1H41
 [8:0:5:0]	disk3		sdg	4.00TB	ST4000VN000-1H41
 [8:0:6:0]	disk2		sdh	4.00TB	ST4000VN000-1H41


                             *** END OF REPORT ***

 

Edited by reapola
Link to comment

I just had to rebuild a data disk for the second time in less than a week. It finished at a blazing speed of ~50 MB/s both times.🙁

This seems really slow. My parity checks usually average around 80 MB/s which isn't great but better than 50. Am I comparing apples to oranges here? Not sure if these slow speeds are related to the recent changes I've made to tunables, or possibly the fact that the disk is failing, or maybe none of the above.

Link to comment
14 minutes ago, wgstarks said:

I just had to rebuild a data disk for the second time in less than a week. It finished at a blazing speed of ~50 MB/s both times.🙁

This seems really slow. My parity checks usually average around 80 MB/s which isn't great but better than 50. Am I comparing apples to oranges here? Not sure if these slow speeds are related to the recent changes I've made to tunables, or possibly the fact that the disk is failing, or maybe none of the above.

Install the DiskSpeed docker app in my sig and run a benchmark on all your spinners. It will help you identify if you have a drive performing slowly. It might be the drive you had to rebuild, it might be another.

Link to comment

In my experience, a rebuild should be similar in time to a parity check.  The parity check reads from all drives simultaneously, while a rebuild writes to one and reads from all others.  Total bandwidth is close to identical, as is parity calculation load on the CPU.

 

As jbartlett advised, one of your drives could be running slow.

Link to comment
10 minutes ago, jbartlett said:

Install the DiskSpeed docker app in my sig and run a benchmark on all your spinners. It will help you identify if you have a drive performing slowly. It might be the drive you had to rebuild, it might be another.

 

5 minutes ago, Pauven said:

In my experience, a rebuild should be similar in time to a parity check.  The parity check reads from all drives simultaneously, while a rebuild writes to one and reads from all others.  Total bandwidth is close to identical, as is parity calculation load on the CPU.

 

As jbartlett advised, one of your drives could be running slow.

I'll run another speed test and post in that thread. Thanks.

Link to comment
  • 2 weeks later...

Looks like the 6.8 series will present a new set of challenges on tuning...  In particular:

 

Quote

 

md/unraid driver
Introduced "multi-stream" support:

Reads on devices which are not being written should run at full speed.  In addition, if you have set the md_write_method tunable to "reconstruct write", then while writing, if any read streams are detected, the write method is switched to "read/modifywrite".

Parity sync/check should run at full speed by default.

Parity sync/check can be throttled back in presence of other active streams.

The "stripe pool" resource is automatically shared evenly between all active streams.

 

As a result got rid of some Tunables:

md_sync_window

md_sync_thresh

 

and added some tunables:

md_queue_limit

md_sync_limit

 

Please refer to Settings/Disk Settings help text for description of these settings.

 

 

 

 

Edited by BRiT
Link to comment
7 hours ago, BRiT said:

Looks like the 6.8 series will present a new set of challenges on tuning...  In particular:

v.6.8 should autotune and use the best settings and there's not much to manually tune for parity check speed, there are some new tunables for priority tuning, though on my test server it's currently not as fast as v6.7 during a parity check.

 

 

Edited by johnnie.black
  • Like 1
Link to comment
7 hours ago, johnnie.black said:

v.6.8 should autotune and use the best settings and there's not much to manually tune for parity check speed, there are some new tunables for priority tuning, though on my test server it's currently not as fast as v6.7 during a parity check.

 

 

So this is something to report to Limetech and they will have to make the changes required in order to get solid performance on wider variety of hardware?

Link to comment
4 minutes ago, BRiT said:

So this is something to report to Limetech and they will have to make the changes required in order to get solid performance on wider variety of hardware?

From the 6.8 OP

Quote

Remaining issue: some users have reported slower parity sync/check rates for very wide arrays (20+ devices) vs. 6.7 and earlier releases - we are still studying this problem.

Note that my system (21 data, 2 parity) is the same or faster on parity checks

Link to comment
9 minutes ago, Squid said:

From the 6.8 OP

Note that my system (21 data, 2 parity) is the same or faster on parity checks

Was your system performance changed much through changes in these tunables on the older versions? My system performance wasn't drastically changed on older versions, just minor improvements so I suppose I'll be alright.

 

But seeing how vastly different other systems are I would think the new unraid would require some sort of knobs to tune.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.