unraid-tunables-tester.sh - A New Utility to Optimize unRAID md_* Tunables


Recommended Posts

I'll check again.  Balls to the Wall, I was at 7:35 on RC16c, and my Best Bang was 7:40.  I had no idea your times had climbed back up from sub 7:30's.

 

It's definitely frustrating.  On an earlier release (I think it was ~ RC13 or 14) -- back when you and I were both running a lot of tests to tune things (before your very handy script), I had the time down to just under 7:30.    Then it jumped to ~ 7:40 with the release after that;  and jumped a good bit when the new kernel was introduced in RC15.    I was also playing a little bit with the tunables, so I may have changed some relationship that I didn't realize.    With the B4B setting after running your script (v2) I got back under 8 hrs, which I was quite glad to see [i suspect my 7:57 vs. your 7:40 is probably due to my trusty little Atom not having as much horsepower as your CPU] ... but then it jumped back up with v5.0 !!  I've seen other reports of increased times with v5.0 as well, so SOMETHING is going on, although I'm not sure just what.

 

I'm fine with ~ 8 hr checks ... it's just frustrating to know that 7:30 is POSSIBLE, and I can't get it !!

 

... it would definitely be interesting to see what your time is with v5.0

Link to comment

Are they plugged in to PCIe x4 (or greater) slots?

 

Man! I was hoping I wouldn't have to open up the tower since I decided to upgrade the SSD (external drive bays).... *snerk* I'm pretty sure they are, I wouldn't short-change a card in a slot.

 

-John

 

ETA: Specs report the slots as PCIe 2.0 x16

Link to comment

Are they plugged in to PCIe x4 (or greater) slots?

 

Man! I was hoping I wouldn't have to open up the tower since I decided to upgrade the SSD (external drive bays).... *snerk* I'm pretty sure they are, I wouldn't short-change a card in a slot.

 

-John

 

What motherboard do you have?    I'm sure if the board has enough > x1 slots you used them;  but many boards only have one x16 slot and the rest are x1 ... and those cards have a slot that will allow them to be used in an x1 slot (with, of course, a significant reduction in their performance, since they'd only have 1 PCIe lane to use).

 

Link to comment

... it would definitely be interesting to see what your time is with v5.0

 

I'm running a parity check now, will know in the morning, but so far it looks the same, maybe a smidge faster.

 

Since we have the same hard drives, I'm guessing it might be hd controller driver related.  What driver is used on your box?  MVSAS is used on mine.  Maybe there's been some changes with different kernel releases.  Unless your CPU's are peaking, I don't think CPU horsepower has much affect.  Any chance your memory speeds changed on you?  I've had some memory modules that don't always run their rated speed, and a reboot might bring me up in a slower speed.

 

My Balls to the Wall and Best Bang values inched up a bit, as did the reported speeds, but for the most part it looks similar to earlier test runs.

 

Tunables Report from  unRAID Tunables Tester v2.2 by Pauven

Test | num_stripes | write_limit | sync_window |   Speed 
--- FULLY AUTOMATIC TEST PASS 1 (Rough - 20 Sample Points @ 3min Duration)---
   1  |    1408     |     768     |     512     | 105.7 MB/s 
   2  |    1536     |     768     |     640     | 120.5 MB/s 
   3  |    1664     |     768     |     768     | 128.7 MB/s 
   4  |    1920     |     896     |     896     | 132.3 MB/s 
   5  |    2176     |    1024     |    1024     | 135.0 MB/s 
   6  |    2560     |    1152     |    1152     | 135.8 MB/s 
   7  |    2816     |    1280     |    1280     | 136.7 MB/s 
   8  |    3072     |    1408     |    1408     | 137.0 MB/s 
   9  |    3328     |    1536     |    1536     | 137.3 MB/s 
  10  |    3584     |    1664     |    1664     | 137.7 MB/s 
  11  |    3968     |    1792     |    1792     | 137.7 MB/s 
  12  |    4224     |    1920     |    1920     | 137.8 MB/s 
  13  |    4480     |    2048     |    2048     | 138.1 MB/s 
  14  |    4736     |    2176     |    2176     | 138.2 MB/s 
  15  |    5120     |    2304     |    2304     | 138.4 MB/s 
  16  |    5376     |    2432     |    2432     | 138.7 MB/s 
  17  |    5632     |    2560     |    2560     | 138.8 MB/s 
  18  |    5888     |    2688     |    2688     | 138.8 MB/s 
  19  |    6144     |    2816     |    2816     | 139.1 MB/s 
  20  |    6528     |    2944     |    2944     | 139.1 MB/s 
--- Targeting Fastest Result of md_sync_window 2816 bytes for Final Pass ---
--- FULLY AUTOMATIC TEST PASS 2 (Final - 16 Sample Points @ 4min Duration)---
  21  |    5984     |    2696     |    2696     | 139.0 MB/s 
  22  |    6008     |    2704     |    2704     | 138.9 MB/s 
  23  |    6024     |    2712     |    2712     | 139.0 MB/s 
  24  |    6040     |    2720     |    2720     | 138.8 MB/s 
  25  |    6056     |    2728     |    2728     | 139.1 MB/s 
  26  |    6080     |    2736     |    2736     | 139.0 MB/s 
  27  |    6096     |    2744     |    2744     | 138.9 MB/s 
  28  |    6112     |    2752     |    2752     | 139.1 MB/s 
  29  |    6128     |    2760     |    2760     | 139.0 MB/s 
  30  |    6144     |    2768     |    2768     | 139.0 MB/s 
  31  |    6168     |    2776     |    2776     | 138.9 MB/s 
  32  |    6184     |    2784     |    2784     | 139.1 MB/s 
  33  |    6200     |    2792     |    2792     | 138.9 MB/s 
  34  |    6216     |    2800     |    2800     | 139.0 MB/s 
  35  |    6240     |    2808     |    2808     | 139.1 MB/s 
  36  |    6256     |    2816     |    2816     | 139.2 MB/s 

Completed: 2 Hrs 6 Min 55 Sec.

Best Bang for the Buck: Test 7 with a speed of 136.7 MB/s

     Tunable (md_num_stripes): 2816
     Tunable (md_write_limit): 1280
     Tunable (md_sync_window): 1280

These settings will consume 198MB of RAM on your hardware.

Unthrottled values for your server came from Test 36 with a speed of 139.2 MB/s

     Tunable (md_num_stripes): 6256
     Tunable (md_write_limit): 2816
     Tunable (md_sync_window): 2816

These settings will consume 439MB of RAM on your hardware.
This is 25MB more than your current utilization of 414MB.

Link to comment

Well I was hoping it would be a bit better than this but I guess it's not too bad. Any reason I shouldn't consider the slightly higher speed of run 23? Also any thoughts on what i could do to increase this value? I am assuming my 4x 2tb wd green and my 2x 2tb 5900rpm seagates are keeping the speeds down?

 

 

Tunables Report from  unRAID Tunables Tester v2.2 by Pauven

NOTE: Use the smallest set of values that produce good results. Larger values
      increase server memory use, and may cause stability issues with unRAID,
      especially if you have any add-ons or plug-ins installed.

Test | num_stripes | write_limit | sync_window |   Speed 
--- FULLY AUTOMATIC TEST PASS 1 (Rough - 20 Sample Points @ 3min Duration)---
   1  |    1408     |     768     |     512     | 120.5 MB/s 
   2  |    1536     |     768     |     640     | 120.8 MB/s 
   3  |    1664     |     768     |     768     | 121.2 MB/s 
   4  |    1920     |     896     |     896     | 121.4 MB/s 
   5  |    2176     |    1024     |    1024     | 121.6 MB/s 
   6  |    2560     |    1152     |    1152     | 121.7 MB/s 
   7  |    2816     |    1280     |    1280     | 121.9 MB/s 
   8  |    3072     |    1408     |    1408     | 121.8 MB/s 
   9  |    3328     |    1536     |    1536     | 121.9 MB/s 
  10  |    3584     |    1664     |    1664     | 121.9 MB/s 
  11  |    3968     |    1792     |    1792     | 121.9 MB/s 
  12  |    4224     |    1920     |    1920     | 121.8 MB/s 
  13  |    4480     |    2048     |    2048     | 121.8 MB/s 
  14  |    4736     |    2176     |    2176     | 121.8 MB/s 
  15  |    5120     |    2304     |    2304     | 121.9 MB/s 
  16  |    5376     |    2432     |    2432     | 121.9 MB/s 
  17  |    5632     |    2560     |    2560     | 121.9 MB/s 
  18  |    5888     |    2688     |    2688     | 121.9 MB/s 
  19  |    6144     |    2816     |    2816     | 121.9 MB/s 
  20  |    6528     |    2944     |    2944     | 121.9 MB/s 
--- Targeting Fastest Result of md_sync_window 1280 bytes for Final Pass ---
--- FULLY AUTOMATIC TEST PASS 2 (Final - 16 Sample Points @ 4min Duration)---
  21  |    2576     |    1160     |    1160     | 121.9 MB/s 
  22  |    2592     |    1168     |    1168     | 121.9 MB/s 
  23  |    2608     |    1176     |    1176     | 122.0 MB/s 
  24  |    2624     |    1184     |    1184     | 121.9 MB/s 
  25  |    2648     |    1192     |    1192     | 121.8 MB/s 
  26  |    2664     |    1200     |    1200     | 121.9 MB/s 
  27  |    2680     |    1208     |    1208     | 122.0 MB/s 
  28  |    2696     |    1216     |    1216     | 122.0 MB/s 
  29  |    2720     |    1224     |    1224     | 122.0 MB/s 
  30  |    2736     |    1232     |    1232     | 122.0 MB/s 
  31  |    2752     |    1240     |    1240     | 122.0 MB/s 
  32  |    2768     |    1248     |    1248     | 121.9 MB/s 
  33  |    2784     |    1256     |    1256     | 122.0 MB/s 
  34  |    2808     |    1264     |    1264     | 122.0 MB/s 
  35  |    2824     |    1272     |    1272     | 122.0 MB/s 
  36  |    2840     |    1280     |    1280     | 121.9 MB/s 

Completed: 2 Hrs 7 Min 2 Sec.

Best Bang for the Buck: Test 1 with a speed of 120.5 MB/s

     Tunable (md_num_stripes): 1408
     Tunable (md_write_limit): 768
     Tunable (md_sync_window): 512

These settings will consume 55MB of RAM on your hardware.


Unthrottled values for your server came from Test 23 with a speed of 122.0 MB/s

     Tunable (md_num_stripes): 2608
     Tunable (md_write_limit): 1176
     Tunable (md_sync_window): 1176

These settings will consume 101MB of RAM on your hardware.
This is 51MB more than your current utilization of 50MB.
NOTE: Adding additional drives will increase memory consumption.

In unRAID, go to Settings > Disk Settings to set your chosen parameter values.

Link to comment

Are they plugged in to PCIe x4 (or greater) slots?

 

Man! I was hoping I wouldn't have to open up the tower since I decided to upgrade the SSD (external drive bays).... *snerk* I'm pretty sure they are, I wouldn't short-change a card in a slot.

 

-John

 

ETA: Specs report the slots as PCIe 2.0 x16

 

I just read up on your motherboard, the M4A88TD-V EVO/USB3:

  • 1 x PCIe 2.0 x16 (blue)
  • 1 x PCIe 2.0 x16 (x4 mode, gray)
  • 1 x PCIe 2.0 x1
  • 3 x PCI

Most likely you had to plug into the two x16 sized slots.  The gray one is only capable of x4, but that lines up well with your card.

 

If you have lspci installed, you can run lspci -vv to get the actual bandwidth that the cards are running.

 

I checked out the card specs, and they have the Marvell 88SX7042 chip powering them.  Seems like a decent budget chip, but I haven't had much luck finding tech specs on it.  It should be running the sata_mv drivers.

 

If I counted correctly, you have 11 drives in your array (not counting the cache drive).  Since you only have 8 ports available on these two cards, that means some of your drives on on the motherboard, right?

 

The motherboard has 5 x SATA 6Gb/s ports controlled by the AMD SB850 chipset.  Not quite sure what driver it runs.

 

Regardless, you've got two different hd controller chipsets in your system, and either one, or the combination of both, is probably what is driving your server's bizarro behavior with a fondness for small bytes.

 

Not that I think it is a problem, just very interesting.  Looking forward to your sub 128 test results.

 

-Paul

Link to comment

Well I was hoping it would be a bit better than this but I guess it's not too bad. Any reason I shouldn't consider the slightly higher speed of run 23? Also any thoughts on what i could do to increase this value? I am assuming my 2x 2tb wd green and my 2x 2tb 5900rpm seagates are keeping the speeds down?

 

I would say your speeds max out at test # 7, and all the results after that are pretty much identical (normal variation in test results).  Those values are low enough that I think you could set them there and call it a day.

 

Interestingly, even the lowest values in the test produced good speeds.  Considering that you have 2TB drives in the mix, these really are great results.

 

The difference between test 1 and test 7, over a full length parity check, is probably less than a minute.  Consider yourself lucky that your server doesn't appear to be afflicted with the throttled... er, I mean ball-less performance that some of our servers exhibit when running stock values.

 

-Paul

Link to comment

It is initiated with '/root/mdcmd check CORRECT'.  If the driver detects one invalid disk it begins rebuilding it, if all disks are valid it does a correcting check.

 

unevent thanks for confirming.

 

John, sounds like you could just change the NOCORRECT to CORRECT in the utility.  Keep in mind that the current version never sets md_write_limit below 768, but you could change that easily - first line in the CalcValues, WriteLimit sets the lower limit.  You could change that to 128 and the script would handle the rest.

 

Keep us posted if you decide to do this!

 

-Paul

Link to comment

If the difference is only around a minute then I should probably just stick with the best bang for the buck settings. I will be excited to see the results of my parity check. Maybe it will actually be able to finish before I wake up now.

 

Also it wasn't that I was complaining about the speeds because I see there are definitely others who are having some serious speed issues. It was more I was hoping to pull in the 130-140 range. However if my parity check decreases in time safely then I will be happy and maybe when I pull the 2tb drives out the speeds will increase.

 

Now to run it on my other server that has all 3tb drives and see how that turns out.

Link to comment

Just a thought but what if we created a separate thread to have people post their system specs with the speeds they achieved and their UnRaid version, script version tested, and ultimately the values they used? This might be a good way for people to get an idea of potential parity check speeds with different hardware, especially if they are looking to upgrade or purchase a new system.

Link to comment

If the difference is only around a minute then I should probably just stick with the best bang for the buck settings. I will be excited to see the results of my parity check. Maybe it will actually be able to finish before I wake up now.

 

Also it wasn't that I was complaining about the speeds because I see there are definitely others who are having some serious speed issues. It was more I was hoping to pull in the 130-140 range. However if my parity check decreases in time safely then I will be happy and maybe when I pull the 2tb drives out the speeds will increase.

 

Now to run it on my other server that has all 3tb drives and see how that turns out.

 

Not only would ditching the 2TB drives get you close to 140 MB/s, it would also eliminate the dreaded slowdown partway through the parity check.  This will save hours on your parity check.  I was closer to 14 hours just a few months ago, and ditching the 1.5TB and 2TB drives, combined with tuning the md_* settings, almost cut my parity in half.

 

Of course, I'm simply amazed that servers based on 4TB 7200RPM drives can churn out 6.5 hour parity checks.  Blazing!

Link to comment

Just a thought but what if we created a separate thread to have people post their system specs with the speeds they achieved and their UnRaid version, script version tested, and ultimately the values they used? This might be a good way for people to get an idea of potential parity check speeds with different hardware, especially if they are looking to upgrade or purchase a new system.

 

I was actually thinking almost the same thing about half an hour ago, though I was thinking about logging what hd controller chipsets were responding well to which values.  That's probably a little too nitty gritty, and your idea seems more helpful.

Link to comment

After discovering that you can't stop an array with the preclear running, I said buggars and dropped the 4 TB in. I'll let a full parity check after the rebuild tell me if the drive gets any pending sectors. I'm not worried about the data, it's backed up elsewhere.

 

Running the script with full-auto and modified to perform a parity rebuild. After it finishes, I'll let the parity fully rebuild and then run the script again unmodified.

 

ETA: Correction, decided to do a 4 minute test run with 128 byte spread from 8 to 2944 bytes.

 

ETA: Scratch that. New post below.

Link to comment

Very interesting results.  I set my tunables back to what I had originally decided were the best values (after 2 weeks of parity checks a few months ago ... probably ran 25 full checks with various settings to find them) ==> and it's back to 8:05.    Guess I'll now have to try a few more combinations to see if I can at least get below 8 hours  :)

 

I did remember one factor that added about 8-10 minutes (and I'm just going to leave it that way) ... I've got cache_dirs running without limits (i.e. nothing's excluded).

 

I've read the details of what these tunables mean (in Tom's post here: 

http://lime-technology.com/forum/index.php?topic=4625.msg42091#msg42091

 

... and I've concluded the following ...

 

=>  The md_write_limit parameter does NOT need to be modified at all.  Once it's set to a value that provides a reasonable write buffer, there's no reason to make it larger, as long as the total number of stripes is enough larger than the sync_buffer setting that the specified number of strips for writing will always be available.  I've found that 1500 is a good value for my array, although your mileage may vary.  I'll outline how I determined that number at the end of this post.

 

=>  There's an "implied" setting for minimum read stripes -- the total stripes minus md_write_limit minus md_sync_window.  If you make this implied setting large enough ... I've found 1000 is very good ... then read performance will also be fine during parity checks -- and making it larger won't really help read performance.  Note also that if no writes are being done, read operations can also use all the stripes that would be allocated for writes (an extra 1500 stripes if wd_write_limit is 1500).

 

BOTH read and write performance can be impacted by simple resource availability (e.g. disk thrashing) if the md_sync_window setting is so "perfect" that the disks are fully engaged at all times.  But changing the number of stripes available for reads or writes won't impact that.  I suppose "de-tuning" the sync performance may, but I can't really think of a good reason to do that !!

 

So ...  what I'd love to have is a version of this utility that lets the user fix the md_write_limit value;  fix the minimum number of stripes available for reading;  and then simply sets md_num_stripes = the sum of those two numbers plus the current md_sync_window value as it's doing its tests.

 

You can independently "tune" the number of write stripes by simply testing writes with various settings ... but since this is a maximum number you don't need to change anything else (as long as md_write_limit < md_num_stripes).    But once you've decided on a value, there's no reason to change it as you tune other parameters (notably md_sync_window).  I don't think this is a reasonable thing to do with a script, since you need to do the copies from your PC ... but it's very easy to do manually.

 

Note that writes to the array are influenced by quite a few things;  so if you're going to "tune" this parameter, you should try to be as consistent on all the other things as possible (i.e. are you writing to inner cylinders or outer cylinders? ... is there any network contention?  ... etc.).    I'd create a folder on your PC's hard drive to hold a few large files (3GB is enough to write -- I used a single 3GB file);  then write it to a specific disk on UnRAID (NOT a user share, which could alter the location of the writes);  then delete what you just wrote; alter the parameter; and repeat the process.

 

FWIW I found that 1500 stripes worked well for writes.  A 3GB file took about 1:21 with the default 768 stripes;  took 1:01 with 1500; and dropped to 0:57 at 2500 stripes, with no improvement with larger values (I tried up to 5000).  I decided 1500 was a good choice.    I also tried this with a bunch of different values of md_sync_window, and it makes NO difference what it's set at (as I suspected would be the case, since the md_write_limit sets the maximum number of stripes used for writes).

 

With md_write_limit set to 1500, and md_num_stripes set to md_sync_windows + 2500, there are up to 2500 stripes available for reads during a sync, and 1500 available for writes.  I see no reason to ever use values higher than those.  Note that with these settings, reading the same 3GB file I used for write testing takes 27 seconds (which may be limited by the write speed on my PC's drive).

 

Now all I need to do is find a value of md_sync_window that cuts 5 more minutes off my parity checks !!  :)

 

Link to comment

I'm not sure what might work best for parity syncs and/or rebuilds.  Note that in both of these cases, it's really much closer to a parity check in terms of disk activity than anything else ... the difference is that a parity check is doing N reads (in an N-disk array), whereas a parity sync or disk rebuild is doing (N-1) reads and one write.

 

My intuition is that if you have an optimized md_sync_window it's going to be very close to the best you can do for either of the other activities as well.    Especially if you've already optimized your md_write_limit as I noted above.

 

Link to comment

I'm not sure what might work best for parity syncs and/or rebuilds.  Note that in both of these cases, it's really much closer to a parity check in terms of disk activity than anything else ... the difference is that a parity check is doing N reads (in an N-disk array), whereas a parity sync or disk rebuild is doing (N-1) reads and one write.

 

My intuition is that if you have an optimized md_sync_window it's going to be very close to the best you can do for either of the other activities as well.    Especially if you've already optimized your md_write_limit as I noted above.

 

The difference is seeing what the effect of the different values on a parity rebuild vs parity checks using the tunables script which plugs in different numbers for each iteration. It's about finding out the optimized value for a rebuild to see if it's on par for the optimized value for sync. I just need to know the correct way to do it.

Link to comment

Paul may have some ideas on what to modify ... but as I noted, I doubt it's different than the optimal value for parity checks, since the only difference functionally is a write is substituted for one of the reads.  The other N-1 disks are doing the same reads they'd do for a parity check.

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.