Jump to content
Pauven

unraid-tunables-tester.sh - A New Utility to Optimize unRAID md_* Tunables

1065 posts in this topic Last Reply

Recommended Posts

I just rebooted and re-enabled Plex...my family members might be itchy if it wasnt available tonight.  Ill find some time to run another test.

 

Just for the hell of it, Im running at full blast right now: 

 

    Tunable (md_num_stripes): 5648

    Tunable (md_write_limit): 2544

    Tunable (md_sync_window): 2544

 

Could the fact that Im running my parity on a hardware RAID affect the numbers?

Share this post


Link to post

most people run a weekly or monthly parity check. anything that cuts this down from the multi-hour event is a welcomed thing... assuming it doesnt greatly affect day-to-day performance.

speaking of, i would assume the md_ variables dont really affect the cache drive. thus you could always recommend tuning for parity then rely on cache drive for day-to-day?

Share this post


Link to post

I still think we should draw the line at using integer values for measuring MB/s.  The thing is, while I care to find such settings that will get me from 70MB/s to 120MB/s, I don't care which settings will get me from 120.4MB/s to 120.7MB/s -- I much rather use the lowest possible setting that got me to 120MB/s.

 

120.5 and 121.4 are pretty much a full megabyte apart, but both round to 121.  The extra decimal allows my script to have greater awareness of throughput trends.

 

Since you don't care about extracting the last iota of performance, the new 'Best Bang for the Buck' recommendation addresses your desire to have good enough performance at lowest possible setting.

Share this post


Link to post

I for one would like to be able to stream a few BluRays from the server without stuttering while a parity check is running.

 

 

This is my entire reason for this.  My parity check starts at midnight on the 2nd day of the month.  Right now, my parity check takes ~15 hours to complete. If that happens to fall on a weekend, that's almost a whole day that I'm having to deal with it. 

 

I wish there was a way to schedule my monthly parity check for something like the 1st Monday of each month.  I would love for my parity check to not fall on the weekend.

 

I will report back on Monday to see if my parity check speed has increased.

Share this post


Link to post

Nice improvement on my case. Thanks a lot!

 

Test | num_stripes | write_limit | sync_window |   Speed 
---------------------------------------------------------------------------
   1  |    1280     |     768     |     384     |  90.5 MB/s 
   2  |    1408     |     768     |     512     |  98.3 MB/s 
   3  |    1536     |     768     |     640     | 102.7 MB/s 
   4  |    1664     |     768     |     768     | 106.3 MB/s 
   5  |    1920     |     896     |     896     | 108.0 MB/s 
   6  |    2176     |    1024     |    1024     | 108.9 MB/s 
   7  |    2560     |    1152     |    1152     | 108.9 MB/s 
   8  |    2816     |    1280     |    1280     | 108.9 MB/s 
   9  |    3072     |    1408     |    1408     | 108.8 MB/s 
  10  |    3328     |    1536     |    1536     | 109.0 MB/s 
  11  |    3584     |    1664     |    1664     | 109.0 MB/s 
  12  |    3968     |    1792     |    1792     | 109.0 MB/s 
  13  |    4224     |    1920     |    1920     | 109.1 MB/s 
  14  |    4480     |    2048     |    2048     | 107.9 MB/s 
  15  |    4736     |    2176     |    2176     | 109.1 MB/s 
  16  |    5120     |    2304     |    2304     | 108.9 MB/s 
  17  |    5376     |    2432     |    2432     | 108.7 MB/s 
  18  |    5632     |    2560     |    2560     | 109.0 MB/s 
  19  |    5888     |    2688     |    2688     | 108.8 MB/s 
  20  |    6144     |    2816     |    2816     | 108.9 MB/s 
  21  |    6528     |    2944     |    2944     | 109.0 MB/s 

Share this post


Link to post

These 100MB/s+ speeds are pissing me off. I need to figure out what my bottleneck is!

My guess is your Areca card... try using the parity disc on the SAS2LP as well.

Why use this RAID card anyway?

Share this post


Link to post

I for one would like to be able to stream a few BluRays from the server without stuttering while a parity check is running.

That is a nice goal.  I know others share it.

 

Personally, I just run my Parity Checks during the middle of the night when I'm sleeping and household temperatures are cooler.  I would also much rather have a parity check take less time, so my drives are spun up a shorter period of time.

 

So, you're testing with different values of write_limit and sync_window, but you are hardcoding the num_stripes to 11% of the sum.  What is your rationale for that 11%? 

Yes, I hard coded to 11% of that sum, same as you hard coded in your script to 15% of that sum.  My rational was maintaining the same percentage of overhead that Tom ships unRAID with.

 

When Tom played around with these tunables, he settled on having num_stripes to be about 20% bigger than the sum of the other two:

md_num_stripes=1280

md_write_limit=768

md_sync_window=288

Your 4-year-old quote doesn't represent the values currently shipping with unRAID.  They were changed years ago.  The current md_num_stripes is 11% bigger than the other two.  Sound familiar?

 

In the past I've found values of up to 50% bigger to have positive effect if I am transferring files to/from the server while parity check is running.  So I am inclined to think that testing with different values of ALL three tunables may be meaningful, especially if we introduce something like a large read of files from the server while the parity check speeds are being measured.

Agreed, testing of all three tunables would be meaningful, but the methodology for such a test has not been defined.  Since one of the tunables involves writing, the only way to test all three would be to run a parity check, read files, and write files, all simultaneously.  Oh, and you would want to do that while excluding any outside variables like LAN connectivity and desktop performance.  If you want to go ahead and code it, I'll be happy to test with it.  After all: [ $Your_bash_programming_skills -gt (( $My_bash_programming_skills**2 )) ];

 

So [Tom] only says that md_num_stripes should be bigger than (md_write_limit+md_sync_window).  But how much bigger?  1% bigger?  Three times bigger?  That too can be a subject of our test utility.

You are correct, Tom has never clarified his statement.  I suspect he doesn't know the answer any more than we do.  Tests are the only way to determine the appropriate values, which is the goal of this utility primarily for testing the md_sync_window with a known quantifiable touch point.  But before you can test, you have to identify your touch points for the tests. What are the known quantifiable touch points for md_write_limit and md_num_stripes?

 

In the past have found my server to be nicely responsive when I'm using something like this:

md_num_stripes=3072

md_write_limit=1024

md_sync_window=1024

 

Just something to think about.

If you take a look at zoggy's results, you will see that his server got slower with any values beyond md_sync_window=512.  At this point I don't think we understand enough about these values to make any generalized assumptions about what is right, as what works for one server doesn't necessarily work for another.

 

Regarding the relationship between the 3 tunables, it can be viewed in multiple ways.

 

For example, Tom ships unRAID with md_num_stripes=md_write_limit+md_sync_window+11%.  But another way to look at those numbers is as a balanced equation:  60% writes, 30% syncs, 10% reads.

 

Keep in mind that with the reads, they get whatever is left over, up to the maximum of md_num_stripes (since there is no md_read_limit).  So if all you are doing is reading, you get 100% reads.  If you are reading while doing a parity sync, you get 70% reads vs. 30% sync.  If you are reading while writing, you get 40% reads vs. 60% writes.

 

My formula basically adjusts the values so the ratio (for values 768 and beyond) is 45% syncs, 45% writes and 10% reads.  If you are reading while syncing (your watching a Blu-Ray example), you get 55% reads vs 45% syncs.

 

In your example values (33% syncs, 33% writes, 33% reads), if you are reading while syncing, you get 66% reads vs 33% syncs.  Those numbers are not that far off from 55% reads vs 45% syncs. 

 

Interestingly enough, even at 50% bigger your read allocation is worse than stock unRAID.

 

 

Share this post


Link to post

These 100MB/s+ speeds are pissing me off. I need to figure out what my bottleneck is!

My guess is your Areca card... try using the parity disc on the SAS2LP as well.

Why use this RAID card anyway?

 

I don't have an SAS2LP card...its an SASLP.  I use the Arcea for a RAID0 4TB parity and a 1TB RAID1 cache.  I'm pretty sure its not the Areca slowing things down. It even has 256MB of cache.

 

root@nas:~# hdparm -tT /dev/sdb

/dev/sdb:
Timing cached reads:   26486 MB in  1.99 seconds = 13284.33 MB/sec
Timing buffered disk reads: 830 MB in  3.10 seconds = 268.14 MB/sec

 

Share this post


Link to post

Still, wouldn't hurt to attach your parity drive to the same controller the rest of you drives is on... maybe the Areca is fast enough on its own, but unraid can't handle the thing properly. I know a another user was always having problems with his transfer speeds on unraid, until he removed his Areca card...

 

Sorry for the OT.

Share this post


Link to post

I wish there was a way to schedule my monthly parity check for something like the 1st Monday of each month.  I would love for my parity check to not fall on the weekend.

 

There is a way! Your scheduled parity check is just a cron job.  The crontab format is:

#minute hour mday month wday command

 

So if you used the following values:

 

0 0 1-7 * * test $(date +%u) -eq 1 && /root/mdcmd check CORRECT

 

Then the job would run at midnight (0 0), sometime between the 1st-7th of each month (1-7), every month (*), every day (*), but only if the test shows it is a Monday (test $(date +%u) -eq 1 &&) and it calls the parity check directly.

 

I add this to my go file so it's always in my cron job list on reboot:

# Add unRAID Fan Control & Monthly Parity Check Cron Jobs
crontab -l >/tmp/crontab
echo "# Run unRAID Fan Speed script every 5 minutes" >>/tmp/crontab
echo "*/5 * * * * /boot/unraid-fan-speed.sh >/dev/null" >>/tmp/crontab
echo "# Run a Parity Check on the First Monday of each Month at 12:00am" >>/tmp/crontab
echo "0 0 1-7 * * test $(date +%u) -eq 1 && /root/mdcmd check CORRECT" >>/tmp/crontab
crontab /tmp/crontab

 

Note: I also add my unraid-fan-speed.sh script as a cron job, you don't need that part.

 

Doing it this way doesn't require any packages or scripts or add-ons.  Just a few lines in your go file.

 

-Paul

 

Share this post


Link to post

I wish there was a way to schedule my monthly parity check for something like the 1st Monday of each month.  I would love for my parity check to not fall on the weekend.

 

There is a way! Your scheduled parity check is just a cron job.  The crontab format is:

#minute hour mday month wday command

 

So if you used the following values:

 

0 0 1-7 * * test $(date +%u) -eq 1 && /root/mdcmd check CORRECT

 

Then the job would run at midnight (0 0), sometime between the 1st-7th of each month (1-7), every month (*), every day (*), but only if the test shows it is a Monday (test $(date +%u) -eq 1 &&) and it calls the parity check directly.

 

I add this to my go file so it's always in my cron job list on reboot:

# Add unRAID Fan Control & Monthly Parity Check Cron Jobs
crontab -l >/tmp/crontab
echo "# Run unRAID Fan Speed script every 5 minutes" >>/tmp/crontab
echo "*/5 * * * * /boot/unraid-fan-speed.sh >/dev/null" >>/tmp/crontab
echo "# Run a Parity Check on the First Monday of each Month at 12:00am"
echo "0 0 1-7 * * test $(date +%u) -eq 1 && /root/mdcmd check CORRECT"
crontab /tmp/crontab

 

Note: I also add my unraid-fan-speed.sh script as a cron job, you don't need that part.

 

Doing it this way doesn't require any packages or scripts or add-ons.  Just a few lines in your go file.

 

-Paul

 

 

Thank you!!  Thank you!!  I'm not entirely familiar with decoding the crontab format. I figured out how to change the date and time, but couldnt figure out the way I really wanted.

Share this post


Link to post

add the redirect part of the commands

echo "# Run a Parity Check on the First Monday of each Month at 12:00am" >> /tmp/crontab
echo "0 0 1-7 * * test $(date +%u) -eq 1 && /root/mdcmd check CORRECT"   >> /tmp/crontab 

 

or via here document with cat (can also be done with echo.)

 

cat <<-EOF >> /tmp/crontab
# Run unRAID Fan Speed script every 5 minutes
*/5 * * * * /boot/unraid-fan-speed.sh >/dev/null
# Run a Parity Check on the First Monday of each Month at 12:00am
0 0 1-7 * * test $(date +%u) -eq 1 && /root/mdcmd check CORRECT
EOF

 

 

Share this post


Link to post

add the redirect part of the commands

 

Aaaarggghh!  Cut-n-paste, you foil me again!  Good catch, updated my post to correct.

Share this post


Link to post

zoggy,

 

I've been giving some more thought to your results.

 

On the one hand, the utility did as designed an helped identify the correct values for your server - it is just that they were unexpected low, and lower than where you had them configured.

 

On the other hand, I'm suspicious of the results.  I would have expected that Test 1 and Test 36, which both sampled the 512 byte value, would have had the nearly same result.

 

Instead, it almost appears that your server is experiencing some kind of issue that is affecting the results.

 

Two things:

 

1) Anything in your log file, error wise?

 

2) If you have the time, can you run another test:  (N)ormal length, 16 Byte Interval, 384 Byte Start and 768 Byte End.  This test should take about 50 minutes.

 

 

Share this post


Link to post

Did a before and after parity check. Before is with stock settings:

 

Last checked on Wed Aug 21 18:04:00 2013 CEST (today), finding 0 errors.

> Duration: 8 hours, 24 minutes, 50 seconds. Average speed: 132.1 MB/sec

 

After is a full parity check after first optimizing settings with the script 2.0:

 

Last checked on Tue Aug 27 20:41:17 2013 CEST (today), finding 0 errors.

> Duration: 8 hours, 24 minutes, 13 seconds. Average speed: 132.2 MB/sec

 

As you can see... no difference at all on the parity check speed.

(4TB parity disk)

 

Share this post


Link to post

jowi, that's a great time for a 4TB drive!

 

Looks like your server runs fine with stock values.  Many do.

 

Would you mind posting your TunablesReport.txt file?

Share this post


Link to post

Here it is.

 

Tunables Report from  unRAID Tunables Tester v2.0 by Pauven

NOTE: Use the smallest set of values that produce good results. Larger values
      increase server memory use, and may cause stability issues with unRAID,
      especially if you have any add-ons or plug-ins installed.

Test | num_stripes | write_limit | sync_window |   Speed 
--- FULLY AUTOMATIC TEST PASS 1 (Rough - 20 Sample Points @ 3min Duration)---
   1  |    1408     |     768     |     512     | 151.7 MB/s 
   2  |    1536     |     768     |     640     | 157.7 MB/s 
   3  |    1664     |     768     |     768     | 159.7 MB/s 
   4  |    1920     |     896     |     896     | 160.3 MB/s 
   5  |    2176     |    1024     |    1024     | 161.4 MB/s 
   6  |    2560     |    1152     |    1152     | 161.8 MB/s 
   7  |    2816     |    1280     |    1280     | 161.0 MB/s 
   8  |    3072     |    1408     |    1408     | 161.9 MB/s 
   9  |    3328     |    1536     |    1536     | 161.9 MB/s 
  10  |    3584     |    1664     |    1664     | 161.9 MB/s 
  11  |    3968     |    1792     |    1792     | 162.2 MB/s 
  12  |    4224     |    1920     |    1920     | 162.4 MB/s 
  13  |    4480     |    2048     |    2048     | 162.5 MB/s 
  14  |    4736     |    2176     |    2176     | 162.4 MB/s 
  15  |    5120     |    2304     |    2304     | 162.5 MB/s 
  16  |    5376     |    2432     |    2432     | 162.4 MB/s 
  17  |    5632     |    2560     |    2560     | 162.5 MB/s 
  18  |    5888     |    2688     |    2688     | 162.5 MB/s 
  19  |    6144     |    2816     |    2816     | 162.6 MB/s 
  20  |    6528     |    2944     |    2944     | 162.6 MB/s 
--- Targeting Fastest Result of md_sync_window 2816 bytes for Medium Pass ---
--- FULLY AUTOMATIC TEST PASS 2 (Final - 16 Sample Points @ 4min Duration)---
  21  |    5984     |    2696     |    2696     | 162.5 MB/s 
  22  |    6008     |    2704     |    2704     | 162.5 MB/s 
  23  |    6024     |    2712     |    2712     | 162.5 MB/s 
  24  |    6040     |    2720     |    2720     | 162.4 MB/s 
  25  |    6056     |    2728     |    2728     | 162.5 MB/s 
  26  |    6080     |    2736     |    2736     | 162.5 MB/s 
  27  |    6096     |    2744     |    2744     | 162.4 MB/s 
  28  |    6112     |    2752     |    2752     | 162.5 MB/s 
  29  |    6128     |    2760     |    2760     | 162.2 MB/s 
  30  |    6144     |    2768     |    2768     | 162.1 MB/s 
  31  |    6168     |    2776     |    2776     | 162.2 MB/s 
  32  |    6184     |    2784     |    2784     | 162.2 MB/s 
  33  |    6200     |    2792     |    2792     | 162.4 MB/s 
  34  |    6216     |    2800     |    2800     | 162.2 MB/s 
  35  |    6240     |    2808     |    2808     | 162.0 MB/s 
  36  |    6256     |    2816     |    2816     | 162.0 MB/s 

Completed: 2 Hrs 7 Min 1 Sec.

Best Bang for the Buck: Test 3 with a speed of 159.7 MB/s

     Tunable (md_num_stripes): 1664
     Tunable (md_write_limit): 768
     Tunable (md_sync_window): 768

These settings will consume 45MB of RAM on your hardware.


Unthrottled values for your server came from Test 21 with a speed of 162.5 MB/s

     Tunable (md_num_stripes): 5984
     Tunable (md_write_limit): 2696
     Tunable (md_sync_window): 2696

These settings will consume 163MB of RAM on your hardware.
This is 41MB more than your current utilization of 122MB.
NOTE: Adding additional drives will increase memory consumption.

In unRAID, go to Settings > Disk Settings to set your chosen parameter values.

Share this post


Link to post

Hey jowi,

 

I'm confused.  You said you ran the first parity check at stock values, but the report indicates you already had an md_num_stripes of about 4460 - which definitely is not stock.

 

The small increase from 4460 to 5984 (assuming that is the value you tested with) would have only provided a few seconds reduction in parity check time (as you reported).

 

But had you really tested with stock unRAID values (1280/768/384) I think the processing time would have been much longer.

 

Are you sure you tested correctly with stock unRAID values?

Share this post


Link to post

I ran the first parity check with stock values; after that i experimented with your scripts.

Basically, i did not run your script with stock values. Should i have done that?

 

So:

- Parity check 1 with stock values aug. 21st

- ran the 1.1 script a few times for testing and used those values after that

- ran the 2.0 script this morning in FULLAUTO, rebooted using the values it gave

- ran the 2nd parity check.

Share this post


Link to post

No need to use stock values when running the utility, since it sets it's own values for testing.

 

While there clearly is an improvement in speed from increasing the variables, that improvement is small, and it did not result in a worthwhile decrease in parity check speed.

 

Since stock values give you great results, you should use the stock values (md_num_stripes=1280, md_write_limit=768, md_sync_window=384). No need to waste memory on zero tangible improvement.

Share this post


Link to post

So these 3 parameters only influence the parity check speed, and have nothing to do with overall transfer speed or disk/data reading or writing speed?

Share this post


Link to post

So these 3 parameters only influence the parity check speed, and have nothing to do with overall transfer speed or disk/data reading or writing speed?

 

They can affect all, but the md_sync_window directly affects parity check speed, making it the easiest one to test, and since unRAID ships with a low value for md_sync_window, the parity check speed is the most likely to be afflicted with low performance.

 

If your parity check speed is fine, then typically all else will be fine too (from an md_* tunables sizing point).  Even if your md_sync_window value is low, disk transfers and writing speed may still be fine, since they ship with significantly higher values.

Share this post


Link to post

I gained a lot of responsiveness via standard copies, and 50MB/s on parity checks by running this tool.

 

I now get 110MB/s on parity checks.

 

Happy  :)

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.