Pauven

I laughed out-loud! Good one! Thank you!

With upgrading the 32GB SSD, I would only need to let the parity rebuild run for less than 10 minutes to see how long it takes to rebuild the 32GB range. If you really want to test the rebuild timings, create a script that'll kick off the parity rebuild, let it run to 32 GB, then abort the rebuild and repeat looping over the different values. I'm game with running it on my server - as long as one of my normal drives doesn't fail I'm not sure I want to create a script for that, sounds like a inappropriate tool to put out there. But testing it manually wouldn't be too hard. Basically, set the three params, start a rebuild, cancel after 10 minutes measuring what position was reached. Rinse and repeat with new params. I did this dozens of times on a parity check when I first tested the md_sync_window. The first time you could run with stock values (1280/768/384) and the second time with tuned values (288/128/128). If the second run is looking good after 10 minutes, no need to cancel, just let it continue to the end. Otherwise, cancel, go back to stock values, and rebuild again. If there is any impact at all, it should be very noticeable.

Yes, halfway. I think the SSD change is so significant that it completely changes your baseline. If you did two upgrades, one replacing the SSD, and one replacing a 1.5TB drive, with stock vs. tuned values, that would be best.

Hey Steven, I just went back and looked at your results, and I would agree that your parity drive array is probably not the problem. Most likely those 1.5 TB drives are the main culprit. Having a mix of drive sizes impacts parity check / rebuild speed in multiple ways. Primarily, the slowest drive sets the pace for the whole array. Additionally, you get multiple slow-downs as each drive reaches the inner cylinders at different points during the parity check: so you would have slowdowns approaching 1.5TB, 2TB and 4TB. This doesn't necessarily affect read or write performance unless you're accessing data on one of those drives. Unless your parity check/rebuild times are unfathomably long, upgrades may not be cost effective. Anyway, I'm interested to hear how your upgrades go. -Paul

I really like this idea. I've been thinking about doing a dual-ssd cache drive setup (not even using a cache drive today), but that's obviously expensive and requires functionality Tom has not yet released. Do you have a link, or instructions, for mounting a drive outside the array? I wouldn't mind having a smaller SSD for everyday file transfers, and then I can use bigger, cheaper hard drives for the cache drive role to speed up my BR rips. Thanks, Paul

Alright John, I think your server is from Bizarro World. Your results are pretty much the opposite of what should be happening. Higher values make your sync slower, not faster. Bizarro. Longer tests produce higher speeds, not slower. Bizarro. As I described in a post above, longer tests should produce slower results because hard drives get slower the further into a parity check you traverse (due to the the tracking of the read head from outside cylinders to inside cylinders on the platters). Longer tests are more accurate, as the noise floor is reduced, but they "should" be slightly slower. But your 10-minute Extreme level tests produced significantly higher results compared to your 3-minute results. I think you're right about your SSD skewing the results. I think your SSD is SLOOOOOW, so slow that it is dragging down your results. But because your SSD is so small, longer tests give less weight to the SSD speed. By my math, approximately the first 6 minutes of your parity check includes the SSD, beyond which your parity check position has passed the 32GB SSD. So your 10-minute tests have about 4 minutes of HD only tests, pulling your avg speed back up. I don't know that this is a problem, so to speak, but for accurate testing on your server, you need to be running extreme length tests (10 minute), or maybe even longer. Your results are also slightly inconsistent. On pass 1 and 1b, the 512 byte point gets tested twice, both times at 3 minutes. The result varied by 1 MB/s (90.5 vs. 89.5). This is enough variance to make it difficult to choose a best value. Again, longer tests should help here. I see similar variance on my server, but typically at shorter run times around 2 minutes or less. I was also expecting 128 bytes to be the winner from pass 1b, allowing pass 2 to test from 8 to 128 bytes. That didn't happen, so we've yet to see what results look like on your server below 128 bytes (I'm very interested to see). If you're willing, please run the following test: (E)xtreme, 8 byte interval, starting at 8 bytes, ending at 256 bytes. This would be a 5.5 hour test, but I think it targets the right range for your server and each test is long enough to cut through the noise the SSD is generating. I think you may also want to consider some of the options proposed by others, removing the SSD from your protected array, not so much because it is slowing down your parity check for a few minutes, but because the slow speed of an SSD may be indicating it is in poor health (maybe a lack of proper maintenance like trim or other garbage collection, or possibly even getting closer to failure). I don't think that a slow 32GB drive is worth risking parity on your whole array just for instant access capability. Wow, you slipped that one in there, and it almost got by me! What are these two 'PCI-E SATA extenders'?!! I think whatever these things are is what is causing your server to work well with lower values. Also, I am excited to hear that you are upgrading drives. If you're willing, I propose a test during your next upgrade. Hopefully you already know how long it takes to upgrade a drive, if not, you need to get a baseline. Once you have a baseline, I propose performing an upgrade with an md_write_limit of 128 instead of the unRAID stock 768, and see how that affects the running time. If that's a test you're willing to try, before we do it I think we need to test lower md_write_limit values just in everyday writing before going whole hog on a rebuild - but higher values seem to work well on servers that respond well to higher md_sync_window values, so I think this is a reasonable test on your server. This test would help establish 2 things: That rebuilds are connected to md_write_limit in the same way that parity checks are connected to md_sync_window, and that md_write_limit values should be set to md_sync_window values for optimal results. Obviously this test is not without risks, so if you are not interested I understand. -Paul

Correct, I don't compare pass 2 results to pass 1, since they run for different lengths of time. Pass 2 results are expected to be slightly lower, as the test extends farther into the parity check, which gradually slows from beginning to end. It is expected that pass 1 gets you into the right range, and pass 2 finds the best value in that range. If I compared pass 2 to pass 1, the majority of the time the logic would probably pick the pass 1 result since it would probably have the faster time due to the shorter test length. Unfortunately some servers are not testing well (he's not slow, he just doesn't test well...). It's not so much about comparing pass 1 to pass 2, rather it's more about servers producing inconsistent results, which no amount of logic can power through. Myself, I'm looking at RockDawg's results and have no freaking idea which values are good values... My point exactly, inconsistent results. Test 36 should have been pretty close, but slightly below test 1. Actually, if you look at the bigger picture (and RockDawg's server is not the first to show this behavior), the 1st test started off at a nice speed, and each subsequent test gets slower, until a lower threshold is reached, and then all results hover around that lower threshold. I would hazard a guess that these md_* values are not actually affecting anything on RockDawg's server - he could run a 512 byte test 10 times in a row, and each subsequent test would be a little bit slower. I think VM is highly suspect. zoggy had nearly identical behavior in his test results: Tunables Report from unRAID Tunables Tester v2.0 by Pauven Test | num_stripes | write_limit | sync_window | Speed --- FULLY AUTOMATIC TEST PASS 1 (Rough - 20 Sample Points @ 3min Duration)--- 1 | 1408 | 768 | 512 | 88.0 MB/s 2 | 1536 | 768 | 640 | 87.8 MB/s 3 | 1664 | 768 | 768 | 87.4 MB/s 4 | 1920 | 896 | 896 | 87.0 MB/s 5 | 2176 | 1024 | 1024 | 87.2 MB/s 6 | 2560 | 1152 | 1152 | 86.8 MB/s 7 | 2816 | 1280 | 1280 | 86.6 MB/s 8 | 3072 | 1408 | 1408 | 86.2 MB/s 9 | 3328 | 1536 | 1536 | 86.0 MB/s 10 | 3584 | 1664 | 1664 | 85.7 MB/s 11 | 3968 | 1792 | 1792 | 85.7 MB/s 12 | 4224 | 1920 | 1920 | 86.1 MB/s 13 | 4480 | 2048 | 2048 | 86.2 MB/s 14 | 4736 | 2176 | 2176 | 85.7 MB/s 15 | 5120 | 2304 | 2304 | 85.3 MB/s 16 | 5376 | 2432 | 2432 | 85.3 MB/s 17 | 5632 | 2560 | 2560 | 85.1 MB/s 18 | 5888 | 2688 | 2688 | 85.1 MB/s 19 | 6144 | 2816 | 2816 | 84.8 MB/s 20 | 6528 | 2944 | 2944 | 84.8 MB/s --- Targeting Fastest Result of md_sync_window 512 bytes for Medium Pass --- --- FULLY AUTOMATIC TEST PASS 2 (Final - 16 Sample Points @ 4min Duration)--- 21 | 1288 | 768 | 392 | 84.9 MB/s 22 | 1296 | 768 | 400 | 84.8 MB/s 23 | 1304 | 768 | 408 | 84.7 MB/s 24 | 1312 | 768 | 416 | 84.7 MB/s 25 | 1320 | 768 | 424 | 84.4 MB/s 26 | 1328 | 768 | 432 | 84.7 MB/s 27 | 1336 | 768 | 440 | 84.7 MB/s 28 | 1344 | 768 | 448 | 84.4 MB/s 29 | 1360 | 768 | 456 | 84.6 MB/s 30 | 1368 | 768 | 464 | 84.7 MB/s 31 | 1376 | 768 | 472 | 84.3 MB/s 32 | 1384 | 768 | 480 | 84.5 MB/s 33 | 1392 | 768 | 488 | 84.7 MB/s 34 | 1400 | 768 | 496 | 84.6 MB/s 35 | 1408 | 768 | 504 | 84.5 MB/s 36 | 1416 | 768 | 512 | 84.7 MB/s Notice that the 512 byte test is both the fastest and one of the slowest! I talked to zoggy the other day about his build, and I don't think he mentioned VM, but I didn't think to ask either. -Paul

Yup, the Best Bang for the Buck logic is pretty much broken in two scenarios: servers that respond well to lower values, like John's (this issue is easily fixable, though); and servers that have spikey results. Currently the Best Bang recommendation only looks at pass 1, and takes the last set of values that shows a nice improvement (at least 1% over the previous values), but stops checking at soon as the the tests turn up a sub-par improvement. While this logic worked well on my server with a very smooth bell curve, on spikey servers the logic stops checking too early. I have some ideas on how to handle this, but probably the best way is with an array of values that I analyze after the test is done. I guess I disagree with you there (but I'm open to better words if you have any suggestions). If you look at my test results with extra low values, my performance drops below 10MB/s with values below 128 bytes. If that isn't throttling, then what is it? I understand your point that throttling wasn't the intent of that parameter, and that true throttling (for the purpose of prioritizing reads vs. writes vs syncs) has not been implemented or at least exposed as parameters. But I would prefer to call that wished for feature 'load prioritization', not throttling. Semantics...

Got it now, thanks for pointing it out. I haven't been maintaining the 2nd post, mainly because the utility has been changing frequently enough that it felt like a moving target. There's a lot more I need to update in the 2nd post.

I have updated the utility to version 2.2 (see the main post). The changes are very minor, so feel free to skip this version unless you need one of the three changes. New Features in v2.2: Added support for starting position byte values down to 8 bytes (previously 128). You will have to select (M)anual on the Starting Position Override screen, then you can manually enter the starting value. You can also select down to 8 on the Ending Position Override screen too. Updated the FULLAUTO routine to add a special pass if the fastest value after Pass 1 is 512 bytes. The extra pass basically extends pass 1 down to 128 bytes, and adds 12 minutes to the running time. If 128 bytes is the new fastest value after special pass 1b, then the final pass (unmodified) will effectively test values from 8 bytes to 128. This allows the FULLAUTO routine to correctly handle 'special' servers like John's, which for reasons unknown works better with lower values. I fixed a small bug that caused the utility to not restore the original configured parameters after the test when the user failed to select either the (U)nthrottled or (B)est Bang for the Buck values. You can download the file from post #1: http://lime-technology.com/forum/index.php?topic=29009.msg259087#msg259087 -Paul John - this version is for you. While you can run the ultra low values in Manual mode, I would appreciate it if you could also run a FULLAUTO, as I added logic to handle your server. I am not able to test this logic since my server doesn't behave better with lower values.

Yes, I believe so. Hopefully it would be a positive effect. I first documented an issue where parity checks took significantly longer than parity rebuilds. I took my case to Tom, and he pointed me to these md_* tunables. I "believe" that since a rebuild is writing data, it is limited by the md_write_limit instead of the md_sync_window. If I am correct, and if increasing your md_sync_window beyond 768 bytes improves your parity check speed, then also increasing the md_write_limit beyond 768 would improve your rebuild speed. Or maybe Tom has some other control in place on rebuilds... Obviously a rebuild is much harder to test than a parity check (I don't think you want me aborting and restarting a rebuild a few dozen times to test it, do ya?). Hopefully someone with a couple disk upgrades to perform could run each with different values for md_write_limit and report back. I just went through a bunch of these myself, and it didn't even occur to me to test this. Too bad, since all my drives are upgraded now.

DROOOL. You're spot on about the DX and DM's. I actually have 4, two of each, in a RAID 10. Maybe not the best to mix different rpms in a RAID 10, but I got the two new drives as partial payment for my old server, so I couldn't be too picky. Works a charm, though. 8TB at up to 286MB/s does wonders for editing 1080p video.

That's a possibility I hadn't considered, and I think it is quite likely. I was thinking that maybe the hd controller could only accept so many bytes per request. At 128 stripes times 4k, that's 512k per drive. I was thinking that too much data at once overwhelms the controller, and I think that every controller has that limit at some point (be it a limit per drive or a limit upon the sum of all drives), and the limit is unique to that controller. My controller seems to hit that limit around 24 MB per drive, or 480 MB as a sum of all drives I have installed. This idea also helps explain some of the sporadic/spikey results some users have reported - I've noticed that these users appear to have multiple controllers (mb + add-in), and each controller has different limits/needs, and what makes one controller faster might be making the other controller slower. My results have been smooth as butter, but I only have the one controller. What a nightmare if you have one controller that wants 128, and another that wants 2600. It's impossible to satisfy both at once. Maybe I'm way off base.

JarDo, garycase is spot-on in his assessment. Areal density makes a huge impact on performance. In my desktop I have two different versions of 4TB drives, one is a 5 platter 800GB/platter 7200RPM beast, and the other is a 4 platter 1TB/platter 5400RPM eco type drive. The 5400 rpm drive is the faster drive. Go figure. I recently upgraded all my unRAID drives to 3TB units, and I am very pleased with the consistent level of performance I get, plus a much faster parity check. That said, swapping your drives will primarily only affect parity checks at the 'whole system level'. Otherwise you would only experience slower performance when reading or writing to one of the slow drives. If these are drives you don't use much, you may not find much benefit from replacing them, especially if you only run a monthly parity check.

Fascinating! This was completely unexpected! I've now got a few thoughts: a) I should have opened up even lower values below 128... but how low should I go? 64? 32? 16? 8? b) John, what crazy hardware are your running? You should put your build in your profile for convenience. c) If higher numbers are slowing down your syncs, are higher numbers also slowing down your writes? For c), I don't ever drop the md_write_limit below the unRAID stock of 768, but my theory is that the value that works best for syncs is the value that works best for writes too. At 768 bytes, your Parity Check speed is limited to 60.6 MB/s, climbing 38% to 83.5 MB/s at 128 bytes. I can't help but wonder if writing to your array would see similar levels of gain? Of course, to test you would need a gigabit connection, anything slower and your network will skew the results. You also need a source that can send the file faster than unRAID can write it. You would also want to write to an empty drive to accurately measure the impact of the parameter changes, preferably the fastest model drive in your build. -Paul

Thanks for the heads up! I gave up on watching... Exciting!!!

I have updated the utility to version 2.1 (see the main post). The changes are very minor, so feel free to skip this version unless you need one of the two changes. New Features in v2.1: Added support for starting position byte values down to 128 bytes (previously 384). You will have to select (M)anual on the Starting Position Override screen, then you can manually enter the starting value. You can also select down to 128 on the Ending Position Override screen too. Fixed a typo on the FULLAUTO option on the Test Type screen - it now correctly indicates the test will take 2.1 hours. You can download the file from post #1: http://lime-technology.com/forum/index.php?topic=29009.msg259087#msg259087 -Paul

Hey John, I could be wrong, but I don't think the Cache Drive has any impact on Parity Checks, so it wouldn't influence these test results. Longer tests just naturally provide smoother results. I just posted v2.1 with support down to 128 bytes. You'll have to use the (M)anual option to access it. Here's my results for comparison: Test | num_stripes | write_limit | sync_window | Speed --------------------------------------------------------------------------- 1 | 992 | 768 | 128 | 20.1 MB/s 2 | 1024 | 768 | 160 | 21.7 MB/s 3 | 1056 | 768 | 192 | 24.8 MB/s 4 | 1088 | 768 | 224 | 32.3 MB/s 5 | 1120 | 768 | 256 | 36.5 MB/s 6 | 1152 | 768 | 288 | 45.4 MB/s 7 | 1184 | 768 | 320 | 55.5 MB/s 8 | 1216 | 768 | 352 | 61.4 MB/s 9 | 1280 | 768 | 384 | 74.3 MB/s

Yikes! JarDo, if I'm reading your build correctly, you've installed a PCI-X hd controller card into a motherboard that doesn't have any PCI-X slots. That means you've installed it into a regular PCI slot, with dramatically limited bandwidth! Since you have 5 drives on the controller card, which is being limited to 133 MB/s by the PCI slot it is installed in, you are getting 133/5=26.6 MB/s, maximum! After accounting for overhead and such, your 18.5 MB/s number makes perfect sense! Replace that PCI-X card with a proper PCIe (PCI-Express) card, which your motherboard supports! -Paul

JarDo, could you be more specific on the hard drives: how many of each, what size & RPM they are, and which controller each one is connected to?

Hey unevent, those are good looking results. I wouldn't expect much of an improvement on your server since you are already tuned, but by dropping the values to the Best Bang values, you can free up some memory while keeping performance relatively the same. I know you are concerned about the recommended values being md_sync_window centric, with no regard for the other two metrics, but there actually is regard for the other parameters, just not a method to directly test them in this utility. What is happening behind the scenes is that we are finding how many stripes is needed to allow unthrottled access to your array, and this number is found by running a parity check while tuning the md_sync_window. This same number of stripes, by extension, is also a good value for the md_write_limit, as it should represent the point at which writes are unthrottled. At this point, admittedly this is just a theory, but it is a very rational theory that simply needs to be proven, or disproven as the case may be. The md_num_stripes represents two things: the total impact to server memory, as well as any additional overage, beyond syncs and writes, that remain for reads. I have maintained that same percentage allocation (10%) that stock unRAID allocated. Is this the best allocation? Probably not, but if it isn't then neither is the unRAID stock allocation, since they are one and the same. If anything, I theorize that reads (i.e. watching a Blu-Ray) while running a parity check may be slightly worse on a tuned system for one simple fact: Since the hard drives are now being accessed at their maximum potential by any one task (be it reads, writes or syncs), they have no spare processing capacity to handle multiple tasks. It is at this point that the balancing of reads vs writes vs syncs, handled by the mdcmd driver, ultimate impacts your viewing enjoyment of a Blu-Ray while running a parity check. It seems that your idea of tuning for playback enjoyment is to starve the parity check, leaving excess processing capacity on the hard drives so that reads have no competition. While that is a valid proposition, I don't think purposely de-tuning one process to get performance on another process is the greatest of solutions. I think your assertion that Tom de-tuned parity checks on purpose is conjecture, as we can see in many of these test results that optimum performance comes from unRAID stock values. If anything, Tom chose nice default values that work great on many servers, and work decent on all servers, while being sized for the smallest of servers. I would be interested to know how Tom prioritizes reads vs. writes vs. syncs: is this something hard coded into mdcmd, or is it controlled by the allocation % of these three tunable parameters. Perhaps someone would care to peruse the mdcmd.c source file and see if they can see a methodology. If it turns out that prioritization is controlled by these three parameters, since I never allocate more than 45% to syncs or writes, if you are running a parity check while playing a Blu-Ray, the Blu-Ray is getting 55% of the throughput, while the parity check gets 45%. And if you have a server that can deliver 100 MB/s, then in theory the Blu-Ray should be getting more than enough throughput at 55 MB/s theoretical maximum, since Blu-Rays only need about 5 MB/s. Of course, having hard drives thrashing about trying to do two things at once is never optimal, so theoretical maximum would never be reached. Obviously we've [all?] experienced streaming issues while running a parity check. As my 'back of the napkin' calculations show above, the cause isn't readily apparent, and we need more insight into the inner workings, and/or more tests to evaluate more combinations of factors. You've proposed some interesting sounding tests, but I'm not sure how you would go about building those tests. Primarily, I've never even see the number of IO's exposed in unRAID. How do you test something you can't even see? We also shouldn't be lulled into thinking that these three tunables are the only factors at play - these are just what Tom gave us to play with. NCQ may affect playback quality while running parity checks. So may cache memory parameters, available memory, driver revisions, etc. Any additional tests will probably have to be coded by someone else. This was primarily a utility for myself, and a nice little programming challenge to fill up some spare time. Since I don't really have any more spare time, nor a desire for anything more, I think it is time for someone else to take up the candle. And many unRAID users, like Patilan, make me look like a newb when it comes to Bash programming... and rightly so. They would be better choices moving forward. As far as making a statement in the initial post that this utility has no regard for the md_num_stripes and md_write_limit parameters, I will make no such statement, since I have clearly put consideration into how all three values are set. I freely described my methodology, and everyone is free to use or ignore this utility as they feel want to do.

Hey jbartlett, It certainly doesn't hurt to do a stop&start before the test. In all honesty, I haven't seen the rampant memory use I expected from running just these tests, so my recommendation may be overkill. In some of my earlier manual testing, when I was purposely trying to crash my server by running pre-clears and reads and parity checks simultaneously, my memory use would skyrocket and the server would get slow - a simple stop&start restored normal memory utilization without needing a reboot, so that is why I threw that recommendation in there. Maybe it is just me, but I think I see a bell curve on your hardware, it is just that the peak of your curve is at 384... or perhaps even lower! I think it would be interesting to run a test below 384 on your hardware - but the current version of the utility won't let you. If you're interested, I'll make a new version with a lower limit and post it today. Maybe you can go faster by going lower... -Paul

They can affect all, but the md_sync_window directly affects parity check speed, making it the easiest one to test, and since unRAID ships with a low value for md_sync_window, the parity check speed is the most likely to be afflicted with low performance. If your parity check speed is fine, then typically all else will be fine too (from an md_* tunables sizing point). Even if your md_sync_window value is low, disk transfers and writing speed may still be fine, since they ship with significantly higher values.

No need to use stock values when running the utility, since it sets it's own values for testing. While there clearly is an improvement in speed from increasing the variables, that improvement is small, and it did not result in a worthwhile decrease in parity check speed. Since stock values give you great results, you should use the stock values (md_num_stripes=1280, md_write_limit=768, md_sync_window=384). No need to waste memory on zero tangible improvement.

Hey jowi, I'm confused. You said you ran the first parity check at stock values, but the report indicates you already had an md_num_stripes of about 4460 - which definitely is not stock. The small increase from 4460 to 5984 (assuming that is the value you tested with) would have only provided a few seconds reduction in parity check time (as you reported). But had you really tested with stock unRAID values (1280/768/384) I think the processing time would have been much longer. Are you sure you tested correctly with stock unRAID values?