Pauven

Awesome, thanks unevent. So after creating a ramdisk, you could load it up with test files, then run the write tests. Do you think there is a way to turn those steps into a script? How would you make a script smart enough to choose conservatively? Sounds like there are several variables to consider. Not that I understand it, but WeeboTech's suggestion might be better. I don't think you have to create a ramdisk or allocate memory. I think it might be a way to create test data out of thin air, so to speak. I'm concerned that allocating some memory to a ramdisk could influence the results, not to mention the possibility of crashing the server if you run out of memory.

Sounds great! But I have no idea what any of that means. Can you put it in layman terms? That makes sense. So a newly added, empty drive is a great place to start? I would think you would also want the drive to be the same model as your fastest drive, otherwise if you are testing a slow drive you might come up with values that don't work well for faster drives. Of course, I am also making the assumption that values that work well on your fastest drive will also work well on your slowest drive. Any chance you might make a script out of this? This is way beyond my technical ability.

So I googled the term and WeeboTech is right about the technical definition of Thrashing. I've certainly been using the term incorrectly. I was looking for a single word that describes high seek activity with very large head movements caused when a hard drive is responding to competing requests that send the actuator arms back and forth between the inner and outer cylinders of a platter, ultimately causing total data transfer rates (summed together) to be far below the achievable rate if the hard drive was able to focus on a single request. Basically, a drive that is so busy seeking instead of reading and writing that data transfer rates fall. While in my mind the mental image of a hard drive's actuator arm rapidly seeking over the platters, like a teenager flailing their arms about, was best described by the word thrashing (like something you would do at a metal band concert), technically this is not Thrashing at all. From what I've just read, a more technically correct definition of the scenario I am describing is a drive Saturated with Random I/O from Competing Requests, though I would still love a single word that describes this condition, and perhaps WeeboTech is right and the word is Competing. Regardless, though some of us were using the wrong term, we were all talking about the same problem: drives saturated with random I/O from competing requests, and the need to prioritize some requests over others. Anyway, back to the core topic.

That's an elegant solution to eliminate network performance as a factor. Having a very high performance cache drive, perhaps a modern fast ssd, would also help mitigate external influence. A RAM drive, with extremely low latency and high transfer speeds, would still be best. Any way to create a RAM drive and assign it as a cache drive (or simply mount it at all) in Linux?

After tuning on my server, I found that read checks were slightly faster than rebuilds. Stock reads checks were much slower. At least in my case, with my hardware, the tunables appeared to improve read check performance to hardware maximum, while rebuilds were already at maximum.

Sorry I failed to mention that. It simply increments to the next set of test values. I forgot it doesn't call CalcValues. I should probably make a version that does all the calcs in one place so yo don't have to go hunting.

Awesome, thanks! I don't think you need to repeat with the SSD. You can analyze it's impact on the next parity check. -Paul

You make some good points, but I'm concerned how many variables you have in play here. Network speed and/or desktop speed might be the limiting factors preventing higher values from improving write performance. I believe some benchmark programs fabricate data out of thin air when performing write tests, eliminating all variables except for HD write performance. Short of that, having a 10Gb/s LAN connection, transferring data from a RAM drive, might minimize those variables. Since you have not eliminated those variables, I do not think we can conclusively say whom is correct. It read like you though md_max_read_stripes=md_num_stripes - md_sync_window, which is what you thought I was referring to. You make some excellent points, that since reads can empty twice as fast as writes, the stripes empty faster. Does that mean you can do with half the number of read stripes vs. write stripes, since it is harder to fill them up? Or do you need to double them up since they empty twice as fast? Keep this in mind, though: Parity Checks/Rebuilds and Reads are all unidirectional - each drive is only accessed in a single direction. Since the number of stripes allocated is 'per drive', then it stands to reason that reads and syncs need a similar number of stripes. That would make writes the only outlier, since writes are bidirectional (data has to be both read and written from the parity and data disk to calculate the new parity bit on the fly). So do you need double the write stripes to maintain full speed, or half as many? -Paul

I have NCQ disabled, as I found it made my parity checks slower. But I've been meaning to test again while trying to stream a movie to see if it makes a difference.

John, thank you very much! Your results have proven very valuable! Once again, your Bizarro server strikes again. Lower values have produced slower times. This might be because you replaced the 32GB SSD. We won't know for sure until your rebuild is done and you can test normal parity checks again. What is interesting is that we see throughput changes even when md_write_limit is static at 768. I've never seen that in my results. That means that rebuilds are linked to either md_sync_window or md_num_stripes (or both). At least we know rebuilds are not connected to md_write_limit, and for that I thank you. If you wanted to, you could run one more test, from 8 to 520 (where the values changed quite dramatically), but with the md_num_stripes hard coded to 1544 (you can hard code this in the CalcValues procedure at the top of the program). This would show whether it is md_sync_window or md_num_stripes that is impacting rebuild speed. -Paul

I think we are all describing the same thing, just using different words. Semantics. I'm using the term thrashing to describe the behavior of disk heads inefficiently moving around to different cylinders/sectors to satisfy competing requests. In all my years of IT, I've never heard the term thrashing used to describe a stall, but perhaps I've just never heard the correct technical definition. I think we all agree work is getting done, it's just inefficient due to competing resources. This is where load prioritization in unRAID, allowing you to put one queue at a higher priority over another, would be the best possible solution. Work still gets done, but in the order the user prefers, and the work becomes somewhat more efficient since there will be less head movement to satisfy competing requests, as complementary requests with the same priority get bundled together.

This is what I would call 'load optimization'. Basically Tom just needs to give a priority to each of the queues, and if he exposed those priority assignments to us as tunable values, then we could choose which queue gets processed first. If you set parity checks to the lowest priority, then reads and writes would process through the queue first, minimizing thrashing and providing faster read/write performance. If you prioritized reads over writes over parity checks, then your Blu-Ray might actually play back smoothing while running other tasks! There is a way to move the heads differently - it is called Native Command Queuing (NCQ). This allows the drive to re-order requests, grouping them together based upon the area of the platter the data resides, to minimize thrashing and improve overall throughput. Tom makes this selectable as a disk tunable. It's quite possible that enabling this will allow you to watch movies while running a parity check, so it's worth playing with. I haven't tested recently, but in my earlier manual tests, having NCQ enabled changed the parity check performance at various md_sync_window values - so you would want to do a FULLAUTO again with NCQ enabled to find the right values with NCQ. That's very interesting. I haven't been stopping anything on my server when doing my tests, and I get very smooth test results. I don't have a lot running, but I do have cache-dirs, unMENU, and a few other items. I'm not sure if running barebones is the right answer, as running with plug-ins makes the results applicable to your normal, every day use model. But certainly allowing enough time for your plug-ins to settle down is very wise.

Interesting ... are you SURE about that? Pretty much positive, at least on my server. Remember how I originally documented the problem with parity checks because they were so much slower than rebuilds. In addition to testing parity checks with the md_sync_window values, I also tested rebuilds (I upgraded all my Samsung drives to the 3TB Reds) and my rebuild speed never changed when I tried different md_sync_window values. I had also increased md_num_stripes during those rebuilds as well to no effect, which basically leaves just md_write_limit. I never tested changing the md_write_limit values during my rebuilds, which is why I'm helping John conduct that specific test. If the rebuild speed changes at all, then most likely it is directly connected to the md_write_limit. Of course, that's just a guess. For all I know, Tom considered a rebuild so critical that he set the the rebuild to use md_num_stripes... all of em, to maximize rebuild speed. Seems a logical thing to do, but again my test results during rebuild didn't hint that the md_num_stripes had any impact on rebuild speed, but since that value was so high to begin with, maybe any increase was too small for me to notice. I'm a dumbass who though I knew what I was talking about. Turns out I was wrong wrong wrong. But I'm not too ashamed to admit it. John just did some tests to check if md_sync_window affects parity rebuilds, and sure enough it does. Interestingly, the rebuild speed appears to scale faster than parity check speed using the same values, and speed maxes out with lower values. That might be why I didn't observe a relationship in my earlier testing.

garycase, your ideas on the writes and reads are similar to mine. Let me compare and contrast: Your statement is a little confusing, as you first say md_write_limit doesn't need to be modified at all, then you go on to describe modifying it to positive results. I think what you intended to say is that you don't want it modified as part of this utility, and that it should be independently tested and set based upon a series of write tests. I agree that write tests are the way to verify the correct setting of md_write_limit, but let me explain why I am setting it in this utility: The md_sync_window is the maximum number of stripes available to the parity check. The md_write_limit is the maximum number of stripes available to writing operations. My hypothesis is that the same number of stripes that maximizes parity check performance IS THE SAME number of stripes that maximizes write performance. My hypothesis is unproven, of course... it's nothing more than a theory. But if my theory is correct, then you don't have to test writes at all, just test the parity check. Once you find the right value, set it to both parameters, sync window and write limit. I think you've almost proven my hypothesis: You indicated that a md_write_limit of 1500 gave you good results, and 2500 basically maxed out performance. How do those numbers compare to your most recent FULLAUTO test results? I would expect that md_sync_window 1500 gives you good results, and about 2500 basically maxes out performance. Can you verify? I mostly agree with your statements here. Yes: "md_min_read_stripes" = md_num_stripes - md_write_limit - md_sync_window But Also: "md_max_read_stripes" = md_num_stripes (this is slightly different that what you stated) I agree that making it larger than necessary won't further improve performance, but because this is a dynamic allocation of stripes, varying between min_read and max_read depending upon what other tasks are being performed, you have to make a choice as to whether you target optimum read performance on the min_read end of the spectrum, the max_read end of the spectrum, or somewhere in the middle. Extending my hypothesis on md_write_limit, I believe that reads reach max performance at the same number of stripes that writes reach max performance, which is the same number of stripes that parity checks reach max performance. For easy numbers, let's pretend max parity check performance is reached at 1000 stripes. md_sync_window=1000 And since my theory states that write stripes = sync stripes: md_write_limit=1000 Then, if you want to provide for max read performance while both writing and running a parity check: md_num_stripes=3000 (calculated as md_sync_window + md_write_limit + 1000). But keep in mind, with those settings, if all you are doing is reading (not writing and not running a parity check) then the number of stripes allocated to reading is 3000, more than 3x what you need for optimum performance. So we can lower it and still get good read performance as long as you are not also writing and running a parity check. So if we set md_num_stripes=2000, you still have 2x what you need for optimum read performance when you are only reading. If you start up a parity check, it would drop the available read stripes down to 1000, so it would still be optimal. If instead of running a parity check you were writing files, that too would drop the reads down to 1000, so it would still be optimal. Problems only occur if you start a parity check and write some files while reading, as that would drop your available read stripes down to 0 - they would get starved and reads would stall. So we want to make md_num_stripes > md_sync_window + md_write_limit. In the above scenario, I personally think 3000 is excessive, as you now have completely maximized reads, writes and syncs, but since the hard drives are thrashing around trying to do 3 things at once, you are not getting good performance, and you wasted a lot of memory trying to maximize something that can't really be maximized. My formula follows Tom's original formula by making md_num_strips ~11% bigger (technically 10% of the whole value dedicated to reads). So instead of 3000, we would set md_num_stripes=2222. That way, when only reading, the read stripes are more than 2x what you need for optimum read performance. And if you are either reading and writing, or reading while running a parity check, the read stripes are still more than what you need for optimum read performance. It's only when you are reading, writing and running a sync, all at the same time, that read performance would drop below optimal, with only 222 read stripes available in the above scenario. I think that is a fair compromise that reins in memory use a bit, but I'm sure others disagree. I hear ya, but if my theories about how these three parameters work are correct, then my utility is already applying the logic I laid out (md_write_limit=md_sync_window, and md_num_stripes=(md_write_limit + md_sync_window)*1.11, which means it is doing a better job than you can do manually. If anything, I can see making it selectable how much larger to make md_num_stripes. Some people may want 11%, some may want 50%, etc. You don't need to do any other tests - no read tests, and no write tests - because the parity check test already told you everything you need to know about the correct number of stripes, and you simply use a formula to set the other two values based upon md_sync_window test results. That's the nice thing about my theory - if I'm correct, then you don't have to worry about any of that! Anyway, I think it would be great if some users (like yourself) could independently test my theories. I don't think people are too willing to believe me if I am both the source of the theories and the tester, as I am biased on the results. -Paul

I'm running a parity check now, will know in the morning, but so far it looks the same, maybe a smidge faster. 7:34:51, so a few seconds faster with the new values on 5.0 final.

Hey John, Sorry you caught me sleeping. I'm east coast. From earlier tests, I already know that rebuilds are not affected by md_sync_window. [EDIT: I was wrong, rebuilds are affected by the md_sync_window. Tests performed by John have proven it.] The goal of this test is to see if they are impacted by md_write_limit. Don't need a full spectrum test, not worth it. I recommend extreme length, 128 byte interval, start at 128 and end at 768. If you made the two changes in the code (change NOCORRECT to CORRECT, and WriteLimit=768 to WriteLimit=128) then for byte values 128 and above, md_sync_window and md_write_limit will be set to the exact same test values. The test results will then show if there is any difference in speed during a parity rebuild. Don't worry about actually trying to find the best value for the write limit, as garycase was suggesting, that's not the goal of this particular test. We are merely trying to establish that write limit is connected to rebuilds. The test above should only take an hour. Did that answer your questions? -Paul

I was actually thinking almost the same thing about half an hour ago, though I was thinking about logging what hd controller chipsets were responding well to which values. That's probably a little too nitty gritty, and your idea seems more helpful.

Not only would ditching the 2TB drives get you close to 140 MB/s, it would also eliminate the dreaded slowdown partway through the parity check. This will save hours on your parity check. I was closer to 14 hours just a few months ago, and ditching the 1.5TB and 2TB drives, combined with tuning the md_* settings, almost cut my parity in half. Of course, I'm simply amazed that servers based on 4TB 7200RPM drives can churn out 6.5 hour parity checks. Blazing!

unevent thanks for confirming. John, sounds like you could just change the NOCORRECT to CORRECT in the utility. Keep in mind that the current version never sets md_write_limit below 768, but you could change that easily - first line in the CalcValues, WriteLimit sets the lower limit. You could change that to 128 and the script would handle the rest. Keep us posted if you decide to do this! -Paul

I would say your speeds max out at test # 7, and all the results after that are pretty much identical (normal variation in test results). Those values are low enough that I think you could set them there and call it a day. Interestingly, even the lowest values in the test produced good speeds. Considering that you have 2TB drives in the mix, these really are great results. The difference between test 1 and test 7, over a full length parity check, is probably less than a minute. Consider yourself lucky that your server doesn't appear to be afflicted with the throttled... er, I mean ball-less performance that some of our servers exhibit when running stock values. -Paul

Man! I was hoping I wouldn't have to open up the tower since I decided to upgrade the SSD (external drive bays).... *snerk* I'm pretty sure they are, I wouldn't short-change a card in a slot. -John ETA: Specs report the slots as PCIe 2.0 x16 I just read up on your motherboard, the M4A88TD-V EVO/USB3: 1 x PCIe 2.0 x16 (blue) 1 x PCIe 2.0 x16 (x4 mode, gray) 1 x PCIe 2.0 x1 3 x PCI Most likely you had to plug into the two x16 sized slots. The gray one is only capable of x4, but that lines up well with your card. If you have lspci installed, you can run lspci -vv to get the actual bandwidth that the cards are running. I checked out the card specs, and they have the Marvell 88SX7042 chip powering them. Seems like a decent budget chip, but I haven't had much luck finding tech specs on it. It should be running the sata_mv drivers. If I counted correctly, you have 11 drives in your array (not counting the cache drive). Since you only have 8 ports available on these two cards, that means some of your drives on on the motherboard, right? The motherboard has 5 x SATA 6Gb/s ports controlled by the AMD SB850 chipset. Not quite sure what driver it runs. Regardless, you've got two different hd controller chipsets in your system, and either one, or the combination of both, is probably what is driving your server's bizarro behavior with a fondness for small bytes. Not that I think it is a problem, just very interesting. Looking forward to your sub 128 test results. -Paul

Yes, I'm very interested. Hope it goes well! -Paul

I'm running a parity check now, will know in the morning, but so far it looks the same, maybe a smidge faster. Since we have the same hard drives, I'm guessing it might be hd controller driver related. What driver is used on your box? MVSAS is used on mine. Maybe there's been some changes with different kernel releases. Unless your CPU's are peaking, I don't think CPU horsepower has much affect. Any chance your memory speeds changed on you? I've had some memory modules that don't always run their rated speed, and a reboot might bring me up in a slower speed. My Balls to the Wall and Best Bang values inched up a bit, as did the reported speeds, but for the most part it looks similar to earlier test runs. Tunables Report from unRAID Tunables Tester v2.2 by Pauven Test | num_stripes | write_limit | sync_window | Speed --- FULLY AUTOMATIC TEST PASS 1 (Rough - 20 Sample Points @ 3min Duration)--- 1 | 1408 | 768 | 512 | 105.7 MB/s 2 | 1536 | 768 | 640 | 120.5 MB/s 3 | 1664 | 768 | 768 | 128.7 MB/s 4 | 1920 | 896 | 896 | 132.3 MB/s 5 | 2176 | 1024 | 1024 | 135.0 MB/s 6 | 2560 | 1152 | 1152 | 135.8 MB/s 7 | 2816 | 1280 | 1280 | 136.7 MB/s 8 | 3072 | 1408 | 1408 | 137.0 MB/s 9 | 3328 | 1536 | 1536 | 137.3 MB/s 10 | 3584 | 1664 | 1664 | 137.7 MB/s 11 | 3968 | 1792 | 1792 | 137.7 MB/s 12 | 4224 | 1920 | 1920 | 137.8 MB/s 13 | 4480 | 2048 | 2048 | 138.1 MB/s 14 | 4736 | 2176 | 2176 | 138.2 MB/s 15 | 5120 | 2304 | 2304 | 138.4 MB/s 16 | 5376 | 2432 | 2432 | 138.7 MB/s 17 | 5632 | 2560 | 2560 | 138.8 MB/s 18 | 5888 | 2688 | 2688 | 138.8 MB/s 19 | 6144 | 2816 | 2816 | 139.1 MB/s 20 | 6528 | 2944 | 2944 | 139.1 MB/s --- Targeting Fastest Result of md_sync_window 2816 bytes for Final Pass --- --- FULLY AUTOMATIC TEST PASS 2 (Final - 16 Sample Points @ 4min Duration)--- 21 | 5984 | 2696 | 2696 | 139.0 MB/s 22 | 6008 | 2704 | 2704 | 138.9 MB/s 23 | 6024 | 2712 | 2712 | 139.0 MB/s 24 | 6040 | 2720 | 2720 | 138.8 MB/s 25 | 6056 | 2728 | 2728 | 139.1 MB/s 26 | 6080 | 2736 | 2736 | 139.0 MB/s 27 | 6096 | 2744 | 2744 | 138.9 MB/s 28 | 6112 | 2752 | 2752 | 139.1 MB/s 29 | 6128 | 2760 | 2760 | 139.0 MB/s 30 | 6144 | 2768 | 2768 | 139.0 MB/s 31 | 6168 | 2776 | 2776 | 138.9 MB/s 32 | 6184 | 2784 | 2784 | 139.1 MB/s 33 | 6200 | 2792 | 2792 | 138.9 MB/s 34 | 6216 | 2800 | 2800 | 139.0 MB/s 35 | 6240 | 2808 | 2808 | 139.1 MB/s 36 | 6256 | 2816 | 2816 | 139.2 MB/s Completed: 2 Hrs 6 Min 55 Sec. Best Bang for the Buck: Test 7 with a speed of 136.7 MB/s Tunable (md_num_stripes): 2816 Tunable (md_write_limit): 1280 Tunable (md_sync_window): 1280 These settings will consume 198MB of RAM on your hardware. Unthrottled values for your server came from Test 36 with a speed of 139.2 MB/s Tunable (md_num_stripes): 6256 Tunable (md_write_limit): 2816 Tunable (md_sync_window): 2816 These settings will consume 439MB of RAM on your hardware. This is 25MB more than your current utilization of 414MB.

I'll check again. Balls to the Wall, I was at 7:35 on RC16c, and my Best Bang was 7:40. I had no idea your times had climbed back up from sub 7:30's. I've been saving my TunablesReport.txt reports, so I should be able to spot any differences. I'm baffled at the differences you're seeing - nothing in Tom's change log would have made me think this would be affected, but maybe he's doing some under the hood tweaking that he doesn't post in the change logs. -Paul

I just looked in the md.c source code, and here are the commands I've found: set (md_trace, md_num_stripes, md_write_limit, md_sync_window, invalidateslot, resync_start, resync_end, rderror, wrerror, spinup_group, rdev_size) import start stop check (with CORRECT or NOCORRECT options) nocheck clear dump spindown spinup I don't see a parameter that looks like it kicks off a rebuild. I read through the source code, and I don't see a command line entry point for a rebuild. It seems you might only be able to do this through the GUI. It's possible that running a mdcmd check CORRECT, if exactly one disk is 'invalid', then it "might" start a rebuild, but that is purely a guess. From looking at the source code, I don't think it connects.