Pauven

Good work Patilan, I think you figured it out! It appears Tom is allocating a smidgen of extra memory, for whatever reason only he knows... about 3MB with stock settings. Probably thought we would never notice. I like you new math much better, I can easily add that formula to my code. Thanks!

I just spent a few minutes working the math. I had never looked at Tom's numbers before, and sure enough the numbers didn't add up. I think you are right about the Highest Disk + 1 for the Parity, as I have 15 drives (highest in slot 18) and mine reports 19 disks. Sep 2 14:57:57 Tower kernel: unraid: allocating 491584K for 6256 stripes (19 disks) In your case, the math behaves as if there are 206 more stripes than reported (1486 instead of 1280). In my case, 212 more (6468 instead of 6256). Interesting in that with the very different md_num_stripes values that we have, the core variance is very similar at about 209 stripes. You have 4.65k per stripe actual allocation, and I have 4.14k per stripe. Ah, I think I found it! The math works if you use 4.6445 for the number of disks, and same for me if I use 19.6445 disks. There's an extra 0.6445 disks in there. So the revised formula is: (Highest Disk + 1.6445) * 4096 * md_num_stripes Tom sure has some wonky math going on...

I just wanted to give a quick progress update and also say thank you to everyone who has been posting their results. Based upon the various results, I'm continuing to tweak the FULLAUTO routine in the attempt to make it better at finding the best options. My server produces a smooth bell curve, so basing the logic primarily on my server has made the routine less than ideal for other servers. Continue posting your test results, as they help me refine the FULLAUTO test logic to handle more scenarios. The improved routine does mean the new test routine will take longer. Unfortunately, I don't think there's anyway around that aspect, as shorter test intervals produce inaccurate results, and a smarter routine needs to check more test points. I already have the new write test coded, and will be extending it soon to perform read tests as well. Most of my work ahead of me is designing the menus and run-time options. I hope to have version 3.0 out this week. -Paul

Sounds like your Areca is not the problem. 60 MB/s write speeds is phenomenal for an unRAID server. Hopefully we can tune your write speeds soon for even better performance!

Hey John, Were both of these tests after your recent drive rebuild, when you upgraded your 32GB SSD? I was curious if the upgrade was going to affect your server's preference for lower md_sync_window values on parity checks. I find it interesting that your parity rebuilds increased in speed with higher md_sync_window values, and your parity checks increased in speed with lower.md_sync_window values. Your Bizarro server continues to baffle me. You might even find it helpful to have two sets of values that you switch between: one for everyday use and parity checks, and another for parity rebuilds. I still see a bit of inconsistency in your results, and I'm not sure that this was absolutely a result of your reboot. I expect that each time you run your tests, slightly different answers would result. I too see slightly different results with my tests (and boy have I done a lot of them!) but the peaks always fall the neighborhood of 2500-2900. Maybe should view these results as good enough to get you into the right ballpark, but that there is no single 'perfect value'. -Paul

Hey tyrindor, those are some great looking results. I see you have six of the AOC-SAS2LP-MV8 controller cards, which respond extremely well to adjusting these settings. Your free lowmem is actually much higher than what I typically see. In my post here I showed only 8MB of free lowmem: http://lime-technology.com/forum/index.php?topic=29009.msg260911#msg260911 In my experience, stock unRAID has been very stable when running short of lowmem, and it seems to swap/page into the highmem area effectively enough to keep my server from throwing errors, though it can get very slow if I push it too far. I know my experience is not shared with all, and it seems that users who install certain 3rd party plug-ins and add-ons most commonly have lowmem issues. I think we all need to be concerned about lowmem, especially when running add-ons. Adjusting these md_* tunables increases your risk of having lowmem issues, and unfortunately I don't believe that there is any way I can predict issues or even gauge the likelihood of issues based upon your server build. This is why I came up with the idea of analyzing the results to suggest 'Best Bang' values, trying to straddle that middle ground between ultimate performance and judicious memory use. Probably the best advice I can give is to keep an eye on your syslog, especially after making adjustments, for any types of errors. -Paul

I have 15 drives, with the last drive installed in position Disk18. I haven't counted how many files I have in a long time, but estimate it to be less than a million. I've never performed an rsync nor a locate. The majority of my data is comprised of Blu-Ray and DVD iso's, mp3's, and large backup files produced by backup programs. So my total number of files is somewhat limited, and I have very little use for locate or rsync. Our usage models are very different.

I understand the lowmem is static after boot, based upon the configuration you booted with. That's not what I was asking. What % of lowmem should I draw a line at for allocating memory to stripes? Obviously if I allocated 100%, we got problems. I could allocate up to 50%, and leave a nice buffer, or I could be more aggresive. I understand that there will be other processes competing for the same memory, so there's no hard and fast rule. I'm just looking for general guidance like "don't allocate more than 80% of total lowmem to stripes". This would primarily be to stop a FULLAUTO test at lower stripe values in case a particular server booted with significantly less lowmem (for whatever reason).

Total lowmem? 857184, and that's after a fresh boot. unRAID 5.0 + unMenu + screen + apcupsd + pciutils + dmidecode + bwm-ng root@Tower:~# free -l total used free shared buffers cached Mem: 4043520 1132884 2910636 0 335060 247944 Low: 867184 858224 8960 High: 3176336 274660 2901676 -/+ buffers/cache: 549880 3493640 Swap: 0 0 0 Is total lowmem a value I should check, and should I use it to limit how far the tests go? If so, how much spare lowmem should I leave, and would the formula be something like this? $SpareLowMem > ($TotLowMem - ( $md_num_stripes * 4 * $HighestDriveNum ))

Thanks for sharing jack0w. Looks like your server is also immune to influence from the md_sync_window. I really need to add 384 back into the FULLAUTO routine so I can see if there's any improvement over stock. I was skipping it based upon the assumption that higher values are faster, but so many results have proven that assumption wrong. -Paul

I find it interesting that one parameter showed benefit while the other did not. Glad to hear you got some benefit! When the new version comes out, you'll be able to tune your writes much more effectively. Interestingly, the AOC-SAS2LP-MV8 is the card that uses the same chipset as my HighPoint 2760A (and if you think the SAS2 is special, you really should take a gander at the 2760A). This chipset is highly influenced by the md_sync_window values, which has caused users to complain about performance for a long long time, and the discovery that the solution is adjusting the md_* tunables became the foundation of this utility. So while your motherboard might be immune to the parameters, the SAS2 most definitely is not! I appreciate it! -Paul

RAID1, correct. I'm prone to reverse the two in my head. I didn't mean to imply that unRAID is built to do something different when you have a single drive parity protected array, but my point was that the math collapsed to the point it was effectively mirroring, so the math is extremely fast.

My syslog is spotless so far. Are you asking me to double again, to 11776?!!!

Don't worry, I will have nice output. What I showed above wasn't actual output, just some data I cobbled together to get forum feedback. Formatting the data and performing the results analysis takes more coding than actually running the write tests! Choosing 100% performance will always be an option, but I also want to make it easy for people to choose options that still give great results with much lower memory usage. That's the best bang.

I'm looking for Best Bang feedback. Below are a few sample points from my write test. Which would you consider the Best Bang for the MB? Or do you have another criteria in mind? A) 100%: 49.7 MB/s @ md_write_limit=5568 uses 435MB <-- Fastest recorded write speed B) 95%: 47.3 MB/s @ md_write_limit=2688 uses 210MB <-- Half the memory usage with 95% of the speed C) 90%: 45.2 MB/s @ md_write_limit=1792 uses 140MB <-- 1/3rd the memory usage with 90% of the speed D) 85%: 43.3 MB/s @ md_write_limit=1280 uses 100MB <-- 1/4th the memory usage with 85% of the speed E) 79%: 39.1 MB/s @ md_write_limit= 768 uses 60MB <-- unRAID Stock Keep in mind that the memory usage above is for the writes only - total memory usage would be this plus sync plus an overage for reads. Also, the memory usage above is calculated for my server, with 20 drives. Each server is different.

I can't say for sure that adding additional drives will have an impact. Only one way to find out. It's certainly possible that your system works great at any combination of md_* values, possibly as a result of using the 6Gbps ports of the ASRock FM2A85X-ITX. We are seeing that different hd controllers has an impact on what set of values brings optimum performance. I know that this is a SFF build, but if you were adding more drives than the MB could handle, forcing you to include an add-in card, most likely these parameters would have an impact. Since your system works well with stock values, you're in great shape.

Thanks for sharing RockDawg. There's something wonky in your results. The speed increase seen in the 1152 to 1408 range was not repeated in Pass 2. And from test 10 on, your results appear to be stuck at about 91.5 MB/s, all the way through the end of Pass 2. Does anyone have any ideas on what may be causing this behavior, and/or any suggestions on how to prevent it? I wouldn't think cache would be coming into play on a parity check, especially one running for 3-4 minutes, as that is a lot of data from a lot of drives, but I really don't know. -Paul

Nothing wrong with the script, or your build, but you have only two drives in your test build: 1 parity and 1 data. You have effectively created a RAID 0 using unRAID. This means unRAID doesn't have to do any calculations or anything. When building or checking parity, unRAID simply has to make parity=datadrive. Mirrors are fast. That is why you are seeing parity check speeds at full disk speeds. That is also why these parameters are not doing anything on this specific build. Had you selected either Unthrottled or Best Bang, the very next prompt would have allowed you to revert to original values. In fact, I programmed so that if you fail to select either Unthrottled or Best Bang, it automatically reverts to original values. Had you simply hit ENTER instead of CTRL-C, you would be back to the original settings, but since you did a CTRL-C, your settings were left at the last test values. You can always hit APPLY again from the unRAID Disk Setttings page to get your settings back to configured values. I'll try and make the options friendlier in the next version.

As a teaser of what's coming in the next version, here's my write test results. Big thanks to WeeboTech for sharing his knowledge. The dd command is fantastic, and this was easy enough to program that I will be doing both read tests and write tests. I still have quite a bit of programming to do, so it won't be out for a few days. I think I hit peak write performance just shy of 50MB/s, which is a nice 25%+ boost over stock unRAID settings. This was a 6 hour test, with 10GB written for each pass. This was conducted on a WD Red 3TB NAS drive (5400RPM) that was 48% full. Unfortunately it was my emptiest drive. I might have to move some data around to be able to test on the fast part of the drive. -Paul md_write_limit=128, 301.676 s, 33.9 MB/s md_write_limit=192, 317.47 s, 32.3 MB/s md_write_limit=256, 283.058 s, 36.2 MB/s md_write_limit=320, 298.919 s, 34.3 MB/s md_write_limit=384, 297.67 s, 34.4 MB/s md_write_limit=448, 302.064 s, 33.9 MB/s md_write_limit=512, 297.018 s, 34.5 MB/s md_write_limit=576, 306.282 s, 33.4 MB/s md_write_limit=640, 290.022 s, 35.3 MB/s md_write_limit=704, 273.539 s, 37.4 MB/s md_write_limit=768, 261.84 s, 39.1 MB/s md_write_limit=832, 269.1 s, 38.1 MB/s md_write_limit=896, 272.334 s, 37.6 MB/s md_write_limit=960, 276.062 s, 37.1 MB/s md_write_limit=1024, 268.788 s, 38.1 MB/s md_write_limit=1088, 267.094 s, 38.3 MB/s md_write_limit=1152, 260.519 s, 39.3 MB/s md_write_limit=1216, 245.766 s, 41.7 MB/s md_write_limit=1280, 236.244 s, 43.3 MB/s md_write_limit=1344, 235.364 s, 43.5 MB/s md_write_limit=1408, 238.502 s, 42.9 MB/s md_write_limit=1472, 239.953 s, 42.7 MB/s md_write_limit=1536, 236.889 s, 43.2 MB/s md_write_limit=1600, 234.248 s, 43.7 MB/s md_write_limit=1664, 232.905 s, 44.0 MB/s md_write_limit=1728, 232.573 s, 44.0 MB/s md_write_limit=1792, 226.406 s, 45.2 MB/s md_write_limit=1856, 224.929 s, 45.5 MB/s md_write_limit=1920, 227.07 s, 45.1 MB/s md_write_limit=1984, 230.45 s, 44.4 MB/s md_write_limit=2048, 226.18 s, 45.3 MB/s md_write_limit=2112, 226.925 s, 45.1 MB/s md_write_limit=2176, 223.399 s, 45.8 MB/s md_write_limit=2240, 223.267 s, 45.9 MB/s md_write_limit=2304, 223.151 s, 45.9 MB/s md_write_limit=2368, 219.552 s, 46.6 MB/s md_write_limit=2432, 223.168 s, 45.9 MB/s md_write_limit=2496, 220.98 s, 46.3 MB/s md_write_limit=2560, 220.718 s, 46.4 MB/s md_write_limit=2624, 218.409 s, 46.9 MB/s md_write_limit=2688, 216.448 s, 47.3 MB/s md_write_limit=2752, 216.608 s, 47.3 MB/s md_write_limit=2816, 217.598 s, 47.1 MB/s md_write_limit=2880, 216.989 s, 47.2 MB/s md_write_limit=2944, 216.578 s, 47.3 MB/s md_write_limit=3008, 216.179 s, 47.4 MB/s md_write_limit=3072, 213.888 s, 47.9 MB/s md_write_limit=3136, 212.857 s, 48.1 MB/s md_write_limit=3200, 211.932 s, 48.3 MB/s md_write_limit=3264, 212.881 s, 48.1 MB/s md_write_limit=3328, 213.732 s, 47.9 MB/s md_write_limit=3392, 214.099 s, 47.8 MB/s md_write_limit=3456, 214.81 s, 47.7 MB/s md_write_limit=3520, 212.939 s, 48.1 MB/s md_write_limit=3584, 212.713 s, 48.1 MB/s md_write_limit=3648, 211.289 s, 48.5 MB/s md_write_limit=3712, 210.06 s, 48.7 MB/s md_write_limit=3776, 211.82 s, 48.3 MB/s md_write_limit=3840, 209.016 s, 49.0 MB/s md_write_limit=3904, 210.22 s, 48.7 MB/s md_write_limit=3968, 212.187 s, 48.3 MB/s md_write_limit=4032, 210.77 s, 48.6 MB/s md_write_limit=4096, 208.326 s, 49.2 MB/s md_write_limit=4160, 209.331 s, 48.9 MB/s md_write_limit=4224, 208.988 s, 49.0 MB/s md_write_limit=4288, 207.678 s, 49.3 MB/s md_write_limit=4352, 210.226 s, 48.7 MB/s md_write_limit=4416, 209.887 s, 48.8 MB/s md_write_limit=4480, 210.053 s, 48.7 MB/s md_write_limit=4544, 210.036 s, 48.8 MB/s md_write_limit=4608, 209.409 s, 48.9 MB/s md_write_limit=4672, 210.029 s, 48.8 MB/s md_write_limit=4736, 206.886 s, 49.5 MB/s md_write_limit=4800, 208.978 s, 49.0 MB/s md_write_limit=4864, 206.788 s, 49.5 MB/s md_write_limit=4928, 207.969 s, 49.2 MB/s md_write_limit=4992, 208.136 s, 49.2 MB/s md_write_limit=5056, 207.687 s, 49.3 MB/s md_write_limit=5120, 207.471 s, 49.4 MB/s md_write_limit=5184, 206.618 s, 49.6 MB/s md_write_limit=5248, 210.428 s, 48.7 MB/s md_write_limit=5312, 210.248 s, 48.7 MB/s md_write_limit=5376, 208.4 s, 49.1 MB/s md_write_limit=5440, 209.255 s, 48.9 MB/s md_write_limit=5504, 207.281 s, 49.4 MB/s md_write_limit=5568, 205.834 s, 49.7 MB/s md_write_limit=5632, 207.783 s, 49.3 MB/s md_write_limit=5696, 209.198 s, 48.9 MB/s md_write_limit=5760, 208.117 s, 49.2 MB/s md_write_limit=5824, 208.329 s, 49.2 MB/s md_write_limit=5888, 208.249 s, 49.2 MB/s

Awesome. Thanks for testing John. And thanks for making a liar out of me. Now I get to spend 15 minutes correcting my mistakes in earlier posts...

Moves from the cache drive to the array would not be limited by network speed, so there is a reason to test beyond what a network can provide. I used to be a network engineer, and I don't trust LAN connections as a variable in testing hard drive speed. Too variable. The only time to run a performance test that includes LAN connections is when you are actually testing LAN connections. I'm not saying you aren't able to get good results testing transfers from a Windows PC over a LAN connection, but I wouldn't engineer a test script around it, way too many variables. It's also more complex than the dd method. As far as I know, since the cache drive is outside the array, md_* parameters don't affect it.

All good info, thanks WeeboTech. I was thinking, regarding selecting an empty drive - it's not just about selecting the emptiest drive, but rather about selecting a drive with free outer sectors, where performance will be greatest. It's possible (not likely, but possible) to have a drive 90% full, but because some files were deleted the outer cylinders are empty. Any way to check for that condition? And if the answer is yes, is there any way to control where on a drive the file gets written to?

Yup, I caught and corrected that typo about 30 seconds before you posted. My post above is correct now. Er, I mean I was just checking if you were awake...

Yup. That's a pretty good time considering your mix of drives.

Alright, so I read through some documentation on the dd command (completely new to me), and I can't believe I'm gonna say this, but I'm thinking about incorporating a routine to test md_write_limit using dd. There are some logistical problems that I didn't have to worry about with a parity check, so I'm open to suggestions on the following: How to choose what drive to write to? A) Write to All Drives and select the fastest for the test B) Let the user specify C) Automatically select the emptiest drive D) Automatically select the largest drive, and if more than one select the emptiest E) Make A-D selectable options and make the user choose F) Another option I failed to think of dd also has a nickname - data destroyer. Misuse of the tool can easily cause a loss of data. I need to make sure that the dangerous options are not user selectable. It looks all too easy to perform a full disk wipe by specifying of=/dev/sda instead of of=/mnt/disk18/testfile10gbzeros.txt. Scary. cp is child's play next to dd. I didn't see any logic in your script to prevent someone from specifying a whole drive or partition as the output file... Since we will be writing a file, I think we need to make sure that the filename doesn't already exist (with real user data). I bet that would really piss someone off. Do the kernel caches need to be dropped after each test, or can I run 10 tests back to back with different md_write_limit values without dropping the caches? I assume with dd, I would log the time, call dd to write the test file, and when it completes log the time again, right? Something like this? testloop (){ /root/mdcmd set md_write_limit $WriteLimit sync && echo 3 > /proc/sys/vm/drop_caches StartTime=$(date +%s.%N) dd if=/dev/zero bs=1024 count=10485760 of=/mnt/$DiskNum/testfile10gbzeros.txt EndTime=$(date +%s.%N) Duration=`echo "$EndTime $StartTime" | awk '{ printf( "%0.3f\n", ($1 - $2) )}'` Speed=`echo "$Duration" | awk '{ printf( "%0.1f\n", ((10737418240 / $1) / 1024) )}'` } I appreciate any insight. -Paul I EDITED TO CORRECT A STUPID MISTAKE ON SPECIFYING THE FILE PATH...