Pauven

Awesome, thanks! I see swings of +/- 10 MB/s in these results for a handful of repeated test points. Doesn't look like the server has quite settled down with 30 sec duration tests to provide any accurate answers. That said, it got surprisingly close to your current configured values. -Paul

Yes, 4 disks. Okay, thanks. I was hopeful that the lshw output was arranging everything hierarchically, but that doesn't seem to be the case. I may try it without the -businfo flag in v4b3. Paul

Yay, the first Normal Auto! Unfortunately, really bad results (I trimmed them out above, these posts are getting too long for the forum character limit). There is a ton of randomness here, it didn't settle down at all. Even worse, look at this: Test | num_stripes | sync_window | nr_requests | sync_thresh | Speed --------------------------------------------------------------------------- 1 | 1536 | 768 | 128 | 767 | 110.5 MB/s <-- nr_requests test pass 1, 10 minutes ----------------------------------------------------------------------------- 1 | 1536 | 768 | 128 | 767 | 112.2 MB/s <-- nr_requests test pass 2, 10 minutes Test | RAM | num_stripes | sync_window | nr_reqs | sync_thresh | Speed ----------------------------------------------------------------------------- 7a | 81 | 1536 | 768 | 128 | 767 | 239.0 MB/s <-- md_sync_window test pass 1, 5 minutes --------------------------------------------------------------------------- 17a | 81 | 1536 | 768 | 128 | 767 | 89.8 MB/s <-- md_sync_window test pass 2, 5 minutes The exact same test happened to get run 4 times, with wildly varying results. I can't even begin to understand how that 239 MB/s result got in there, it's just so different, and high. 5 minutes at 239 MB/s is about 70 GB of data processed. Can't imagine this is caching. You also posted earlier you were having power issues to your cages. Any chance the power went out just long enough to let one test parity check run without the slow drives? This is actually my best guess at how this could have happened, but I would have expected lots of issues (redballs) if this occurred. Did you notice any errors at all during the test? They wouldn't be in the report file, just on the screen. If that had been a Short Auto, I wouldn't be surprised. But this being a Normal Auto, I'm not sure what to make of this. Those top two tests I listed were from the nr_requests tests 1 and 2, and those run for 10 minutes instead of 5, and they "might" have produced more consistent results (or maybe that's just a coincidence). Those bottom two tests were from Pass 1 and 2 of the md_sync_window tests, which "only" ran for 5 minutes. Perhaps that's not long enough for your server to settle down. Anyone have any ideas? Long Auto is a big time commitment, and I'm not ready to recommend that just yet. If you are willing, I would recommend you run a Manual test, Start at 768, End at 768, any interval, with a time of 20 minutes. Repeat 1 more time. Because it will pre-run the 4 nr_requests, doing this will take 100 minutes each time, so if you do this twice you're looking at almost 3.5 hours of testing. What I'm looking for is: Does the server settle down with longer tests? (hopefully 20 minutes will tell us), and by running the same values I listed above, we can compare 20 minute tests back to 10 and 5 minute tests. But if you are still having power issues, please resolve those first. Thanks, Paul

Is anything connected to the 1430A? From the lshw output, it looks like it is going unused (see above), but not sure if that is reporting correctly. On this server, when nr_requests=128, high md_sync_thresh values return horrible results. While the script did run test 1c on this server, getting very close to your original values, the second test of the nr_requests switched from 8 to 128 as the better value, so the rest of Test 2 was run at 128. I have a feeling this was just the random noise of a Short Auto test, and that you'll get much better results from the Full Auto. Thanks, Paul

Fascinating server. Just like Tower7, there was too much randomness with the Short Auto to trigger Test 1c, so Tower5 never tested the upper ranges. Hopefully a Normal Auto will smooth things out enough to trigger Test 1c. Thanks, Paul

I think I see the beginnings of a curve there, so definitely worth running a Normal Auto. Just an FYI for all reading, here is an overview of the test logic being used: 1a) Test 1a range is 384-1152. If 384 is 98%+ of the fastest measured speed, then Test 1b is triggered. If 1152 is 98%+ of the fastest measure speed, then Test 1c is triggered. 1b) Test 1b is 64-320 1c) Test 1c is 1216-2304 If 2304 is 99.5%+ of the fastest measured speed, then Test 1d is triggered. 1d) Test 1d is 2368-3456, which is as high as Pass 1 goes. On my server, when running a Normal Auto, I get tests 1a, 1c and 1d. When I run a Short Auto, I typically only get 1a, as the results are too random to trigger 1c. I see this Short Auto test here on Tower7 was too random to trigger 1c (1152 was 92.3% of fastest), so it didn't test the original value of 2048 (except in the baseline). This seems odd: Current Values: md_num_stripes=4096, md_sync_window=2048, md_sync_thresh=2000 Global nr_requests=8 sdc nr_requests=128 sdd nr_requests=128 sde nr_requests=128 sdf nr_requests=128 sdg nr_requests=128 The global nr_requests setting (in your disks.ini file) is 8, but the individual drives are set to 128. Any chance you started and aborted a test, or had a test fail on you? This looks to me like a nr_requests test started and set the drives individually to 128, and that the script never finished to return the drives back to 8. Looks like CPU/RAM are reporting correctly now, and the new "Recommended" is showing up. Thanks! Paul

Amazing detective work. Yes I do have the Beta Preclear plugin installed, but haven't been running it. Wouldn't have expected that to modify anything. I also have Unassigned Devices plugin installed and running, and that was originally developed by the same person, though I think it has since changed ownership. Perhaps that is the actual culprit. Regardless, I do have that symlink. Found it there a couple days ago when I was still thinking about changing the mdcmd paths. I wrongly assumed [again!] that this was a Lime-Tech change. Shame on me for not testing in safe mode. Excellent, thank you. When v4b3 drops, it will include this change. Paul

I don't think anyone knows the answer...yet. Hope you can help find the answer. Thanks!

unRAID version? Your signature says 6.0. 6.2-RC4 And your signature still says 6.0 That's a graphic. Im not going to update it every version. You could accomplish most of that in bbcode, or if not, maybe next time you do update the graphic you could just remove the version completely and display it separately since versions change pretty frequently around here. I had had to comment, how times have changed! Remember when we went years without a version change...

UTTv4b Testers, there is an updated beta ready for download, v4 Beta 2. Same download location, but now has 4b2 in the name. Changes: Fixed the CPU/RAM output Fixed some cosmetic spacing issues Added a new "Recommended" Sync Window value It's all minor cosmetic stuff, no need to abort a currently running test just for this. Just use this version before you start your next test. The new Recommend is the lowest md_sync_window value that returns 99.5% of peak speed. Should help find that leading edge of the peak while reducing memory. Thanks to jumperalex for suggesting it. -Paul

Absolutely, you're spot on regarding diminishing returns. That's why in my script I'm checking for the "Thriftiest" value, which is the lowest md_sync_window that returns at least 95% of maximum tested speed. (yes, I know that's an arbitrary threshold, but gotta draw the line somewhere, right?) From the same test run that produced those charts, here's what my script reported: The Fastest Sync Speed tested was md_sync_window=3392 at 131.6 MB/s Tunable (md_num_stripes): 6784 Tunable (md_sync_window): 3392 Tunable (md_sync_thresh): 3391 Tunable (nr_requests): 8 This will consume 547 MB with md_num_stripes=6784, 2x md_sync_window. This is 157MB more than your current utilization of 390MB. NOTE: Adding additional drives will increase memory consumption. The Thriftiest Sync Speed tested was md_sync_window=832 at 126.2 MB/s Tunable (md_num_stripes): 1664 Tunable (md_sync_window): 832 Tunable (md_sync_thresh): 831 Tunable (nr_requests): 8 This will consume 134 MB with md_num_stripes=1664, 2x md_sync_window. The "Thriftiest" value (832) was significantly lower than the value you chose (1920), and saves a ton of memory. It might be that 95% is too low of a threshold for the Thriftiest value. The value you chose is 99.6% of the maximum speed observed. Or maybe I need to change my algorithm to report the leading edge of the peak, instead of the absolute peak. I could look for the lowest value that gives 99%+ of peak observed speed, which would have returned 1408, or 99.5% which would give 1856. I still like to test for and report absolute maximums, but your absolutely right about using a lot of memory for very little gain. -Paul

Wow, that's a surprising amount of consistency for an Auto Short test. The nr_requests test is very telling, I think your server likes 8 better than 128, but need a Normal length test to be sure. I think this script is going to do wonders for your server. And of course your output is also missing the CPU/RAM values. I guess that makes my server the oddball. Thanks!!!

Yikes. Well, that was unexpected. Not sure why CPU was missing, you had plenty of values to choose from. Perhaps my code couldn't make a decision... The memory line I was looking for is in there. Wonder why it didn't use it? Maybe a spacing issue... EDIT, yes it is a spacing issue. I was expecting 9 spaces between the word memory and the beginning of the actual value, there were only 5. Teach me a lesson for hard coding values. memory 32GiB System Memory

Cool, thanks. I see enough volatility in those results to say you should definitely run the Normal test. Interesting, I see that the CPU and RAM values are missing. This comes from lshw. When you get a chance, can you give me the output of: lshw -quiet -businfo -c processor -c memory Thanks, Paul

While the beta testers are busy testing beta, here's a couple charts from one of my test runs. I made these in Excel, using the new CSV output file as the source. This first chart is the overview of the entire test. The red and blue lines are both from Pass 1, in which the script tested md_sync_windows from 384 to 3456. The red line is md_sync_thresh being set to 50% of md_sync_window, while the blue line is being set to md_sync_window-1. The highest blue peak is at md_sync_window 2496, which becomes the central point for Pass 2. The Pass 2 is the green and purple lines, running from 2368 to 2624. With 4 lines occupying the same space, it gets pretty hard to tell what's going on, so here's a second chart where I zoomed in: If you look closely, you may be able to tell that in this range, the red/blue Pass 1 lines only have 5 overlapping test points, while the green/purple Pass 2 lines have 33. You can measure repeatability/accuracy by comparing blue to green, and red to purple. Pass 2 is retesting a range that shows promise with 16x more sample points, checking to see if a particular value in the neighborhood works better than the rest. The peaks/valleys at each test point probably aren't better/worse values, just natural run to run variances, which in this chart is +/- 0.2 MB/s for the same test at the same test point. For the most part, any value in this range is a good value (130.7 MB/s to 131.7 MB/s), and the impact from best to worst is probably a minute or two on a full parity check. I'm starting to think this second pass may be overkill. Won't know for sure until after others post their results.

UTTv4b Testers, check your messages, you should have a download link. I can't tell if the PM worked, when I go to view the 'Sent Messages', it's blank. Paul

UTTv4.0 Beta Testers: I have sent the PM's with the download link. Please read through the following before beginning beta testing. Let me know if you have questions. Apologies if you find any of this condescending, it's not meant to be. I know most of you are smarter than I am. I just like to be thorough. Obligatory Statements: If at anytime you feel uncomfortable with the tool, then stop it. Either cancel out or CTRL-C or stop the array, your choice. If you aborted during a parity check, that parity check will still be running, and you can cancel it through the GUI. Rebooting the server should restore all parameter settings. I can't imagine what may occur that would cause you to take such actions, but feel free to do so if you don't like what's going on. There is unused code in the script (read/write tests) that you could run if you really wanted to. I suggest you don't, and stay on the guided path by following displayed menu options. These hidden options were tests developed on unRAID v5.0, they were never publicly released or tested, and haven't been properly updated for 6.2. I didn't want to strip them out. Installation: Download the unraid-tunables-tester.sh from the link I sent you via PM. Put the file onto your flash drive (I put mine in a folder named "utt" off the root of the flash drive). Make sure the file is executable, if it isn't, you can "chmod 777 unraid-tunables-tester.sh" to make it so. Running It: On the console or telnet session (screen!), just navigate to the location (i.e. cd /boot/utt) and type the name to run it (unraid-tunables-tester.sh). Easy. Testing Strategy: Before testing, please know how long a full parity check takes to complete on your server in the current configuration and unRAID version. The primary purpose of this tool is to improve parity check speeds (if possible), so that little detail may be important later. I recommend that you use "screen" to run the tests, or use the console directly, so that a lost telnet session won't impact your ability to monitor the test progress. After installing, I recommend the first test you run on any (and every) server be the Short Auto test. It's too short to provide useful results (so don't save or apply them), but it will at least expose if your server responds to these parameters being changed (many servers do not). So instead of spending 16 hours to find out your server doesn't work with this tool, you can find out in 60-90 minutes. No need to publish any of these results (unless you really want to) - this is just a quick(er) test to figure out if you should run the Normal Auto test. If every test reports about the same speed, that's a big clue your server doesn't respond to these parameters. If the Short Auto test looks promising, go ahead and run the Normal Auto. It will take a minimum of 9 hours, and possibly up to 16 hours (it self adjusts to run more tests as necessary), and for best results you should do this while this system is not being accessed, so plan this out ahead of time. I would say start it up when you go to bed, and have results by lunchtime, but you may want to run this during daytime hours so you can monitor (I like to keep an eye on drive temps). I don't anticipate anyone will run the Long Auto or Manual tests at this time, but I won't stop you if you want to. Long Auto is same as Short/Normal Auto, just twice as long as Normal (so up to 32 hours!!!). Long Auto is primarily there in case someone has a server that doesn't seem to settle down and produce clean results with the Normal Auto test. Longer tests produce smother, more accurate results, something you'll see yourself when comparing the Short to the Normal test results. The script now records the current server settings and writes these to the report file before beginning any tests, and you can monitor the report file as it is being written. It will be in the same directory you placed the script, and will be time-stamped so as not be overwritten from test to test. Additionally, at the end of the test, if you don't apply the new Thrifty or Fastest setting, the script will automatically revert the parameters back to the original settings. Otherwise, if you abort the script, rebooting may be the easiest way to get your parameters back to the starting values. Something new I added was a staggered spin-up of array drives not already spun-up. This provides a smoother menu response (previously it looked like the script froze when unRAID auto spun up drives). To test this, you'll need to have some or all drives spun down. This logic was so easy, I'm now pretty upset Lime-Tech has never given us staggered spin-up (or have they? I really don't know). This is something I wanted and asked for, as I had to install a 650W power supply in my server just to handle concurrent spin-ups, and during a parity check my server only uses about 150W. But I digress... sorry. Desired Feedback: Please share all feedback directly in this thread. I lose track of PM's, so that's not recommended. If you experience any problems, please detail them as much as possible, provide the output file, and if you can, copy information from the screen (which is different from what goes into the report file) and share that too. I would really appreciate it if you read every word on every screen, and let me know if anything is not clear or needs corrections. I'm also interested in your critique of the test data presented both on the screen and in the output file. Feel free to be blunt in your critique, I can handle it (but that's not an excuse to be rude...). After your Normal test completes, please post the report file here. If you are testing more than one system (cough...johnnie.black...cough) please detail what system the test was performed on when you post the results. No need to post the output of the telnet session (unless there were problems), as the output file has everything I want to see. If the Thriftiest/Fastest test results look like something you want to try, go ahead and apply them (saving is optional) and run a full parity check. Post before/after times for the parity check. Optional Extra Credit: In addition to the report file, there is a separate CSV file with all of the tested speeds, perfect for charting the results. Use whatever tool you like (I've been using Excel), chart them up, and share the chart here. I find the charts to be very helpful. Last Notes: UTTv4 represent a pretty significant rewrite of the old v2.2 script. Not only is it changed to be compatible with unRAID v6.2, I've added a LOT of new functionality and greatly modified the testing routines. I won't post here everything that is new, I'll let you discover it for yourself. Thank you gentlemen, your help is greatly appreciated. Paul

I've selected the beta testers. Apologies to those who weren't selected, the list below is a little bigger than I wanted to go, but with so many good choices I had a hard time excluding. ljm42 (because you have a nice boring server with drives on the motherboard) StevenD (because you have 13 identical drives on a dual Xeon box with tons of memory, dual M1015's, and running ESXi) jack0w (because you have single parity and both a HBA and an extender) Squid (because you have AMD cpus, a SAS2LP, and a nice mix of drives) scottc (because you have AMD CPU's, a 9211-8i and a H310, and a nice mix of drives) johnnie.black (because of all your servers, you're like 7 beta testers in one!!!) I'm thinking the above will provide a nice cross section of hardware. I'll send PM's to each of you with download instructions later today. I'll provide testing guidelines in a post at that time. If you weren't included in the beta, please be patient. I believe we're very close to a public release. Thanks, Paul

Very close, may seed the beta today. Thought it was going to be yesterday, but I switched ISP's yesterday morning, and wasted time not only during the install, but also spent a few hours trouble-shooting it when it all went tits-up a few hours after the install guy left. Fun. -Paul

It took me a couple minutes to understand it, but I was able to figure out the turbo write script. I like it. PHP looks a lot closer to the languages I typically program in. Bash is aptly named, as it makes me want to bash my head in because it is so finicky and has such weird syntax. Thanks for sharing. Are there other languages I should be considering, or is the choice simply bash vs PHP? -Paul

To be honest, I would have to say I don't know PHP well enough to switch over immediately. It has been many years since I've played with it, and off the top of my head I can't recall even one PHP command. Not that I know bash all that well, but right now I'm stronger in bash than PHP. And the script is basically done, so there's that. I took one CS course in college (learned a bubble sort on Pascal), and beyond that I have zero programming education. Not that I let that stop me. I've learned a handful of languages and scripts over the years, and I'm pretty quick to pick up or relearn languages as needed. And I always appreciate the kind help of the community. If you have some sample PHP unRAID scripts I could review, that would be appreciated. I don't think I've ever even seen one. One look might be all it takes to make me a convert. Maybe the right path is to go PHP when we convert to a plug-in. A lot of the bash script is all about menus, and that all has to be redone in the plug-in. For now, the bash script is a great starting point for getting the testing routine down, and will give us an immediate tool to utilize until we can get a plug-in. It may not be pretty or elegant, but it works. -Paul

Totally agree, writing the above in PHP is basically a one-liner. Just wanted to point out that things are possible in bash. It's up to Pauven to decide how to move forward! Okay, I'm totally curious... what's the one line? I do a little amateur PHP coding on web pages, but it never even crossed my mind that I could do it in a unRAID script.

I think I got what I need. Not sure how you could do it in 3 lines (unless you mean I can call your script with just 3 lines in my script), as I think 21 is the best I could do if I really tried. Here's what I came up with, which is meeting my needs: count=-1 while read line do if [[ $line == "["*"]" ]]; then ((count++)) #Increase the counter if [ $count -gt 0 ]; then DiskNamePretty[$idx]=$name DiskType[$idx]=$type DiskName[$idx]=$device if [ $color == "green-on" ]; then DiskSpin[$idx]="UP" else DiskSpin[$idx]="down" fi fi else eval $line #convert the line into a var fi done < /var/local/emhttp/disks.ini #process the final disk from the file DiskNamePretty[$idx]=$name DiskType[$idx]=$type DiskName[$idx]=$device if [ $color == "green-on" ]; then DiskSpin[$idx]="UP" else DiskSpin[$idx]="down" fi Oh, look I see bonienl showed me up!!! Very nice. I see it was a similar solution, but you grabbed all values and threw then into an array, and silly me I used separate arrays to hold just the values I wanted. Eh, for now it is working. But I really do like what you did there. Thanks, Paul

Have you checked they are not on eBay, craigslist or something similar? Good call, I just checked and found a new ebay listing for a bunch of used drives: 2x Seagate 4TB, 9x Seagate 3TB, 4x Seagate 2TB, 3x Seagate 1TB, 2x WD 3TB, 1x WD 2TB, 1x Hitachi 2TB, and 1x Hitachi 1TB. Score!

PITA. That's the truth. I guess I was hoping for a miracle one-liner, like how: eval $(egrep 'md_num_stripes|md_sync_window|md_sync_thresh|nr_requests' /boot/config/disk.cfg | tr -d '\r') creates 4 vars with values assigned. Freaking awesome. I was also hopeful to find the data in the "mdcmd status" output, as at least that identifies the drive # on each line. But I don't see spin status or name there. If it is simply a matter of looping through, finding one value, then trusting the next value is the one I'm looking for, I can do that. PITA, but doable. Thanks, Paul