Pauven

Well, good job guys. The conversation has prompted me to find and review my testing documentation, which includes my strategy for the next test routine. And it just so happens that at this rare moment, I find myself with a bit of free time, and craving working on something different for a change of pace. So I think I'm going to try implementing the new testing strategy. I'll also take a look at Xaero's revision to see what best practices I need to apply.

Almost 36 hours is indeed too long. My mixture of 5400 RPM 3TB & 4TB drives, and 7200 RPM 8TB drives finishes in 18.5 hours. A pure 7200 RPM 8TB setup should complete in under 16.5 hours. Even slower 5400 RPM 8TB drives should finish in under 22 hours. I'm not sure that your math accounts for typical communication protocol overhead, nor the inefficiency inherent in port expanders. I'm happy you did find a 10MB/s improvement. This is exactly the reason I avoided port expanders. 60-70 MB/s sounds close to what I would expect. This is absolutely correct. Though I think many users struggle to understand this without a visual: This chart is from a HDD review, showing throughput speed (MB/s) over the course of the disk. The lime green line starts off high around 200MB/s at the beginning (outside edge of the disk platter), then tapers off to around 95 MB/s at the end of the disk (inside edge of the disk platter). This is just a random sample, and doesn't necessarily represent your drives, so this is just to show the concept of what is going on. The average speed of the drive (and the resulting Parity Check) would be around 155 MB/s, not the 200 MB/s peak. The Unraid Tunables Tester only tests the very beginning of the drive (i.e. the first 5-10%, where speeds are the highest). This is way above the average speed of an entire drive, beginning to end. On this chart I drew three dashed lines. The green line at the top represents Unraid Tunables set to a value that doesn't limit performance at all. This is what we are trying to achieve with the Unraid Tunables Tester. The yellow line in the middle represents how a typical system (one without any real performance issue) might perform with stock Unraid Tunables. Notice that while peak performance is reduced from 200 MB/s to perhaps around 190 MB/s, this slight reduction is only for the first 17% of the drive, beyond which the performance is no longer limited. A 5% speed reduction for 17% of the drive only reduces average throughput (for the entire drive) by less than 1%, so fixing this issue might only increase average throughput for the entire drive by 1-2 MB/s. Sure, it's an improvement, but a very small one. The red line at the bottom represents how some controllers have major performance issues when using the stock Unraid Tunables - like my controller. In this case, the throughput is so constrained, over 90% of the drive performs slower than it is capable of performing. Fixing the Tunables on my system unleashes huge performance gains. Hopefully that helps show why most systems see extremely little improvement from adjusting the Unraid Tunables - these systems are already performing so close to optimum that any speed increase will hardly make a dent in parity check times. It's only the systems that are misbehaving that truly benefit. Paul

While your comments are very interesting, they are really outside the scope of the tunables tester. Going back to the car analogy, the tunables tester is trying to identify the best gas and oil to use to get the manufacturer's rated horsepower. You bought a car with 300 horsepower, but for some reason you're only getting 150 hp, so we test all the variables to find out why you're not getting what you bought. Perhaps your car needs 93 octane and 5w40, but you've been running 87 octane and 10w30. Or maybe you need a tuneup with new spark plugs. Simple fixes to eliminate performance issues, and each car might have slightly different issues. What you're talking about is swapping out camshafts, porting and polishing, upgrading the fuel system, and adding NOS, trying to get the maximum amount of power from the engine and hoping it doesn't blow. The tunables tester should be limited to testing the user settable tuning values that Limetech provides access to in the GUI. Hence the name - Unraid Tunables Tester. Everything else that you're talking about sounds like a discussion you should be having with Limetech, or developing brand new tuning tools way beyond the scope of the tunables tester. Years ago, when I upgraded from v4.x to v5.x, my parity check times immediately increased from 6.5 hours to 12+ hours. I was fine with 6.5 hours, but knew there was a problem with the nearly doubled parity check times. Thus the tunables tester was born, and I was able to get back to my 6.5 hour parity checks. My system does everything I ask of it, and while it's certainly not the fastest server in the world, performance is fine and I can watch movies while a parity check is running with no stuttering. I can simultaneously run 4 Windows 2016 server VM's too. Not sure what else I could ask for. I'm the type of person that is not interested in modifying hidden tuning parameters trying to eek out a bit of extra performance beyond what stock Unraid offers, and I would also be slow to upgrade to a new Unraid version that has made major changes to the I/O scheduler. My 67 TB of data, and my time not spent doing data recovery, is much more important to me than a bit more performance that I probably won't even notice. I'm still running 6.6.6 because I've seen things in 6.6.7 and 6.7.x that have me waiting for a more robust update. Don't get me wrong, I think your ideas are great and I hope you pursue them. If you can get Limetech to implement these enhancements in core Unraid, that would benefit us all, and after a few months of letting everyone else beta test it for me, I would happily upgrade for the free performance boost. But elevating Unraid to a higher level of performance is not what the tunables tester is about - it's about fixing problematic settings that are dramatically hurting performance on certain machines.

While the testing ideology isn't broken, due to the v6.x changes in tunables, the script no longer provides a complete picture. Going from memory, nr_requests is a new tunable that affects performance, and I had planned to include it in my v6 compatible beta, and in testing we saw some interesting results around 4 on some machines. And while md_write_limit is gone in v6, we have a new md_sync_thresh that needs to be tested. So instead of 3 tunables, with v6 there's really at least 4 - which only further complicates testing. You can find them on the Disk Settings. Note, some of them wouldn't be part of this type of testing, like md_write_method is beyond the scope of what this tunables tester is doing (and is irrelevant for parity checks anyway). I vaguely recall doing some type of testing with NCQ - but I don't think I ever automated that. I think that was more of a manual effort, test with it both on and off, and see what works for your machine. poll_attributes I think is a newer tunable added in a fairly recent version - I don't think it existed back when I was working on my v6.x beta tunables tester. I don't even know what this does. Believe it or not, the v5 script does a lot of that. At runtime it provides multiple test types to choose from, and the quicker tests do exactly as you describe. All of the tests try to skip around, hunting for a value region that performs better, then focusing in on that region and testing more values. Older versions of the tunables tester made big jumps and tested much faster, but as I refined the tool I added more detailed testing that didn't skip around as much, because I've also found, from examining dozens upon dozens of user submitted results from different Unraid builds, that it is a mistake to make any major assumptions about what values work best. There are some machines that get faster with smaller values (below 512). Performance is not always best at powers of two intervals - for some machines yes, but not all. These tunables seems to be controlling how Unraid communicates with the drive controller. And there are dozens of different drive controllers, each with unique behaviors and tuning requirements. Add in that many users have "Frankenstein" builds using different combinations of controller cards (some from the CPU/northbridge, some from the chipset/southbridge, and the rest from sometimes mismatched controller cards), and what you end up with is an entirely unpredictable set of tuning parameters to make that type of machine perform well. While I don't disagree with your sentiment, making a tunables tester that works equally well on every type of build in the real world doesn't align very well with expedited testing that skips around too much - what works great on one machine doesn't work at all on another. It was very frustrating trying to identify a testing pattern that worked well on any and all machines. To me, the big picture is that it's better to spend 8 hours identifying a set of parameters that provide a real-world benefit, rather than wasting 1 hour to come up with parameters that aren't really that great. It's not like you run this thing every day - for most users it is a run once and then never again. Trying to save time for something you run once, at the cost of accuracy, isn't ideal. Paul

To add to jonathanm's answer, the script starts a series of partial, read-only, non-correcting parity checks, each with slightly tweaked parameters, and logs the performance of each combination of settings. Essentially, it is measuring the peak performance of reading from all discs simultaneously, and showing how that can be tweaked to improve peak performance. Improving peak performance is not the same thing as improving the time of a full parity check, as your parity check only spends a few minutes at the beginning of the drives where peak performance has an impact, as performance gradually tapers off from the beginning of your drive to the end. If your peak performance was abnormally slow (i.e. 50 MB/s), then that would affect a much larger percentage of the parity check, and improving that to 150MB/s would make a huge improvement in parity check times, but increasing from 164 MB/s to 173 MB/s won't make much of a difference, since essentially you were already close to max performance and that small increase will only affect perhaps the first few % of the drive space. In a similar way, I could improve aerodynamics on my car to increase top speed from 164 MPH to 173 MPH, but that won't necessarily help my work commute where I'm limited to speeds below 65 MPH. But if for some reason my car couldn't go faster than 50 MPH, any increase at all would help my commute time. There are a handful of drive controllers (like the one in my sig) that suffer extremely slow parity check speeds with stock Unraid settings, so I see a huge performance increase from tweaking the tunables. There is also some evidence that tweaking these tunables can help with multi-tasking (i.e. streaming a movie without stuttering during a parity check), and for some users this seems to be true. I know there are some users who have concerns that maximizing parity check speed takes away bandwidth for streaming, though I don't think we ever actually saw evidence of this. That's a shame, as that is really what is needed to make this script compatible with 6.x. LT changed the tunables from 5.x to 6.x, and the original script needs updating to work properly with the 6.x tunables. Fixing a few obsolete code segments to make it run without errors on 6.x doesn't mean you will get usable results on 6.x. I had created a beta version for Unraid 6.x a while back, but testing showed it was not producing usable results. I documented a new tunables testing strategy based on those results, but never did get around to implementing them. It seems that finding good settings on 6.x is harder than it was for 5.x - possibly because 6.x just runs better and there's less issues to be resolved. I still have my documented changes for the next version around here somewhere... That's another shame. Seems like you know what you're doing, more so than I do with regards to Linux. I'm a Windows developer, and my limited experience with Linux and Bash (that's what this script is, right?) is this script. For me to pick it up again, I have to essentially re-learn everything. I keep thinking a much stronger developer than I will pick this up someday. I'm not trying to convince users not to use this tool, and I certainly appreciate someone trying to keep it alive, but I did want to clarify that the logic needs improvement for Unraid 6.x, and you may not get accurate recommendations with this Unraid 5.x tunables tester. Paul

Simply awesome, thanks for doing this!

I can't see the gallery you created, or any galleries at all. Is this section of the forum invite only?

That's a great idea. That way it's either half stretched or half squished, so it will look more normal. I went ahead and created a 1080p version, and even stretched to 4K it looks okay, so for this one I'll just stick with 1080p. Since we seemed to be missing some AMD love in here:

I don't GUI boot, and rarely view the GUI on a mobile device (rare enough that I don't care how it looks there). 90% of the time, I'm viewing in Chrome or Firefox on a 4K monitor. Sometimes I'm viewing full screen, and sometimes I pin the browser to one half of the screen. So effectively I'm viewing at both 3840 and 1920 on the same monitor. Occasionally I view on my 1080p TV. To test, I just created a 4K banner (3840 x 90) and it looks perfect at fullscreen, but when I shrink the browser to half the screen width the banner gets smooshed. I also noticed that due to the smooshing/stretching, the server info at the right and Unraid logo at left cover different portions of the banner depending upon how smooshed/stretched the banner is. The banner I created is very dark, and needs a lighter box at both ends to make the text overlays readable, but the size of the necessary box changes depending upon the smoosh/stretch factor. Would be nice to be able to adjust the transparency of the server info background box instead - is that possible? It is too transparent. I'm still on 6.6.6 - not sure if 6.7 changes any of this behavior. Paul

I've got a mixture of 1080p and 4K monitors. Is there a solution to one banner that displays correctly on 4K and 1080p?

I gotcha beat. I live in a gated community, and you have to dial 666 at the gate to ring us to let you in. Needless to say, we don't get many visitors. 😈

Hey gang, sorry I've been absent. Luckily I've been healthy, so no concerns there. Instead, I've been sidetracked by by work and other projects. My biggest sidetrack has been a new program I wrote to replace the old My Movies Media Center Plugin. That may be of some interest to unRAID users like me who store their large movie collections on their server: If you're interested, you can check it out here: MM Browser MM Browser was supposed to be a quick little 2-4 week programming project, just for myself, but then I went crazy and decided to sell it online, which required a ton more programming and a website. User support has been much more time consuming than I ever fathomed. I've easily spent 6 months full time on MM Browser. MM Browser pulled me away from my other project, the Chameleon Pinball Engine. I had planned to have it ready for the next big Southern Fried Gameroom Expo here in Atlanta. Somehow the time slipped away. The show is in 4 weeks, and I'm realizing that there's too much work to make up to make to the show. That's a big disappointment for me. I've also got a a small enterprise software suite that I've worked on for the past decade, and I'm currently working on my first big sale. Trying to sell enterprise software to, uhm, big enterprises, has been eye opening to say the least. So many hurdles, and I'm spending more time doing documentation than anything else. Right now this is my biggest priority. Plus I've got a full time consulting gig at the moment. Long story short, I just haven't a moment to spare. I would release the private beta, but to be honest it just didn't work well, so that version was scrapped. I have documented plans for a new version that hopefully would fix the problems of the private beta. Every so often I think about trying to knock it out, and I've come close to working on it a few times, but it just fell too low on my priority list. There's always a chance I may get to it soon, but I can't make any promises. I know that this isn't the answer anyone was looking for. Sorry. If anyone else wants to run with it, please feel free. You have my blessing. Paul

For you, definitely.

Thanks guys! I think the camera running out of storage is exactly what happened! Cody (of 908 Pinball Zine) was filming and was very disappointed. He had caught me at the very end of the show, after I had pretty much packed up. I showed him under the hood, but he never actually got to see it turned on... and I couldn't be motivated to do so since his storage was full. We'll go for round 2 in a few weeks. If you want to see under the hood of Modern Firepower, here's a good post: http://pinballchameleon.blogspot.com/2013/03/the-modern-firepower-pinball-project.html That was before the playfield was installed, but otherwise it runs pretty much complete (just needs 3 cables to playfield: USB, 5V/12V power and 50V power). If you compare the wiring of my playfield with a typical pinball playfield, this is a HUGE improvement. Black Knight Rises will be even better! While I haven't gone to market, I have developed my own kit, and it is in Alpha Testing (Black Knight is the Alpha test). I have a new IO Controller for the LED's and Switches, both superior and cheaper to anything else I could find on the market (except for what maybe Stern uses internally). I have a new version of my Power Driver, shrunken to the size of a business card yet more capable (and Arduino/Pi compatible to boot). I've developed my own pinball LED lighting solution, though I think I need another revision to make it smaller. I'm really proud of my PCB designs. I went from through-hole 1980's style circuit boards to surface mount SMT designs, ready for mass production - but a PITA to hand assemble, especially with resistors the size of fleas!!! I can't wait to show it all off, only a handful of people have seen anything I've done. And the crown jewel is my software. I've been making it better so much better. There's a guy in the Atlanta area that makes custom pinball machines, and I have him lined up as a Beta Tester. Once that's done, then I'm ready to try and market it. Oh, and the biggest problem I had with Modern Firepower is that it looked too good! Most people walked by and didn't realize how unique and special it was in a sea of pinball machines. This time, I'm making my cabinet look very different, hopefully to draw more attention to Black Knight Rises. -Paul

That has definitely been a factor, yes, but just one of many. I have started development of the next UTT release, but have not got to the testing stage yet. I also have a solid road map for coding my changes. For the moment, I've had to halt progress to work on another project. What could possibly be more important than a new unRAID Tunables Tester? Pinball, of course! Okay, okay, I get it, I have messed up priorities. I'll try to explain. For those of you in the Southeast (USA), the 4th annual "Southern Fried Gameroom Expo" (SFGE) is June 9-11 in Atlanta. Tons of pinball and arcade machines. You should go if you can. A few years ago I built my own pinball machine from parts, wrote my own software and used off the shelf, non-pinball circuitry combined with a few circuit boards of my own design. It came out great. I took it to SFGE 3 years ago, and it did really well. After that show, I designed my own control circuit boards to replace the off the shelf parts I had used. My stuff is both more capable and cheaper. I also had picked up a Williams Black Knight playfield that had been basically trashed, with the intention of building a machine around it and adding my own circuit boards and software to it. Nothing like waiting until the last minute, but I've finally started the build. With the show 3 weeks from today, I'm completely focused on building this pinball machine. My goal is to take it to SFGE. Hopefully I make it, but it is a massive project. The good news is that if I don't make the show, I've got another year to get it finished... If you interested in reading more, you can check out my blog. Currently all my posts are from my previous build, Modern Firepower, 3 years ago, but I have a lot of new posts I'm working on detailing the new build: The Black Knight Rises. http://pinballchameleon.blogspot.com/ And right after the show, I'm going on vacation. TL;DR: unRAID Tunables Tester development is on hold until the end of June. Sorry. -Paul

Hey gang, Just wanted to drop a quick update. For those of you that have been reading the "Anybody planning a Ryzen build" message thread, you've realized that I've purchased a Ryzen system to upgrade my server. You would also know that there is a problem with Ryzen + unRAID, a combination that causes random crashes. For the past 2+ weeks, I've been troubleshooting these issues and trying to find a solution. This has consumed all my free time. In addition, my unRAID server is offline due to my testing. Long story short, I got sidetracked and haven't been able to do anything with the Tunables Tester script. I plan to wrap up my Ryzen testing this week, regardless of the outcome, and worst case I will get the old motherboard back in place to get my server back up. Then I will resume development of this script. Sorry for the delay after the big tease. -Paul

I still see the pictures, they are in this post: https://forums.lime-technology.com/topic/25692-unmenu-16-now-available-a-major-upgrade-in-appearance/?do=findComment&comment=245390 Can you see them there?

If theone is taking feature requests, I still have the one I listed above, to color code the drives according to temperature to create a heat map: https://forums.lime-technology.com/topic/38543-server-layout-plugin-for-unraid-v6/?do=findComment&comment=485756 Otherwise, I love this plugin as is, thanks!

Thanks for all the happy greetings and well wishes, much appreciated! Squid, I especially liked the Welcome Back theme song! Careful, though, it has me wanting to binge watch the whole series, and I'm easily distracted... Recovery is a work in progress, but so is life. I'm certainly feeling good enough now to try and tackle this challenge. I really would like to get it knocked out pretty quick, as I've got a long list backlogged projects I need to get restarted. On topic, my parity check times have stayed constant through 6.2 to 6.3.2, all within a few seconds of each other. I think I read some of you have experienced slowdowns related to different releases. Is anyone aware of Lime-Tech making any tuneables related changes over the past 6-months? Thanks again! -Paul

Hey BRiT and all the rest of the gang. Apologies for my absence. I had some health issues crop up unexpected back in September, and it took me out of commission for a while. Plus I've been pretty busy in the new year. I would like to resume work on the v4 script. I recall I had ideas back in September, getting ready to completely revamp the testing method. What I had coded just wasn't working right. Of course, half a year later, I don't recall what my ideas were. But fortunately, I have a habit of sharing my ideas in writing, so I think I can get back to where I was in September by re-reading my old posts. Hopefully my absence brought back fond memories of Tom disappearing for long periods of time between unRAID releases... Probably best I don't make any promises of dates just yet. It'll take me a while to get my bearings and remember how to code again. Once I get a good enough v4, I think it would be a great idea to get it into a git repository so the community can maintain it. None of ever know when life attacks. Even better, hopefully it can become a true plugin. -Paul

Any thoughts on the time frame? Probably next week.

Looks like it shaved a couple hours off. Nice. I'm curious why your elapsed time is in seconds and not hours/minutes/seconds. I'm still working on new testing logic.

Last night I ran a parity check using the same settings from last time, but instead of sync_thresh=sync_window-1, I used sync_window-64, which my recent testing showed was a few MB/s faster. Before: 2016-08-21, 17:16:08 7 hr, 40 min, 37 sec 108.6 MB/s OK After: 2016-09-01, 09:35:19 7 hr, 35 min, 18 sec 109.8 MB/s OK This change resulted in another 5 minutes saved. The server is now within 15 minutes of the fastest parity check I ever accomplished on 5.0, back when I had single parity. I don't know if the dual parity is adding 15 minutes, or if there is still performance left to be discovered. Not willing to go back to single parity to find out. I'm working on a new version of the tunables tester with the recently discussed logic changes. -Paul