garycase

Thought Limetech was a Limey eh

Actually it's effectively a RAID-1, not a RAID-0 ... and while the impact of that is a mirror (since UnRAID uses even parity), it's NOT built that way .. i.e. UnRAID doesn't simply "make parity = datadrive". It still does the parity calculations ... so writes aren't any faster than they would be with more drives. But parity checks are very fast because there are only 2 drives involved, so the calculation is very quick.

Agree. It certainly makes sense to tune your write parameters for the highest possible speeds if you're doing writes internal to the array. I'm not sure that isn't the same thing I was doing, however ... since the network can easily exceed the speed that writes maxed out at -- indeed reading the same file back to my PC transfers at over double the speed it was written at (well over 100MB/s). If I had a spare SATA port (I don't), I'd pop an SSD in as a cache for testing and see if it writes to the array any faster. My expectation is that it would not.

data warehousing perhaps ?? A Gb network with UnRAID is NOT "... limiting writes to your network speed ..." (except possibly to the cache drive if it's an SSD or a fast high areal density unit) That would certainly be true with a hardware RAID-5 or -6 controller with a high drive count where the bandwidth can easily get well over 1GB (not Gb) per second ... indeed they can saturate even a 10Gb network. But UnRAID's writes -- unless your array is composed of all SSDs -- aren't going to come even close to Gb network speed.

While it does seem that the most common (but not the only) definition of thrashing on the 'net is virtual memory thrashing, that word has been used since I was a young CS student 50 years ago to mean exactly what you described above ... my CS Profs used to compare it to a needle on an LP jumping back & forth between songs so much that you couldn't tell what either one of the songs was. They called it thrashing the disk, and it's stuck with me ever since, and still seems like a good use of the word. Once virtual memory systems became common in the 70's, page file thrashing became a significant problem ... and that's what the current Wikipedia definition refers to. [but note that disk thrashing was discussed in CS BEFORE there were ever virtual memory systems.] This was also often called "system thrashing", to differentiate it from the same problem that was found with database systems with "index thrashing", "buffer cache thrashing", and other problems that would significantly slow down database accesses. They ALL refer to essentially the same thing -- a LOT of disk head movement. My mental picture has always been exactly that phonograph arm jumping around on the platter so much that everything simply runs slower. Regardless of what you call it, when you have a lot of activities causing significant head movement, the time for those activities will be longer than if they were done sequentially, simply because seeks are the slowest part of a disk access. By the way, your teenager is indeed thrashing according to Webster ["... or stir about violently : toss about ..."] Enough on this ... I understand the modern usage that tends to equate "thrashing" to "system level virtual memory thrashing" => but it's still a word that means exactly what you described to me

r.e. write testing from the cache drive (or an internal RAM drive) ==> agree this would provide the best test of the maximum possible write speed of the array with zero impact from the network. However ... in real life writes are done over the network And if the writes are at a speed significantly below what the network can sustain (which is certainly the case with Gb networks), then there's no benefit to using md_write_limit settings higher than whatever maximizes your writes across the network. There IS one other factor I didn't consider and/or test (because I can't) => writes TO the cache drive ... these should be very fast as they are (a) done at drive speed without any parity computations; and (b) don't have any impact parity checks, since the cache is "outside" of any parity checks. Not sure how much impact md_write_limit might have on this, but it might be interesting to test it.

I don't have a cache drive.

Very unlikely, since it's a Gb network. I can do writes over 100MB/s between my Windows machines ... so the ~50MB/s I'm getting on UnRAID isn't even close to what the desktop and network can provide. Just to be sure, I just wrote the same test file I've been using to my wife's machine across the network, and it completed in 28 seconds.

Definitely agree. As I noted earlier (almost as an aside), if parity checks, rebuilds, and parity syncs were designed to automatically pause during reads or writes, the system would have maximum performance for the user ... and if reads were prioritized over writes, then there'd be virtually no "stalls" during media streaming, etc. This set of priorities wouldn't necessarily satisfy everyone ... but IMHO it would work very well. Providing users the ability to set these priorities would be even nicer. And of course this would also virtually eliminate "unnecessary head movement" [whatever you want to call it ].

A semantic question. Whenever you have multiple disk accesses ongoing that each require a seek, the heads are doing a lot of movement. The result is a slowdown ... for example, the 30 minutes extra a test takes if I don't wait 5 minutes for cache_dirs to fill its cache after a reboot. I'd call that thrashing -- you don't, since there IS some work being done. You're getting 60MB/s reads ... compared to ~ double that you'd likely get without the parity check. You indicated writes slow to 10-20MB/s ... 1/4th to 1/2 what they typically should be. Those slowdowns are clearly due to excessive seeks ... whether you call them "thrashing" or not is semantics.

This is done in Windows as well whenever you enable AHCI. In fact, it's probably the most important reason to use AHCI. [Hot swap capability is another, but I'm "old school" on that and still don't hot swap drives] In any event, the results I noted include the use of NCQ.

Yes, that's exactly what I was saying ... I don't think it's related to the optimal md_sync_window. When I was testing the write values I wasn't very granular at all -- I just roughly doubled the default; then added 1000 to it ... so I only tested 1500, then 2500. Then I doubled it again to see if there was any additional benefit. So ... I just tried 2048, and it also results in 57 seconds => so on my system I get very good results (1:01) with 1500; could shave a bit off at 2048; and get zero gain with numbers larger than that (as I noted before, I tested 5000 as well). The last Fullauto showed a balls-to-the-wall value [not called that, of course ] of 2688. Just for grins, I did 5 minutes of a parity check with that value; then did 5 minutes of a parity check with double that (5376) ... and get about 5% improvement with the higher value !! So for parity checks, I get continued improvement with higher values. I'm confident that if I set my md_write_limit to ~ 2048 it would be maxed out, with no further improvement ... and at what I've elected to use (1536) it's at the "plenty good enough" point. But I do NOT think the optimal values for these two parameters are the same. Obviously md_max_read_stripes = md_num_stripes ... what did I say that sounded different?? ... actually, looking at my earlier note, I noted you could also use the write stripes, but didn't note that you could also use the sync stripes (since it was in the context of a parity check -- which would be using those => I assume that's what you're referring to. But yes, I'm well aware of how these parameters interact Definitely don't agree that reads optimize at the same number of stripes as writes. Reads are FAR faster, since there's no parity involved, so the dynamics are a good bit different. The stripes can empty much quicker (assuming a Gb network). I haven't done any testing on what different numbers do here, as I can generally max out my Gb network with reads anyway. I suspect the optimal number for reads is likely SMALLER than for writes, since the network can likely empty them at almost the same rate they can fill ... but that wouldn't be true on outer cylinders with high areal density disks (like our 1TB/platter Reds), so perhaps the number is actually larger. In any event, as long as both reads and writes aren't done simultaneously during a parity check, I suspect the stripes allocated for writes, together with some minimal number of "extra" stripes allocated for "md_min_read_stripes" is plenty. I agree with your comments about not needing to have too much headroom vis-à-vis a "md_min_read_stripes" ... but do think it should be a reasonable number to account for those who might want to do both read and writes during parity syncs [not something I do]. Assuming a Gb network, 250 or 500 is plenty ... and if you assume that simultaneous reads and writes are unlikely, then even small values would be okay (since the writes stripes are available in that case). Clearly you do NOT want it to be zero (or, for that matter, I'd say no lower than 100), as that would starve reads if writes were underway.

I agree that with AHCI access is reasonably well optimized; but nevertheless if you have reads and writes queued during a parity check, you will do a good bit of thrashing, as each time slice given to the reads or writes will result in a disk seek on the impacted disk(s), which will effectively stop the parity check while those are satisfied; then the disk will seek back to where it needs to be for the next parity check read ... and this process will continue back-and-forth until there are no pending reads/writes. The only way to avoid the thrashing would be to stop the parity check for the duration of any pending read/write operations [which, by the way, would completely eliminate any impact of a parity check on system performance ... so from one perspective that wouldn't be a bad idea at all !! ] By the way, I've done quite a bit of testing on desktops to see the impact of various types of simultaneous reads/writes with/without AHCI, and regardless of the setting, one thing is ALWAYS true: The total time for a series of reads that impact the same disk is always lower if they're done sequentially than if you start them all at the same time. It's closer with AHCI enabled; but it's still better to do them sequentially. While my testing was with Windows, I'm sure the Linux results would be the same ... there's simply no way to move the disk heads any differently That was a "tongue in cheek" comment -- I have no plans to do anything else to change my parity times Agree that stopping services that may access the disk arbitrarily would eliminate a potential for skewing the results. Booting to UnRAID Safe Mode for performing this test is probably a good idea, although there are still a few things that doesn't stop. Mentioning cache_dirs reminds me of a result I found some time ago: I tested the impact of cache_dirs on a parity check, and found that it was relatively minor (less than 10 minutes) on my system (with ~ 300,000 files) UNLESS I started a parity check immediately after a reboot ... in that case it adds over 30 minutes. If I wait 5 minutes after a reboot and do the check, it finishes much quicker. The reason is fairly obvious: cache_dirs will, after a reboot, read all of the directories to fill its cache. If you wait for this to happen (~ 5 minutes), then it's relatively dormant from then on. But if you start a parity check before this is done, it then causes a LOT of disk thrashing, which slows down BOTH the caching reads and the parity check reads as the head moves back-and-forth a LOT to satisfy both requirements.

Interesting ... are you SURE about that? I'd think the logic would be very close, since both a parity sync (Not check) and a rebuild does almost exactly the same thing as a parity check with the exception of one disk (the disk being rebuilt). There is, of course, a timing synchronization difference -- the write is always "one behind" in terms of sector # (since it can't be done until the reads are done and the appropriate bit values determined). Even if it's not directly impacted by the md_sync_window (i.e. just does "reads" instead of reading to the allocated md_sync_window stripes), it would still be indirectly related, since all of the md_sync_window stripes will be available for reads.

I suspect in your case it has something to do with the fact you added the drive "hot" ... nothing wrong with doing that on a hot-swappable setup, but most folks power off to add drives, so the pre-clear is after a reboot. And in that case, the behavior with v5 is you have to start the array; then access the web GUI; and then it works perfectly. ... at least on every system I've seen (perhaps a dozen) and those that have reported the behavior on the forum.

Paul may have some ideas on what to modify ... but as I noted, I doubt it's different than the optimal value for parity checks, since the only difference functionally is a write is substituted for one of the reads. The other N-1 disks are doing the same reads they'd do for a parity check.

I'm not sure what might work best for parity syncs and/or rebuilds. Note that in both of these cases, it's really much closer to a parity check in terms of disk activity than anything else ... the difference is that a parity check is doing N reads (in an N-disk array), whereas a parity sync or disk rebuild is doing (N-1) reads and one write. My intuition is that if you have an optimized md_sync_window it's going to be very close to the best you can do for either of the other activities as well. Especially if you've already optimized your md_write_limit as I noted above.

Very interesting results. I set my tunables back to what I had originally decided were the best values (after 2 weeks of parity checks a few months ago ... probably ran 25 full checks with various settings to find them) ==> and it's back to 8:05. Guess I'll now have to try a few more combinations to see if I can at least get below 8 hours I did remember one factor that added about 8-10 minutes (and I'm just going to leave it that way) ... I've got cache_dirs running without limits (i.e. nothing's excluded). I've read the details of what these tunables mean (in Tom's post here: http://lime-technology.com/forum/index.php?topic=4625.msg42091#msg42091 ... and I've concluded the following ... => The md_write_limit parameter does NOT need to be modified at all. Once it's set to a value that provides a reasonable write buffer, there's no reason to make it larger, as long as the total number of stripes is enough larger than the sync_buffer setting that the specified number of strips for writing will always be available. I've found that 1500 is a good value for my array, although your mileage may vary. I'll outline how I determined that number at the end of this post. => There's an "implied" setting for minimum read stripes -- the total stripes minus md_write_limit minus md_sync_window. If you make this implied setting large enough ... I've found 1000 is very good ... then read performance will also be fine during parity checks -- and making it larger won't really help read performance. Note also that if no writes are being done, read operations can also use all the stripes that would be allocated for writes (an extra 1500 stripes if wd_write_limit is 1500). BOTH read and write performance can be impacted by simple resource availability (e.g. disk thrashing) if the md_sync_window setting is so "perfect" that the disks are fully engaged at all times. But changing the number of stripes available for reads or writes won't impact that. I suppose "de-tuning" the sync performance may, but I can't really think of a good reason to do that !! So ... what I'd love to have is a version of this utility that lets the user fix the md_write_limit value; fix the minimum number of stripes available for reading; and then simply sets md_num_stripes = the sum of those two numbers plus the current md_sync_window value as it's doing its tests. You can independently "tune" the number of write stripes by simply testing writes with various settings ... but since this is a maximum number you don't need to change anything else (as long as md_write_limit < md_num_stripes). But once you've decided on a value, there's no reason to change it as you tune other parameters (notably md_sync_window). I don't think this is a reasonable thing to do with a script, since you need to do the copies from your PC ... but it's very easy to do manually. Note that writes to the array are influenced by quite a few things; so if you're going to "tune" this parameter, you should try to be as consistent on all the other things as possible (i.e. are you writing to inner cylinders or outer cylinders? ... is there any network contention? ... etc.). I'd create a folder on your PC's hard drive to hold a few large files (3GB is enough to write -- I used a single 3GB file); then write it to a specific disk on UnRAID (NOT a user share, which could alter the location of the writes); then delete what you just wrote; alter the parameter; and repeat the process. FWIW I found that 1500 stripes worked well for writes. A 3GB file took about 1:21 with the default 768 stripes; took 1:01 with 1500; and dropped to 0:57 at 2500 stripes, with no improvement with larger values (I tried up to 5000). I decided 1500 was a good choice. I also tried this with a bunch of different values of md_sync_window, and it makes NO difference what it's set at (as I suspected would be the case, since the md_write_limit sets the maximum number of stripes used for writes). With md_write_limit set to 1500, and md_num_stripes set to md_sync_windows + 2500, there are up to 2500 stripes available for reads during a sync, and 1500 available for writes. I see no reason to ever use values higher than those. Note that with these settings, reading the same 3GB file I used for write testing takes 27 seconds (which may be limited by the write speed on my PC's drive). Now all I need to do is find a value of md_sync_window that cuts 5 more minutes off my parity checks !!

Just saw your edit => I assume that means the motherboard has multiple x16 slots, and you used those for the cards. I'm sure you wouldn't have plugged them in to an x1 slot if there was an x4, x8, or x16 available

Man! I was hoping I wouldn't have to open up the tower since I decided to upgrade the SSD (external drive bays).... *snerk* I'm pretty sure they are, I wouldn't short-change a card in a slot. -John What motherboard do you have? I'm sure if the board has enough > x1 slots you used them; but many boards only have one x16 slot and the rest are x1 ... and those cards have a slot that will allow them to be used in an x1 slot (with, of course, a significant reduction in their performance, since they'd only have 1 PCIe lane to use).

Are they plugged in to PCIe x4 (or greater) slots?

It's definitely frustrating. On an earlier release (I think it was ~ RC13 or 14) -- back when you and I were both running a lot of tests to tune things (before your very handy script), I had the time down to just under 7:30. Then it jumped to ~ 7:40 with the release after that; and jumped a good bit when the new kernel was introduced in RC15. I was also playing a little bit with the tunables, so I may have changed some relationship that I didn't realize. With the B4B setting after running your script (v2) I got back under 8 hrs, which I was quite glad to see [i suspect my 7:57 vs. your 7:40 is probably due to my trusty little Atom not having as much horsepower as your CPU] ... but then it jumped back up with v5.0 !! I've seen other reports of increased times with v5.0 as well, so SOMETHING is going on, although I'm not sure just what. I'm fine with ~ 8 hr checks ... it's just frustrating to know that 7:30 is POSSIBLE, and I can't get it !! ... it would definitely be interesting to see what your time is with v5.0

Joe -- the array not only has to be started, but you have to access the Web GUI one time before the pre-clear script excludes the array disks. I have absolutely no idea why that's the case, but if you simply Start the array (i.e. set it to auto-start), pre-clear will still list all of the disks until the Web GUI has been looked at one time. But if you simply look at the Web GUI one time, pre-clear will then list the disks correctly -- i.e. it will only list those that are not part of the array.

I was curious r.e. whether there'd be any change in the recommendations with the v2.2 script and v5.0 vs. my previous run with the v2.0 script and RC16c .. and WOW were there ever. I'm not sure what the changes in the various RCs have been that impacted parity (I think the newer kernel had the most impact ~ RC15) ... but my system used to do parity checks in ~ 7:40 and is now up to 8:15 !! I was at about 8:10 with RC16c (I did a LOT of tunables testing about 3 months or so ago, so had fairly good settings before running this script). Ran the Tunables-Tester script, and adjusted my parameters to the B4B values, and was pleased to see my time drop back down to just under 8 hours (7:57). Then upgraded to v5.0 ... and the time jumped up to 8:17 !! Reran Tunables-Tester and the suggestions changed dramatically !! Just for grins, I tried the very high UnT values .. and it still took 8:12 FWIW these were the Tunables-Tester results: With v2 running on RC16c: B4B: 3072/1408/1408 UnT: 4408/1984/1984 With v2.2 running on v5.0: B4B: 1408/768/512 UnT: 5968/2688/2688 What I HAD been using (after all my prior testing): 2560/768/1024 I've just changed back to the settings I had originally used and am going to kick off yet-another parity test. Details in 8 hrs or so Paul -- since both of our sysems have all 3TB WD Reds, I'm curious what kind of parity check times you're getting these days. (with v5.0)

Do I read that correctly that you have a 32GB SSD as one of the array drives?