UhClem

I just did a little reading on the RES2SV240 [http://www.intel.com/support/motherboards/server/res2sv240/sb/CS-033969.htm], and connecting it to a (single) controller with 2 cables will potentially allow for increased bandwidth IF the controller supports it. Intel says their controllers do; I don't know about the M1015, but it probably does. Of course, actually realizing that increased bandwidth is totally dependent on the controller having the oomph! to exploit it. I suspect that the M1015 is maxed out on a single cable [4x6Gbps] (but I hope I'm wrong!). Maybe you could test the two-cable theory now, with just the existing 12 drives. There is no need to do a full parity check--just watch the speed report for the first five minutes, and note what it "averages" (it should be pretty steady) [then terminate it]. Then connect the 2nd cable and test again. I expect the current (single cable) setup will be 150-160 MB/s. If the 2-cable setup is "successful", the test should be 175-190.

That's about what I would expect for 19 of those drives on a single M1015 (note that 20th drive, the cache, is not a factor in a parity check). Because the (single) M1015 is saturated, you are not able to utilize the full speed of the outer ~50% of those drives. If there were 3 x M1015 (and no expander), I'd expect it to complete in 36-38Ksec. With that mix of drives, that's about as good as you can expect (regardless of MB and controller). You might get a slightly faster completion if you had 5 array drives on the N40L itself (and only 4 on the MV8), but logistics might preclude that. If you ever add a 6th array drive (of that same ilk) to the MV8, expect your time to increase 15-20%. With the current layout, I don't think a controller upgrade will buy you much performance (<5-10%) [pls let me know if I'm wrong]. Actually about 15% faster completion for the "weakling" ... which highlights the effect of "overburdening" a controller. I presume that is with a single M1015 + expander. Because, with 2 x M1015, I'd expect 20-22Ksec. Those drives are capable of 160-195 MB/sec across the outer ~30-40%, but they are being limited to a max of ~150-160 (with 12 of them on a single M1015).

and what make/model?

How many drives total, and what make/model? Be careful of extrapolating from BobP's 12-drive test to your (say) 20-24 drive set-up. My (gut) prediction is that BobP's comparison will show no (or very little--~10%) difference in (parity-check) performance. But, with 24 drives, the performance of 3 x M1015 will be ~twice as good as one expanded M1015. Bob's 12 (fast) drive scenario is right at the "tipping point" of the M1015. --UhClem

Now that I've familiarized myself with your rack's drive connectivity, I agree. Given the infeasibility of using the mobo ports, don't waste your time testing with the SASLP (with 8 drives connected). The operative shortcoming of the SASLP is not that it is (only) x4, but that its x4 is only PCIe V1. In fact, it would not surprise me if the SAS2LP (PCIe x8 v2) in the x4 slot (phys x8) performed just as fast as in a x8 slot. --UhClem

Auggie, Using 3 x SAS2LP is overkill. You can get the same performance [with less added expense] (on your mobo) by using 2xSAS2LP + 1xSASLP + 4xSATA2 mobo ports; only put 4 drives on the SASLP. You might also get additional improvement by only putting 7 drives on each SAS2LP, and using the other 2 (SATA3) mobo ports. As I mentioned to BobP above, comparing the observed speeds of the initial 5 minutes of a parity-check is sufficient to choose an optimal confuguration. --UhClem

Well, you are definitely saturating the throughput limits of your M1015. Consider: Those drives (ST3000DM001) do an average of 150 MB/s (200max/100min). If they were not hampered by controller limits, you would have completed in 20000-21000 secs. A simple, but effective, improvement you can make is to move 4-6 drives onto your onboard SATA ports. It shouldn't be necessary to run a complete parity-check--just start it, and note the speed for the first 5 minutes. Do that before and after the above drive migration. --UhClem

Auggie, Keep in mind that the total actual throughput is limited by the controller/HBA, and using an expander just spreads that limited throughput over more drives, resulting in a lower max speed (for a parity check). I believe that the m1015 controller(s) that BobPhoenix uses have better throughput than the SAS2LP-MV8 (I don't want to hear how they are both PCIe x8v2, so they both have 4GB/s--crap!!), so be careful in extrapolating his results to your situation. I don't know how many drives BobP is pushing through his m1015+expander set-up(s), but it is possible/likely that even his better throughput is still a bottleneck (for him)--but, since he is slot-limited, he has no choice. You don't have that limitation--so, why constrain yourself with that "solution"? Unless you are OK with slower-than-need-be parity-checks ... --UhClem

Hence my caveat regarding the value of the source... My remark was not directed at you [else, I would have quoted you.] (Anyway, you clearly expressed some doubt to the veracity.) Common sense tells me it is hogwash. If there was an issue, you would have heard about it--and experienced it. The market would never accept such crap (and there would be a specification for it.) --UhClem

"I read it on the Internet, so it must be true."

Those ARE the bad blocks, in more detail and only the last 5. UNC is short for UNCorrectable, so "Error: UNC at LBA = 0x0fffffff = 268435455" roughly means "bad block at 268435455". Not this time ... Look at the fuller "picture"-- first, always be a little suspicious of numbers that are "all ones" (ie, 0x0fffffff); then look carefully at the preceding commands in the error log for the conclusive clue. "The devil is in the details." --UhClem

Try this: /boot/usbreset /dev/bus/usb/`lsusb|sed -n "/0ccd:0097/s/^Bus \(0..\) Device \(0..\).*$/\1\/\2/p"` [Note that the (hexadecimal) <Mnfr:Devc> string is unique to the device you are interested in; whereas you might just have more than one "TerraTec ..."-described device. So that is how we select the line to process.] --UhClem

tr = "translate" transliterate [cf: Unix man page, 1970s]

I'd put this in the "so what" category => the time it takes to do a pre-clear is very much irrelevant. Maybe to you ... but how about the rest of unRAID users: 1. What about the saving in power (and heat) [Pardon me for getting back on-topic ] 2. Reduction in array downtime (for those who don't keep a spare) 2a. Reduced time for running unprotected, because spouse/kids need to watch a flick 3. [unproductive] wear-and-tear on the new drive and the rest of the system 4. (All other things being equal) Faster is better!! [And speaking of the (...) in #4] ... To start is absolutely false. As I originally pointed out, they contribute a very negligible 0.5% overhead, AND I am not removing that from the procedure. [Note (above): same exact functionality] In addition to taking minimal time, it also doesn't impact the memory usage (because it is accomplished using Direct I/O). Something else you failed to comprehend was that the "head dislocation dance" does NOT contribute to "confidence level" (read the original explanation). But it is harmless (and takes negligible time), and keeps the drive from getting bored . Sour grapes ?? Obviously, you don't have the knowledge or the creativity/ingenuity to take/solve the challenge. But I think someone else will. And, once that light bulb flashes, it's only about 20 lines of sh and less than an hour of time. Pretty damn good ROI !! [not that I'm trying to get back on-topic, and attempting to put a damper on all the "fun" you Green is god fans seem to be having . I know Paul gets that one--anyone else?] --UhClem

Take my own challenge?? Where's the fun in that? Maybe you don't think I already know the answer [but I do!] or Maybe you're trying to goad me into spilling the beans ... Which would deprive someone who (like you) loves a challenge, but also does have an inkling. That person should get the satisfaction (of the solution) and the appreciation from their community. [Remember, I don't use unRAID--think of me as a "stranger from a strange land" ] --UhClem "Welcome to the Future Fair--a fair for all, and no fair to anybody"

[ all you want ... but,] on the subject of disk subsystem performance (measurement/analysis), you are at the novice stage! You can continue to deny that, or you can embrace it and endeavor to educate yourself on the subject. While you might not like my approach (think of it as a corollary to "tough love"), it is my way of helping you. Consider ... isn't it better to learn that the premise upon which you are basing your claims, and even technical advice to others, is incorrect? (Rather than continue to misinform others, and (ultimately) embarass yourself.) As for ~ 50 years ... So what!! I'm an old fart myself (received my first software royalties in 1969, and was supporting Internet development when there were 8 machines connected [1970]). But something I learned back then was the extreme importance of knowing what you don't know ... and the value of admitting your mistakes. Sounds like back-pedalling to me. You had stated in an attempt to bolster your (flawed) argument that and by using the (presumed) authority of Limetech choosing the same (antiquated) [chip-level] components for their "new" product. Then (like I said), stick to 11 or 12 drives maximum. Because the 14 drive config (that you are so sure of!) will drop that average by 20+%. --UhClem

Your problem, regarding disk subsystem performance, is that you don't know how little you know. For example, you haven't even progressed past the arithmetic-based-on-([raw] PCIe)spec stage. New? Feh! Based on a 5+ years old mobo chipset and a 4+ years old controller chip. They just barely kept up with disk drives of that era. Like I said, you don't know ...

Like I just said, your board makes an excellent 6-drive energy efficient server, But scale it up to 16 drives and I think it will start to choke. 16?? Hah!! You're far too kind. Try 11 or 12 !! But, I commend you. You do appreciate the basics of performance (measurement/analysis), and seem to take the right approach to learning, and understanding, the concepts behind it. (Certain others will hopefully follow your example.) --UhClem

Yeah, it's called Direct I/O. badblocks uses it by default, but it's disabled by -B dd doesn't use it by default, but it's enabled by iflag=direct and/or oflag=direct Preclear could invoke dd using Direct I/O during pre-read & zeroing with no ill-effect (but significant reduction in buffer/memory use). However, it can't do so during the post-read phase because of all it needs to accomplish. But there's little benefit from reducing the memory usage of Preclear during only 2 of its 3 phases; it's really all or nothing, especially since the post-read phase typically takes the longest. None of my comments here should be taken as critical of Preclear. It is a superb tool and an excellent example of shell programming. JoeL deserves your continued appreciation and support for his efforts and contributions. Why don't any of you help--by building upon that excellent foundation, and adding these improvements? Reminds me of that Little Red Hen children's story. [Note: I don't use unRAID myself, so I don't think I'm being a hypocrite here. (I don't want to eat the bread )] For any who missed it, I'd like to repeat a "challenge" I made at the end of this posting[link]: Note: if you combine the correct solution to the "challenge", with the simple addition of having dd use Direct I/O during the pre-read and zeroing phases, you get both better performance and a significant reduction in memory usage. --UhClem

Hey, good for you! That's more like excavating (than digging). A favor please: could you post with an attachment of your lspci -vv output. Thanks. (Apologies to all others, but PM was too restricting [no attachments; msg size too limited]) --UhClem

Thanks. But I must ... Murphy is a master criminal--and, even when you're good/lucky enough to catch him, he never goes to jail.

There are many reasons for a disk to perform below its specs. But there are hardly any reasons for a disk to perform above its specs, especially by the amount indicated in your report. Not only that, but it is not just the one disk, but both! (The other disk is only 20-25% over spec [vs ~30% for the quoted one].) Additionally, both 3rd passes are notably faster than the average for all 3 passes. Far-fetched as it seems, I've got to ask ... Did you set the system clock (back by about 1 hour) sometime Saturday morning? [via date -s XXX]

Those odds are 0.00 (at least, with Powerball, you have > 0 odds ) The buffer cache keeps the most recent data. Doesn't preclear.sh do strictly increasing-LBA sequential operations (for all buffered I/O) ? [Regardless, as you noted, (even in the most optimally perverse case,) a few "fast" GB out of 3TB would have immeasurable negligible effect.]

Your disks looked perfectly normal to me. Except, maybe, the Read performance--it's too fast . Specifically, the Pre Read Time speed. 209 MB/s as an AVERAGE for the entire drive, is fantastic. I'd have expected something closer to 150-160. "There's something going on here, but I don't know what it is ..." Ideas?

They may have to take more abuse but I'll bet the expected USE is less. They get disconnected and therefor are not powered up as much. They have less warranty--and they have no performance specifications to meet. A drive can be inferior for quantitative reasons as well as qualitative reasons.