UhClem

The whole time, or just post-read verify ?? (I don't use Unraid, but I vaguely recollect the details of Joe's preclear.) No, it will not affect the CPU usage. It does (effectively) eliminate the I/O bottleneck of the on-board (chipset) Sata sub-system. That CPU usage you saw during pre-clear (x2) should not guide any (re-configure) decision you make.

"Was that nine chips, or only eight? In all this excitement ..." -- Dirty Harry 😀

Sort of ... it shouldn't be used for one of the six array drives, since that would further divide the 650-700. But, it could/should be used for the cache drive; then it could only (slightly) impact mover operations, and only if TurboWrite was enabled. The (2-port?) add-in card would connect array drives 5 and 6. A "full-spec" PCIe x1 Gen2 card (e.g. ASM1061-based) , giving ~350 MB/sec, would not lower the "ceiling" of ~160 (for 4) on the mobo SATA. The only improvement would be that the cache SSD could operate at full (SataIII [~550]) speed, but that is moot, since, as your cache, it is inherently limited by your 1GbE network (on input) and your array (on output). Very little bang for the extra bucks, since you'd need a PCIe >=x2 card to handle >2 drives, and not lower the "ceiling". Here's a neat idea, if you really want to eliminate the speed bottleneck: (Assuming that your x16 slot is available,) Get a UNRAID-friendly LSI-based card (typically PCIe x8, at >= Gen2), and connect the built-in 4-bay Sata backplane to it. Note that the thick cable/connector (left of Sata-5), which connects that backplane to the mobo, is actually a Mini-SAS 8087. And can instead be connected to (one of the connections on) an add-in LSI card. That completely eliminates the bottleneck for SATA 1-4. "But, wait, there's more ..." Then you put a standard SAS-to-SATA breakout cable into the now-empty mobo connector and use 3 (of the 4) SATAs for drives 5 and 6, and your cache SSD. That gives you full SataIII for the SSD (FWIW) and you've still got SATA-5 and the eSata, plus "breakout #4", to play with for whatever. "But, wait ...." If you get a 4i4e card, you can (later) add 4+ more drives externally, and still no bottleneck! (Pretty neat, huh?) [Important, LSI card must have low-profile bracket.] I don't use UNRAID, so I can't be certain about the re-config issue. Once you've cleared that, I strongly encourage you to start with what you've already got. Not only will it give you time to think it all through, and find exactly the pieces that will serve you best, but it will give you an opportunity to assess your N40L's performance with current version of UNRAID. Then you can extrapolate from that initial CPU/throughput ratio to be sure that the CPU won't become a new/unexpected bottleneck if/when the throughput increases from 650 to 1000 (2-port card) or 1300+ (LSI).

No limitation! -- for at least another decade (Since you mentioned that you'd be running 6 newer (ie, faster) array drives,) The SATA controller in the N40L (and 36L + 54L) has a maximum (combined) throughput limitation of ~650 MB/sec. With 4 array drives on the built-in SATAs (and 2 on the add-in), that would limit your "parallel" operations (parity-check, rebuild, Turbo-write) to 160 MB/sec, but many/most newer drives are capable of 200-250 MB/sec max (130-150 min). To maximize your array performance, you'd want to be very selective in your choice of add-in SATA card. The one you linked is based on the SiI3132, which is a notable under-performer (80MB/sec max for each of 2 drives [150-170 total])--scratch that one. Better would be any Asmedia ASM1061-based 2-port card; 170MB/sec each of 2 drives. At least that would preserve the 160 limitation imposed by the built-in. [Note: these configs don't utilize the N40L's eSata, so it would stay available for a backup/import-export external drive.]

How about this: [from your 800k syslog] lines 759-763 Mar 11 07:50:54 Tower kernel: ahci 0000:04:00.0: SSS flag set, parallel bus scan disabled Mar 11 07:50:54 Tower kernel: ahci 0000:04:00.0: AHCI 0001.0200 32 slots 2 ports 6 Gbps 0x3 impl SATA mode Mar 11 07:50:54 Tower kernel: ahci 0000:04:00.0: flags: 64bit ncq sntf stag led clo pmp pio slum part ccc sxs Mar 11 07:50:54 Tower kernel: scsi host7: ahci Mar 11 07:50:54 Tower kernel: scsi host8: ahci Then searching for 0000:04:00 leads to: [line 373] Mar 11 07:50:54 Tower kernel: pci 0000:04:00.0: [1b21:0612] type 00 class 0x010601 And [1b21:0612] is the Vendor ID (Asmedia) : Device ID (ASM1062) pair for that controller. "A rose by any other name ... is still a rose."

Maybe ... maybe not. Consider that the 9211-8i (and its LSI SAS2008-based brethren) appear to have an ultimate bottleneck of their internal CPU/memory/firmware which precludes them achieving the maximum (real-world/measurable) PCIe Gen2 x8 throughput of 3200 MB/s. Your tests, on the H310, show 2560 (8x320) MB/s. Whereas the 9207-8 (and its SAS2308 brethren) do have the "muscle-power" to achieve, and exceed, full Gen2x8 throughput (3200); you measured, on a Gen3x8, 4200 (8x525) MB/s--and that is limited, not by the 2308, but by maxing out the Sata3 (real-world) speed of ~525 [I suspect that SAS-12Gb SSDs would do better, no?***] So, for 9211 vs 9207, using Gen2, it comes down to a cost/benefit decision (plus, an appropriate degree of future-proof factor; re-use following a mobo upgrade). *** [Edit] Actually, NO -- it looks like 12G SAS connectivity was not offered until the 93xx series.

Wouldn't the max throughput be (slightly) limited by the PCIe Gen2 of the R710? [ie, to ~400 MB/s each for 8xSata3-SSDs]

[my mistake] Yes, all those "generic" references to the Sonnet Allegro Pro card clouded my thinking. [Note: the one hyper-link to the Pro card (on first page) now 404's--shame on Sonnet--they should have "preserved" the link, redirecting it to its new "home", under their LegacyProducts, and suggested the new "Pro".] I totally agree with you regarding the need for documented specificity before taking action on a reference/suggestion for a functioning solution. And, NO, I'm not using any of these cards--just a passing interest in atypical "solutions". [I was aware of this thread, but never actually dug into it--then I was trying to find out more about the Pericom Semiconductor PI7C9X2G608GP switch for this UnRAID thread ... and here we are. I don't know what chips are used on the Allegro Pro 3.0 card, but the 3.1 card uses the Pericom PI7C9x2G308GP (can see it in the picture @ the Amzn lnk) [4<==>2x2]. I'm real curious to know which chip is used for the (2) USB controller chips. If anyone does get one to try, please do post a relevant snippet from either the kernel meggages or from lspci. tnx [Edit] Just found it--the 3.1 Pro uses 2 Asmedia ASM1142 Sorry for the confusion/distraction. Good luck to all on your quest.

Amazon link Something interesting about the Sonnet card: it uses a PCIe switch chip to split a PCIe_V2 x4 connection into two x2's each going to one of two USB controller chips. Apparently, each of those USB chips has two "sub-devices" each of which can be passed (as desired here). Compare that to the Startech card, which also has a PCIe switch chip (from the same company), but its switch splits a PCIe_V2 x4 connection into four x1's, each going to its own USB chip. Yet, this one causes grief. In addition to working as you all desire, the Sonnet card has the advantage of higher maximum single-port throughput, since its (2) x2 chips can share their two-lane bandwidth on-demand/as-needed to its two "sub-devices". Whereas, the Startech, even if it worked for you, would limit each "sub-device" to an x1 lane of throughput. [I realize that connectivity (vs bandwidth) is more the goal here, but ... just saying ...] --UhClem "How can you be in two places at once, when you're not anywhere at all."

Those ARE the bad blocks, in more detail and only the last 5. UNC is short for UNCorrectable, so "Error: UNC at LBA = 0x0fffffff = 268435455" roughly means "bad block at 268435455". Not this time ... Look at the fuller "picture"-- first, always be a little suspicious of numbers that are "all ones" (ie, 0x0fffffff); then look carefully at the preceding commands in the error log for the conclusive clue. "The devil is in the details." --UhClem

tr = "translate" transliterate [cf: Unix man page, 1970s]

Thanks. But I must ... Murphy is a master criminal--and, even when you're good/lucky enough to catch him, he never goes to jail.

There are many reasons for a disk to perform below its specs. But there are hardly any reasons for a disk to perform above its specs, especially by the amount indicated in your report. Not only that, but it is not just the one disk, but both! (The other disk is only 20-25% over spec [vs ~30% for the quoted one].) Additionally, both 3rd passes are notably faster than the average for all 3 passes. Far-fetched as it seems, I've got to ask ... Did you set the system clock (back by about 1 hour) sometime Saturday morning? [via date -s XXX]

Those odds are 0.00 (at least, with Powerball, you have > 0 odds ) The buffer cache keeps the most recent data. Doesn't preclear.sh do strictly increasing-LBA sequential operations (for all buffered I/O) ? [Regardless, as you noted, (even in the most optimally perverse case,) a few "fast" GB out of 3TB would have immeasurable negligible effect.]

Your disks looked perfectly normal to me. Except, maybe, the Read performance--it's too fast . Specifically, the Pre Read Time speed. 209 MB/s as an AVERAGE for the entire drive, is fantastic. I'd have expected something closer to 150-160. "There's something going on here, but I don't know what it is ..." Ideas?

There are numerous reports (on other forums' MicroServer threads [homeservershow.com & overclockers.com.au]) of people successfully installing one or another unlocked ("modified") BIOS (initially intended, and used, on the N36L/N40L) on the new N54L. If you stay with the stock BIOS, SATA ports 4 & 5 are locked in IDE mode and limited to SATA I speeds. Also they are incapable of hot-swap usage and port-multiplier controllability (both of which require the port to be in AHCI mode). Just Do It!!

Note that I "stuck my nose in" here because it seemed like both RobJ and JoeL agreed that the time added to the script/cycle run (by the "feature" you described) was of some significance. It was only in the process of composing my previous (ie 2nd) response that I made an effort to quantify it (~0.5%). I'll bet neither of you realized it was so negligible--if I had, I wouldn't have bothered ... but then, look at all the fun we'd have missed. I still contend that a more apt rationale for adding the "feature" would be "I didn't want the drive to get bored." [Those 6 seeks every ~20 seconds are not going to affect the temperature. Here's a little experiment I just did. I had a spinning, but idle, drive at 30C. I did 5 minutes worth of flat-out reading (similar to your pre-read w/o the dance). When it finished, the drive was at 31C. I let the drive rest for 15 minutes; back to 30C. Then did 5 minutes of flat-out seeking (seektest); when finished, the drive was at 35C (that was ~1400 seeks per 20 sec.).] No, it does not. In even the most perverse case, where each and every seek resulted in a read-retry, it would not even have doubled that overhead. Ie, instead of ~0.5% extra, it would have been <~1.0%. How is that going to "show in the speed of the preclear process"? Also, it does not show in a SMART report. (In anticipation of a misguided reply ... Seek_Error_Rate is not only undocumented, but also looks to not even be "implemented" on most drives (only Seagate)) Huh? 995/1000 is a fraction, right? Seriously, any measurable speed difference was not caused in any way by the dislocation dance. ========== And, now for something completely different ... Here's a challenge: Add 10-20 lines to the preclear script that will cause the post-read phase to run just as fast as the pre-read phase for many/most users. For the rest, no change (but there would be such clamoring that they could easily/quickly join the party). [same exact functionality/results as now.] Who's wants to be the hero??? Think about it-- ~5-10 hours saved per cycle for a 2TB drive. [No questions ... just think ] --UhClem

I know you did, and I respect that. (I've been in that position many times, and) That is precisely why I said "take a step back"--meaning to try to get a different view/perspective. But, here's the problem (as I see it): For a marginal drive, that little head-fake () might just cause the subsequent read attempt to be off-track and/or un-settled, BUT, if so, the drive will detect that; also, even if the drive doesn't detect that it hasn't settled sufficiently, and proceeds with the read, it will obviously get ECC failure. In all of those cases, the drive will merely RETRY the read (but this time with no/negligible prior head motion), and succeed. That little dislocation-dance (every 200 "cylinders") is just a (very minor) "waste of time" [looks like only about 0.5% extra], but has no chance of leading to any "feedback". Remember, a drive will RETRY a read 10-20 times before giving up and returning UNC to the driver (and the driver will RETRY 4-5 times before giving up and returning error to the calling program). However, I definitely agree that a new drive should really get a mechanical pummeling!! (But, instead of a couple of gnats, how about a swarm of horseflies.) I give my drives 5+ minutes of constant seeking, which also serves to verify that the drive's seek time is within spec. [seektest -n 20000 /dev/sdX -- my own little hack; don't know if Linux has something like it]. If any relevant component is sub-par, or inclined to be, now's the time to find out. To quote Crocodile Dundee: "Now, that's a torture test." (compared to the little twitches in preclear). I'll repeat this test occasionally (at least a few times per year). Following that initial torture session, I do a thorough surface integrity test (xfrtest ... /dev/sdX -- another personal hack). I try to repeat this once/twice per year, tracking the (very quantitative) results. --UhClem

Joe, I understand what you intend the above dislocation enhancement to accomplish, but I'd suggest that you (take a step back and really) think about it. What it does do is cause a slight (probably undetectable) seek noise, and increase the elapsed time of those phases (apparently noticeably). [Neither of which were your actual intent (but unavoidable side effects of your intent).] --UhClem

Two months older news now, than when I tried telling you ... Have you tried plugging the SansDigital into the MicroServer's eSATA port? [surprise!! ] Probably will, but performance (parity check, etc.) might be yucky. Even there, you will be limited to about 120 MB/s total bandwidth with the PM enclosure. The SiI3132 has a bogus transfer rate limit of ~120 MB/s (even though it is PCIe x1 v1, and should be able to get 180-200 MB/s).

Agreed. But I am a software person, and a seemingly analogous motherboard's BIOS would be an easy starting point for comparison. I had done that too, but only within my limitations (I studied EE [almost 50 years ago] but wasn't good at it). One thing that caught my eye (in the DataSheet) was Table 61 (pg 112) -- Performance mode. But then there is the last entry in Table 28 (on pg 77) for AZ_SDOUT which implies (to me) that Performance mode is always available. Isn't it possible that HP decided to omit/remove a 6Gb/s setting from their BIOS so as to avoid that slight bump in TDP power draw (5.3W vs 4.9W--ref Table 61)? Or, as you surmised, maybe just to cripple the MicroServer market-segment-wise, relative to its more macho Proliant brethren? Hence, that is why I suggested checking another (SB820M-motherboard) BIOS--to see if anything jumps out at you. For example, in the MicroServer BIOS, can you tell me what effect the (SATA) 1.5G setting has, relative to the 3G choice? --UhClem

If it was easy, we wouldn't be having this discussion . There are motherboards that use the same SB820M and DO support 6Gbps SATA. I suspect their BIOS might have some good clues.

Is it possible to enable SATA III/6Gbps? It is documented as being available in the AMD SB820M. Thanks.

Have you tried plugging the SansDigital into the MicroServer's eSATA port? [surprise!! ] Seems a pity to put a PCIe v1 card (possibly further constrained by a PCI dependency) into a PCIe v2 slot, though I empathize with the attraction for that card's connectivity [2 SATA + 2 eSATA]. Yes, they are. The MicroServer's 6 port SATA subsystem (via the SB820M SouthBridge) is pretty decent. You're still ~100 MB/s shy of saturation [(2x170)+(2x130) = 600]. I measure about a 675-700 MB/s ceiling (on a N40L). Envying your 50% faster CPU ...

[To you "youngsters": Please allow us two old farts to reminisce; maybe it'll be of some interest to a few of you.] Before I respond, allow me to put this in a chronological perspective (using a human lifecycle): I'd say that today, Unix is solidly in middle-age; it's healthy, accomplished, and has a long future ahead. The PWB/Unix that you referred to was first released in 1976, which I would characterize as Unix's early puberty; the time period you went hands-on, 1979-80, is getting close to its adolescence. When I first had hands-on with Unix and the kernel was Nov, 1974; that was the first release of the code from Western Electric (aka AT&T). Seven sets of tapes were mailed from Murray Hill; one came to me (on behalf of my employer). I would call that Unix's early weaning . Were you one of those who used "adb -w" on the kernel while loaded in memory? I fully respect anyone with that skill. adb? That's like solid food [cf: weaning]; there was no adb. There was only db, and truthfully, it is a misnomer to call db a debugger. Its limited functionality was to examine a (binary) executable (pre-execution), or to analyze a core-dump (post-execution). It could do nothing to help you in debugging a running program, or (obviously) a live kernel. [i will always remember my phone call to Ken Thompson, asking him where the (real) debugger was. When he told me there was none, I was honestly incredulous. I instinctively asked him "How did you and Dennis get all of this stuff working?" His answer: "We read each other's code."] And that is part of what identifies true software deities. But, as a mere mortal, I still needed to debug my modified kernel. So, I hacked the header of DEC's ODT-11 (octal debugging tool for PDP-11) so that it was compatible with Unix ld (the [link-]loader). Adding this odt11.o to the rest of the kernel object files, ld gave me a kernel that I could debug, albeit painfully (no symbols--constantly referring to a nm listing for a symbol map). I believe I was the first person to actually debug (breakpoints, single-step, etc.) on Unix (kernel or user-space). Not to evade your question , but by the time adb came around (1976-77), I was winding down my kernel hacking. [i guess I don't get your respect, huh? :)] Oh my, time-traveling is tiring ... this geezer has to take a nap. I'll follow-up the on-topic discussion of this (computer) memory exhaustion issue a little later. --UhClem "Forward! Into the past... back to the shadows, again..."