I'm trying to figure out the maximum array size my HP N40L Microserver can support. In other words, at what HDD size will the hardware become unusable? I know the motherboard has a x16 slot that could be used for an HBA card, but I'd like to stick to the native 6x SATA II ports. Usage is NAS only. Assuming identical drives, the solution seems very roughly based on:

 

Max acceptable time of limiting operation x Max realistic average drive speed on SATA II = Max drive size

 

I'll spare you the minutiae of my rationale, but I assumed the parity check was the slowest/limiting operation. My SWAG numbers are:

 

18 hours x 250MB/s = 15.8TB formatted capacity

 

To be on the conservative side, do 16TB SMR drives seems like a reasonable maximum?

I have the N54L, but expect the N40L to be the same, due to the A-link, maximum parity check speed you'll get using the 6 onboard sata ports is 125MB/s.

Thanks for the reply. If you don't mind answering:

 

1) What is the A-Link? I'm referring to parity sync/check speed, just to be clear. 125MB/s makes me think you might be referring to max parity write speed over the network?

2) What is the largest drive you'd consider upgrading to using the native SATA ports?

A-link is the connection between the south and north bridge chipsets, essentially is a pcie 1.1 x4 link, maximum theoretical bandwidth is 1000MB/s, real world is ~750MB/s.

 

I have one with 4 x 8TB seagates, parity check takes 15h, I would be ok with anything under 24h.

 

You can see the limit on the screenshot below, at that time it had 3 x 8TB and 1 x 3TB, hence the speed change, but it's visible in the beginning.

Your assumption that drives will hit areal densities and rotation speeds that let them average 250MB/s is probably reasonable ... the current crop of 1.33TB/platter units can already average over 150MB/s (over 200 on the outer cylinders).

 

But why would you consider 18 hours a "limit" for the parity checks?  Granted, if that's your "I don't want it to ever take longer" factor, then 16TB drives would indeed be a limit if the interface was SATA-II.    But that "limit" isn't really due to the interface, it's because you want to put an upper limit on the parity check time.

 

A-link is the connection between the south and north bridge chipsets, essentially is a pcie 1.1 x4 link, maximum theoretical bandwidth is 1000MB/s, real world is ~750MB/s.

 

I have one with 4 x 8TB seagates, parity check takes 15h, I would be ok with anything under 24h.

 

You can see the limit on the screenshot below, at that time it had 3 x 8TB and 1 x 3TB, hence the speed change, but it's visible in the beginning.

 

Ah! Thank you. That was exactly the information I was looking for.

 

But why would you consider 18 hours a "limit" for the parity checks?  Granted, if that's your "I don't want it to ever take longer" factor, then 16TB drives would indeed be a limit if the interface was SATA-II.    But that "limit" isn't really due to the interface, it's because you want to put an upper limit on the parity check time.

 

You're right. Sorry, I was trying to stay concise. By "max acceptable time" I meant preferred upper time limit. I was trying to encourage others to state theirs and provide a reason.

I actually agree that there's an upper limit to what I'd consider an acceptable parity check time.

 

I agree with Johnnie ... anything under 24 hr is okay with me.

 

Fortunately, technology has a way of helping with those desires ... there are already 1200MB/s SAS interfaces (with SATA not far behind);  and disk drive densities will likely continue to improve, along with higher rotation rates => so 24 hr parity checks will probably be achievable even as drives hit 16TB or well beyond.

 

For that matter, I wouldn't be surprised in time to see 16TB or larger SSDs 

 

My back of the napkin calculations:

 

johnnie.black's 4x 8TB server

1) 750MB/s max. bandwidth / 4x drives = 187.5 MB/s max. throughput

2) 8TB = ~7.24TB actual

3) 7.24TB / 15hr parity = 140.6MB/s realized throughput

4) 140.6MB/s / 187.5MB/s = ~75% of theoretical max. throughput

 

N40L Max Capacity

1) 750MB/s max. bandwidth / 6x drives = 125 MB/s max. throughput

2) 75% of max. throughput = 93.72 MB/s realized throughput

3) 7.24TB / 93.72 MB/s = 22.5 hours parity check speed

 

So realistically, the N40L maxes out with 6x 8TB drives for a maximum usable capacity of 36.2TB.

 

Again, thanks for the help everyone.

4) 140.6MB/s / 187.5MB/s = ~75% of theoretical max. throughput

 

750MB/s is the actual maximum bandwidth, but you have to remember that disk speed is not constant, parity check goes from the outer (faster) cylinders to the inner (slower) cylinders, see the 8tb Seagate speed graph below, with my 4 disks, average speed is 148MB/s, with 6 I estimate average speed at ~122MB/s, and a parity check will take a little over 18h.

 

So with 4x 8TB drives you only saturate the port for the outer 25% of the drive. With 6x 8TB you'd saturate the port for the outer 80+% of the drive.

 

Conventional 6 platter 10TB disks would require ~1.67TB platters, or 25% faster sequential speed over current 8TB disks. In other words, 10TB+ would use (and be stuck at) the full 125MB/s bandwidth throughout the disk.

 

10TB -->  ~9.05TB formatted capacity / 125MB/s average throughput = ~21h parity check

12TB --> ~10.86TB formatted capacity / 125MB/s average throughput = ~25h parity check

 

With 10TB drives next on manufacturers' roadmaps and <24h parity check speeds, that sounds like a good maximum for me. Time to be patient and ponder why ~45TB of storage seems like so much... yet so little. 

So with 4x 8TB drives you only saturate the port for the outer 25% of the drive. With 6x 8TB you'd saturate the port for the outer 80+% of the drive.

 

Conventional 6 platter 10TB disks would require ~1.67TB platters, or 25% faster sequential speed over current 8TB disks. In other words, 10TB+ would use (and be stuck at) the full 125MB/s bandwidth throughout the disk.

 

Exactly, I believe that’s the practical limit for these servers, 10 or 12TB disks.

... Exactly, I believe that’s the practical limit for these servers, 10 or 12TB disks.

 

However ... when 12TB disks are common, I suspect 1200MB/s SATA ports will be also => and likely won't have the bus speed limitations you're looking at with the current Nxx series servers.    But I agree that with the current N54L/N40L servers the practical limit is indeed 10 or 12TB drives.    Anything larger would just be far too slow on parity checks/rebuilds.

 

To fully flesh out this thought exercise, what is the maximum array size if an HBA card is used? It would be an ~$100 upfront expense and an additional ~7W power load, but if it extends the server's potential it may be worth considering.

 

My research suggests that the PCIe 2.0 x16 slot has a direct connection the the RS785E Northbridge, bypassing the "A-Link Express II" bottleneck. The Northbridge has a 41.6GB/s max. bidirectional link to the CPU, but the unidirectional link is 16-bit and limited to a max. of 10.4GB/s. Case in point, there's a lot of theoretical bandwidth available.

 

There don't appear to be any half-height PCIe 2.0 x16 12Gb/s HBA cards; everything seems to be staying on PCIe x8. Based on johnnie.black's excellent work, the fastest PCIe 2.0 x8 card tested (LSI 2008 based) appears to have a real-world throughput of ~377.5MB/s for 6x drives. The aggregate bandwidth of ~2.2GB/s is well within the capabilities of the system bus.

 

377.5MB/s x 24hr parity check limit x 3600s/hr = ~31.1TB drive (34TB nominal)

 

In short, the HP G7 Microservers have a realizable array limit of ~155TB, with 34TB drives probably available around the 2025 timeframe -- if a HDD-industry apocalypse doesn't happen first. Do these calculations seem reasonable? Are there any other bottlenecks I'm not considering (e.g. CPU speed)? Does anyone think that the new 12Gb/s PCIe 3.0 x8 HBA cards can capture any residual 6Gb/s bandwidth?

I’m pretty sure the CPU would be a bottleneck, when I tested the controllers I also tested the N54L with an additional Adaptec 1430SA, it’s not the fasted HBA but is capable of 200MB/s+ with 4 disks, you can see that the CPU is a bottleneck with 4 onboard + 4 on the 1430SA, I didn't test with the LSI but I’d be surprised at anything above 200MB/s with 6 disks on that CPU, still it would be a lot better than the 125MB/s max. using the onboard ports.

 

These are tests I did at the time, HP N54L, Unraid v6.1.3:

 

4 x 185MB/s

5 x 150MB/s

6 x 125MB/s

8 x 150MB/s (4 onboard + 4 on 1430SA)

 

If the CPU is limited to ~1.2GB/s of throughput, there are probably a few options worth considering:

 

1) A ~$15 PCIe 2.0 x1 card that does at least 370MB/s would allow 185MB/s and 16TB drives. This would likely have the lowest upfront and power costs.

2) A ~$20 used Adaptec 1430SA, with three drives on the HBA and three on the MoBo. It would allows 200MB/s and 18TB drives.

3) A ~$25 used SASLP-MV8 card, with all drives on the HBA to simplify wiring (reuse SAS cable) and potentially shut off onboard SATA (saving power?).

 

 

 

FWIW, I found a block diagram and learned that the Hypertransport bus in the Proliant servers have 8.8GB/s bandwidth in each direction.

 

Options 1 and 2 are good, the SASLP on the other hand with 6 disks would be slower than the onboard ports.

 

So realistically, the N40L maxes out with 6x 8TB drives for a maximum usable capacity of 36.2TB.

 

Again, thanks for the help everyone.

I have 10 drives in mine (well, one is external and 3 are 2.5"). 1430SA card to attach 4 in addition to the 6 onboard SATA ports.

Options 1 and 2 are good, the SASLP on the other hand with 6 disks would be slower than the onboard ports.

 

Thanks. You've been an immense help. I've summarized my conclusions below for any future Google lurkers.

 

I'm going to settle with option #1. Cards using the ASM1061 controller are cheap, seem well supported in UnRAID and are just fast enough (~370MB/s across 1 drive based on various reviews; ~200MB/s across 2 drives based on johnnie.black's thread linked earlier). It'll also allow me to maximize the UnRaid Basic license, keep all cables inside, open the option of hot-plugging a non-array SSD on the internal SATA for VMs/Dockers and lastly, allow Pre-Clearing disks via the eSATA port.

 

The Adaptec 1430SA will allow a slightly larger max array size (highest i/o) and more SATA ports for cache pools for UnRAID+ license holders. The downside is higher power consumption (assumed based on chip size/#), difficulty connecting to the SAS backplane (as necessary for max i/o), slightly higher upfront cost and the loss of the x16 slot.

Only comment on your summary is I think the 1430SA is a better choice ... and with this card routinely available on e-bay for under $20 cost is hardly a factor.  [e.g. http://www.ebay.com/itm/Adaptec-1430SA-4-Port-SATA-PCI-E-x4-RAID-0-1-10-JBOD-Controller-Card-/181629244484?hash=item2a49f25444:g:qhEAAOSw7ThUmRwr ]

 

 

So realistically, the N40L maxes out with 6x 8TB drives for a maximum usable capacity of 36.2TB.

 

Again, thanks for the help everyone.

I have 10 drives in mine (well, one is external and 3 are 2.5"). 1430SA card to attach 4 in addition to the 6 onboard SATA ports.

 

As noted above, I've come to the conclusion that 6x 16TB drives (i.e. ~72.4TB array) is about the largest configuration the G7 Microserver can attain for <24hr. parity checks. What is your array size and parity check time? AFAIK, adding more disks spreads the peak obtainable bus bandwidth and as a result doesn't net more capacity, just the same peak capacity with smaller disks.

 

I think the simplest strategy (for me) is to add 8TB disks as my data load warrants, stop at 6x drives and wait until ~2018/9, buy 4x 16TB drives, consolidate array to new disks, sell old disks, add new 16TB disks as my data load warrants... then junk Microserver. 

Only comment on your summary is I think the 1430SA is a better choice ... and with this card routinely available on e-bay for under $20 cost is hardly a factor.  [e.g. http://www.ebay.com/itm/Adaptec-1430SA-4-Port-SATA-PCI-E-x4-RAID-0-1-10-JBOD-Controller-Card-/181629244484?hash=item2a49f25444:g:qhEAAOSw7ThUmRwr ]

 

Thanks, but what are your reasons for feeling that way?

Only comment on your summary is I think the 1430SA is a better choice ... and with this card routinely available on e-bay for under $20 cost is hardly a factor.  [e.g. http://www.ebay.com/itm/Adaptec-1430SA-4-Port-SATA-PCI-E-x4-RAID-0-1-10-JBOD-Controller-Card-/181629244484?hash=item2a49f25444:g:qhEAAOSw7ThUmRwr ]

 

Thanks, but what are your reasons for feeling that way?

 

The extra ports.