Which speed do I get on the drives when using expander backplane.

[MvL]

If I have a LSI SAS 9300-4i and I connect it to a expander backplane (12Gb) what speeds do I get on the drives? I'm not sure if I calculate this correctly..

 

LSI SAS 9300-4i

PCI-e 3.0 x8

 

Backplane 12Gb 

 

drives SATA 600=SATA III 6Gbit/s - 600 MB/s (theoretical, you never get this speed with a spinner).

 

 

PCI-e 3.0 = 985MB/s per lane | 985MB/s x8 = 7880MB/s x8 8 lanes.

[JorgeB]

Using SATA3 drives you'll get around 2200MB/s usable (from a theoretical max of 2400MB/s) for single link, double that for dual link.

[MvL]

How did you calculated that value? 

[JorgeB]

Single SAS expander link = 4 x disks link speed, so using SATA3 disks = 4 x 600MB/s = 2400MB/s, of those max usable speed will be around 2200MB/s due to protocol overhead.

[MvL]

oke, understood.

 

but, if the backplane is 12Gb and the HBA card is 12Gb per channel? Then you have 4 channels of 12Gb thus 48Gb = 6000MB...

[pwm]

19 minutes ago, MvL said:

oke, understood.

 

but, if the backplane is 12Gb and the HBA card is 12Gb per channel? Then you have 4 channels of 12Gb thus 48Gb = 6000MB...

But your drives are SATA 3 (6Gbit/s) and not SAS-3 (12Gbit/s).

[hawihoney]

Sorry for jumping in, I'm highly interested too.

 

According to the OP the single link from the adapter goes to a 12Gbit/s expander backplane. So from the theoretical 7,8 GByte/s only 4x SATA III performance reach the expander chip of the backplane?

 

I mean, the expander and it's chip can host lot's of drives. What's the limiting factor between both high speed components? The single SFF8643 port on the adapter card and the single adapter cable?

 

Comparing e.g. 9300-4i and 9300-8i. Both slurp up to 7,8 GByte/s from the bus, the chip on both cards is identical, the expander is identical and the only difference are 1 vs. 2 output ports.

 

Veryinteresting. Need to rethink my own plans.

 

[JorgeB]

26 minutes ago, MvL said:

but, if the backplane is 12Gb and the HBA card is 12Gb per channel?

Not with SATA3 drives, only if you use SAS3 drives.

[JorgeB]

1 minute ago, hawihoney said:

According to the OP the single link from the adapter goes to a 12Gbit/s expander backplane. So from the theoretical 7,8 GByte/s only 4x SATA III performance reach the expander chip of the backplane?

See reply above.

[JorgeB]

Some tests I did with SAS2/SATA2 and SAS2/SATA3 expanders here:

 

 

Edited March 18, 2018 by johnnie.black

[MvL]

Ah, interesting. You guys helping me a lot!

 

So a expander backplane is not a good idea.., you can better use several HBA cards?

[JorgeB]

Just now, MvL said:

So a expander backplane is not a good idea.., you can better use several HBA cards?

Depends on how many disks you're using, a dual linked expander provides adequate speed for around 24/30 disks, depending also on the disks used.

[MvL]

If I use a expander backplane then the following is true? so theoretical you'll get 4800MB/sec (4400MB/sec usable )dual link connection. That is 4400MB/sec / 24 drives = 183,33MB/sec usable per drive.

 

If I use separate HBA cards then:

 

I'll use a LSI sas9201-16i:

 

channel 1 || 4 drives x 600MB/sec = 2400MB/sec 

channel 2 || 4 drives x 600MB/sec = 2400MB/sec

channel 3 || 4 drives x 600MB/sec = 2400MB/sec 

channel 4 || 4 drives x 600MB/sec = 2400MB/sec 

 

that is a total 9600MB/sec and the card is PCI-E 2.0 x8 thus 500MB/sec per channel. It has 8 channels x 500MB/sec = 4000MB/sec max. So the bus is bottleneck. So max 4000MB/sec / 16 drives = 250MB/sec per drive.

 

I'll use a LSI sas9201-8i:

 

channel 1 || 4 drives x 600MB/sec = 2400MB/sec 

channel 2 || 4 drives x 600MB/sec = 2400MB/sec

 

that is total 4800MB/sec and the card is PCI-E 2.0 x8 thus 500MB/sec per channel. It has 8 channels x 500MB/sec = 4000MB/sec max. So the bus is bottleneck. So max 4000MB/sec / 4 drives = 1000MB/sec per drive.

 

Is this correct?

 

[JorgeB]

19 minutes ago, MvL said:

that is a total 9600MB/sec and the card is PCI-E 2.0 x8 thus 500MB/sec per channel. It has 8 channels x 500MB/sec = 4000MB/sec max. So the bus is bottleneck. So max 4000MB/sec / 16 drives = 250MB/sec per drive.

PCIe has an high overhead, max usable you can expect from a x8 PCIe2 link is around 3000MB/s

 

 

[JorgeB]

22 minutes ago, MvL said:

So max 4000MB/sec / 4 drives = 1000MB/sec per drive.

It would be divided by 8, same 3000MB/s usable, so around 375MB/s per drive.

[MvL]

yeah your right. Little calculation error.

 

So with:

 

expander backplane dual link --> 183MB/sec per drive

sas9211-16i --> 187MB/sec per drive

sas9201-8i --> 375MB/sec per drive

 

So you have the best speeds with a 2 (8 channels)  ports HBA card. So parity checks and rebuilds have the best performance in this situation. The only factor now is how fast is the drive.  I think most drive are around the 200MB/sec. Correct? Very interesting! Thank you Johnnie for you time. Much appreciated!

[JorgeB]

1 hour ago, MvL said:

I think most drive are around the 200MB/sec. Correct?

Yes, there are a few models a little faster, at around 250/270, but also keep in mind they only achieve those speeds in the outer cylinders, so say your drives max at 200MB/s like the example below, if you were using a dual linked expander there would be a small bottleneck, but only for the first 20% or so of a parity check or rebuild, then the limit would be the disks.

 

 

 

 

 

[JorgeB]

16 hours ago, johnnie.black said:

PCIe has an high overhead, max usable you can expect from a x8 PCIe2 link is around 3000MB/s

 

 

A little late but wanted to get permission before posting since it's from the alpha version on the new diskspeed test, here you can see the LSI 9211 or similar usable bandwidth in an x8 PCIe 2.0 slot.

 

 

There was a problem with this version detecting the 8th drive, but it's easy to see that in this case the limit is reached with 6 devices.

[MvL]

That's cool Johnnie, and indeed your absolutely right. Is the DiskSpeed test a plugin or is it part of a new UnRaid OS version?

 

Did some thinking when driving home from work.

 

The LSI sas9300-8i uses a PCI-E 3.0 x8 bus. So the bandwidth of this card is 1000MB/s x 8 lanes = 8000MB/s. If I connect the expander backplane dual link then I have 8 channels x 600MB/s = 4800MB/s. So there is no bottleneck. So we have a speed of 4800MB/s / 24 = 200MB/s per drive. Also we do not saturate the PCI-E bus. I assume we do not suffer of any overhead problems. True? 

 

 

 

20 hours ago, MvL said:

If I use a expander backplane then the following is true? so theoretical you'll get 4800MB/sec (4400MB/sec usable )dual link connection. That is 4400MB/sec / 24 drives = 183,33MB/sec usable per drive.

 

 

I was talking in some posts back of a speed of 183MB/s per drive using the expander backplane, but that is not true because of the speed of the HBA card (4000MB/s / 24 = 166MB/s per drive. I did 8 x 600MB/s = 4800MB/s but yeah the HBA card is slower.

[JorgeB]

17 minutes ago, MvL said:

So we have a speed of 4800MB/s / 24 = 200MB/s per drive. Also we do not saturate the PCI-E bus. I assume we do not suffer of any overhead problems. True? 

Yes. Didn't read the firs part, correct that you won't be hitting the PCIe overhead, but there's some SAS/SATA overhead, like I said you'll have around 4400MB/s usable on a dual link connection.

 

 

Edited March 19, 2018 by johnnie.black

[MvL]

Ah, damn. True!

 

What if I do a sas9211-8i in a PCI-E 3.0 x4 (in x8) slot?

[MvL]

Hmmm,

 

Normaly it is 8 lanes (500MB/s x 8 = 4000MB/s), but we have overhead so 3000MB/s thus 375MB/s per drive

 

I assume it just uses 4 lanes (500MB/s x 4 = 2000MB/s). No overhead thus 250MB/s per drive.

 

250MB/s is okay probably the drive never reach this speed...

 

Edited March 19, 2018 by MvL

[JorgeB]

Overhead is independent of lanes, It will work at PCIe 2.0 on a x4 slot, so limited to 2000MB/s (minus the overhead).

 

If you check the link I posted above I did almost all those tests, you can check the values.

[MvL]

Checked you post and is was very valuable!

 

Thanks for the support Johnnie!

[MvL]

If the times come when ssd's of 50TB are affordable what speed do we get with these ssd's? Because ssd's are faster as hdd's. I assume that the sata interface is also the limit. So that should be again 4x600MB/s = 2400MB/s (theoretical). 80-100MB/s single link. That's a pity cause the ssd's are a lot faster. I'm not sure if a expander chassis is a good idea...