This is maybe gonna sound dumb but can I check some logic with you guys..............?

 

In respect of PCI/PCI-E cards, logic would suggest that the throughput of a SATA card is shared by the number of devices attached to it. So for example, parity checks using a PCI card with 1 SATA drive attached would be (roughly!) twice as fast at parity checks than the same card with 2 drives attached?

 

If this is true, then is it better (from a parity check perspective) to place HDDs on different PCI/PCI-E devices rather than have, say, 4 HDDs on one device...or.... does the answer depend on whether the MB shares the PCI lane(s)? Do all MBs work in the same way, i.e. are lanes always shared?

 

MB is the Asus A8n-sli Deluxe and my real-world example is that I currently have 4 of my drives drives attached to a 4-port PCI card. I have another 4 port PCI card available, plus (recently) two 2-port PCI-Ex1 cards which were cheap so I thought 'what the hell'. My question is would I be better off having one of these 4 drives on each of the 4 devices, or all four drives on the two PCI-E x1 cards? I do not intend on increasing the number of drives in future, but rather swapping the current drives for larger capacity ones.

 

Thanks for any insight.....

MB SATA > PCI-E SATA > PCI SATA in terms of available lane bandwidth and bus speed.  Your parity drive gets worked every time you write to any of the disks in your array, but the data disk involved is the only data drive that also has a load.  The rest of the disks are in standby, unless you have other requests doing reads and writes as well.  So while in theory the bus will bottleneck easily if all disks are going full bore, the reality is much less that way.  That being said, I wouldn't put anything on a PCI bus unless all my PCI-E slots were full.

 

 

Thanks but this doesn't answer my question which was 'parity check specific' (all drives spun-up and accessed).

Day to day speed is fine (streaming bluray etc) but my parity checks take about 26 hours so i'm looking to improve this without buying new hardware.

cheers

From other conversations here, it seems common that ALL PCI slots share the same channel.  So if you have two PCI sata controllers they'll be fighting for bandwidth. 

 

As I understand it, PCIx and PCIe both generally have dedicated channels so each card has it's own path.  This is why the advice of "PCI last".  Others know much more but I wanted to give you a head start until they get here to straighten us out.

Well, I was trying to indicate where your bottlenecks are.  Obviously, one drive per PCI bus is preferrable to 4 drives over a single card on that bus.  

 

Maximum device bus bandwidths (assumes PCI-E version 1.1)

PCI                  133 MB/s

PCI Express 1x 250 MB/s

PCI Express 2x 500 MB/s

PCI Express 4x 1000 MB/s

PCI Express 8x 2000 MB/s

 

Maximum HDD bandwidths

IDE (ATA100)      100 MB/s

IDE (ATA133)      133 MB/s

SATA                 150 MB/s

SATA II              300 MB/s

 

Reference

 

PCI Express is a dedicated serial bus, so I have expressed the parallel mode numbers above to reflect read/write speeds.  It's also important to note that it is dedicated.  Many times motherboard manufacturers will not bother putting PCI 133 slots on dedicated channels, thus increasing congestion.  

 

So, if you run 2 SATA II drives over PCI-E 1x, you will have a theortical bottleneck around ~250 MB/s because of bus limitations.  Without indicating what your drive speeds are, it's difficult to see if you hit a wall with respect to the drives, but my 7200rpm drives top out around 130 MB/s on the edge of the platter.  That means at full speed 2 drives would be fine on the bus.  

 

Assuming again that you're using 4 SATA II drives that are 7200 rpm, you will KILL the older PCI bus if all 4 drives are being hit at once, because you only have 133 MB/s to deal with on the entire bus, and your drives can saturate that individually.

 

Specific to your hardware, you have 4 dedicated SATA ports on your motherboard.  Those should be utilized before you ever get to the external device busses.  Once you need the extra ports, use your PCI-E 1x slots for the 2 cards you bought.  This assumes you're already using the PCI-E 16x slot for your video card, since that board has no onboard video.  Beyond 8 drives I would recommend getting a higher end server motherboard that sports at least 6 SATA slots and has at least one PCI-E 4x slot that can easily handle an 8 port expander.  But if you were to stick with what you've got, then you should put the smallest, least used drives on the PCI card, because that will be your bottleneck.

 

One thing you haven't mentioned is the size of the drives and their speeds.  This is important when benchmarking against other systems for comparison purposes.

 

Does that help?

Pretty sure wholly is right,  Standard 32bit PCI bus is shared.  Also remember that when you do a parity check it has to read every single byte of every disk... with lots of disks that takes a long time whatever hardware you are using.  Mine takes 15 hours, 4 drives on a PCI card and 6 on the motherboard. ( 25mb/s )

Pretty sure wholly is right,  Standard 32bit PCI bus is shared.  Also remember that when you do a parity check it has to read every single byte of every disk... with lots of disks that takes a long time whatever hardware you are using.  Mine takes 15 hours, 4 drives on a PCI card and 6 on the motherboard. ( 25mb/s )

 

Wholly is right.  PCI is shared.

What size drives?  What rotational speed?  Which drives are on the card and which drives are on the motherboard?

and you can test individual drive performance with hdparm :-

 

root@Monkey:/boot/smarthistory# hdparm -t /dev/sdf

/dev/sdf:
Timing buffered disk reads:  250 MB in  3.01 seconds =  83.13 MB/sec
root@Monkey:/boot/smarthistory# 

 

That might help you identify where your bottle necks are.

 

Thanks to everyone for the replies - gives me something to work with. I'll do some testing with hdparm and then look at spreading the load & testing again (just wanted to check it was worth doing)

 

For info, all drives are 7200rpm, various sizes (2TB to 500GB) and mix of SATA and SATA II and I've used the onboard SATA for largest and parity drives.

 

Aiden, quite right on the new MB front, but the Mrs thinks a new bathroom should take priority over a new toy that "does exactly what the old one does". Guess she has a point......grrrr.....

 

Thanks again.

MB SATA > PCI-E SATA > PCI SATA in terms of available lane bandwidth and bus speed.

 

Not 100% sure this is true all of the time. It depends on the motherboard design, I have three MBs all with six sata ports and parity checks using the same disk set range from 70MB/s to 40MB/s. I think the 40MB/s is because all six ports are sharing a x1 PCIe slot. The other MBs probably have x2 or x4 bandwidth available for the six sata ports.

 

Moving two HDD to PCIe controller on the slow MB, results in the parity speed going back up to 65MB/s.

 

For good parity performance limiting the PCI bus to two or three HDDs seems reasonable. Use HDPARAM to show the relative performance of each single drive. Remember PCI is good for 133MB/s, so two 65MB/s drives or three 45MB/s drives on a single card is probably the limit. PCI is either read or write whereas PCIe can read and write concurrently (both 250MB/s per x1). Ideally the parity drive should be on a PCIe or MB (PCIe based) sata port.

 

My parity checks take around 5 hours for 7 HDDS, 5TB all @7200RPM.

 

 

 

 

 

 

 

 

 

 

 

 

MB SATA > PCI-E SATA > PCI SATA in terms of available lane bandwidth and bus speed.

 

Not 100% sure this is true all of the time. It depends on the motherboard design, I have three MBs all with six sata ports and parity checks using the same disk set range from 70MB/s to 40MB/s. I think the 40MB/s is because all six ports are sharing a x1 PCIe slot. The other MBs probably have x2 or x4 bandwidth available for the six sata ports.

 

Moving two HDD to PCIe controller on the slow MB, results in the parity speed going back up to 65MB/s.

 

For good parity performance limiting the PCI bus to two or three HDDs seems reasonable. Use HDPARAM to show the relative performance of each single drive. Remember PCI is good for 133MB/s, so two 65MB/s drives or three 45MB/s drives on a single card is probably the limit. PCI is either read or write whereas PCIe can read and write concurrently (both 250MB/s per x1). Ideally the parity drive should be on a PCIe or MB (PCIe based) sata port.

 

My parity checks take around 5 hours for 7 HDDS, 5TB all @7200RPM.

 

I can appreciate what you're saying.  I'm providing the theory and design as laid down by the engineers who developed the standard.  You're relaying real world experience.  Generally, those two positions rarely coincide completely.  Motherboard designers always step outside the box.  I mean, look at this poor little thing that doesn't know if it's a P45 or an ION.  I guarantee you the Intel engineers did not talk to the nVIDIA engineers.