Hi all,

 

I have the ASUS M4A785T-M motherboard from the Australian budget build. (Some stuff here is harder to find like the supermicro motherboards.) I'll also be purchasing a Norco 4224 so drive space isn't an issue. The motherboard has 6 onboard SATA ports and I'm looking to expand and also allow for the future.The Asus mother board has which from what I've read will allow a maximum without bottlenecking:

1 x PCIe 2.0 x16  -8SATA - Going to get a Supermicro AOC SASLP MV8

1 x PCIe x1        -2SATA - Yet to pick one

2 x PCI 2.2        -1SATA  -Yet to pick a card (I think the pci ports share the same bandwidth so I can only use one)

Is that correct?

 

That will allow me 1 cache, 1 parity and 15 data. How can I get the extra 5 drives? I know I will lose some performance if 2 drives are sharing bandwidth  at the same time but I'll try and minimise that by how I put the files on the drives.

 

What are my options to get those extra 5 drives? Use a 4 port PCIe x1 and multiport pci 2.2?

 

Thanks Josh

Just out of curiosity: do you have 20 or even 15 drives' worth of data that you need to dump onto the array right away?

The PCI bus has limited bandwidth, which is shared among all drives connected to it.  Sometimes the PCI bus is also used for network access, in which case it shares the bandwidth as well.  (It is likely your network is on the PCIe bus and not impacted by the PCI bus.)

 

You can have many drives on the PCI bus (up to 8 per card).  And the performance on each individual disk will be good, so long as you aren't accessing several at the same time at full speed.  For most users that means that performance is good except when running parity checks.  Parity checks represent a worst case for the PCI bus because each and every disk is running full tilt.  And this will impact parity check speed.  But remember you only do parity checks once a month, so maybe you can handle slower parity checks knowing that the normal drive accesses will be fast.

 

One drive on the PCI bus will run full speed, because it isn't sharing.  Two will run very close to full speed.  I ran with 6 for a short while and the slow speed got annoying.  And that was when I had a 750G parity, not 2T.  My thought is that you'd find performance acceptable with somewhere between 3 and 5 disks on the PCI bus, depending on your personal tolerance.

 

The multi-port option is more complicated.  I know that only certain PCIe ports can be used with these multi-port cards, and I think there were some issues / limitations.  Perhaps WeeboTech would chime in on this option, or you may be able to search for some of his old posts on the subject.

 

Good luck!

kenoka

 

Not at the moment. I think I'll need more than 4 data, parity and cache in a month or so once I get all my media onto there. They are pretty much all .iso so they do use a bit of space.

 

I'm about to go buy more parts but just don't want to buy something and find later on that it doesn't get used or doesn't work. I'd like to have a rough idea of what I can and or can't do.

 

Thanks bjp999

 

so maybe run 2x 2port cards on the pci slots and that would take me to 20 disks.

 

Thanks Josh

I think you can likely find a 4 port PCI card.  Promise TX4 sticks in my head.

Thanks bjp999,

 

I remember seeing something about the promise (I think it is) tx4.

 

What about with the PCIe x1 slot. I assume if running 4 off a pci is ok then 4 sata or even more would be fine with the PCIe x1.

 

I know I can run more off the PCIe x16 slot (the supermicro one only runs at x4) but those cards got fairly pricey pretty quick.

 

Thanks Josh

The bandwidth and performance limitations become a factor in other situations than just a parity check.

 

You should plan for the worst case situation, having to recover from a drive failure. All drives will fail. It would be a shame to lose your data because of poor planning.

 

The most important and vital situation includes simulating a drive in a failed drive situation and a failed drive rebuild. I would never want to be in a situation where rebuilding a failed drive would take more than a day, which is quite possible if you place 8 larger drives (2TB) on the PCI Bus. The fastest a drive rebuild could occur in that case would be 33.4 hours; realistically it will take longer perhaps more than 48 hours. With 4 drives on the PCI bus, the fastest a rebuild could occur would be 16.7 hours; realistically it will take longer, perhaps more than 24 hours.

 

The list of performance limitation situations includes: parity checks, parity builds, simulated drive, drive rebuilds, concurrent reads, and concurrent writes.

 

The entire PCI Bus provides 133 MB/s of bandwidth shared between all devices for both directions. If you're reading and writing on the PCI Bus, the bandwidth is shared, so reading and writing would have to split the bandwidth.

 

The PCI-Express bus provides for bidirectional bandwidth. The bandwidth in each direction is separate. If you're reading and writing on the PCI-Express bus, then each activity would have full bandwidth available.

 

A single lane (1x) PCI-Express provides 250 MB/s of bandwidth bidirectionally.

A four lane (4x) PCI-Express provides 1000 MB/s of bandwidth bidirectionally.

 

Those are the numbers for PCI-Express 1.x slots and cards. Once PCI-Express 2.x slots and cards become more plentiful, the bandwidth will double to be 500 MB/s for 1x, and 2000 MB/s for 4x.

Thanks for explaining Brit, starting to make some sense now.

 

So I would be better off looking for a 4 port card for the PCIe x1 slot.

 

At 250Mb/s with 4 hdd that would mean around 62.5Mb/s read and 62.5Mb/s write. Isn't that fairly close to the max speed for hdd (around 80Mb/s I remember reading somewhere or am I way off)

 

For parity rebuild with the PCI slot place perhaps 3 SATA hdd in a 4 port card  (133Mb/s would mean 2 would read at 44.3 and write at 44.3 taking around 12.5 hours theoretically for 2TB) and place the smallest drives there.

 

Thanks Josh

Thanks for explaining Brit, starting to make some sense now.

 

So I would be better off looking for a 4 port card for the PCIe x1 slot.

 

At 250Mb/s with 4 hdd that would mean around 62.5Mb/s read and 62.5Mb/s write. Isn't that fairly close to the max speed for hdd (around 80Mb/s I remember reading somewhere or am I way off)

 

For parity rebuild with the PCI slot place perhaps 3 SATA hdd in a 4 port card  (133Mb/s would mean 2 would read at 44.3 and write at 44.3 taking around 12.5 hours theoretically for 2TB) and place the smallest drives there.

 

Thanks Josh

 

The problem is the availabilty of products (supported products) with the right combination of features you want.  As far as I know, there are no 4 port PCIe x1 cards.  I also don't recall seeing any 16 port PCIe x8 (or x16) cards - at least no one here has ever posted that they have one).

 

The PCI approach definitely has its limitations.  BRIT is right that it will slow things down on parity checks, drive rebuilds, and drive simulations.  But again I could argue that these situations represent a very small fraction of the time, and that using PCI drives for a modest number of drives is not a bad way to go.  There is a certain snobbery about avoiding PCI ports here (not totally undeserved), and bragging rights to those with the fastest parity check speeds.  But if you need the space and don't want to upgrade your motherboard ....

 

Look at this thread.  Read it all the way through.  WeeboTech posts that he was getting 59 MB/sec parity checks with 4 PCI drives (his MB supports a faster PCI bus, not sure if your MB runs at that speed).  Even if it does not, you could have 2-3 drives and still get reasonably good parity check speeds.  And this doesn't include the cache disk.  Put it on the PCI bus and it will have no impact on parity check speeds.

 

The alternatives?  Replace the motherboad with one that has more expandability.  Replace smaller capacity hard disks with larger capacity ones (maybe you're already full of 2T drives, but especially if you have some <= 750G drives, this is a reasonable way to go).  The latter won't get you more drives, but will get you more capacity which after all is the goal.

 

Personally, I'd get the 4 port PCI card and start adding drives one at a time.  Follow WeeboTech's advice to not have the PCI drives as consecutive disks.  (You don't want disk4, disk5, disk6, and disk7 to all be on the PCI controler.  Intersperse them so disk4, disk6, disk8 and disk10 (for example, with other disks in between) are the PCI disks.  This gives the controller a short pause between parity check accesses and speeds things up.)  Hook up your cache disk to the PCI bus, if you have one, as well as your lowest capacity disks.

 

Do some research.  This Syba controller seems to be well priced at $40 (shipped).  This promise controller is $78.  Search the forum to make sure there are no compatibilty issues before buying a specific card.

Yes, the performance limited situations represent a very small fraction of the time, but you've already realized they occur and planned for those otherwise you wouldn't be using unRAID or a parity protected array at all.

 

There might be another alternative. WeeboTech said the SuperMicro card supports Port Multipliers and didn't seem to impact performance. It was the PCI-X version of the card, but it's very probable the PCI-Express version supports Port Multipliers as well. http://lime-technology.com/forum/index.php?topic=6590.0 / http://lime-technology.com/forum/index.php?topic=4351.msg38757#msg38757

 

Unfortunately no one seems to have ventured into the 16 port PCI-Express 8x or the 4 port PCI-Express 1x controllers.

 

You can support quite a few drives without getting into performance concerns (18-19).

 

6 Ports Motherboard

8 Ports PCI-Express 16x

2 Ports PCI-Express 1x

2 Ports PCI Bus

 

The PCI Bus can certainly be used for 2 drives without feeling any or much of a real performance impact. If you stagger the drives in disk slots it might be more tolerable for 20 drives (4 on PCI Bus).

 

There's nothing wrong and there is no shame with using the PCI Bus as long as you're educated about the situation. The worst thing to have is to be unaware and disappointed.

 

The faster PCI Bus doubles the normal PCI Bus speeds and is mostly only on server-grade motherboards. So that 59 MB/s parity checks would be 1/2 of that on the normal PCI Bus for 29.5 MB/s parity checks.

 

 

Thanks guys,

 

That was the info I was after, clears up what my options are and the path to upgrading.

 

Thanks Josh

Honestly, if you're not married to that motherboard, you might want to trade it for one with two PCIe slots with 4 lanes or better.  Or plan to switch when you use up your first 14 ports.  As you've already figured out, 1-lane PCIe 1.0 is good for 2 drives, PCI is good for 1 drive.  8 drives on a 4-lane controller is pushing it but it's hard to ignore the value of that Supermicro card.  I've got one in transit right now.

 

If you just can't spend the money, I think your best bet would be to buy a second Supermicro card once you fill up the first.  Melt a notch into the back of your 1-lane PCIe slot and run the 4-lane card in the 1-lane slot.  It looks like the motherboard is free of obstructions directly behind the port so it should fit fine.  Any 4-port 1-lane card will cost you nearly as much as the supermicro but it'll be a sunk cost that can't be recycled in a new build.  When you get to the point that your performance becomes unacceptable, you can just replace the motherboard and keep everything else.  Multiport 1-lane PCIe and PCI controllers would have to go on the scrap pile with the motherboard.

 

When it's all said and done, your motherboard is good for 16-17 drives before performance starts to suffer.  That's a whole lot of space with 2tb drives.  You can start with the motherboard you've got and a Supermicro 8 port card.  When you fill those ports, get another Supermicro card and put it in the 1-lane slot with the expectation of replacing the motherboard when your performance drops off too far.  I wouldn't spend money on a 1-lane PCIe controller or a PCI controller because they'll be of no use if you have to upgrade the motherboard and they won't save any money even if you don't upgrade the motherboard.

Thanks jtown

 

That was exactly the path I ws thinking of going down. Getting a supermicro SASLP MV8 card now. When I run out I'll change to a different motherboard. The cost (especially here in Australia) of some of the PCIe and PCI SATA cards would be easier to just buy a new motherboard.

 

Thanks Josh

I agree, I would just switch out the motherboard if you don't want to have to worry about bottlenecks.  I've seen one or two with three PCIe x4 or faster slots (for example.  That means you could run three of the SuperMicro cards at 8 ports each plus the 6 ports on the motherboard for a total of 30 SATA ports with no bottlenecks (and you would only use at most 24 of those in the Norco 4224).

Have to disagree. Since you have the board, stick with it until it is time to replace it. Until then spend the money on putting in more controllers/drives until you run out of slots/ports. You can then go port multipliers or SAS expanders or replace the board. That board will let you run 20 drives.

 

6 onboard.

8 on the PCI-e x16.

4 on the PCI-e x1 (highpoint x1 cards work).

4 on the PCI (TX4 will deliver 60MB/s if the board supports 66MHz, keep to two if it doesnt).

 

20/22 drives. Without looking at SAS expander or port multipliers. Stick the cache drive on the IDE connector to free up a sata port.

I'm gonna have to go ahead and disagree with the highpoint and TX4 card advice for the reasons I've already posted.

 

Those 4 port highpoint cards cost nearly as much as the supermicro 8 port cards.  What's the point of spending the same amount for less?  You can put a 4-lane PCIe card in a 1-lane slot by notching the back of the slot.  If performance turns out to be a problem, he can then move the whole thing to a new build, using both of his supermicro 4-lane 8-port cards.  If he buys the 1-lane, 4-port highpoint card plus a 4-port PCI card, both of those cards are both useless in a new build if it turns out that he's unhappy with the performance.

 

The 4 port highpoints start around $125 in the states and that TX4 card runs around $70.  That's nearly $200 in controllers that will have to be scrapped if he's not happy with performance.  You can get the 8 port supermicro and cables for under $150.  And it will see a 4x performance increase in a 4-lane slot on a new motherboard.  So it costs less up front and it has the potential to provide superior performance in a new build.

 

There's just no financial or performanced advantage to buying a 1-lane PCIe SATA controller plus a PCI SATA controller.

 

My $0.03 (adjusted for inflation.)

Thanks guys, just an update. I've gone with upgrading the board. Part of the reason is that board doesn't have pwm fan control which I'd like for the case. Also as jtown mentioned in the long run it would be less expensive.