At the moment I have 3 M1015s on my X9SCM motherboard an I like to free up at least one PCI-E slot.

 

What would you recommend? Get some 12 port controllers? Which? Are there 12 port M1015 alikes that work in unraid?

 

or get a SAS expander?

 

What option would give best parity check speeds?

http://lime-technology.com/forum/index.php?topic=32991.msg303634#msg303634

At the moment I have 3 M1015s on my X9SCM motherboard an I like to free up at least one PCI-E slot.

 

What would you recommend? Get some 12 port controllers? Which? Are there 12 port M1015 alikes that work in unraid?

 

or get a SAS expander?

 

What option would give best parity check speeds?

 

i had a similar situation when my x9SCM-iif with 4 PCIe slots are full of cards.. you have some options IMHO:

- replace your board with X9SCM-iif - you will get one additional slot

- get expander - but you have so many hdd, if you add expander, may be you will see some performance degrade when all disks are up(parity check) but it depends how you connect it.

- get new dual cpu board - this was my route.. see my sig for model numbers - got it all used from ebay.. now i have 7 PCIe slots + one SAS 2008 built-in controller. on negative side of this route is power usage - dual CPU uses much more power, generates much more heat..

I might consider going to larger spindles.

Upgrade smallest spindles and sell them off.

 

If that's not a viable option, Areca high count controllers or SAS expanders.

SAS expanders might be the best option if running under ESX.

Something to think about if you go with a SAS expander:  I use them and like them but I see a 25% speed reduction on Parity Check/Sync operations where every drive is being accessed at the same time.  I have up to 24 drives off of 1 M1015 controller so that I can use my other PCIe slots for other devices passed through to VMs in ESXi.

One forum member is using a 24-port Highpoint 2760A and it works very well:

http://www.newegg.com/Product/Product.aspx?Item=N82E16816115095

 

You'd have to CAREFULLY "Dremel" the plastic end piece off of one of your x8 slots to use an x16 card (I'm assuming they're not already open-ended) ... and of course the card would only run at x8 speed, but even with only 8 lanes you'd still have 167MB/s bandwidth for each drive.    That's a bit of a bottleneck on the outer cylinders with modern drives that have 1TB (or higher) platters, but not with drives using smaller platters.  And it's certainly better than using extenders.

 

If you only need to free up one slot, the x8 card suggested by c3 above will work without any motherboard modifications, and will have plenty of bandwidth for even the highest density drives you can get.

 

Expanders are a less expensive alternative to free up some slots, but as already noted these will result in significantly slower speeds when all drives are active at once (i.e. parity checks & rebuilds).

 

One forum member is using a 24-port Highpoint 2760A and it works very well:

http://www.newegg.com/Product/Product.aspx?Item=N82E16816115095

The only thing I would worry about on this is if it still uses Marvel chips to control it and V6 of unRAID.  Until the Marvel slow down problem is fixed a SAS expander actually becomes the better SHORT TERM solution.  Long Term I would rather use something like this myself.  It also looks like said problem doesn't occur in every instance so that is good to know as well.

Pauven is the user I know about that uses the Highpoint.    To my knowledge he hasn't had any issues with v6, but you may want to ask him -- in fact I'll PM him and ask him to comment in this thread.

 

Hey guys,

 

The 24-port 2760A still runs like a champ under 6.0.  No issues here.

 

I might be mistaken, but I thought you could always plug a 16x card into 8x and 4x slots, and it automatically connects at the appropriate number of lanes.  Not sure I've ever been in that situation myself.

 

What is the Marvel slowdown that is referenced?  If I know some details I'll be happy to check.

 

Unless the slowdown is in reference to the need to tweak the unRAID tunables, for which I even wrote a tool to find the right settings.  I had massive slowness, and Tom told me to tweak the settings.  Since there was no guidance on how to set them, I wrote a tool to test them all!  The tweaked settings still work great under 6.0.

 

Haven't tried 6.1 yet...

 

Paul

... I might be mistaken, but I thought you could always plug a 16x card into 8x and 4x slots, and it automatically connects at the appropriate number of lanes.  Not sure I've ever been in that situation myself.

 

You can -- as long as it physically fits.  The problem is that unless the smaller slots are "open ended" (i.e. not closed at the back) the card won't fit into the slot.    It's an easy thing to fix -- just CAREFULLY cut away the bit of plastic at the end of the slot (a Dremel tool works well for that).    But electrically there's no issue -- the x16 card will simply run at x4 or x8 speed, depending on the slot.      Note that a some motherboards use open-ended connections on the smaller slots -- but a lot don't.

 

You're absolutely right.  I just read up on PCIe on Wikipedia.  They show a picture of an open ended slot, and mentioned physical modification or riser cards as possible solutions for allowing a longer card to fit into a physically smaller slot.

 

Seems silly that open ended slots, or a physical design with larger gaps on the card to fit over the ends, wasn't made standard.  While certainly not optimal to run a card at below design speed, there are times when sometimes (i.e. troubleshooting) you just need to get something plugged in and working, even if it is slower than intended.

 

It's also quite common for an 8x PCIe slot to be physically a 16x size with only half the wire connections.  Since the physical 8x slots are not as commonly used, sometimes it's cheaper for the manufacturer to use a 16x slot wired for 8x.

 

I should also mention that there's a 16-port version of the 2460A (2440A?, 1660A, can't remember).  Since you're only looking to free up one slot, this will do it for less money, and since you are limited to 8x, you'll have more bandwidth per drive with only 16 drives instead of 24.  Might be the perfect compromise.

 

It's also interesting to note that the 2460A is simply 3 cards in one, combined with a PCIe hub.  It actually shows up as three separate cards in the system.  The 16-port version is simply 2 cards in one.

Seems silly that open ended slots, or a physical design with larger gaps on the card to fit over the ends, wasn't made standard.

If the board layout designer is mindful of the possibility of something occupying that space, everything works out ok with open ended slots. If the decision is made to use that space for headers or components, then even if you open the end of the slot, the board still isn't going to fit, thus closed slots. Often times the same board layout will be used with multiple models, and the added functionality of extra components on some models means that some boards have the room, others don't, but since the possibility exists for the slot to be blocked, they use closed slots even though they could have used open ended.

 

Also, this is pure speculation, but the mechanical pinching required to ensure good contact may mean an open ended slot is more expensive because of the lack of an end cap keeping the sides a fixed distance apart means more engineering to make sure it won't spread over time.

The easiest and least expensive option for me would be getting an expander.

 

Am I right that the Intel expander can be mounted in an open PCI slot on the back of the case and powered through a Molex power connector?

 

Does it make much difference, speedwise, if it is connected through 1 or 2 8087 cables to a M1015?

 

When connecting through one 8087 cable I can connect 20 disks to the expander and 4 disks to the M1015, right? Or am I overseeing something?

 

Or perhaps getthis card?

 

http://www.ebay.de/itm/LSI-Logic-SAS-9201-16i-16Port-6Gb-s-SAS-SATA-Single-Controller-Card/151814474161?_trksid=p2045573.c100033.m2042&_trkparms=aid%3D111001%26algo%3DREC.SEED%26ao%3D1%26asc%3D20131017132637%26meid%3D0a122dee89b2497bb0b0a7b5b0db1cfd%26pid%3D100033%26rk%3D3%26rkt%3D4%26sd%3D141703264505

The easiest and least expensive option for me would be getting an expander.

 

Am I right that the Intel expander can be mounted in an open PCI slot on the back of the case and powered through a Molex power connector?

 

Does it make much difference, speedwise, if it is connected through 1 or 2 8087 cables to a M1015?

 

When connecting through one 8087 cable I can connect 20 disks to the expander and 4 disks to the M1015, right? Or am I overseeing something?

 

Or perhaps getthis card?

 

http://www.ebay.de/itm/LSI-Logic-SAS-9201-16i-16Port-6Gb-s-SAS-SATA-Single-Controller-Card/151814474161?_trksid=p2045573.c100033.m2042&_trkparms=aid%3D111001%26algo%3DREC.SEED%26ao%3D1%26asc%3D20131017132637%26meid%3D0a122dee89b2497bb0b0a7b5b0db1cfd%26pid%3D100033%26rk%3D3%26rkt%3D4%26sd%3D141703264505

I have mine connect up with a single cable to my M1015.  Doing this allows me to get 4 drives connected directly to the M1015 and 20 connected to the RES2SV240 expander (I.E. 24 drives off of ONE M1015).  When I tested I had 16 drives connected so could test with the expander and without but with a second M1015 and still have all drives connected.  The drives were ST3000DM01 and in my testing I got about a 25% reduction in speed when using the RES2SV240 over using 2 M1015s.

What is the Marvel slowdown that is referenced?  If I know some details I'll be happy to check.

This thread and a few others like it.  http://lime-technology.com/forum/index.php?topic=42629.0

Better examples here http://lime-technology.com/forum/index.php?topic=39125.msg386462#msg386462

 

Found on the SAS2LP-MV8 which uses a Marvel controller.

... When connecting through one 8087 cable I can connect 20 disks to the expander and 4 disks to the M1015, right?

 

Yes, that's correct => and the performance of the array in normal use will be just fine.    You'll just get appreciably lower performance with parity checks and rebuilds, since you're effectively connecting 5 disks to each SATA port (and sharing the bandwidth between them).  But with 6Gb ports, that still leaves ~ 120MB/disk of bandwidth, which isn't "bad" at all.

 

 

The easiest and least expensive option for me would be getting an expander.

 

Am I right that the Intel expander can be mounted in an open PCI slot on the back of the case and powered through a Molex power connector?

 

Does it make much difference, speedwise, if it is connected through 1 or 2 8087 cables to a M1015?

 

When connecting through one 8087 cable I can connect 20 disks to the expander and 4 disks to the M1015, right? Or am I overseeing something?

 

Or perhaps getthis card?

.....

 

 

It's my understanding that if you connect 2 cables to the expander you'll have increased bandwidth and may see minimal degradation.

I would keep the parity and cache off the expander if possible.

 

Given the suggested LSI card and the ability to have 16 devices and 512 with expanders.

I might go that way depending on cost and maximum drives intended to support.

unRAID can only support 26 drives at the current time.

You can always add expanders to the LSI card if you need to grow larger.

In contrast, adding the expander today with 2 cables should satisfy today's and tomorrow's needs.

 

It all depends on costs vs the number of intended spindles to support.

In addition, how you may consider doing virtualization and passing through controllers.

 

SAS and SAS expanders have a benefit of also having two host connections. Thus being able to share the bus between hosts.

One huge disk array could have half the disks on one host and have some on the other.

From what I've read, SATA disks can only be allocated to one host, but SAS disks can be shared.

I've never done this, but I've read about it. Cool stuff if doing virtualization.

 

http://www.sasexpanders.com/