February 27, 201610 yr Hi all, I recently switched out my 10 year old CPU and Motherboard cos UNRAID 6 would not recognize both cores on the AMD X2 3800+ on an ASUS A8N32-SLI I switched to an Intel Core 2 Duo Quad Core (Q6600 @2.4Ghz) on an ASUS IPIBL-LA Motherboard. PROBLEM: My old ASUS A8N32-SLI had: 2 x PCI Express x16 ( SLI: full x16, x16 mode ) 1 x PCIe x4 The ASUS IPIBL-LA has: 1x PCI Express x16 1x PCI Express x1 1x PCI I have one SUPERMICRO AOC-SASLP-MV8 PCI-Express x4 and I used to use a Adaptec 1430SA Both of which require a PCI Express x4 slot. I have kept the SUPERMICRO AOC-SASLP-MV8 PCI-Express x4 on the x16 slot and have switched to an old PCI SATA controller from PROMISE which supports 4 drives. I am currently using 1 port on this but plan to add 2 SSDs to be used as CACHE POOL. QUESTION 1: Is it worth getting a PCIe x1 card that will support 4 drives? will it have more bandwidth / speed? or should I stick with PCI card. QUESTION 2: If I keep PCI card should I move the CACHE POOL to the SuperMiCRO? Thank you in advance for your help.
February 27, 201610 yr According to specs. the ASUS IPIBL-LA has PCI-E 2.0 ports. PCI-E 2.0 speed is 500MB/sec pr lane PCI speed is usually 133MB/sec
February 27, 201610 yr Wow, I haven't had to write this out for nearly 6 years, so here goes from what I can remember. The PCI bus is a sharing bus. All devices (controller cards) on the PCI Bus share a maximum bandwidth of 133 MB/s. If you have 2 drives on PCI card and are being used at the same time, each one is limited to a maximum of 66.6 MB/s. If you have 4 devices on PCI, and need to use each one at the same time, each one would be limited 33 MB/s. In reality, the device bandwidths will be lower. The situations for this are Parity Builds/Syncs, Parity Checks, Failed Drive emulation, Failed Drive Rebuilds, simultaneous reads, and simultaneous writes. It would be a nightmare in those situations taking multiple days to get your array back to a safe situation. PCI-Express is NOT a sharing bus. Each device (controller cards) on the PCI Express bus does not share bandwidth. A PCI Express 1.0 device has a limit of 250 MB/s PER LANE. A PCI Express 2.0 device has a limit of 500 MB/s PER LANE. Multiply the width of the PCI-Express by the per lane limit. A 4 lane PCI-Express 1.0 has a limit of 1000 MB/s. A 4 lane PCI-Express 2.0 has a limit of 2000 MB/s. A 8 lane PCI-Express 1.0 has a limit of 2000 MB/s. A 8 lane PCI-Express 2.0 has a limit of 4000 MB/s. A 16 lane PCI-Express 2.0 has a limit of 4000 MB/s. A 16 lane PCI-Express 2.0 has a limit of 8000 MB/s. PCI is a dead bus. I would avoid it like the plague.
February 27, 201610 yr Here is a post from 2010 explaining in more details the severe limits of the PCI bus. http://lime-technology.com/forum/index.php?topic=7520.msg72845#msg72845
February 28, 201610 yr QUESTION 1: Is it worth getting a PCIe x1 card that will support 4 drives? will it have more bandwidth / speed? or should I stick with PCI card. Absolutely -- as noted above, a PCIe v2 bus has 500MB/lane of bandwidth, so an x1 slot will give you 500MB compared to 133MB for a PCI slot -- more than 3 times the bandwidth. While a card with 4 attached drives will still be a bottleneck, that bottleneck will be 125MB/s instead of the 33MB/s of a PCI slot. QUESTION 2: If I keep PCI card should I move the CACHE POOL to the SuperMiCRO? No. Although the SuperMicro card is a 4 lane card, it's a PCIe v1 card, so even with 4 lanes it's only got 1000MB/s of bandwidth for 8 drives ... the same you'll have per/drive with a PCIe v2 card supporting 4 drives on an x1 slot. I'd just use the motherboard SATA ports for your cache pool => even though they're just SATA-1 ports, that's still 150MB/s of bandwidth. [Or use the x1 card but only plug in 2 drives to that card, which will provide 250MB/s of bandwidth for each of the drives]
February 28, 201610 yr Bottom line r.e. where to plug in your drives ... Do a bit of math If you get a PCIe x1 card that supports PCIe v2, then you'll have 500MB of bandwidth to share among the drives plugged in to that card. Assuming that card supports 4 drives, then that means you can have 500MB (1 drive), 250MB (2 drives), 167MB (3 drives), or 125MB (4 drives) of bandwidth per drive. On the SuperMicro you have 1000MB of bandwidth to share among up to 8 drives ... just divide 1000 by the number of drives you're going to plug in to the card. And your motherboard ports each provide 150MB/s of bandwidth. Note that there won't be any bottlenecks from the add-in cards in normal operation ... since you won't generally have more than one or two drives in operation at a time. The bottlenecks come into play when you're using multiple drives at once ... e.g. during a parity sync, parity check, or drive rebuild operation, or when using a cache pool.
February 28, 201610 yr Good advice above, avoid the PCI slot (or use it for a single disk only), just want to point out that the IPIBL-LA is PCIe v1.1, so take that into account when doing your calculations.
February 28, 201610 yr Good advice above, avoid the PCI slot (or use it for a single disk only), just want to point out that the IPIBL-LA is PCIe v1.1, so take that into account when doing your calculations. I hadn't actually looked this up -- assumed isvein had, since he said "... According to specs. the ASUS IPIBL-LA has PCI-E 2.0 ports ..." However, since there are competing views, I had a look -- and according to this the PCIe ports on the board are indeed v2 ports: http://www.pc-specs.com/mobo/Asus/Asus_IPIBL-LA/1414 Johnnie => Do you have another source that disagrees with that?
February 28, 201610 yr All intel Bearlake chipsets (P3x/G3x/Q3x) are PCIe 1.1 only. PCIe 2.0 on Intel chipsets first appeared on Eaglelake (P4x/G4x/Q4x), though limited to the graphics card slot. Edit: Let me just add for accuracy that highend Intel Bearlake X38/X48 also have PCIe 2.0 on the graphics slots and Intel Eaglelake G41 is PCIe 1.1 only.
February 28, 201610 yr daytona235 => Given that the PCIe slots are NOT v2 slots, the math changes just a bit ... (a) Doesn't change anything for the SuperMicro board ... since it's a v1 board, I had already shown the v1 bandwidth for it [e.g. 1000MB/s total is what's available to share among the 8 ports] (b) the motherboard SATA ports aren't impacted ... they still SATA-I, so they each have 150MB/s bandwidth © the PCIe x1 slot will only have 250MB/s to share, NOT the 500MB that a v2 spec'd board would provide. So the available bandwidth per port will be half of what I noted earlier when you're using multiple drives. Bottom line: You can do the math to optimize the available per/drive bandwidth -- but to keep it simple with the configuration you've indicated, then IF your goal is to minimize the bottlenecks for operations when all drives are in use, I'd try and limit the # of drives on your add-in cards to a number that doesn't drop the bandwidth below the on-board SATA ports (unless that doesn't provide all the ports you need). i.e. connect 6 drives to the SuperMicro; 1 to the PCIe x1 card; and 6 to the motherboard ports. HOWEVER, if the goal is to maximize the bandwidth for your drives during "normal" operations and not worry about the parity check and drive rebuild speeds, then you'll want to do it a bit differently: i.e. you'll want to connect your 2 SSDs to the SuperMicro card, and your parity drive to the PCIe x1 card as the only drive on that card.
February 28, 201610 yr Author Wow guys - THANK YOU! Thank you for all the info and advice. Sounds like I should get a PCIe x1 SATA controller that supports 4 drives (I need only 3) As the system is configured now - I need 3 more drives supported in the system the 2 cache pool + 1 storage. Data drives: 14 Parity: 1 Cache Pool: 2 (SSD) Total: 17 drives Would this be a good card to get? IO Crest 4 Port SATA III PCI-e 2.0 x1 Controller Card Marvell Non-Raid with Low Profile Bracket SI-PEX40064 ($30) http://www.amazon.com/IO-Crest-Controller-Non-Raid-SI-PEX40064/dp/B00AZ9T3OU/ref=sr_1_2?ie=UTF8&qid=1456663147&sr=8-2&keywords=x1+sata Chipset: Marvell 88SE9215 Compliant with PCI-Express Specification v2.0 and backwards compatible with PCIex1 Compliant with Serial ATA Specification 3.0 Supports Communication Speeds of 6.0 Gbps, 3.0 Gbps, and 1.5 Gbps Hot Plug and Hot Swap Supports Port Multiplier FIS Based Switching or Command Based Switching 4 Internal SATA 6Gb/s Ports OR IO Crest 4 Port SATA III PCIe 2.0 x 2 Controller Card Components SI-PEX40062, Green ($37) Chipset: Marvell 88SE9235 Installed with a full profile bracket, low profile included with the package Port Multiplier FIS-based and Command-based switching supported Supports SATA Rev 3.0 transfer speeds up to 6.0Gbps and backwards compatible with SATA I/II at 1.5/3.0Gbps Supports Native Command Queue (NCQ) 4 Internal SATA 6Gb/s Ports Compatible with SATA 6G, 3G and 1.5G Hard Drives PCI-Express x2 Interface is Compatible with PCI-Express x4, x8, and x16 slots Any other recommendations for PCIe X1 v1.0 card(s)? Summary (please correct me) MB SATA1 ports: 6 drives = 150MB per drive [ put PARITY - CACHE POOL here?] The SuperMicro board (1000MB): 8 drives = 125MB per drive Question: How best to install 3 more drives? Option1: 1 PCIe V2 (x1) card that supports 4 devices = 1000/4 = 250MB per drive (did I do that math correctly?) 2 PCIe V2 (x1) cards that support 2 devices each = 1000/2 = 500MB per drive If this works then I should move the Parity Drive to the new cards I would purchase. I guess one other thing I could do is just consolidate my old 500gb drives into one 3TB drive and then I don't have any problems (but at some point I would want to support all the drives my 3 (5x3) cages can hold) Note: in current config Parity check completed in: Duration: 12 hours, 37 minutes, 1 second. Average speed: 66.1 MB/s
February 28, 201610 yr Author One other question. If I were to say go mad and wanna spend more $$$ which motherboard + controllers would allow me to setup 1 - Parity 14 - Data 2 - Cache Pool SSD with the most optimal bandwidth allocation? ps. mad cos - really this system as it is serving my needs very well. 1 - Serve music up to my sonos systems 2 - Plex to two TVs (rarely are they both used) (Docker) 3 - Backup laptops 4 - Setup one Virtual machine running windows (Not setup yet)
February 28, 201610 yr Don’t forget that your board is PCIe v1.1, this is how I'd do it: SASLP – 6 Disks + Cache pool Onboard – Parity + 5 Disks PCIe 1x controller #1 – 2 Disks PCIe 1x controller #2 – 1 Disk (or use your current PCI card for it) For PCIe 1x controller I would recommend something asmedia 1601 based, e.g., IOCrest SY-PEX40039. With this config your max bandwidth during a parity check should be ~90MB/s.
February 28, 201610 yr Question: How best to install 3 more drives? Option1: 1 PCIe V2 (x1) card that supports 4 devices = 1000/4 = 250MB per drive (did I do that math correctly?) 2 PCIe V2 (x1) cards that support 2 devices each = 1000/2 = 500MB per drive No -- these are both x1 cards, so they only have ONE PCIe lane to use = 250MB/s total bandwidth, since your motherboard uses PCIe v1 (doesn't matter that the cards support v2). If you install 2 drives on that card that's 125MB/drive; if you install 3 that's only 83MB/drive. But if you have two x1 slots available, you could indeed use 2 cards, and then you'd get 125MB/drive bandwidth for 2 connected drives.
February 28, 201610 yr Note: As Jonnie already noted, you can always keep your PCI card and just connect one drive to it -- that will give you 133MB/s for that drive.
February 28, 201610 yr Author Ok folks - THANK YOU for all the help think this is the way forward. SASLP – 6 Disks + Cache pool (2 drives) Onboard – Parity + 5 Disks PCIe 1x controller #1 – 2 Disks PCI card - 1 Disk (shall move one of my 500gb drives to this) IO Crest 2 Port SATA III PCI-Express x1 Card (SY-PEX40039) - $15 http://www.amazon.com/IO-Crest-Port-PCI-Express-SY-PEX40039/dp/B005B0A6ZS/ref=sr_1_1?ie=UTF8&qid=1456677451&sr=8-1&keywords=IOCrest+SY-PEX40039 ASM1061 Chipset (Asmedia 1061 SATA Host Controller) Supports Hot Plug and Hot Swap Supports Communication Speeds of 6.0Gbps, 3.0Gbps, and 1.5Gbps Supports 2 Ports Serial ATA, Native Command Queue (NCQ), Port Multiplier Works with SSDs
February 28, 201610 yr Given your hardware, that's a good set of choices, and only requires a $15 purchase.
March 6, 201610 yr Author Latest time for parity check: Duration: 19 hours, 58 seconds. Average speed: 43.8 MB/s Previous with two drives on PCI card and no Cache Pool and no PCIe 1x card: Duration: 12 hours, 37 minutes, 1 second. Average speed: 66.1 MB/s SASLP – 6 Disks + Cache pool (2 drives) Onboard – Parity + 5 Disks PCIe 1x controller #1 – 2 Disks PCI card - 1 Disk Why did it slow down? The Cache Pool is NOT involved in the parity check right?
March 6, 201610 yr The Cache Pool is NOT involved in the parity check right? No, is that and the new PCIe card the only things that changed between checks? It doesn’t make much sense.
March 6, 201610 yr Note that as Johnnie had indicated your PCIe slots are only v1.1, so the PCIe card is sharing 250MB/s of bandwidth -- not the 500MB/s that I had earlier suggested. That means those drives are limited to 125MB/s each -- which is LESS than the drive on the PCI card has available (although not by much -- 133MB/s). You can confirm whether or not this is causing your issue by moving your cache pool drives to the PCIe card and moving the two drives on it back to the SASLP, then doing a parity check.
Archived
This topic is now archived and is closed to further replies.