bnevets27 Posted August 13, 2016 Share Posted August 13, 2016 Firstly I would like to thank all the info provided by johnnie.black, any reference to speeds are taken from his real world tests post here. I'm still trying to wrap my head around the different configurations on how to hook up 24+ drives. I've also noticed a lot of question around using expanders. I've made some terrible diagrams in paint to help visually. I've been hung up on the different ways to connect the controllers to the sas exapnders, and what max speed the resulting drives will have available to them. I'm going to try to have the questions and hopefully the answers universal so it's helpful to all. These examples are how to connect 24 drives. Of course there is also on board ports/controllers, some people may want to add a couple SSDs for cache and increasing the needed amount of ports. Everyone will have their own optimal configuration. Hopefully we can have some discussion about different configurations in this thread. Some info on the diagrams bellow: Each black line is a cable. 8 Port controller is actually a 2 port controller with 4 channels/lanes per port for a total of 8 channels/lanes per card. This can be a Dell Perc H310 or IBM M1015 for example. There are others but these seem to be the most popular and been used with the expanders before so they are compatible. Connection from the expander or controller (if not using expander) to the hard drives will use a SFF8087 to SATA (forward breakout) cable, therefore allowing 4 drives per expander/controller port. Connection from controller to expander uses SFF8087 to SFF8087 cable Controller to backplane with built in expander would also use the same SFF8087 to SFF8087 cable 2 Controllers and 2 Intel RAID SAS Expander RES2SV240 (Shown in Dual Link) 1 Controller and 1 HP 6Gb (3Gb SATA) SAS Expander (Dual Link) 3 Controllers, Direct Connect 1 Controller 2 Intel RAID SAS Expander RES2SV240 Expected but not tested speeds from jonnie.black using a PCIe 2.0 HBA the bottleneck is the PCIe bus, max speed ~110/125MB/s using a PCIe 3.0 HBA the bottleneck are the SAS2 links, 2200 * 2 / 24 = 185MB/s The M1015 and H310 are both PCIe 2.0 x8 cards 2 Controllers 1 HP 6Gb (3Gb SATA) SAS Expander - Not Possible This configuration is not possible, you can't connect the same expander to more than one controller. - jonnie.black Some more info General Expander info. General Expander info 2 HP vs Intel. Intel RES2SV240 Wiki Some info on the HP 6Gb (3Gb SATA) SAS Expander. It does require a PCI-E x4 port for power only. I do suspect that you could use one of following type of adapters but I haven't seen anyone confirm that it does work. Type 1 Type 2 Type 3 Type 4 Now I know some cases have expanders built in, I don't have one (yet) so I don't know a lot about them yet, as far as dual link, single link, speed etc. I would like to add some info/discussion on disk speed and what is considered too much of a bottle neck for current drives, future drives and how much the speed decrease will effect unraid usage. With most people using gigabit lan which has a theoretical max of 125 MB/s and a real world max close to that. Any disk speed over 125MB/s won't make any difference writing to the array. Only benefit of speeds over 125MB/s is decreased time for parity checks. Will add more info as I find it and hopefully some of the smarter people start to chime in. 2 1 Quote Link to comment
bnevets27 Posted August 13, 2016 Author Share Posted August 13, 2016 (edited) Configurations listed in order of speed 3 Controllers, Direct Connect - 320MB/s - 3 PCIe x8 slots 2 Controllers and 2 Intel Expanders (dual link) - 205MB/s - 2 PCIe x8 slots 2 Controllers and 2 HP Expanders (dual link) - 182MB/s - 2 PCIe x8 slots 2 Controllers and 1 Expander, HP or Intel - 16 drives connected through expander (dual link) - 140MB/s - 2 PCIe x8 slots - 8 drives via breakout cable - 320MB/s 1 Controller and 2 Intel Expanders - 110/125MB/s - 1 PCIe x8 slots 1 Controllers and 1 Intel Expander - 20 drives connected through expander (single link) - 110MB/s - 1 PCIe x8 slots - 4 drives via breakout cable - 320MB/s 1 Controller and 1 HP Expander (dual link) - 95MB/s - 1 PCIe x8 slot and 1 PCIe x4 slots for power but could possibly be powered externally/from adapter Possible bottlenecks: Expander PCIe bus DMI bus Maximum Theoretical Speed vs "Real World" PCIe x8 2.0 4000MB/s maximum theoretical speed, johnnie.black tests found that the max "real world" speed is between 2600 and 3000MB/s. 4port SAS2 link 2400MB/s (4x 600MB/s) max theoretical bandwidth real world speed is ~2200MB/s The IBM M1015 is PCIe 2.0 The Dell H310 is PCIe 2.0 Intel RAID SAS Expander RES2SV240 (4000MB/s max theoretical speed, 2600 and 3000MB/s real world) HP 3Gb SATA (6Gb SAS) Expander (2400MB/s max theoretical speed, 2200MB/s real world)(running SATA, didn't include SAS speed due to little chance anyone here will use SAS) 3 Controllers, Direct Connect - 320MB/s - No PCIe or expander bottleneck possible DMI bus bottleneck 2 Controllers and 2 Intel Expanders - 205MB/s - PCIe 2.0 bus is the bottleneck (12 drives on one PCIe slot) 1 Controller and 2 Intel Expanders - 110/125MB/s - PCIe 2.0 bus is the bottleneck (24 drives on one PCIe slot) 1 Controller and 1 HP Expander - 95MB/s - Expander is the bottleneck So a single card with 2 4port SAS2 links would be able to provide connection of 24 drives at 185MB/s with the use of an expander if no bottlenecks are present. Each of the of the ports of the HBA will need to connect to 12 drives via an expander. The HBA would need a total 4400MB/s of bandwidth to run at max speeds. But the PCIe 2.0 x8 bus max speed is between: 2600MB/s /24 = 108MB/s per disk 3000MB/s /24 = 125MB/s per disk So the max speed per disk on a single PCIe x8 2.0 slot is 108-125MB/s On 8/5/2016 at 4:18 AM, johnnie.black said: A 4port SAS2 link has 2400MB/s (4x 600MB/s) max theoretical bandwidth, in my tests I've found real world speed is ~2200MB/s, so: 2200 / 12 = 185MB/s Quote You main bottleneck is DMI, which in your board is first gen, up to 1000MB/s, all 6 onboard ports + how many drives are in the bottom pcie x16(x4) controller have to share it, still should give you more than 15MB/s, even with 14 disks going through it it should be roughly 55Mb/s, you may be needing some tuning. Speed won't improve by replacing the controllers, the most you can do with that board is to connect only 6 to the bottom controller plus the 6 onboard, with some tuning rebuilds should start at ~65MB/s. The SASLP with no other bottlenecks is not the fastest card but it's capable of 80MB/s fully loaded. Edited October 3, 2018 by bnevets27 1 Quote Link to comment
bnevets27 Posted August 13, 2016 Author Share Posted August 13, 2016 Make and model of all suitable LSI2008 chipset HBA's LSI 9211-8i IBM M1015 Dell Perc H310 Dell Perc H200 Please respond with any other variations that are possible. Any more suitable expanders would also be helpful. DMI bus bottleneck This will be very different for each machine. No one size fits all here. But best to refrence your motherboard manual and johnnie.blacks post here. Quote Link to comment
JorgeB Posted August 13, 2016 Share Posted August 13, 2016 1 Controller 2 Intel RAID SAS Expander RES2SV240 Never tested as I only have 1 Intel expander but I would expect these speeds: using a PCIe 2.0 HBA the bottleneck is the PCIe bus, max speed ~110/125MB/s using a PCIe 3.0 HBA the bottleneck are the SAS2 links, 2200 * 2 / 24 = 185MB/s 2 Controllers 1 HP 6Gb (3Gb SATA) SAS Expander This configuration is not possible, you can't connect the same expander to more than one controller. Quote Link to comment
makarai Posted May 27, 2019 Share Posted May 27, 2019 (edited) Hello, I am trying to figure out how to get the most out of my 14 HDD drives, I do however have a 24HDD suitable case. However, how would i connect 24 drives to get the most out of it. When i use regular HDDs (Wd reds) is the HBA or the Expander the limiting factor? Would this be possible ? And what would the speeds be? Is there a good 16port controller out there so I could just skip the expander? Edited May 28, 2019 by makarai Quote Link to comment
makarai Posted May 28, 2019 Share Posted May 28, 2019 @johnnie.black what would you recommend? Quote Link to comment
JorgeB Posted May 28, 2019 Share Posted May 28, 2019 17 hours ago, makarai said: Is there a good 16port controller out there so I could just skip the expander? LSI 9300-16i, though an 8 port HBA + expander might be cheaper on ebay. Quote Link to comment
makarai Posted May 28, 2019 Share Posted May 28, 2019 @johnnie.black so it makes sense to go for the 12gb adapters tho, mostlikely i'll only use hdds? Something like 9201-16i + another 8i or expander wont do? Quote Link to comment
JorgeB Posted May 28, 2019 Share Posted May 28, 2019 31 minutes ago, makarai said: to go for the 12gb adapters tho, mostlikely i'll only use hdds? It does for 16 ports if you don't want to risk a bottleneck, not because it's 6gbps but because it's PCIe 2.0, though for WD Reds up to 6TB it shouldn't be much of one, but it can be for faster disks. Quote Link to comment
squirrelslikenuts Posted January 12, 2020 Share Posted January 12, 2020 On 8/13/2016 at 2:58 AM, johnnie.black said: Never tested as I only have 1 Intel expander but I would expect these speeds: using a PCIe 2.0 HBA the bottleneck is the PCIe bus, max speed ~110/125MB/s I am using a PCIe 3.0 HBA on a PCIe 2.0 server, connected to a SAS1 expander (hp dl180 g6). I can speed test at ~205-210 MB/s on a single drive, but transferring disk to to disk its limited to 85-90 MB/s. Does this make sense? Quote Link to comment
JorgeB Posted January 12, 2020 Share Posted January 12, 2020 6 hours ago, squirrelslikenuts said: Does this make sense? Not enough info, is this array disk to array disk? With parity? Turbo write enable? Quote Link to comment
squirrelslikenuts Posted January 12, 2020 Share Posted January 12, 2020 (edited) 8 hours ago, johnnie.black said: Not enough info, is this array disk to array disk? With parity? Turbo write enable? Array disk to array disk 8TB Red to 8TB Red No parity Turbo write enabled HP H220 HBA (SAS2308) -> SINGLE SFF-8087 Cable -> HP 12 Drive Backplane with integrated expander. Interestingly - While using unBALANCE to move files from drive to drive - at ~81 MB/s (lots of smaller files) I am able to speed test one of the other drives in the array at 200+ MB/s with only slight impact on performance. Slight speed hit to the drive being tested, but the 2 drives that are doing a transfer aren't really affected in performance. Edit: My model shows this in the specs 12HDD Models HP Smart Array P212/256MB Controller (RAID 0/1/1+0/5/5+0) NOTE: Available upgrades: P410 with FBWC, 256MB with BBWC, 512MB with FBWC, Battery kit upgrade (for the 256MB cache), and Smart Array Advanced Pack (SAAP). NOTE: Support transfer rate up to 3Gb/s SAS or 3Gb/s SATA I am using an HP H220 HBA capable of 6Gb/s sas or 3Gb/s SATA - but It appears the 12 drive backplane will only negotiate 3gb/s SAS. The P212 supports 6Gb/s SAS so I assume its the expander/backplane that does not. Edit2: hba connected at single link not dual Edited January 12, 2020 by squirrelslikenuts Quote Link to comment
JorgeB Posted January 13, 2020 Share Posted January 13, 2020 Turbo write doesn't make any difference without parity, what Unraid version are you using? V6.7.x has known performance issues. Quote Link to comment
UhClem Posted January 13, 2020 Share Posted January 13, 2020 On 1/12/2020 at 11:54 AM, squirrelslikenuts said: ... While using unBALANCE to move files from drive to drive - at ~81 MB/s (lots of smaller files) ... Bad way to test for bandwidth limits/bottlenecks. ===== Fud's First Law: "If you push something hard enough, it will fall over." Quote Link to comment
cbr600ds2 Posted September 20, 2020 Share Posted September 20, 2020 I didn't see the answer but you seem to be using the SAS expanders in PCIe slots. Would speed change if you powered via molex and not plugged in? I'm guessing you'd be limited by the HBA card and the HDD or am I way off? Quote Link to comment
JorgeB Posted September 21, 2020 Share Posted September 21, 2020 14 hours ago, cbr600ds2 said: Would speed change if you powered via molex and not plugged in? No, some expanders can be powered either way, it won't affect performance. 1 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.