October 31, 201510 yr Hi everyone, I'm so confused about HBA, RAID, SAS Expanders, IR/IT Mode, option ROM. I've done many hours of reading but still lost. For example, the Intel RAID Twenty-four port Expander Card RES2SV240 has 6 x SFF-8087 ports, supporting 24 drives. That makes complete sense since each SFF-8087 cable can support 4 SAS/SATA drives. Is this considered a RAID card or just an HBA card or SAS expander? But then the LSI 9300 MegaRAID SAS 9361-8i (LSI00416) has 2 x SFF-8643 ports, supporting 128 drives. How is this possible? How can 2 SFF-8643 ports possibly connect to 128 drives? I'm also confused about SAS expanders, especially when they are talked about in a backplane vs a controller card. For example, the X-case 424 Pro Rev 2 with SAS expander has a single port to connect to 24 drives. How is this possible and how is there not a bottleneck? Am I also correct in that these controller cards come in IR (integrated RAID) mode by default and must be flashed to IT mode? I don't even know how to ask about option ROM. If someone could please explain some of the above in layman terms or point to another thread that already has good explanations, that would be most appreciated! Thank you.
October 31, 201510 yr Where are you seeing the lsi 9361-8i supports 128 drives? It only supports 8 drives. The Intel 24 port expander card is an expander. Expander cards do create a bottleneck as you are squeezing more data through the the same number of avaliable SAS cables. They are the SAS equivalent of a sat port multiplier.
October 31, 201510 yr The card would support 128 drives through using expanders. Without expanders, it would only support 8
October 31, 201510 yr Author Where are you seeing the lsi 9361-8i supports 128 drives? It only supports 8 drives. On the specifications page of the Newegg link I posted it says "Connect up to 128 3Gb/s, 6Gb/s, or 12Gb/s SATA and SAS devices" The card would support 128 drives through using expanders. Without expanders, it would only support 8 Ok, so do you mean you would connect the LSI 9300 MegaRAID SAS 9361-8i to SAS expanders such as the Intel RAID Twenty-four port Expander Card RES2SV240 I mentioned? I'm having a hard time understand why anyone would do that, why not just connect the Intel 24 port expander directly to the motherboard.
October 31, 201510 yr The card would support 128 drives through using expanders. Without expanders, it would only support 8 Ok, so do you mean you would connect the LSI 9300 MegaRAID SAS 9361-8i to SAS expanders such as the Intel RAID Twenty-four port Expander Card RES2SV240 I mentioned? I'm having a hard time understand why anyone would do that, why not just connect the Intel 24 port expander directly to the motherboard. Same thing (If you motherboard has SAS connectors)
October 31, 201510 yr Because the SAS card or motherboard controller needs to also support the use of SAS expander. The SAS controller and expander also need to be compatible with eachother. If you are looking to get 24 drives with no bottleneck I would suggest 3 LSI 9211-8i cards it would be cheaper than the LSI card and Intel expander that you have listed. To flash to IT mode call LSI tech support they will walk you through the procedure over the phone, they did for me.
October 31, 201510 yr Author Sorry folks, I'm just not getting it. Let me state what I think I know and I think it might help you experienced guys point out the flaw in my logic. SAS Expander = HBA controller = SAS/SATA controller = Port multipler (it's all the same thing right?) So a SAS expander connects to a motherboard via PCIe. The SAS expander has SFF-8087 ports, each of which can connect to 4 drives. So a SAS expander with 6 SFF-8087 ports can connect 24 drives. Correct so far? Intel RAID Twenty-four port Expander Card RES2SV240 is considered a SAS Expander as well as a RAID card because it has hardware RAID built into it. Right? LSI 9300 MegaRAID SAS 9361-8i (LSI00416) has 2 x SFF-8643 ports. So these 2 SFF-8643 ports can connect to 2 different expanders, and let's say each of those SAS expanders can connect 24 drives. Then the LSI 9300 MegaRAID has now connected to 48 drives. What's the point of the LSI 9300 MegaRAID card? Why not just connect the two expanders to the motherboard directly? Same result without the middleman (LSI 9300 MegaRAID card).
October 31, 201510 yr Sorry folks, I'm just not getting it. Let me state what I think I know and I think it might help you experienced guys point out the flaw in my logic. SAS Expander = HBA controller = SAS/SATA controller = Port multipler (it's all the same thing right?) So a SAS expander connects to a motherboard via PCIe. The SAS expander has SFF-8087 ports, each of which can connect to 4 drives. So a SAS expander with 6 SFF-8087 ports can connect 24 drives. Correct so far? Intel RAID Twenty-four port Expander Card RES2SV240 is considered a SAS Expander as well as a RAID card because it has hardware RAID built into it. Right? LSI 9300 MegaRAID SAS 9361-8i (LSI00416) has 2 x SFF-8643 ports. So these 2 SFF-8643 ports can connect to 2 different expanders, and let's say each of those SAS expanders can connect 24 drives. Then the LSI 9300 MegaRAID has now connected to 48 drives. What's the point of the LSI 9300 MegaRAID card? Why not just connect the two expanders to the motherboard directly? Same result without the middleman (LSI 9300 MegaRAID card). A SAS Expander needs a SAS Controller/RAID card to work, so it is not a raid card it is used in conjunction with a RAID or HBA card to get more drives with one raid/HBA card. Typicaly you get bottlenecks with this setup if accessing more than 4 drives on the expander at the same time, you will see slower parity checks. A HBA controller is similar to a RAID card but a RAID card typical has more sophisticated RAID like LSI's megaRAID cards. HBA and RAID cards are both SAS/SATA controllers. With unRAID you want and HBA card because they are cheaper in general and you don't use the RAID features on either HBA or RAID cards with unRAID A Port multiplier is a where you take a SATA port and hook it up to multiple drives, I just used that as an example of what a SAS expander is similar to but not the same. What are you trying to accomplish? If you give us a idea of what you are doing we can give you more targeted answers instead of just general ones.
October 31, 201510 yr Author A SAS Expander needs a SAS Controller/RAID card to work, so it is not a raid card it is used in conjunction with a RAID or HBA card to get more drives with one raid/HBA card. Typicaly you get bottlenecks with this setup if accessing more than 4 drives on the expander at the same time, you will see slower parity checks. A HBA controller is similar to a RAID card but a RAID card typical has more sophisticated RAID like LSI's megaRAID cards. HBA and RAID cards are both SAS/SATA controllers. With unRAID you want and HBA card because they are cheaper in general and you don't use the RAID features on either HBA or RAID cards with unRAID A Port multiplier is a where you take a SATA port and hook it up to multiple drives, I just used that as an example of what a SAS expander is similar to but not the same. This helps clear a lot of confusion. Understanding the definition of a HBA controller, RAID card, SAS expander, port multiplier seems basic but was very confusing to me. Thank you. I still don't understand how the LSI 9300 MegaRAID SAS 9361-8i (LSI00416) can connect to 128 drives ... maybe each of the two SFF-8643 ports connects to a SAS expander that each has 16 SFF-8087 ports (2 SFF-8643 ports x 2 SAS expanders x 16 SFF-8087 ports x 4 drives per SFF-8087 = 128 drives)? What are you trying to accomplish? If you give us a idea of what you are doing we can give you more targeted answers instead of just general ones. I'm just trying to create glorified NAS and Plex server, but I love to know about everything even if I won't be using everything. A lot of my questions are more for academic purposes - I'm just not satisfied with buying the parts I need, I have to understand all the options that exist.
November 3, 201510 yr Author How many drives do you want to have, what kind of MB do you have. I haven't built the system, but I'm planning to eventually have 24 drives in a Norco 4224 with a Xeon E5-1650v3 and a SUPERMICRO MBD-X10SRA-F-O.
November 3, 201510 yr I still don't understand how the LSI 9300 MegaRAID SAS 9361-8i (LSI00416) can connect to 128 drives ... maybe each of the two SFF-8643 ports connects to a SAS expander that each has 16 SFF-8087 ports (2 SFF-8643 ports x 2 SAS expanders x 16 SFF-8087 ports x 4 drives per SFF-8087 = 128 drives)? Yes, that would be the way how 128 drives are hooked up on the SAS controller. RAID cards vs. HBA (Host Bus Adapter) RAID cards have more or less (depends on the money you're willing to pay) an own beefy processor that will take over the parity calculations. Those cards also have their own RAM and a so called BBU (Battery backup unit). HBA is just a means to plug more drives to the system. The card is dumb and is just passing the drive through to the OS. All parity calculations have to be done on the CPU. And finally we have cards that can do both, depending on the firmware used. IR or IT. Those cards are usually fake RAIDs because they don't do active parity calculations in RAID mode. For the use with unRAID we need plain passthrough and unRAID will do the "RAID" magic (CPU based). In order to hook up 24 drives you should watch out for a board with 3 PCIe slots and plug 3 of those 8-port HBA's in. There are also boards with plenty of SATA ports (up to 11 AFAIK) - this would save you an HBA. As already mentioned, you should try to avoid a bootleneck by hooking up all drives on one PCIe port. See here for some additional info on bandwith and controller cards.
November 3, 201510 yr How many drives do you want to have, what kind of MB do you have. I haven't built the system, but I'm planning to eventually have 24 drives in a Norco 4224 with a Xeon E5-1650v3 and a SUPERMICRO MBD-X10SRA-F-O. I would just get one 16 port or two 8 port SAS cards and not mess with a SAS expander. I currently use a LSI 9211-8i that is flashed to IT mode. There is a supermicro AOC-SASLP-MV8 3Gbps SAS card that a lot of user use here it is cheaper than the LSI cards and since we use mechanical hard drives there really isn't any bottleneck. I was able to get the AOC-SASLP-MV8 for about $75.
November 3, 201510 yr 16-port HBA is quite expensive IMHO. 8-port LSI is also expensive compared to the alternatives available. One can get an 8i LSI based card from IBM (1015 or 1115) or DELL (H310 or H200) for far less. People pull this cards from OEM machines and replace them with real RAID controllers. You can find them quite often on ebay. The only thing that has to be done is to flash them to IT mode. There are good instructions around here how to do that and it's worth the effort - again IMHO.
November 3, 201510 yr How many drives do you want to have, what kind of MB do you have. I haven't built the system, but I'm planning to eventually have 24 drives in a Norco 4224 with a Xeon E5-1650v3 and a SUPERMICRO MBD-X10SRA-F-O. I would just get one 16 port or two 8 port SAS cards and not mess with a SAS expander. I currently use a LSI 9211-8i that is flashed to IT mode. There is a supermicro AOC-SASLP-MV8 3Gbps SAS card that a lot of user use here it is cheaper than the LSI cards and since we use mechanical hard drives there really isn't any bottleneck. I was able to get the AOC-SASLP-MV8 for about $75. The SASLP is somewhat bandwidth limited, in my experience a fully loaded card will limit a parity check/sync and disk rebuild to 80MB/s max, and while that it’s a perfectible acceptable speed it will be a bottleneck for the slowest disks on the market today and bigger one for the fastest ones that can do 200MB/s+ in the outer tracks. Just so you are not surprised after getting one and decide now if that’s an acceptable speed for you.
November 3, 201510 yr The SASLP is somewhat bandwidth limited, in my experience a fully loaded card will limit a parity check/sync and disk rebuild to 80MB/s max, and while that it’s a perfectible acceptable speed it will be a bottleneck for the slowest disks on the market today and bigger one for the fastest ones that can do 200MB/s+ in the outer tracks. That seems incorrect to me. Even with her only 4 PCIe lanes (assuming PCIe v2) that card offers a bandwidth of ~2TB/s ~2GB/s. That should be enough for 8 drives with ~200MB/s (that I haven't yet seen). Supermicro even states 300MB/s per channel - referring to the 8 SATA channels. Parity check is just as fast as your slowest drive read rate. Maybe that's your 80MB/s cap?
November 3, 201510 yr The SASLP is somewhat bandwidth limited, in my experience a fully loaded card will limit a parity check/sync and disk rebuild to 80MB/s max, and while that it’s a perfectible acceptable speed it will be a bottleneck for the slowest disks on the market today and bigger one for the fastest ones that can do 200MB/s+ in the outer tracks. That seems incorrect to me. Even with her only 4 PCIe lanes (assuming PCIe v2) that card offers a bandwidth of ~2TB/s ~2GB/s. That should be enough for 8 drives with ~200MB/s (that I haven't yet seen). Supermicro even states 300MB/s per channel - referring to the 8 SATA channels. Parity check is just as fast as your slowest drive read rate. Maybe that's your 80MB/s cap? The SASLP is PCIe gen1 4x, max theoretical bandwidth is 1000MB/s, max I could get in Unraid V6 was ~580MB, with the same tweak that helps the SAS2LP it goes to ~640MB/s max. You can see more details in my post where I tested various controller with SSDs, the SASLP was particularly slow when compared for example with the Adaptec 1430SA, also PCIe gen1 4x but can deliver ~840MB/s max.
December 26, 201510 yr Author I still don't understand how the LSI 9300 MegaRAID SAS 9361-8i (LSI00416) can connect to 128 drives ... maybe each of the two SFF-8643 ports connects to a SAS expander that each has 16 SFF-8087 ports (2 SFF-8643 ports x 2 SAS expanders x 16 SFF-8087 ports x 4 drives per SFF-8087 = 128 drives)? Yes, that would be the way how 128 drives are hooked up on the SAS controller. RAID cards vs. HBA (Host Bus Adapter) RAID cards have more or less (depends on the money you're willing to pay) an own beefy processor that will take over the parity calculations. Those cards also have their own RAM and a so called BBU (Battery backup unit). HBA is just a means to plug more drives to the system. The card is dumb and is just passing the drive through to the OS. All parity calculations have to be done on the CPU. And finally we have cards that can do both, depending on the firmware used. IR or IT. Those cards are usually fake RAIDs because they don't do active parity calculations in RAID mode. For the use with unRAID we need plain passthrough and unRAID will do the "RAID" magic (CPU based). In order to hook up 24 drives you should watch out for a board with 3 PCIe slots and plug 3 of those 8-port HBA's in. There are also boards with plenty of SATA ports (up to 11 AFAIK) - this would save you an HBA. As already mentioned, you should try to avoid a bootleneck by hooking up all drives on one PCIe port. See here for some additional info on bandwith and controller cards. Sorry for the late response, life put unRAID stuff on the low priority stuff. This is an excellent explanation sir, thank you. I also found this long post helpful: https://forums.freenas.org/index.php?threads/confused-about-that-lsi-card-join-the-crowd.11901/ I ended up building my unRAID this week (after replacing a DOA Seasonic PSU and frustratingly discovering that you can't use Seasonic PSU modular cables on a Corsair PSU). The server has a Xeon 1650 v3 CPU, Supermicro MBD-X10SRA-F-O motherboard, 32GB DDR4 ECC RAM, Noctua 120mm and 80mm PWM case fans, Samsung 850 EVO 500GB SSD for cache, two WD Red 6TB drives, all in a Norco 4224 case. After Black Friday bargain shopping, was able to get the entire set including the unRAID OS purchase for $2,272.85. My motherboard has 8 SATA ports so I should only have to get two 8-port HBAs. The server is currently doing the first of two pre-clear cycles. Thanks to everyone who helped me with my first unRAID build. Happy holidays.
Archived
This topic is now archived and is closed to further replies.