May 16, 20233 yr Author 8 hours ago, Allexz said: If you connect 6/6 sata ports to SSDs and only one ssd transfers at this moment in time will it get the full throughput of the pci-express slot ? Yes, you'll get around 560MB/s
July 21, 20232 yr On 10/20/2021 at 12:33 AM, rosswaters said: I am not daisy chaining to another enclosure and I don't need redundancy on the SFF-8088s, I am just looking for better than 100MBs per drive. Did you ever figure this out, I have the same 36 bay enclosure with the same SAS2-846EL2 controller, and fully loaded I'm maxed out at ~90MB/s using SATA drives.
September 5, 20232 yr Love this topic! I read all the posts here and I am still a bit confused. Please forgive my ignorance about hardware, but I am trying 🙂 I have been trying to figure out how to increase my disk speeds on my array and was told in a discord channel that I was stuck at my current speeds and there was nothing to be done about it. BUT, I kept on looking and found this wonderful article which I have learned a great deal from. My problem, my array of 18 disks is doing read/write at ~100mb and am currently shrinking my array using the second non-volatile procedure of moving the data using unbalance and then 0'ing out the drive to update parity, etc, etc. but am only getting 69.8 MB/s. As can be seen here: Here is my array: Here is my setup Unraid 6.12.3: GA-7PESH2 GIGABYTE Intel Rev1.0 LGA2011 E-ATX Server Motherboard LSI2008 8x SAS/SATA, 2x 10G X540-AT2, Avocent IPMI 2.0 (GA-7PESH2) 2x Intel Xeon E5-2690 v2 3.0GHz 10 Core 25MB Cache Processors 128GB DDR3 1600MHz ECC Registered Memory 8x 16GB HP 487738-001 468405-001 24-Bay 3Gb SAS Expander Card Samsung 860 QVO 1TB 2.5 Inch SATA III Internal SSD (MZ-76Q1T0B/AM) x2 Cache 24 x 8+ TB Drives I want to try to maximize bandwidth since it is taking me ~40 hours to "0" out an 8TB drive. From what I have read, I am now thinking I need to get 2 SAS cards to do this, but was hoping you might confirm that a good option would be 2x LSI 9305-24i's.
September 6, 20232 yr Author With 18 disks connected to the HP expander max speed will max out at roughly 120MB/s, still should be better then 100MB/s, possibly the CPU bottlenecking, also post the output of: lspci -d 1000: -vv to check HBA link speed/width
September 6, 20232 yr 9 hours ago, JorgeB said: With 18 disks connected to the HP expander max speed will max out at roughly 120MB/s, still should be better then 100MB/s, possibly the CPU bottlenecking, also post the output of: lspci -d 1000: -vv to check HBA link speed/width Here you go. I was hoping to get something over 200MB or more with a change. lspci.txt
September 7, 20232 yr Author The HBA is only linking at x4: Speed 5GT/s, Width x4 (downgraded) That means around 1600MB/s usable total, with 16 drives it's 100MB/s, so it checks out. To be able to get around 200MB/s the HBA would need to be on an x8 slot and you'd also need a faster expander, current one will max out at around 120MB/s as mentioned.
September 7, 20232 yr 10 hours ago, JorgeB said: The HBA is only linking at x4: Speed 5GT/s, Width x4 (downgraded) That means around 1600MB/s usable total, with 16 drives it's 100MB/s, so it checks out. To be able to get around 200MB/s the HBA would need to be on an x8 slot and you'd also need a faster expander, current one will max out at around 120MB/s as mentioned. What would you suggest to replace it?
September 7, 20232 yr Author 5 minutes ago, Kirkbuilder said: What would you suggest to replace it? First you need to get the HBA linking at x8, since that's the current bottleneck, is that possible with your board?
September 7, 20232 yr 15 minutes ago, Kirkbuilder said: What would you suggest to replace it? Thank you for your analysis! I have had a great deal of luck with HP so probably want HP 12G SAS3 EXPANDER (761879-001) or would you suggest another card?
September 7, 20232 yr 25 minutes ago, JorgeB said: First you need to get the HBA linking at x8, since that's the current bottleneck, is that possible with your board? https://download1.gigabyte.com/Files/Manual/server_manual_7pesh_e_1003.pdf?v=102d7cc391a19a751aff685e16f3ee87 I believe so.
September 7, 20232 yr 3 hours ago, JorgeB said: First you need to get the HBA linking at x8, since that's the current bottleneck, is that possible with your board? Yes. The expander was in an 16x PCI slot. I switched it to an 8x slot. here is my new lspci. Still shows LnkSta: Speed 5GT/s, Width x4 (downgraded) lspci.txt
September 8, 20232 yr Author Not clear if you moved the expander only or also the HBA, but if it's still linking at x4 it will still bottleneck.
September 8, 20232 yr 7 hours ago, JorgeB said: Not clear if you moved the expander only or also the HBA, but if it's still linking at x4 it will still bottleneck. I am having trouble figuring out how to fix this. Googling and googling with no joy 😞
September 8, 20232 yr Author Basically the HBA needs to be on an x8 or x16 electrical slot, some boards have x8 or x16 physical slots that are only x4 electrically.
September 8, 20232 yr Possibly if you already have an extension card using x16 (graphics card maybe), your motherboard wouldn't or couldn't provide you x8 but only x4 ?
September 14, 20232 yr On 9/8/2023 at 12:48 PM, ChatNoir said: Possibly if you already have an extension card using x16 (graphics card maybe), your motherboard wouldn't or couldn't provide you x8 but only x4 ? No other cards are installed in the PCI slots. I have been looking for a way to determine if there is a setting which is keeping the power low on the motherboard or on the expander card. Any ideas?
September 14, 20232 yr If the card is plugged in the first PCIe slot (generally used for Graphics Cards) I wouldn't know why it would not link at full speed.
September 20, 20232 yr Curious if anyone knows anything about an LSI SAS3x36R expander? I have been looking at a 2U 24bay chassis that uses this expander with the intention of using it as a JBOD to connect SSD's to my server. I'm just trying to make sure that it will have the throughput to use SSD's. Fairly sure it should be absolutely fine during normal use but in a worst case if I'm hitting all 24 drives at once what's the theoretical throughput? EDIT* I would use a 9500-16e HBA to get 3 connections and PCIE4 link to HBA. Edited September 20, 20232 yr by tazire
September 20, 20232 yr Author 1 hour ago, tazire said: it as a JBOD to connect SSD's to my server. SAS roughly 9600MB/s theoretical max, I would expect around 8800MB/s usable. SATA devices can make use of LSI's Databolt for better than 4800MB/s, but don't expect same as SAS, should be capable of at least 6000 to 7000MB/s. 1 hour ago, tazire said: I would use a 9500-16e HBA to get 3 connections and PCIE4 link to HBA. You can just connect two cables from the HBA to the expander.
September 20, 20232 yr 1 hour ago, JorgeB said: SAS roughly 9600MB/s theoretical max, I would expect around 8800MB/s usable. SATA devices can make use of LSI's Databolt for better than 4800MB/s, but don't expect same as SAS, should be capable of at least 6000 to 7000MB/s. You can just connect two cables from the HBA to the expander. The particular expander allows for up to 3 connections. Would this make any difference?
September 20, 20232 yr Author 10 minutes ago, tazire said: The particular expander allows for up to 3 connections. Only two wide connections are supported by expanders, the other one can be for cascading.
September 22, 20232 yr Can we get 32 HDDs x 140MB/s by this way? , just plug one HBA card in PCIEX16 slot.
September 22, 20232 yr Author 47 minutes ago, DNU said: Can we get 32 HDDs x 140MB/s by this way? , You can, note that speed will be limited to roughly 140MB/s with all disks used simultaneously.
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.