JorgeB Posted May 16, 2023 Author Share Posted May 16, 2023 8 hours ago, Allexz said: If you connect 6/6 sata ports to SSDs and only one ssd transfers at this moment in time will it get the full throughput of the pci-express slot ? Yes, you'll get around 560MB/s Quote Link to comment
coolspot Posted July 21, 2023 Share Posted July 21, 2023 On 10/20/2021 at 12:33 AM, rosswaters said: I am not daisy chaining to another enclosure and I don't need redundancy on the SFF-8088s, I am just looking for better than 100MBs per drive. Did you ever figure this out, I have the same 36 bay enclosure with the same SAS2-846EL2 controller, and fully loaded I'm maxed out at ~90MB/s using SATA drives. Quote Link to comment
Kirkbuilder Posted September 5, 2023 Share Posted September 5, 2023 Love this topic! I read all the posts here and I am still a bit confused. Please forgive my ignorance about hardware, but I am trying 🙂 I have been trying to figure out how to increase my disk speeds on my array and was told in a discord channel that I was stuck at my current speeds and there was nothing to be done about it. BUT, I kept on looking and found this wonderful article which I have learned a great deal from. My problem, my array of 18 disks is doing read/write at ~100mb and am currently shrinking my array using the second non-volatile procedure of moving the data using unbalance and then 0'ing out the drive to update parity, etc, etc. but am only getting 69.8 MB/s. As can be seen here: Here is my array: Here is my setup Unraid 6.12.3: GA-7PESH2 GIGABYTE Intel Rev1.0 LGA2011 E-ATX Server Motherboard LSI2008 8x SAS/SATA, 2x 10G X540-AT2, Avocent IPMI 2.0 (GA-7PESH2) 2x Intel Xeon E5-2690 v2 3.0GHz 10 Core 25MB Cache Processors 128GB DDR3 1600MHz ECC Registered Memory 8x 16GB HP 487738-001 468405-001 24-Bay 3Gb SAS Expander Card Samsung 860 QVO 1TB 2.5 Inch SATA III Internal SSD (MZ-76Q1T0B/AM) x2 Cache 24 x 8+ TB Drives I want to try to maximize bandwidth since it is taking me ~40 hours to "0" out an 8TB drive. From what I have read, I am now thinking I need to get 2 SAS cards to do this, but was hoping you might confirm that a good option would be 2x LSI 9305-24i's. Quote Link to comment
JorgeB Posted September 5, 2023 Author Share Posted September 5, 2023 Please post the diagnostics. Quote Link to comment
Kirkbuilder Posted September 5, 2023 Share Posted September 5, 2023 tower-diagnostics-20230905-1721.zip Quote Link to comment
JorgeB Posted September 6, 2023 Author Share Posted September 6, 2023 With 18 disks connected to the HP expander max speed will max out at roughly 120MB/s, still should be better then 100MB/s, possibly the CPU bottlenecking, also post the output of: lspci -d 1000: -vv to check HBA link speed/width Quote Link to comment
Kirkbuilder Posted September 6, 2023 Share Posted September 6, 2023 9 hours ago, JorgeB said: With 18 disks connected to the HP expander max speed will max out at roughly 120MB/s, still should be better then 100MB/s, possibly the CPU bottlenecking, also post the output of: lspci -d 1000: -vv to check HBA link speed/width Here you go. I was hoping to get something over 200MB or more with a change. lspci.txt Quote Link to comment
JorgeB Posted September 7, 2023 Author Share Posted September 7, 2023 The HBA is only linking at x4: Speed 5GT/s, Width x4 (downgraded) That means around 1600MB/s usable total, with 16 drives it's 100MB/s, so it checks out. To be able to get around 200MB/s the HBA would need to be on an x8 slot and you'd also need a faster expander, current one will max out at around 120MB/s as mentioned. Quote Link to comment
Kirkbuilder Posted September 7, 2023 Share Posted September 7, 2023 10 hours ago, JorgeB said: The HBA is only linking at x4: Speed 5GT/s, Width x4 (downgraded) That means around 1600MB/s usable total, with 16 drives it's 100MB/s, so it checks out. To be able to get around 200MB/s the HBA would need to be on an x8 slot and you'd also need a faster expander, current one will max out at around 120MB/s as mentioned. What would you suggest to replace it? Quote Link to comment
JorgeB Posted September 7, 2023 Author Share Posted September 7, 2023 5 minutes ago, Kirkbuilder said: What would you suggest to replace it? First you need to get the HBA linking at x8, since that's the current bottleneck, is that possible with your board? Quote Link to comment
Kirkbuilder Posted September 7, 2023 Share Posted September 7, 2023 15 minutes ago, Kirkbuilder said: What would you suggest to replace it? Thank you for your analysis! I have had a great deal of luck with HP so probably want HP 12G SAS3 EXPANDER (761879-001) or would you suggest another card? Quote Link to comment
Kirkbuilder Posted September 7, 2023 Share Posted September 7, 2023 25 minutes ago, JorgeB said: First you need to get the HBA linking at x8, since that's the current bottleneck, is that possible with your board? https://download1.gigabyte.com/Files/Manual/server_manual_7pesh_e_1003.pdf?v=102d7cc391a19a751aff685e16f3ee87 I believe so. Quote Link to comment
Kirkbuilder Posted September 7, 2023 Share Posted September 7, 2023 3 hours ago, JorgeB said: First you need to get the HBA linking at x8, since that's the current bottleneck, is that possible with your board? Yes. The expander was in an 16x PCI slot. I switched it to an 8x slot. here is my new lspci. Still shows LnkSta: Speed 5GT/s, Width x4 (downgraded) lspci.txt Quote Link to comment
JorgeB Posted September 8, 2023 Author Share Posted September 8, 2023 Not clear if you moved the expander only or also the HBA, but if it's still linking at x4 it will still bottleneck. Quote Link to comment
Kirkbuilder Posted September 8, 2023 Share Posted September 8, 2023 7 hours ago, JorgeB said: Not clear if you moved the expander only or also the HBA, but if it's still linking at x4 it will still bottleneck. I am having trouble figuring out how to fix this. Googling and googling with no joy 😞 Quote Link to comment
JorgeB Posted September 8, 2023 Author Share Posted September 8, 2023 Basically the HBA needs to be on an x8 or x16 electrical slot, some boards have x8 or x16 physical slots that are only x4 electrically. Quote Link to comment
ChatNoir Posted September 8, 2023 Share Posted September 8, 2023 Possibly if you already have an extension card using x16 (graphics card maybe), your motherboard wouldn't or couldn't provide you x8 but only x4 ? Quote Link to comment
Kirkbuilder Posted September 14, 2023 Share Posted September 14, 2023 On 9/8/2023 at 12:48 PM, ChatNoir said: Possibly if you already have an extension card using x16 (graphics card maybe), your motherboard wouldn't or couldn't provide you x8 but only x4 ? No other cards are installed in the PCI slots. I have been looking for a way to determine if there is a setting which is keeping the power low on the motherboard or on the expander card. Any ideas? Quote Link to comment
ChatNoir Posted September 14, 2023 Share Posted September 14, 2023 If the card is plugged in the first PCIe slot (generally used for Graphics Cards) I wouldn't know why it would not link at full speed. Quote Link to comment
tazire Posted September 20, 2023 Share Posted September 20, 2023 (edited) Curious if anyone knows anything about an LSI SAS3x36R expander? I have been looking at a 2U 24bay chassis that uses this expander with the intention of using it as a JBOD to connect SSD's to my server. I'm just trying to make sure that it will have the throughput to use SSD's. Fairly sure it should be absolutely fine during normal use but in a worst case if I'm hitting all 24 drives at once what's the theoretical throughput? EDIT* I would use a 9500-16e HBA to get 3 connections and PCIE4 link to HBA. Edited September 20, 2023 by tazire Quote Link to comment
JorgeB Posted September 20, 2023 Author Share Posted September 20, 2023 1 hour ago, tazire said: it as a JBOD to connect SSD's to my server. SAS roughly 9600MB/s theoretical max, I would expect around 8800MB/s usable. SATA devices can make use of LSI's Databolt for better than 4800MB/s, but don't expect same as SAS, should be capable of at least 6000 to 7000MB/s. 1 hour ago, tazire said: I would use a 9500-16e HBA to get 3 connections and PCIE4 link to HBA. You can just connect two cables from the HBA to the expander. Quote Link to comment
tazire Posted September 20, 2023 Share Posted September 20, 2023 1 hour ago, JorgeB said: SAS roughly 9600MB/s theoretical max, I would expect around 8800MB/s usable. SATA devices can make use of LSI's Databolt for better than 4800MB/s, but don't expect same as SAS, should be capable of at least 6000 to 7000MB/s. You can just connect two cables from the HBA to the expander. The particular expander allows for up to 3 connections. Would this make any difference? Quote Link to comment
JorgeB Posted September 20, 2023 Author Share Posted September 20, 2023 10 minutes ago, tazire said: The particular expander allows for up to 3 connections. Only two wide connections are supported by expanders, the other one can be for cascading. 1 Quote Link to comment
DuckServer Posted September 22, 2023 Share Posted September 22, 2023 Can we get 32 HDDs x 140MB/s by this way? , just plug one HBA card in PCIEX16 slot. Quote Link to comment
JorgeB Posted September 22, 2023 Author Share Posted September 22, 2023 47 minutes ago, DNU said: Can we get 32 HDDs x 140MB/s by this way? , You can, note that speed will be limited to roughly 140MB/s with all disks used simultaneously. 1 Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.