New Inexpensive 24 port SATA PCIE Card

[bing281]

NOTE: i own this card but have not currently tested it in my fully populated 24 drive array system to see how it performs for speed to each drive and under UNRAID on XFS using RECONSTRUCTIVE WRITE aka all drive same time live.   I appreciate the discussion i had below and will report back on this card as soon as I get it tested.   

 

IMPORTANT: CURRENTLY UNTESTED on UNRAID do not purchase thinking it will work until I do my testing unless you want to take that risk.   See replies to this post below.  

 

I recently had my LSI 9305-24i fail and I purchased the following card and cables to replace it.   I believe this is fully supported and I also believe it is a really inexpensive option over LSI HBA's for 24 port internal 

 

 

1x Amazon.com: IO CREST 24 Port SATA III to PCIe 3.0 x4 Non-RAID Expansion Card JMB575 JMB582 Low Profile Bracket,SI-PEX40169 @ ~$150

3x Amazon.com: ChenYang CY SFF-8654 8i 74Pin PCI-E Ultraport Slimline SAS Slim 4.0 to Dual SFF-8087 Mini SAS Cable PCI-Express @ ~$36 each

 

note may need different cables depending on what you are connecting to 

note the card for ~$150 comes with 3x SFF-8654 (big connector on board) to 8x sata port cables included in price.  

 

Total ~$250 for 24 port internal JBOD SATA with cables

 

 

 

I will report back after i actually install this however I thought it would be interesting to others 

 

 

Edited October 28, 2022 by bing281
Add note that i need to test this card before actually recommending it as working for speed to all drives and stability of UNRAID XFS array itself.

[ChatNoir]

1 hour ago, bing281 said:

1x Amazon.com: IO CREST 24 Port SATA III to PCIe 3.0 x4 Non-RAID Expansion Card JMB575 JMB582 Low Profile Bracket,SI-PEX40169 @ ~$150

Well this card uses a port multipler, those are not recommended with Unraid. That's probably why it's cheaper than a HBA.

 

 

[bing281]

i agree those are not recommended and i don't even use them after my HBA's but i also was reading lately that these specific port multipliers are supported and they also are fast enough to max out all the ports at once on spinning hardware but not on SSD's.   

 

Can you correct me here if I am wrong because that was my understanding on these specific chips from this forum and from storage review 

 

Here is the basic setup from this card 

 

 

Goes JMB585 -> 5x JMB575 -> 25/24 SATA ports 

 

on the thread you referenced they call out JMB585 chip cards as recommended.  Difference here it does't have the secondary breakdown with the JMB575 however to my knowledge this should only be a speed issue for SSD's.   

 

Perhaps I am uneducated or missing something and I look forward to understanding better.   Thanks for the help. 

Edited October 28, 2022 by bing281

[bing281]

Shoot i see the issue now i think hmm 

 

Basically it goes JMB585 which is PCIE to 5xSATA ports (6Gbps)

then it goes on each of the 5xSATA to a JMB575 which is a SATA port multiplier from 1xSATA to 5xSATA (6Gbps) total of 25 SATA however it is only using 24 of them and neglecting 1.   

 

So you are saying that UNRAID doesn't like the JMB575 multiplier in this case.  What is the reason that UNRAID doesn't like port multipliers is it a speed thing or is it another inherent reason? 

 

Let me do some math real quick here.  

 

PCIE x4 = 3.938 GB/s or 31.504 Gbps 

PCIE x4 31.504 Gbps / 5xSATA = 6.308 Gbps to each multiplier over the sata link

6.308 Gbps / 5xSATA port multiplier = 1.26016 Gbps to each of the final 24 SATA ports

 

1.26016 Gbps on each of the 24 ports is approximately equal to 157.5 MB/s max speed on each port.   

 

On my spinners I see them at parity check when on the beginning / inside of the plates run around 160 MB/s max speed for parity checking 

 

Now continuing my thinking out loud here there is clearly some loss in there during all of this however theoretically it can basically push all 24 of my spinners at max speed which they can only do for about the first 0-25% of the parity check.    Therefore I am guessing that while I would see some decrease in max sustainable speed I don't think it would bring them down below 140 MB/s or around 10% and i think it could maintain that speed easily.   Therefore for a 10% loss in max sustainable speed I can stop purchasing $1000 cards.   However this only holds true if there is not some other unknown to me issue for why UNRAID can't use port multipliers.   

 

Thoughts?  

 

I currently am running my older -8i IT cards on my drives and i think they bring down the max sustainable speed to around 120 MB/s 

Edited October 28, 2022 by bing281

[JorgeB]

4 hours ago, bing281 said:

So you are saying that UNRAID doesn't like the JMB575 multiplier in this case. 

It's not Unraid, it's a general issue, though it Unraid in can b worse since all the devices are accessed simultaneously (if they are in the array).

 

4 hours ago, bing281 said:

PCIE x4 31.504 Gbps / 5xSATA = 6.308 Gbps to each multiplier over the sata link

6.308 Gbps / 5xSATA port multiplier = 1.26016 Gbps to each of the final 24 SATA ports

Each SATA port 600MB/s is split into 5 ports, so 600/5=125MB/s max per port if all used at the same time in the same multiplier, also the JMB585 is PCIe x2 (it's only x4 physically) max usable bandwidth is around 1750MB/s, so if all drives are accessed at the same time that's around 73MB/s max per device.

[bing281]

7 hours ago, JorgeB said:

It's not Unraid, it's a general issue, though it Unraid in can b worse since all the devices are accessed simultaneously (if they are in the array).

 

Each SATA port 600MB/s is split into 5 ports, so 600/5=125MB/s max per port if all used at the same time in the same multiplier, also the JMB585 is PCIe x2 (it's only x4 physically) max usable bandwidth is around 1750MB/s, so if all drives are accessed at the same time that's around 73MB/s max per device.

yeah i think you are right on this one for the 73MB/s that card says it requires 4x lanes but who knows if it really does.   

 

I think i am going to install it and run it on one of my test UNRAID servers and see how it does.   I am interested to see how it operates on a 24 hard drive system and the speed it will really give.   Because if it did give 160MB/s per drive and was stable it is an excellent inexpensive UNRAID solution.   Since I have one I might as well find out for others.   However I went on the eBay yesterday and purchased a 24i HBA card which was surprisingly inexpensive at only $200 without any cables at all.  

 

Therefore my current comment is that no one should run out and buy this for UNRAID until i put it in my system and see how it runs and report back.   

 

Appreciate your help and comments thank you.   

[bing281]

I updated the original post with notes about not currently recommended to purchase until my testing and peer reviewed testing is complete.   

[DotJun]

If you don't mind used, Adaptec 72405 can be gotten cheap and it's also 24 internal ports.

[camprr]

For all those concerned about bandwidth (and saying solution is bad, because it's limited), also think about the PCIe bus connection. 1x PCIe lane is around 1Gbyte/sec (more like 750Mbytes/sec in practice). So this card is 4 lanes, maximum ~3000Mbytes/sec. Which is ~125Mbytes/sec/drive. That's just about golden for 5400rpm disks.
If you have a x8 Adaptec/LSI card, it's not going to be pushing much faster when using all drives at the same time. I think that last thing is key. When are you likely to use 24 drives _at the same time_? Quite unlikely unless you have a 24-drive Raid-5/6 set (and that is nuts from a reliability point of view.)

@bing281, did you come to a conclusion of your tests? (I ask because I have one laying on my desk, about to be used for 8Tbyte QVO SATA SSDs as a long-term storage for 8x Intel P4150 drives which are going to be connected through this extra cool/special card: https://www.amazon.nl/-/en/PCIe-ports-Switch-chipset-Profile/dp/B097HRQJZ8 (PCIe 3.0 card 16x for 8 SSD U.2 NVMe (U2 NGFF) or 8 ports PCIe x4 Multi Host Switch Card chipset PLX PEX 8749 High and Low Profile.). The idea is to use the NVMe disks as a caching layer in front of the SATA disks.  Basically the PCIe switch card allows me to use 8 drives, on a 16x PCIe bus. Limited to ~1500Mbytes/drive, and if all used at the same time, around 12Gbytes/sec. So the SATA backup is a factor ~4x slower than the cache. The idea is to have the cards in the server and add disks as my requirements grow.

Edited February 23, 2023 by camprr
Adding more information about the complete setup for Unraid

[trurl]

14 minutes ago, camprr said:

When are you likely to use 24 drives _at the same time_?

Most don't, but there are certainly some who do with Unraid. 

 

15 minutes ago, camprr said:

24-drive Raid-5/6 set (and that is nuts from a reliability point of view.)

Unraid doesn't do it like that. But dual parity definitely recommended if you have that many disks.

[Michael_P]

1 hour ago, camprr said:

When are you likely to use 24 drives _at the same time_?

 

Every time I upgrade/replace a drive, all 24 are used to re-build it - and it already takes over 30 hours

 

[camprr]

But it's not like you are sitting next to it, waiting. Are you? I am talking about performance bottlenecks that cause a loss in 'exterior' performance. I mean, after all, you have to have something that processes data at that speed. So if we manage to reach 10Gbit, basically all is good. Or not? And 10Gbit can go through a PCIe x2 slot. My internet connection is 1Gbit, I do a but of unparring (but that's all read from memory, as they are recent writes), and then the media is watched at a 'mellow' pace (10-50Mbit). 

My question is more this: "for a hobby environment, a PCIe x4 to 24xSATA card is plenty good". Yes, I think is the answer to that. For enterprise server grade stuff not, but would you be running Unraid for that? More likely ceph or something that provides multi-node redundancy and increased performance.

[Michael_P]

34 minutes ago, camprr said:

a PCIe x4 to 24xSATA card is plenty good"

 

It's not x4, it's x2 in x4 clothes. Will you notice a performance difference, depends on if you ever need to do more than 1 or 2 things at once on any of the drives, and more so for drives slung off of the multiplier.

[Kilrah]

47 minutes ago, camprr said:

But it's not like you are sitting next to it, waiting. Are you? I am talking about performance bottlenecks that cause a loss in 'exterior' performance.

When you run in reconstruct write mode and do any write to the array all drives are being hit at the same time, i.e. full load scenario.

Edited February 24, 2023 by Kilrah

[Cerberus9]

I'm currently only planning, but I won't use this card and not so many disks. I'm planning on using 2x delock card: https://www.delock.com/produkt/90010/merkmale.html?setLanguage=en

 

I did the math on IO, I calculate that I get roughly 200MB/s /device, and HDD speeds are close to maxed with that. So I think that will be fine for me. However I don't know abount how Unraid can hangle this. Can anyone update me with this? Should I scrap the idea?

 

Thanks.

[Vr2Io]

7 hours ago, Cerberus9 said:

I'm currently only planning, but I won't use this card and not so many disks. I'm planning on using 2x delock card: https://www.delock.com/produkt/90010/merkmale.html?setLanguage=en

 

I did the math on IO, I calculate that I get roughly 200MB/s /device, and HDD speeds are close to maxed with that. So I think that will be fine for me. However I don't know abount how Unraid can hangle this. Can anyone update me with this? Should I scrap the idea?

 

Thanks.

ASM1064 work well with Unraid generally, x1 PCIe 3.0 also fine for 4 spinner disk.