Hello everyone.  I am trying to figure out how to add a dual NIC 2.5GbE to my build.  I have a Asus Prime B550-PLUS motherboard.  an LSI SAS2008 HBA running 8 spinning drives.  4 spinning drives and 2 SATA SSD connected directly to the MB.  I have 2 NVME, one on the MB and one using a PCIE to M2 adapter.  I have put together a kind of diagram, and I am wondering if what I am proposing will significantly limit the speed of the HBA.  The slot the HBA is in is normally a x4 slot (I believe this refers to speed) that drops to x2 if either or both of the bottom to x1 slots are occupied.  I think I need 1 of these slots to accomplish all of the hardware i need.  So the question is, will dropping the HBA to x2 instead of x4 slow anything down in a significant way?

 

If there is anything I can clarify to help me get this optimized please let me know.

Thank you!

40 minutes ago, Rhuarc said:

So the question is, will dropping the HBA to x2 instead of x4 slow anything down in a significant way?

 

It will limit the HBA to around 800MB/s usable, so yes, it should be noticeable with 8 disks during a parity check/rebuild.

Would it be any better if I swapped the 1060 into the bottom slot and put the HBA in the top slot?  The onyl thing I use the 1060 for is outputting the GUI to a local monitor and pass through to a PLEX docker for transcoding.

 

Theoretically this would probably use less bandwidth than the HBA?

 

Thank you for your help!

Probably a better option.

Is there room to fit anything in pciex_2 or 3 with the 1060 in the lower position?

 

You're fundamentally up against the hardware not quite having what you need. Is 2.5Gb that important?

5 hours ago, _cjd_ said:

Is there room to fit anything in pciex_2 or 3 with the 1060 in the lower position?

 

You're fundamentally up against the hardware not quite having what you need. Is 2.5Gb that important?

Yeah, this is my main problem!  I do want the faster NIC.  I do photo and video editing and keep my main library and files on the NAS.  Being able to edit with little delay to the full size files is greatly increased with the faster direct connection.

 

I think I have space to add the NIC to the bottom slot even with the 1060 in the bottom 16x slot.  The main thing I need to check also is that the bottom 16x2 slot can output over a video card.  I'm assuming it can and that the system won't think the HBA is a video device and try to output over that.

 

I wonder also if the HBA will run faster in the top 16x slot that in the bottom one even when the bottom one is running at full speed.  Is there an easy way to do a speed test on the HBA now and then rerun later?  A parity check I know can indicate but I get wildly difference speeds during parity checks depending on what else I am doing on the server.  I guess I could disable docker and vm's and run a 20 minute parity check to see what speed I get, then redo everything and run the same 20 minute parity check to see what speeds I get then.

From experience... Jump to 10gb networking. Assuming you're not using spinning disks for photo access etc... if you can do SFP+ DAC (not rj45) its almost cheap to get into. And far more stable.

 

Video card will do it's thing wherever it is.

 

Diskspeed docker may help on perf measurement numbers.

5 hours ago, _cjd_ said:

From experience... Jump to 10gb networking. Assuming you're not using spinning disks for photo access etc... if you can do SFP+ DAC (not rj45) its almost cheap to get into. And far more stable.

 

Video card will do it's thing wherever it is.

 

Diskspeed docker may help on perf measurement numbers.

Thanks for this!

 

I actually just did the switch.  Now have dual 2.5GbE, the GPU is now showing 2 lanes instead of the 16 before I made the swap and added the NIC.  As long as the 2 lanes are plenty for just transcoding in PLEX this should not be a problem!  And as a bonus after running Diskspeed on the controller I actually gained around 10 percent performance on all disk reads and the Controller is no longer being the bottleneck!

 

Before:

 

After:

Alright, so my idea for putting the graphics card in the second x16 slot and running it all the way down at x2 speeds and putting the HBA in the top 16x slot does not seem to have worked.  The GPU, while recognized and seeming to be passing through correctly to Plex and doing transcoding, is not working correctly.  The transcode seems to be software only even though GPU Stats is saying it is using the GPU.  To test I swapped the 2 cards again and right back to full transcoding speeds.

 

Basically now I need to figure out how to run all 12 of my drives only using the HBA.  This will free up the second M2 on my second NVME drive in, and then the NIC can go in the PCIEx1 slot currently occupied by the M2 to NVME adapter. 

 

My questions are:

1) If I get an Expander, can I cover the pins in kapton tape and then put it into an X1 slot only to hold it and then power it off of a Molex?

 

2) Would I even need to do #1, or would the Expander just get power and the board would not even recognize that it has a card running therefor dropping the speed of the second x16 slot?

 

3) Would I then run 8 of the drives off the Expander and the other 4 off the breakout cable of the HBA, or if you have an Expander do you never have any drives connected directly to the HBA?

 

Thanks again!

SAS expander won't use PCIe lane, but some expander only accept power by PCIe, and it need 3.3v and 12v, so in this case you can use apapter or DIY to power it, as result won't waste a PCIe slot.

 

82885t is SAS3, and I always recommend, you can power it by PCIe or molex. Recently I got two more which cost only usd7 each, total have four now.

 

For hba connect disk directly or not, it just depends on which you like, not a issue. Currently I use 9300-4i4e to backplane expander and external 82885t, no disk connect to onboard.

Edited July 31, 20241 yr by Vr2Io

Thanks for the advice!  Here is the one I am specifically looking at:

 

https://www.amazon.com/Intel-RAID-Expander-Card-RES2SV240/dp/B0042NLUVE

 

In this case I could connect 4 drives to my HBA, the HBA to this expander, and then my other 8 drives to this expander.  

 

I believe this can also be powered over a molex connecter.  I would probably test to see if using power from an x1 slot would drop the speed of the x16 slot.  If it does then I would kapton tape the pins and power over molex.  If it does not then I would just power using the x1 slot.

 

Oh, also, on these expanders do they use a different type of cable to connect the drives than what I currently have connecting the drives to the HBA?  I know I still the the cable to connect between the HBA and the expander also.

Edited July 31, 20241 yr by Rhuarc

Use below adapter (something like) to power the expander will be more reliable and simple, extend the PCIe 1x to power up expander aren't recommend. As mention, expander not use PCIe lane, so you don't need connect that “USB” cable. This won't affect speed.

 

But you also can DIY, a SATA power plug already have the need of 3.3v and 12v.

 

 

SAS 2 use 8087 / 8088 and SAS 3 use 8643 / 8644. SAS 3 double the bandwidth, so future prove.

Edited July 31, 20241 yr by Vr2Io

5 hours ago, Rhuarc said:

I believe this can also be powered over a molex connecter.

You can, and no need to connect it to a PCIe slot if that is being used.