Help picking SFP+ 10Gbps network adapters

[UnraidServer2020]

Hey guys,

Im new to the world or servers and Unraid. Been testing 4x120GB SSD cash pool and some dockers like NextCloud etc and im really happy with the OS. I will buy a Plus licence when my trail runs out soon. Im also new to SFP+ but know a thing or two about networking and have been building PCs for many years.

The idea is to connect the Unraid server to a video editing workstation (peer-to-peer), and connect both of the machines to the switch aswell. I should preferably get two identical cards, right? I will also help a friend setup the same solution. So I would like to buy 4 of the same cards without going broke. Used or refurbished is fine.

After several hours online at Unraids wiki, Mellanox's community pages etc Im still uncertain about what cards to get. Could someone please help me make a good pick from eBay (shipped from and to the US) ?

Any advice is much appriciated!:)

 

https://www.ebay.com/sch/i.html?_from=R40&_trksid=m570.l1313&_nkw=sfp%2B+10gbps+network+adapter&_sacat=0&LH_TitleDesc=0&LH_PrefLoc=1&_sop=15&_osacat=0&_odkw=sfp%2B+10gbps

Edited June 4, 2020 by UnraidServer2020

[JorgeB]

Mellanox ConnectX-2 and X-3 are plug play and work great with Unraid, also cheap on Ebay.

[UnraidServer2020]

4 minutes ago, johnnie.black said:

work great with Unraid

Thanks for the confimation. The card is short for server cases:
https://www.ebay.com/itm/Mellanox-MNPH29D-XTR-ConnectX-2-2x-Port-10GbE-SFP-Network-Adapter-Low-Profile/153121594290?epid=4030566940&hash=item23a6c21bb2:g:eDIAAOSw1lBbYOgd
 

but I could buy some high brackets to be able to fit it in a PC case (will upgrade to server rack cases in the future)
https://www.ebay.com/itm/Full-High-Bracket-for-Mellanox-CX312A-MCX312A-XCBT-MNPH29C-XTR-MNPH29D-XTR/184044929669?_trkparms=aid%3D1110001%26algo%3DSPLICE.SIM%26ao%3D1%26asc%3D20131231084308%26meid%3D132dc397385b4d2cade927361a3309f5%26pid%3D100010%26rk%3D5%26rkt%3D12%26sd%3D372582936866%26itm%3D184044929669%26pmt%3D0%26noa%3D1%26pg%3D2047675&_trksid=p2047675.c100010.m2109

[tjb_altf4]

Yeah I bought dual x3s with short brackets and just ordered full-size brackets.

They weren't genuine, but they were cheap and decent enough to do the job.

[UnraidServer2020]

Sharing some of my notes from the research today.

Correct me if im wrong. Hope someone find it useful.

- SFP supports only up to 4.25 Gbps

- SFP+ supports data rates up to 16 Gbps

http://www.cables-solutions.com/sfp-vs-sfp-plus.html

 

- PCI Gen1.0 (x8) is fine when using a 1 port card.

- PCI Gen2.0 (x8) is fine when using a 2 port card
(The PCI lane/slot maxes out at 32Gbps and 2.0 is enough for a dual port).
 

Edited June 4, 2020 by UnraidServer2020

[UnraidServer2020]

11 hours ago, johnnie.black said:

ConnectX-2

Considering single or dual port cards since I’m buying four of them.
 

Do you or anyone have experience with dual port cards; either using both ports directly peer-to-peer for possibly 20Gbps (SMB multichannel), or one cable between two machines and the spare port from each machine to the switch? 

Edited June 5, 2020 by UnraidServer2020

[UnraidServer2020]

Have been considering this single port card as well but it looks a bit sketchy. Read on different sources that people have good experience with them though, but I like the options and expandability with dual ports. 

Mellanox ConnectX-3 MCX341

https://www.ebay.com/itm/Mellanox-ConnectX-3-MCX341-SFP-10Gigabit-Ethernet-RMDA-RoCE/192981457274?hash=item2cee97597a:g:eo8AAOSw31pdI~00

[JorgeB]

13 hours ago, UnraidServer2020 said:

- PCI Gen1.0 (x8) is fine when using a 1 port card.

- PCI Gen2.0 (x8) is fine when using a 2 port card

That's correct, but note that the single port ConnectX-3 adapter is PCIe 3.0 x4, so it would bottleneck on a PCI gen1 slot.

 

2 hours ago, UnraidServer2020 said:

Do you or anyone have experience with dual port cards; either using both ports directly peer-to-peer for possibly 20Gbps (SMB multichannel), or one cable between two machines and the spare port from each machine to the switch? 

AFAIK no one has been able to get SMB multichannel working with Unraid yet, but it should be possible, I do have a few servers with dual port NICs, one port goes to the switch the other is for a peer-2-peer link to that server's backup, since I have no need to access the backup server @ 10GbE and this way the backups goes faster.

[UnraidServer2020]

1 hour ago, johnnie.black said:

PCIe 3.0 x4, so it would bottleneck

Yeah I see it now. Great that you saw that.  I was focusing on everything els and overlooked what PCI slot connection that X3 had.

 

1 hour ago, johnnie.black said:

multichannel

Okey, Want to test and learn about multichannel at some point.

 

1 hour ago, johnnie.black said:

one port goes to the switch the other is for a peer-2-peer link to that server's backup

Sounds nice. Should definitely invest that little extra for 2 port cards. Thanks for the help guys. This made the choice a whole lot easier:)