Mellanox ConnectX-3 support


32 posts in this topic Last Reply

Recommended Posts

Interested as well...
I scored a ConnectX-3  card a little while back and been wondering if i should toss it into my NAS box. A quick search revealed that even pfSense V1.2 supports it, so hopefully unraid can as well.

*EDIT:

Card in question is the HP 764736-001 - HPE InfiniBand FDR/Ethernet 2-Port 544+QSFP/FLR-QSFP and 547FLR-QSFP Adapter.
these cards are readily available on eBay for under $100

Edited by _Adrian_
Link to post
On 12/4/2020 at 12:51 AM, JorgeB said:

Ethernet ConnectX-3 are supported for a long time, but Unraid doesn't support Infiniband.

I have no interest in infiniband but rather if unRaid can bifurcate the 40G ports ( QSFP - 4x 10GBe ) in order to eliminate the need for a core network switch and use my unraid box as a Router/10G switch. I have 3 Miktotik CSS610-8G+2S+ ( 8 Gigabit + 2SFP+ ports ) that are linked to my pfSense Box. Im soon adding a 10Gigabit card to the pfSense box and would like to upgrade the backbone to 10Gigabit but that would require a 10Gigabit switch. IF i can achieve this with my unRaid box in software then i wont bother purchasing a $400 Switch ( Mikrotik CRS309-1G-8S+ ) and can sink it into something like a 3080 :)

Link to post
  • 2 months later...
On 12/5/2020 at 10:57 PM, _Adrian_ said:

I have no interest in infiniband but rather if unRaid can bifurcate the 40G ports ( QSFP - 4x 10GBe ) in order to eliminate the need for a core network switch and use my unraid box as a Router/10G switch. 

As far as I am aware this isn't even a question if UnRaid supports it or not. The network cards don't support it. Splitting 40G into 4x 10G is reserved for Switches and some of the newer models.

I am currently getting around 1.8 -1.5 GB/s to my NVMe RAID 1. So fairly close to @Siren. On Windows Server I can get to around 3 GB/s with RDMA and RSS working correctly which as near as makes no difference maxes out the drives. I suspect 2 GB/s is going to be a fairly hard limit without either RDMA or RSS.

Link to post
On 12/6/2020 at 5:57 AM, _Adrian_ said:

I have no interest in infiniband but rather if unRaid can bifurcate the 40G ports ( QSFP - 4x 10GBe ) in order to eliminate the need for a core network switch and use my unraid box as a Router/10G switch. I have 3 Miktotik CSS610-8G+2S+ ( 8 Gigabit + 2SFP+ ports ) that are linked to my pfSense Box. Im soon adding a 10Gigabit card to the pfSense box and would like to upgrade the backbone to 10Gigabit but that would require a 10Gigabit switch. IF i can achieve this with my unRaid box in software then i wont bother purchasing a $400 Switch ( Mikrotik CRS309-1G-8S+ ) and can sink it into something like a 3080 :)

 

Its not bifurcation in this context, it's SR-IOV that you want to look at.

Its supported in Linux, and by the X3 cards (see link), but in an Unraid context I don't know of anyone that's been successful, or if we have all the prerequisites in place, even with the Mellanox toolset installed... maybe @ich777 knows?

 

https://community.mellanox.com/s/article/howto-configure-sr-iov-for-connectx-3-with-kvm--ethernet-x

Link to post
On 2/23/2021 at 4:45 AM, tjb_altf4 said:

maybe @ich777 knows?

No, haven't done anything related to SR-IOV but after reading through the link that @tjb_altf4 has posted it should be possible if you install the Mellanox Firmware Tools if you are on 6.9.0RC2 from the CA App (be sure to replace 'mlxconfing' with 'mstconfig' my package includes the open source version of the Mellanox Firmware Tools).

Link to post
  • 3 months later...

JFYI:

I've been playing with a cx3, the (mainline) in-kernel driver is v4.0.0 which contains a bug which makes sriov not possible for windows guests (other guests work well). This bug was fixed with mlx4 v4.3.x which is only available via OFED.

Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.