What I learned about installing a Mellanox ConnectX-3

[rubicon]

I'm relatively new to Unraid and owe a huge thank you to everyone who contributes on this forum. I've got Unraid running on an old QNAP TS-453A and custom PC build and couldn't be happier with the results.

 

I recently installed a Mellanox ConnectX-3 adapter in my PC server to go along with a new Mac Studio which has a 10 GbE port. There is a lot of information here, and elsewhere, about getting the ConnectX-3 working in Unraid but I got sidelined with details I don't recall seeing anywhere. Or perhaps I needed to do more homework. Regardless, here is what I've learned. Apologies if I'm not 100% accurate with naming devices, parts, etc.

 

I bought a Mellanox ConnectX-3, model MCX354A-QCBT. It has two slots for network transceivers. The one I bought came with a SAS expander which was an option I didn't know existed. Presumably, you'd connect a cable from your existing SAS card (i.e.: LSI HBA) to this port for external SAS connection. But SAS is a topic for another post.

 

The ports on the ConnectX-3 support QSFP transceivers for multiple InfiniBand (IB) connections at 40 Gb. It's my understanding Unraid doesn't support IB therefore the card needs to run at 10 Gb (10Base-whatever).

Updating the Firmware

I used the Mellanox Firmware Tools plug-in from Community Apps to update the firmware to the latest version. Searching for the firmware took me on a web page redirection journey to Mellanox and nVidia branded websites, but I found it. Installing the firmware using the Unraid plug-in was easy.

Change the Port Protocol to Ethernet

The ConnectX-3 defaults to an Auto interface mode to switch between IB and Ethernet. Unraid doesn't recognize the Auto mode, therefore won't select Ethernet automatically. Forum posts indicate the card settings need to be changed from Auto to Ethernet. This is easier said than done because the Mellanox Firmware Tool plug-in doesn't provide this functionality.

 

I decided to boot from an Ubuntu USB drive (on the NAS), install the Linux version of the Mellanox firmware tools, and change the mode. I wasn't successful because the versions of the firmware installers and Ubuntu (20 and 23) wouldn't cooperate. I moved the ConnectX-3 card to a Windows PC and installed the official Mellanox Firmware Tools on that system. The directions at this site were the solution. Below are the key points if that page vanishes.

 

Open Windows Terminal, PowerShell, etc. and enter this commend to ensure the card is recognized and note the name for it.

 

PS C:\Users\Administrator> mst status
MST devices:
------------
  mt4099_pci_cr0
  mt4099_pciconf0

 

In System Devices, look for the ConnectX-3 card and open the Port Protocol tab. It will show Auto, IB, or ETH. Most likely the options are greyed-out and cannot be changed.

 

Go back to Windows Terminal and enter the following command and be sure to use the name you found above.

 

PS C:\Users\Administrator\Desktop> mlxconfig -d mt4099_pci_cr0 query
 
Device #1:
----------
 
Device type:    ConnectX3
Device:         mt4099_pci_cr0
 
Configurations:                              Next Boot
         SRIOV_EN                            False(0)
         NUM_OF_VFS                          8
         LINK_TYPE_P1                        IB(1)
         LINK_TYPE_P2                        IB(1)
         LOG_BAR_SIZE                        3
         BOOT_PKEY_P1                        0
         BOOT_PKEY_P2                        0
(snip)

 

We need to change the Link Type values on the card firmware so it defaults to Ethernet. At Windows Terminal, we'll set the Link Type for both ports to Ethernet which is value 2.

 

PS C:\Users\Administrator\Desktop> mlxconfig -d mt4099_pci_cr0 set LINK_TYPE_P1=2 LINK_TYPE_P2=2
 
Device #1:
----------
 
Device type:    ConnectX3
Device:         mt4099_pci_cr0
 
Configurations:                              Next Boot       New
         LINK_TYPE_P1                        IB(1)           ETH(2)
         LINK_TYPE_P2                        IB(1)           ETH(2)
 
 Apply new Configuration? (y/n) [n] : y
Applying... Done!
-I- Please reboot machine to load new configurations.

 

I rebooted the PC, re-checked the ConnectX-3 Port Protocol tab, and it was selectable and set to Ethernet. I moved the card back into the NAS and booted Unraid. It recognized both ports as 10 Gb.

Transceiver Types

Earlier in this post I mentioned the ConnectX-3 uses QSFP transceivers. During my research about the ConnectX-3 card and Unraid, I don't recall reading anything about transceiver types - and not knowing any better - I bought the only 10 GbE transceiver I could find which was SFP+. The SFP+ transceiver is smaller and doesn't fit in the QSFP slot.

I couldn't find a single QSFP transceiver which supports Ethernet, but there was a QSFP to SFP+ adapter. I inserted the SFP+ into the QSFP adapter, QSFP into the ConnectX-3 port, and then connected the Mac Studio 10 GbE (RJ-45 copper, Cat-8) cable into the NAS. Success.

I'm happy to have it working but I probably spent more $ on it than expected.

 

There are other multi-port Mellanox cards which support 10GbE using traditional RJ-45 connectors - no transceivers needed - but maybe those aren't supported in Unraid.

 

I've also found breakout cables which are QSFP to SFP+ which might be more cost effective for connecting to an SFP+ Ethernet switch.

 

I hope this information is useful for anyone looking to use the ConnectX-3 cards.