Mellanox MHQH19B-XTR


Leondre

Recommended Posts

Does anyone have experience using this with unraid? I've got it showing up as a pci device, however it is not showing up as an ethernet device so I don't see an interface. It looks like it can operate in either infiniband or ethernet modes, but swapping between the two is a bit beyond me in unraid at this point. Any tips?

06:00.0 InfiniBand [0c06]: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] [15b3:673c] (rev b0)
                Subsystem: Mellanox Technologies MT26428 [ConnectX VPI PCIe 2.0 5GT/s - IB QDR / 10GigE] [15b3:0022]
                Kernel driver in use: mlx4_core
                Kernel modules: mlx4_core

 

 

Edited by Leondre
Link to comment

The ethernet models work great, link for the mode change if for the connectx-3, are you sure it also works for the connectx-2?

 

From the manual:

 

"The ConnectX-2 adapter card is compliant with the IEEE Std 802.3. The VPI adapter cards listed above, based on this device, provide access to the Ethernet ports by means of QSFP connectors.The Mellanox QSA (QSFP to SFP+) adapter modules can be ordered through Mellanox Technologies authorized dealers."

 

Do you have the adapter? If not probably cheaper to get the the ethernet connectx-2 on ebay, they are available for under 25$.

Link to comment
  • 7 months later...
  • 3 months later...

Sorry to resurrect an old thread, did you have any luck getting a ConnectX-2 VPI card working with unRAID @wheelbase or @Leondre ?

 

 

From my limited linux experience it appears that the mlx4_core module is loaded and used b mlx4_en, however attempting to load mlx4_ib fails iwth error:

 

Quote

FATAL: Module mlx4_ib not found in directory /lib/modules/4.14.35-unRAID

 

 

Searching suggests that I need to compile a new MLNX_OFED kernel module that includes mlx4_ib, which looks like a steep learning curve.  I'll try changing the port mode in the first instance as the Mellanox documented instructions are pretty straight forward...

 

Quote

Port Type Management/VPI Cards Configuration:
Device ports can be individually configured to work as InfiniBand or Ethernet ports. By default, device ports are initialized as InfiniBand ports. If you wish to change the port type, use the mstflint tool after the driver is loaded.

5. Install mstflint tools:

  • Zypper install mstflint.

6. Check the PCI address.

  • # lspci | grep Mellanox
  • 00:06.0 Infiniband controller: Mellanox Technologies MT27520 Family [ConnectX-3 Pro]

7. Use mstconfig to change the link type as desired IB – for InfiniBand, ETH – for Ethernet.

  • # mstconfig –d s LINK_TYPE_P1/2=
  • Example: # mstconfig -d 82:00.1 s LINK_TYPE_P1=ETH

8. Reboot your machine.

Edited by fredhoon
Link to comment

After doing a bit more digging, the current version of the mlx_core module has dropped support for ConnectX-2 cards, I'm not sure what version unRAID runs however I would assume it to be fairly recent due to the kernel version.  It would be a very steep learning curve for me to be able to compile and install a previous version of the OFED driver and possibly even require an older kernel / version of unRAID.

 

I tried a ConnectX-3 card which was detected by unRAID but only provided a single network interface despite being a dual port VPI card.  Both ports were in VPI / auto detect mode and despite being directly connected via a QSFP DAC cable to an identical CX-3 card in another machine, the card didn't show a link light.  Compiling / resolving dependencies / installing the mft utilities in unRAID (or even mint) was beyond me so I threw the card in a windows machine and changed Port 1 from "Auto / VPI" to "Eth" & did the same for the card at the other end - now I have a 40Gb/s ethernet link from UnRAID to my Win7 Games / backup box!

 

 

My cache SSDs should be good for 2-300MB/s and Turbo writes for 150MB/s so I can use at least some of the CX-3 bandwidh.

Edited by fredhoon
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.