Mellanox 10GBe Adapter Problem

[Srengr]

Hi All, I am trying to setup a direct connect 10GBe link between my to unRAID servers.  Both are running v6.4.1.  I installed a Mellanox Connectx 2 card in each machine. 

 

unRAID is not recognizing the cards as ethernet adapters:

 

IOMMU group 10:

[15b3:0191] 01:00.0 Memory controller: Mellanox Technologies MT25408 [ConnectX IB Flash Recovery] (rev b0)

Any ideas?

 

Thanks, srengr

[Ambrotos]

What is the exact model of the ConnectX card you got? Are you sure you didn't get one of their Infiniband or FibreChannel cards instead of an Ethernet card? Does yours say "MNPA19-XTR" anywhere on it? My ConnectX-2 detects as follows in the system devices list:

 

[15b3:6750] 01:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)

 

[Ambrotos]

Actually, assuming you did get the correct Ethernet variant of the ConnectX-2, it's possible your card might be in Firmware Recovery mode. Some quick Googling suggests that the "ConnectX IB Flash Recovery" device shows up if your card failed to properly load its firmware image.

 

Maybe you could try flashing the latest firmware image into the card using the flint utility.

 

http://www.mellanox.com/page/management_tools

 

...although Mellanox only makes DEB and RPM installers available for Linux, so unless you can find a pre-compiled binary somewhere it might be easier to move the card to a Windows machine to try to recover the firmware.

[Srengr]

13 minutes ago, Ambrotos said:

Actually, assuming you did get the correct Ethernet variant of the ConnectX-2, it's possible your card might be in Firmware Recovery mode. Some quick Googling suggests that the "ConnectX IB Flash Recovery" device shows up if your card failed to properly load its firmware image.

 

Maybe you could try flashing the latest firmware image into the card using the flint utility.

 

http://www.mellanox.com/page/management_tools

 

...although Mellanox only makes DEB and RPM installers available for Linux, so unless you can find a pre-compiled binary somewhere it might be easier to move the card to a Windows machine to try to recover the firmware.

 

Both cards say MNPA19-XTR.  I'll try to recover the firmware on my Win machine.

 

Thanks, srengr

[Srengr]

Windows wouldn't recognize that the cards were even installed.  I think they are bricked.

 

My donation to the www for the week...

 

Thanks, srengr

[Ambrotos]

I'm not sure what drivers (if any) Windows would load if the card were in recovery mode. What is shown if you execute "mst status" from the WinMFT installation directory? If you get any Mellanox device names listed there, you can do a device query to see if it's in "failsafe" mode with "flint -d /dev/mst/<device_name> query"

 

http://www.mellanox.com/page/firmware_HCA_FW_identification

 

If flint can identify it, there's a good chance it can flash it.

 

Failing all that, you're probably right that the card's a brick. You don't sound optimistic that you'll be able to get your money back. I guess you didn't buy it on eBay, where worst case if you get a crusty vendor you can exercise your buyer protection?

 

Cheers, 

 

-A

[Srengr]

13 hours ago, Ambrotos said:

I'm not sure what drivers (if any) Windows would load if the card were in recovery mode. What is shown if you execute "mst status" from the WinMFT installation directory? If you get any Mellanox device names listed there, you can do a device query to see if it's in "failsafe" mode with "flint -d /dev/mst/<device_name> query"

 

http://www.mellanox.com/page/firmware_HCA_FW_identification

 

If flint can identify it, there's a good chance it can flash it.

 

Failing all that, you're probably right that the card's a brick. You don't sound optimistic that you'll be able to get your money back. I guess you didn't buy it on eBay, where worst case if you get a crusty vendor you can exercise your buyer protection?

 

Cheers, 

 

-A

 

Mst status says no device found, or something like that.  I will try to RMD them.  Got them on Amazon.

 

Thanks, srengr

[Can0n]

On 2/26/2018 at 4:36 PM, Ambrotos said:

What is the exact model of the ConnectX card you got? Are you sure you didn't get one of their Infiniband or FibreChannel cards instead of an Ethernet card? Does yours say "MNPA19-XTR" anywhere on it? My ConnectX-2 detects as follows in the system devices list:

 

[15b3:6750] 01:00.0 Ethernet controller: Mellanox Technologies MT26448 [ConnectX EN 10GigE, PCIe 2.0 5GT/s] (rev b0)

 

It appears mine i got off ebay are Infiniband (they show as Dual port but really only have one)

any way to get these to work?

 

IOMMU group 20:

[15b3:673c] 08:00.0 InfiniBand: Mellanox Technologies MT25408A0-FCC-QI ConnectX, Dual Port 40Gb/s InfiniBand / 10GigE Adapter IC with PCIe 2.0 x8 5.0GT/s In... (rev b0)

[eschultz]

1 hour ago, Can0nfan said:

It appears mine i got off ebay are Infiniband (they show as Dual port but really only have one)

any way to get these to work?

 

IOMMU group 20:

[15b3:673c] 08:00.0 InfiniBand: Mellanox Technologies MT25408A0-FCC-QI ConnectX, Dual Port 40Gb/s InfiniBand / 10GigE Adapter IC with PCIe 2.0 x8 5.0GT/s In... (rev b0)

Unfortunately we don't have Infiniband or FibreChannel drivers in the Unraid kernel at this time.

[Can0n]

3 minutes ago, eschultz said:

Unfortunately we don't have Infiniband or FibreChannel drivers in the Unraid kernel at this time.

darn i have 300 of these cards and bought a $100 DAC cable for them any chance they can be added manually if i can somehow grab the driver?

 

They were listed as Connect-X2 cards which i saw all over youtube (space invader1) and the unraid forums as compatible

Edited August 30, 2018 by Can0nfan

[eschultz]

23 minutes ago, Can0nfan said:

darn i have 300 of these cards

wow, what kind of InfiniBand switch are you using?  This sounds like poor product naming from Mellanox since there seems to be a ConnectX-2 model for Ethernet and InfiniBand.

 

If I just got two of these cards off ebay, can I just connect two machines directly using a DAC cable?  Besides the Mellanox InfiniBand driver I see another driver in the kernel 'IP-over-InfiniBand protocol'... wonder if that's needed or if your using InfiniBand you have your own identifiable addressing besides IPv4/6?  (I'm new to IB)

[Can0n]

2 hours ago, eschultz said:

wow, what kind of InfiniBand switch are you using?  This sounds like poor product naming from Mellanox since there seems to be a ConnectX-2 model for Ethernet and InfiniBand.

 

If I just got two of these cards off ebay, can I just connect two machines directly using a DAC cable?  Besides the Mellanox InfiniBand driver I see another driver in the kernel 'IP-over-InfiniBand protocol'... wonder if that's needed or if your using InfiniBand you have your own identifiable addressing besides IPv4/6?  (I'm new to IB)

no switch just won an auction from a local computer recycler here in Calgary, Alberta. i am using two, one in each of my unRAID servers and an Infiniband DAC cable i purchased from fs.com the cable cost 3x more than i paid for the 300 cards...

[Can0n]

2 hours ago, eschultz said:

 wonder if that's needed or if your using InfiniBand you have your own identifiable addressing besides IPv4/6?  (I'm new to IB)

network settings isn't seeing the Infiniband as a Network card though thats why im wondering how i can get drivers installed.

@jonp can i put a pretty please with sugar on top request for the Mellonox Infiniband drivers for unRAID 6.6??? I would be happy to send any of you or your staff some of my cards if you need them for testing...like i said i have 300 cards. ill send two each for testing if needed. PM me we can make arrangements