Jump to content
unsainted

10Gbe Mellanox issues

13 posts in this topic Last Reply

Recommended Posts

I purchased 2 Mellanox 10GBe cards from ebay:

 

Link: https://www.ebay.com/itm/MNPA19-XTR-10GB-MELLANOX-CONNECTX-2-PCIe-X8-10Gbe-SFP-NETWORK-CARD/131634470127?ssPageName=STRK%3AMEBIDX%3AIT&_trksid=p2060353.m2749.l2649

 

I also purchased this cable from ebay: https://www.ebay.com/itm/SFP-H10GB-CU3M-CISCO-SFP-TWINAX-COPPER-CABLE-3M-PASSIVE-DAC-CABLE/282060432334?ssPageName=STRK%3AMEBIDX%3AIT&_trksid=p2060353.m2749.l2649

 

unRaid detects the cards and loads drivers and I have assigned them IP addresses on a separate subnet. My goal is to use 10GBe connection to backup Primary unRaid server to a backup unRaid server.

 

I see blinking lights when cable is plugged in and cards are online, but i am unable to ping between them. DId I purchase the wrong type of cable?

 

Not sure where to look to troubleshoot but appears that there is no link. I can ping each cards own IP from the server its connected to, but cant ping across to the other.

 

Any assistance is appreciated.

 

 

Share this post


Link to post

Looks like to correct cable to me, if you run:

 

ethtool ethX

 

Do you get link detected? (replace X with correct number)

Share this post


Link to post

root@JUPITER:~# ethtool eth2
Settings for eth2:
        Supported ports: [ FIBRE ]
        Supported link modes:   10000baseT/Full
        Supported pause frame use: No
        Supports auto-negotiation: No
        Supported FEC modes: Not reported
        Advertised link modes:  10000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Advertised FEC modes: Not reported
        Speed: Unknown!
        Duplex: Unknown! (255)
        Port: FIBRE
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: off
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000014 (20)
                               link ifdown
        Link detected: no

 

 

appears no link.

Share this post


Link to post

Yes, I use the same NIC but between Windows and unRAID, and since there's no switch sometimes after booting or rebooting no link is detect, I just disable and re-enable the NIC in Windows and the link is detected again, you might need to do the same on unRAID, it should be enough to go to Settings -> Network Settings and click "Link Down" on the correct interface, then "Link Up"

Share this post


Link to post

i ran ifup eth2 and then ethtool eth2 again and now appears link is up, but still unable to ping.

 

 for eth2:
        Supported ports: [ FIBRE ]
        Supported link modes:   10000baseT/Full
        Supported pause frame use: No
        Supports auto-negotiation: No
        Supported FEC modes: Not reported
        Advertised link modes:  10000baseT/Full
        Advertised pause frame use: No
        Advertised auto-negotiation: No
        Advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Full
        Port: FIBRE
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: off
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000014 (20)
                               link ifdown
        Link detected: yes

Share this post


Link to post

With link up it should work, make sure the IP addresses are correctly configured.

Share this post


Link to post

Here is config. Seems that when both interfaces are up, i run ping command, it fails and then takes link back down.

 

 

1135334663_ScreenShot2018-07-09at11_19_55AM.thumb.png.6d0b6267ace625c8c6ece9b80d8bec67.png830066125_ScreenShot2018-07-09at11_19_32AM.thumb.png.ec389f498f89d87be8103f7380fe3ef4.png

Share this post


Link to post

weird, seems every now and then it will ping successfully.. Thoughts?

Screen Shot 2018-07-09 at 11.34.38 AM.png

Share this post


Link to post
19 minutes ago, unsainted said:

Thoughts?

Not really, maybe if you can try between unRAID and a Windows desktop to see if it's the same.

Share this post


Link to post

thanks for the help as always, I will see if i can move my desktop around to get them connected.

 

Share this post


Link to post

one other thing of note. I ran tail -f /var/log/syslog on each system..

Screen Shot 2018-07-09 at 1.00.08 PM.png

Share this post


Link to post

Yep, link is not staying up, that would explain it, but no clue why it's happening, I never used them between 2 unRAID server, but I would expect that once up the link would stay up, until possibly a reboot or power cycle.

Share this post


Link to post
8 minutes ago, johnnie.black said:

Yep, link is not staying up, that would explain it, but no clue why it's happening, I never used them between 2 unRAID server, but I would expect that once up the link would stay up, until possibly a reboot or power cycle.

 

 

I've used these same cards between 2 unraid servers for backup. Once its up, it was rock solid. That is, until one card started to die. Then it would drop connection....

Share this post


Link to post

Create an account or sign in to comment

You need to be a member in order to leave a comment

Create an account

Sign up for a new account in our community. It's easy!

Register a new account

Sign in

Already have an account? Sign in here.

Sign In Now