10gb network... how to get speed


Recommended Posts

i just got myself 2 Mellanox 10 cards in my unraid servers. Have made subnet for them and Connected them up and testing a bit with iperf3. 

Got this results who is clearly wrong...

 

Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.11.140, port 53738
[  5] local 192.168.11.20 port 5201 connected to 192.168.11.140 port 53742
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   472 KBytes  3.86 Mbits/sec    3   8.74 KBytes       
[  5]   1.00-2.00   sec  0.00 Bytes  0.00 bits/sec    1   8.74 KBytes       
[  5]   2.00-3.00   sec  0.00 Bytes  0.00 bits/sec    0   8.74 KBytes       
[  5]   3.00-4.00   sec  0.00 Bytes  0.00 bits/sec    1   8.74 KBytes       
[  5]   4.00-5.00   sec  0.00 Bytes  0.00 bits/sec    0   8.74 KBytes       
[  5]   5.00-6.00   sec  0.00 Bytes  0.00 bits/sec    0   8.74 KBytes       
[  5]   6.00-7.00   sec  0.00 Bytes  0.00 bits/sec    1   8.74 KBytes       
[  5]   7.00-8.00   sec  0.00 Bytes  0.00 bits/sec    0   8.74 KBytes       
[  5]   8.00-9.00   sec  0.00 Bytes  0.00 bits/sec    0   8.74 KBytes       
[  5]   9.00-10.00  sec  0.00 Bytes  0.00 bits/sec    0   8.74 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   472 KBytes   387 Kbits/sec    6             sender

 

 

how do i troubleshoot this? 

Link to comment

One place to start is using 'ethtool' from the UNRAID cli.  It will

tell you about negotiated speed, duplex, other parameters.  This

may tell you if something is misconfigured.

 

# ip link show

will list all the interfaces on your system (devname is usually eth0  eth1, eth2,  etc.)

 

# ethtool devname

will list the parameters for that specific interface

 

#ethtool -i devname

will tell you what driver it's using 

 

# ethtool -h

will give you brief help.  Suggest reading up on ethtool online.

https://linuxhint.com/ethtool_commands_examples/

 

-- Tom

 

 

  • Thanks 1
Link to comment

yeah the info looks ok, i have link on the 10 gb card.

 

root@battlestar:~# ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
3: gre0@NONE: <NOARP> mtu 1476 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/gre 0.0.0.0 brd 0.0.0.0
4: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
5: erspan0@NONE: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff
6: ip_vti0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/ipip 0.0.0.0 brd 0.0.0.0
7: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default qlen 1000
    link/sit 0.0.0.0 brd 0.0.0.0
11: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP mode DEFAULT group default qlen 1000
    link/ether 70:85:c2:28:bb:f2 brd ff:ff:ff:ff:ff:ff
12: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc fq state DOWN mode DEFAULT group default qlen 1000
    link/ether 70:85:c2:28:bb:f4 brd ff:ff:ff:ff:ff:ff
13: eth2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 9004 qdisc mq master br2 state UP mode DEFAULT group default qlen 1000
    link/ether e4:1d:2d:7f:22:f0 brd ff:ff:ff:ff:ff:ff
30: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 70:85:c2:28:bb:f2 brd ff:ff:ff:ff:ff:ff
34: eth2.3010@eth2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 9004 qdisc noqueue master br2.3010 state UP mode DEFAULT group default qlen 1000
    link/ether e4:1d:2d:7f:22:f0 brd ff:ff:ff:ff:ff:ff
35: br2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9004 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether e4:1d:2d:7f:22:f0 brd ff:ff:ff:ff:ff:ff
36: br2.3010: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9004 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether e4:1d:2d:7f:22:f0 brd ff:ff:ff:ff:ff:ff
37: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default 
    link/ether 02:42:fb:bd:60:82 brd ff:ff:ff:ff:ff:ff
40: tailscale0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc fq state UNKNOWN mode DEFAULT group default qlen 500
    link/none 
42: veth21c8498@if41: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT group default 
    link/ether 5a:4e:e9:75:c6:1d brd ff:ff:ff:ff:ff:ff link-netnsid 1
52: veth1ebc0d3@if51: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT group default 
    link/ether ea:00:4f:15:b4:a1 brd ff:ff:ff:ff:ff:ff link-netnsid 0
root@battlestar:~# 

root@battlestar:~# ethtool eth0
Settings for eth0:
        Supported ports: [ TP ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Supported pause frame use: Symmetric
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Speed: 1000Mb/s
        Duplex: Full
        Auto-negotiation: on
        Port: Twisted Pair
        PHYAD: 1
        Transceiver: internal
        MDI-X: on (auto)
        Supports Wake-on: pumbg
        Wake-on: g
        Current message level: 0x00000007 (7)
                               drv probe link
        Link detected: yes

 

root@battlestar:~# ethtool eth2
Settings for eth2:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseKX/Full
                                10000baseKR/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: No
        Supported FEC modes: Not reported
        Advertised link modes:  1000baseKX/Full
                                10000baseKR/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: No
        Advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Full
        Auto-negotiation: off
        Port: FIBRE
        PHYAD: 0
        Transceiver: internal
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000014 (20)
                               link ifdown
        Link detected: yes

 

 

Link to comment

Hi Orlando - interesting.    ethtool reports 10000baseKR (which is the single lane

backplane electrical interface version of 10GbE).  The coding on that is the same as

10GBaseSR, so it's likely just an unusual way to report the interface.

 

You don't report which driver the 10G interface is using. (ethtool -i devname)

Not all NIC cards are supported in Linux, you need to check to see if the driver and card

you are using is supported by UNRAID.

 

With optical interfaces there are some compatibility issues that need to be taken account

of:

 

1. Both ends need to be compatible (same wavelength, same fiber type, correct fiber type).

If you are using 10GBaseSR (short reach) multimode fiber (MMF) then the fiber needs to be the

right type.  10GBaseSR is not too stressful on the fiber characteristics, I think up to about
500m is normally OK.  The two optical modules need to be compatible.

 

2. That interface can support FEC (forward error correction).  FEC needs to be set the

same (On / Off) at both ends. Depending on the software, it may need to be manually

configured.  Your card may or may not support FEC - ethtool is showing 'Not reported'.

Perhaps you may need to set if to Off at both ends.

 

3. Fiber connector cleanliness is critical. Dirty connectors are far and away the most

common problem with fiber and can cause all kinds of strange symptoms. Both the fiber

end-face and the barrels need to be clean and unscratched..

 

4. ethtool message on eth2 reports that the interface is down, but ip show reports it up.

 

-- Tom

 

  • Thanks 1
Link to comment
53 minutes ago, Tom3 said:

Hi Orlando - interesting.    ethtool reports 10000baseKR (which is the single lane

backplane electrical interface version of 10GbE).  The coding on that is the same as

10GBaseSR, so it's likely just an unusual way to report the interface.

 

You don't report which driver the 10G interface is using. (ethtool -i devname)

Not all NIC cards are supported in Linux, you need to check to see if the driver and card

you are using is supported by UNRAID.

 

With optical interfaces there are some compatibility issues that need to be taken account

of:

 

1. Both ends need to be compatible (same wavelength, same fiber type, correct fiber type).

If you are using 10GBaseSR (short reach) multimode fiber (MMF) then the fiber needs to be the

right type.  10GBaseSR is not too stressful on the fiber characteristics, I think up to about
500m is normally OK.  The two optical modules need to be compatible.

 

2. That interface can support FEC (forward error correction).  FEC needs to be set the

same (On / Off) at both ends. Depending on the software, it may need to be manually

configured.  Your card may or may not support FEC - ethtool is showing 'Not reported'.

Perhaps you may need to set if to Off at both ends.

 

3. Fiber connector cleanliness is critical. Dirty connectors are far and away the most

common problem with fiber and can cause all kinds of strange symptoms. Both the fiber

end-face and the barrels need to be clean and unscratched..

 

4. ethtool message on eth2 reports that the interface is down, but ip show reports it up.

 

-- Tom

 

 

im using optical patch cord im3-lc-lc-d1m om3 LC to LC MMF duplex  1m from 10gtek. all new, and all new 10gb sfp+ from https://www.amazon.co.uk/gp/product/B013WFH7C8/ref=ppx_yo_dt_b_asin_title_o00_s01?ie=UTF8&psc=1

 

Link to comment
1 hour ago, Tom3 said:

Here are from the other server on type of card and driver: same driver on both of them.

 

root@Tower:~# ethtool eth1
Settings for eth1:
        Supported ports: [ FIBRE ]
        Supported link modes:   1000baseKX/Full
                                10000baseKR/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: No
        Supported FEC modes: Not reported
        Advertised link modes:  1000baseKX/Full
                                10000baseKR/Full
        Advertised pause frame use: Symmetric
        Advertised auto-negotiation: No
        Advertised FEC modes: Not reported
        Speed: 10000Mb/s
        Duplex: Full
        Auto-negotiation: off
        Port: FIBRE
        PHYAD: 0
        Transceiver: internal
        Supports Wake-on: d
        Wake-on: d
        Current message level: 0x00000014 (20)
                               link ifdown
        Link detected: yes

 

ethtool -l eth1:

 

driver: mlx4_en
version: 4.0-0
firmware-version: 2.42.5000
expansion-rom-version: 
bus-info: 0000:0b:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: no
supports-register-dump: no
supports-priv-flags: yes

 

btw: i use the same sfp+ in the microtik 10gb switch inbetween the servers and the UDM pro 

Edited by orlando500
Link to comment
3 hours ago, Tom3 said:

Hi Orlando - it looks like the Mellanox MCX311A-XCAT has been reported by others to work

correctly in Unraid (based on google search):

 

Unraid Forum Post

Reddit Post

 

I'm puzzled why ethtool on both systems reports the interface (if) as link down.

 

-- Tom

 

 

 

yeah good question. AS im getting traffic when i disconnect the 1 gb cable. But the speed was not there.

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.