orlando500 Posted December 19, 2021 Share Posted December 19, 2021 i just got myself 2 Mellanox 10 cards in my unraid servers. Have made subnet for them and Connected them up and testing a bit with iperf3. Got this results who is clearly wrong... Server listening on 5201 ----------------------------------------------------------- Accepted connection from 192.168.11.140, port 53738 [ 5] local 192.168.11.20 port 5201 connected to 192.168.11.140 port 53742 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 472 KBytes 3.86 Mbits/sec 3 8.74 KBytes [ 5] 1.00-2.00 sec 0.00 Bytes 0.00 bits/sec 1 8.74 KBytes [ 5] 2.00-3.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes [ 5] 3.00-4.00 sec 0.00 Bytes 0.00 bits/sec 1 8.74 KBytes [ 5] 4.00-5.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes [ 5] 5.00-6.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes [ 5] 6.00-7.00 sec 0.00 Bytes 0.00 bits/sec 1 8.74 KBytes [ 5] 7.00-8.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes [ 5] 8.00-9.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes [ 5] 9.00-10.00 sec 0.00 Bytes 0.00 bits/sec 0 8.74 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 472 KBytes 387 Kbits/sec 6 sender how do i troubleshoot this? Quote Link to comment
Tom3 Posted December 19, 2021 Share Posted December 19, 2021 One place to start is using 'ethtool' from the UNRAID cli. It will tell you about negotiated speed, duplex, other parameters. This may tell you if something is misconfigured. # ip link show will list all the interfaces on your system (devname is usually eth0 eth1, eth2, etc.) # ethtool devname will list the parameters for that specific interface #ethtool -i devname will tell you what driver it's using # ethtool -h will give you brief help. Suggest reading up on ethtool online. https://linuxhint.com/ethtool_commands_examples/ -- Tom 1 Quote Link to comment
orlando500 Posted December 20, 2021 Author Share Posted December 20, 2021 yeah the info looks ok, i have link on the 10 gb card. root@battlestar:~# ip link show 1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 2: tunl0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ipip 0.0.0.0 brd 0.0.0.0 3: gre0@NONE: <NOARP> mtu 1476 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/gre 0.0.0.0 brd 0.0.0.0 4: gretap0@NONE: <BROADCAST,MULTICAST> mtu 1462 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff 5: erspan0@NONE: <BROADCAST,MULTICAST> mtu 1450 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ether 00:00:00:00:00:00 brd ff:ff:ff:ff:ff:ff 6: ip_vti0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/ipip 0.0.0.0 brd 0.0.0.0 7: sit0@NONE: <NOARP> mtu 1480 qdisc noop state DOWN mode DEFAULT group default qlen 1000 link/sit 0.0.0.0 brd 0.0.0.0 11: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP mode DEFAULT group default qlen 1000 link/ether 70:85:c2:28:bb:f2 brd ff:ff:ff:ff:ff:ff 12: eth1: <BROADCAST,MULTICAST> mtu 1500 qdisc fq state DOWN mode DEFAULT group default qlen 1000 link/ether 70:85:c2:28:bb:f4 brd ff:ff:ff:ff:ff:ff 13: eth2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 9004 qdisc mq master br2 state UP mode DEFAULT group default qlen 1000 link/ether e4:1d:2d:7f:22:f0 brd ff:ff:ff:ff:ff:ff 30: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether 70:85:c2:28:bb:f2 brd ff:ff:ff:ff:ff:ff 34: eth2.3010@eth2: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 9004 qdisc noqueue master br2.3010 state UP mode DEFAULT group default qlen 1000 link/ether e4:1d:2d:7f:22:f0 brd ff:ff:ff:ff:ff:ff 35: br2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9004 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether e4:1d:2d:7f:22:f0 brd ff:ff:ff:ff:ff:ff 36: br2.3010: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9004 qdisc noqueue state UP mode DEFAULT group default qlen 1000 link/ether e4:1d:2d:7f:22:f0 brd ff:ff:ff:ff:ff:ff 37: docker0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default link/ether 02:42:fb:bd:60:82 brd ff:ff:ff:ff:ff:ff 40: tailscale0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1280 qdisc fq state UNKNOWN mode DEFAULT group default qlen 500 link/none 42: veth21c8498@if41: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT group default link/ether 5a:4e:e9:75:c6:1d brd ff:ff:ff:ff:ff:ff link-netnsid 1 52: veth1ebc0d3@if51: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP mode DEFAULT group default link/ether ea:00:4f:15:b4:a1 brd ff:ff:ff:ff:ff:ff link-netnsid 0 root@battlestar:~# root@battlestar:~# ethtool eth0 Settings for eth0: Supported ports: [ TP ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Supported pause frame use: Symmetric Supports auto-negotiation: Yes Supported FEC modes: Not reported Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full Advertised pause frame use: Symmetric Advertised auto-negotiation: Yes Advertised FEC modes: Not reported Speed: 1000Mb/s Duplex: Full Auto-negotiation: on Port: Twisted Pair PHYAD: 1 Transceiver: internal MDI-X: on (auto) Supports Wake-on: pumbg Wake-on: g Current message level: 0x00000007 (7) drv probe link Link detected: yes root@battlestar:~# ethtool eth2 Settings for eth2: Supported ports: [ FIBRE ] Supported link modes: 1000baseKX/Full 10000baseKR/Full Supported pause frame use: Symmetric Receive-only Supports auto-negotiation: No Supported FEC modes: Not reported Advertised link modes: 1000baseKX/Full 10000baseKR/Full Advertised pause frame use: Symmetric Advertised auto-negotiation: No Advertised FEC modes: Not reported Speed: 10000Mb/s Duplex: Full Auto-negotiation: off Port: FIBRE PHYAD: 0 Transceiver: internal Supports Wake-on: d Wake-on: d Current message level: 0x00000014 (20) link ifdown Link detected: yes Quote Link to comment
Tom3 Posted December 20, 2021 Share Posted December 20, 2021 Hi Orlando - interesting. ethtool reports 10000baseKR (which is the single lane backplane electrical interface version of 10GbE). The coding on that is the same as 10GBaseSR, so it's likely just an unusual way to report the interface. You don't report which driver the 10G interface is using. (ethtool -i devname) Not all NIC cards are supported in Linux, you need to check to see if the driver and card you are using is supported by UNRAID. With optical interfaces there are some compatibility issues that need to be taken account of: 1. Both ends need to be compatible (same wavelength, same fiber type, correct fiber type). If you are using 10GBaseSR (short reach) multimode fiber (MMF) then the fiber needs to be the right type. 10GBaseSR is not too stressful on the fiber characteristics, I think up to about 500m is normally OK. The two optical modules need to be compatible. 2. That interface can support FEC (forward error correction). FEC needs to be set the same (On / Off) at both ends. Depending on the software, it may need to be manually configured. Your card may or may not support FEC - ethtool is showing 'Not reported'. Perhaps you may need to set if to Off at both ends. 3. Fiber connector cleanliness is critical. Dirty connectors are far and away the most common problem with fiber and can cause all kinds of strange symptoms. Both the fiber end-face and the barrels need to be clean and unscratched.. 4. ethtool message on eth2 reports that the interface is down, but ip show reports it up. -- Tom 1 Quote Link to comment
orlando500 Posted December 20, 2021 Author Share Posted December 20, 2021 53 minutes ago, Tom3 said: Hi Orlando - interesting. ethtool reports 10000baseKR (which is the single lane backplane electrical interface version of 10GbE). The coding on that is the same as 10GBaseSR, so it's likely just an unusual way to report the interface. You don't report which driver the 10G interface is using. (ethtool -i devname) Not all NIC cards are supported in Linux, you need to check to see if the driver and card you are using is supported by UNRAID. With optical interfaces there are some compatibility issues that need to be taken account of: 1. Both ends need to be compatible (same wavelength, same fiber type, correct fiber type). If you are using 10GBaseSR (short reach) multimode fiber (MMF) then the fiber needs to be the right type. 10GBaseSR is not too stressful on the fiber characteristics, I think up to about 500m is normally OK. The two optical modules need to be compatible. 2. That interface can support FEC (forward error correction). FEC needs to be set the same (On / Off) at both ends. Depending on the software, it may need to be manually configured. Your card may or may not support FEC - ethtool is showing 'Not reported'. Perhaps you may need to set if to Off at both ends. 3. Fiber connector cleanliness is critical. Dirty connectors are far and away the most common problem with fiber and can cause all kinds of strange symptoms. Both the fiber end-face and the barrels need to be clean and unscratched.. 4. ethtool message on eth2 reports that the interface is down, but ip show reports it up. -- Tom im using optical patch cord im3-lc-lc-d1m om3 LC to LC MMF duplex 1m from 10gtek. all new, and all new 10gb sfp+ from https://www.amazon.co.uk/gp/product/B013WFH7C8/ref=ppx_yo_dt_b_asin_title_o00_s01?ie=UTF8&psc=1 Quote Link to comment
orlando500 Posted December 20, 2021 Author Share Posted December 20, 2021 (edited) 1 hour ago, Tom3 said: Here are from the other server on type of card and driver: same driver on both of them. root@Tower:~# ethtool eth1 Settings for eth1: Supported ports: [ FIBRE ] Supported link modes: 1000baseKX/Full 10000baseKR/Full Supported pause frame use: Symmetric Receive-only Supports auto-negotiation: No Supported FEC modes: Not reported Advertised link modes: 1000baseKX/Full 10000baseKR/Full Advertised pause frame use: Symmetric Advertised auto-negotiation: No Advertised FEC modes: Not reported Speed: 10000Mb/s Duplex: Full Auto-negotiation: off Port: FIBRE PHYAD: 0 Transceiver: internal Supports Wake-on: d Wake-on: d Current message level: 0x00000014 (20) link ifdown Link detected: yes ethtool -l eth1: driver: mlx4_en version: 4.0-0 firmware-version: 2.42.5000 expansion-rom-version: bus-info: 0000:0b:00.0 supports-statistics: yes supports-test: yes supports-eeprom-access: no supports-register-dump: no supports-priv-flags: yes btw: i use the same sfp+ in the microtik 10gb switch inbetween the servers and the UDM pro Edited December 20, 2021 by orlando500 Quote Link to comment
Tom3 Posted December 20, 2021 Share Posted December 20, 2021 Hi Orlando - it looks like the Mellanox MCX311A-XCAT has been reported by others to work correctly in Unraid (based on google search): Unraid Forum Post Reddit Post I'm puzzled why ethtool on both systems reports the interface (if) as link down. -- Tom Quote Link to comment
orlando500 Posted December 20, 2021 Author Share Posted December 20, 2021 3 hours ago, Tom3 said: Hi Orlando - it looks like the Mellanox MCX311A-XCAT has been reported by others to work correctly in Unraid (based on google search): Unraid Forum Post Reddit Post I'm puzzled why ethtool on both systems reports the interface (if) as link down. -- Tom yeah good question. AS im getting traffic when i disconnect the 1 gb cable. But the speed was not there. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.