December 21, 20241 yr Recently upgraded part of my network to 10Gb. My router and Unraid server had a Mellanox MCX322A card installed and they're connected with OM3 fiber. There's a 5x2.5GE+2x10G sfp+ Tenda switch in between, both router and Unraid server are connected to the sfp+ ports. LC-LC transceivers on each side. The router interface reports 10Gbase-CX4 instead of 10G-SR, but I guess that's not that big of a deal. UnRaid reports 10000Mbps, full duplex and 1500 mtu. The problem is when testing the bandwidth with iperf, the results speed show 1Gbps at best and average around 750-800 Mbps!! Prior to "upgrading" my network, iperf tests averaged at 950Mbps. Not sure how to proceed, is there anything I should edit on the Unraid side to work with Mellanox cards? They were detected just fine, plug and play.
December 21, 20241 yr You may want to check whether both SFP+ optics modules are the same type. 10Gbase-CX4 is a very old standard, it uses 4 copper lanes over a bulky twinax cable. My recollection is that there are also some format differences going back that far. Run the ethtool command on each interface, for example: $ ethtool eth0 and see what it reports. If both modules are actually SFP+ and are both labeled 10Gbase-SR, and ethtool reports that one is -CX4, then it's possible that module is damaged, had its MDIO overwritten, or you may be selecting the wrong interface (i.e. asking about eth1 when perhaps the module is plugged into eth2 or some such). -- Tom
December 21, 20241 yr Author To isolate the issue, I've temporarily replaced the fiber+transceiver for twinax DAC cables. The ethtool output: Supported ports: [ FIBRE ] Supported link modes: 10000baseKX4/Full 1000baseX/Full 10000baseCR/Full 10000baseSR/Full Supported pause frame use: Symmetric Receive-only Supports auto-negotiation: No Supported FEC modes: Not reported Advertised link modes: 10000baseKX4/Full 1000baseX/Full 10000baseCR/Full 10000baseSR/Full Advertised pause frame use: Symmetric Advertised auto-negotiation: No Advertised FEC modes: Not reported Speed: 10000Mb/s Duplex: Full Auto-negotiation: off Port: Direct Attach Copper PHYAD: 0 Transceiver: internal Supports Wake-on: d Wake-on: d Current message level: 0x00000014 (20) link ifdown Link detected: yes iperf speeds are now down from 800 to 780 Mbps. [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 96.4 MBytes 809 Mbits/sec 0 223 KBytes [ 5] 1.00-2.00 sec 94.5 MBytes 793 Mbits/sec 0 223 KBytes [ 5] 2.00-3.00 sec 74.7 MBytes 627 Mbits/sec 0 223 KBytes [ 5] 3.00-4.00 sec 96.9 MBytes 813 Mbits/sec 0 221 KBytes [ 5] 4.00-5.00 sec 93.4 MBytes 784 Mbits/sec 0 221 KBytes [ 5] 5.00-6.00 sec 95.7 MBytes 803 Mbits/sec 0 223 KBytes [ 5] 6.00-7.00 sec 83.2 MBytes 698 Mbits/sec 0 223 KBytes [ 5] 7.00-8.00 sec 95.1 MBytes 798 Mbits/sec 0 221 KBytes [ 5] 8.00-9.00 sec 97.1 MBytes 815 Mbits/sec 0 243 KBytes [ 5] 9.00-10.00 sec 98.5 MBytes 826 Mbits/sec 0 5.66 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 926 MBytes 776 Mbits/sec 0 sender [ 5] 0.00-10.00 sec 922 MBytes 774 Mbits/sec receiver I'm just not sure what's going on... Link negotiated speed seems like it's 10Gbps. But bw tests shows 1Gbps at best.
December 22, 20241 yr You also need check what speed negotiate at Router port. 6 hours ago, Mad_cyclist said: There's a 5x2.5GE+2x10G sfp+ Tenda switch in between And most case, problem usually there. Edited December 22, 20241 yr by Vr2Io
December 22, 20241 yr Author As I said, the router port autoselects 10Gbls with CX4 protocol. The sfp+ port on the switch shows a green light flashing that indicates a 10Gbps link.
December 22, 20241 yr Author 8 hours ago, Vr2Io said: You also need check what speed negotiate at Router port. And most case, problem usually there. After having replaced the transceivers, fiber, and now the Mellanox NICs the problem persists (provided the configuration is ok which by the looks of it, it is). There's only one piece of hw that I haven't changed and that is the switch. I'll replace it tomorrow and report back.
December 23, 20241 yr 21 hours ago, Mad_cyclist said: As I said, the router port autoselects 10Gbls with CX4 protocol. Those are DAC cable name ID only, you need confirm actual speed too.
December 23, 20241 yr Author On 12/22/2024 at 9:44 AM, Mad_cyclist said: I'll replace it tomorrow and report back. I've installed a similar switch, a 4x2.5G+ 2x10G sfp+. On the sfp+ ports the 1G light is flashing instead of the 10G one next to it. Triple-checked everything, tested a different cable, different transceivers. I have no idea what's going on. Ethtool reports 10000Mbps and the OPNsense router shows 10Gbase-CX4 for that interface. Switch in between shows 1G instead of 10G, and iperf still stuck at 850Mbps.
December 23, 20241 yr 7 hours ago, Mad_cyclist said: On the sfp+ ports the 1G light is flashing instead of the 10G one next to it. That means the switch is not compatible with the cable/SFP-module you're inserting into it. Try connecting the cable between two computers directly, and see if they negotiate at 10gbit. If they do, your router doesn't like the brand of the cable, or the cable itself. Also, SFP tends to stay in the same state once you insert something sometimes, so try shutting down both computer and switch, remove the cable between them, start them up and once up, insert the cable first in the switch, then in the computer, and see if it stays at 1gbit.
December 30, 20241 yr Author Solution The pcie slot I was using on the Unraid server motherboard was doing weird stuff. I tested the adjacent slot and everything worked fine, 9.9 Gbps on iperf as expected!
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.