jena Posted May 30, 2021 Share Posted May 30, 2021 (edited) Router: EdgeRouter X Switch 1 (10G): MikroTik CRS305-1G-4S+IN, Switch 2: NetGear GS305 PC - Switch1: 2.5G RJ45 - SFP+ adapter, CAT6 cable UNRAID - Switch 1: 10G Tek DAC cable I currently don't have extra DAC cable to try PC MB: ASUS B550i On Board NIC: Intel I225-V UNRAID Recently migrated from a old DELL Optiplex 3010 (with default NIC offloading and flow setting, i.e. ON). It has no problem to reach sustained 110MB/s. MB: HP Z240 (C236 chipset) Bond0 (fail-over) eth0 - 10G NIC: Mellanox MCX-311A eth1 - On board NIC: Intel I219-LM Symptoms: (using Switch 1) iperf3 -c IP-address -f M UNRAID side test file is on NVME SSD cache only. PC side test file is on SATA SSD. iperf3 speed for 10G MCX-311A NIC host PC, client UNRAID, 30MB/s (~240Mbps) host PC, client UNRAID, 220MB/s File copy UNRAID to PC 30MB/s PC to UNRAID 280MB/s iperf3 speed for for onboard Intel I219-LM NIC iperf3 speed at 80MB/s (~640Mbps) File copy UNRAID to PC 106MB/s, almost steady. PC to UNRAID 100MB/s, with momentary speed drop during copy What I have tried: [2.5G NIC seems good, Switch 1 and adaptor works] iperf3 from PC-2.5G NIC to another server with 2.5G NIC, achieved 230MB/s, almost full speed. File transfer speed are similar to iperf. [No improvement] Swap DAC cable in different ports in Switch 1 [No improvement] Unplug 10G DAC cable (using eth1, 1G), Swap different CAT6 cable in different ports in Switch 1, still 80MB/s, Windows to SMB file transfer speed is also around 40-80MB/s. Later NIC offloading fixed some of this issue. [No improvement] change eth0 to onboard 1G Intel I219-LM, unplug 10G DAC cable [Some improvement] NIC offloading OFF, NIC flow OFF, also typed in "ethtool -K eth0 tso off" "ethtool -K eth0 gso off" for both eth0 and eth1. iperf3 onboard 1G Intel I219-LM, 112MB/s, seems normal, expect for the first data point ~40MB/s in the "host UNRAID, client PC" test (this behavior has been consistent). But PC to UNRAID file transfer speed (with single 100GB file) starts at 110MB/s and drops to 40MB/s in the half way through and come back up to 110MB/s in a minute, and later drops again and come back. [No improvement] (with NIC offloading OFF, NIC flow OFF, buffer size 256 for TX and RX) change eth0 back to 10G NIC, still 30MB/s (UNRAID to PC) [No change] test with another PC with its onboard intel 1G NIC, still 30MB/s for 10G NIC. [No change] test all 1G test with Switch 2. 1. 10G - Host PC, Client UNRAID Connecting to host PC-2.5G-NIC-IP, port 5201 [ 5] local UNRAID-10G-IP port 59950 connected to PC-IP port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 34.6 MBytes 34.6 MBytes/sec 12221 106 KBytes [ 5] 1.00-2.00 sec 23.7 MBytes 23.7 MBytes/sec 4609 103 KBytes [ 5] 2.00-3.00 sec 27.8 MBytes 27.8 MBytes/sec 6516 211 KBytes [ 5] 3.00-4.00 sec 25.4 MBytes 25.4 MBytes/sec 6037 150 KBytes [ 5] 4.00-5.00 sec 24.0 MBytes 24.0 MBytes/sec 4322 97.0 KBytes [ 5] 5.00-6.00 sec 25.7 MBytes 25.7 MBytes/sec 6001 99.8 KBytes [ 5] 6.00-7.00 sec 34.4 MBytes 34.4 MBytes/sec 8015 200 KBytes [ 5] 7.00-8.00 sec 36.6 MBytes 36.6 MBytes/sec 10178 106 KBytes [ 5] 8.00-9.00 sec 29.8 MBytes 29.8 MBytes/sec 9167 111 KBytes [ 5] 9.00-10.00 sec 24.8 MBytes 24.8 MBytes/sec 6076 91.2 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 287 MBytes 28.7 MBytes/sec 73142 sender [ 5] 0.00-10.00 sec 285 MBytes 28.5 MBytes/sec receiver 2. 10G - Host UNRAID, Client PC Connecting to host UNRAID-10G-NIC-IP, port 5201 [ 4] local PC-2.5G-IP port 60951 connected to UNRAID-10G-NIC-IP port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.00 sec 283 MBytes 283 MBytes/sec [ 4] 1.00-2.00 sec 283 MBytes 283 MBytes/sec [ 4] 2.00-3.00 sec 283 MBytes 283 MBytes/sec [ 4] 3.00-4.00 sec 283 MBytes 283 MBytes/sec [ 4] 4.00-5.00 sec 283 MBytes 283 MBytes/sec [ 4] 5.00-6.00 sec 283 MBytes 283 MBytes/sec [ 4] 6.00-7.00 sec 283 MBytes 283 MBytes/sec [ 4] 7.00-8.00 sec 283 MBytes 283 MBytes/sec [ 4] 8.00-9.00 sec 281 MBytes 281 MBytes/sec [ 4] 9.00-10.00 sec 283 MBytes 283 MBytes/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.00 sec 2.76 GBytes 283 MBytes/sec sender [ 4] 0.00-10.00 sec 2.76 GBytes 283 MBytes/sec receiver 3. 1G - Host PC, Client UNRAID Connecting to host PC, port 5201 [ 5] local UNRAID port 59964 connected to PC port 5201 [ ID] Interval Transfer Bitrate Retr Cwnd [ 5] 0.00-1.00 sec 112 MBytes 112 MBytes/sec 0 314 KBytes [ 5] 1.00-2.00 sec 112 MBytes 112 MBytes/sec 0 308 KBytes [ 5] 2.00-3.00 sec 112 MBytes 112 MBytes/sec 0 305 KBytes [ 5] 3.00-4.00 sec 112 MBytes 112 MBytes/sec 0 297 KBytes [ 5] 4.00-5.00 sec 112 MBytes 112 MBytes/sec 0 297 KBytes [ 5] 5.00-6.00 sec 113 MBytes 113 MBytes/sec 0 305 KBytes [ 5] 6.00-7.00 sec 111 MBytes 111 MBytes/sec 0 299 KBytes [ 5] 7.00-8.00 sec 112 MBytes 112 MBytes/sec 0 302 KBytes [ 5] 8.00-9.00 sec 112 MBytes 112 MBytes/sec 0 297 KBytes [ 5] 9.00-10.00 sec 112 MBytes 112 MBytes/sec 0 299 KBytes - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.10 GBytes 112 MBytes/sec 0 sender [ 5] 0.00-10.00 sec 1.09 GBytes 112 MBytes/sec receiver 4. 1G - Host UNRAID, Client PC Connecting to host UNRAID, port 5201 [ 4] local PC port 49534 connected to UNRAID port 5201 [ ID] Interval Transfer Bandwidth [ 4] 0.00-1.00 sec 44.4 MBytes 44.4 MBytes/sec [ 4] 1.00-2.00 sec 112 MBytes 112 MBytes/sec [ 4] 2.00-3.00 sec 112 MBytes 112 MBytes/sec [ 4] 3.00-4.00 sec 111 MBytes 111 MBytes/sec [ 4] 4.00-5.00 sec 111 MBytes 111 MBytes/sec [ 4] 5.00-6.00 sec 110 MBytes 110 MBytes/sec [ 4] 6.00-7.00 sec 111 MBytes 111 MBytes/sec [ 4] 7.00-8.00 sec 111 MBytes 111 MBytes/sec [ 4] 8.00-9.00 sec 112 MBytes 112 MBytes/sec [ 4] 9.00-10.00 sec 112 MBytes 112 MBytes/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bandwidth [ 4] 0.00-10.00 sec 1.02 GBytes 105 MBytes/sec sender [ 4] 0.00-10.00 sec 1.02 GBytes 105 MBytes/sec receiver Edited May 30, 2021 by jena Quote Link to comment
John_M Posted May 31, 2021 Share Posted May 31, 2021 13 hours ago, jena said: Bond0 (fail-over) What happens if you disable the bonding and use iperf3 to test eth0 on its own, and eth1 on its own? Quote Link to comment
Vr2Io Posted May 31, 2021 Share Posted May 31, 2021 16 hours ago, jena said: [2.5G NIC seems good, Switch 1 and adaptor works] iperf3 from PC-2.5G NIC to another server with 2.5G NIC, achieved 230MB/s, almost full speed. File transfer speed are similar to iperf. No reason you will got 2.5G speed if all switch not support 2.5G. For 10G ( or 1G ) pls, NIC offload (on), NIC flowcontrol (off) and all client/switch use same MTU. Quote Link to comment
jena Posted May 31, 2021 Author Share Posted May 31, 2021 12 hours ago, Vr2Io said: No reason you will got 2.5G speed if all switch not support 2.5G. For 10G ( or 1G ) pls, NIC offload (on), NIC flowcontrol (off) and all client/switch use same MTU. "from PC-2.5G NIC to another server with 2.5G NIC" this was tested on 10G switch with two RJ45-SFP+ adapter. The adapter has auto negotiation and can work at 1/2.5/5/10G. Also PC and server have default 1500MTU, no jumbo frame. Quote Link to comment
jena Posted May 31, 2021 Author Share Posted May 31, 2021 14 hours ago, John_M said: What happens if you disable the bonding and use iperf3 to test eth0 on its own, and eth1 on its own? I will test that. Also test on a different MB and report back. Quote Link to comment
rodan5150 Posted May 31, 2021 Share Posted May 31, 2021 (edited) 1 hour ago, jena said: "from PC-2.5G NIC to another server with 2.5G NIC" this was tested on 10G switch with two RJ45-SFP+ adapter. The adapter has auto negotiation and can work at 1/2.5/5/10G. Also PC and server have default 1500MTU, no jumbo frame. In my experience, 9000 (9014 is what I have set, per Intel driver in my Windows box, same card in my Unraid server, so I set it identically) MTU is necessary for 10Gbe to see full bandwidth. Though, IIRC, I saw much faster than 1Gbe with 1500 MTU set. edit: For reference, I have same 10GBe switch as you as well, and Unraid server is on DAC cable to switch. Windows box is over copper CAT6A in the walls, close to 20M+ in length, and I get at or near theoretical max for 10Gbe both directions. NICs are Intel X540 T2's in both machines. Edited May 31, 2021 by rodan5150 Quote Link to comment
jena Posted June 1, 2021 Author Share Posted June 1, 2021 (edited) UPDATE: borrowed some hardware so I can do more isolated test. First I took UNRAID out of equation. PC2: I borrowed a buddy's i5-7500 and B150M motherboard. 10G NIC is plugged in slot 1 x16. PC1: (ASUS B550 Intel I225V 2.5G), TX/RX buffer = 1024, flow control enabled. Server2: OMV, RTL8125 2.5G NIC, just uses putty to SSH. Treat it as just a Linux. Adapter settings: see detail at the end All default, MTU 1500. iperf3 host Server2 (RTL8125) to client PC1 (Intel I225V) 283 MBytes/sec, no Retr displayed in output (maybe issue with windows version of iperf3?) host Server2 (RTL8125) to client PC2 (10G NIC) 259 MBytes/sec, total Retr = ~60000 host PC1 (I225V) to client PC2 (10G NIC) 166 MBytes/sec (interesting...), total Retr = ~30000 This was "host PC, client UNRAID (10G), 30MB/s". My guess there was multiple factors led to this. host PC1 (I225V) to client Server2 (RTL8125) 221 MBytes/sec, total Retr = 28 host PC2 (10G NIC) to client PC1 (I225V) 283 MBytes/sec, no Retr displayed in output host PC2 (10G NIC) to client Server2 (RTL8125) 229 MBytes/sec, Retr = 0 So it looks like I215V has some problem (there are quite a bit report on Rev1 and 2 of I225, dropping packets). Mine is Rev3, which is "supposed" to be free of previous issue. But seems to be ok from 2.5G Intel I215V to RTL8125. I am seeking diagnose directions. Thanks! ####### Server2 2.5G RTL8125 Settings for enp5s0: Supported ports: [ TP MII ] Supported link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full 2500baseT/Full Supported pause frame use: Symmetric Receive-only Supports auto-negotiation: Yes Supported FEC modes: Not reported Advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Full 2500baseT/Full Advertised pause frame use: Symmetric Receive-only Advertised auto-negotiation: Yes Advertised FEC modes: Not reported Link partner advertised link modes: 10baseT/Half 10baseT/Full 100baseT/Half 100baseT/Full 1000baseT/Half 1000baseT/Full 10000baseT/Full 2500baseT/Full 5000baseT/Full Link partner advertised pause frame use: No Link partner advertised auto-negotiation: Yes Link partner advertised FEC modes: Not reported Speed: 2500Mb/s Duplex: Full Port: MII PHYAD: 0 Transceiver: internal Auto-negotiation: on Supports Wake-on: pumbg Wake-on: d Link detected: yes ####### PC2 10G NIC ethtool enp1s0 Settings for enp1s0: Supported ports: [ FIBRE ] Supported link modes: 1000baseKX/Full 10000baseKR/Full Supported pause frame use: Symmetric Receive-only Supports auto-negotiation: No Supported FEC modes: Not reported Advertised link modes: 1000baseKX/Full 10000baseKR/Full Advertised pause frame use: Symmetric Advertised auto-negotiation: No Advertised FEC modes: Not reported Speed: 10000Mb/s Duplex: Full Port: Direct Attach Copper PHYAD: 0 Transceiver: internal Auto-negotiation: off Cannot get wake-on-lan settings: Operation not permitted Current message level: 0x00000014 (20) link ifdown Link detected: yes Edited June 1, 2021 by jena Quote Link to comment
John_M Posted June 2, 2021 Share Posted June 2, 2021 7 hours ago, jena said: I am seeking diagnose directions. On 5/31/2021 at 2:54 AM, John_M said: What happens if you disable the bonding and use iperf3 to test eth0 on its own, and eth1 on its own? Posting all those test data doesn't really help because it takes a lot of time and effort to sift through it and it doesn't seem to lead anywhere useful. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.