Abnormally Slow and Inconsistent Network (10G, 1G) Speed


Recommended Posts

Router: EdgeRouter X

Switch 1 (10G): MikroTik CRS305-1G-4S+IN,

Switch 2: NetGear GS305

 

PC - Switch1: 2.5G RJ45 - SFP+ adapter, CAT6 cable

UNRAID - Switch 1: 10G Tek DAC cable

I currently don't have extra DAC cable to try

 

PC

MB: ASUS B550i 

On Board NIC: Intel I225-V

 

UNRAID

Recently migrated from a old DELL Optiplex 3010 (with default NIC offloading and flow setting, i.e. ON).

It has no problem to reach sustained 110MB/s.

 

MB: HP Z240 (C236 chipset)

Bond0 (fail-over)

eth0 - 10G NIC: Mellanox MCX-311A

eth1 - On board NIC: Intel I219-LM

 

Symptoms: (using Switch 1)

iperf3 -c IP-address -f M

UNRAID side test file is on NVME SSD cache only.

PC side test file is on SATA SSD.

 

iperf3 speed for 10G MCX-311A NIC

  • host PC, client UNRAID, 30MB/s (~240Mbps)
  • host PC, client UNRAID, 220MB/s

File copy

  • UNRAID to PC 30MB/s
  • PC to UNRAID 280MB/s

 

iperf3 speed for for onboard Intel I219-LM NIC

iperf3 speed at 80MB/s (~640Mbps)

File copy

  • UNRAID to PC 106MB/s, almost steady.
  • PC to UNRAID 100MB/s, with momentary speed drop during copy

 

What I have tried:

  1. [2.5G NIC seems good, Switch 1 and adaptor works] iperf3 from PC-2.5G NIC to another server with 2.5G NIC, achieved 230MB/s, almost full speed. File transfer speed are similar to iperf.
  2. [No improvement] Swap DAC cable in different ports in Switch 1
  3. [No improvement] Unplug 10G DAC cable (using eth1, 1G), Swap different CAT6 cable in different ports in Switch 1, still 80MB/s, Windows to SMB file transfer speed is also around 40-80MB/s. Later NIC offloading fixed some of this issue. 
  4. [No improvement] change eth0 to onboard 1G Intel I219-LM, unplug 10G DAC cable
  5. [Some improvement] NIC offloading OFF, NIC flow OFF, also typed in "ethtool -K eth0 tso off" "ethtool -K eth0 gso off" for both eth0 and eth1. iperf3 onboard 1G Intel I219-LM, 112MB/s, seems normal, expect for the first data point ~40MB/s in the "host UNRAID, client PC" test (this behavior has been consistent). But PC to UNRAID file transfer speed (with single 100GB file) starts at 110MB/s and drops to 40MB/s in the half way through and come back up to 110MB/s in a minute, and later drops again and come back. 
  6. [No improvement] (with NIC offloading OFF, NIC flow OFF, buffer size 256 for TX and RX) change eth0 back to 10G NIC, still 30MB/s (UNRAID to PC)
  7. [No change] test with another PC with its onboard intel 1G NIC, still 30MB/s for 10G NIC. 
  8. [No change] test all 1G test with Switch 2.

 

1. 10G - Host PC, Client UNRAID
Connecting to host PC-2.5G-NIC-IP, port 5201
[  5] local UNRAID-10G-IP port 59950 connected to PC-IP port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  34.6 MBytes  34.6 MBytes/sec  12221    106 KBytes       
[  5]   1.00-2.00   sec  23.7 MBytes  23.7 MBytes/sec  4609    103 KBytes       
[  5]   2.00-3.00   sec  27.8 MBytes  27.8 MBytes/sec  6516    211 KBytes       
[  5]   3.00-4.00   sec  25.4 MBytes  25.4 MBytes/sec  6037    150 KBytes       
[  5]   4.00-5.00   sec  24.0 MBytes  24.0 MBytes/sec  4322   97.0 KBytes       
[  5]   5.00-6.00   sec  25.7 MBytes  25.7 MBytes/sec  6001   99.8 KBytes       
[  5]   6.00-7.00   sec  34.4 MBytes  34.4 MBytes/sec  8015    200 KBytes       
[  5]   7.00-8.00   sec  36.6 MBytes  36.6 MBytes/sec  10178    106 KBytes       
[  5]   8.00-9.00   sec  29.8 MBytes  29.8 MBytes/sec  9167    111 KBytes       
[  5]   9.00-10.00  sec  24.8 MBytes  24.8 MBytes/sec  6076   91.2 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec   287 MBytes  28.7 MBytes/sec  73142             sender
[  5]   0.00-10.00  sec   285 MBytes  28.5 MBytes/sec                  receiver

2. 10G - Host UNRAID, Client PC
Connecting to host UNRAID-10G-NIC-IP, port 5201
[  4] local PC-2.5G-IP port 60951 connected to UNRAID-10G-NIC-IP port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   283 MBytes   283 MBytes/sec
[  4]   1.00-2.00   sec   283 MBytes   283 MBytes/sec
[  4]   2.00-3.00   sec   283 MBytes   283 MBytes/sec
[  4]   3.00-4.00   sec   283 MBytes   283 MBytes/sec
[  4]   4.00-5.00   sec   283 MBytes   283 MBytes/sec
[  4]   5.00-6.00   sec   283 MBytes   283 MBytes/sec
[  4]   6.00-7.00   sec   283 MBytes   283 MBytes/sec
[  4]   7.00-8.00   sec   283 MBytes   283 MBytes/sec
[  4]   8.00-9.00   sec   281 MBytes   281 MBytes/sec
[  4]   9.00-10.00  sec   283 MBytes   283 MBytes/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  2.76 GBytes   283 MBytes/sec                  sender
[  4]   0.00-10.00  sec  2.76 GBytes   283 MBytes/sec                  receiver

3. 1G - Host PC, Client UNRAID
Connecting to host PC, port 5201
[  5] local UNRAID port 59964 connected to PC port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   112 MBytes   112 MBytes/sec    0    314 KBytes       
[  5]   1.00-2.00   sec   112 MBytes   112 MBytes/sec    0    308 KBytes       
[  5]   2.00-3.00   sec   112 MBytes   112 MBytes/sec    0    305 KBytes       
[  5]   3.00-4.00   sec   112 MBytes   112 MBytes/sec    0    297 KBytes       
[  5]   4.00-5.00   sec   112 MBytes   112 MBytes/sec    0    297 KBytes       
[  5]   5.00-6.00   sec   113 MBytes   113 MBytes/sec    0    305 KBytes       
[  5]   6.00-7.00   sec   111 MBytes   111 MBytes/sec    0    299 KBytes       
[  5]   7.00-8.00   sec   112 MBytes   112 MBytes/sec    0    302 KBytes       
[  5]   8.00-9.00   sec   112 MBytes   112 MBytes/sec    0    297 KBytes       
[  5]   9.00-10.00  sec   112 MBytes   112 MBytes/sec    0    299 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.10 GBytes   112 MBytes/sec    0             sender
[  5]   0.00-10.00  sec  1.09 GBytes   112 MBytes/sec                  receiver
4. 1G - Host UNRAID, Client PC
Connecting to host UNRAID, port 5201
[  4] local PC port 49534 connected to UNRAID port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec  44.4 MBytes  44.4 MBytes/sec
[  4]   1.00-2.00   sec   112 MBytes   112 MBytes/sec
[  4]   2.00-3.00   sec   112 MBytes   112 MBytes/sec
[  4]   3.00-4.00   sec   111 MBytes   111 MBytes/sec
[  4]   4.00-5.00   sec   111 MBytes   111 MBytes/sec
[  4]   5.00-6.00   sec   110 MBytes   110 MBytes/sec
[  4]   6.00-7.00   sec   111 MBytes   111 MBytes/sec
[  4]   7.00-8.00   sec   111 MBytes   111 MBytes/sec
[  4]   8.00-9.00   sec   112 MBytes   112 MBytes/sec
[  4]   9.00-10.00  sec   112 MBytes   112 MBytes/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  1.02 GBytes   105 MBytes/sec                  sender
[  4]   0.00-10.00  sec  1.02 GBytes   105 MBytes/sec                  receiver

 

Edited by jena
Link to comment
16 hours ago, jena said:

[2.5G NIC seems good, Switch 1 and adaptor works] iperf3 from PC-2.5G NIC to another server with 2.5G NIC, achieved 230MB/s, almost full speed. File transfer speed are similar to iperf.

No reason you will got 2.5G speed if all switch not support 2.5G.

 

For 10G ( or 1G ) pls, NIC offload (on), NIC flowcontrol (off) and all client/switch use same MTU.

Link to comment
12 hours ago, Vr2Io said:

No reason you will got 2.5G speed if all switch not support 2.5G.

 

For 10G ( or 1G ) pls, NIC offload (on), NIC flowcontrol (off) and all client/switch use same MTU.

"from PC-2.5G NIC to another server with 2.5G NIC" this was tested on 10G switch with two RJ45-SFP+ adapter. The adapter has auto negotiation and can work at 1/2.5/5/10G.

Also PC and server have default 1500MTU, no jumbo frame. 

Link to comment
1 hour ago, jena said:

"from PC-2.5G NIC to another server with 2.5G NIC" this was tested on 10G switch with two RJ45-SFP+ adapter. The adapter has auto negotiation and can work at 1/2.5/5/10G.

Also PC and server have default 1500MTU, no jumbo frame. 

In my experience, 9000 (9014 is what I have set, per Intel driver in my Windows box, same card in my Unraid server, so I set it identically) MTU is necessary for 10Gbe to see full bandwidth. Though, IIRC, I saw much faster than 1Gbe with 1500 MTU set.

 

edit: For reference, I have same 10GBe switch as you as well, and Unraid server is on DAC cable to switch. Windows box is over copper CAT6A in the walls, close to 20M+ in length, and I get at or near theoretical max for 10Gbe both directions. NICs are Intel X540 T2's in both machines.

Edited by rodan5150
Link to comment

UPDATE: borrowed some hardware so I can do more isolated test.

 

First I took UNRAID out of equation. 

 

PC2: I borrowed a buddy's i5-7500 and B150M motherboard. 10G NIC is plugged in slot 1 x16. 

 

PC1: (ASUS B550 Intel I225V 2.5G), TX/RX buffer = 1024, flow control enabled. 

 

Server2: OMV, RTL8125 2.5G NIC, just uses putty to SSH. Treat it as just a Linux. Adapter settings: see detail at the end

 

All default, MTU 1500. 

 

iperf3

  • host Server2 (RTL8125) to client PC1 (Intel I225V) 
    • 283 MBytes/sec, no Retr displayed in output (maybe issue with windows version of iperf3?)
  • host Server2 (RTL8125) to client PC2 (10G NIC) 
    • 259 MBytes/sec, total Retr = ~60000
  • host PC1 (I225V) to client PC2 (10G NIC)
    • 166 MBytes/sec (interesting...), total Retr = ~30000
    • This was "host PC, client UNRAID (10G), 30MB/s". My guess there was multiple factors led to this. 
  • host PC1 (I225V) to client Server2 (RTL8125)
    • 221 MBytes/sec, total Retr = 28
  • host PC2 (10G NIC) to client PC1 (I225V)
    • 283 MBytes/sec, no Retr displayed in output
  • host PC2 (10G NIC) to client Server2 (RTL8125)
    • 229 MBytes/sec, Retr = 0

 

So it looks like I215V has some problem (there are quite a bit report on Rev1 and 2 of I225, dropping packets).

Mine is Rev3, which is "supposed" to be free of previous issue. 

But seems to be ok from 2.5G Intel I215V to RTL8125.

 

I am seeking diagnose directions.

Thanks!

 

####### Server2 2.5G RTL8125
Settings for enp5s0:
        Supported ports: [ TP MII ]
        Supported link modes:   10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
                                2500baseT/Full
        Supported pause frame use: Symmetric Receive-only
        Supports auto-negotiation: Yes
        Supported FEC modes: Not reported
        Advertised link modes:  10baseT/Half 10baseT/Full
                                100baseT/Half 100baseT/Full
                                1000baseT/Full
                                2500baseT/Full
        Advertised pause frame use: Symmetric Receive-only
        Advertised auto-negotiation: Yes
        Advertised FEC modes: Not reported
        Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                             100baseT/Half 100baseT/Full
                                             1000baseT/Half 1000baseT/Full
                                             10000baseT/Full
                                             2500baseT/Full
                                             5000baseT/Full
        Link partner advertised pause frame use: No
        Link partner advertised auto-negotiation: Yes
        Link partner advertised FEC modes: Not reported
        Speed: 2500Mb/s
        Duplex: Full
        Port: MII
        PHYAD: 0
        Transceiver: internal
        Auto-negotiation: on
        Supports Wake-on: pumbg
        Wake-on: d
        Link detected: yes


####### PC2 10G NIC
ethtool enp1s0 
Settings for enp1s0:
	Supported ports: [ FIBRE ]
	Supported link modes:   1000baseKX/Full 
	                        10000baseKR/Full 
	Supported pause frame use: Symmetric Receive-only
	Supports auto-negotiation: No
	Supported FEC modes: Not reported
	Advertised link modes:  1000baseKX/Full 
	                        10000baseKR/Full 
	Advertised pause frame use: Symmetric
	Advertised auto-negotiation: No
	Advertised FEC modes: Not reported
	Speed: 10000Mb/s
	Duplex: Full
	Port: Direct Attach Copper
	PHYAD: 0
	Transceiver: internal
	Auto-negotiation: off
Cannot get wake-on-lan settings: Operation not permitted
	Current message level: 0x00000014 (20)
			       link ifdown
	Link detected: yes

 

Edited by jena
Link to comment
7 hours ago, jena said:

I am seeking diagnose directions.

 

On 5/31/2021 at 2:54 AM, John_M said:

What happens if you disable the bonding and use iperf3 to test eth0 on its own, and eth1 on its own?

 

Posting all those test data doesn't really help because it takes a lot of time and effort to sift through it and it doesn't seem to lead anywhere useful.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.