Only 1 Gbit/s with 10 Gbit/s Mellanox card via iperf3


Recommended Posts

I've been trying for some time to find the reason why I can't get speeds above 1 Gbit/s to my Unraid server with a built-in Mellanox ConnectX-3 10GB SFP+ Ethernet card (MCX311A).
My macOS client is connected via a 2.5G USB NIC with a Realtek RTL8156 chip. In between is a Zyxel XGS1010-12 switch with the server connected to the SFP+ port via a DAC cable. A 10G link is shown on the SFP+ port and a 2.5G link on the 2.5G port.
However, via SMB and iperf3 I only get about 1 Gbit/s in both directions:

 

Unraid:
iperf3 -s --bind 192.168.1.205

Client:
iperf3 -c 192.168.1.205                                                                                                                                                    ~ 18:34:28
Connecting to host 192.168.1.205, port 5201
[  5] local 192.168.1.120 port 52161 connected to 192.168.1.205 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.01   sec   116 MBytes   962 Mbits/sec                  
[  5]   1.01-2.00   sec   111 MBytes   940 Mbits/sec                  
[  5]   2.00-3.00   sec   112 MBytes   941 Mbits/sec                  
[  5]   3.00-4.01   sec   113 MBytes   941 Mbits/sec                  
[  5]   4.01-5.00   sec   111 MBytes   941 Mbits/sec                  
[  5]   5.00-6.01   sec   112 MBytes   939 Mbits/sec                  
[  5]   6.01-7.00   sec   112 MBytes   944 Mbits/sec                  
[  5]   7.00-8.01   sec   112 MBytes   937 Mbits/sec                  
[  5]   8.01-9.00   sec   112 MBytes   943 Mbits/sec                  
[  5]   9.00-10.00  sec   112 MBytes   941 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec  1.10 GBytes   943 Mbits/sec                  sender
[  5]   0.00-10.00  sec  1.10 GBytes   941 Mbits/sec                  receiver

iperf Done.


iperf3 -c 192.168.1.205 -R                                                                                                                                                 ~ 18:40:38
Connecting to host 192.168.1.205, port 5201
Reverse mode, remote host 192.168.1.205 is sending
[  5] local 192.168.1.120 port 52205 connected to 192.168.1.205 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   112 MBytes   939 Mbits/sec                  
[  5]   1.00-2.00   sec   112 MBytes   939 Mbits/sec                  
[  5]   2.00-3.00   sec   112 MBytes   941 Mbits/sec                  
[  5]   3.00-4.00   sec   112 MBytes   941 Mbits/sec                  
[  5]   4.00-5.00   sec   112 MBytes   941 Mbits/sec                  
[  5]   5.00-6.00   sec   112 MBytes   941 Mbits/sec                  
[  5]   6.00-7.00   sec   112 MBytes   941 Mbits/sec                  
[  5]   7.00-8.00   sec   112 MBytes   941 Mbits/sec                  
[  5]   8.00-9.00   sec   107 MBytes   898 Mbits/sec                  
[  5]   9.00-10.00  sec   101 MBytes   843 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  1.08 GBytes   930 Mbits/sec   26             sender
[  5]   0.00-10.00  sec  1.08 GBytes   927 Mbits/sec                  receiver

iperf Done.

 

My other 4 internal Intel NICs are also connected to the same switch, but the IP used is that of the Mellanox card.

Even with multiple parallel streams it does not get faster:

 

iperf3 -c 192.168.1.205 -P 4                                                                                                                                               ~ 18:42:09
Connecting to host 192.168.1.205, port 5201
[  5] local 192.168.1.120 port 52233 connected to 192.168.1.205 port 5201
[  7] local 192.168.1.120 port 52234 connected to 192.168.1.205 port 5201
[  9] local 192.168.1.120 port 52235 connected to 192.168.1.205 port 5201
[ 11] local 192.168.1.120 port 52236 connected to 192.168.1.205 port 5201
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec  29.9 MBytes   251 Mbits/sec                  
[  7]   0.00-1.00   sec  29.8 MBytes   250 Mbits/sec                  
[  9]   0.00-1.00   sec  29.8 MBytes   250 Mbits/sec                  
[ 11]   0.00-1.00   sec  29.3 MBytes   246 Mbits/sec                  
[SUM]   0.00-1.00   sec   119 MBytes   997 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   1.00-2.00   sec  28.1 MBytes   236 Mbits/sec                  
[  7]   1.00-2.00   sec  27.6 MBytes   231 Mbits/sec                  
[  9]   1.00-2.00   sec  28.1 MBytes   236 Mbits/sec                  
[ 11]   1.00-2.00   sec  28.1 MBytes   236 Mbits/sec                  
[SUM]   1.00-2.00   sec   112 MBytes   939 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   2.00-3.00   sec  28.2 MBytes   236 Mbits/sec                  
[  7]   2.00-3.00   sec  28.1 MBytes   236 Mbits/sec                  
[  9]   2.00-3.00   sec  28.2 MBytes   236 Mbits/sec                  
[ 11]   2.00-3.00   sec  28.2 MBytes   236 Mbits/sec                  
[SUM]   2.00-3.00   sec   113 MBytes   945 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   3.00-4.00   sec  27.9 MBytes   234 Mbits/sec                  
[  7]   3.00-4.00   sec  27.3 MBytes   229 Mbits/sec                  
[  9]   3.00-4.00   sec  27.9 MBytes   234 Mbits/sec                  
[ 11]   3.00-4.00   sec  28.3 MBytes   238 Mbits/sec                  
[SUM]   3.00-4.00   sec   111 MBytes   935 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   4.00-5.00   sec  28.1 MBytes   236 Mbits/sec                  
[  7]   4.00-5.00   sec  28.1 MBytes   236 Mbits/sec                  
[  9]   4.00-5.00   sec  28.1 MBytes   235 Mbits/sec                  
[ 11]   4.00-5.00   sec  28.1 MBytes   236 Mbits/sec                  
[SUM]   4.00-5.00   sec   112 MBytes   942 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   5.00-6.00   sec  28.2 MBytes   236 Mbits/sec                  
[  7]   5.00-6.00   sec  28.1 MBytes   236 Mbits/sec                  
[  9]   5.00-6.00   sec  28.1 MBytes   236 Mbits/sec                  
[ 11]   5.00-6.00   sec  28.2 MBytes   236 Mbits/sec                  
[SUM]   5.00-6.00   sec   113 MBytes   944 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   6.00-7.00   sec  28.1 MBytes   235 Mbits/sec                  
[  7]   6.00-7.00   sec  28.0 MBytes   235 Mbits/sec                  
[  9]   6.00-7.00   sec  28.1 MBytes   236 Mbits/sec                  
[ 11]   6.00-7.00   sec  28.1 MBytes   236 Mbits/sec                  
[SUM]   6.00-7.00   sec   112 MBytes   942 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   7.00-8.00   sec  28.2 MBytes   236 Mbits/sec                  
[  7]   7.00-8.00   sec  28.3 MBytes   237 Mbits/sec                  
[  9]   7.00-8.00   sec  28.2 MBytes   236 Mbits/sec                  
[ 11]   7.00-8.00   sec  27.7 MBytes   232 Mbits/sec                  
[SUM]   7.00-8.00   sec   112 MBytes   942 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   8.00-9.00   sec  27.7 MBytes   232 Mbits/sec                  
[  7]   8.00-9.00   sec  27.6 MBytes   232 Mbits/sec                  
[  9]   8.00-9.00   sec  27.6 MBytes   232 Mbits/sec                  
[ 11]   8.00-9.00   sec  27.8 MBytes   233 Mbits/sec                  
[SUM]   8.00-9.00   sec   111 MBytes   928 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[  5]   9.00-10.00  sec  27.3 MBytes   229 Mbits/sec                  
[  7]   9.00-10.00  sec  29.2 MBytes   245 Mbits/sec                  
[  9]   9.00-10.00  sec  27.3 MBytes   229 Mbits/sec                  
[ 11]   9.00-10.00  sec  26.1 MBytes   219 Mbits/sec                  
[SUM]   9.00-10.00  sec   110 MBytes   922 Mbits/sec                  
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-10.00  sec   282 MBytes   236 Mbits/sec                  sender
[  5]   0.00-10.00  sec   281 MBytes   235 Mbits/sec                  receiver
[  7]   0.00-10.00  sec   282 MBytes   237 Mbits/sec                  sender
[  7]   0.00-10.00  sec   281 MBytes   236 Mbits/sec                  receiver
[  9]   0.00-10.00  sec   281 MBytes   236 Mbits/sec                  sender
[  9]   0.00-10.00  sec   280 MBytes   235 Mbits/sec                  receiver
[ 11]   0.00-10.00  sec   280 MBytes   235 Mbits/sec                  sender
[ 11]   0.00-10.00  sec   279 MBytes   234 Mbits/sec                  receiver
[SUM]   0.00-10.00  sec  1.10 GBytes   944 Mbits/sec                  sender
[SUM]   0.00-10.00  sec  1.09 GBytes   940 Mbits/sec                  receiver

iperf Done.

 

I first thought it was due to the missing macOS Big Sur drivers of the RTL8156, but I learned in another forum that despite the missing drivers, other users can achieve speeds over 1 GBit/s with it even if only a 1000BaseT connection is shown in the system settings.

 

I tried to pass the USB NIC through to a Windows VM, but even there I get the same result with activated jumbo frames and installed drivers.

 

What could be the reason for this? I can only explain this with the fact that despite using the correct IP in reality only one of the 1 Gbit/s Intel NICs is used.

 

Additionally I have attached a diagnostics file.

unraid-diagnostics-20210503-1848.zip

Link to comment

I have now tried to disconnect the internal NICs from the switch and suddenly get about 1.37 Gbit/s to the server 1.8 Gbit/s in the other direction with 4 parallel connections. With one it is only 1.23 Gbit/s. Still no good values but at least faster than 1 Gbit/s. If I connect the NICs to the switch again, the speed drops back to the old level. 
Does Unraid still combine several interfaces internally despite deactivated bonding and is the speed set to the lowest common value? I can't explain it any other way at the moment. Or am I missing something in iperf?

Link to comment

Thanks, that seems to work. Now I get up to 2.33 Gbit/s with 4 parallel connections. Via SMB I get around 1,27 Gbit/s to the cache drive with a 4GB file, but think this could be related to the missing macOS driver that enables jumbo frames. Maybe I will try this later with the Windows VM.

 

But now I have to be connected with 2 network interfaces to reach both subnets. Is there a way to combine these? Or is this not possible because my switch is unmamaged? Are there other ways to achive higher speeds without a separate 10GbE subnet? Sorry for that question but i never worked with different subnets before.

 

I noticed that Unraid lost the DNS server config after I changed the ip configuration although my router ip was shown as the DNS server for all interfaces. I had this issue before and setting the DNS server assignment to manual and automatic again solved it.

Link to comment

The problem is that that the USB interface will not always be connected to the client. I'm looking for a way to automatically choose the 10gbe subnet ip for the server when the usb interface is connected and the router subnet ip when I'm connected via wifi. I googled I little bit about DNS load balancing Looks like this could be the solution for this. 

Link to comment

Use two subnet usually because no 10G switch to bridge 1G/2.5G and 10G network.

 

You have Zyxel XGS1010-12 switch, than all different speed device 1/2.5/10 can connect to same switch. You shouldn't use the onboard 1G NIC (disconnect them), and put the 10G NIC be your 1st NIC.

 

If you need use onboard 1G, then a common way was set them be different subnet.

Edited by Vr2Io
Link to comment

Disconnecting all 1G NICs or putting them into a different subnet is not an option for me. They need to communicate with other devices of the 1GbE subnet.

I read that round robin dns by default is not checking which IP is reachable. So I have to find another solution.

Link to comment
9 minutes ago, kennymc.c said:

Disconnecting all 1G NICs or putting them into a different subnet is not an option for me. They need to communicate with other devices of the 1GbE subnet.

If they are not connect through switch then it will be fine.

 

Otherwise, VLAN is one solution, set VLAN at both end NIC, no need switch support. If your swtch are management support VALN then you can set that at switch only.

Edited by Vr2Io
Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.