kennymc.c Posted May 3, 2021 Posted May 3, 2021 I've been trying for some time to find the reason why I can't get speeds above 1 Gbit/s to my Unraid server with a built-in Mellanox ConnectX-3 10GB SFP+ Ethernet card (MCX311A). My macOS client is connected via a 2.5G USB NIC with a Realtek RTL8156 chip. In between is a Zyxel XGS1010-12 switch with the server connected to the SFP+ port via a DAC cable. A 10G link is shown on the SFP+ port and a 2.5G link on the 2.5G port. However, via SMB and iperf3 I only get about 1 Gbit/s in both directions: Unraid: iperf3 -s --bind 192.168.1.205 Client: iperf3 -c 192.168.1.205 ~ 18:34:28 Connecting to host 192.168.1.205, port 5201 [ 5] local 192.168.1.120 port 52161 connected to 192.168.1.205 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.01 sec 116 MBytes 962 Mbits/sec [ 5] 1.01-2.00 sec 111 MBytes 940 Mbits/sec [ 5] 2.00-3.00 sec 112 MBytes 941 Mbits/sec [ 5] 3.00-4.01 sec 113 MBytes 941 Mbits/sec [ 5] 4.01-5.00 sec 111 MBytes 941 Mbits/sec [ 5] 5.00-6.01 sec 112 MBytes 939 Mbits/sec [ 5] 6.01-7.00 sec 112 MBytes 944 Mbits/sec [ 5] 7.00-8.01 sec 112 MBytes 937 Mbits/sec [ 5] 8.01-9.00 sec 112 MBytes 943 Mbits/sec [ 5] 9.00-10.00 sec 112 MBytes 941 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 1.10 GBytes 943 Mbits/sec sender [ 5] 0.00-10.00 sec 1.10 GBytes 941 Mbits/sec receiver iperf Done. iperf3 -c 192.168.1.205 -R ~ 18:40:38 Connecting to host 192.168.1.205, port 5201 Reverse mode, remote host 192.168.1.205 is sending [ 5] local 192.168.1.120 port 52205 connected to 192.168.1.205 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 112 MBytes 939 Mbits/sec [ 5] 1.00-2.00 sec 112 MBytes 939 Mbits/sec [ 5] 2.00-3.00 sec 112 MBytes 941 Mbits/sec [ 5] 3.00-4.00 sec 112 MBytes 941 Mbits/sec [ 5] 4.00-5.00 sec 112 MBytes 941 Mbits/sec [ 5] 5.00-6.00 sec 112 MBytes 941 Mbits/sec [ 5] 6.00-7.00 sec 112 MBytes 941 Mbits/sec [ 5] 7.00-8.00 sec 112 MBytes 941 Mbits/sec [ 5] 8.00-9.00 sec 107 MBytes 898 Mbits/sec [ 5] 9.00-10.00 sec 101 MBytes 843 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate Retr [ 5] 0.00-10.00 sec 1.08 GBytes 930 Mbits/sec 26 sender [ 5] 0.00-10.00 sec 1.08 GBytes 927 Mbits/sec receiver iperf Done. My other 4 internal Intel NICs are also connected to the same switch, but the IP used is that of the Mellanox card. Even with multiple parallel streams it does not get faster: iperf3 -c 192.168.1.205 -P 4 ~ 18:42:09 Connecting to host 192.168.1.205, port 5201 [ 5] local 192.168.1.120 port 52233 connected to 192.168.1.205 port 5201 [ 7] local 192.168.1.120 port 52234 connected to 192.168.1.205 port 5201 [ 9] local 192.168.1.120 port 52235 connected to 192.168.1.205 port 5201 [ 11] local 192.168.1.120 port 52236 connected to 192.168.1.205 port 5201 [ ID] Interval Transfer Bitrate [ 5] 0.00-1.00 sec 29.9 MBytes 251 Mbits/sec [ 7] 0.00-1.00 sec 29.8 MBytes 250 Mbits/sec [ 9] 0.00-1.00 sec 29.8 MBytes 250 Mbits/sec [ 11] 0.00-1.00 sec 29.3 MBytes 246 Mbits/sec [SUM] 0.00-1.00 sec 119 MBytes 997 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 1.00-2.00 sec 28.1 MBytes 236 Mbits/sec [ 7] 1.00-2.00 sec 27.6 MBytes 231 Mbits/sec [ 9] 1.00-2.00 sec 28.1 MBytes 236 Mbits/sec [ 11] 1.00-2.00 sec 28.1 MBytes 236 Mbits/sec [SUM] 1.00-2.00 sec 112 MBytes 939 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 2.00-3.00 sec 28.2 MBytes 236 Mbits/sec [ 7] 2.00-3.00 sec 28.1 MBytes 236 Mbits/sec [ 9] 2.00-3.00 sec 28.2 MBytes 236 Mbits/sec [ 11] 2.00-3.00 sec 28.2 MBytes 236 Mbits/sec [SUM] 2.00-3.00 sec 113 MBytes 945 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 3.00-4.00 sec 27.9 MBytes 234 Mbits/sec [ 7] 3.00-4.00 sec 27.3 MBytes 229 Mbits/sec [ 9] 3.00-4.00 sec 27.9 MBytes 234 Mbits/sec [ 11] 3.00-4.00 sec 28.3 MBytes 238 Mbits/sec [SUM] 3.00-4.00 sec 111 MBytes 935 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 4.00-5.00 sec 28.1 MBytes 236 Mbits/sec [ 7] 4.00-5.00 sec 28.1 MBytes 236 Mbits/sec [ 9] 4.00-5.00 sec 28.1 MBytes 235 Mbits/sec [ 11] 4.00-5.00 sec 28.1 MBytes 236 Mbits/sec [SUM] 4.00-5.00 sec 112 MBytes 942 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 5.00-6.00 sec 28.2 MBytes 236 Mbits/sec [ 7] 5.00-6.00 sec 28.1 MBytes 236 Mbits/sec [ 9] 5.00-6.00 sec 28.1 MBytes 236 Mbits/sec [ 11] 5.00-6.00 sec 28.2 MBytes 236 Mbits/sec [SUM] 5.00-6.00 sec 113 MBytes 944 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 6.00-7.00 sec 28.1 MBytes 235 Mbits/sec [ 7] 6.00-7.00 sec 28.0 MBytes 235 Mbits/sec [ 9] 6.00-7.00 sec 28.1 MBytes 236 Mbits/sec [ 11] 6.00-7.00 sec 28.1 MBytes 236 Mbits/sec [SUM] 6.00-7.00 sec 112 MBytes 942 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 7.00-8.00 sec 28.2 MBytes 236 Mbits/sec [ 7] 7.00-8.00 sec 28.3 MBytes 237 Mbits/sec [ 9] 7.00-8.00 sec 28.2 MBytes 236 Mbits/sec [ 11] 7.00-8.00 sec 27.7 MBytes 232 Mbits/sec [SUM] 7.00-8.00 sec 112 MBytes 942 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 8.00-9.00 sec 27.7 MBytes 232 Mbits/sec [ 7] 8.00-9.00 sec 27.6 MBytes 232 Mbits/sec [ 9] 8.00-9.00 sec 27.6 MBytes 232 Mbits/sec [ 11] 8.00-9.00 sec 27.8 MBytes 233 Mbits/sec [SUM] 8.00-9.00 sec 111 MBytes 928 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ 5] 9.00-10.00 sec 27.3 MBytes 229 Mbits/sec [ 7] 9.00-10.00 sec 29.2 MBytes 245 Mbits/sec [ 9] 9.00-10.00 sec 27.3 MBytes 229 Mbits/sec [ 11] 9.00-10.00 sec 26.1 MBytes 219 Mbits/sec [SUM] 9.00-10.00 sec 110 MBytes 922 Mbits/sec - - - - - - - - - - - - - - - - - - - - - - - - - [ ID] Interval Transfer Bitrate [ 5] 0.00-10.00 sec 282 MBytes 236 Mbits/sec sender [ 5] 0.00-10.00 sec 281 MBytes 235 Mbits/sec receiver [ 7] 0.00-10.00 sec 282 MBytes 237 Mbits/sec sender [ 7] 0.00-10.00 sec 281 MBytes 236 Mbits/sec receiver [ 9] 0.00-10.00 sec 281 MBytes 236 Mbits/sec sender [ 9] 0.00-10.00 sec 280 MBytes 235 Mbits/sec receiver [ 11] 0.00-10.00 sec 280 MBytes 235 Mbits/sec sender [ 11] 0.00-10.00 sec 279 MBytes 234 Mbits/sec receiver [SUM] 0.00-10.00 sec 1.10 GBytes 944 Mbits/sec sender [SUM] 0.00-10.00 sec 1.09 GBytes 940 Mbits/sec receiver iperf Done. I first thought it was due to the missing macOS Big Sur drivers of the RTL8156, but I learned in another forum that despite the missing drivers, other users can achieve speeds over 1 GBit/s with it even if only a 1000BaseT connection is shown in the system settings. I tried to pass the USB NIC through to a Windows VM, but even there I get the same result with activated jumbo frames and installed drivers. What could be the reason for this? I can only explain this with the fact that despite using the correct IP in reality only one of the 1 Gbit/s Intel NICs is used. Additionally I have attached a diagnostics file. unraid-diagnostics-20210503-1848.zip Quote
kennymc.c Posted May 5, 2021 Author Posted May 5, 2021 I have now tried to disconnect the internal NICs from the switch and suddenly get about 1.37 Gbit/s to the server 1.8 Gbit/s in the other direction with 4 parallel connections. With one it is only 1.23 Gbit/s. Still no good values but at least faster than 1 Gbit/s. If I connect the NICs to the switch again, the speed drops back to the old level. Does Unraid still combine several interfaces internally despite deactivated bonding and is the speed set to the lowest common value? I can't explain it any other way at the moment. Or am I missing something in iperf? Quote
JorgeB Posted May 5, 2021 Posted May 5, 2021 10 minutes ago, kennymc.c said: Does Unraid still combine several interfaces internally despite deactivated bonding and is the speed set to the lowest common value? It shouldn't, are you using a different IP subnet for the 10GbE link? Quote
kennymc.c Posted May 5, 2021 Author Posted May 5, 2021 No, all Intel NICs are using 192.168.1.201-204 and the Mellanox card is using 192.168.1.205. Quote
JorgeB Posted May 5, 2021 Posted May 5, 2021 You need to put the 10GbE NIC on a different subnet (without gateway), then use that IP for 10GbE. Quote
kennymc.c Posted May 5, 2021 Author Posted May 5, 2021 Thanks, that seems to work. Now I get up to 2.33 Gbit/s with 4 parallel connections. Via SMB I get around 1,27 Gbit/s to the cache drive with a 4GB file, but think this could be related to the missing macOS driver that enables jumbo frames. Maybe I will try this later with the Windows VM. But now I have to be connected with 2 network interfaces to reach both subnets. Is there a way to combine these? Or is this not possible because my switch is unmamaged? Are there other ways to achive higher speeds without a separate 10GbE subnet? Sorry for that question but i never worked with different subnets before. I noticed that Unraid lost the DNS server config after I changed the ip configuration although my router ip was shown as the DNS server for all interfaces. I had this issue before and setting the DNS server assignment to manual and automatic again solved it. Quote
JorgeB Posted May 5, 2021 Posted May 5, 2021 I'm really not a network guy, but always read you need different subnets for that, you can also add the server name with the 10GbE IP to the hosts file on the client computer, so you can access it by name still using 10GbE, that's what I do. Quote
kennymc.c Posted May 5, 2021 Author Posted May 5, 2021 The problem is that that the USB interface will not always be connected to the client. I'm looking for a way to automatically choose the 10gbe subnet ip for the server when the usb interface is connected and the router subnet ip when I'm connected via wifi. I googled I little bit about DNS load balancing Looks like this could be the solution for this. Quote
Vr2Io Posted May 5, 2021 Posted May 5, 2021 (edited) Use two subnet usually because no 10G switch to bridge 1G/2.5G and 10G network. You have Zyxel XGS1010-12 switch, than all different speed device 1/2.5/10 can connect to same switch. You shouldn't use the onboard 1G NIC (disconnect them), and put the 10G NIC be your 1st NIC. If you need use onboard 1G, then a common way was set them be different subnet. Edited May 5, 2021 by Vr2Io Quote
kennymc.c Posted May 5, 2021 Author Posted May 5, 2021 Disconnecting all 1G NICs or putting them into a different subnet is not an option for me. They need to communicate with other devices of the 1GbE subnet. I read that round robin dns by default is not checking which IP is reachable. So I have to find another solution. Quote
Vr2Io Posted May 5, 2021 Posted May 5, 2021 (edited) 9 minutes ago, kennymc.c said: Disconnecting all 1G NICs or putting them into a different subnet is not an option for me. They need to communicate with other devices of the 1GbE subnet. If they are not connect through switch then it will be fine. Otherwise, VLAN is one solution, set VLAN at both end NIC, no need switch support. If your swtch are management support VALN then you can set that at switch only. Edited May 5, 2021 by Vr2Io Quote
kennymc.c Posted May 5, 2021 Author Posted May 5, 2021 So instead of using the 4 physical 1GbE interfaces I create 4 vlans with 2 subnets for the Mellanox card? Since only my router support VLANs and my switch is unmanaged I will have to configure this in Unraid. Quote
Vr2Io Posted May 5, 2021 Posted May 5, 2021 My first idea was not multiplex several VLAN in one 10G physical link, this shouldn't work because Unraid management LAN can't set with VLAN. If those 4 independent 1G NIC are connect to other devive directly then you would fine. Quote
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.