Nanobug Posted August 29 Share Posted August 29 Hello, First of, I'm not entirely sure where the problem is, it might not even be on Unraid, so I'm posting all the information I can think of. I'm trying to setup LAG/LACP with 2 x 10 Gbit NIC. Some hardware information: CPU is 5900X in both servers. The NIC in each server is an Intel X540 T2. The switch is a USW-EnterpriseXG-24. I'm testing it between an Unraid server, and a VM in Proxmox. The VM has 4 CPU's and 8 GB of RAM, and only runs Speedtest and iPerf. I'm testing wit iPerf3. So far, Unraid has these settings on the NIC: # Generated settings: IFNAME[0]="br0" DHCP_KEEPRESOLV="yes" DNS_SERVER1="1.XXX.XXX.1" DNS_SERVER2="1.XXX.XXX.1" DHCP6_KEEPRESOLV="yes" BRNAME[0]="br0" BRNICS[0]="eth0" BRSTP[0]="no" BRFD[0]="0" PROTOCOL[0]="ipv4" USE_DHCP[0]="yes" USE_DHCP6[0]="yes" VLANID[0,1]="30" DESCRIPTION[0,1]="DMZ - 30" PROTOCOL[0,1]="ipv4" VLANID[0,2]="40" DESCRIPTION[0,2]="Storage - 40" PROTOCOL[0,2]="ipv4" VLANID[0,3]="50" DESCRIPTION[0,3]="Smart Home - 50" PROTOCOL[0,3]="ipv4" VLANID[0,4]="60" DESCRIPTION[0,4]="Surveilance - 60" PROTOCOL[0,4]="ipv4" VLANID[0,5]="70" DESCRIPTION[0,5]="Services - 70" PROTOCOL[0,5]="ipv4" VLANID[0,6]="100" DESCRIPTION[0,6]="Server - 100" PROTOCOL[0,6]="ipv4" VLANID[0,7]="200" DESCRIPTION[0,7]="North Hosting - 200" PROTOCOL[0,7]="ipv4" VLANS[0]="8" IFNAME[1]="br1" BONDNAME[1]="bond1" BONDING_MIIMON[1]="100" BRNAME[1]="br1" BRSTP[1]="no" BRFD[1]="0" DESCRIPTION[1]="10 Gbit aggregation" BONDING_MODE[1]="4" BONDNICS[1]="eth1 eth2" BRNICS[1]="bond1" PROTOCOL[1]="ipv4" USE_DHCP[1]="yes" MTU[1]="9000" DESCRIPTION[1,1]="DMZ - 30" VLANID[1,1]="30" PROTOCOL[1,1]="ipv4" DESCRIPTION[1,2]="Storage - 40" VLANID[1,2]="40" PROTOCOL[1,2]="ipv4" DESCRIPTION[1,3]="Smart Home - 50" VLANID[1,3]="50" PROTOCOL[1,3]="ipv4" DESCRIPTION[1,4]="Surveilance - 60" VLANID[1,4]="60" PROTOCOL[1,4]="ipv4" DESCRIPTION[1,5]="Services - 70" VLANID[1,5]="70" PROTOCOL[1,5]="ipv4" DESCRIPTION[1,6]="Server - 100" VLANID[1,6]="100" PROTOCOL[1,6]="ipv4" DESCRIPTION[1,7]="North Hosting - 200" VLANID[1,7]="200" PROTOCOL[1,7]="ipv4" VLANS[1]="8" SYSNICS="2" I've also added the diagnostics zip file. The Proxmox network settings: # Loopback interface auto lo iface lo inet loopback # enp7s0 interface with static IP auto enp7s0 #iface enp7s0 inet manual iface enp7s0 inet static address 10.0.20.3 netmask 255.255.255.0 gateway 10.0.20.1 # Bonding setup for enp8s0f0 and enp8s0f1 auto bond0 iface bond0 inet manual bond-slaves enp8s0f0 enp8s0f1 bond-miimon 100 bond-mode 802.3ad bond-xmit-hash-policy layer2-3 bond-lacp-rate 1 # vmbr1 using bond0 auto vmbr1 iface vmbr1 inet manual bridge-ports bond0 bridge-stp off bridge-fd 0 bridge-vlan-aware yes bridge-vids 2-4094 Along with ethtool -K <interface> lro on ifconfig <interface> mtu 9000 on both enp8s0f0 and enp8s0f1. The same settings are made on the VM, it just has a different IP. The vmbr1 is attatched to the VM within the vNIC. For the switch, I've attatched a screenshot of the settings: Only the jumbo frames have been enabled otherwise. No matter what I do, I get an iPerf test similar to this: I know I'm not supposed to get the full 20 Gbit, but I should be able to get ~2x that give or take. Accordingly to Ubiquiti, I should get more with aggregation enabled. https://help.ui.com/hc/en-us/articles/360007279753-Port-Aggregation-FAQs What I've tried so far: Disabling jumbo frames. Enabling/disabling the Large Receive Offload (LRO). Enabling/disabling TCP Segmentation Offload (TSO). Rebooting each device one at a time. iPerf setting with parallel streams (cores). iPerf setting with UDP. iPerf setting higher TCP window size. iPerf setting a longer time. iPerf setting Nagle's Algorithm disabled. iPerf setting with taskset to bind a specific core. Tried to look into the drivers for the NICs, but it seems like they were included in the kernels a while back. As mentioned in the begining, I'm not sure where the problems is exactly. Can someone check if it's correctly configured in Unraid at least? Maybe even the switch and Proxmox as well? storage-03-diagnostics-20240829-2251.zip Quote Link to comment
SimonF Posted August 29 Share Posted August 29 15 minutes ago, Nanobug said: Hello, First of, I'm not entirely sure where the problem is, it might not even be on Unraid, so I'm posting all the information I can think of. I'm trying to setup LAG/LACP with 2 x 10 Gbit NIC. Some hardware information: CPU is 5900X in both servers. The NIC in each server is an Intel X540 T2. The switch is a USW-EnterpriseXG-24. I'm testing it between an Unraid server, and a VM in Proxmox. The VM has 4 CPU's and 8 GB of RAM, and only runs Speedtest and iPerf. I'm testing wit iPerf3. So far, Unraid has these settings on the NIC: # Generated settings: IFNAME[0]="br0" DHCP_KEEPRESOLV="yes" DNS_SERVER1="1.XXX.XXX.1" DNS_SERVER2="1.XXX.XXX.1" DHCP6_KEEPRESOLV="yes" BRNAME[0]="br0" BRNICS[0]="eth0" BRSTP[0]="no" BRFD[0]="0" PROTOCOL[0]="ipv4" USE_DHCP[0]="yes" USE_DHCP6[0]="yes" VLANID[0,1]="30" DESCRIPTION[0,1]="DMZ - 30" PROTOCOL[0,1]="ipv4" VLANID[0,2]="40" DESCRIPTION[0,2]="Storage - 40" PROTOCOL[0,2]="ipv4" VLANID[0,3]="50" DESCRIPTION[0,3]="Smart Home - 50" PROTOCOL[0,3]="ipv4" VLANID[0,4]="60" DESCRIPTION[0,4]="Surveilance - 60" PROTOCOL[0,4]="ipv4" VLANID[0,5]="70" DESCRIPTION[0,5]="Services - 70" PROTOCOL[0,5]="ipv4" VLANID[0,6]="100" DESCRIPTION[0,6]="Server - 100" PROTOCOL[0,6]="ipv4" VLANID[0,7]="200" DESCRIPTION[0,7]="North Hosting - 200" PROTOCOL[0,7]="ipv4" VLANS[0]="8" IFNAME[1]="br1" BONDNAME[1]="bond1" BONDING_MIIMON[1]="100" BRNAME[1]="br1" BRSTP[1]="no" BRFD[1]="0" DESCRIPTION[1]="10 Gbit aggregation" BONDING_MODE[1]="4" BONDNICS[1]="eth1 eth2" BRNICS[1]="bond1" PROTOCOL[1]="ipv4" USE_DHCP[1]="yes" MTU[1]="9000" DESCRIPTION[1,1]="DMZ - 30" VLANID[1,1]="30" PROTOCOL[1,1]="ipv4" DESCRIPTION[1,2]="Storage - 40" VLANID[1,2]="40" PROTOCOL[1,2]="ipv4" DESCRIPTION[1,3]="Smart Home - 50" VLANID[1,3]="50" PROTOCOL[1,3]="ipv4" DESCRIPTION[1,4]="Surveilance - 60" VLANID[1,4]="60" PROTOCOL[1,4]="ipv4" DESCRIPTION[1,5]="Services - 70" VLANID[1,5]="70" PROTOCOL[1,5]="ipv4" DESCRIPTION[1,6]="Server - 100" VLANID[1,6]="100" PROTOCOL[1,6]="ipv4" DESCRIPTION[1,7]="North Hosting - 200" VLANID[1,7]="200" PROTOCOL[1,7]="ipv4" VLANS[1]="8" SYSNICS="2" I've also added the diagnostics zip file. The Proxmox network settings: # Loopback interface auto lo iface lo inet loopback # enp7s0 interface with static IP auto enp7s0 #iface enp7s0 inet manual iface enp7s0 inet static address 10.0.20.3 netmask 255.255.255.0 gateway 10.0.20.1 # Bonding setup for enp8s0f0 and enp8s0f1 auto bond0 iface bond0 inet manual bond-slaves enp8s0f0 enp8s0f1 bond-miimon 100 bond-mode 802.3ad bond-xmit-hash-policy layer2-3 bond-lacp-rate 1 # vmbr1 using bond0 auto vmbr1 iface vmbr1 inet manual bridge-ports bond0 bridge-stp off bridge-fd 0 bridge-vlan-aware yes bridge-vids 2-4094 Along with ethtool -K <interface> lro on ifconfig <interface> mtu 9000 on both enp8s0f0 and enp8s0f1. The same settings are made on the VM, it just has a different IP. The vmbr1 is attatched to the VM within the vNIC. For the switch, I've attatched a screenshot of the settings: Only the jumbo frames have been enabled otherwise. No matter what I do, I get an iPerf test similar to this: I know I'm not supposed to get the full 20 Gbit, but I should be able to get ~2x that give or take. Accordingly to Ubiquiti, I should get more with aggregation enabled. https://help.ui.com/hc/en-us/articles/360007279753-Port-Aggregation-FAQs What I've tried so far: Disabling jumbo frames. Enabling/disabling the Large Receive Offload (LRO). Enabling/disabling TCP Segmentation Offload (TSO). Rebooting each device one at a time. iPerf setting with parallel streams (cores). iPerf setting with UDP. iPerf setting higher TCP window size. iPerf setting a longer time. iPerf setting Nagle's Algorithm disabled. iPerf setting with taskset to bind a specific core. Tried to look into the drivers for the NICs, but it seems like they were included in the kernels a while back. As mentioned in the begining, I'm not sure where the problems is exactly. Can someone check if it's correctly configured in Unraid at least? Maybe even the switch and Proxmox as well? storage-03-diagnostics-20240829-2251.zip 335.26 kB · 0 downloads Normally only one of the links will be used for two endpoint i.e. mac address pairs or ip pairs. if you have differnet macs or ips for another transfer you would get two 10g streams but for a single stream will be limited to one of the physical paths. Quote Link to comment
Nanobug Posted August 30 Author Share Posted August 30 (edited) 9 hours ago, SimonF said: Normally only one of the links will be used for two endpoint i.e. mac address pairs or ip pairs. if you have differnet macs or ips for another transfer you would get two 10g streams but for a single stream will be limited to one of the physical paths. Thank you for the reply I agree. But Ubiquiti says differently on the Port Aggregation FAQ: Quote Port aggregation can increase maximum throughput, and allow for network redundancy. It does this by splitting traffic across multiple ports instead of forcing clients to use a single uplink port on a switch. Note that these performance improvements will only occur when multiple clients are passing traffic simultaneously through the aggregated ports. Edited August 30 by Nanobug Quote Link to comment
JonathanM Posted August 30 Share Posted August 30 3 hours ago, Nanobug said: But Ubiquiti says differently How is that different? It looks like they say exactly the same thing with different wording. Quote Link to comment
Nanobug Posted August 30 Author Share Posted August 30 3 hours ago, JonathanM said: How is that different? It looks like they say exactly the same thing with different wording. Then I guess it's just how you understand the wording. Guess I'll upgrade the other switch and do 2 iPerf switch at the same time to see if I can get something along the lines of 2 x 10 Gbit then. Thank you Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.