Is this normal for 10gbe speed transfer for 2 unraid server both have 10gbe nic connected to udmse using dac sfp+?

It show interface is 10000 Mbps but transfer rate on iperf3 is half.

Edited November 4, 20241 yr by hellbhoy

Should get about double that.  Bottleneck somewhere.

The fact that you're seeing a throughput of around 5 Gbps instead of the expected 10 Gbps could be due to several reasons, even if your interfaces are configured for 10 Gbps (10,000 Mbps). Here are some possibilities:

Cabling and SFP+ DACs: Ensure that you're using proper SFP+ direct attach cables (DACs) rated for 10 Gbps. Lower quality or improperly rated cables might limit the speed to lower rates like 5 Gbps.

Link Speed Negotiation: Some network devices may not always negotiate the highest possible link speed due to various factors such as cable length, device capabilities, or auto-negotiation issues. You can try forcing the link speed on both NICs and UDM-SE switch ports using manual configuration if supported by your devices.

Flow Control Settings: Ensure that flow control settings are configured properly across all network interfaces involved (NICs and switch). Incorrect flow control settings could potentially limit throughput.

iPerf3 Parameters: Check the iperf3 parameters you're using for testing. If you're not specifying the number of parallel connections (-P), this might limit the bandwidth used during the test.

Network Overhead: Don't forget that some network overhead is normal and expected, so even if your link is running at 10 Gbps, actual throughput will be slightly less due to protocol overhead (around 3% for UDP traffic with iperf3).

To investigate further:

Try using a different cable or SFP+ module to rule out hardware issues.

Monitor the network interfaces' status and speed on all devices involved. You can use commands like ip link on Linux-based systems or check the device manager in Windows.

Check flow control settings (tx-usc, rx-usc) and ensure they are configured consistently across all devices.

Run iperf3 with different parameters to see if you can achieve higher throughput (e.g., use -P 16 for 16 parallel connections).

Use other network tools like ping or netcat (nc) to verify the connection speed, as they might behave differently than iperf3.

On 11/3/2024 at 6:37 PM, RoboCanvas said:

The fact that you're seeing a throughput of around 5 Gbps instead of the expected 10 Gbps could be due to several reasons, even if your interfaces are configured for 10 Gbps (10,000 Mbps). Here are some possibilities:

Cabling and SFP+ DACs: Ensure that you're using proper SFP+ direct attach cables (DACs) rated for 10 Gbps. Lower quality or improperly rated cables might limit the speed to lower rates like 5 Gbps.

Link Speed Negotiation: Some network devices may not always negotiate the highest possible link speed due to various factors such as cable length, device capabilities, or auto-negotiation issues. You can try forcing the link speed on both NICs and UDM-SE switch ports using manual configuration if supported by your devices.

Flow Control Settings: Ensure that flow control settings are configured properly across all network interfaces involved (NICs and switch). Incorrect flow control settings could potentially limit throughput.

iPerf3 Parameters: Check the iperf3 parameters you're using for testing. If you're not specifying the number of parallel connections (-P), this might limit the bandwidth used during the test.

Network Overhead: Don't forget that some network overhead is normal and expected, so even if your link is running at 10 Gbps, actual throughput will be slightly less due to protocol overhead (around 3% for UDP traffic with iperf3).

To investigate further:

Try using a different cable or SFP+ module to rule out hardware issues.

Monitor the network interfaces' status and speed on all devices involved. You can use commands like ip link on Linux-based systems or check the device manager in Windows.

Check flow control settings (tx-usc, rx-usc) and ensure they are configured consistently across all devices.

Run iperf3 with different parameters to see if you can achieve higher throughput (e.g., use -P 16 for 16 parallel connections).

Use other network tools like ping or netcat (nc) to verify the connection speed, as they might behave differently than iperf3.

I tried iperf3 parallel connections see results on the photo. I also disable flow control on the UDMSe  but enable it on both unraid. ( disabling it on unraid side doesnt get any improvement). MTU both on 9014. Tried different TxRx buffer size not much different. Tried Nic offload on and off not much different.

If i run iperf3 single it only giving me 5Gbps.

 

Edited November 5, 20241 yr by hellbhoy
Wrong photo upload