Hi,

 

Windows workstation and Unraid server are connected directly by a DAC Copper cable (I ruled out the cable issue by using 3 differents brands and cable length).

 

XA520-DA2 Cards and Driver 4.1.197.0 - 27/11/2019.

 

Windows 10 Pro / Unraid (latest version)

 

The problem is that Bandwith results  are below expectations and not symmetrical and that they are well below what I get on Ubuntu. Using two Ubuntu live USB keys and doing a Iperf on both machines, I get a constant robust symmetrical 9.49 Gbits/sec between the two workstations (see below).

 

I ruled out the PCIe ports restrictions by leaving just the NIC on the 16x port of each machine and by using a Remote Desktop to connect to Windows.

 

Here are the results on Windows. I tried every possible settings on the Device manager Driver tab. The only thing that seems to improve things a bit are Disabling Flow Control and setting the Interrupt Moderation Rate to Off. The other settings, jumbo frame and other Buffers have zero impact on the performance.

 

Anybody with experience with the Intel X520 cards would have an idea how to improve things?

 

>iperf3 -c 10.10.10.3 (Windows Machine)
Connecting to host 10.10.10.3, port 5201
[ 4] local 10.10.10.2 port 9356 connected to 10.10.10.3 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 604 MBytes 5.07 Gbits/sec
[ 4] 1.00-2.00 sec 678 MBytes 5.69 Gbits/sec
[ 4] 2.00-3.00 sec 731 MBytes 6.13 Gbits/sec
[ 4] 3.00-4.00 sec 745 MBytes 6.25 Gbits/sec
[ 4] 4.00-5.00 sec 747 MBytes 6.27 Gbits/sec
[ 4] 5.00-6.00 sec 742 MBytes 6.22 Gbits/sec
[ 4] 6.00-7.00 sec 769 MBytes 6.45 Gbits/sec
[ 4] 7.00-8.00 sec 764 MBytes 6.41 Gbits/sec
[ 4] 8.00-9.00 sec 758 MBytes 6.36 Gbits/sec
[ 4] 9.00-10.00 sec 788 MBytes 6.61 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 7.16 GBytes 6.15 Gbits/sec sender
[ 4] 0.00-10.00 sec 7.16 GBytes 6.15 Gbits/sec receiver

iperf Done.

 

iperf3 -c 10.10.10.2 (Unraid)
Connecting to host 10.10.10.2, port 5201
[ 4] local 10.10.10.3 port 49799 connected to 10.10.10.2 port 5201
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 324 MBytes 2.72 Gbits/sec
[ 4] 1.00-2.00 sec 334 MBytes 2.80 Gbits/sec
[ 4] 2.00-3.00 sec 350 MBytes 2.93 Gbits/sec
[ 4] 3.00-4.00 sec 327 MBytes 2.74 Gbits/sec
[ 4] 4.00-5.00 sec 324 MBytes 2.72 Gbits/sec
[ 4] 5.00-6.00 sec 276 MBytes 2.31 Gbits/sec
[ 4] 6.00-7.00 sec 321 MBytes 2.69 Gbits/sec
[ 4] 7.00-8.00 sec 323 MBytes 2.71 Gbits/sec
[ 4] 8.00-9.00 sec 336 MBytes 2.82 Gbits/sec
[ 4] 9.00-10.00 sec 336 MBytes 2.82 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth
[ 4] 0.00-10.00 sec 3.17 GBytes 2.73 Gbits/sec sender
[ 4] 0.00-10.00 sec 3.17 GBytes 2.73 Gbits/sec receiver

iperf Done.

 

Ubuntu Live USB Key (20.04 version)

 

Connecting to host 10.10.10.3, port 5201
[ 5] local 10.10.10.2 port 59388 connected to 10.10.10.3 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.09 GBytes 9.32 Gbits/sec 85 1.77 MBytes
[ 5] 1.00-2.00 sec 1.10 GBytes 9.42 Gbits/sec 7 1.77 MBytes
[ 5] 2.00-3.00 sec 1.10 GBytes 9.42 Gbits/sec 10 1.77 MBytes
[ 5] 3.00-4.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.77 MBytes
[ 5] 4.00-5.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.77 MBytes
[ 5] 5.00-6.00 sec 1.09 GBytes 9.41 Gbits/sec 0 1.77 MBytes
[ 5] 6.00-7.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.77 MBytes
[ 5] 7.00-8.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.77 MBytes
[ 5] 8.00-9.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.77 MBytes
[ 5] 9.00-10.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.77 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 11.0 GBytes 9.41 Gbits/sec 102 sender
[ 5] 0.00-10.00 sec 10.9 GBytes 9.40 Gbits/sec receiver

iperf Done.

Connecting to host 10.10.10.2, port 5201
[ 5] local 10.10.10.3 port 46954 connected to 10.10.10.2 port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 1.08 GBytes 9.32 Gbits/sec 37 1.36 MBytes
[ 5] 1.00-2.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.36 MBytes
[ 5] 2.00-3.00 sec 1.09 GBytes 9.41 Gbits/sec 0 1.36 MBytes
[ 5] 3.00-4.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.36 MBytes
[ 5] 4.00-5.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.36 MBytes
[ 5] 5.00-6.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.36 MBytes
[ 5] 6.00-7.00 sec 1.09 GBytes 9.41 Gbits/sec 0 1.37 MBytes
[ 5] 7.00-8.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.37 MBytes
[ 5] 8.00-9.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.37 MBytes
[ 5] 9.00-10.00 sec 1.10 GBytes 9.42 Gbits/sec 0 1.38 MBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 10.9 GBytes 9.40 Gbits/sec 37 sender
[ 5] 0.00-10.00 sec 10.9 GBytes 9.40 Gbits/sec receiver

iperf Done.

 

Thank you.

Install the Tips and Tweaks plugin and try changing some of the NIC settings, there are some reports that performance with those NICs is better with NIC Offload disable.

10 hours ago, johnnie.black said:

Install the Tips and Tweaks plugin and try changing some of the NIC settings, there are some reports that performance with those NICs is better with NIC Offload disable.

Thank you for the suggestion, I will try but my feeling that it comes from the Windows side, not the unraid side.

 

Anybody has experience with that card under Windows? Any idea on the Windows side?

Just tried an Ubuntu 20.04 "iperfing" Unraid and bingo 9.49 GBits/sec. So Unaid is not the limiting factor, Windows is!