Unable to get more than 3Gbits/s using 10GbE on iperf3 (SOLVED)

[Adam S]

Hi All,

 

I have been fiddling with this for a while and I can't seem to get anywhere above 3Gb/s.

 

Both devices are connected to the home network over 1GbE. (10.0.0.0/24)

Both devices are directly connected using 10GbE with a 10M (32ft) Duplex Multimode OM3 fiber cable. (100.0.0.0/24)

I've set both MTUs at 9000.

 

I hope this info I provided is sufficient if not please let me know.

If anyone could point me at the right direction it would be greatly appreciated.

 

This is the best speed I could manage.

Accepted connection from 100.0.0.37, port 40052
[  5] local 100.0.0.23 port 5201 connected to 100.0.0.37 port 40054
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   367 MBytes  3.08 Gbits/sec
[  5]   1.00-2.00   sec   372 MBytes  3.12 Gbits/sec
[  5]   2.00-3.00   sec   370 MBytes  3.11 Gbits/sec
[  5]   3.00-4.00   sec   368 MBytes  3.09 Gbits/sec
[  5]   4.00-5.00   sec   370 MBytes  3.10 Gbits/sec
[  5]   5.00-6.00   sec   367 MBytes  3.08 Gbits/sec
[  5]   6.00-7.00   sec   366 MBytes  3.07 Gbits/sec
[  5]   7.00-8.00   sec   367 MBytes  3.08 Gbits/sec
[  5]   8.00-9.00   sec   366 MBytes  3.07 Gbits/sec
[  5]   9.00-10.00  sec   368 MBytes  3.09 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.00  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.00  sec  3.60 GBytes  3.09 Gbits/sec                  receiver

 

Server specifications:

MB: ASRock B450M Pro4
CPU: AMD Ryzen 5 3600
RAM: 32GB
Video: Nvidia Quadro P400
Networking:
	Onboard 1GbE 10.0.0.37
	Mellanox MCX311A-XCAT 10GbE w/ MFM1T02A-SR SPF+ 100.0.0.37
Storage:
	Samsung 970 EVO 500GB
	4 x 4TB WD Red

 

Windows 10 PC Specs:

MB: Asrock X470 TAICHI
CPU: AMD Ryzen 7 2700x
RAM: 32GB
Networking:
	Onboard 1GbE 10.0.0.23
	Mellanox MCX311A-XCAT 10GbE w/ MFM1T02A-SR SPF+ 100.0.0.23

 

Thank you in advance!

uss-ibn-majed-diagnostics-20210603-0037.zip

Edited June 4, 2021 by Adam S
Solved

[Adam S]

I found out my problem, so I will leave this here in case anyone else has the same issue.

 

I ran HWiNFO64 on my windows and I found out that my 10GbE card is using PCIe 3.0 x4 correctly.

I checked my unraid server and it turns out that the slot I had the card in was a PCIe 2.0.

 

Here is how I found out.

• Open a terminal and type "lspci | grep Ethernet", that will give you a list of devices. Copy the number at the beginning for the device you are trying to get detailed info for.
• Run the following. Replace 06:00.0 with the numbers for yours.

• sudo lspci -vv -s 06:00.0 | grep "LnkSta:"

 

This is the result I got.

LnkSta: Speed 5GT/s (downgraded), Width x4 (ok)

I was expecting Speed 8GT/s for PCIe 3.0. I double checked my MB manual and yep it's a 2.0 slot.

 

[Adam S]

I thought I found the problem with my post above, but now doing some math pcie 2.0 x4 should still be more than enough for 10GbE.

PCIe 2.0 1x can handle up to: 5GT/s

So 5 x 4 lanes = 20GT/s

Converting to Gbps:

20GT/s * (8b/10b) = 16Gbps.

Take away 20% for PCIe 2.0 overhead:

16Gbps - (16Gbps * 0.2) = 12.8 Gbps.

That is still more than 10Gbps.

 

Any suggestions are appreciated.

 

[Adam S]

After many days of battling with this I finally reached a conclusion.

I decided to eliminate as many variables as possible. I started with my Windows 10 installation. I installed Windows To Go on a USB SSD and booted from it into a fresh installation.

I set my static IP on the 10GbE NIC, downloaded iperf3 and ran it.

Lo and behold 9.90 Gbits/sec.

So after all the problem was my windows 10 installation. I will now reinstall my OS then install one application at a time while testing to figure out which software was the offender (if any).

 

If I find anything I will add it in a reply hopefully it'll help someone.

 

[Adam S]

I reinstalled Windows 10 and started adding one application at a time then running iperf3.

I got to Norton Anti Virus and boom... Speed down to 3.5Gbps. (Even though I thought I ruled it out previously because I disabled it).

I started digging deeper with Norton (I didn't want to have my PC unprotected).

The exact culprit was "Intrusion Prevention". (Settings -> Firewall -> Intrusion and Browser Protection -> Intrusion Prevention).

As soon as I turned it off.... Bam! 9.87 Gbps.

 

P.S. Their customer service are idiots. I called before figuring out the root cause. I told the guy what's going on.

Him: Well what do you expect your subscription to do?

Me: Not slow down my network traffic.

Him: Well we don't provide internet, it's an antivirus software.

Me: Yes, a SECURITY suite software that includes firewall with network and browsing protection.

Him: I don't know how we can help with your speed.

........... I hung up.

Edited June 5, 2021 by Adam S
Nicer language