10Gbps transfer speed limited - Mellanox ConnectX-3

[Taramu]

Hello together,

 

I finally built my Unraid rig, which is based on a Dell T20 with following hardware:

- CPU: Xeon E3-1225 v3 @3.20 GHz

- 16 GB DDR ECC RAM @1600 MHz

- 10Gbps NIC Mellanox ConnectX-3

- Cache Drive: Samsung MZFLV256 NVMe PCIe M.2 256GB (Should reach 10Gbps easily, at least for read-only)

- HDD: WD Red 6TB

 

Sadly, I'm not even close to transfer files at 10 Gbps, I'm reaching around 300 - 400 Mbyte/s when transfering files via SMB to the cache drive, from a local SSD of my system (Crucial MX 500). The transfer was tested between two systems, both using a Mellanox ConnectX-3 Card and both connected via SFP+ to the same switch in the same VLAN.

 

Running iPerf (which should measure the raw network transfer rate, excluding a potential bottleneck of a SSD) gives me following result shown in the screenshot.

 

Could someone help me to find the bottleneck of my system?

 

Thanks a lot and stay healthy!

 

Edited April 4, 2020 by Taramu

[JorgeB]

There are various possibilities, for a recent user it was the AV software, basically you need to try various different things until you find the culprit.

[Taramu]

Thanks, this topic was an interesting read. Sadly I'm not using an Antivirus which could limit the transfer speed. I'm testing with a clean Windows 10 installation on which Windows Defender was disabled completely.

 

The switch between both systems is a Mikrotik CRS305-1G-4S+IN running SwOS with the latest software. Both systems are connected via SFP+ and shown with 10Gbps link speed. Since there is no routing between and both systems are in a plain Layer 2 network, the bottleneck should not come from the network. I hope.

[JorgeB]

The most likely culprit would be the Win desktop, try to get a different computer to test with, but it can also be a NIC, switch, cable, etc.

[Taramu]

I booted the second system also with Unraid and connected both Mellanox ConnectX-3 cards via SFP+ together. Set static IPs for both systems in the same subnet and installed iPerf 3 on both systems. The result is even worse. The transfer rate is below 3.0 Gbit/s now.

 

Is there anything I can do directly at the Unraid systems? I guess something seems wrong here, the results should be much better. I tried the latest stable 6.8.3 and the latest 6.9-beta1.

[Taramu]

I stopped troubleshooting at the Windows side and set both systems to Unraid. Same problem sadly, even with a direct connection between both systems.

[Vr2Io]

11 minutes ago, Taramu said:

I stopped troubleshooting at the Windows side and set both systems to Unraid. Same problem sadly, even with a direct connection between both systems.

In fact Unraid almost no NIC settiing for change. I try different NIC and platform ( AMD nor Intel ) also no issue, iperf drop only found at Windows side.

 

Pls check PCIe slot link width or provide diagnostics for ref.

 

What TX RX optical power show on Mikrotik ?

Edited April 4, 2020 by Benson

[Taramu]

I guess I can't lookup the optical power at my Mikrotik, because I'm using SwOS. What I'm seeing are the RX / TX rates and the error count. There are no errors and data gets transfered. I don't have another SFP+ cable to check if this one is the culprit. I'm using following cable:

https://www.fs.com/de/products/30760.html

 

Is it possible to get diagnostic values for the PCIe slots via Unraid?

[Vr2Io]

I also use Mikrotik SwOS, in SFP page should got the optical figure.

 

Suggest install tips and tweak plugin to turn off NIC flow control.

 

6 minutes ago, Taramu said:

Is it possible to get diagnostic values for the PCIe slots via Unraid?

Seems not if Mellanox, but Intel X520 have.

Edited April 4, 2020 by Benson

[Taramu]

SwOS doesn't display any values for TX / RX optical power on my switch. I changed the cabling and replaced the SFP+ passive cable from FS with an active cable from Netgear, which I use to connect my Switch Uplinks. Same issue with the speed, I'm starting to think the network is not the problem.

 

Does someone have a suggestion what to check next? I'll probably try every PCIe slot of both PCs, even if the Mellanox cards are connected to 16x Gen3 PCIe slots right now.

[gennro]

I know this is a bit old but I have ran into the same issue with a ConnectX-3 and could not figure out a solution at all. From what I can tell it is something with unraid and the connectX-3 drivers.

[JorgeB]

9 hours ago, gennro said:

From what I can tell it is something with unraid and the connectX-3 drivers.

If it was a driver problem it would most likely affect everyone using them, and that's not the case, I have multiple ConnectX-3 working at or close to line speed.

[gennro]

I passed the ConnectX-3 card straight to the win10 VM on the exact same Unraid server that I was getting the 2.5gbps and I was able to get 10gbps speeds I was expecting. I was using iperf3 to confirm all of this along with 20gb file transfers. There has to be a bug or something within Unraid and I spent about a week troubleshooting this issue.

[BRiT]

It wouldn't be in unRaid per-se, it would be with the Linux Kernel Drivers for it. So try passing it through to a UNIX Flavor VM, maybe Ubuntu and see what that can do? Maybe some sort of CPU governor scaling issues with what you have set for unraid?

Edited July 14, 2020 by BRiT

[JorgeB]

3 hours ago, gennro said:

There has to be a bug or something within Unraid and I spent about a week troubleshooting this issue.

Again, if that was the case everyone using those NICs would have the same problem, no?