Windows 10gbe Card Issue

[crowdx42]

Hi all,

 so I just got two Mellanox MCX311A-XCAT CX311A 10gbe cards, I installed one in my Unraid server and the other in a Windows 11 machine. Unraid seems to be working fine. Both machines are connected via fiber to a TP-Link 10gbe switch,  but the Windows 11 machine is not seeing the Unraid machine when it is using the 10gbe card (I disabled it's normal NIC). Windows 11 seems to have installed the drivers for the card correctly and it shows as a 10gbe NIC in the network section but when I run a speedtest it gives me really low numbers (200mbps vs normally 900mbps).
Anyone got any idea why Windows 11 is not working correctly? Is there a better driver for Windows 11 that I should be using?
Thanks for any help.
Patrick

[crowdx42]

As an update here, I did a fresh Windows 11 install and I think I have a 10gbe connection BUT it is slow, only about 200mbytes writing from an NVME SSD on Windows to an NVME SSD Cache drive on Unraid. I would think I should be getting much better speeds than that (both drives are capable of 5000mbs r/w . 
I found the issue with the internet speed was vpn related as the ip was using the vpn connection instead of bypassing it, which is normally what I do with my main desktop.
Is there anything else I can check to figure out why the connection is so slow? 

[JorgeB]

Start by running a single stream iperf test in both directions.

[crowdx42]

Iperf3 is giving me 6gbps, the switch I have is TP-Link TL-SX3008F which when controlled by Omada (Ubiquiti software controller clone) it does not support Jumbo frames, only in standalone mode. Having said that, if I could get 6gbps in a transfer from Windows to Unraid, I would be more than happy with that. 
I did disable Windows Firewall and Virus scanner for testing and I get about 220mbytes in Windows, but that is still a long way off 600mbytes that iperf3 is reporting as what is possible between the two cards in standalone transfers.
 

[JorgeB]

You should usually get closer to iperf numbers if the devices are not the problem, looking at the Windows explorer graph does the tranfer start faster and slow down after a few seconds or it's always slow from the beginning?

[crowdx42]

I hits maybe 220mbytes in Windows explorer dropping down to 180 or so over the transfer of an 8gb file. I am considering installing Linux to see what speeds I get with it.

[JorgeB]

That suggests the devices are not the problem, if they were you'd see closer to 6gbps in the beginning while it's being cache to RAM and then slow down.

[crowdx42]

Yeah,  Windows seems to be the issue,  unless I have something wrong with Unraid which installing Linux would help eliminate OS issue, if there is one.

[crowdx42]

So I took the switch out of the equation and I hooked up to Unraid from the Windows PC directly and hit 800mbyts per sec, so there is either a setting in the switch or the switch is just not up to it.
I may just order a Microtek switch and see how it works and return the TP-Link switch.

[crowdx42]

As  a further update, I installed an Intel 540 T2 NIC in my livingroom PC yesterday and it is performing much better than the Mellanox in the Office PC. I have a Microtik switch arriving the middle of next week, and so I will be able to see how it performs with the Mellanox NICs. I am hoping to get 600 - 700 mbytes. The intel 540 is doing about 400mbytes, so over double what I am getting with the Mellanox. I also switched out the transceivers on the switch to TP-Link brand, hoping that there could be some sort of compatibility there, but it had no effect.