Slow transfer speeds between PC and NAS over 10Gbe

[Hellomynameisleo]

I'm having slower than expected speeds between my Windows 10 and UNRAID NAS which both have a Intel x520-DA1 NIC in them connected directly over a 3m 10Gb/s SPF+ DAC Twinax Cable.

I used "iperf3 NASip -P 8" and "iperf3 Windowsip -P 8" on both W10 and the NAS to see if there any difference but they both reach only 3.50Gb/s speeds which is far from the expected 10Gb/s speed I should be getting.

I've already enabled jumbo frames to 9014 bytes already on UNRAID and W10 PC which helped speeds get to 3.50Gb/s. Also disabled offloading, enabling full-duplex and increased buffers within Windows 10 but there isn't any change in speeds. The latest drivers (version 4.1.249.0) has been installed on the Windows 10 x520 NIC already and CPU usage is below 10% so there isn't a bottleneck there either.

I've tested over SMB file transfers and it indeed caps out at around 450MB/s.

 

Any ideas what the issue could be?

 

 

Edited March 12, 2023 by Hellomynameisleo

[Vr2Io]

You should enable offload and I would recommend use fiber instead 3m DAC.

Edited March 12, 2023 by Vr2Io

[Hellomynameisleo]

14 minutes ago, Vr2Io said:

You should enable offload and I would recommend use fiber instead 3m DAC.

Is the DAC causing the issue? It said on the seller amazon website that it has been tested on switches already and its branded 10Gtek DAC reputable. Also I don't really need a seperate module as its just 2 PCs.

Edited March 12, 2023 by Hellomynameisleo

[Zonediver]

Maybe a cable issue...

I reach 1169MB/sek max. between Win 10 and unraid.

Edited March 12, 2023 by Zonediver

[Vr2Io]

24 minutes ago, Hellomynameisleo said:

Is the DAC causing the issue? It said on the seller amazon website that it has been tested on switches already and its branded 10Gtek DAC reputable.

I suspect 3m DAC would be problem cause but I can't confirm, I don't like DAC at all and mainly use X520 ( also ConnectX-3 ) without problem.

 

Pls also note, screen capture indicate have retry error too. "Retr column"

Edited March 12, 2023 by Vr2Io

[Hellomynameisleo]

26 minutes ago, Vr2Io said:

I suspect 3m DAC would be problem cause but I can't confirm, I don't like DAC at all and mainly use X520 ( also ConnectX-3 ) without problem.

 

Pls also note, screen capture indicate have retry error too. "Retr column"

not sure why the DAC would cause it because I've seen a lot of people recommend and use a DAC as its much less costly. I can try buy another DAC and see if it makes a difference.

The rety is because I've sent too many request at once, doesn't show up if I do 4 parallel.

[Vr2Io]

8 minutes ago, Hellomynameisleo said:

The rety is because I've sent too many request at once, doesn't show up if I do 4 parallel.

Great.

 

Could you try make both end in Unraid and test again to see any different, trial / non-register Unraid also could make iperf3 test.

 

DAC use much less power, but due to its copper cable, so not good to against noise. ( I use 1m to 6m fiber, I like could use different length for different need )

Edited March 12, 2023 by Vr2Io

[Hellomynameisleo]

4 minutes ago, Vr2Io said:

Great.

 

Could you try make both end in Unraid and test again to see any different, trial / non-register Unraid also could make iperf3 test.

I've did listen to 192.168.11.1 (unraid IP) on unraid and then ran iperf3 on unraid again "iperf3 -c 192.168.11.1 -P 4". So it pinged itself if this is what you meant?

[Vr2Io]

16 minutes ago, Hellomynameisleo said:

So it pinged itself if this is what you meant?

No, I never test in this way and I confirm no packet will transmit in/out. ( and 44.3G result only )

 

Pls try run Unraid in Windows machine, this should ensure no other issue to slow down i.e. firewall / virus application.

 

Connecting to host 192.168.9.6, port 5201
[  5] local 192.168.9.6 port 60408 connected to 192.168.9.6 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  5.13 GBytes  44.1 Gbits/sec    0   1023 KBytes       
[  5]   1.00-2.00   sec  5.16 GBytes  44.3 Gbits/sec    0    895 KBytes       
[  5]   2.00-3.00   sec  5.17 GBytes  44.4 Gbits/sec    0   1023 KBytes       
[  5]   3.00-4.00   sec  5.16 GBytes  44.3 Gbits/sec    0   1023 KBytes       
[  5]   4.00-5.00   sec  5.16 GBytes  44.3 Gbits/sec    0   1023 KBytes       
[  5]   5.00-6.00   sec  5.15 GBytes  44.2 Gbits/sec    0    895 KBytes       
[  5]   6.00-7.00   sec  5.15 GBytes  44.2 Gbits/sec    0    895 KBytes       
[  5]   7.00-8.00   sec  5.16 GBytes  44.3 Gbits/sec    0    895 KBytes       
[  5]   8.00-9.00   sec  5.16 GBytes  44.3 Gbits/sec    0    895 KBytes       
[  5]   9.00-10.00  sec  5.16 GBytes  44.3 Gbits/sec    0   1023 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  51.6 GBytes  44.3 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  51.6 GBytes  44.3 Gbits/sec                  receiver

 

Edited March 12, 2023 by Vr2Io

[Hellomynameisleo]

9 minutes ago, Vr2Io said:

No, I never test in this way and I confirm no packet will transmit in/out.

 

Pls try run Unraid in Windows machine, this should ensure no other issue to slow down i.e. firewall / virus application.

I don't have another spare USB around. I'll try booting to ubuntu and using iperf3 from my spare SSD

[Hellomynameisleo]

3 hours ago, Vr2Io said:

No, I never test in this way and I confirm no packet will transmit in/out. ( and 44.3G result only )

 

Pls try run Unraid in Windows machine, this should ensure no other issue to slow down i.e. firewall / virus application.

 

Connecting to host 192.168.9.6, port 5201
[  5] local 192.168.9.6 port 60408 connected to 192.168.9.6 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec  5.13 GBytes  44.1 Gbits/sec    0   1023 KBytes       
[  5]   1.00-2.00   sec  5.16 GBytes  44.3 Gbits/sec    0    895 KBytes       
[  5]   2.00-3.00   sec  5.17 GBytes  44.4 Gbits/sec    0   1023 KBytes       
[  5]   3.00-4.00   sec  5.16 GBytes  44.3 Gbits/sec    0   1023 KBytes       
[  5]   4.00-5.00   sec  5.16 GBytes  44.3 Gbits/sec    0   1023 KBytes       
[  5]   5.00-6.00   sec  5.15 GBytes  44.2 Gbits/sec    0    895 KBytes       
[  5]   6.00-7.00   sec  5.15 GBytes  44.2 Gbits/sec    0    895 KBytes       
[  5]   7.00-8.00   sec  5.16 GBytes  44.3 Gbits/sec    0    895 KBytes       
[  5]   8.00-9.00   sec  5.16 GBytes  44.3 Gbits/sec    0    895 KBytes       
[  5]   9.00-10.00  sec  5.16 GBytes  44.3 Gbits/sec    0   1023 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  51.6 GBytes  44.3 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  51.6 GBytes  44.3 Gbits/sec                  receiver

 

I've tried it in ubuntu fresh install to unraid and it still stuck at 3.5Gb/s. I've also installed the docker openspeedtest to my 10Gb NIC and it still shows the usual slow speed. Any ideas what is going wrong? I didn't expect it to be this hard to set the 10Gb NIC connection up

Edited March 12, 2023 by Hellomynameisleo

[Vr2Io]

As eliminate Wimdows issue, pls as previous suggestion to buy / change the cable.

[Hellomynameisleo]

5 hours ago, Vr2Io said:

As eliminate Wimdows issue, pls as previous suggestion to buy / change the cable.

I'll get one ordered but I'm thinking the intel x520-DA1 maybe fakes as the speeds are capping right at 3.5Gb/s

[Hellomynameisleo]

I managed to fix it finally. It was my Windows 10 PC I put the NIC 10Gb on a electrical x1 gen 3.0 slot which was limiting the throughput. I've switched it to a electrical x16 slotinstead.