10gbe throughput at 2.5gbe

[feins]

Hi all,

 

Need some help.

Ive been using 10Gbe since previous version.

Its been a while i notice that during copy files i only get to 100MB and sometime drop below 40MB and rarely goes up as previous usually can saturate until my SSD speed which is around 350-400MB.

I did iperf and notice that both site transfer only cap at 2.5Gbe and I'm wonder why and how to resolve this issue?

Under unraid i can see the Lan connection is 10Gbe same with my Windows 11 Lan connection show 10Gbe.

 

ts-p500-diagnostics-20230517-2004.zip

[JorgeB]

iperf just does a LAN test, so could be NIC (or NIC settings), cables, switch, client PC, etc.

[MAM59]

Note that there is no real "2,5G" defined.

 

Some Devices always show 10G connection but transfer at lower speeds. Depends on the driver / OS.

 

So, with bad/too long cables the devices could agree to step down to 5G or 2,5G transmission speed, whilst the connection still shows full 10G.

 

On all speeds the bitrate is the same, the difference is done by inserting more or less pauses to give the cable time to recover.

(whilst the real encoding now becomes really complicated, you can simply think of 10G=transmit all the time, 5G=transmit,pause,transmit,pause... and 2,5G=transmit,pause,pause,pause,transmit...)

 

You did not tell us the type of cards and switches involved, so no educated guess besides "check the cables" can be offered.

 

[pidg30n]

Check on the (managed) switch or gateway side. Look to see if flow control is enabled. The settings for flow control will affect the maximum throughput. If it's on, try disabling it. See what happens. Play with that control, if it's there.

[feins]

17 hours ago, pidg30n said:

Check on the (managed) switch or gateway side. Look to see if flow control is enabled. The settings for flow control will affect the maximum throughput. If it's on, try disabling it. See what happens. Play with that control, if it's there.

Its an Unmanage Qnap QWS-2104-2T switch. no setting is required. On Unraid is there such setting? I know Windows i could try to disable flow control.

[feins]

17 hours ago, MAM59 said:

Note that there is no real "2,5G" defined.

 

Some Devices always show 10G connection but transfer at lower speeds. Depends on the driver / OS.

 

So, with bad/too long cables the devices could agree to step down to 5G or 2,5G transmission speed, whilst the connection still shows full 10G.

 

On all speeds the bitrate is the same, the difference is done by inserting more or less pauses to give the cable time to recover.

(whilst the real encoding now becomes really complicated, you can simply think of 10G=transmit all the time, 5G=transmit,pause,transmit,pause... and 2,5G=transmit,pause,pause,pause,transmit...)

 

You did not tell us the type of cards and switches involved, so no educated guess besides "check the cables" can be offered.

 

The cable is Genuine AMP Cat6E cable at 0.5M length. the card is HP Intel X540-2T. connected to a Qnap QSW-2104-2T unmanage switch.

 

[feins]

18 hours ago, JorgeB said:

iperf just does a LAN test, so could be NIC (or NIC settings), cables, switch, client PC, etc.

How could i pinpoint the issue? where should i start to troubleshoot?

[MAM59]

14 minutes ago, feins said:

the card is HP Intel X540-2T. connected to a Qnap QSW-2104-2T unmanage switch

can we affirm that both machines are plugged into the 2 10G ports of the switch (port #1 and #2) ? ? ?

 

The others are 2,5G only, so if one of them is in a "wrong" port (#3 to #6) it will result in 2,5g speed everywhere.

 

Edited May 18, 2023 by MAM59

[feins]

5 hours ago, MAM59 said:

can we affirm that both machines are plugged into the 2 10G ports of the switch (port #1 and #2) ? ? ?

 

The others are 2,5G only, so if one of them is in a "wrong" port (#3 to #6) it will result in 2,5g speed everywhere.

 

Yes both Unraid and my Workstation is connected to the 10gbit port. Picture attach. 
I’ve also try to change the flow control to off as suggested from my Windows 11 Setting the issue still the same. 
I’ve also try to enable jumbo frame to 9000 on both windows 11 and unraid tested also same result. 

Edited May 18, 2023 by feins

[MAM59]

ok, I just wanted to make sure... 😁

 

But, are you sure, your cables are correct? those look very cheap to me the one in #6 is even unshielded!

"real" 10G cables look different...

 

[Faceman]

If you run some traffic between those two 10g ports do any other lights flash on the switch? perhaps something funny with the routing forcing traffic to flow through a 2.5g connection? is your local traffic going from 10gbe to switch, then 2.5gbe to router and back, then 10gbe to the other end? 

 

If you disconnect every other device (and any secondary NICS) and just run server to windows through that switch does it still cap out at 2.5?

 

Cables shouldn't be an issue, you can run 10GBaseT over UTP CAT6 if the distances aren't too long, and 2.5gbe over UTP Cat5e is just fine for short hops too, and those sorts of issues arent going to cut your throughput that far, a badly grounded FTP cable can cause more problems than that.

[MAM59]

one more cable hint: try to pull the cable locked into the switch back and forth gently. If they move, they are wrong!

"real" Cat6 plugs are minimal longer than the old ones (even if you put them side by side, its hard to tell the difference), they are really fixed whereas "wrong" plugs can still be moved.

 

The problem is, that until a year ago or so, NO cable was really good enough for 10G. The cable type itself was ok, but the plugs on both ends were not. They have sold the stuff as "cat 7 raw cable" (or even cat8 sometimes).

 

10G on Twisted Pair is no child's play... I gave up and switched to fibre, no problems since then...

 

[feins]

On 5/19/2023 at 12:04 AM, MAM59 said:

ok, I just wanted to make sure... 😁

 

But, are you sure, your cables are correct? those look very cheap to me the one in #6 is even unshielded!

"real" 10G cables look different...

 

Those are AM Cat6E cable.

 

[feins]

On 5/19/2023 at 8:25 AM, Faceman said:

If you run some traffic between those two 10g ports do any other lights flash on the switch? perhaps something funny with the routing forcing traffic to flow through a 2.5g connection? is your local traffic going from 10gbe to switch, then 2.5gbe to router and back, then 10gbe to the other end? 

 

If you disconnect every other device (and any secondary NICS) and just run server to windows through that switch does it still cap out at 2.5?

 

Cables shouldn't be an issue, you can run 10GBaseT over UTP CAT6 if the distances aren't too long, and 2.5gbe over UTP Cat5e is just fine for short hops too, and those sorts of issues arent going to cut your throughput that far, a badly grounded FTP cable can cause more problems than that.

Both the 10Gbe port are connect to my Win11 Workstation and Unraid. The other 2 white cable are going to my Qnap NAS qhich is power down at meantime and the last port are going to my Telco Router so as now there is only 2 10Gbe connection are active only.

When i try to copy file from my windows 11 workstation to Unraid only both the 10Gbe port are blinking.

[feins]

On 5/19/2023 at 12:40 PM, MAM59 said:

one more cable hint: try to pull the cable locked into the switch back and forth gently. If they move, they are wrong!

"real" Cat6 plugs are minimal longer than the old ones (even if you put them side by side, its hard to tell the difference), they are really fixed whereas "wrong" plugs can still be moved.

 

The problem is, that until a year ago or so, NO cable was really good enough for 10G. The cable type itself was ok, but the plugs on both ends were not. They have sold the stuff as "cat 7 raw cable" (or even cat8 sometimes).

 

10G on Twisted Pair is no child's play... I gave up and switched to fibre, no problems since then...

 

That's what i have AM cat6E 0.5M cable which I'm using or the Krone Cat6E cable which is 2M in length.

[MAM59]

40 minutes ago, feins said:

Those are AM Cat6E cable.

yeah, you did not get my message. Of course these are suitable cables, but that does not mean a thing. Could even be Cat 7 or Cat 8. Does not mean a thing too (ok, its relevant too but only 50% of the problem)

 

The problems are the plugs on both ends. Before 2022 there were no cat6 plugs available, so they took real cables and attached bad plugs and sold it as "patch cable (raw cable)".

 

If you bought one of these you should read it as "does not work properly".

 

(sadly there is nothing written on the plugs, thats why I added the trick with moving the plug whilest it is attached, If if moves, you should consider to throw away that cable)

 

Btw, price is also no real way to tell. I've bought dozends of different brands before I realized that the rather cheap "Amazon basics, nylon covered" worked flawlessly.

 

[feins]

57 minutes ago, MAM59 said:

yeah, you did not get my message. Of course these are suitable cables, but that does not mean a thing. Could even be Cat 7 or Cat 8. Does not mean a thing too (ok, its relevant too but only 50% of the problem)

 

The problems are the plugs on both ends. Before 2022 there were no cat6 plugs available, so they took real cables and attached bad plugs and sold it as "patch cable (raw cable)".

 

If you bought one of these you should read it as "does not work properly".

 

(sadly there is nothing written on the plugs, thats why I added the trick with moving the plug whilest it is attached, If if moves, you should consider to throw away that cable)

 

Btw, price is also no real way to tell. I've bought dozends of different brands before I realized that the rather cheap "Amazon basics, nylon covered" worked flawlessly.

 

I’ve just order two cat 8 Shielded cable and will try it out to see hows it’s going. 

[feins]

Today my Cat8 cable arrived and did a test and similar result.

 

C:\iperf3>iperf3 -c 10.10.2.10
Connecting to host 10.10.2.10, port 5201
[  4] local 10.10.2.127 port 56119 connected to 10.10.2.10 port 5201
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-1.00   sec   372 MBytes  3.12 Gbits/sec
[  4]   1.00-2.00   sec   378 MBytes  3.17 Gbits/sec
[  4]   2.00-3.00   sec   367 MBytes  3.08 Gbits/sec
[  4]   3.00-4.00   sec   384 MBytes  3.22 Gbits/sec
[  4]   4.00-5.00   sec   391 MBytes  3.28 Gbits/sec
[  4]   5.00-6.00   sec   402 MBytes  3.38 Gbits/sec
[  4]   6.00-7.00   sec   375 MBytes  3.14 Gbits/sec
[  4]   7.00-8.00   sec   386 MBytes  3.24 Gbits/sec
[  4]   8.00-9.00   sec   381 MBytes  3.20 Gbits/sec
[  4]   9.00-10.00  sec   382 MBytes  3.21 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  4]   0.00-10.00  sec  3.73 GBytes  3.20 Gbits/sec                  sender
[  4]   0.00-10.00  sec  3.73 GBytes  3.20 Gbits/sec                  receiver

iperf Done.

 

root@c9449a0498ba:/# iperf3 -c 10.10.2.127
Connecting to host 10.10.2.127, port 5201
[  5] local 172.17.0.6 port 58088 connected to 10.10.2.127 port 5201
[ ID] Interval           Transfer     Bitrate         Retr  Cwnd
[  5]   0.00-1.00   sec   293 MBytes  2.46 Gbits/sec    0    342 KBytes       
[  5]   1.00-2.00   sec   290 MBytes  2.43 Gbits/sec    0    325 KBytes       
[  5]   2.00-3.00   sec   302 MBytes  2.54 Gbits/sec    0    354 KBytes       
[  5]   3.00-4.00   sec   319 MBytes  2.67 Gbits/sec    0    354 KBytes       
[  5]   4.00-5.00   sec   319 MBytes  2.67 Gbits/sec    0    342 KBytes       
[  5]   5.00-6.00   sec   324 MBytes  2.72 Gbits/sec    0    339 KBytes       
[  5]   6.00-7.00   sec   321 MBytes  2.69 Gbits/sec    0    337 KBytes       
[  5]   7.00-8.00   sec   325 MBytes  2.73 Gbits/sec    0    334 KBytes       
[  5]   8.00-9.00   sec   296 MBytes  2.49 Gbits/sec    0    334 KBytes       
[  5]   9.00-10.00  sec   324 MBytes  2.72 Gbits/sec    0    334 KBytes       
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bitrate         Retr
[  5]   0.00-10.00  sec  3.04 GBytes  2.61 Gbits/sec    0             sender
[  5]   0.00-10.00  sec  3.04 GBytes  2.61 Gbits/sec                  receiver

iperf Done.

 

Ive no idea what went wrong.

[MAM59]

too bad, but I tell everybody that 10G with twisted pair is nothing for the wimp 🙂 You need strong nerves before you give up and move to fiber 🙂

 

But if I would be you, I would just leave out that switch and connect the computers back to back. See what they do then.

 

I just realize you are running iperf from within a docker or a vm? thats would I not call a serious environment for speed check.

?!?!?!?

 

[Faceman]

Don't buy anything labelled CAT7 or CAT8 on amazon, they are all fake. buy CAT6A (not CAT6E that doesn't exist) UTP patch cables from a respected data equipment supplier.

 

If the cable is "flat" it would barely meet cat5e specs.

 

 

All that aside, the cables are not your problem, the issue is a bottleneck somewhere in the chain, has someone ever done a proper throughput test on that switch? can it actually switch 10gbe at full speed through those ports? is it one 10gbe channel on the switch chip and they are splitting it for convenience?

Is there a CPU bottleneck somewhere, a PCIE lane limitation? have you tried a direct connection between the machines?

 

 

[aaronwt]

I have a couple of those QNAP switches. I've never had any problem getting over 8Gbps throughput, between two PCs, using Commscope Cat6 patch cables. On the 10GbE ports. And then Cat5e is rated for up to 100 meters at 2.5 GbE. My unRAIDs are connected to the 2.5 GbE ports. Some using generic Cat5e. At 2.5 GbE, they work just the same as the ones using Commscope Cat6 cables.

Edited May 24, 2023 by aaronwt

[MAM59]

3 hours ago, Faceman said:

Don't buy anything labelled CAT7 or CAT8 on amazon, they are all fake. buy CAT6A (not CAT6E that doesn't exist) UTP patch cables from a respected data equipment supplier.

 

If the cable is "flat" it would barely meet cat5e specs.

Outdated info.

 

Since 2022 there are good cables available. The problem is to find them because the old stuff is still around and claims to be good to.

And, since this year there is also a flat cat8 cable on the market (without plugs of course).

Anyway, proper cables for 10G are a pain in the ass, my waste bin is full of them. The real solution for sleeping calm and tight is to move over to fiber or DA (but DA can be tricky too, not every module works well in every equipment).

 

But I agree, the problem here are not switch or cables, Maybe there is a small detail he forgot to tell us about?

 

[feins]

Today ive tested direct connect to Unraid bypass my 10Gbe Switch still getting the same result.

Means its nothing to do with the Switch since the connection is direct from my Windows 11 to Unraid.

Any idea whats might be the issue here?

 

[MAM59]

still dont understand why you are running iperf in a docker without host mode???

the additional overhead for NATting and rerouting the packets will surely slow down the connection (but it should not be that much)

 

 

AAAAH! I just see one more potential point of failure!!! You are using eth0 and eth1 as a single bridge. You should be aware that this bridge only can transport at the speed of the slowest interface. So, if your eth1 is just 2,5G, you know where it comes from...

 

Edited May 24, 2023 by MAM59

[feins]

On 5/23/2023 at 6:53 PM, MAM59 said:

too bad, but I tell everybody that 10G with twisted pair is nothing for the wimp 🙂 You need strong nerves before you give up and move to fiber 🙂

 

But if I would be you, I would just leave out that switch and connect the computers back to back. See what they do then.

 

I just realize you are running iperf from within a docker or a vm? thats would I not call a serious environment for speed check.

?!?!?!?

 

I'm connecting Back to Back actually no more goes through the 10Gbe switch.

Now iperf3 for unraid is in docker now. no more under Nerdtools.

Both eth0 and eth1 are 10Gbe its the same card X540-2T comes with 2 10Gbe Ether port.

Edited May 25, 2023 by feins