Hi,

 

I have 2 Unraid Pro servers. I put a TP Link Tx401 in one (10.10.20.50) and it runs at the 10Gbe Speeds to my desktop, which also uses an Aquantia aqc107 (Z690 Aorus Master). My main server (10.10.20.186) has both pcie x16 slots full, so I had to use the OWC 10G Ethernet PCIe Network Adapter Card since it an X2 card and can use PCIE 4.0 x1 with the Aquantia aqc113cs chipset. The switch is set to MTU 9022. My windows desktop wont connect to either of my server's webGUI if the switch MTU is lower than 9018. The only jumbo frame option for the desktop is 4088 or 9014. I'm using 9014 for the destop MTU

 

Network settings are the same for both servers

 

I've tested multiple MTU configurations and nothing really changes for the OWC 10GBe card. I've seen it go to 7 gigabits, but nothing close to 9.

 

My desktop is able to ping the offending server at 8972 bytes, but not at 9000 bytes without fragmenting... not sure if that's relevant.

 

The CPU on the offending server is a 3900x. No cores are pinned to 100% during iperf testing. The "good" server with the tx401 is a 6600k and handles the full 10gbe no problem.

 

I've attached diagnostics as well as lspci -vv showing that the OWC card is linked at 16GT/s as well as some iperf3 tests. It does say downgraded, but I'm not sure if that is causing the problem. The card can run PCIE 3.0 x2 or PCIE 4.0 x1, and I'm doing the latter. The offending server is connected to the switch with a Cable Matters 25ft CAT6A cable and the "good" server is connected to the switch with the included 1.5m CAT6A cable that came with the TX401.

 

 

Any help would be greatly appreciated.

jupiter-diagnostics-20230912-0018.zip

1st of all: get rid of jumbo frames! They are just a relict from ancient networking times and are not recommended anymore since many years already.

 

2nd: make sure, you have turned on "flow control" everywhere! 10G LANs need it, if there is at least one slower device on the network somewhere (those evil 2,5G cards....)

If there is a problem with flow control, connections will "stutter" a lot.

 

3rd: check the cabling! 10G Twisted Pair is no kid's game and much much more sensible than 1G before. A loose plug, a not well enough shielded cable... Connections will break, reset and continue after a significant delay. They will look like you have a stable link, but once you transfer a lot of date, they will break... (hint: fiber makes you sleep much more relax)

 

4th: avoid Realtek and Aquatia chipsets in Servers. They do not have the needed buffers and interrupt "downcalmer" to handle multi-client simultanous requests. They are meant for workstations (and cheap). A good server card handles tcp protocol onboard, freeing the server's OS from all these calculations.

(they WILL work, but only within a very limited level of stress).

 

3 hours ago, MAM59 said:

1st of all: get rid of jumbo frames! They are just a relict from ancient networking times and are not recommended anymore since many years already.

 

2nd: make sure, you have turned on "flow control" everywhere! 10G LANs need it, if there is at least one slower device on the network somewhere (those evil 2,5G cards....)

If there is a problem with flow control, connections will "stutter" a lot.

 

3rd: check the cabling! 10G Twisted Pair is no kid's game and much much more sensible than 1G before. A loose plug, a not well enough shielded cable... Connections will break, reset and continue after a significant delay. They will look like you have a stable link, but once you transfer a lot of date, they will break... (hint: fiber makes you sleep much more relax)

 

4th: avoid Realtek and Aquatia chipsets in Servers. They do not have the needed buffers and interrupt "downcalmer" to handle multi-client simultanous requests. They are meant for workstations (and cheap). A good server card handles tcp protocol onboard, freeing the server's OS from all these calculations.

(they WILL work, but only within a very limited level of stress).

 

 

1. Without jumbo frames, the "bad" server only get 1.5 - 2 gigabit speeds on iperf3. The "good" server still has 10Gbe with a MTU of 1500 but it seems to affect the "bad" server.

 

2. I have a managed 10gbe switch. The description on the tips and tweaks plugin suggests turning it on on the switch. Should it turn them on only for the 2.5gbe ports or only the 10gbe ports? or all?

 

3. Cabling is fine. They're all brand new. I already bought the switch and NICs which are all rj45... too steep of an investment to switch to sfp+ after buying all this gear

 

4. Intel NICs are great, but the only reasonably affordable ones are older gen and only support PCIE 2.0 x8. Like I said, the only reason I went with the OWC 10Gbe ethernet card is for the X1 slot compatibility. Switching to a server motherboard and cpu would be another large investment.

 

I'm just wondering if there's anything I'm missing to cause the speed to be so slow or is it just the Aquantia 113 chipset? Does the unraid drivers support the 113 chipset? 

11 hours ago, Quadrupole said:

1. Without jumbo frames, the "bad" server only get 1.5 - 2 gigabit speeds on iperf3. The "good" server still has 10Gbe with a MTU of 1500 but it seems to affect the "bad" server.

Good! 🙂

Naah, but it shows that "something is wrong". Jumbo Frames just cover the problem a bit. So I think the problem is with header calculations and checks. As I said, the cheap cards cannot handle these very good.

 

11 hours ago, Quadrupole said:

2. I have a managed 10gbe switch. The description on the tips and tweaks plugin suggests turning it on on the switch. Should it turn them on only for the 2.5gbe ports or only the 10gbe ports? or all

ALL. Flow control works between the two clients, the switch only asks them to turn it on or off. So if one port is on, and the counterpart is off, the messages produced by #1 are not handled by #2.

 

11 hours ago, Quadrupole said:

3. Cabling is fine. They're all brand new.

you think.

Sadly there are not many real 10G cable cables on the market still. It gets better since last year but still most chinese based patch cables are still wrong.

Sadly too, there is nothing you can do before buying. Be alarmed if the cables are offered as "raw cable". This means, the cable itself is ok, but the plugs are still <10G.

They will work a bit, but often lose link and reset.

There is a check you can do to see if your cables are ok:

Plug the cable into the NIC, let it snap in.

Now gently try to pull it and push it.

If it moves, its a "bad cable"

(10G Ports are slighly longer. old plugs do not reach the back so they can be moved. A "real" 10G Plug sits absolutely tight)

(this is an absolute nightmare for Admins, the difference is so minimal, you cant even see it)

Edited September 13, 20232 yr by MAM59

11 hours ago, Quadrupole said:

10Gbe ethernet card is for the X1 slot compatibility.

I cannot find any hint that this chipset should work with 4.0x1. The usual connection is at least 3.0x2 or 3.0x4

Maybe THIS is your problem? you CAN plug it into an 1x slot because the gap in the plug, but it cannot run with full speed this way.

 

are we talking about "OWC 10G LAN-Adapter, RJ-45, PCIe 3.0 x2"