Intel X540-T1 10Gbps NIC with Jumbo Frames (9000) breaking all connections to unraid

[citizengray]

I have 2 NIC, the Intel mentioned in the title and some onboard 1 Gbps NIC.

The onboard was configured with 1500 MTU and the Intel with 9000 MTU.

They are both connected to different networks, and for the life of me I could never figure out why I could never access unraid's admin page (or ssh for that matter) through the Intel sub-network. Until I played enough with it that I realized that when I lowered the MTU to the defaults 1500 it suddenly started working !?! So I removed all my Jumbo frames configuration on my network - I do get slightly less performance (~8k Gbps vs ~9.5 Gbps) but now I can access unraid from the right network (the other sub-network on the 1 Gbps onboard NIC was meant to be a temporary transition).

 

I find this so weird and could not find any online content about it... anyone else noticed this ?

[MAM59]

Jumbo Frames are Dinosaurs, left over from an ancient period.

They are not needed anymore today, and, like you have already noticed, they can produce a lot of grief.

 

First of all, make yourself clear why they have been invented some decades ago. CPUs were single core mostly and slow. NICs were dumb. The CPU had to do all the computations for checking and signing the frames and also to re-request them when something was wrong.

Building and checking the Frame Headers was a tough and time consuming job.

This became more and more ugly with the arrival of 1G ethernet.

So the Idea came up that bigger ("Jumbo") frames would carry more data with fewer headers (and computations).

This wasn`t too bad, but had some serious glitches:

* more data will statistically produce more errors. So more retransmissions and more recalculations would be needed on not-so-stable lines

* all devices on the transmission path (computers, switches and so on) need to be informed about this and support the larger frame type (THIS will be your current problem, it is NOT sufficient to set the Intel Card to 9000, all other Devices on the LAN need to be set to the same value! If one is missing (unmanaged switches are often a real big problem, you cannot see or check what they do or not), the packets are lost!)

 

After some time people realized that the idea was bad. Many old devices did not play along and even newer devices often were incompatible because the frame size is reported different between vendors (your Intel "9000" reads "9024" elsewhere)

 

In real life, jumbo frames never worked really well if enough devices were in the LAN.

 

Technics moved on instead. CPU became faster, but more important: NICs became intelligent!

Now even the cheapest NIC on the cheapest Motherboard offers "offloading", which means, all the computations, checkings and retransmission handlings formerly done by the CPU are now done by the NIC itself and do not bother anymore. The NIC is fast enough to handle full traffic of the desired speed now.

"offloading" usually does Layer2 stuff, but very expensive (Server) NICs even offer Layer 3 offloading, meaning the whole TCP/IP stuff is done on the NIC.

This made Jumbo Frames totally unnecessary anymore and almost everybody stopped using them.

 

So, either turn them off (EVERYWHERE) or try to find the device(s) on your LAN, that do not support them properly and change them.

 

(And before you asked: yeah, they still exist just because they have once existed. Like the 10Mbit/s Ethernet that still is to be handled by every current Device)

Edited September 9, 2024 by MAM59

[citizengray]

Well ! A very big thank you for giving me all the details about Jumbo frame... I guess I am that old that "back in my days" we would always aim to enable jumbo frames to get the max speed out of the ethernet transfers... but since I haven't dabbled in networking stuff for a long while - this is my first run with 10Gbps - it does feel kinda useless now ! I'll leave it off then  Problem solved ! Thx