I have been messing with this a while. I have workstation connected with Agg switch SFP+ 10GB to unraid connected SFP+ 10GB. I am copying large files (70GB) from workstation to server. I have this folder going to cache drive (nvme NVME 1TB Samsung pro) and it starts at 200MB a second and other times it does 1.2 GB a second then dies down to 200 mb a second. I was troubleshooting and wondering if it could be the setup of my cache drive? I have plenty of free space (700 GB on nvme). Could it be the "copy on write" set to auto?

 

Any help is appreciated.

 

Please post a Windows explorer graph sowing a large file transfer, but by the description, looks like the device is not keeping up with the writes.

I think I figured it out. I had to disable flow control on the nic on my windows PC. That seems to have drastically increase speed. Now getting 950 MB/sec with no drop off.

 

Related. Should I be using Jumbo frames (9014)? What is the downside?

This is the best explanation I've seen in a while regarding jumbo frames:

*************

MAM59

Posted September 9

Jumbo Frames are Dinosaurs, left over from an ancient period.

They are not needed anymore today, and, like you have already noticed, they can produce a lot of grief.

 

First of all, make yourself clear why they have been invented some decades ago. CPUs were single core mostly and slow. NICs were dumb. The CPU had to do all the computations for checking and signing the frames and also to re-request them when something was wrong.

Building and checking the Frame Headers was a tough and time consuming job.

This became more and more ugly with the arrival of 1G ethernet.

So the Idea came up that bigger ("Jumbo") frames would carry more data with fewer headers (and computations).

This wasn`t too bad, but had some serious glitches:

* more data will statistically produce more errors. So more retransmissions and more recalculations would be needed on not-so-stable lines

* all devices on the transmission path (computers, switches and so on) need to be informed about this and support the larger frame type (THIS will be your current problem, it is NOT sufficient to set the Intel Card to 9000, all other Devices on the LAN need to be set to the same value! If one is missing (unmanaged switches are often a real big problem, you cannot see or check what they do or not), the packets are lost!)

 

After some time people realized that the idea was bad. Many old devices did not play along and even newer devices often were incompatible because the frame size is reported different between vendors (your Intel "9000" reads "9024" elsewhere)

 

In real life, jumbo frames never worked really well if enough devices were in the LAN.

 

Technics moved on instead. CPU became faster, but more important: NICs became intelligent!

Now even the cheapest NIC on the cheapest Motherboard offers "offloading", which means, all the computations, checkings and retransmission handlings formerly done by the CPU are now done by the NIC itself and do not bother anymore. The NIC is fast enough to handle full traffic of the desired speed now.

"offloading" usually does Layer2 stuff, but very expensive (Server) NICs even offer Layer 3 offloading, meaning the whole TCP/IP stuff is done on the NIC.

This made Jumbo Frames totally unnecessary anymore and almost everybody stopped using them.

 

So, either turn them off (EVERYWHERE) or try to find the device(s) on your LAN, that do not support them properly and change them.

 

(And before you asked: yeah, they still exist just because they have once existed. Like the 10Mbit/s Ethernet that still is to be handled by every current Device)

Edited September 9 by MAM59

 

 

Great thanks.