Extremely uneven network performance when writing to the array

[eighty1]

New Unraid box setup, two 16TB drives, two NVMe drives in a pool, a single 1TB SATA SSD for downloads and such, an LSI SAS/SATA HBA, a dual-port Intel gigabit NIC with 802.3ad LACP bonding (switch supports it). I'm transferring around 8TB of data from my old NAS, and writes start out at around 110MB/s and then nosedive to around 30MB/s; they'll peak back up a bit but then slow down again or stall out entirely. Here's what I've done so far to try and remediate:

 

- Connected a third NIC between the two NASes to bypass the switch/bonding/any potential network issues

- Enabled "turbo" mode

- Removed the parity drive

- Set MTU on both dedicated NICs to 9000 (Jumbo Frames)

- Reconfigured the array to use the 1TB SSD as a cache (Fill-up)

- Mover rescheduled to run every three hours to get copied data over to the array

 

The results are the same regardless of each step attempted: max bandwidth copy for maybe a minute or two, then it drops and goes very "peaky." There are no Dockers active, no VMs, and very little else on the array; this is a totally fresh server. I've seen this kind of thing come up which is why I tried all of the steps above, but I'm still experiencing the issue. Perhaps some setting in the BIOS or on the LSI card? Write cache settings for the drives in the array? I'm stumped.

[JorgeB]

First note that bonding (if correctly configured and supported) will only speed up multiple transfers, single transfer will still only use one NIC.

 

Transfer starting at 110MB/s suggests the LAN is working correctly, then the devices are not able to keep up, do you get the same speed if transferring to array or cache? You should also post the diagnostics mostly to look at the hardware used.