I run 3 servers on a weakly bases. Since I find myself changing the disk configurations 1-2 times a year, and must recopy all the data from one server to another, I am looking for the fastest syncspeed from kjell to egon.

kjell: 24-7, relativly low power, media/backup/vm/containers

2 x 25G nics

10 x sata ssd 8GB

helmaxx: Runs once a week. secon version of media library/Game VM/transcoding

egon: A Truenas server simce the Exos 2x18 do not work well with Unraid. Mostly backupserver.

4 x 10Gb nics.

8 x Exox2x18 (each disk runs at about 500 Gb each

Switch is USW EnterpriseXG 24.

I have been playing around with 4 different Freefilesync container on the Unraid to sync thru each 10Gb nic on the TN server. I believ I have worn down an AI server rack, so now searching for human feedback. To my understanding this is mainly a network question,

Happy for any feedback.

What speeds are you observing?

In the end you'll still be write speed constrained I think.

That assumes your network is optimized, jumbo frames etc.

If your disks can sustain higher write speeds (and also read speeds in this case) than a 10gbe connection, link aggregation may help on the 10gb side. Assuming your 500Gb is read speed and a typo, interpreting as 500Mb/sec and in RAID5, that's only 30% faster max read than optimized 10Gbe. All theoretical maximum. If I'm doing my math right, lots of assumptions being made... I'd be surprised to see write speeds saturating in practice, though large files may manage.

Anything that isn't using all the 2x18 in RAID5 will be write speed constrained. Any parallel operations will slow things overall in a full RAID scenario and be write speed constrained.

According to ChatGPT:

Source:

Taking ZFS overhead and RAIDZ math into account:

👉 ~4.5–5.0 GB/s sustained read is realistic from lageret

(with bursts higher if ARC/L2ARC is actively serving)

Destination:

Your xxxx pool, with 8 mirrored vdevs from 8 Exos 2×18 dual-actuator disks, should sustain:

👉 ~3.2–3.6 GB/s real-world write speed
👉 Up to ~4.0 GB/s in ideal conditions

The Unraid server has E810 nic / 2 x 25Gb, to switch 2 x 25Gb to source Truenas 4 x 10Gb nics, I´m aiming at 3 GB/s, running 4 simultaneously Freefilesync containers, I have worn out a ChatGPT serverpark, so its time to call in the experts on how to set up best possible.

A key question is: “How to get the FFsync containers to utilize a specific nic? Currently enabled bridge mode for eth2/eth3, wonder if custom network type is the way to go?

I´m getting about 1.5 GB/s when running 4 x ffsync containers simultaneously, about half of what I would like to see.

What up does iperf server to server say?

What does one ffsync alone with large files achieve?

I can't think of an easy way to dedicate a nic to a docker unless they're on different networks. I still suspect multiple parallel writes will hit limits due to head movements required. The above two bits for info may help isolate what's holding things back.

What ChatGPT said about the run:

What your results say (TL;DR)

• You’re getting ~4.8–6.5 Gbit/s aggregate, fluctuating per second.

• Each client flow sits around ~0.9–1.7 Gbit/s.

• Retransmits are huge (e.g., 800k–900k total in 60s). That screams packet loss / queue drops / sub-optimal TCP tuning, not a raw link limit.

• So the bottleneck is likely loss/queues/offloads/MTU (or hash pinning if traffic isn’t really hitting all links).

Let’s quickly isolate and then tune.

"The run" is what exactly? Let's try just one question at a time.

Please run iperf3 server to server and share the results.

In the source, Unraid server, it is dual 25G nics and target, Truenas Scale, it is 4 x 10G nics, setup as 2 bonds LACP. Ran iferp, let ChatGPT do the summary, see below. Iperf server to server is fine, allthough a lot of retries source to target, but non for target to source.

Testing pool to pool, I found one wrong drive in the TN server, I had mixed up Exos 2x18 with a X18. The correct hdd is in the server, currently resilvering.

Summary from ChatGPU of some iperf run:
ACP is finally up. Both slaves show Aggregator ID: 1 and no churn. Your 8-stream test hits ~19.8 Gb/s, which confirms the 2×10G LAG is working.

But client → NAS (forward) still shows lots of TCP retransmits (single stream ~25k; multi-stream huge), while reverse is perfect (0 retrans @ 9.9 Gb/s).
That means the remaining issue is on the client→switch→NAS receive path.

Single stream: ~9.9 Gbit/s each way — exactly what you’d expect when one TCP flow hashes to one 10 Gb link in the LACP bundle.

8 streams (client → TN): ~19.8 Gbit/s aggregate — that’s both 10 Gb links working in parallel. ✔️

Reverse: still ~9.9 Gbit/s for a single stream — normal.

The packet loss only one direction might indicate a cabling issue, nic issue, or perhaps that MTU tuning is required. Or something else entirely. What are your MTU settings? Cabling you may be able to try some single port testing and cable swapping at least. Possibly also nic/port testing this way.

The chatgpt summary is not particularly helpful to me, but it seems like a reasonable analysis assuming it's trustworthy and that's what the data says.