New to Unraid and seem to be going down a networking rabbit hole with slow SMB transfers to a 2TB ZFS mirror NVME write cache.

I'm using Unraid as purely storage (so no docker etc on the Unraid box) with compute on a mini pc.

Unraid: Asus Pro WS W680-ACE (i226 NIC), Intel 13500T, 32GB ECC RAM, 2x Seagate 530R (ZFS Mirror), 12x WD Red Pro (array, 2 drive parity)

Mini PC: Lenovo P3 Tiny (i219 NIC), Proxmox, Ubuntu server VM (24.04)

Windows PC: Asus Crosshair X670e (i225 NIC), Win11

I initially spun up the array (no cache) without parity transferred all my data from a Synology then added parity and cache drives. This was transferred by SMB mount via unassigned devices then rclone. For the few days this was running I got full 1GbE line speed direct to the array.

Following this I noticed slow write speeds (40 MB/s) to the unraid cache from windows firstly, this is write speed to the NVME cache (not the now parity protected array), I confirmed this by making a new share just on the cache only and testing that, same speed. However the same is true to the array, but with dual parity that is some what expected.

This speed is consistently slow to the cache from multiple devices including the mini pc, so different NICs, OS's etc.

I use a few plugins, powertop and force the SMB version/encryption in smb extras. So I next tried deleting the changes (keeping plugins) no change, next safe mode, no change, next clean install (copying across share and drive assignments+ license), no change, increase zfs_arc_max, no change, NIC flow control on/off, no change.

On the network side I've run iperf3, one way and -R, both show full speed (TCP) and steady. Using -bidir (TCP) on the mini pc shows TX at line speed but very slow to start RX (10 Mbits/sec) ending at line speed. On the windows pc, TX is at line speed and RX is consistently slow (230 Mbits/s).

First thought was network switches and cables, so I changed those. Both an Aruba switch (the normal one in use) and a Qnap one showed no difference.

To make it more confusing I ran iperf3 as UDP and got full line speed in -bidir, further suggesting the network set up is not the issue here.

When the cache writes back to the array, speed doesn't seem to be an issue and the drive temps all seem okay too.

I'm not completely sure but I'm guessing this low RX is the cause of slow smb transfers but could be wrong??

I'm running out of things to try here, any suggestions or help would be appreciated.

diagnostics-20250814-2301.zip

That Windows Explorer graph suggests a network issue, despite the good iperf results.

1st is a good transfer for 10GbE, 2nd shows consistently slower performance, network problem in this case created on purpose, 3rd shows initial fast transfer while it's being cached to RAM, and then slowed by the device's actual speed. Yours looks like the 2nd one.

I would normally agree but the initial transfer from the Synology was a line speed. Just ran a copy via the SMB mount in windows file explorer and then by SCP and SSH, both to the same location, both with a substantial speed difference.

With the initial transfer from the Synology the SMB server would have been the Synology side and the client Unraid, when Unraid is acting as the server the speed drops.

The external network to the Unraid machine doesn't seem to be the issue here, maybe a network stack/ smb server issue?

17 minutes ago, Zerix said:

maybe a network stack/ smb server issue?

It's possible.

FWIW i've never had bidir iperf work "right", usually i spin up both a server and a client on each side and run them at the same time, seems to be more reliable to assess the network.

3 hours ago, Kilrah said:

FWIW i've never had bidir iperf work "right", usually i spin up both a server and a client on each side and run them at the same time, seems to be more reliable to assess the network.

So I've now tried this, both from the windows machine and mini pc, one after the other.

Windows client -> Unraid Server:

Unraid client -> Windows server:

Then the set from the mini pc, Mini pc client -> Unraid Server:

Unraid client -> Mini pc server:

Potentially just a windows issue or an i225 issue on that front. Anything to be gleaned from the higher cwnd on the unraid -> min pc test or is this just due to the higher transfer speed?

The network side/stack is probably okay then, just windows/i225 wonkiness, SMB is a different question.

On the SMB side, an rsync transfer from the mini pc to the unraid box (vers=3.1.1, cache=none)= ~19-22MB/s transfer speed:

Remount and retry without the cache=none argument, starts fast but after 5-10 seconds stabilises out at the same ~20 MB/s.

There is plenty of cpu power/ nvme on both ends, scp works at line rate and SMB is showing low speed from two sources to the Unraid box = SMB issue? what can I look at?

4 minutes ago, Zerix said:

Potentially just a windows issue or an i225 issue on that front.

Did you try updating the NIC driver on the Windows side?

1 hour ago, Kilrah said:

Did you try updating the NIC driver on the Windows side?

Gotta love motherboard manufacturers, according to them yes but just updated it from the intel website. No change unfortunately, I think the issues with this NIC were more drop outs anyway.

The windows transfer speed and rsync from the mini pc are still way below line speed and pretty stable below line speed too, both using different NICs with nothing maxing out on the system resources.

So this is ASPM related, maybe just my setup, maybe i226-LM. Not powertop or any tweaks (tested with a clean install) just OS controlled ASPM.

This is a modern board so OS controlled ASPM is the default in the bios, thus the kernel allows L0s/L1 states. Apparently some of the ASPM states that the NIC advertises as supported don't work very well, at least not with my system combo.

Disabling ASPM completely on just the NIC appears to have solved the issue, SMB copies are showing the full 113MB/s 1Gbe line rate, iperf bidir is still funky though.

Given SCP worked before but SMB didn't this is probably a latency/timing thing messing with the way SMB specifically works.

Potentially a BIOS, igc driver, NIC firmware update or other low level thing can fix it in the future but specifically disabling L0s/L1 bits fixes the issue for the moment (will test a little more).