I tried many things suggested on this forum (and other forums) and this setting is the one that solved my problem with slow writing speed over SMB on macOS
Stop the array and then go to the Settings → SMB. In the "Samba extra configuration" put this:
[global]
strict sync = no
Apply settings, start the array, enjoy!
DISCLAIMER: I can be wrong or very wrong. I'm not an expert. Please do you own research
macOS using synchronized writes when transfering files over SMB. AFAIK Windows doesn't and that's why this problem is specific to macOS
My explanation can be wrong, this is just how I understand it. Please correct me if you know better.
Sync writes are useful in databases, VMs or some critical appliances where application must know that data is on stable storage before it continues to write next portion. macOS sends such request with every write and this is very slow. I guess it doesn't matter much for file transfers if Windows not using it and everybody seems OK with that
I can get around 500Mbps without this setting and 800-900Mbps with strict sync = no (on 1Gpbs connection)
BTW this option does not mean sync is disabled, it's just... relaxed(?)
I should also mention that I have this in my /etc/nsmb.conf on my mac
[default]
signing_required=no
I don't know if it makes any difference
Sources:
1. Old reddit post: https://www.reddit.com/r/MacOS/comments/d6obqs/a_possible_fix_for_some_of_you_with_terrible_smb/
2. Various posts on old truenas forum, for example: https://www.truenas.com/community/threads/smb-performance-on-macos.85175/#post-589857