Found the problem, reflinking over SMB is now enable by default with Samba, me adding "vfs objects = btrfs" was causing the problem, it's no longer needed for reflinks, though it's still a valid option, so server side copy making Samba crash with it enable now looks like a Samba bug.
Having a second vfs object "fixed" the problem because apparently Unraid only loads the last one, so basically wasn't loading "vfs objects = btrfs", on a related note if I needed to set two or more vfs objects for one share how could I do that?
P.S. reflinks over SMB now also work with XFS, assuming it's a relatively recent XFS filesystem with reflink support, so you can just right-click one or more files with Windows explorer and select copy here (or copy/paste) to make reflinks, it works with disk shares and user shares.