I found the problem, updating here in case anyone else has the same issue in the future.
My network is set up as follows:
VLAN 6: MGMT
VLAN 10: Data
Other VLANs: not relevant here.
My PC is on the "Data" VLAN. Unraid has interfaces on several VLANs, including MGMT and Data, but the intention is to manage it through VLAN 6 so I can control access to it at the network level. All my management traffic (switch interfaces, Unraid, firewall, WAPs, SNMP etc.) is on the MGMT VLAN, so this makes sense to me.
When I used SSH from my laptop on VLAN 10 to access Unraid on VLAN 6, it would go through my firewall as expected, but the reply traffic was weird, because Unraid would send the replies out directly on VLAN 10, instead of on the interface they were received on to be routed back through the firewall. This caused all sorts of havoc with TCP. I'm surprised it worked at all tbh.
To resolve it, I've now set up a reverse proxy on my firewall using HAProxy, which I use to access the web interface, and a SSH proxy/jumpbox (also on the firewall) that I use for SSH access. The problem has now gone away.
I used this tutorial to set up the SSH jump box: https://medium.com/@nathan.c.cole/pfsense-ssh-proxy-edd03f9dbe60
Hope that helps someone.