Ok, so I think I got it, maybe.
I turned off arc cache on all datasets and then I restarted the server.
When started, the arc used 7% of ram and it started to build up to 34% and as it did the read and writes increased too.
So I think this has something to do with that zfs reads and writes data to ram "all the time".
Clearly some data is cached to ram even if you tun it off on the datasets, else the zfs ram usage would not increase.
But it would be good to get some info on this from someone who knows zfs better