Noticed the behavior since I changed the cache pool filesystem to ZFS. They are mirrored 2x2TB SSD drives. Before was using BTRFS without issues.
File copies from Windows to Unraid sometimes started normal, then gradually dropped to 0 bytes/s and eventually failed.
I created a share that's exposed a specific disk in the array, then copy files from Windows to this share directly is normal, at about 60MB/s. This is to eliminate any network issues.
Once done copying, then go into terminal to copy files to cache to test write speed, it is extremely slow. A 70GB file takes over an hour to complete. And often block other activities, such as updating docker containers. It behaves like the file copying task is blocking any other writes to the cache disks. In one severe case, the front GUI became completely non-responsive and threw 500 server error.
Reading from cache pool is always fine, speed is normal, can reach 115MB/s. This box has been running for about 11 years, with 32GB non-ECC memory. In which, 4GB is allocated to ZFS, as the ratio seems fine. Don't have any VMs, just dockers. Ran Fix Common Problems and did not find any errors or warnings.
Attached the diagnostics file.
tower-diagnostics-20240328-0607.zip