I recently created a ZFS pool with 3 2TB SSDs to use for appdata, torrents and general cache. 2 MX500 and 1 cheaper teamgroup T-Force SSD.

I created the pool in Raidz1, which seems like the best bang for both read/write speeds and data loss prevention. I have appdata and torrent going to that cache and I have noticed that whenever there is any writing to that cache pool my CPU threads spike to 100% and dockers, webgui become unresponsive. I can see using TOP that WA is spiking to over 20% which indicates I have high IO waits happening. I ran a couple of tests to confirm. I took a large ISO directory from NVME storage and copied to a couple of different shares that all have different primary storage. Writing to that pool has terrible throughput (30 MB/s) once cache fills, I have 48GB allocated for ZFS cache.

Writing to Array (no issues)

Wait IO ~5%

Throughput ~110 MB/s

Writing to ZFS SSD Pool (system sometimes will go unresponsive)

Wait IO ~20% CPU spikes

Throughput ~30 MB/s

Writing to ZFS SSD Pool - non-compressed (system sometimes will go unresponsive)

Wait IO ~22% CPU spikes - sometimes drops to below 5%

Throughput ~35 MB/s

Writing from ZFS SSD Pool

Wait IO <5%

Throughput 900 MB/s

So issue definitely seems like writing to my ZFS pool, I can write FROM the SSD pool to NVME and getting 900 MB/s so reads seem to be normal. Could it be an issue with that cheaper T-Force SSD. What else could the issue be? Should I go to Raid0 and risk losing a disk for speed? Attaching Diags...

debo-server-diagnostics-20250721-1402.zip

During a large transfer to the pool post the output from zpool iostat -vyl 10 1

I guessing this proves that Teamgroup SSD is the problem here? 3s wait vs 4ms for other two SSDs.

Yep, that's what it looks like.