December 15, 20232 yr Hi, I made a cache array with 4 x HP EX950 2TB (pcie 3.0), formatted in ZFS (raidz1, compression on, encryption on), connected dirrectly to Ryzen 9 5900x cpu (pcie x16 -> 4x pcie x4 bifurcation card), 64GB 3200MHz RAM. The problem is, I am getting really slow write/read speeds from this array, even though when testing beforehand on windows, I could achieve ~8.6GB/s write speeds and 10.4GB/s read speeds from windows raid array. I know that with zfs's raidz1 I could expect raw write speeds only up to single drive write speeds (that's should be around 2900MB/s), and the read speeds should be around 3x read speed of a single drive (that is, up to 9.45GBs). Also, I expected a drop from compression + encryption, but not that big! Here is my test results: Transfering 10gb video file between 2 cache-only shares resulted in 600MB/s, that really shocked me a bit. First thought - maybe shares overlay fs having a huge overhead, so I opened terminal and transfered file manually between 2 cache folders: pv /mnt/cache_nvme/share1/test.mkv > /mnt/cache_nvme/share2/test.mkv => 1.12GiB/s Second thought - mabe only the write performance is terrible, let's try read from nvme to RAM pv /mnt/cache_nvme/share1/test.mkv > /tmp/test.mkv => 1.29GiB/s That's like 1/7 of the theoretical max read speed and a bit more than 1/3 of theoretical write speed - And that's for a big sequential write of 10GB. What could be wrong with my setup?
December 19, 20232 yr Author Solution UPD: Somehow, when running Windows 11 inside VM with raw vfio partition, I am getting full performance, which I could expect from raidz1 array. Maybe I am just testing speed on unraid incorrectly? UPD2: Aha, I think I got it. My testing method was really bad. That was the only single-thread performance. I tried to run tests via fio utility -- and I got the numbers I expected for 12 threads. Sick! fio --name=test --rw=read --size=1G --numjobs=12 > READ: bw=12.8GiB/s (13.7GB/s), 1092MiB/s-1410MiB/s (1145MB/s-1479MB/s), io=12.0GiB (12.9GB), run=726-938msec fio --name=test --rw=write --size=1G --numjobs=12 > WRITE: bw=4625MiB/s (4849MB/s), 385MiB/s-404MiB/s (404MB/s-424MB/s), io=12.0GiB (12.9GB), run=2535-2657msec Edited December 19, 20232 yr by pacmancoder
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.