Hi all,

I have recently reshuffled my unraid server with zfs. I went from a non-encrypted zpool to a zpool with almost all datasets encrypted. I noticed a quite significant and non stable overhead on my system (see image with overall load = 85%), when copying my data (large photos of +- 30MB each) back onto the new encrypted zpool from another (single) harddisk.

 

As a debugging step I have tried the 'top' command to see what is eating CPU resources, giving me the 'z_wr_iss' process. I read that this process handles the writes to the zpool and also handles the encryption.

My CPU is a AMD Ryzen 5 5500, checking for the encryption hardware give me a positive result:

root@Deimos:~# grep -m1 -o aes /proc/cpuinfo
aes
aes

root@Deimos:~# grep module /proc/crypto | grep -v kernel | sort | uniq
module       : aesni_intel
module       : crc32_pclmul
module       : crc32c_intel
module       : crct10dif_pclmul
module       : ghash_clmulni_intel
module       : sha512_ssse3

 

Then I tried to copy that same data to an unencryted dataset to check if this cpu overhead was due to the encryption. The result is shown in the image with overall load = 30%. So it looks like the encryption is adding a significant overhead!!!

The system at idle (stopped copy tasks) is shown in the image with overall load = 2%

 

 

Finding out what encryption is used for my datasets:

root@Deimos:/boot# zfs get encryption phobos/nino
NAME         PROPERTY    VALUE        SOURCE
phobos/nino  encryption  aes-256-gcm  -

 

I read that zfs encryption has only a 1-10% performance hit vs. non-encrypted dataset in terms of speed. I read that this encryption could and/or should be handled by the aesni_intel crypto-acceleration-hardware. I copy with rsync.

 

What I'd like to know: is this large overhead normal behaviour for encryption, or is something misconfigured?

Thank you in advance!

 

 

Which Unraid version? There's a zfs bug affecting performance with some CPUs, especially with encryption. 

@JorgeB im running the latest stable version at the moment, which is version 6.12.4. I upgraded from 6.12.2 two weeks ago. If this bug is fixed in a later version, I assume that the performance is automatically fixed? E.g. I do not have to reshuffle the data again?

That is the affected release, post the output of:

 

cat /sys/module/zfs/parameters/zfs_vdev_raidz_impl
cat /sys/module/zcommon/parameters/zfs_fletcher_4_impl

 

34 minutes ago, NytoxRex said:

I assume that the performance is automatically fixed?

Correct

Running the suggested code

cat /sys/module/zfs/parameters/zfs_vdev_raidz_impl
cat /sys/module/zcommon/parameters/zfs_fletcher_4_impl

 

Gives:

cycle [fastest] original scalar sse2 ssse3
[fastest] scalar superscalar superscalar4 sse2 ssse3

 

Though, I dont really know what this means

AVX2 is not recognized, so it uses a slower implementation, it's this issue:

 

https://github.com/openzfs/zfs/issues/15223

 

Downgrading to v6.12.3 should confirm it's this, but you can also just wait for a newer Unraid release with a newer fixed kernel.

Thanks! Ill just wait for the new release. Ill reply to this thread if the issue is resolved.

Okay, waiting around for a new version of unraid got me a little impatient... Via the Tools -> Update OS, I reverted back to version 6.12.2. I am happy to report that the encryted datasets can write data much more efficiently now. Here a screenshot of the cpu usage now with the same workload:

This is more in line with my expectation.
Thanks for the help!