Best approach to benchmark write/read speed on zfs with/without compression enabled?

[Swarles]

TLDR: How can I create a repeatable test for read/write performance on a zfs file system (6.12-RC8) to compare the results when compression is enabled and disabled on a disk? Intending to test the array and a mirrored cache pool.

 

Hey everyone, 

So I recently put together my low-power unraid build and ultimately used 6.12.0-rc8 to create a zfs array and a mirrored cache pool also using zfs. I'm struggling to decide whether or not I turn on compression for all my drives, and want to compare it. In general I want compression for extra space and transfer speed benefits, but before I commit and move data onto the server with compression I want to see if this will have significant impact on my power consumption. My server will host movies (hevc) as well as system backups, documents, photos, etc. I know that for already compressed data having compression on is pointless, but from research it seems that lz4 compression is pretty quick to give up on already compressed data so it might be a non-issue.

 

Ideally what I want to achieve is a test where I can iterate a file read/write process for compressible and non-compressible data on both the cache and the array. This way the test is easily repeatable and I can simultaneously monitor CPU load as well as my power meter to determine the impact on power draw. I tried using the DiskSpeed docker with the intention to see if there was a difference between compression enabled and disabled, but it appears the tests do not work on zfs. I'm also not sure what type of data it writes and if it would be relevant for my purposes.

 

Currently my system idles at about 10W with my dockers running and the disks spun down. I reach about 21W when both disks are spinning (no read/writes). If on-the-fly compression and decompression adds only a couple extra watts then it will be worth enabling it for the storage and read/write benefits.

 

Appreciate any help, thanks!

[JorgeB]

I would create a folder with representative sample data like the one you plan to use, make sure it's well over the ARC size, then copy that with compression enabled, reboot and re-copy with compression disabled and compare.

[Swarles]

@JorgeB Okay yeah I was thinking that would probably be the simplest way to test it. I'm wondering what would be the best location to copy the data from? I definitely cannot saturate the write/read speeds of the server as my network is only 1gbe currently. If I connect ethernet directly to my server I can achieve 2.5gbe speeds with a USB 2.5gbe adapter on a personal PC, I suppose this is the best I can do. Or perhaps I am over thinking it and I only need to move data between the cache pool and the array? 

By ARC size you are referring to "adaptive replacement cache"? If this is the case, my memory is only 16GB but I will probably try collate sample data of 30GB+.

 

I'm also wondering the impact on the CPU for decompression for each user that may be using Jellyfin, as there is a lot of files being accessed in this case (media, posters, audio, subtitles, bif, etc). Do you think the best way to test this would be to clone a media share and additional jellyfin docker; One share with compression enabled and one with it disabled?

 

Thank you

[JonathanM]

2 minutes ago, Swarles said:

(media, posters, audio, subtitles, bif, etc)

Why are you trying to compress already highly compressed and therefore relatively incompressible files? Or is a large percentage of your media in old formats with poor native compression?

[JorgeB]

Cache to array will likley be limited by array write speed, if LAN is not fast enough best you'd be to a fast UD device and copy from that to the pool.

 

32 minutes ago, Swarles said:

By ARC size you are referring to "adaptive replacement cache"? If this is the case, my memory is only 16GB

By default with Unraid ARC size will be 1/8 installed RAM, so just 2G.

 

Like mentioned by Jonatham don't expect much compression from already compressed data, though it should slow down writes noticeably, unless CPU is very low power.

[Swarles]

18 minutes ago, JonathanM said:

Why are you trying to compress already highly compressed and therefore relatively incompressible files? Or is a large percentage of your media in old formats with poor native compression?

While I do have some older media, it’s certainly not enough to warrant trying to compress it. It’s not that I want to compress this media, it’s more that I just want to set and forget “compression: on” for each of my zfs drives in my configuration (which is all of them at this point).

 

This reddit post combined with other research has lead me to believe that lz4 compression (from my understanding Unraid zfs is using lz4 by default) is pretty quick to stop attempting to compress an already compressed file. My thought process is that performance and power impact is probably negligible in this case as it skips compression on most of these files. But I’m also not sure if this will actually be the case and figure I should test it.

[JorgeB]

9 minutes ago, Swarles said:

My thought process is that performance and power impact is probably negligible in this case as it skips compression on most of these files

Should be, but let us know what you find.

[Swarles]

Thank you for the feedback, both of you, I really appreciate it. 

10 minutes ago, JorgeB said:

Cache to array will likley be limited by array write speed, if LAN is not fast enough best you'd be to a fast UD device and copy from that to the pool.

Yeah I think this approach makes sense for testing the cache pool. I may need to find a way to add an additional m.2 to my system as I don’t own  any sata ssd.

 

16 minutes ago, JorgeB said:

By default with Unraid ARC size will be 1/8 installed RAM, so just 2G.

This is good to know thanks. Although, the memory dashboard reports that “zfs” is always using 30-50% (usually whatever I have left free).

 

Overall it is my hope that by enabling compression on the drives it saves space where it can (for not that much more power increase) and quickly skips files that would be ineffective to do so. I’ll see what I can do to get some conclusive results. Thanks

[JorgeB]

2 minutes ago, Swarles said:

This is good to know thanks. Although, the memory dashboard reports that “zfs” is always using 30-50% (usually whatever I have left free).

That can be confusing, it represents 30-50% of the used RAM, not installed RAM.

[Swarles]

Okay so I did some initial basic testing to get an idea of which route I would like to take but ideally would like to do more rigorous and quantitative testing soon.

Using one sample data set, I tested 3 primary behaviours:

1. PC to Cache (over 2.5Gbe smb)
2. Cache to PC (over 2.5Gbe smb)
3. Cache to Array

The data set was 52.4 GB consisting of some tv series taken directly from Jellyfin with metadata (35.6 GB) as well as a variety of personal documents (word, pdf, programming, CAD files, etc : 16.7 GB). For simplicity I will call the 35.6GB portion “TV” and the 16.7 GP portion “Files”. I only performed each test once, logging time taken and average transfer speed, while manually observing CPU load and power consumption.

 

RESULTS:

In each case the compression resulted in slightly faster transfer speeds, but it should be noted in this case, the primary bottleneck for transfer speed was not the read or write speeds and instead the network so it is expected they should be similar. I observed that power consumption was similar perhaps +3W in general with compression turned on.

I should have tested from the array to my PC and not the Cache to PC because again, this result is limited by the LAN and not the read/write speeds. Interestingly the decompression took a bit longer, I am not sure why this is the case. CPU load was similar as well as power consumption, maybe +3W on average.

The results of compression start to show here where compression allows for faster transfer time onto the array for data that can be compressed. Interestingly I observed the CPU load with compression turned off to fluctuate a lot more and reached much higher loads and watts than previously seen in any test. Not sure if this is just a coincidence as I ran the test once, this was also only for part of the test where it then returned to expected figures. Again, in general compression turned on may only be increasing loads by ~3W.

 

I have ordered an m.2 enclosure so I can perform these tests again but with a faster connection. I would also like to properly log CPU load and power consumption for a quantitative comparison, so will investigate ways to do this.

 

In general, it would appear enabling compression is beneficial for both space savings and transfer speed (except for reads off cache, will need to investigate this) for a little bit of extra CPU load and power consumption. I’d recommend having compression turned on for anyone running zfs on their cache and array in new unraid.

[ivanavich]

Some interesting info here regarding compression from @SpaceInvaderOne

 

https://unraid.net/blog/zfs-guide

Edited September 13, 2023 by ivanavich