Consider using compress-force instead of compress for Btrfs compression

Don't have a strong opinion one way or the other, from previous searches I see some recommendations to use compress, others to use compress-force, I believe Arch Linux also defaults to compress=zstd:3, I have two file systems that I am considering compressing (the other ones have mostly media files so compression doesn't really help), one is used for docker containers, the other for my VMs, did some tests using those files since it's the ones I'm interested in how they perform, test was done copying all the files from both filesystems (around 2k small files from the containers and some larger ones and 5 larger vdisks) from one NVMe device (formatted xfs) to another, CPU in this test server is a quad core Skylake Xeon (E3-1220 v5), cp speed is based on the uncompressed size:

 

                             time     used space      cp speed
uncompressed              7m57.281s     311GB        651.61MB/s
compress=zstd:3           7m41.138s     306GB        674.42MB/s
compress-force=zstd:3    13m43.869s     163GB        377.49MB/s

 

While this shows that like you mentioned some files (most in this case) will not get compressed with just the compress option, it also shows that the performance is clearly affected with compress-force, of course for the actual use case, i.e., docker/VMs speed doesn't really get near the copy speeds, so possibly not a big impact, curiously, and since I'm also considering converting these to zfs, also tested zfs uncompressed and compressed and with zfs the performance is much better with compression enabled, better than the uncompressed speed, with a compression ratio just a little lower than btrfs compress-force, but it uses lz4, not zstd, so differences are expected.

 

Anyway it's just one data point, and it does look like just compress might skip a lot of compressible files, but performance suffers, I guess the best setting depends on your priority, space or speed.

4 hours ago, JorgeB said:

Don't have a strong opinion one way or the other, from previous searches I see some recommendations to use compress, others to use compress-force, I believe Arch Linux also defaults to compress=zstd:3, I have two file systems that I am considering compressing (the other ones have mostly media files so compression doesn't really help), one is used for docker containers, the other for my VMs, did some tests using those files since it's the ones I'm interested in how they perform, test was done copying all the files from both filesystems (around 2k small files from the containers and some larger ones and 5 larger vdisks) from one NVMe device (formatted xfs) to another, CPU in this test server is a quad core Skylake Xeon (E3-1220 v5), cp speed is based on the uncompressed size:

 

                             time     used space      cp speed
uncompressed              7m57.281s     311GB        651.61MB/s
compress=zstd:3           7m41.138s     306GB        674.42MB/s
compress-force=zstd:3    13m43.869s     163GB        377.49MB/s

 

While this shows that like you mentioned some files (most in this case) will not get compressed with just the compress option, it also shows that the performance is clearly affected with compress-force, of course for the actual use case, i.e., docker/VMs speed doesn't really get near the copy speeds, so possibly not a big impact, curiously, and since I'm also considering converting these to zfs, also tested zfs uncompressed and compressed and with zfs the performance is much better with compression enabled, better than the uncompressed speed, with a compression ratio just a little lower than btrfs compress-force, but it uses lz4, not zstd, so differences are expected.

 

Anyway it's just one data point, and it does look like just compress might skip a lot of compressible files, but performance suffers, I guess the best setting depends on your priority, space or speed.

The reason the compress option instead of compress-force was faster is *because* it straight up just didn't compress your files for the most part  . You might as well not use compression with the compress mount option, as the algorithm is known to be poor. Many files can have headers or first portions not compressible, yet the file straight up gets skipped based on the first 64KiB.

 

Keep in mind, filesystem fragmentation can also play a part in these benchmarks. I'm not certain how fragmented it was, particularly free space. If you did a full balance in particular, free space would be less fragmented and that could improve performance of compress-force. The referenced benchmark was performed on a ramdisk to eliminate that bottleneck.

 

At any rate, the fact ZFS is faster is just yet another example of the downsides of btrfs in my opinion. ZFS is always going to be faster. However, in my opinion, if people want compression, it's a given it will have CPU overhead. The point is to trade CPU for disk space, so in my opinion, this just further shows how the force option should be used.

 

I've actually been doing this on UnRAID manually via a remount script with user scripts. I was hopeful the built in compress option would allow me to eliminate that.

 

Most in the btrfs community recommend it over the standard compress option, especially with zstd compression. Like everything btrfs, the official documentation can be outdated.

Edited March 27, 2023 by JSE

10 hours ago, JSE said:

Keep in mind, filesystem fragmentation can also play a part in these benchmarks.

It should not affect these tests since the files are being copied from a new xfs filesystem to a new btrfs/zfs filesystem.

 

10 hours ago, JSE said:

At any rate, the fact ZFS is faster is just yet another example of the downsides of btrfs in my opinion. ZFS is always going to be faster. However, in my opinion, if people want compression, it's a given it will have CPU overhead. The point is to trade CPU for disk space, so in my opinion, this just further shows how the force option should be used.

 

Yeah, but zfs with compression with this data is faster that zfs/btrs without compression, these were the results of the same test with zfs:

 

                        time      used space        cp speed
uncompressed zfs      8m37.722s      320GB         618.09MB/s
compressed zfs        5m08.942s      185GB        1035.80MB/s

 

 

1 hour ago, JorgeB said:

It should not affect these tests since the files are being copied from a new xfs filesystem to a new btrfs/zfs filesystem.

Hmmm interesting, then it should also have been faster tbh.

 

1 hour ago, JorgeB said:

Yeah, but zfs with compression with this data is faster that zfs/btrs without compression

What I meant by that is ZFS is just a better fs when it comes to performance. ZFS' ARC caching alone is a big difference. Btrfs has a lot of performance downsides.

 

That said, many people do see performance upsides with Btrfs transparent compression and compress-force, since often the CPU is capable of compressing data faster and the block device is the bottleneck. I wonder if a lower compression level would help. If you could try :2 or :1, you might see much better performance.

 

I'm not personally asking for adding compression level support in this release, it seems to be planned for later. I would just prefer compress would be switched out for compress-force so that compression can actually work

10 minutes ago, JSE said:

I would just prefer compress would be switched out for compress-force so that compression can actually work

Yeah, it would be nice to see more tests with different data but I'm inclined to agree, looks like just compress might not actually do much, LT plans to add compression levels later, so possibly this can be changed at that time, or have both options with compress-force as the default.

I think it would be nice to have it swapped out for compress-force now, before the next stable release. The current option should just straight up be forgot about imo. It's kinda useless.