How to setup BTRF compression automatic apply for usershare

[Hellomynameisleo]

I've created a new user share, and my disks are formatted to BTRFS. How do I make it so UNRAID does ZSTD compression for all the files in the usershares and does it for all new ones including all files in the subdirectories as well automatically? I'm unable to see an option for it in the GUI

[Dubai Housekeeping]

BTRFS is a modern file system that supports transparent compression to reduce disk usage and improve read/write performance. You can enable BTRFS compression on a per-subvolume basis, which means that you can enable compression for specific directories or file systems.

To set up automatic BTRFS compression for a user share, you can follow these steps:

1. Identify the subvolume that contains the user share that you want to compress. You can use the btrfs subvolume list command to view a list of all subvolumes in your BTRFS file system.
2. Enable compression for the subvolume by setting the compress attribute to one of the supported compression algorithms (e.g., zstd, lzo, zlib, none). You can use the btrfs property set command to set the attribute:

 

Python Code

sudo btrfs property set -ts /path/to/subvolume compress <algorithm>
For example, to enable zstd compression for the /mnt/userdata subvolume, you can use the following command:

sudo btrfs property set -ts /mnt/userdata compress zstd

• Modify the mount options for the subvolume to enable automatic compression for newly created files. You can add the compress-force=<algorithm> option to the subvolume's mount options in the /etc/fstab file:

UUID=<UUID> /path/to/mount btrfs subvol=/path/to/subvolume,compress-force=<algorithm> 0 0
For example, to enable zstd compression for newly created files on the /mnt/userdata subvolume, you can modify the corresponding line in /etc/fstab as follows:

UUID=<UUID> /mnt/userdata btrfs subvol=/mnt/userdata,compress-force=zstd 0 0
Alternatively, you can use the mount command to modify the mount options for the subvolume:

sudo mount -o remount,compress-force=<algorithm> /path/to/mount
For example, to modify the mount options for the /mnt/userdata subvolume, you can use the following command:

sudo mount -o remount,compress-force=zstd /mnt/userdata
After following these steps, all newly created files and directories in the user share will be automatically compressed using the specified compression algorithm. Existing files will not be compressed automatically, but you can compress them manually using the btrfs filesystem defragment command with the --compress option.

Hope it will help.

[JorgeB]

1 hour ago, Hellomynameisleo said:

How do I make it so UNRAID does ZSTD compression for all the files in the usershares and does it for all new ones including all files in the subdirectories as well automatically? I'm unable to see an option for it in the GUI

GUI option will be available for v6.12, for the complete filesystem, not just a share, fou now you can enable that manually for a specific share or the whole fs.

[Hellomynameisleo]

1 hour ago, Dubai Housekeeping said:

BTRFS is a modern file system that supports transparent compression to reduce disk usage and improve read/write performance. You can enable BTRFS compression on a per-subvolume basis, which means that you can enable compression for specific directories or file systems.

To set up automatic BTRFS compression for a user share, you can follow these steps:

1. Identify the subvolume that contains the user share that you want to compress. You can use the btrfs subvolume list command to view a list of all subvolumes in your BTRFS file system.
2. Enable compression for the subvolume by setting the compress attribute to one of the supported compression algorithms (e.g., zstd, lzo, zlib, none). You can use the btrfs property set command to set the attribute:

 

Python Code

sudo btrfs property set -ts /path/to/subvolume compress <algorithm>
For example, to enable zstd compression for the /mnt/userdata subvolume, you can use the following command:

sudo btrfs property set -ts /mnt/userdata compress zstd

• Modify the mount options for the subvolume to enable automatic compression for newly created files. You can add the compress-force=<algorithm> option to the subvolume's mount options in the /etc/fstab file:

UUID=<UUID> /path/to/mount btrfs subvol=/path/to/subvolume,compress-force=<algorithm> 0 0
For example, to enable zstd compression for newly created files on the /mnt/userdata subvolume, you can modify the corresponding line in /etc/fstab as follows:

UUID=<UUID> /mnt/userdata btrfs subvol=/mnt/userdata,compress-force=zstd 0 0
Alternatively, you can use the mount command to modify the mount options for the subvolume:

sudo mount -o remount,compress-force=<algorithm> /path/to/mount
For example, to modify the mount options for the /mnt/userdata subvolume, you can use the following command:

sudo mount -o remount,compress-force=zstd /mnt/userdata
After following these steps, all newly created files and directories in the user share will be automatically compressed using the specified compression algorithm. Existing files will not be compressed automatically, but you can compress them manually using the btrfs filesystem defragment command with the --compress option.

Hope it will help.

I've tried the code in the unraid terminal but it doesn't seem to work as its in python. Is there a native code that works in the terminal right out of the gate?

[JorgeB]

Use

btrfs property set /mnt/disk_path/share compression zstd

 

Replace disk_path with disk1, disk2, cache, etc, note that you need to use the disk path, it won't work with /mnt/user, and if the shares resides in more than one disk you must set that for all disks where it exists.

[Hellomynameisleo]

6 minutes ago, JorgeB said:

Use

btrfs property set /mnt/disk_path/share compression zstd

 

Replace disk_path with disk1, disk2, cache, etc, note that you need to use the disk path, it won't work with /mnt/user, and if the shares resides in more than one disk you must set that for all disks where it exists.

thanks I used "btrfs property set /mnt/disk1/Backup compression zstd" and "btrfs property set /mnt/disk2/Backup compression zstd". I copied over a test file over to the shared named "Backup" but the filesize hasn't changed.

[JorgeB]

File size won't change, used size can be less, assuming the files are compressible.

[Hellomynameisleo]

3 minutes ago, JorgeB said:

File size won't change, used size can be less, assuming the files are compressible.

I've tested documents and images but they remain the same "size on disk" on properties. How do I see if comprression is enabled or the differenence between the uncompressed/compressed size to confirm because byte for byte there isn't a reduction at all or change compared to the original file.

[JorgeB]

It should be working but let me take a look later, it's been some time since I've tested.

[Hellomynameisleo]

2 hours ago, JorgeB said:

It should be working but let me take a look later, it's been some time since I've tested.

I've tried a bunch some other things but can't get confirmation if its working. Is there a way to see if comrpession is actually enabled and see how much its actually comrpessed?

I'm looking through the SMB share on windows 10 and don't see the files I copy over to UNRAID disk 1 being compressed at all

[JorgeB]

Give me a couple of hours and if I have some time I'll make some tests, to see the actual compression ratio you'd need an external package, someone mentioned it the other day, but don't remember the name.

[JonathanM]

Try creating a very large empty file, and see if the actual used space increases. Maybe make the file half the total size of the remaining space on the drive.

https://www.windows-commandline.com/how-to-create-large-dummy-file/

[Hellomynameisleo]

50 minutes ago, JonathanM said:

Try creating a very large empty file, and see if the actual used space increases. Maybe make the file half the total size of the remaining space on the drive.

https://www.windows-commandline.com/how-to-create-large-dummy-file/

I've created an 5GB empty text tile and it doesn't compress anything as shown in the picture. I've even set it to the highest compression setting too.

Edited February 17, 2023 by Hellomynameisleo

[JonathanM]

Try 500GB, see if available space reduces by .5TB.

5GB is too small to see a difference.

[Hellomynameisleo]

16 minutes ago, JonathanM said:

Try 500GB, see if available space reduces by .5TB.

5GB is too small to see a difference.

I created a 5GB text.txt file on Window 10 pc and copied it over and it didn't comrpess at all but if I make it directly on the UNRAID smb share it has 0 bytes size on disk as shown in test1.txt. Any idea why it behaves like this?

[primeval_god]

47 minutes ago, Hellomynameisleo said:

I've created an 5GB empty text tile and it doesn't compress anything as shown in the picture. I've even set it to the highest compression setting too.

Enabling compression in BTRFS will not automatically compress existing files. Compression will only apply to files created after the setting is applied. 

[Hellomynameisleo]

15 minutes ago, primeval_god said:

Enabling compression in BTRFS will not automatically compress existing files. Compression will only apply to files created after the setting is applied. 

I've copied over files to the UNRAID smb share and it doesn't compress them at all. 

[JorgeB]

Like mentioned before the reported size of the file won't change, this is part of my dropbox folder totaling 5.71GB.

 

Disk1 is btrfs with zstd=10, disk2 is btrfs no compression, disk3 is zfs with lz4 compression

 

P.S. since was compressing the whole disk I used

mount -o remount,compress=zstd:10 /mnt/disk1

to enable compression

[Hellomynameisleo]

9 minutes ago, JorgeB said:

Like mentioned before the reported size of the file won't change, this is part of my dropbox folder totaling 5.71GB.

 

Disk1 is btrfs with zstd=10, disk2 is btrfs no compression, disk3 is zfs with lz4 compression

 

P.S. since was compressing the whole disk I used

mount -o remount,compress=zstd:10 /mnt/disk1

to enable compression

I've used both "btrfs property set /mnt/disk2/Backup/test/ compression zstd:15" and "mount -o remount,compress=zstd:15 /mnt/disk2". And then checked it and it still doesn't work... it may be buggy for me. I've tested comrpession with 7zip and it compresses a good 30%. With both set it has no compression as far as I can see when checking unraid GUI, or command line.

[JorgeB]

5 minutes ago, Hellomynameisleo said:

And then checked it and it still doesn't work

Don't see how that's possible, again it won't change the file size you see, including the files size on disk as reported by Windows.

[Kilrah]

btrfs doesn't have a built-in tool to display compression ratio, so basically you can't easily see that it's working other than seeing free space go down less (and btrfs free space also has its own caveats/inaccuracies).

[Hellomynameisleo]

43 minutes ago, JorgeB said:

Don't see how that's possible, again it won't change the file size you see, including the files size on disk as reported by Windows.

it seems it does display it its just I had to use "mount -o remount,compress-force=zstd:15 /mnt/disk2" as it didn't compress with just the compress option

[Hellomynameisleo]

14 hours ago, JorgeB said:

Like mentioned before the reported size of the file won't change, this is part of my dropbox folder totaling 5.71GB.

 

Disk1 is btrfs with zstd=10, disk2 is btrfs no compression, disk3 is zfs with lz4 compression

 

P.S. since was compressing the whole disk I used

mount -o remount,compress=zstd:10 /mnt/disk1

to enable compression

With that command do you have to apply it every time you start up the array or reboot?

[JorgeB]

21 minutes ago, Hellomynameisleo said:

With that command do you have to apply it every time you start up the array or reboot?

Yep, it will be possible to set compression on disk by disk (and pools) using the GUI with v6.12

 

[Kilrah]

Note that zstd at level 15 causes loads of CPU usage for little benefit. There's a reason the default is 3, very diminishing returns beyond that.