compression on unraid


Recommended Posts

5 minutes ago, itimpi said:

It is worth thinking if compression is likely to gain anything to justify the extra complexity.   Most unRAID users seem to use their servers primarily for storing media and since media is normally already compressed little is gained by adding file system level compression.

 

It's normally when I need to store large log files that compression is meaningful - I want the data to be easily accessible without having to stream through a decompression software.

Source code files are normally much too small to make a difference - It would make a difference if the machine is hosting a huge public Git server with millions or billions of source files.

Databases needs constant updates where the database update logic interferes with the block-level compression of the file system.

And my largest disks normally stores huge but pre-compressed media files.

 

One note is that many SSD performs compression on the inside, to reduce media wear. That's because a lot of SSD writes are made to easily compressed structures like the meta-data used by the file system layer. And writes to files that changes are normally done on files that compresses well. And a SSD is much more likely used as a system disk for a PC than a storage volume of large media files. So quite a lot of files traditionally stored on a SSD can compress quite well.

Link to comment

I was looking at https://btrfs.wiki.kernel.org/index.php/Compression 

 

Quote

Can I force compression on a file without using the compress mount option?

Yes. The utility chattr supports setting file attribute c that marks the inode to compress newly written data. Setting the compression property on a file using btrfs property set <file> compression <zlib|lzo|zstd> will force compression to be used on that file using the specified algorithm.

 

Would this possibly be used by the shfs bit (that does all the /mnt/users magic) per filetype? Imagine setting it for *.mov, *.wav, etc. only.

 

Quote

How can I recursively compress/uncompress a directory (including guessed/forced-compression)

Uset the btrfs filesystem defrag command, the option -r will process the files recursively in a directory. This is independent of the mount options compress or compress-force, and using the option -c you can set the compression algorithm.

Currently (v4.14), it's not possible to select "no compression", using the defrag command. This may change in the future.

 

Or a plugin to run this per share?

 

Note: I'm new to unRAID, I just had my first installation and I'm midway through my very first parity-sync run... I'm on the default xfs and strongly considering to reformat to btrfs to experiment with compression.

Link to comment

You can enable compression different ways, for example remounting the disk with compression enable after array start, e.g. by using the user script plugin, but probably the easiest way is use chattr to set a share (or folder) to compressed, then all files inside that share/folder will inherit compression.

  • Like 2
Link to comment
5 hours ago, johnnie.black said:

You can enable compression different ways, for example remounting the disk with compression enable after array start, e.g. by using the user script plugin, but probably the easiest way is use chattr to set a share (or folder) to compressed, then all files inside that share/folder will inherit compression.

is any tutorial in the forums that i can do ?

Link to comment
7 minutes ago, JohnPeace said:

is any tutorial in the forums that i can do ?

 

Just set the share or folder you want to use compression with chattr, just note that since each disk is a separate filesystem it needs to be done on every disk that the share is using (if you want to use compression on all) e.g.:

 

chattr +c /mnt/disk1/Share
chattr +c /mnt/disk2/Share

etc

  • Upvote 1
Link to comment
2 minutes ago, johnnie.black said:

 

Just set the share or folder you want to use compression with chattr, just note that since each disk is a separate filesystem it needs to be done on every disk that the share is using (if you want to use compression on all) e.g.:

 


chattr +c /mnt/disk1/Share

chattr +c /mnt/disk2/Share

etc

in other word create a script that run on the start of the array with that

Link to comment
Just now, JohnPeace said:

in other word create a script that run on the start of the array with that

No script needed, you just do it one time and it stays on, but it will only start compressing new files added to that share after +c is set, if at some point in the future you want to turn it off use chattr -c, but similarly it will only affect new/changed files.

Link to comment

Lovely, I'll try that with the user script plugin :)

 

Question: I stopped the array, changed for each drive the fs from xfs to btrfs, started the array and reformatted my drives, but the parity is reported valid... I was expecting that a reformat with change of fs to invalidate the parity? Should I run parity again before I add any files?

Link to comment
1 hour ago, johnnie.black said:

I did some testing at work, and since there's not an easy way to check the compress size, I used two empty disks on my work server, both have same 512x2MB text files, these as expected are very compressible and you can see the difference: 1.09GB vs 86.3MB used

 

1348689476_btrfscompression.png.a10cd855ed3c6d2e0863e24a5e3a725d.png

 

 

 

Was it log files? Not too often to get better than 10x compression unless there are lots of regular information in the files.

  • Like 1
Link to comment
12 hours ago, Laurian said:
Apparently this https://github.com/kilobyte/compsize can be used to check the compressed size.

 

Hmmm, but I have to compile it, I can get gcc from the Dev Tools plugin, but I also need btrfs headers...

 

I also saw that but would expect to be difficult to compile it for unRAID, if you're using compression on the whole disk you can check the global compression with this, e.g.:

 

Total (uncompressed) file size:

root@Tower9:~# du -sh /mnt/disk4
1.1G    /mnt/disk4
 

Actual used space:

 

root@Tower9:~# df -h /mnt/disk4
Filesystem      Size  Used Avail Use% Mounted on/dev/md4
		466G   83M  465G   1% /mnt/disk4
 

To get compression ratio divide the outputs, in this case it's around 13:1

 

 

 

 

Edited by johnnie.black
  • Like 1
Link to comment

OK, so with df I get

 

root@CUBE:~# du -sh /mnt/disk1
304G    /mnt/disk1

root@CUBE:~# df -h /mnt/disk1
Filesystem      Size  Used Avail Use% Mounted on
/dev/md1        3.7T  291G  3.4T   8% /mnt/disk1

I managed to compile compsize, and it shows me this

root@CUBE:~# /root/compsize/compsize /mnt/disk1
Processed 5060 files, 1075817 regular extents (1075817 refs), 176 inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL       95%      289G         303G         303G
none       100%      226G         226G         226G
zlib        81%       63G          77G          77G

I compiled compsize like this:

  1. somewhere (I was in /root) I cloned https://github.com/kilobyte/compsize
  2. I installed the Dev Pack plugin, and installed all the packages inside (you could probably cherry picking them)
  3. I downloaded btrfs-progs (the one already installed had no headers) from (see note in red at the end of this post) https://slackware.pkgs.org/14.2/slackware-i486/btrfs-progs-v4.5.3-i586-1.txz.html
  4. I upgraded btrfs-progs with upgradepkg --install-new btrfs-progs-v4.5.3-i586-1.txz (upgradepkg reported: btrfs-progs-v4.15.1-x86_64-1 upgraded with new package ./btrfs-progs-v4.5.3-i586-1.txz)
  5. In the cloned compsize repo, I ran make and got by that the compsize binary in there

 

Note: I'm on unRAID 6.5.3 (as per /etc/unraid-version) and apparently that's Slackware 14.2 (as per /etc/slackware-version), this version should match in the URL of the package you download.

 

And I just noticed I installed from slackware-i486 instead of slackware-x86_64, upgraded now to the one from https://slackware.pkgs.org/14.2/slackware-x86_64/btrfs-progs-v4.5.3-x86_64-1.txz.html for consistency (and safety I guess); I recompiled compsize and the report on compression is the same.

 

Edited by Laurian
style fixes
  • Like 2
Link to comment
  • 2 months later...

Thanks so much for this info, I've been desperate to implement compression on my huge collection of Sony RAW image files.

 

I copied across 20GB of files as a test and my compsize output looks like this:

Processed 568 files, 32187 regular extents (32187 refs), 0 inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL       86%       16G          19G          19G
none       100%       15G          15G          15G
zlib        25%      948M         3.6G         3.6G

 

I think it's saying that the folder is 86% of what it was, i.e. it's 16G on disk rather than 19GB which is the uncompressed size. Is that correct?

 

Link to comment
  • 5 years later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.