JohnPeace Posted July 21, 2018 Share Posted July 21, 2018 Hello is any plugin to enable compression on unraid? Quote Link to comment
JorgeB Posted July 22, 2018 Share Posted July 22, 2018 Not AFAIK, but you can use btrfs as the filesystem and it does support compression. Quote Link to comment
itimpi Posted July 22, 2018 Share Posted July 22, 2018 It is worth thinking if compression is likely to gain anything to justify the extra complexity. Most unRAID users seem to use their servers primarily for storing media and since media is normally already compressed little is gained by adding file system level compression. Quote Link to comment
pwm Posted July 22, 2018 Share Posted July 22, 2018 5 minutes ago, itimpi said: It is worth thinking if compression is likely to gain anything to justify the extra complexity. Most unRAID users seem to use their servers primarily for storing media and since media is normally already compressed little is gained by adding file system level compression. It's normally when I need to store large log files that compression is meaningful - I want the data to be easily accessible without having to stream through a decompression software. Source code files are normally much too small to make a difference - It would make a difference if the machine is hosting a huge public Git server with millions or billions of source files. Databases needs constant updates where the database update logic interferes with the block-level compression of the file system. And my largest disks normally stores huge but pre-compressed media files. One note is that many SSD performs compression on the inside, to reduce media wear. That's because a lot of SSD writes are made to easily compressed structures like the meta-data used by the file system layer. And writes to files that changes are normally done on files that compresses well. And a SSD is much more likely used as a system disk for a PC than a storage volume of large media files. So quite a lot of files traditionally stored on a SSD can compress quite well. Quote Link to comment
Laurian Posted July 24, 2018 Share Posted July 24, 2018 I was looking at https://btrfs.wiki.kernel.org/index.php/Compression Quote Can I force compression on a file without using the compress mount option? Yes. The utility chattr supports setting file attribute c that marks the inode to compress newly written data. Setting the compression property on a file using btrfs property set <file> compression <zlib|lzo|zstd> will force compression to be used on that file using the specified algorithm. Would this possibly be used by the shfs bit (that does all the /mnt/users magic) per filetype? Imagine setting it for *.mov, *.wav, etc. only. Quote How can I recursively compress/uncompress a directory (including guessed/forced-compression) Uset the btrfs filesystem defrag command, the option -r will process the files recursively in a directory. This is independent of the mount options compress or compress-force, and using the option -c you can set the compression algorithm. Currently (v4.14), it's not possible to select "no compression", using the defrag command. This may change in the future. Or a plugin to run this per share? Note: I'm new to unRAID, I just had my first installation and I'm midway through my very first parity-sync run... I'm on the default xfs and strongly considering to reformat to btrfs to experiment with compression. Quote Link to comment
JorgeB Posted July 24, 2018 Share Posted July 24, 2018 You can enable compression different ways, for example remounting the disk with compression enable after array start, e.g. by using the user script plugin, but probably the easiest way is use chattr to set a share (or folder) to compressed, then all files inside that share/folder will inherit compression. 2 Quote Link to comment
JohnPeace Posted July 24, 2018 Author Share Posted July 24, 2018 5 hours ago, johnnie.black said: You can enable compression different ways, for example remounting the disk with compression enable after array start, e.g. by using the user script plugin, but probably the easiest way is use chattr to set a share (or folder) to compressed, then all files inside that share/folder will inherit compression. is any tutorial in the forums that i can do ? Quote Link to comment
JorgeB Posted July 24, 2018 Share Posted July 24, 2018 7 minutes ago, JohnPeace said: is any tutorial in the forums that i can do ? Just set the share or folder you want to use compression with chattr, just note that since each disk is a separate filesystem it needs to be done on every disk that the share is using (if you want to use compression on all) e.g.: chattr +c /mnt/disk1/Share chattr +c /mnt/disk2/Share etc 1 Quote Link to comment
JohnPeace Posted July 24, 2018 Author Share Posted July 24, 2018 2 minutes ago, johnnie.black said: Just set the share or folder you want to use compression with chattr, just note that since each disk is a separate filesystem it needs to be done on every disk that the share is using (if you want to use compression on all) e.g.: chattr +c /mnt/disk1/Share chattr +c /mnt/disk2/Share etc in other word create a script that run on the start of the array with that Quote Link to comment
JorgeB Posted July 24, 2018 Share Posted July 24, 2018 Just now, JohnPeace said: in other word create a script that run on the start of the array with that No script needed, you just do it one time and it stays on, but it will only start compressing new files added to that share after +c is set, if at some point in the future you want to turn it off use chattr -c, but similarly it will only affect new/changed files. Quote Link to comment
Laurian Posted July 24, 2018 Share Posted July 24, 2018 Lovely, I'll try that with the user script plugin Question: I stopped the array, changed for each drive the fs from xfs to btrfs, started the array and reformatted my drives, but the parity is reported valid... I was expecting that a reformat with change of fs to invalidate the parity? Should I run parity again before I add any files? Quote Link to comment
JorgeB Posted July 24, 2018 Share Posted July 24, 2018 4 minutes ago, Laurian said: but the parity is reported valid. It is valid, parity is updated during the format. 1 Quote Link to comment
JorgeB Posted July 24, 2018 Share Posted July 24, 2018 I did some testing at work, and since there's not an easy way to check the compress size, I used two empty disks on my work server, both have same 512x2MB text files, these as expected are very compressible and you can see the difference: 1.09GB vs 86.3MB used 1 Quote Link to comment
pwm Posted July 24, 2018 Share Posted July 24, 2018 1 hour ago, johnnie.black said: I did some testing at work, and since there's not an easy way to check the compress size, I used two empty disks on my work server, both have same 512x2MB text files, these as expected are very compressible and you can see the difference: 1.09GB vs 86.3MB used Was it log files? Not too often to get better than 10x compression unless there are lots of regular information in the files. 1 Quote Link to comment
Laurian Posted July 24, 2018 Share Posted July 24, 2018 Apparently this https://github.com/kilobyte/compsize can be used to check the compressed size. Hmmm, but I have to compile it, I can get gcc from the Dev Tools plugin, but I also need btrfs headers... 1 Quote Link to comment
JorgeB Posted July 25, 2018 Share Posted July 25, 2018 Was it log files? Not too often to get better than 10x compression unless there are lots of regular information in the files. Yes, I used those since I knew they would compress a lot. 1 Quote Link to comment
JorgeB Posted July 25, 2018 Share Posted July 25, 2018 (edited) 12 hours ago, Laurian said: Apparently this https://github.com/kilobyte/compsize can be used to check the compressed size. Hmmm, but I have to compile it, I can get gcc from the Dev Tools plugin, but I also need btrfs headers... I also saw that but would expect to be difficult to compile it for unRAID, if you're using compression on the whole disk you can check the global compression with this, e.g.: Total (uncompressed) file size: root@Tower9:~# du -sh /mnt/disk4 1.1G /mnt/disk4 Actual used space: root@Tower9:~# df -h /mnt/disk4 Filesystem Size Used Avail Use% Mounted on/dev/md4 466G 83M 465G 1% /mnt/disk4 To get compression ratio divide the outputs, in this case it's around 13:1 Edited July 25, 2018 by johnnie.black 1 Quote Link to comment
Laurian Posted July 26, 2018 Share Posted July 26, 2018 (edited) OK, so with df I get root@CUBE:~# du -sh /mnt/disk1 304G /mnt/disk1 root@CUBE:~# df -h /mnt/disk1 Filesystem Size Used Avail Use% Mounted on /dev/md1 3.7T 291G 3.4T 8% /mnt/disk1 I managed to compile compsize, and it shows me this root@CUBE:~# /root/compsize/compsize /mnt/disk1 Processed 5060 files, 1075817 regular extents (1075817 refs), 176 inline. Type Perc Disk Usage Uncompressed Referenced TOTAL 95% 289G 303G 303G none 100% 226G 226G 226G zlib 81% 63G 77G 77G I compiled compsize like this: somewhere (I was in /root) I cloned https://github.com/kilobyte/compsize I installed the Dev Pack plugin, and installed all the packages inside (you could probably cherry picking them) I downloaded btrfs-progs (the one already installed had no headers) from (see note in red at the end of this post) https://slackware.pkgs.org/14.2/slackware-i486/btrfs-progs-v4.5.3-i586-1.txz.html I upgraded btrfs-progs with upgradepkg --install-new btrfs-progs-v4.5.3-i586-1.txz (upgradepkg reported: btrfs-progs-v4.15.1-x86_64-1 upgraded with new package ./btrfs-progs-v4.5.3-i586-1.txz) In the cloned compsize repo, I ran make and got by that the compsize binary in there Note: I'm on unRAID 6.5.3 (as per /etc/unraid-version) and apparently that's Slackware 14.2 (as per /etc/slackware-version), this version should match in the URL of the package you download. And I just noticed I installed from slackware-i486 instead of slackware-x86_64, upgraded now to the one from https://slackware.pkgs.org/14.2/slackware-x86_64/btrfs-progs-v4.5.3-x86_64-1.txz.html for consistency (and safety I guess); I recompiled compsize and the report on compression is the same. Edited July 26, 2018 by Laurian style fixes 2 Quote Link to comment
ThatDude Posted October 1, 2018 Share Posted October 1, 2018 Thanks so much for this info, I've been desperate to implement compression on my huge collection of Sony RAW image files. I copied across 20GB of files as a test and my compsize output looks like this: Processed 568 files, 32187 regular extents (32187 refs), 0 inline. Type Perc Disk Usage Uncompressed Referenced TOTAL 86% 16G 19G 19G none 100% 15G 15G 15G zlib 25% 948M 3.6G 3.6G I think it's saying that the folder is 86% of what it was, i.e. it's 16G on disk rather than 19GB which is the uncompressed size. Is that correct? Quote Link to comment
JorgeB Posted October 1, 2018 Share Posted October 1, 2018 9 minutes ago, ThatDude said: I think it's saying that the folder is 86% of what it was, i.e. it's 16G on disk rather than 19GB which is the uncompressed size. Is that correct? Correct. Quote Link to comment
ThatDude Posted October 1, 2018 Share Posted October 1, 2018 Great! Thank you. Hopefully the team will add this functionality into the unRAID shares tab for BTRFS volumes in the future. Quote Link to comment
ThatDude Posted November 5, 2023 Share Posted November 5, 2023 (edited) Sorry to resurrect this old thread, I rebuilt my unRAID and forgot to backup my compsize binary, is there any chance one of you could link to yours? I recall building it being a PITA to compile Edited November 5, 2023 by ThatDude grammar Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.