[Dynamix File Integrity plugin]

I would love a different hashing algorithm: XXH64 https://github.com/Cyan4973/xxHash

 

It is not only faster but that data is compatible with https://mediahashlist.org/mhl-specification/ 

(yes, that spec does MD5 too, but other tools using that spec use XXH64)

 

Actually MHL file support would be useful too, to test if the files were not corrupted on copy to unRAID or to other remote disks.

 

On macOS I use https://hedge.video/ for my media transfer and it creates such files.

 

In the meantime I'll see what I can do with custom scripts and https://github.com/pomfort/mhl-tool 

 

[compression on unraid]

OK, so with df I get

 

root@CUBE:~# du -sh /mnt/disk1
304G    /mnt/disk1

root@CUBE:~# df -h /mnt/disk1
Filesystem      Size  Used Avail Use% Mounted on
/dev/md1        3.7T  291G  3.4T   8% /mnt/disk1

I managed to compile compsize, and it shows me this

root@CUBE:~# /root/compsize/compsize /mnt/disk1
Processed 5060 files, 1075817 regular extents (1075817 refs), 176 inline.
Type       Perc     Disk Usage   Uncompressed Referenced
TOTAL       95%      289G         303G         303G
none       100%      226G         226G         226G
zlib        81%       63G          77G          77G

I compiled compsize like this:

1. somewhere (I was in /root) I cloned https://github.com/kilobyte/compsize
2. I installed the Dev Pack plugin, and installed all the packages inside (you could probably cherry picking them)
3. I downloaded btrfs-progs (the one already installed had no headers) from (see note in red at the end of this post) https://slackware.pkgs.org/14.2/slackware-i486/btrfs-progs-v4.5.3-i586-1.txz.html
4. I upgraded btrfs-progs with upgradepkg --install-new btrfs-progs-v4.5.3-i586-1.txz (upgradepkg reported: btrfs-progs-v4.15.1-x86_64-1 upgraded with new package ./btrfs-progs-v4.5.3-i586-1.txz)
5. In the cloned compsize repo, I ran make and got by that the compsize binary in there

 

Note: I'm on unRAID 6.5.3 (as per /etc/unraid-version) and apparently that's Slackware 14.2 (as per /etc/slackware-version), this version should match in the URL of the package you download.

 

And I just noticed I installed from slackware-i486 instead of slackware-x86_64, upgraded now to the one from https://slackware.pkgs.org/14.2/slackware-x86_64/btrfs-progs-v4.5.3-x86_64-1.txz.html for consistency (and safety I guess); I recompiled compsize and the report on compression is the same.

 

[compression on unraid]

Apparently this https://github.com/kilobyte/compsize can be used to check the compressed size.

 

Hmmm, but I have to compile it, I can get gcc from the Dev Tools plugin, but I also need btrfs headers...

[compression on unraid]

Lovely, I'll try that with the user script plugin

 

Question: I stopped the array, changed for each drive the fs from xfs to btrfs, started the array and reformatted my drives, but the parity is reported valid... I was expecting that a reformat with change of fs to invalidate the parity? Should I run parity again before I add any files?

[compression on unraid]

I was looking at https://btrfs.wiki.kernel.org/index.php/Compression 

 

Quote

Can I force compression on a file without using the compress mount option?

Yes. The utility chattr supports setting file attribute c that marks the inode to compress newly written data. Setting the compression property on a file using btrfs property set <file> compression <zlib|lzo|zstd> will force compression to be used on that file using the specified algorithm.

 

Would this possibly be used by the shfs bit (that does all the /mnt/users magic) per filetype? Imagine setting it for *.mov, *.wav, etc. only.

 

Quote

How can I recursively compress/uncompress a directory (including guessed/forced-compression)

Uset the btrfs filesystem defrag command, the option -r will process the files recursively in a directory. This is independent of the mount options compress or compress-force, and using the option -c you can set the compression algorithm.

Currently (v4.14), it's not possible to select "no compression", using the defrag command. This may change in the future.

 

Or a plugin to run this per share?

 

Note: I'm new to unRAID, I just had my first installation and I'm midway through my very first parity-sync run... I'm on the default xfs and strongly considering to reformat to btrfs to experiment with compression.