Dedupe, Compression, VM Snapshots, Backups

[vagrantprodigy]

Is there a plan to support any of this in the future? 

 

Deduplication is a big one for me, I recently deduped my box by hand, and gained back about 20% of my storage.

 

Compression similarly could gain back some space, which would be critical for someone like me who has filled up every drive bay, and added too many sata power adapters onto his power supply.

 

VM Snapshots are something that I didn't realize unRAID didn't have until recently, and I thought for sure I must be missing something, but I was unable to find it. If it's there, and I'm missing it, please let me know.

 

Backups - It would be great if there was an integrated backup solution in unRAID, even if it was just for virtual machines. 

 

I'm actually in charge of a project at work right now to bring in new files servers, and while I'd love to consider unRAID there, my team members will never got for something that does not provide features like deduplication and compression.

[JorgeB]

All of those features are supported if you select btrfs as the filesystem, but there's currently no GUI support, I believe at least VM snapshots are planned for the near future, not sure about the others, so you can use them now but need to use the console/scripts.

 

For example, there's a recent thread discussing compression here.

 

One thing to keep in mind, since with unRAID all data disks are a separate filesystem these only work on a disk by disk basis, i.e., you can't dedupe a share spanning multiple disks.

[vagrantprodigy]

50 minutes ago, johnnie.black said:

All of those features are supported if you select btrfs as the filesystem, but there's currently no GUI support, I believe at least VM snapshots are planned for the near future, not sure about the others, so you can use them now but need to use the console/scripts.

 

For example, there's a recent thread discussing compression here.

 

One thing to keep in mind, since with unRAID all data disks are a separate filesystem these only work on a disk by disk basis, i.e., you can't dedupe a share spanning multiple disks.

BTRFS is a non starter for me. Dedupe is really only good across the entire array. Unfortunately the way this is currently implemented just isn't useful for any situation I would ever use personally or professionally.

[JorgeB]

7 minutes ago, vagrantprodigy said:

Dedupe is really only good across the entire array.

Each disk being a contained filesystem is one of the biggest advantages of unRAID, but like everything it has is its downsides, like for example performance and implementing features like these, I don't see how for example dedupe can work across multiple filesystems, but I could be wrong.

[martinf]

Hi

I'm new to unraid but have used linux for a long time.

There is a tool called fdupes in linux.

Not sure how hard it would be to add to unraid standard cmd commands.

But for now, you could nfs mount a share to a unix/linux computer and run fdupes agains the nfs share.

When / if fdupes can be included in unraid, you could run it agains /mnt/user/sharename and find duplicated per share or across all shares

fdupes /mnt/user

 

My $0.02

/Martin

[trurl]

40 minutes ago, martinf said:

There is a tool called fdupes in linux.

Not sure how hard it would be to add to unraid standard cmd commands.

Included in Nerd Pack plugin:

 

https://forums.unraid.net/topic/35866-unraid-6-nerdpack-cli-tools-iftop-iotop-screen-kbd-etc/