It would be nice to have Lessfs, or other block deduplication, in unraid.  Handy for backups.

 

So, out of curiosity, are there any other unraid users out there interested in block de-duplication?

 

So, out of curiosity, are there any other unraid users out there interested in block de-duplication?

 

I find it very interesting. I'm not sure where it fits in with unRAID at the moment.

Has anyone tried this on a 2TB filesystem with movies to see how much space it saves?

 

I this were used at my job for archival backups. I'm sure it would help.

The data there contains all sorts of redundant strings.

 

In comparison, some sort of compression filesystem may be better for that.

 

In thought, the de-duplication may be helpful for rsync linked type backups.

I'm not convinced this would be helpful for movies, Music files or others which are already compressed a great deal.

De-duplication doesn't offer many beneficial effects (and in fact, may hinder performance) when dealing with media files. Fact of the matter is that there aren't a lot of similar 4K blocks that can be de-duplicated to produce any real benefit (and probably costing valuable CPU cycles).

 

If I were to lobby for a new FS for unRAID, I would suggest btrfs (pronounced ButterFS) which is an upcoming Linux FS that very closely resembles ZFS for Solaris/BSD.

 

De-duplication is good for backups.  Works great for drive image backups.  You don't have the complexity differential backups -- simply back-up the entire disk image to a storage server with de-duplication -- only the changes need to actually consume space.  If you want to removed older backups, simply delete them,  only the blocks that are no longer used will be deleted.

 

I don't think de-duplication has a place in movies, music, etc -- unless you're making backups of them.

 

Despite being 'stable', all this would do is scare me about data integrity.

De-duplication is good for backups.  Works great for drive image backups.  You don't have the complexity differential backups -- simply back-up the entire disk image to a storage server with de-duplication -- only the changes need to actually consume space.  If you want to removed older backups, simply delete them,  only the blocks that are no longer used will be deleted.

 

I don't think de-duplication has a place in movies, music, etc -- unless you're making backups of them.

 

 

I see where the drive image backups and de-duplication have merit. I wonder how many people do this to unRAID to make it worth the effort of getting it to work.

 

I know rsync has a method of using hard links to eliminate duplication too.

With the use of the --link-dest= parameter, you can point to a prior directory.

I believe it hard links the unchanged files from the prior directory to the new directory.

Then rsyncs the changed files into the new directory.

 

I have a script that does an rsync file backup to dated directories on a daily basis.

It does a rolling set of dated directories and while keeping duplicates to a minimum.

Despite being 'stable', all this would do is scare me about data integrity.

 

NetApp and others have been doing this for years, and making good money at it.  Lessfs is a bit green, but appears to maturing quickly.  ZFS also recently added de-duplication.

 

I see where the drive image backups and de-duplication have merit. I wonder how many people do this to unRAID to make it worth the effort of getting it to work.

 

If unraid supported de-duplication out of the box, it may move unraid out of the realms of a home media server to a cost effective business back-up server.  A low cost alternative to the likes of NetApp.  I would say virtual disk image storage as well, but performance of unraid is a bit too low for that (though striped parity sets may be able to help).

 

I currently image nightly to unraid, using incremental built into the drive imaging software.  This is a pain to manage, and having > 2x the image space available seems unavoidable when retiring old images.

 

I know rsync has a method of using hard links to eliminate duplication too.

With the use of the --link-dest= parameter, you can point to a prior directory.

I believe it hard links the unchanged files from the prior directory to the new directory.

Then rsyncs the changed files into the new directory.

 

Unfortunately, this is file level de-duplication.  Helpful for some circumstances, but not as flexible and transparent as block level.

 

I see where the drive image backups and de-duplication have merit. I wonder how many people do this to unRAID to make it worth the effort of getting it to work.

 

If unraid supported de-duplication out of the box, it may move unraid out of the realms of a home media server to a cost effective business back-up server.  A low cost alternative to the likes of NetApp.  I would say virtual disk image storage as well, but performance of unraid is a bit too low for that (though striped parity sets may be able to help).

 

Since this is a FUSE module, the chances of installing with unRAID are pretty good. (I thought it was a replacement for reiserfs).

 

With the larger drives, I can see this being useful for system/image/data backups.

 

I know rsync has a method of using hard links to eliminate duplication too.

With the use of the --link-dest= parameter, you can point to a prior directory.

I believe it hard links the unchanged files from the prior directory to the new directory.

Then rsyncs the changed files into the new directory.

 

Unfortunately, this is file level de-duplication.  Helpful for some circumstances, but not as flexible and transparent as block level.

 

 

I dunno if this is unfortunate. it works very well for me. When I need to restore a file I can locate it see all the dates for it and merely review, copy or rsync it back.

 

I do see the usefulness of lessfs.    Especially where I have to archive large uncompressed files of mortgage backed securities data.  There is allot of redundancy throughout the span of the data's lifetime.

 

I may experiment with this to see how well it does.

Since this is a FUSE module, the chances of installing with unRAID are pretty good. (I thought it was a replacement for reiserfs).

 

Seemed like it could be a good fit for unraid.

 

I was trying to figure out if lessfs, or Tokyo Cabinet (the DB lessfs uses) could easily span multiple drives, making use of available space on them without having to pre-allocate it.   Use on even one drive would be nice though.

 

I dunno if this is unfortunate.

Bad choice of words on my part.

 

I may experiment with this to see how well it does.

 

I look forward to any results you may have.

 

From what I understand, de-duplication of 4k blocks in lessfs seems to be a performance killer on "low spec" hardware -- but I'm not sure what is "low spec" in regards to lessfs.  Increasing the block size appears to help greatly -- but would also likely make de-duplication less effective for some types of data.

 

From what I understand, de-duplication of 4k blocks in lessfs seems to be a performance killer on "low spec" hardware -- but I'm not sure what is "low spec" in regards to lessfs.  Increasing the block size appears to help greatly -- but would also likely make de-duplication less effective for some types of data.

 

In my case, we have tape backup to this massive robotic array of tapes.

It takes about a day or so for an admin to handle the backup request.

A file server with autocompression or deduplication may prove to make the request at my location faster.

I could have daily backups near line and have customer service restore files to customer directories without our involvement. Thanks for the heads up.

Though I doubt you were planning on approaching it this way, here's a note I stumbled on about tape backups and lessfs:

http://www.lessfs.com/wordpress/?p=259&cpage=1#comment-219