January 20, 20179 yr This isn't necessarily UnRAID specific but I thought it fit better here than in the lounge. I'm trying to get opinions on a good duplicate checker/search program or even some sort of plugin for UnRAID. I'm not talking about issues specific to UnRAID like a duplicate file in /mnt/disk1/myfile and /mnt/disk2/myfile but just in general. I know the file integrity plugin checks for duplicates but it isn't very robust. Some of the things I am looking for in a solution: 1. Check by blake2 checksum - Practically all duplicate checkers use checksums to check for dupes such as MD5 and SHA1 but blake2 is much faster and that makes a difference when it's 25-30TB. 2. Run as a service in the background - New files and changed files automatically get hashed and stored. I don't want to have to manually run a search every so often and wait for it to hash my entire server again. 3. Supports hard links - Not only can it replace duplicates with hard links but also has an option to not report hard linked files as duplicates. Currently, I use Dupscout Server from one of my Windows VMs. It's an older version so maybe it isn't an issue now (support hasn't responded) but it doesn't fully meet my third criteria, at least not that I can see. It will replace duplicate files with hard links but when running the search again it sees the hard links as duplicates again. It can be free or paid Linux or Windows. I'm looking for suggestions.
January 20, 20179 yr This really works well. https://lime-technology.com/forum/index.php?topic=35183.0
January 20, 20179 yr This really works well. https://lime-technology.com/forum/index.php?topic=35183.0 even though I am the author of that script I would say it does not meet the OP requirements. He is specifically not interested in files on different disks with the same name which is what the script is intended to handle.
January 20, 20179 yr Author This really works well. https://lime-technology.com/forum/index.php?topic=35183.0 even though I am the author of that script I would say it does not meet the OP requirements. He is specifically not interested in files on different disks with the same name which is what the script is intended to handle. Yes, I did specifically mention that. That script does work very well for that purpose though as I've used it before. I did just find out however that the problem I was having with Dupscout counting hard links as dupes has to do with the fact that while I can create hard links over SMB 3.0, it will not report them as hard links. Some earlier versions work so I am looking at that or switching to NFS. I am still open to suggestions though.
Archived
This topic is now archived and is closed to further replies.