[WIP] De-Duplicator script for UnRAID (eventually a plugin)

[bobbintb]

Right now, this is a Python script for UnRAID to search for duplicates. It's very fast.

 

Files found: 53,647
3.80 TB (4,181,981,773,439 bytes)
After omitting 1,881 hard linked files: 51,766
After omitting 37,258 unique file sizes: 14,508
     Hashing file 14,508 of 14,508...done.
After omitting 968 unique hashes: 13,540
     Hashing file 13,540 of 13,540...done.
After omitting 185 unique hashes: 13,355
Time elapsed: 13 minutes and 10.19 seconds

 

Files found: 26,602
2.05 TB (2,257,131,773,393 bytes)
After omitting 0 hard linked files: 26,602
After omitting 22,461 unique file sizes: 4,141
     Hashing file 4,141 of 4,141...done.
After omitting 3,539 unique hashes: 602
     Hashing file 602 of 602...done.
After omitting 96 unique hashes: 506
Time elapsed: 3 minutes and 22.26 seconds

I plan to eventually complete the other parts to create a full plugin with a daemon and UI. Right not it's just a one-time scan and you view the results. I appreciate any feedback or polite criticisms.

https://github.com/bobbintb/UnRAID-DeDuper