Duplicate files on multiple disks in a user share question

[kopuz]

I have the following setup with 2 disks

 

Tower > Movies > A

 

What happens if I was to copy the same file to

 

Disk1 > Movies > A and Disk2 > Movies > A

 

Both disks are in the user share Movies.  The same exact file is copied directly too 2 different disks to the same user share with the same path.

 

Would the file take up double the space in the array?

 

Hope I explained my scenario clearly enough.  Thanks in advance for the help!

 

 

 

[itimpi]

Yes, the file uses twice the space, even though you will only see it once at the User Share level.  I think unRAID shows the one on the lowest number disk but I am not sure about that. 

 

You can get strange behaviour in that if you try to delete (or rename) the file from the share it still appears to remain as unRAID will now present the second copy.

[kopuz]

Anyone recommend a good duplicate file scanner?  Docker maybe?

 

[kopuz]

Dug this up.

 

Working very well so far.

 

http://lime-technology.com/forum/index.php?topic=35183.0

[itimpi]

I wrote that script originally to address exactly this issue when I was moving data around and managed to make mistakes and ended up with duplicates.

 

I think I have seen that the "Fix Common Problems" plugin will also now show if you have duplicates.

[Sn3akyP3t3]

Two applications I often turn to in duplicate file removal situations are:

• Duplicate Cleaner (Digital Volcano Software) - https://www.digitalvolcano.co.uk/duplicatecleaner.html  (Free or Paid versions available) Its pretty amazing software!  Its flexible in finding duplicates in ways that work for you covering different possible scenarios.  It groups found duplicates together for verification.  Cleanup processing has plenty of options available to the user to decide which file survives with criteria like age, size, or location.  The pro version unlocks features such as finding dupes in photos which can be tedious to do manually.
• VisiPics - http://www.visipics.info/index.php?title=Main_Page (Free) Another one I find useful from time to time, but mostly now on other users computers and often after migration from old computers to new computers.  It pretty much restricts duplicate file cleanup to pictures only, but its really good at it!  I usually do this cleanup in steps starting with strict mode then knocking it down a notch to find like matches that may have been exports of the original or reduced for sharing.

[Philip Gore]

Try Duplicate Files Deleter, a free and good software.

[steve1977]

On 8/24/2016 at 3:25 PM, itimpi said:

I wrote that script originally to address exactly this issue when I was moving data around and managed to make mistakes and ended up with duplicates.

 

I think I have seen that the "Fix Common Problems" plugin will also now show if you have duplicates.

You sure about the "fix common problems" plugin? I suspect I have duplicates (same file on different disks, shows up once in user share). What's the best way to look into this?

[superloopy1]

Dynamix file integrity plugin can give you a list is all duplicate files, I use it regularly, never had a problem. Quick and simple to use.

[steve1977]

2 hours ago, superloopy1 said:

Dynamix file integrity plugin can give you a list is all duplicate files, I use it regularly, never had a problem. Quick and simple to use.

Thanks. I installed it, but not fully clear to me what it does and how I get to the list of duplicate files. Is this looking more broadly for "real" duplicates? I am looking for files with same folder/file name across disks (that then only show up once when in same user share). Thanks!

[Squid]

8 hours ago, steve1977 said:

You sure about the "fix common problems" plugin? I

Yup.  Extended tests

[superloopy1]

Sorry, missed this one a while back, oops!

 

Yes,  dynamix file integrity has a 'Find' option which, once you've built/exported the array disks, will identify duplicates across all disks producing a logfile.