I'm trying to find duplicate files between my cache and array (basically where mover wrote the file to the array but didn't successfully remove it from the cache so now its on both cache and array), by default it doesn't seem to do this.

it looks like -D should do what I want, but it seems to be broken

root@UnraidHost:/mnt/user/Media# ./unRAIDFindDuplicates.sh -v -D /mnt/cache_unprotected

============= STARTING unRAIDFIndDuplicates.sh ===================

Included disks:
   /mnt/disk/mnt/cache_unprotected
   /mnt/disk1
   /mnt/disk2
   /mnt/disk3
   /mnt/disk4
   /mnt/disk5
   /mnt/disk6
   /mnt/disk7
   /mnt/disk8
   /mnt/disk9
   /mnt/disk10
   /mnt/disk11
   /mnt/disk12
   /mnt/disk13
   /mnt/disks

Included shares:
   Backup
   CommunityApplicationsAppdataBackup
   LibVirtBackup
   Media
   ...

List duplicate files
./unRAIDFindDuplicates.sh: line 373: verbose_to_bpth: command not found
./unRAIDFindDuplicates.sh: line 404: cd: /mnt/disk/mnt/cache_unprotected: No such file or directory
checking /mnt/disk/mnt/cache_unprotected
    Backup
^C
root@UnraidHost:/mnt/user/Media# 

it seems to concat /mnt/disk onto the path specified by -D which seems incorrect?

 

if I let it run it seems to list every file in the array as duplicate after it gives an error on line 404.

It might do what you want if you change line 215 that reads:

    D ) include_disks[$include_disks_count]="/mnt/disk${OPTARG}"

to be:

    D ) include_disks[$include_disks_count]="${OPTARG}"

 

In addition line 373 should be

  verbose_to_both ""

although that is just a cosmetic change

 

Could you let know if that does the job as if it does I can change the linked script to include these changes.  Unfortunately not in a good position to check the changes myself at the moment.

 

EDIT:  Updated the script attached to the first post to include the above changes.

 

On 6/3/2024 at 2:38 PM, itimpi said:

 

EDIT:  Updated the script attached to the first post to include the above changes.

 

The 1.6 version has a couple of issues. Missing closing quotation marks on lines 95 an 101 prevent it from running, and "preogram_dir" is misspelled on line 49 which makes that check fail silently. The version number returned from -V was incorrect, also. 

 

Here's an updated file with those issues fixed.

 

unRAIDFindDuplicates v1.7.zip

Edited June 5, 20242 yr by WalkerJ

Thanks for the fixed version - I will update the first post in the thread with this version.    I hope with those fixes you got useful results?   

 

It is nearly 10 years since I originally developed this script early in my Unraid days (using an early v5 beta release) when I had unreliable disks and ended up with lots of duplicate files while sorting this out.  Despite all the changes that have happened in this time at the Unraid level it is interesting to see that the basic concepts have remained constant so that this script still works.

 

BTW:  Another interesting fact - assuming you username is based on your surname then we have the same surname 🙂

Nice script. You save my day :D. It would be even better if this script can group all files to the corresponding folders because my files was all over disks and It look so messy.

Edited October 28, 20241 yr by Taipt5

Hello @itimpi,
can you reupload your script? 

When i hit the download button it gives me an error.
And yes i'm logged in.

 

thanks

The script is able to tell me the duplicate files at the share level, but can the script tell me which drives?

I need to know the drives I can delete the duplicated files from.

 

Thanks.

@itimpi

The script in Unraid 7 seems to completely fail to work. First, it skips ZFS arrays, so you have to add them via the -D parameter.

 

 

 

But still, it doesn't find duplicates at all, even when I deliberately created them:

 

 

 

 

Edited April 25, 20251 yr by jaclas

The script ignores pools - it predates pools being generally used for anything other than caching so only checks the main Unraid array.

 

I guess at some time it might be worth considering updating it to handle pools, and also make it a plugin and provide some sort of GUI front end.   I will put it on my 'to do' list, but no promises.

@itimpi I'll just add some more questions just to be sure.

So even though the pool is explicitly pointed out it is ignored? And even though there is information in the log about checking this pool it is really bypassed?

1 hour ago, jaclas said:

@itimpi I'll just add some more questions just to be sure.

So even though the pool is explicitly pointed out it is ignored? And even though there is information in the log about checking this pool it is really bypassed?

Your question made me look at the code - you are right there is some explicit code at around lines 311-312 which looks for /mnt/cache and if present adds that to the list of disks (as at the time the script was developed that was the only pool available and its name was fixed).

 

That looks like it is the area at which one could add any additional pools if present to get them included in the check to make the script able to handle other pool names.

Your script was super helpful for me.

 

I ended up writing a script to follow up and act on (delete) duplicates.

#!/bin/bash

dupeList=(
)

remove_dupes() {
  dupe="$1"

  dupePaths=$(find /mnt/disk*/"${dupe}")
  dupeCount=$(echo "${dupePaths}" | wc -l)

  if [[ "${dupeCount}" -gt 1 ]]; then
    removeFiles=$(echo "${dupePaths}" | sort -R | tail -n +2)
    keepFile=$(grep -v -F -x -f <(echo "${removeFiles}") <(echo "${dupePaths}"))
    echo "------------------------------------"
    echo "Found ${dupeCount} duplicates for:"
    echo "${dupe}"
    echo ""
    echo "Keeping one file:"
    echo "${keepFile}"
    echo ""
    echo "Removing duplicate file(s):"
    echo "${removeFiles}"
    # rm "${removeFiles}"
    echo ""
    echo "Remaining files (confirmation):"
    find /mnt/disk*/"${dupe}"
    echo "------------------------------------"
  fi
}

for dupe in "${dupeList[@]}"; do
  remove_dupes "${dupe}"
done

The rm line in the above is commented out. If anyone decides to use this script, be sure you know what you're doing (or run it first with the rm commented).

Also the dupeList variable is empty. It doesn't detect the /boot/duplicates.txt, I am just taking the output from there and copying it into the script (with quotes) so it looks like this:

dupeList=(
  "file1.txt"
  "file2.txt"
  "file3.txt"
  "file4.txt"
  "file5.txt"
  "file6.txt"
  "file7.txt"
  "file8.txt"
  "file9.txt"
  "file10.txt"
  # Add more files as needed
)

It could probably be easily modified to actually read from /boot/duplicates.txt but having to manually copy keeps the use of this script very intentional.

 

One last note, it decides which file to keep randomly and deletes the rest. The point isn't to gather/scatter line unbalanced, or keep files together on specific disks, just to get rid of duplicates to recover space.

@iXNyNe

Great script, I just added this just before the RM statement:

echo "Deleting now, you have 2 seconds to break here to stop. Use Ctrl-C to break"

sleep 2

Slows this down, but it at least gives you time to kill it before it deletes in case you see something you don't like.