unRAIDFindDuplicates.sh

[Gunny]

I'm trying to find duplicate files between my cache and array (basically where mover wrote the file to the array but didn't successfully remove it from the cache so now its on both cache and array), by default it doesn't seem to do this.

it looks like -D should do what I want, but it seems to be broken

root@UnraidHost:/mnt/user/Media# ./unRAIDFindDuplicates.sh -v -D /mnt/cache_unprotected

============= STARTING unRAIDFIndDuplicates.sh ===================

Included disks:
   /mnt/disk/mnt/cache_unprotected
   /mnt/disk1
   /mnt/disk2
   /mnt/disk3
   /mnt/disk4
   /mnt/disk5
   /mnt/disk6
   /mnt/disk7
   /mnt/disk8
   /mnt/disk9
   /mnt/disk10
   /mnt/disk11
   /mnt/disk12
   /mnt/disk13
   /mnt/disks

Included shares:
   Backup
   CommunityApplicationsAppdataBackup
   LibVirtBackup
   Media
   ...

List duplicate files
./unRAIDFindDuplicates.sh: line 373: verbose_to_bpth: command not found
./unRAIDFindDuplicates.sh: line 404: cd: /mnt/disk/mnt/cache_unprotected: No such file or directory
checking /mnt/disk/mnt/cache_unprotected
    Backup
^C
root@UnraidHost:/mnt/user/Media# 

it seems to concat /mnt/disk onto the path specified by -D which seems incorrect?

 

if I let it run it seems to list every file in the array as duplicate after it gives an error on line 404.

[itimpi]

It might do what you want if you change line 215 that reads:

    D ) include_disks[$include_disks_count]="/mnt/disk${OPTARG}"

to be:

    D ) include_disks[$include_disks_count]="${OPTARG}"

 

In addition line 373 should be

  verbose_to_both ""

although that is just a cosmetic change

 

Could you let know if that does the job as if it does I can change the linked script to include these changes.  Unfortunately not in a good position to check the changes myself at the moment.

 

EDIT:  Updated the script attached to the first post to include the above changes.

 

[WalkerJ]

On 6/3/2024 at 2:38 PM, itimpi said:

 

EDIT:  Updated the script attached to the first post to include the above changes.

 

The 1.6 version has a couple of issues. Missing closing quotation marks on lines 95 an 101 prevent it from running, and "preogram_dir" is misspelled on line 49 which makes that check fail silently. The version number returned from -V was incorrect, also. 

 

Here's an updated file with those issues fixed.

 

unRAIDFindDuplicates v1.7.zip

Edited June 5 by WalkerJ

[itimpi]

Thanks for the fixed version - I will update the first post in the thread with this version.    I hope with those fixes you got useful results?   

 

It is nearly 10 years since I originally developed this script early in my Unraid days (using an early v5 beta release) when I had unreliable disks and ended up with lots of duplicate files while sorting this out.  Despite all the changes that have happened in this time at the Unraid level it is interesting to see that the basic concepts have remained constant so that this script still works.

 

BTW:  Another interesting fact - assuming you username is based on your surname then we have the same surname 🙂