Jump to content
itimpi

unRAIDFindDuplicates.sh

86 posts in this topic Last Reply

Recommended Posts

I get the following error using -v

./unRAIDFindDuplicates.sh: line 373: verbose_to_bpth: command not found

 

 

Unraid 6.4.1

Share this post


Link to post

That looks like a spelling issue.   It should be ‘both’ rather than ‘bpth’.   It is basically harmless as all that line is trying to do is output a blank line for cosmetic purposes.

Share this post


Link to post

Did the test duplicate file that you created exist in the same place on different disks.

 

So if you have a duplicate file at /mnt/disk1/TV/series/file and /mnt/disk2/TV/series/file (just the name has to match, not the contents) then this script will report it as a duplicate.   This often happens if you re-organise the data in a share by copying rather than moving the files between disks.

 

As the contents of the 2 disks in the above example would be merged together under /mnt/user/TV to create the TV share only one version of the file will be visible at /mnt/user/TV/series/file, which is why it is useful to identify which files are duplicated and remove the duplicate.

 

i suspect that you were expecting it to report files which have the same name and/or content in random locations .... unfortunately this script does not address this problem.

Share this post


Link to post

I used the exact same file(copied the script into a couple of places with no changes) I just need the ability to find duplicate files in the array somehow someway

Share this post


Link to post

Thanks for this script. after a weeklong effort to convert several drives to xfs i found myself with multiple copies of many of my files. this tool made taking care of this issue simple and painless

 

Thanks!

Share this post


Link to post
6 hours ago, EvilSpice said:

Thanks for this script. after a weeklong effort to convert several drives to xfs i found myself with multiple copies of many of my files. this tool made taking care of this issue simple and painless

 

Thanks!

That was exactly the reason I wrote it in the first place as I ended up in the same situation :)

Share this post


Link to post
Posted (edited)

Thanks for this.

 

I ended up creating a real problem for myself with the unbalance plugin (cancelling it in progress causes it to leave copies of files in place) and I needed a real utility to deal with the mess.

 

Unfortunately for me, while the output of this app is useful, I had far, far too many duplicates to deal with by hand.  So I wrote this little Windows (sorry) utility to automate the process for me:

 

Loading file... 
Buffy S01E10.mkv size mismatch!
Chef S01E04.mkv size mismatch!
Read 1515 / 2589 lines.
Finding size mismatches...
/mnt/disk11/media/series/Buffy the Vampire Slayer/Buffy S01E10.mkv : 99,975,168
/mnt/disk12/media/series/Buffy the Vampire Slayer/Buffy S01E10.mkv : 1,981,632,213
/mnt/disk6/media/series/Buffy the Vampire Slayer/Buffy S01E10.mkv : 1,981,632,213
/mnt/disk12/media/series/Chef/Chef S01E04.mkv : 487,030,784
/mnt/disk6/media/series/Chef/Chef S01E04.mkv : 1,045,681,530
Generating list of files that are safe to delete...
Script will delete 955 out of 1,515 files. Total space saved: 1,317,440,808,558 bytes (1.2 TB)
File C:\zzz\delete.txt successfully created.

Essentially, it parses the script's log file, highlights the files that are safe to leave in place and those that are risky and creates a script file that you can execute from the UnRAID console.  The output of the script looks like this:

echo "Keeping /mnt/disk11/media/series/Angel/Season 1/Angel S01E08.mkv"
rm "/mnt/disk12/media/series/Angel/Season 1/Angel S01E08.mkv"
rm "/mnt/disk6/media/series/Angel/Season 1/Angel S01E08.mkv"
echo "Keeping /mnt/disk11/media/series/Angel/Season 1/Angel S01E09.mkv"
rm "/mnt/disk12/media/series/Angel/Season 1/Angel S01E09.mkv"
rm "/mnt/disk6/media/series/Angel/Season 1/Angel S01E09.mkv"
echo "Keeping /mnt/disk11/media/series/Angel/Season 1/Angel S01E10.mkv"
rm "/mnt/disk12/media/series/Angel/Season 1/Angel S01E10.mkv"
rm "/mnt/disk6/media/series/Angel/Season 1/Angel S01E10.mkv"
echo "Keeping /mnt/disk11/media/series/Angel/Season 1/Angel S01E11.mkv"

In the hope that I can save others some time, I have included the app (binary and source) in this post.

 

Be careful, folks! I sincerely hope that no one loses their data with this thing.

UnRAIDdeDupe.zip

Edited by Excessus

Share this post


Link to post
Posted (edited)
On 3/11/2019 at 2:57 PM, Excessus said:

Thanks for this.

 

I ended up creating a real problem for myself with the unbalance plugin (cancelling it in progress causes it to leave copies of files in place) and I needed a real utility to deal with the mess.

 

Unfortunately for me, while the output of this app is useful, I had far, far too many duplicates to deal with by hand.  So I wrote this little Windows (sorry) utility to automate the process for me:

 


Loading file... 
Buffy S01E10.mkv size mismatch!
Chef S01E04.mkv size mismatch!
Read 1515 / 2589 lines.
Finding size mismatches...
/mnt/disk11/media/series/Buffy the Vampire Slayer/Buffy S01E10.mkv : 99,975,168
/mnt/disk12/media/series/Buffy the Vampire Slayer/Buffy S01E10.mkv : 1,981,632,213
/mnt/disk6/media/series/Buffy the Vampire Slayer/Buffy S01E10.mkv : 1,981,632,213
/mnt/disk12/media/series/Chef/Chef S01E04.mkv : 487,030,784
/mnt/disk6/media/series/Chef/Chef S01E04.mkv : 1,045,681,530
Generating list of files that are safe to delete...
Script will delete 955 out of 1,515 files. Total space saved: 1,317,440,808,558 bytes (1.2 TB)
File C:\zzz\delete.txt successfully created.

Essentially, it parses the script's log file, highlights the files that are safe to leave in place and those that are risky and creates a script file that you can execute from the UnRAID console.  The output of the script looks like this:


echo "Keeping /mnt/disk11/media/series/Angel/Season 1/Angel S01E08.mkv"
rm "/mnt/disk12/media/series/Angel/Season 1/Angel S01E08.mkv"
rm "/mnt/disk6/media/series/Angel/Season 1/Angel S01E08.mkv"
echo "Keeping /mnt/disk11/media/series/Angel/Season 1/Angel S01E09.mkv"
rm "/mnt/disk12/media/series/Angel/Season 1/Angel S01E09.mkv"
rm "/mnt/disk6/media/series/Angel/Season 1/Angel S01E09.mkv"
echo "Keeping /mnt/disk11/media/series/Angel/Season 1/Angel S01E10.mkv"
rm "/mnt/disk12/media/series/Angel/Season 1/Angel S01E10.mkv"
rm "/mnt/disk6/media/series/Angel/Season 1/Angel S01E10.mkv"
echo "Keeping /mnt/disk11/media/series/Angel/Season 1/Angel S01E11.mkv"

In the hope that I can save others some time, I have included the app (binary and source) in this post.

 

Be careful, folks! I sincerely hope that no one loses their data with this thing.

UnRAIDdeDupe.zip 65.16 kB · 7 downloads

Can you provide a simple guide on how to use this feature? I have the same issue that's to unbalance and have 500+ GB of duplicated steam games. I have downloaded your zip file, but am unsure what to do at this time.

 

EDIT:

 

I was able to successfully import the code into VS, and get it to run.

 

Thanks for this, just waiting on my first log file to finish. May have questions then!

Edited by mihcox

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.