Identify duplicate files stored on unRAID managed devices


Helmonder

Recommended Posts

That could easily be a year ago... I was copying files from here to there to make drives empty to reformat them to xfs... During that I noticed that I got some copy messages on allready existing files.. So I looked somewhat deeper and I know am sure that I have duplicate files, possibly from a copy going  wrong a long time ago..

 

Because I saw that I did  cat /var/log/syslog | grep duplicate    (jonp: thats commandline dude ;-) but got nothing..

 

I would need some way to find my dupes..

Link to comment

Does unraid 6 still show a message in the syslog when a duplicate file is on the system ?

 

In fact I think it does not... I have several duplicate files and there are no messages..

 

When did this work for you last?

I always assumed this was due to having duplicate "paths" with user shares. For example, /mnt/disk1/music/somesong.mp3 and /mnt/disk2/music/somesong.mp3 would be 2 files that both were at /mnt/user/music/somesong.mp3, even if the actual file contents were different. This is about the only thing that should matter as far as unRAID is concerned. Having files that were actual copies of each other, but having distinct user share paths, should be fine.
Link to comment

Does unraid 6 still show a message in the syslog when a duplicate file is on the system ?

 

In fact I think it does not... I have several duplicate files and there are no messages..

 

When did this work for you last?

I always assumed this was due to having duplicate "paths" with user shares. For example, /mnt/disk1/music/somesong.mp3 and /mnt/disk2/music/somesong.mp3 would be 2 files that both were at /mnt/user/music/somesong.mp3, even if the actual file contents were different. This is about the only thing that should matter as far as unRAID is concerned. Having files that were actual copies of each other, but having distinct user share paths, should be fine.

 

What you are describing is what I mean.. If I am correct then that is no longer working..

Link to comment

I do need some way to make this an easy process... With the way unraid works it is very good possible to have duplicates... There does not necessarily need to be logentry (I basically have allways found that not the most elegant solution), an option in the GUI would be better..

 

But first and foremost.. I think this is a bug right ?  It should be in there.. At least it always was..

Link to comment

Sure it is a bug, actually already existing since the official release of unRAID v5 when the file attributes were changed.

 

The code in the browser function which looks for duplicate file names needs to be changed (note that file content may be different)

 

<?if (strstr($entry['disk'],"duplicate")):?>

  <td style="background-color:orange"><?=$entry['disk'];?></td>

<?else:?>

  <td><?=$entry['disk'];?></td>

<?endif;?>

 

it can be revised to the following:

 

<?if (substr($entry['disk'],-1)=="*"):?>

  <td style="background-color:orange"><?=$entry['disk'];?></td>

<?else:?>

  <td><?=$entry['disk'];?></td>

<?endif;?>

 

For Dynamix I have rewritten the browser function because it was extremely slow when opening a folder with several thousand of entries. Could take half a minute to load, while the revised version does it in sub-second (excluding the time needed to spin-up the drive, when combined with cache_dirs it really gives instant replies).

Link to comment

Does unraid 6 still show a message in the syslog when a duplicate file is on the system ?

 

In fact I think it does not... I have several duplicate files and there are no messages..

 

When did this work for you last?

 

Jonp: Can we get this fixed in the next release possibly ?  I really need some way to find my duplicates.. Especially with everyone copying stuff from here to there while changing file system this is something more people will need ..

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.