July 13, 201213 yr Can someone sanity check this request as I dont have a 5 series to check at hand. When unRAID detects a duplicate file it create a log entry. This is very useful. However the log entry only contains one side of the duplicate and not both. Request that the log entry shows both duplicates for easy correction
July 13, 201213 yr Great idea... but you can always delete the one it gives you and hope the other one (invisible) exists and is same quality .... that's why i also would like to see both positions... so you can compare/ check if the other one is even good....
July 13, 201213 yr Great idea... but you can always delete the one it gives you and hope the other one (invisible) exists and is same quality .... that's why i also would like to see both positions... so you can compare/ check if the other one is even good.... I think you have it backwards, from my recollection, it will list in the syslog the location of the duplicate that is not visible. (the file on the lowest numbered disk is the one visible in the user-share)
July 13, 201213 yr Author Good to know. In a perfect world we would md5sum compare them and just delete 100% match dupes. What i know is thats its a complete pain in the ass to do manually so there has to be a slicker way
July 13, 201213 yr Joe. I am not the one to doubt you as you have much more experience then i have but i think it is the latest added that shows up in your logs i had an small hickup due to my psu and while i was rebuilding my disk with reiserfs rebuild tree my program sickbeard had updated all missing folder.jpg / banner.jpg/ tvshow.jpg on other disks when my disk was rebuild an brought back online the syslog had literraly dozens of duplicates on the disk i brought back.... now it was disk 4... so you might be right ... anyway i deleted all these jpg files as i knew sickbeard would put the missing back
July 13, 201213 yr Good to know. In a perfect world we would md5sum compare them and just delete 100% match dupes. What i know is thats its a complete pain in the ass to do manually so there has to be a slicker way No. Just no. I do not want my array automatically deleting files for me. It would be nice if it did a binary compare and informed me of whether or not they match, but leave the management decision to me.
July 13, 201213 yr Author Each to there own but as far as I am concerned a file with the same md5sum, filename and path is just noise. I estimate I have over 2000 duplicate files this now for some reason
July 13, 201213 yr Each to there own but as far as I am concerned a file with the same md5sum, filename and path is just noise. I estimate I have over 2000 duplicate files this now for some reason Noise to you, backup duplication for me. It provides a way to keep a copy of a file hidden from a user share so it won't get altered or deleted, and it means I'd have to lose 3 physical disks to lose that data. Like you said, to each their own. It's a useful function to me.
July 13, 201213 yr Good to know. In a perfect world we would md5sum compare them and just delete 100% match dupes. What i know is thats its a complete pain in the ass to do manually so there has to be a slicker way No. Just no. I do not want my array automatically deleting files for me. It would be nice if it did a binary compare and informed me of whether or not they match, but leave the management decision to me. It should be a separate plugin... but ... which file should get deleted? Perhaps we should lobby that shfs is altered so that the source file AND it's duplicate are listed. I would rather see which files were duplicated and let me delete what's needed.
July 13, 201213 yr Author Agreed, that was my OP on the topic. If we forget about all automatic actions for now it should at the very least be easy to find all dupes. Currently it is not.
July 13, 201213 yr What about: "ls /mnt/disk*/path/filename"? or the only slightly less intuitive: tail -30000 /var/log/syslog | grep "duplicate object" | cut -d":" -f5- | sed "s/^ //" | sort -u | sed -e "s/^\\/[^\\/]*\\/[^\\/]*\\/\\(.*\\)/ls -lad \\/mnt\\/*\\/\"\\1\"/" | sh - | grep -v "/mnt/user/" Enter it all as one line...
July 13, 201213 yr How about a method of including the duplicate files in a directory listing, e.g., by adding either a prefix or a suffix of some sort to the file name? For example, suppose we have: disk3/backup/afile disk5/backup/afile User share directory listing of 'backup' could show this: afile afile~disk5~ In addition, I can permit all file operations on 'afile~disk5~' such as rename, copy, delete, etc. Sound reasonable? Should there be a different nomenclature for the suffix?
July 13, 201213 yr How about a method of including the duplicate files in a directory listing, e.g., by adding either a prefix or a suffix of some sort to the file name?Can it be optional? I'd like the ability to turn it on or off, or possibly allow a custom suffix?
July 13, 201213 yr In addition, I can permit all file operations on 'afile~disk5~' such as rename, copy, delete, etc. Sound reasonable? Should there be a different nomenclature for the suffix? This would be Great!! Two thumbs up from me on this idea.
July 14, 201213 yr Author So there are two issues here: 1. Making it easy for users to locate and deal with duplicates. This suffix makes this very easy and at same time doesn't cause issues for things like XBMC. Nice 2. Log load. For cache_dirs users these duplicates create a load on logging. root@TOWER:/mnt/user/TV/D# grep -i duplicate /var/log/syslog* | wc -l 14726 Thats from one day. I am not sure we have a solution to that yet.
January 6, 201313 yr What about: "ls /mnt/disk*/path/filename"? or the only slightly less intuitive: tail -30000 /var/log/syslog | grep "duplicate object" | cut -d":" -f5- | sed "s/^ //" | sort -u | sed -e "s/^\\/[^\\/]*\\/[^\\/]*\\/\\(.*\\)/ls -lad \\/mnt\\/*\\/\"\\1\"/" | sh - | grep -v "/mnt/user/" Enter it all as one line... small edit to stop user0 share being output: tail -30000 /var/log/syslog | grep "duplicate object" | cut -d":" -f5- | sed "s/^ //" | sort -u | sed -e "s/^\\/[^\\/]*\\/[^\\/]*\\/\\(.*\\)/ls -lad \\/mnt\\/*\\/\"\\1\"/" | sh - | grep -v "/mnt/user" I wonder how difficult it would be to find the duplicates which sit on a disk alone, or at least with the least other data. My application is media files where I expect all files related to a movie or TV season to be on the same disk. I thinking a small mod to the mover script would be able to achieve this - albeit beyond my feeble abilities
January 8, 201313 yr User share directory listing of 'backup' could show this: afile afile~disk5~ In addition, I can permit all file operations on 'afile~disk5~' such as rename, copy, delete, etc. Sound reasonable? Should there be a different nomenclature for the suffix? I would love an easy way to find out on what disk a file in a user share has been placed regardless of whether it is a duplicate. At the moment I might have to search all my disks (20 in total) if I want to find the physical location of a file.
Archived
This topic is now archived and is closed to further replies.