Revan335 Posted January 4 Share Posted January 4 How can i excluded files, types or Folder? Quote Link to comment
psikic Posted January 8 Share Posted January 8 Enhancement request: It would be great if the Find duplicate files by hash option could output file size and possibly sort with largest file first. I'd be interested in deleting duplicates that were large, but I don't care about smaller duplicate files. Thanks! Quote Link to comment
Vetteman Posted January 18 Share Posted January 18 I was very excited when I found this plugin as I am an avid photographer and have taken many landscape and car show photos over the last 20 years. I also do digital illustrations and 3D modeling. Previously I was backing up all my edited and raw file photos and digital illustrations and 3D assets to external WD and Seagate USB NTFS hard drives. I started to schuck the hard drives leaving them as NTFS and placing them as unassigned devices in my Unraid server with all the files still intact, And continue to backup to them. This has worked quite well. File corruption and bitrot is a concern of any photographer or illustrator. It is mentioned XFS is the preferred file system? Will this work with NTFS or BTRFS or ONLY XFS? Also is there a video any where on setting this up or even a demo of this plugin? Thanks kindly... Quote Link to comment
itimpi Posted January 18 Share Posted January 18 If you btrfs that has built-in check summing for detecting bitrot. 1 Quote Link to comment
Vetteman Posted January 20 Share Posted January 20 (edited) For my array disks I use XFS. For cache and unassigned I use BTRFS and NTFS (NTFS - drives removed from Windows external drives which already have a file system and data). I thought somewhere here it was not recommended to use BTRFS on array disks, only XFS? Perhaps there is a PAR2 plugin for NTFS drives on Unraid? But searching I could not find any... I was hoping SpaceInvader One or someone like the "Bit my Bytes" bloke would have a video on Dynamix File Integrity. Cheers & many thanks.... Edited January 20 by Vetteman Quote Link to comment
primeval_god Posted January 20 Share Posted January 20 1 hour ago, Vetteman said: I thought somewhere here it was not recommended to use BTRFS on array disks, only XFS? Depends on who you ask, but it has been quite a while since that was the general consensus. Quote Link to comment
foo_fighter Posted January 20 Share Posted January 20 On 1/18/2024 at 5:33 AM, Vetteman said: I was very excited when I found this plugin as I am an avid photographer and have taken many landscape and car show photos over the last 20 years. I also do digital illustrations and 3D modeling. Previously I was backing up all my edited and raw file photos and digital illustrations and 3D assets to external WD and Seagate USB NTFS hard drives. I started to schuck the hard drives leaving them as NTFS and placing them as unassigned devices in my Unraid server with all the files still intact, And continue to backup to them. This has worked quite well. File corruption and bitrot is a concern of any photographer or illustrator. It is mentioned XFS is the preferred file system? Will this work with NTFS or BTRFS or ONLY XFS? Also is there a video any where on setting this up or even a demo of this plugin? Thanks kindly... Both NTFS and exFAT can store extended attributes so yes, DFI(or bunker from the command line) can run on those drives and files. 1 Quote Link to comment
foo_fighter Posted January 20 Share Posted January 20 3 hours ago, Vetteman said: For my array disks I use XFS. For cache and unassigned I use BTRFS and NTFS (NTFS - drives removed from Windows external drives which already have a file system and data). I thought somewhere here it was not recommended to use BTRFS on array disks, only XFS? Perhaps there is a PAR2 plugin for NTFS drives on Unraid? But searching I could not find any... I was hoping SpaceInvader One or someone like the "Bit my Bytes" bloke would have a video on Dynamix File Integrity. Cheers & many thanks.... You can find the PAR2 util in nerd-pack/tools. 1 Quote Link to comment
jayw1 Posted January 21 Share Posted January 21 Do my settings look okay? Should I check any boxes under verification tasks? I'm also running a build for each disk. Which I assume is what I should do. Anything else I should do here? Quote Link to comment
Vetteman Posted January 22 Share Posted January 22 On 1/20/2024 at 2:12 PM, foo_fighter said: Both NTFS and exFAT can store extended attributes so yes, DFI(or bunker from the command line) can run on those drives and files. I was using PAR2.EXE but I am always editing my RAW files into new PSD and JPG files. However the folders are constantly changing. As Photoshop and Lightroom evolve with new or better features. Therefore I will re-edited my files to massage them a bit more. Since I have my root folders named by year, 2009, 2010,...2022 and subfolders named under the appropriate year by event/place, A&W CAR SHOW, PEGGYS COVE, BANGOR MAINE, MONCTON NB, running PAR2 can be quite time consuming. AND I have to delete the existing PAR2 files before running PAR2. Quote Link to comment
Vetteman Posted January 22 Share Posted January 22 (edited) A youtube video on this plugin would be welcomed, since there appears to be youtube videos on just about every other plugin and docker. Edited January 23 by Vetteman Quote Link to comment
Vetteman Posted January 23 Share Posted January 23 Instead of selecting disks or shares, I would to prefer the root file within my share that contains all my photos and sub folders. Going through the setup I do not see an option to include a folder only exclude folders? Quote Link to comment
Vetteman Posted January 23 Share Posted January 23 On 1/20/2024 at 2:13 PM, foo_fighter said: You can find the PAR2 util in nerd-pack/tools. I used to have the nerd pack installed but I recall last year or so, another plugin recommended it be removed. I can not remember why or what plugin, so I removed it. Quote Link to comment
Vetteman Posted January 23 Share Posted January 23 (edited) On 11/25/2021 at 10:33 AM, wildfire305 said: I have file integrity set to generate automatically. It seems to keep up on a daily basis. The hashes are stored in the metadata in the filesystem (If I understand the process correctly). Check export, if done after build and export, should verify the hashes. Mine performs with thousands of checks when I do it. I also maintain separate hash catalogs and par2 for the really really important data. You could be safe with par2 as it generates hashes. I'm really surprised to see that not more people are using par2 as an action plan for corruption when restoring from backup. Obviously this is only practical for archival data and not constantly modified data. Are you running PAR2 from a Windows computer on the UNRAID share mapped as a drive? I checked the Unraid APP store and could not find anything for PAR2! i use the Windows gui Multipar which is the replacement for the older QUICKPAR gui. Cheers and many thanks from Nova Scotia.... Edited January 23 by Vetteman Quote Link to comment
Fetal9105 Posted January 29 Share Posted January 29 (edited) hi guys. Is it possible that Dynamix File Integrity Plugin has problems when a file change but the modification date / metadata remains the same? Why do I think that? It is noticeable that I have some corrupt files but: Only music files are affected. Several disks are affected Non music files on same disks are not affected all music files may be replaced, after initial hash check Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0109 - F.U.flac is corrupted Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0116 - Love It When You Hate Me acoustic.flac is corrupted Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0114 - Mercury in Retrograde.flac is corrupted Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0110 - All I Wanted.flac is corrupted Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0118 - Pity Party.lrc is corrupted Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0105 - Love Sux.flac is corrupted Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0102 - Bois Lie.lrc is corrupted Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0116 - Love It When You Hate Me acoustic.lrc is corrupted Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0112 - Break of a Heartache.flac is corrupted Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0101 - Cannonball.lrc is corrupted Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0115 - Bite Me acoustic.flac is corrupted Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0113 - I’m a Mess.flac is corrupted Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0104 - Love It When You Hate Me.flac is corrupted Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0118 - Pity Party.flac is corrupted Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0117 - Bois Lie acoustic.flac is corrupted Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0113 - I’m a Mess.lrc is corrupted Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0107 - Avalanche.flac is corrupted Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0115 - Bite Me acoustic.lrc is corrupted Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0106 - Kiss Me Like the World Is Ending.flac is corrupted Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0108 - Déjà vu.flac is corrupted Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0117 - Bois Lie acoustic.lrc is corrupted Jan 29 03:06:51 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0103 - Bite Me.flac is corrupted Jan 29 03:06:51 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0101 - Cannonball.flac is corrupted Jan 29 03:06:51 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0102 - Bois Lie.flac is corrupted Jan 29 03:06:51 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0111 - Dare to Love Me.flac is corrupted Jan 29 03:06:51 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0114 - Mercury in Retrograde.lrc is corrupted Whole album was replaced/updated yesterday My Unraid OS NON OC BTRFS/ZFS no damaged files reported Memory: 128 GiB DDR5 Single-bit ECC WS W790-ACE Intel w7-2475X Edited January 30 by Fetal9105 Quote Link to comment
JunctionRunner Posted March 12 Share Posted March 12 Hi, little odd quirk I'm having that's not hugely important, but it seems like the "find" function likes to grab some files and keep them active, which in turn means one of my disks and both parity drives are always active. I've rebooted, made sure hashes were up to date and killed the process, but it still seems to latch onto these particular files. Any idea how to fix this? I can't find any other examples of people with this issue so far. It does seem to periodically switch to a different group of files, but then hang onto those for quite a while. I did only run the initial hash of all files a few daysa go so perhaps it's still doing stuff. Quote Link to comment
JonathanM Posted March 12 Share Posted March 12 19 minutes ago, JunctionRunner said: it seems like the "find" function likes to grab some files and keep them active Do you have the cache_dirs plugin installed? Quote Link to comment
JunctionRunner Posted March 13 Share Posted March 13 2 hours ago, JonathanM said: Do you have the cache_dirs plugin installed? Oooh, yeah that's what's doing it. I guess it indexes files a lot slower than everything search does. Thanks, hopefully it doesn't take too much longer. Quote Link to comment
JonathanM Posted March 13 Share Posted March 13 1 minute ago, JunctionRunner said: Oooh, yeah that's what's doing it. I guess it indexes files a lot slower than everything search does. Thanks, hopefully it doesn't take too much longer. If you have too many folders set to be scanned in cache_dirs it will never finish, because old entries will be forced out of RAM before everything is loaded. This will cause drives to never spin down, the exact opposite of what you are intending. Only include folders proven to help the user experience. Your default stance should be to NOT cache unless you need it. Quote Link to comment
JunctionRunner Posted March 13 Share Posted March 13 21 hours ago, JonathanM said: If you have too many folders set to be scanned in cache_dirs it will never finish, because old entries will be forced out of RAM before everything is loaded. This will cause drives to never spin down, the exact opposite of what you are intending. Only include folders proven to help the user experience. Your default stance should be to NOT cache unless you need it. Yeah... I probably have a little much to index everything using this tool sadly, I changed the cache pressure setting to 1 but I saw basically no change in ram usage, I'll try 20 in case that actually works instead of just setting it to never reclaim. I have 128gb of ram right now and only 25% usage. I don't mind it using up a good amount if it makes file explorer on windows more responsive, as it is, it still kind of takes a bit to load everything up, appearing in chunks unlike when I was running windows server. It's too bad it's just stored in ram and can't run off the cache or use a special metadata device. Quote Link to comment
fr500 Posted April 2 Share Posted April 2 Hello Is there any way to pause generation of the integrity data? If not, could it be in the future, also a way to pause verification tasks. I did achieve a pause when I needed using killall with the STOP signal. Quote Link to comment
Joly0 Posted April 3 Share Posted April 3 Hey guys, how are the excluded folders working? I have this in my custom folders for exclusion ".Recycle.Bin, .filerun.thumbnails" (without the quotes) but today i got this in my logs Apr 3 11:49:42 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk3/backup/.Recycle.Bin/Chuwi_Backup/ab_20240318_000001/my-Pihole-DoT-DoH.xml is corrupted Apr 3 11:49:42 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk3/backup/.Recycle.Bin/Chuwi_Backup/ab_20240318_000001/my-homeassistant.xml is corrupted Apr 3 11:49:42 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk3/backup/.Recycle.Bin/Chuwi_Backup/ab_20240318_000001/my-AMP-Controller.xml is corrupted Apr 3 11:49:42 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk3/backup/.Recycle.Bin/Chuwi_Backup/ab_20240318_000001/backup.log is corrupted Apr 3 11:49:42 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk3/backup/.Recycle.Bin/Chuwi_Backup/ab_20240318_000001/my-AMP.xml is corrupted Apr 3 11:49:42 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk3/backup/.Recycle.Bin/Chuwi_Backup/ab_20240318_000001/chuwi1-flash-backup-20240318-0000.zip is corrupted Apr 3 11:49:42 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk3/backup/.Recycle.Bin/Chuwi_Backup/ab_20240318_000001/AMP-Controller.tar.zst is corrupted Apr 3 11:49:42 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk3/backup/.Recycle.Bin/Chuwi_Backup/ab_20240318_000001/Pihole-DoT-DoH.tar.zst is corrupted Apr 3 11:49:42 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk3/backup/.Recycle.Bin/Chuwi_Backup/ab_20240318_000001/config.json is corrupted Why are those files being checked? Shouldnt they be excluded? Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.