How can i excluded files, types or Folder?

Enhancement request:  It would be great if the Find duplicate files by hash option could output file size and possibly sort with largest file first.  I'd be interested in deleting duplicates that were large, but I don't care about smaller duplicate files.

 

Thanks!

I was very excited when I found this plugin as I am an avid photographer and have taken many landscape and car show photos over the last 20 years. I also do digital illustrations and 3D modeling.

 

Previously I was backing up all my edited and raw file photos and digital illustrations and 3D assets to external WD and Seagate USB NTFS hard drives. I started to schuck the hard drives leaving them as NTFS and placing them as unassigned devices in my Unraid server with all the files still intact,  And continue to backup to them.

 

This has worked quite well.

 

File corruption and bitrot is a concern of any photographer or illustrator.

 

It is mentioned XFS is the preferred file system? Will this work with NTFS or BTRFS or ONLY XFS?

 

Also is there a video any where on setting this up or even a demo of this plugin?

 

Thanks kindly...

 

 

 

If you btrfs that has built-in check summing for detecting bitrot.

For my array disks I use XFS. For cache and unassigned I use BTRFS and NTFS (NTFS - drives removed from Windows external drives which already have a file system and data).

 

I thought somewhere here it was not recommended to use BTRFS on array disks, only XFS?

 

Perhaps there is a PAR2 plugin for NTFS drives on Unraid? But searching I could not find any...

 

I was hoping SpaceInvader One or someone like the "Bit my Bytes" bloke would have a video on  Dynamix File Integrity.

 

Cheers & many thanks....

Edited January 20, 20242 yr by Vetteman

1 hour ago, Vetteman said:

I thought somewhere here it was not recommended to use BTRFS on array disks, only XFS?

Depends on who you ask, but it has been quite a while since that was the general consensus. 

On 1/18/2024 at 5:33 AM, Vetteman said:

I was very excited when I found this plugin as I am an avid photographer and have taken many landscape and car show photos over the last 20 years. I also do digital illustrations and 3D modeling.

 

Previously I was backing up all my edited and raw file photos and digital illustrations and 3D assets to external WD and Seagate USB NTFS hard drives. I started to schuck the hard drives leaving them as NTFS and placing them as unassigned devices in my Unraid server with all the files still intact,  And continue to backup to them.

 

This has worked quite well.

 

File corruption and bitrot is a concern of any photographer or illustrator.

 

It is mentioned XFS is the preferred file system? Will this work with NTFS or BTRFS or ONLY XFS?

 

Also is there a video any where on setting this up or even a demo of this plugin?

 

Thanks kindly...

 

 

 

Both NTFS and exFAT can store extended attributes so yes, DFI(or bunker from the command line) can run on those drives and files. 

3 hours ago, Vetteman said:

For my array disks I use XFS. For cache and unassigned I use BTRFS and NTFS (NTFS - drives removed from Windows external drives which already have a file system and data).

 

I thought somewhere here it was not recommended to use BTRFS on array disks, only XFS?

 

Perhaps there is a PAR2 plugin for NTFS drives on Unraid? But searching I could not find any...

 

I was hoping SpaceInvader One or someone like the "Bit my Bytes" bloke would have a video on  Dynamix File Integrity.

 

Cheers & many thanks....

You can find the PAR2 util in nerd-pack/tools.

Do my settings look okay? Should I check any boxes under verification tasks?

 

 

I'm also running a build for each disk.

 

 

Which I assume is what I should do. Anything else I should do here?

On 1/20/2024 at 2:12 PM, foo_fighter said:

Both NTFS and exFAT can store extended attributes so yes, DFI(or bunker from the command line) can run on those drives and files. 

I was using PAR2.EXE but I am always editing my RAW files into new PSD and JPG files. However the folders are constantly changing. As Photoshop and Lightroom evolve with new or better features. Therefore I will re-edited my files to massage them a bit more. Since I have my root folders named by year, 2009, 2010,...2022 and subfolders named under the appropriate year by event/place, A&W CAR SHOW, PEGGYS COVE, BANGOR MAINE, MONCTON NB, running PAR2 can be quite time consuming. AND I have to delete the existing PAR2 files before running PAR2.

A youtube video on this plugin would be welcomed, since there appears to be youtube videos on just about every other plugin and docker.

Edited January 23, 20242 yr by Vetteman

Instead of selecting disks or shares, I would to prefer the root file within my share that contains all my photos and sub folders.

 

Going through the setup I do not see an option to include a folder only exclude folders?

 

 

On 1/20/2024 at 2:13 PM, foo_fighter said:

You can find the PAR2 util in nerd-pack/tools.

 I used to have the nerd pack installed but I recall last year or so, another plugin recommended it be removed. I can not remember why or what plugin, so I removed it.

On 11/25/2021 at 10:33 AM, wildfire305 said:

I have file integrity set to generate automatically. It seems to keep up on a daily basis. The hashes are stored in the metadata in the filesystem (If I understand the process correctly). Check export, if done after build and export, should verify the hashes. Mine performs with thousands of checks when I do it. I also maintain separate hash catalogs and par2 for the really really important data. You could be safe with par2 as it generates hashes. I'm really surprised to see that not more people are using par2 as an action plan for corruption when restoring from backup. Obviously this is only practical for archival data and not constantly modified data.

Are you running PAR2 from a Windows computer on  the UNRAID share mapped as a drive? I checked the Unraid APP store and could not find anything for PAR2! i use the Windows gui Multipar which is the replacement for the older QUICKPAR gui.

 

Cheers and many thanks from Nova Scotia....

Edited January 23, 20242 yr by Vetteman

hi guys.
Is it possible that Dynamix File Integrity Plugin has problems when a file change but the modification date / metadata remains the same?

 

Why do I think that?
It is noticeable that I have some corrupt files but:
Only music files are affected.
Several disks are affected
Non music files on same disks are not affected

all music files may be replaced, after initial hash check

 

Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0109 - F.U.flac is corrupted
Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0116 - Love It When You Hate Me acoustic.flac is corrupted
Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0114 - Mercury in Retrograde.flac is corrupted
Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0110 - All I Wanted.flac is corrupted
Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0118 - Pity Party.lrc is corrupted
Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0105 - Love Sux.flac is corrupted
Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0102 - Bois Lie.lrc is corrupted
Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0116 - Love It When You Hate Me acoustic.lrc is corrupted
Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0112 - Break of a Heartache.flac is corrupted
Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0101 - Cannonball.lrc is corrupted
Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0115 - Bite Me acoustic.flac is corrupted
Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0113 - I’m a Mess.flac is corrupted
Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0104 - Love It When You Hate Me.flac is corrupted
Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0118 - Pity Party.flac is corrupted
Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0117 - Bois Lie acoustic.flac is corrupted
Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0113 - I’m a Mess.lrc is corrupted
Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0107 - Avalanche.flac is corrupted
Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0115 - Bite Me acoustic.lrc is corrupted
Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0106 - Kiss Me Like the World Is Ending.flac is corrupted
Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0108 - Déjà vu.flac is corrupted
Jan 29 03:06:50 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0117 - Bois Lie acoustic.lrc is corrupted
Jan 29 03:06:51 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0103 - Bite Me.flac is corrupted
Jan 29 03:06:51 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0101 - Cannonball.flac is corrupted
Jan 29 03:06:51 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0102 - Bois Lie.flac is corrupted
Jan 29 03:06:51 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0111 - Dare to Love Me.flac is corrupted
Jan 29 03:06:51 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk13/Musik/Avril Lavigne/Avril Lavigne - Album - 2022 - Love Sux/0114 - Mercury in Retrograde.lrc is corrupted

 

Whole album was replaced/updated yesterday

 

My Unraid OS
NON OC
BTRFS/ZFS no damaged files reported

Memory: 128 GiB DDR5 Single-bit ECC
WS W790-ACE
Intel w7-2475X

Edited January 30, 20242 yr by Fetal9105

Hi, little odd quirk I'm having that's not hugely important, but it seems like the "find" function likes to grab some files and keep them active, which in turn means one of my disks and both parity drives are always active.

I've rebooted, made sure hashes were up to date and killed the process, but it still seems to latch onto these particular files.

Any idea how to fix this? I can't find any other examples of people with this issue so far. It does seem to periodically switch to a different group of files, but then hang onto those for quite a while. I did only run the initial hash of all files a few daysa go so perhaps it's still doing stuff.




 

19 minutes ago, JunctionRunner said:

it seems like the "find" function likes to grab some files and keep them active

Do you have the cache_dirs plugin installed?

2 hours ago, JonathanM said:

Do you have the cache_dirs plugin installed?

Oooh, yeah that's what's doing it. I guess it indexes files a lot slower than everything search does. Thanks, hopefully it doesn't take too much longer.

1 minute ago, JunctionRunner said:

Oooh, yeah that's what's doing it. I guess it indexes files a lot slower than everything search does. Thanks, hopefully it doesn't take too much longer.

If you have too many folders set to be scanned in cache_dirs it will never finish, because old entries will be forced out of RAM before everything is loaded.

 

This will cause drives to never spin down, the exact opposite of what you are intending.

 

Only include folders proven to help the user experience. Your default stance should be to NOT cache unless you need it.

21 hours ago, JonathanM said:

If you have too many folders set to be scanned in cache_dirs it will never finish, because old entries will be forced out of RAM before everything is loaded.

 

This will cause drives to never spin down, the exact opposite of what you are intending.

 

Only include folders proven to help the user experience. Your default stance should be to NOT cache unless you need it.

Yeah... I probably have a little much to index everything using this tool sadly, I changed the cache pressure setting to 1 but I saw basically no change in ram usage, I'll try 20 in case that actually works instead of just setting it to never reclaim. I have 128gb of ram right now and only 25% usage.

I don't mind it using up a good amount if it makes file explorer on windows more responsive, as it is, it still kind of takes a bit to load everything up, appearing in chunks unlike when I was running windows server. It's too bad it's just stored in ram and can't run off the cache or use a special metadata device.

Hello

Is there any way to pause generation of the integrity data?

If not, could it be in the future, also a way to pause verification tasks.

 

I did achieve a pause when I needed  using killall with the STOP signal.

Hey guys, how are the excluded folders working? I have this in my custom folders for exclusion ".Recycle.Bin, .filerun.thumbnails" (without the quotes) but today i got this in my logs

Apr  3 11:49:42 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk3/backup/.Recycle.Bin/Chuwi_Backup/ab_20240318_000001/my-Pihole-DoT-DoH.xml is corrupted
Apr  3 11:49:42 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk3/backup/.Recycle.Bin/Chuwi_Backup/ab_20240318_000001/my-homeassistant.xml is corrupted
Apr  3 11:49:42 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk3/backup/.Recycle.Bin/Chuwi_Backup/ab_20240318_000001/my-AMP-Controller.xml is corrupted
Apr  3 11:49:42 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk3/backup/.Recycle.Bin/Chuwi_Backup/ab_20240318_000001/backup.log is corrupted
Apr  3 11:49:42 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk3/backup/.Recycle.Bin/Chuwi_Backup/ab_20240318_000001/my-AMP.xml is corrupted
Apr  3 11:49:42 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk3/backup/.Recycle.Bin/Chuwi_Backup/ab_20240318_000001/chuwi1-flash-backup-20240318-0000.zip is corrupted
Apr  3 11:49:42 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk3/backup/.Recycle.Bin/Chuwi_Backup/ab_20240318_000001/AMP-Controller.tar.zst is corrupted
Apr  3 11:49:42 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk3/backup/.Recycle.Bin/Chuwi_Backup/ab_20240318_000001/Pihole-DoT-DoH.tar.zst is corrupted
Apr  3 11:49:42 Tower bunker: error: BLAKE3 hash key mismatch, /mnt/disk3/backup/.Recycle.Bin/Chuwi_Backup/ab_20240318_000001/config.json is corrupted

Why are those files being checked? Shouldnt they be excluded?

So after a recent overhaul of my server I am looking back into using this plugin again. However the one question I have that I can't seem to find any mention of is moving the hash file location. Seems if I'm reading correctly it is hard coded to store on the flash drive which I'd like to avoid for obvious reasons. Any chance it can get easily pushed to cache instead? 

My exclusions do not work. I have set:

 

Excluded folders: .Trash-99  .incoming  Data_Shadow  Documents_Shadow  Public  Stickware  .Recycle.Bin  .cache

Excluded files: *.nfo , metadata.db

 

But still get:

BLAKE3 hash key mismatch (updated), /mnt/disk7/Books/Calibre/.cache/calibre/server-log.txt was modified
BLAKE3 hash key mismatch (updated), /mnt/disk7/Books/Calibre/metadata.db was modified
 

Any idea how to fix that? Thanks!

I am also seeing the same issue with the excluded folders. Is this plugin still being maintained?  There isn't much going on in their GitHub repository.