Dynamix File Integrity plugin

[karateo]

I still get file integrity check for files excluded.

I had removed/deleted files and reinstalled the plugin but I can't get it right.

 

See bonienl's message two comments ago, should help with exclusions.

 

Just did it and cleared some files.

 

Will get back if I get build-up-to-date = false again

 

[karateo]

It just happened again!

Both disks appear that build is not up to date!

 

[Gizmotoy]

It just happened again!

Both disks appear that build is not up to date!

Same here.  I cleared all disks last night.  I then ran Build on all disks to get them "up to date".  Today I have several disks that show that they're aren't up to date.  I ran clear again, and it cleared more files.

 

Seems like something's broken.  How can I help figure out what?

[bonienl]

After a Clear command, can you do an Export command and verify if any excluded files and folders are still present in the exported file(s)?

[DarkKnight]

I've read through a lot of pages at the beginning and end of this thread, and I can't find an relevant answer to my questions.

 

1) Is there any reason to use one hashing method over another? e.g. Blake2 vs MD5?

 

2) How can I benchmark my system with the different available methods to determine which is fastest?

 

Any of the methods will do the job just fine. If compatibility with other tools is required then MD5 is the safest choice.

 

A simple way to test is to copy a file to your /tmp folder so it stays in RAM memory and do

time md5sum /tmp/testfile
time b2sum /tmp/testfile
time sha256sum /tmp/testfile

 

Thanks. Seems Blake2 is not the fastest method, but I have used it for my entire array. MD5 is about 2x as fast, and SHA256 is about 2x as long. Seems like I should have used MD5.  Recommendations?

[karateo]

After a Clear command, can you do an Export command and verify if any excluded files and folders are still present in the exported file(s)?

 

Did that, opened the exported file with Libre Calc

used Find all for folder Photos which is the only one I need,

deleted all cells contained that text and nothing remained.

 

One thing that could trigger the problem is that I have folders with Greek characters.

I don't know how the plugin works but I thought I should mention just in case!

[Gizmotoy]

After a Clear command, can you do an Export command and verify if any excluded files and folders are still present in the exported file(s)?

Ok, performed a clear.  It found a dozen or so files that needed clearing since yesterday.  Ran export on all disks and grepped for ".nfo" and "CrashPlan", which are in my exclusion list. No instances of either found.

 

So I guess the auto-generation is somehow malfunctioning.  I disabled that feature, then re-enabled it.  We'll see if that does anything.

[Gizmotoy]

After a Clear command, can you do an Export command and verify if any excluded files and folders are still present in the exported file(s)?

Ok, performed a clear.  It found a dozen or so files that needed clearing since yesterday.  Ran export on all disks and grepped for ".nfo" and "CrashPlan", which are in my exclusion list. No instances of either found.

 

So I guess the auto-generation is somehow malfunctioning.  I disabled that feature, then re-enabled it.  We'll see if that does anything.

Just checking in.  No, disabling and then re-enabling "Automatically protect new and modified files" doesn't work.  Same behavior.

 

I tried upgrading to 6.2.1 (which I had planned to do anyway), and tried both tests again.  Same behavior.

 

Edit: Ok, I've done a LOT more testing and I'm nearly certain that there's a serious problem with this plugin, at least in my setup.  I took the following steps:

 

1) Run a build on all disks.  All show green checks.

2) Export all disks to check for excluded files.  All disks still have green checks.

3) Clear all disks.  Hundreds of files cleared since last clear operation yesterday.  All disks still have green checks.

4) Once the clear operation is complete, almost all disks show they need built (see screenshot).

5) Run build again.  Hundreds of new files hashed (see screenshot).  Between #1 and this step the array was idle.

 

I can then alternate between building and clearing, back-to-back, with hashes being added and cleared with every operation.  It seems like the Build and Clear operations are not using the same exclusion lists.

[Spritzup]

I installed this plugin, added the exclusions (Recycle bin and *.nfo), and did a build on disks (56TB).  It took about 30 hours to complete the build, and hit my system pretty hard (~90% CPU usage), but it finished.  I then have it set to check 5 disks at at time, once a week starting on Mondays.

 

So all that being said, my system should be idle now, but I still have 40+ processes of b2sum running that are doing a number on my system.  I assume this isn't normal, so I just want to know the best way to kill them.

 

Thanks!

 

~Spritz

 

EDIT - NVM, rebooted the system and cleared the processes.  That being said, I'm seeing the same issue as the posters above me.  Either we're mis-configuring the plugin in some (consistent) way, or perhaps a bug.  I also excluded .nfo files, but I'm still seeing the above behaviour happen.  The only thing that should be hitting those folders on a regular basis is Plex.

[EdgarWallace]

I'm seeing the same issue as the posters above me.  Either we're mis-configuring the plugin in some (consistent) way, or perhaps a bug.  I also excluded .nfo files, but I'm still seeing the above behaviour happen.

+1

 

Here is a copy&paste of my settings (files to be excluded):

*.nfo,*.dump,*.xml

....maybe a space after each file pattern  is missing? I have also cleared all disks according to your guidance ("After a Clear command, do an Export and verify if any excluded files and folders are still present in the exported file(s)") but still see some of these files.

 

I also enabled automatic hashing but each morning the system is showing a similar picture as shown below.

 

Not an issue to me but maybe I can help investigating the root cause.

[bonienl]

Thanks guys for all the feedback.

 

I'll need to investigate to see what is causing this. Keep you posted (and may ask for additional information).

 

[Perforator]

Sorry if this is a stupid question. Say you have this setup to run on a nightly basis and one morning you get a report that two files are corrupt. How do you find out what files are being reported as being corrupt? I know what disk they are on but no clue on the file names.

 

Thanks.

 

[JorgeB]

Sorry if this is a stupid question. Say you have this setup to run on a nightly basis and one morning you get a report that two files are corrupt. How do you find out what files are being reported as being corrupt? I know what disk they are on but no clue on the file names.

 

Thanks.

 

Check your syslog.

[Squid]

Just a question that relates to another project I'm doing..  (and I'm not too familiar with this plugin as I don't use it and only installed it for a test)

 

Does FIP always operate at the disk level, or does it modify the extended attributes on the user level?

 

IE:  I'm trying to figure out why an inotifywait (inotifywait -m @ /mnt/user/blah/blah) did not trigger any events at all when it created the hash for the file by modifying the extended attributes.  (I was expecting at the very least an open/close (and didn't get one during by telling FIP to create the hashes, but I need to know if it also triggers an ATTRIBUTE event at the user level)

 

[bonienl]

Just a question that relates to another project I'm doing..  (and I'm not too familiar with this plugin as I don't use it and only installed it for a test)

 

Does FIP always operate at the disk level, or does it modify the extended attributes on the user level?

 

IE:  I'm trying to figure out why an inotifywait (inotifywait -m @ /mnt/user/blah/blah) did not trigger any events at all when it created the hash for the file by modifying the extended attributes.  (I was expecting at the very least an open/close (and didn't get one during by telling FIP to create the hashes, but I need to know if it also triggers an ATTRIBUTE event at the user level)

 

FIP makes use of inotifywait to monitor file changes. The command line looks something like this (depending on selected options):

 

inotifywait -dsrqo /var/run/hash.pipe -e close_write --exclude ^/mnt/disk[0-9]+/files --format %w%f /mnt/disk1 /mnt/disk2 /mnt/disk3 ...

 

This let inotifywait run as daemon and act only when (new) files are closed and push their name in a piped file, which on its turn is read and feed bunker to do the hash calculations and store the result in the extended attributes.

 

Ps. changing extended attributes will not trigger a file change.

 

 

[Squid]

Just a question that relates to another project I'm doing..  (and I'm not too familiar with this plugin as I don't use it and only installed it for a test)

 

Does FIP always operate at the disk level, or does it modify the extended attributes on the user level?

 

IE:  I'm trying to figure out why an inotifywait (inotifywait -m @ /mnt/user/blah/blah) did not trigger any events at all when it created the hash for the file by modifying the extended attributes.  (I was expecting at the very least an open/close (and didn't get one during by telling FIP to create the hashes, but I need to know if it also triggers an ATTRIBUTE event at the user level)

 

FIP makes use of inotifywait to monitor file changes. The command line looks something like this (depending on selected options):

 

inotifywait -dsrqo /var/run/hash.pipe -e close_write --exclude ^/mnt/disk[0-9]+/files --format %w%f /mnt/disk1 /mnt/disk2 /mnt/disk3 ...

 

This let inotifywait run as daemon and act only when (new) files are closed and push their name in a piped file, which on its turn is read and feed bunker to do the hash calculations and store the result in the extended attributes.

 

Ps. changing extended attributes will not trigger a file change.

Ok. That's why I wasn't getting and results from inotifywait. Because I'm looking at it on the user level and you're looking at it on the disk level.  Wasn't worried about changes to the file but needed to know if you were triggering an attribute event (which is the only way to know if a hardlink to the file gets deleted)

 

Sent from my LG-D852 using Tapatalk

 

 

[Perforator]

Sorry if this is a stupid question. Say you have this setup to run on a nightly basis and one morning you get a report that two files are corrupt. How do you find out what files are being reported as being corrupt? I know what disk they are on but no clue on the file names.

 

Thanks.

 

Check your syslog.

 

Thank you. I was able to find the two files that were reported corrupt. They were .jpg files and seem fine. The next question would be will it recalculate the checksums for these two files or will they forever, be marked as corrupt and if the latter, how do I fix it?

 

 

[bonienl]

Sorry if this is a stupid question. Say you have this setup to run on a nightly basis and one morning you get a report that two files are corrupt. How do you find out what files are being reported as being corrupt? I know what disk they are on but no clue on the file names.

 

Thanks.

 

Check your syslog.

 

Thank you. I was able to find the two files that were reported corrupt. They were .jpg files and seem fine. The next question would be will it recalculate the checksums for these two files or will they forever, be marked as corrupt and if the latter, how do I fix it?

 

Hash keys are never regenerated for corrupted files. To force a new hash key you need to move or rename the original file.

 

[Perforator]

Sorry if this is a stupid question. Say you have this setup to run on a nightly basis and one morning you get a report that two files are corrupt. How do you find out what files are being reported as being corrupt? I know what disk they are on but no clue on the file names.

 

Thanks.

 

Check your syslog.

 

Thank you. I was able to find the two files that were reported corrupt. They were .jpg files and seem fine. The next question would be will it recalculate the checksums for these two files or will they forever, be marked as corrupt and if the latter, how do I fix it?

 

Hash keys are never regenerated for corrupted files. To force a new hash key you need to move or rename the original file.

 

ok thanks, so rename, wait for it to run, then rename back right?

[bonienl]

Sorry if this is a stupid question. Say you have this setup to run on a nightly basis and one morning you get a report that two files are corrupt. How do you find out what files are being reported as being corrupt? I know what disk they are on but no clue on the file names.

 

Thanks.

 

Check your syslog.

 

Thank you. I was able to find the two files that were reported corrupt. They were .jpg files and seem fine. The next question would be will it recalculate the checksums for these two files or will they forever, be marked as corrupt and if the latter, how do I fix it?

 

Hash keys are never regenerated for corrupted files. To force a new hash key you need to move or rename the original file.

 

ok thanks, so rename, wait for it to run, then rename back right?

 

Right.

 

[Kir]

Hmm.. Just got a notification that Disk 2 needs updating = 2 files didn't have the checksum. No indication which ones.

 

Updated, again, nothing. The only record in the syslog:

bunker: added 2 files from /mnt/disk2. Duration: 00:10:02. Average speed: 96.4 MB/s

 

[bonienl]

Hmm.. Just got a notification that Disk 2 needs updating = 2 files didn't have the checksum. No indication which ones.

 

Updated, again, nothing. The only record in the syslog:

bunker: added 2 files from /mnt/disk2. Duration: 00:10:02. Average speed: 96.4 MB/s

 

You can enable Save new hashing results to flash, this will stored all new file names added on a daily basis.

[Kir]

You can enable Save new hashing results to flash, this will stored all new file names added on a daily basis.

 

But it wouldn't show the files with missing hashes as I understand since it's somehow missed these 2 files. How could it happen in the first place?

[bonienl]

You can enable Save new hashing results to flash, this will stored all new file names added on a daily basis.

 

But it wouldn't show the files with missing hashes as I understand since it's somehow missed these 2 files. How could it happen in the first place?

It would give you more insight in which files were added.

 

Files can get missed when too many files change at the same time. Names are queued for hash processing and the queue isn't infinite. Or the inotifywait process missed the change at all.

 

The recommendation is to exclude files/folders which change frequently. File corruption for those files/folders will be discovered during regular read/write operations.

 

[John_M]

Hmm.. Just got a notification that Disk 2 needs updating = 2 files didn't have the checksum. No indication which ones.

 

Updated, again, nothing. The only record in the syslog:

bunker: added 2 files from /mnt/disk2. Duration: 00:10:02. Average speed: 96.4 MB/s

 

A suggestion: when the plugin notices that a disk needs updating because some files don't have checksums, could the Build process (and the Export process, if enabled - ie. orange circle, not blue cross) be triggered automatically? Perhaps include an option to enable this behaviour or leave it to be done manually.