Dynamix File Integrity plugin

[Squid]

Try deleting /tmp/plugins/dynamix.file.integrity.plg (or whatever its called)

 

 

[John_M]

I've just updated to 2016.02.20b and reapplied my settings but I'm now seeing a pink checkbox:

 

EDIT: I tried clicking the Default button, then re-selecting and reapplying my options, but no change.

 

UPDATE: I uninstalled the plugin and re-installed it. The pink box has gone away.

[bonienl]

I've just updated to 2016.02.20b and reapplied my settings but I'm now seeing a pink checkbox:

 

EDIT: I tried clicking the Default button, then re-selecting and reapplying my options, but no change.

 

UPDATE: I uninstalled the plugin and re-installed it. The pink box has gone away.

 

A pink (red) box means that a background verification process is running for that particular disk. As long as the background verification isn't finished no changes to that disk can be made.

 

See also HELP

 

[bonienl]

A new version 2016.02.21 is available. It is mandatory to re-apply settings after upgrading. See release notes for more details.

 

[wgstarks]

Maybe I'm not re-applying the settings correctly? I can't get exclusions to work though.

 

After installing the update I set all exclusions to none and clicked "Apply".

Re-entered my exclusions and clicked "Apply".

Switched to "File Integrity Control" and clicked "Select All" and then "Clear".

The report was "0 files cleared 0 files skipped".

 

I must have done something wrong here. ???

[bonienl]

Maybe I'm not re-applying the settings correctly? I can't get exclusions to work though.

 

After installing the update I set all exclusions to none and clicked "Apply".

Re-entered my exclusions and clicked "Apply".

Switched to "File Integrity Control" and clicked "Select All" and then "Clear".

The report was "0 files cleared 0 files skipped".

 

I must have done something wrong here. ???

 

My test sequence:

 

1. Create a new share "test"

2. Copy a file to this new share

3. Verify that extended attributes are added to this file

4. Select all disks, run Clear --> 0 files cleared (as expected)

5. Select new share "test" as an excluded folder

6. Select all disks, run Clear --> 1 file cleared (as expected)

 

Not sure why your case doesn't work...

 

[John_M]

I've just updated to 2016.02.20b and reapplied my settings but I'm now seeing a pink checkbox:

 

EDIT: I tried clicking the Default button, then re-selecting and reapplying my options, but no change.

 

UPDATE: I uninstalled the plugin and re-installed it. The pink box has gone away.

 

A pink (red) box means that a background verification process is running for that particular disk. As long as the background verification isn't finished no changes to that disk can be made.

 

See also HELP

 

Ah, OK. Thanks. Maybe I ought to have just left it to do its thing. I do use the Help system and some plugins make better use of it than others, but in this instance I didn't see the reference to the red box - I was simply looking in the wrong place (by the table itself, not higher up the page).

 

[ljm42]

I removed both the custom folder and the custom file, but it still won't "Clear" the attributes from the test files I created in the CrashplanTower share on disk1.

 

Just wanted to let you know that the 2016.02.21 version solved this, Clear now works as expected.

 

I like the change from AND to OR too.

 

Thanks!

[wgstarks]

Looks like the update may have solved my problems as well. Did a clear after resetting the app and it actually reported clearing a large number of files. Previously it always reported 0. Did an automated check and no warnings for excluded shares.

[bonienl]

A new version 2016.02.23 is available. It is mandatory to re-apply settings after upgrading. See release notes for more details.

[danioj]

A new version 2016.02.23 is available. It is mandatory to re-apply settings after upgrading. See release notes for more details.

 

Dude!! Lots of updates at the moment  Can you please define what you mean by "re-apply settings"??

 

I have updated a few times recently and each time I have Remove, Build, Export.

 

Takes a While - is this the correct procedure?

[bonienl]

A new version 2016.02.23 is available. It is mandatory to re-apply settings after upgrading. See release notes for more details.

 

Dude!! Lots of updates at the moment  Can you please define what you mean by "re-apply settings"??

 

I have updated a few times recently and each time I have Remove, Build, Export.

 

Takes a While - is this the correct procedure?

 

Re-apply settings means on the settings page you need to press the apply button. To enable the apply button just make a change and change it back.

 

There is no need to do any of the control commands (build, export, etc..)

 

[danioj]

A new version 2016.02.23 is available. It is mandatory to re-apply settings after upgrading. See release notes for more details.

 

Dude!! Lots of updates at the moment  Can you please define what you mean by "re-apply settings"??

 

I have updated a few times recently and each time I have Remove, Build, Export.

 

Takes a While - is this the correct procedure?

 

Re-apply settings means on the settings page you need to press the apply button. To enable the apply button just make a change and change it back.

 

There is no need to do any of the control commands (build, export, etc..)

 

 

Bugger! LOL!! Oh well, gave the disks some exercise!!

 

Love the new exclude options BTW. .nfo added to exclude and ticked the apple data box!! Great stuff!

 

Thanks!!

[bonienl]

Bugger! LOL!! Oh well, gave the disks some exercise!!

 

Sure your unRAID system likes to get tickled 

[danioj]

bonienl, the plugin ran its scheduled check last night and I got several emails telling me that a number of static media files (.mkv's) had failed their check and were "corrupt". I don't run any software that could change these files.

 

My issue is this, I only re-ran the check to hash these files 48hours ago. If these notifications are to be believed then that means that in the space of 24 hours I have encountered betroth on these files which are held on what is a fairly new drive too?

 

Could I perhaps be experiencing issues because I have rebuilt and exported a few times (see above post when I thought the instruction to re-apply settings was to do a full rebuild and export) when new releases of the plugin came out recently??

[bonienl]

bonienl, the plugin ran its scheduled check last night and I got several emails telling me that a number of static media files (.mkv's) had failed their check and were "corrupt". I don't run any software that could change these files.

 

My issue is this, I only re-ran the check to hash these files 48hours ago. If these notifications are to be believed then that means that in the space of 24 hours I have encountered betroth on these files which are held on what is a fairly new drive too?

 

Could I perhaps be experiencing issues because I have rebuilt and exported a few times (see above post when I thought the instruction to re-apply settings was to do a full rebuild and export) when new releases of the plugin came out recently??

 

There is no relation between Build/Export commands and file corruption.

 

This might have happened due to the changes made in bunker. Did you re-apply the settings or stop/start the service? This will force the latest version of bunker to be used.

 

[danioj]

bonienl, the plugin ran its scheduled check last night and I got several emails telling me that a number of static media files (.mkv's) had failed their check and were "corrupt". I don't run any software that could change these files.

 

My issue is this, I only re-ran the check to hash these files 48hours ago. If these notifications are to be believed then that means that in the space of 24 hours I have encountered betroth on these files which are held on what is a fairly new drive too?

 

Could I perhaps be experiencing issues because I have rebuilt and exported a few times (see above post when I thought the instruction to re-apply settings was to do a full rebuild and export) when new releases of the plugin came out recently??

 

There is no relation between Build/Export commands and file corruption.

 

This might have happened due to the changes made in bunker. Did you re-apply the settings or stop/start the service? This will force the latest version of bunker to be used.

 

Reapplying the settings didn't work so I just rebooted. Checked the Files again, no file corruption or mismatch. All working now.

[JorgeB]

I’m trying to isolate the issue better but on some of my servers new files begin to hash while I’m still copying more files, making the write speed much slower, is there a way to make the plugin wait until copy is over before starting to hash new files?

 

Strangely, like I mentioned, on other servers the hashing begins only when all files are copied.

 

[bonienl]

I’m trying to isolate the issue better but on some of my servers new files begin to hash while I’m still copying more files, making the write speed much slower, is there a way to make the plugin wait until copy is over before starting to hash new files?

 

Strangely, like I mentioned, on other servers the hashing begins only when all files are copied.

 

As soon as inotifywait "sees" a new/modified file completed it will call bunker to do the hashing, this will run in parallel with the copying process when more files are being copied.

 

I understand your observation, but not sure how to change/improve that. Will have a look at it though!

[Squid]

I’m trying to isolate the issue better but on some of my servers new files begin to hash while I’m still copying more files, making the write speed much slower, is there a way to make the plugin wait until copy is over before starting to hash new files?

 

Strangely, like I mentioned, on other servers the hashing begins only when all files are copied.

 

As soon as inotifywait "sees" a new/modified file completed it will call bunker to do the hashing, this will run in parallel with the copying process when more files are being copied.

 

I understand your observation, but not sure how to change/improve that. Will have a look at it though!

When I was doing Checksum Suite, I found that inotifywait could miss the odd new / modified file if it was being bombarded with files.  My solution was to send its output to a FIFO pipe and have another script check the pipe.  Wasn't that elegant, but it never missed a notification

[bonienl]

When I was doing Checksum Suite, I found that inotifywait could miss the odd new / modified file if it was being bombarded with files.  My solution was to send its output to a FIFO pipe and have another script check the pipe.  Wasn't that elegant, but it never missed a notification

 

Thanks for the pointer, and you know ... beauty is in the eye of the beholder

[beardedpants]

Is there any obvious reason why scheduled integrity checking stopped working for me? I'm running the latest version, but this problem has existed for the past couple of versions. I wish I knew exactly when it stopped working, but I don't remember.

 

My schedule is set to run weekly on Monday at 19:00. As far as I can tell, a scan is never even attempted. There are no entries for bunker in the syslog, good nor bad.

 

Things I've tried:

Removing all hashes and rebuilding, multiple times.

Uninstalling the plugin and deleting the /boot/config/plugins/dynamix.file.integrity/ directory.

Switching algorithms from BLAKE2 to SHA2

 

The scheduled time does not conflict with anything else, if that even matters. It did actually work for the first few weeks after I set it up,

[m3ki]

I was wondering...

 

This plugin wont warn me against things like ransomware/cryptolocker will it? especially if "Automatically protect new and modified files" is enabled?

[archedraft]

Is there any obvious reason why scheduled integrity checking stopped working for me? I'm running the latest version, but this problem has existed for the past couple of versions. I wish I knew exactly when it stopped working, but I don't remember.

 

My schedule is set to run weekly on Monday at 19:00. As far as I can tell, a scan is never even attempted. There are no entries for bunker in the syslog, good nor bad.

 

Things I've tried:

Removing all hashes and rebuilding, multiple times.

Uninstalling the plugin and deleting the /boot/config/plugins/dynamix.file.integrity/ directory.

Switching algorithms from BLAKE2 to SHA2

 

The scheduled time does not conflict with anything else, if that even matters. It did actually work for the first few weeks after I set it up,

 

Sort of along the same topic, would it be possible to add notifications like how parity checks work ie ("FIP has started checking the integrity of your files" and "FIP has finished checking all disks and has found XX file modifications and XX file corruptions")?

[bobekdj]

Finally able to read through the entire feed, and was unable to find an answer to the following questions (sorry for a lot of them, tried to split up to be easy to answer).  Maybe the summary of *bitrot, *best method, and *compatability could be added to the sticky at the top of the thread to inform new users these questions?

 

3 different hashing methods, MD5, SHA2, BLAKE2... what are the pros and cons of each?

 

From what I have gathered, related to this plugin it essentially comes down to speed of hash.

 

Any of the options seem to be more than adequate at detecting bitrot?

 

And as for universal compatibility... why does it matter since tools like corz and others can only tell you if your files are corrupt anyway and not do anything to help restore.

 

 

In summary, it appears if your CPU supports it, BLAKE2 would be be the best option most recommend?  Also there was some mention of using CORZ to verify the hashes produced... did anyone ever figure out if the BLAKE2 outputs were similar, or if they still use different variants (s)?

 

Thanks for a great plugin and support!