Dynamix File Integrity plugin

[Marv]

Hi,

 

I've enabled 'Save new hashing results to flash' and get entries from all disks and shares listed there.

As I've excluded all shares but my media share in the plugin settings I'm wondering why this happens.

Shouldn't the plugin only monitor the not excluded shares? Maybe I just don't understand the setting correctly.

Just started using your plugin.

[Marv]

On 16.10.2017 at 3:54 PM, Marv said:

Hi,

 

I've enabled 'Save new hashing results to flash' and get entries from all disks and shares listed there.

As I've excluded all shares but my media share in the plugin settings I'm wondering why this happens.

Shouldn't the plugin only monitor the not excluded shares? Maybe I just don't understand the setting correctly.

Just started using your plugin.

 

Anyone? Just wondering why also files from excluded shares are getting monitored.

[FreeMan]

What's the difference between a mismatch:

 

BLAKE2 hash key mismatch (updated), /mnt/disk7/path/file was modified

 

and corruption:

 

BLAKE2 hash key mismatch, /mnt/disk11/path/file is corrupted

 

I would expect that a mismatch would happen any time a file is updated on the server and that's something I don't need to worry about at all (for files that I expect to change). Is the file corruption a corruption of the stored hash key or a corruption of the file itself? I've looked into many of the files that FIP has reported with corruptions, and I've not had any issues with them that I could detect (many of them are JPGs, and I can pull up the picture and look at it without any software complaining about it).

[Vr2Io]

On ‎2017‎年‎11‎月‎19‎日 at 9:16 PM, Marv said:

 

Anyone? Just wondering why also files from excluded shares are getting monitored.

 

The plugin exclude means exclude for generation hash, so if a file already hash and export, then it will hash_check every time.

You may modify the hash file and remove those entry, then execute clear / import once.

Edited November 22, 2017 by Benson

[Marv]

On 22.11.2017 at 11:51 AM, Benson said:

 

The plugin exclude means exclude for generation hash, so if a file already hash and export, then it will hash_check every time.

You may modify the hash file and remove those entry, then execute clear / import once.

 

Sorry, that's not what I meant or I just understand you correctly

Maybe I was a bit unclear.

 

So I excluded all but my media share in the plugin settings. I did that before the initial build was done.

After writing the hash values to the extended attributes I did an export.

Each file created for each disk only contained the file paths and corresponidng hash values from my media share.

So I think the exclusion worked properly.

 

What I also did was enabling 'Save new hashing results to flash' besides the auto monitoring of new and changed files.

Now each day I get a file saved to my flash drive containing file paths and hash values from all disks and shares.

 

And that's what I'm wondering about. Shouldn't these daily files (just like the export files) only contain information from my media share?

Edited November 23, 2017 by Marv

[Vr2Io]

Would you check the background process look like, I test by exclude "data" & "tv"share, then it will be

 

/bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/watcher a -md5 (data/|tv/)

 

 

If nothing exclude, it would be

 

/bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/watcher a -md5

 

Edited November 23, 2017 by Benson

[Marv]

So I need to use:

 

/bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/watcher a -md5

 

and the output would be the excluded folders?

I don't get any output from this so I guess my exclusions are not recognized?

[Vr2Io]

3 hours ago, Marv said:

So I need to use:

 

/bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/watcher a -md5

 

and the output would be the excluded folders?

I don't get any output from this so I guess my exclusions are not recognized?

 

Those are background process checking, not directly type at command prompt, pls at command prompt type " ps -elf | grep dynamix.file.integrity  "

The setting in GUI actually was change the background process parameter.

 

/bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/watcher a -md5 (data/|tv/)  <--- this show "data" & "tv" share was exclude

/bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/watcher a -md5  <--- this show nothing exclude

[Marv]

10 hours ago, Benson said:

 

Those are background process checking, not directly type at command prompt, pls at command prompt type " ps -elf | grep dynamix.file.integrity  "

The setting in GUI actually was change the background process parameter.

 

/bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/watcher a -md5 (data/|tv/)  <--- this show "data" & "tv" share was exclude

/bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/watcher a -md5  <--- this show nothing exclude

 

Ok thanks.

 

So I get: /bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/watcher a -b2  Backup|Downloads,ISOs,Kodi,Nextcloud,Recordings

[Vr2Io]

5 hours ago, Marv said:

 

Ok thanks.

 

So I get: /bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/watcher a -b2  Backup|Downloads,ISOs,Kodi,Nextcloud,Recordings

 

There are missing "(" ")" and shouldn't have "," between.

 

Below are my some test result for the setting in GUI

/bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/watcher a -b2 (.*123/|.*456/|.*test$|data/|tv/)

 

Suggest you re-apply the setting in setting page and try again.

Edited November 24, 2017 by Benson

[Marv]

6 hours ago, Benson said:

 

There are missing "(" ")" and shouldn't have "," between.

 

Below are my some test result for the setting in GUI

/bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/watcher a -b2 (.*123/|.*456/|.*test$|data/|tv/)

 

Suggest you re-apply the setting in setting page and try again.

 

 

ok thanks a lot.

Now I get: /bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/watcher A -b2 (.*\.Recycle\.Bin/|Backup/|Downloads/|ISOs/|Kodi/|Nextcloud/|Recordings/)

 

So whats the best way to remove the hash values added by the automatic protection from the extended attributes of files from these folders now?

Just use the clear feature? Does the plugin need the stored hashing results on the flash drive for this? Because I deleted most of them.

[Marv]

So after rebooting my server I checked my excluded folders again and unfortunately I get:

 

/bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/watcher A -b2 Backup|Downloads,ISOs,Kodi,Nextcloud,Recordings

 

So the plugin still saves daily files to my flash drive containing file paths and hash values from all disks and shares.

Can someone please confirm if this is how its supposed to be and if the auto hashing ignores the excluded folders?

[bonienl]

What does inotifywait show?

ps -ef | grep inotifywait
...
inotifywait -dsrqo /var/run/hash.pipe -e close_write --exclude ^/mnt/disk[0-9]+/files|media --format %w%f /mnt/disk1 /mnt/disk2 ...

The --exclude option tells which folders to exclude from monitoring.

 

[Marv]

16 hours ago, bonienl said:

What does inotifywait show?

ps -ef | grep inotifywait
...
inotifywait -dsrqo /var/run/hash.pipe -e close_write --exclude ^/mnt/disk[0-9]+/files|media --format %w%f /mnt/disk1 /mnt/disk2 ...

The --exclude option tells which folders to exclude from monitoring.

 

 

 

Getting this:

inotifywait -dsrqo /var/run/hash.pipe -e close_write --exclude ^/mnt/disk[0-9]+/Backup|Downloads,ISOs,Kodi,Nextcloud,Recordings --format %w%f /mnt/disk1 /mnt/disk2 /mnt/disk3

 

Edited December 4, 2017 by Marv

[bonienl]

How are the custom folders entered on the settings page?

 

The above format isn't correct. Commas shouldn't be there in the exclude list. When you re-apply settings does it change?

 

[Marv]

As custom folder I just entered "Recycle.Bin".

The other folders are selected via the drop down menu.

 

When re-applying the settings the format changes as I posted here using "ps -elf | grep dynamix.file.integrity":

On 24.11.2017 at 9:47 PM, Marv said:

Now I get: /bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/watcher A -b2 (.*\.Recycle\.Bin/|Backup/|Downloads/|ISOs/|Kodi/|Nextcloud/|Recordings/)

 

Can't test right now with your inotifywait command above as I'm not at home. But should be the same I guess.

 

After rebooting the server I get commas again.

 

 

edit:

 

Output after re-applying:

 

inotifywait -dsrqo /var/run/hash.pipe -e close_write --exclude ^/mnt/disk[0-9]+/(.*\.Recycle\.Bin/|Backup/|Downloads/|ISOs/|Kodi/|Nextcloud/|Recordings/) --format %w%f /mnt/disk1 /mnt/disk2 /mnt/disk3

 

 

I have no idea why it doesn't keep the settings
 

Edited December 4, 2017 by Marv

[bonienl]

On 12/4/2017 at 12:18 PM, Marv said:

After rebooting the server I get commas again.

 

Okay, this is a bug which I need to fix. Meanwhile the workaround is to go to file integrity settings and disable/enable the automatic protection setting.

 

[Marv]

38 minutes ago, bonienl said:

 

Okay, this is a bug which I need to fix. Meanwhile the workaround is to go to file integrity settings and disable/enable the automatic protection setting.

 

 

okay thanks.

When I want to remove the hash values added by the automatic protection from the extended attributes from folders I wanted to be excluded actually...

Do I just need to use the clear feature after re-applying? Does the plugin need the stored hashing results on the flash drive for this? Because I deleted most of them.

 

Edited December 5, 2017 by Marv

[bonienl]

1 hour ago, Marv said:

When I want to remove the hash values added by the automatic protection from the extended attributes from folders I wanted to be excluded actually...

Do I just need to use the clear feature after re-applying? Does the plugin need the stored hashing results on the flash drive for this? Because I deleted most of them.

 

After you have set the correct "exclude" items you can do a "Clear" to remove the attributes from those excluded items, there is no dependency on the stored hashing results.

 

[bonienl]

An update is available which uses the correct "exclude" settings upon system startup.

 

[Marv]

On 6.12.2017 at 3:08 PM, bonienl said:

An update is available which uses the correct "exclude" settings upon system startup.

 

 

Just wanted to confirm that all is working for me now since the update. Thanks a lot!

[bonienl]

Hash keys are stored together with the files on the array. As long as you don't touch the array content no rebuild is required.

 

[bonienl]

I would exclude the apps share, files in here are regularly updated and adding file integrity has no advantages.

 

The steps to take:

1. Define the hash method to use

2. Define the shares, folders and files which you want to exclude

3. Run "Clear" this removes the attributes for files in the exclude list

4. Run "Build" this adds attributes to any missing files

5. Optionally you can export the file attributes to external files using "Export"

 

[bonienl]

It depends how you have moved the files in the past, extended attributes (which hold the hash key value) may get lost or not. It sounds in your case the extended file attributes are not present anymore, hence they need to be regenerated.

 

The dropdown menu only shows the top level folders (shares). Any folder names on a lower level can be entered manually.

 

[bonienl]

4 minutes ago, zin105 said:

Could it be that if files only exist on cache drive, it doesn't show it?

 

Yes, anything cache only is already excluded. The plugin operates on data disks only.