Dynamix File Integrity plugin


bonienl

Recommended Posts

I would think that it is, but I can't find any info one way or the other. Guess I can try one of the other methods just to test.

 

See this post by Squid.  It might not be what is going on with you... but it can't hurt to look into it.

 

I have removed the dashes from the names of the user shares. Doesn't appear to have had any effect. The exported hash files all have hashes in them so looks like Blake2 is working properly. They also include hashes for excluded shares so those shares weren't really excluded.

Link to comment

I would think that it is, but I can't find any info one way or the other. Guess I can try one of the other methods just to test.

 

See this post by Squid.  It might not be what is going on with you... but it can't hurt to look into it.

 

I have removed the dashes from the names of the user shares. Doesn't appear to have had any effect. The exported hash files all have hashes in them so looks like Blake2 is working properly. They also include hashes for excluded shares so those shares weren't really excluded.

Looks like I finally got this plugin to work. Changed to MD5 and now exclusions are working. I notice that nothing is being logged though??? Maybe only errors are logged??? It would also be nice if the recycle bin could be excluded. Not really a problem, just seems like a waste.

Link to comment

I have had this running for several weeks now.  Yesterday I received a notification of

 

unRAID file corruption: 07-02-2016 08:39
Notice [KIM] - bunker command
Found 2 files with SHA256 hash key corruption

 

However expanding message from the archived notifications page, it only lists one file as found not matching the SHA256 hash.

 

SHA256 hash key mismatch, */mnt/disk1/downloads/Hash_Documentation/Hash process/IMG_2456.JPG is corrupted

Does this archived notification only show 1 of the files corrupted?

 

Here is the related log file from the plugins directory disks.export.20160208.bad.hash

 

Shouldn't this stuff match?

 

SHA256 hash key mismatch, /mnt/disk2/Documents/Design Stuff/Images/Backdrops?Flooring/Design backdrops/Backdrops with logo/psF601.tmp is corrupted
SHA256 hash key mismatch (updated), /mnt/disk2/Documents/Design Stuff/Images/Backdrops?Flooring/Design backdrops/Backdrops with logo/ps9C.tmp was modified
SHA256 hash key mismatch, /mnt/disk2/Documents/Design Stuff/Images/Backdrops?Flooring/Design backdrops/Backdrops with logo/psB9CF.tmp is corrupted
SHA256 hash key mismatch, /mnt/disk2/Documents/Design Stuff/Images/Backdrops?Flooring/Design 

Link to comment

I was doing a defrag on one of my disks which ended up locking up my unRAID box and I had to hard reset it.  I'd like to use this plugin to confirm that all the data is good on the disk.  Do I just do the "check" command on the disk?  What if I've updated the build and exported in the meantime, would that have exported a new hash with bad data? 

Link to comment

I would think that it is, but I can't find any info one way or the other. Guess I can try one of the other methods just to test.

 

See this post by Squid.  It might not be what is going on with you... but it can't hurt to look into it.

 

I have removed the dashes from the names of the user shares. Doesn't appear to have had any effect. The exported hash files all have hashes in them so looks like Blake2 is working properly. They also include hashes for excluded shares so those shares weren't really excluded.

Looks like I finally got this plugin to work. Changed to MD5 and now exclusions are working. I notice that nothing is being logged though??? Maybe only errors are logged??? It would also be nice if the recycle bin could be excluded. Not really a problem, just seems like a waste.

Guess I spoke too soon. The next scheduled check was yesterday and I got flooded with notifications of bad hashes in the excluded folders. Tried switching to sha2 and re-ran the check this morning with the same result. Checked the log, but this plugin doesn't log anything other than the bad hash warning. Not sure what else to try?

Link to comment

I can confirm that exclusions work for me, but none of them have a dash in the share name.  Since a dash is a special character in a regex, it is possible that there is a bug related to that.

 

Hmm, looks like I'm having problems with exclusions now too.

 

Here is my dynamix.file.integrity.cfg file:

 

disks="disk1,disk2,disk3"
service="1"
method="-md5"
cmd="A"
exclude=".fsr,CrashplanTower,VMISO,cache_backup"
schedule="2"
parity="1"
notify="-n"
log="-L -f"
disk1-1="on"
disk2-2="on"
disk3-3="on"
priority="-n19,-c2 -n7"
day="0"
hour="0"
min="0"

But this showed up in my log today, about one of the excluded directories:

 

Feb 14 06:19:43 Tower bunker: warning: MD5 hash key mismatch (updated), /mnt/disk1/CrashplanTower/470662345684159084/cpbf0000000000000000000/cpbmf was modified
<snip>
Feb 14 06:36:14 Tower bunker: warning: MD5 hash key mismatch (updated), /mnt/disk1/CrashplanTower/470662345684159084/cpbf0000000000003747658/cpbdf was modified
Feb 14 06:36:14 Tower bunker: verified 661488 files from /mnt/disk1 with mask *. Found: 75 mismatches (updated), 0 corruptions. Duration: 06:36:13 Average speed: 47.5 MB/s

 

Since it just started, I wonder if the problem is that ".fsr" is excluded but that directory only exists during a defrag.

Link to comment

I can confirm that exclusions work for me, but none of them have a dash in the share name.  Since a dash is a special character in a regex, it is possible that there is a bug related to that.

 

Hmm, looks like I'm having problems with exclusions now too.

 

Here is my dynamix.file.integrity.cfg file:

 

disks="disk1,disk2,disk3"
service="1"
method="-md5"
cmd="A"
exclude=".fsr,CrashplanTower,VMISO,cache_backup"
schedule="2"
parity="1"
notify="-n"
log="-L -f"
disk1-1="on"
disk2-2="on"
disk3-3="on"
priority="-n19,-c2 -n7"
day="0"
hour="0"
min="0"

But this showed up in my log today, about one of the excluded directories:

 

Feb 14 06:19:43 Tower bunker: warning: MD5 hash key mismatch (updated), /mnt/disk1/CrashplanTower/470662345684159084/cpbf0000000000000000000/cpbmf was modified
<snip>
Feb 14 06:36:14 Tower bunker: warning: MD5 hash key mismatch (updated), /mnt/disk1/CrashplanTower/470662345684159084/cpbf0000000000003747658/cpbdf was modified
Feb 14 06:36:14 Tower bunker: verified 661488 files from /mnt/disk1 with mask *. Found: 75 mismatches (updated), 0 corruptions. Duration: 06:36:13 Average speed: 47.5 MB/s

 

Since it just started, I wonder if the problem is that ".fsr" is excluded but that directory only exists during a defrag.

 

Can you post the output of command:

 

ps -ef | grep inotify

 

Link to comment

I can confirm that exclusions work for me, but none of them have a dash in the share name.  Since a dash is a special character in a regex, it is possible that there is a bug related to that.

 

Hmm, looks like I'm having problems with exclusions now too.

 

Here is my dynamix.file.integrity.cfg file:

 

disks="disk1,disk2,disk3"
service="1"
method="-md5"
cmd="A"
exclude=".fsr,CrashplanTower,VMISO,cache_backup"
schedule="2"
parity="1"
notify="-n"
log="-L -f"
disk1-1="on"
disk2-2="on"
disk3-3="on"
priority="-n19,-c2 -n7"
day="0"
hour="0"
min="0"

But this showed up in my log today, about one of the excluded directories:

 

Feb 14 06:19:43 Tower bunker: warning: MD5 hash key mismatch (updated), /mnt/disk1/CrashplanTower/470662345684159084/cpbf0000000000000000000/cpbmf was modified
<snip>
Feb 14 06:36:14 Tower bunker: warning: MD5 hash key mismatch (updated), /mnt/disk1/CrashplanTower/470662345684159084/cpbf0000000000003747658/cpbdf was modified
Feb 14 06:36:14 Tower bunker: verified 661488 files from /mnt/disk1 with mask *. Found: 75 mismatches (updated), 0 corruptions. Duration: 06:36:13 Average speed: 47.5 MB/s

 

Since it just started, I wonder if the problem is that ".fsr" is excluded but that directory only exists during a defrag.

 

Can you post the output of command:

 

ps -ef | grep inotify

 

Sure thing!  Thanks for looking:

 

root@Tower:~# ps -ef | grep inotify
root      5522  5288  0 13:26 pts/0    00:00:00 grep inotify
root      6484     1  0 Feb09 ?        00:00:01 inotifywait -mrq --exclude (^/mnt/disk[0-9]+/(\S+\.img$|.fsr|CrashplanTower|VMISO|cache_backup)/|\.AppleDB|\.DS_Store) --format %w%f -e close_write /mnt/disk1 /mnt/disk2 /mnt/disk3

Hmm, maybe it should be \.fsr instead of .fsr?

 

Any chance you might be willing to exclude that by default, along with the Apple stuff?

  https://lime-technology.com/forum/index.php?topic=44989.msg442250#msg442250

Link to comment

Can you post the output of command:

 

ps -ef | grep inotify

I know this wasn't really directed at me but I thought I'd run it in case it might help with fixing the issue with exclusions.

Brunnhilde login: root
Password: 
Linux 4.1.17-unRAID.
Last login: Mon Feb 15 20:07:47 -0500 2016 on /dev/pts/0 from 10.0.1.190.
root@Brunnhilde:~# ps -ef | grep inotify
root     21768 21082  0 17:03 pts/0    00:00:00 grep inotify
root     32233     1  0 Feb13 ?        00:00:31 inotifywait -mrq --exclude (^/mnt/disk[0-9]+/(\S+\.img$|TM_Jasper|TM_MBP|TM_Maggie|TM_Mini)/|\.AppleDB|\.DS_Store) --format %w%f -e close_write /mnt/disk1 /mnt/disk2 /mnt/disk3 /mnt/disk4 /mnt/disk5 /mnt/disk6 /mnt/disk7 /mnt/disk8 /mnt/disk9
root@Brunnhilde:~# 

I've also attached one of the exported bad hash files. Not sure it really shows much of anything other than the fact that the excluded folder was still checked.

disk9.export.20160213.bad.hash.zip

Link to comment

All of the inotifywait exclude examples I can find have the regex wrapped in single quotes.  Maybe that would help?

 

The original string is between quotes but these are not shown with the 'ps' command.

 

I'll try to reproduce the issue, but so far the exclude works as expected on my system.

 

Link to comment

So from what I can tell, once a file *somehow* gets bunker's file attributes added to it, that file will be processed by bunker's export and verify routines, even if the directory it is in has been excluded.

 

To prevent this, I think we need a "Clean" option that loops through the files in the currently excluded directories and strips off any file attributes that bunker added in the past.  It could either be run manually by the user or bunker could run it automatically before any export or verify routine.

 

As to how the wrong files got the attributes in the first place, I can think of two possibilities:

 

1) If there are special characters in the share name (such as periods or dashes), they could confuse the regex parsers.  Ideally the file integrity plugin would escape those characters before passing the share names to inotifywait or bunker.

 

2) It is also possible that user did a build or enabled the watch feature before they set the exclusions.  Even though I really don't think I did this :) I'll admit it is a possibility.

 

Either way, the "Clean" routine would fix it.

Link to comment

So from what I can tell, once a file *somehow* gets bunker's file attributes added to it, that file will be processed by bunker's export and verify routines, even if the directory it is in has been excluded.

 

To prevent this, I think we need a "Clean" option that loops through the files in the currently excluded directories and strips off any file attributes that bunker added in the past.  It could either be run manually by the user or bunker could run it automatically before any export or verify routine.

 

As to how the wrong files got the attributes in the first place, I can think of two possibilities:

 

1) If there are special characters in the share name (such as periods or dashes), they could confuse the regex parsers.  Ideally the file integrity plugin would escape those characters before passing the share names to inotifywait or bunker.

 

2) It is also possible that user did a build or enabled the watch feature before they set the exclusions.  Even though I really don't think I did this :) I'll admit it is a possibility.

 

Either way, the "Clean" routine would fix it.

 

Good points. I'll work on an update to address these.

 

Link to comment

So from what I can tell, once a file *somehow* gets bunker's file attributes added to it, that file will be processed by bunker's export and verify routines, even if the directory it is in has been excluded.

 

To prevent this, I think we need a "Clean" option that loops through the files in the currently excluded directories and strips off any file attributes that bunker added in the past.  It could either be run manually by the user or bunker could run it automatically before any export or verify routine.

 

As to how the wrong files got the attributes in the first place, I can think of two possibilities:

 

1) If there are special characters in the share name (such as periods or dashes), they could confuse the regex parsers.  Ideally the file integrity plugin would escape those characters before passing the share names to inotifywait or bunker.

 

2) It is also possible that user did a build or enabled the watch feature before they set the exclusions.  Even though I really don't think I did this :) I'll admit it is a possibility.

 

Either way, the "Clean" routine would fix it.

I know this definatly applies to my case. I originally included the TimeMachine backups because the incompatibility hadn't been reported yet. Afterwards, I was not able to exclude the TM shares.

Link to comment

I've created an updated version. Please use 'Check for Updates' in the plugin manager to get it.

 

This release has the following changes:

 

- fixed regular expression when share and file names have 'special' characters

- added *Clear* button, which allows to remove extended attributes from excluded folders and files only

- added *Custom* field, which allows to enter custom file names and/or file extensions

- make exclusion of Apple metadata (netatalk) user selectable

 

IT IS MANDATORY TO RE-APPLY THE SETTINGS AFTER THIS UPDATE

 

Link to comment

I've created an updated version. Please use 'Check for Updates' in the plugin manager to get it.

 

This release has the following changes:

 

- fixed regular expression when share and file names have 'special' characters

- added *Clear* button, which allows to remove extended attributes from excluded folders and files only

- added *Custom* field, which allows to enter custom file names and/or file extensions

- make exclusion of Apple metadata (netatalk) user selectable

 

IT IS MANDATORY TO RE-APPLY THE SETTINGS AFTER THIS UPDATE

 

That was quick. I'll give it a shot later tonight. Thanks.

 

Question

I've got VFS Recycle plugin installed which creates a folder (.Recycle.Bin) when I delete files. Can I enter .Recyle.Bin in the custom field to exclude these temp folders? I haven't seen them appear as shares. Doesn't really create an issue one way or the other. Just seems a waste to check files waiting to be deleted.

Link to comment

Question

I've got VFS Recycle plugin installed which creates a folder (.Recycle.Bin) when I delete files. Can I enter .Recyle.Bin in the custom field to exclude these temp folders? I haven't seen them appear as shares. Doesn't really create an issue one way or the other. Just seems a waste to check files waiting to be deleted.

 

I've made version 2016.02.18b which allows to exclude the folder. You need to enter its name

.Recycle.Bin

in the custom field.

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.