February 5, 201610 yr I would think that it is, but I can't find any info one way or the other. Guess I can try one of the other methods just to test.
February 5, 201610 yr I can confirm that exclusions work for me, but none of them have a dash in the share name. Since a dash is a special character in a regex, it is possible that there is a bug related to that.
February 5, 201610 yr I would think that it is, but I can't find any info one way or the other. Guess I can try one of the other methods just to test. See this post by Squid. It might not be what is going on with you... but it can't hurt to look into it.
February 5, 201610 yr I would think that it is, but I can't find any info one way or the other. Guess I can try one of the other methods just to test. See this post by Squid. It might not be what is going on with you... but it can't hurt to look into it. Thanks. I'll run the command to test later.
February 5, 201610 yr I would think that it is, but I can't find any info one way or the other. Guess I can try one of the other methods just to test. See this post by Squid. It might not be what is going on with you... but it can't hurt to look into it. Thanks. I'll run the command to test later. the path isn't going to be the same as what I posted
February 5, 201610 yr the path isn't going to be the same as what I posted I would just need to replace /checksum/ with /fileintegrity/ right? Might have remembered the exact syntax wrong. Regardless, could I still just look at one of the exported hash files?
February 6, 201610 yr I would think that it is, but I can't find any info one way or the other. Guess I can try one of the other methods just to test. See this post by Squid. It might not be what is going on with you... but it can't hurt to look into it. I have removed the dashes from the names of the user shares. Doesn't appear to have had any effect. The exported hash files all have hashes in them so looks like Blake2 is working properly. They also include hashes for excluded shares so those shares weren't really excluded.
February 8, 201610 yr I would think that it is, but I can't find any info one way or the other. Guess I can try one of the other methods just to test. See this post by Squid. It might not be what is going on with you... but it can't hurt to look into it. I have removed the dashes from the names of the user shares. Doesn't appear to have had any effect. The exported hash files all have hashes in them so looks like Blake2 is working properly. They also include hashes for excluded shares so those shares weren't really excluded. Looks like I finally got this plugin to work. Changed to MD5 and now exclusions are working. I notice that nothing is being logged though??? Maybe only errors are logged??? It would also be nice if the recycle bin could be excluded. Not really a problem, just seems like a waste.
February 9, 201610 yr I have had this running for several weeks now. Yesterday I received a notification of unRAID file corruption: 07-02-2016 08:39 Notice [KIM] - bunker command Found 2 files with SHA256 hash key corruption However expanding message from the archived notifications page, it only lists one file as found not matching the SHA256 hash. SHA256 hash key mismatch, */mnt/disk1/downloads/Hash_Documentation/Hash process/IMG_2456.JPG is corrupted Does this archived notification only show 1 of the files corrupted? Here is the related log file from the plugins directory disks.export.20160208.bad.hash Shouldn't this stuff match? SHA256 hash key mismatch, /mnt/disk2/Documents/Design Stuff/Images/Backdrops?Flooring/Design backdrops/Backdrops with logo/psF601.tmp is corrupted SHA256 hash key mismatch (updated), /mnt/disk2/Documents/Design Stuff/Images/Backdrops?Flooring/Design backdrops/Backdrops with logo/ps9C.tmp was modified SHA256 hash key mismatch, /mnt/disk2/Documents/Design Stuff/Images/Backdrops?Flooring/Design backdrops/Backdrops with logo/psB9CF.tmp is corrupted SHA256 hash key mismatch, /mnt/disk2/Documents/Design Stuff/Images/Backdrops?Flooring/Design
February 9, 201610 yr I was doing a defrag on one of my disks which ended up locking up my unRAID box and I had to hard reset it. I'd like to use this plugin to confirm that all the data is good on the disk. Do I just do the "check" command on the disk? What if I've updated the build and exported in the meantime, would that have exported a new hash with bad data?
February 12, 201610 yr I would think that it is, but I can't find any info one way or the other. Guess I can try one of the other methods just to test. See this post by Squid. It might not be what is going on with you... but it can't hurt to look into it. I have removed the dashes from the names of the user shares. Doesn't appear to have had any effect. The exported hash files all have hashes in them so looks like Blake2 is working properly. They also include hashes for excluded shares so those shares weren't really excluded. Looks like I finally got this plugin to work. Changed to MD5 and now exclusions are working. I notice that nothing is being logged though??? Maybe only errors are logged??? It would also be nice if the recycle bin could be excluded. Not really a problem, just seems like a waste. Guess I spoke too soon. The next scheduled check was yesterday and I got flooded with notifications of bad hashes in the excluded folders. Tried switching to sha2 and re-ran the check this morning with the same result. Checked the log, but this plugin doesn't log anything other than the bad hash warning. Not sure what else to try?
February 12, 201610 yr Quick question... Can this plugin be used after bunker has already created hashes and used the file extended attributes?
February 14, 201610 yr I can confirm that exclusions work for me, but none of them have a dash in the share name. Since a dash is a special character in a regex, it is possible that there is a bug related to that. Hmm, looks like I'm having problems with exclusions now too. Here is my dynamix.file.integrity.cfg file: disks="disk1,disk2,disk3" service="1" method="-md5" cmd="A" exclude=".fsr,CrashplanTower,VMISO,cache_backup" schedule="2" parity="1" notify="-n" log="-L -f" disk1-1="on" disk2-2="on" disk3-3="on" priority="-n19,-c2 -n7" day="0" hour="0" min="0" But this showed up in my log today, about one of the excluded directories: Feb 14 06:19:43 Tower bunker: warning: MD5 hash key mismatch (updated), /mnt/disk1/CrashplanTower/470662345684159084/cpbf0000000000000000000/cpbmf was modified <snip> Feb 14 06:36:14 Tower bunker: warning: MD5 hash key mismatch (updated), /mnt/disk1/CrashplanTower/470662345684159084/cpbf0000000000003747658/cpbdf was modified Feb 14 06:36:14 Tower bunker: verified 661488 files from /mnt/disk1 with mask *. Found: 75 mismatches (updated), 0 corruptions. Duration: 06:36:13 Average speed: 47.5 MB/s Since it just started, I wonder if the problem is that ".fsr" is excluded but that directory only exists during a defrag.
February 16, 201610 yr Author I can confirm that exclusions work for me, but none of them have a dash in the share name. Since a dash is a special character in a regex, it is possible that there is a bug related to that. Hmm, looks like I'm having problems with exclusions now too. Here is my dynamix.file.integrity.cfg file: disks="disk1,disk2,disk3" service="1" method="-md5" cmd="A" exclude=".fsr,CrashplanTower,VMISO,cache_backup" schedule="2" parity="1" notify="-n" log="-L -f" disk1-1="on" disk2-2="on" disk3-3="on" priority="-n19,-c2 -n7" day="0" hour="0" min="0" But this showed up in my log today, about one of the excluded directories: Feb 14 06:19:43 Tower bunker: warning: MD5 hash key mismatch (updated), /mnt/disk1/CrashplanTower/470662345684159084/cpbf0000000000000000000/cpbmf was modified <snip> Feb 14 06:36:14 Tower bunker: warning: MD5 hash key mismatch (updated), /mnt/disk1/CrashplanTower/470662345684159084/cpbf0000000000003747658/cpbdf was modified Feb 14 06:36:14 Tower bunker: verified 661488 files from /mnt/disk1 with mask *. Found: 75 mismatches (updated), 0 corruptions. Duration: 06:36:13 Average speed: 47.5 MB/s Since it just started, I wonder if the problem is that ".fsr" is excluded but that directory only exists during a defrag. Can you post the output of command: ps -ef | grep inotify
February 16, 201610 yr I can confirm that exclusions work for me, but none of them have a dash in the share name. Since a dash is a special character in a regex, it is possible that there is a bug related to that. Hmm, looks like I'm having problems with exclusions now too. Here is my dynamix.file.integrity.cfg file: disks="disk1,disk2,disk3" service="1" method="-md5" cmd="A" exclude=".fsr,CrashplanTower,VMISO,cache_backup" schedule="2" parity="1" notify="-n" log="-L -f" disk1-1="on" disk2-2="on" disk3-3="on" priority="-n19,-c2 -n7" day="0" hour="0" min="0" But this showed up in my log today, about one of the excluded directories: Feb 14 06:19:43 Tower bunker: warning: MD5 hash key mismatch (updated), /mnt/disk1/CrashplanTower/470662345684159084/cpbf0000000000000000000/cpbmf was modified <snip> Feb 14 06:36:14 Tower bunker: warning: MD5 hash key mismatch (updated), /mnt/disk1/CrashplanTower/470662345684159084/cpbf0000000000003747658/cpbdf was modified Feb 14 06:36:14 Tower bunker: verified 661488 files from /mnt/disk1 with mask *. Found: 75 mismatches (updated), 0 corruptions. Duration: 06:36:13 Average speed: 47.5 MB/s Since it just started, I wonder if the problem is that ".fsr" is excluded but that directory only exists during a defrag. Can you post the output of command: ps -ef | grep inotify Sure thing! Thanks for looking: root@Tower:~# ps -ef | grep inotify root 5522 5288 0 13:26 pts/0 00:00:00 grep inotify root 6484 1 0 Feb09 ? 00:00:01 inotifywait -mrq --exclude (^/mnt/disk[0-9]+/(\S+\.img$|.fsr|CrashplanTower|VMISO|cache_backup)/|\.AppleDB|\.DS_Store) --format %w%f -e close_write /mnt/disk1 /mnt/disk2 /mnt/disk3 Hmm, maybe it should be \.fsr instead of .fsr? Any chance you might be willing to exclude that by default, along with the Apple stuff? https://lime-technology.com/forum/index.php?topic=44989.msg442250#msg442250
February 16, 201610 yr Can you post the output of command: ps -ef | grep inotify I know this wasn't really directed at me but I thought I'd run it in case it might help with fixing the issue with exclusions. Brunnhilde login: root Password: Linux 4.1.17-unRAID. Last login: Mon Feb 15 20:07:47 -0500 2016 on /dev/pts/0 from 10.0.1.190. root@Brunnhilde:~# ps -ef | grep inotify root 21768 21082 0 17:03 pts/0 00:00:00 grep inotify root 32233 1 0 Feb13 ? 00:00:31 inotifywait -mrq --exclude (^/mnt/disk[0-9]+/(\S+\.img$|TM_Jasper|TM_MBP|TM_Maggie|TM_Mini)/|\.AppleDB|\.DS_Store) --format %w%f -e close_write /mnt/disk1 /mnt/disk2 /mnt/disk3 /mnt/disk4 /mnt/disk5 /mnt/disk6 /mnt/disk7 /mnt/disk8 /mnt/disk9 root@Brunnhilde:~# I've also attached one of the exported bad hash files. Not sure it really shows much of anything other than the fact that the excluded folder was still checked. disk9.export.20160213.bad.hash.zip
February 16, 201610 yr All of the inotifywait exclude examples I can find have the regex wrapped in single quotes. Maybe that would help?
February 17, 201610 yr Author All of the inotifywait exclude examples I can find have the regex wrapped in single quotes. Maybe that would help? The original string is between quotes but these are not shown with the 'ps' command. I'll try to reproduce the issue, but so far the exclude works as expected on my system.
February 18, 201610 yr So from what I can tell, once a file *somehow* gets bunker's file attributes added to it, that file will be processed by bunker's export and verify routines, even if the directory it is in has been excluded. To prevent this, I think we need a "Clean" option that loops through the files in the currently excluded directories and strips off any file attributes that bunker added in the past. It could either be run manually by the user or bunker could run it automatically before any export or verify routine. As to how the wrong files got the attributes in the first place, I can think of two possibilities: 1) If there are special characters in the share name (such as periods or dashes), they could confuse the regex parsers. Ideally the file integrity plugin would escape those characters before passing the share names to inotifywait or bunker. 2) It is also possible that user did a build or enabled the watch feature before they set the exclusions. Even though I really don't think I did this I'll admit it is a possibility. Either way, the "Clean" routine would fix it.
February 18, 201610 yr Author So from what I can tell, once a file *somehow* gets bunker's file attributes added to it, that file will be processed by bunker's export and verify routines, even if the directory it is in has been excluded. To prevent this, I think we need a "Clean" option that loops through the files in the currently excluded directories and strips off any file attributes that bunker added in the past. It could either be run manually by the user or bunker could run it automatically before any export or verify routine. As to how the wrong files got the attributes in the first place, I can think of two possibilities: 1) If there are special characters in the share name (such as periods or dashes), they could confuse the regex parsers. Ideally the file integrity plugin would escape those characters before passing the share names to inotifywait or bunker. 2) It is also possible that user did a build or enabled the watch feature before they set the exclusions. Even though I really don't think I did this I'll admit it is a possibility. Either way, the "Clean" routine would fix it. Good points. I'll work on an update to address these.
February 18, 201610 yr So from what I can tell, once a file *somehow* gets bunker's file attributes added to it, that file will be processed by bunker's export and verify routines, even if the directory it is in has been excluded. To prevent this, I think we need a "Clean" option that loops through the files in the currently excluded directories and strips off any file attributes that bunker added in the past. It could either be run manually by the user or bunker could run it automatically before any export or verify routine. As to how the wrong files got the attributes in the first place, I can think of two possibilities: 1) If there are special characters in the share name (such as periods or dashes), they could confuse the regex parsers. Ideally the file integrity plugin would escape those characters before passing the share names to inotifywait or bunker. 2) It is also possible that user did a build or enabled the watch feature before they set the exclusions. Even though I really don't think I did this I'll admit it is a possibility. Either way, the "Clean" routine would fix it. I know this definatly applies to my case. I originally included the TimeMachine backups because the incompatibility hadn't been reported yet. Afterwards, I was not able to exclude the TM shares.
February 18, 201610 yr Author I've created an updated version. Please use 'Check for Updates' in the plugin manager to get it. This release has the following changes: - fixed regular expression when share and file names have 'special' characters - added *Clear* button, which allows to remove extended attributes from excluded folders and files only - added *Custom* field, which allows to enter custom file names and/or file extensions - make exclusion of Apple metadata (netatalk) user selectable IT IS MANDATORY TO RE-APPLY THE SETTINGS AFTER THIS UPDATE
February 18, 201610 yr I've created an updated version. Please use 'Check for Updates' in the plugin manager to get it. This release has the following changes: - fixed regular expression when share and file names have 'special' characters - added *Clear* button, which allows to remove extended attributes from excluded folders and files only - added *Custom* field, which allows to enter custom file names and/or file extensions - make exclusion of Apple metadata (netatalk) user selectable IT IS MANDATORY TO RE-APPLY THE SETTINGS AFTER THIS UPDATE That was quick. I'll give it a shot later tonight. Thanks. Question I've got VFS Recycle plugin installed which creates a folder (.Recycle.Bin) when I delete files. Can I enter .Recyle.Bin in the custom field to exclude these temp folders? I haven't seen them appear as shares. Doesn't really create an issue one way or the other. Just seems a waste to check files waiting to be deleted.
February 18, 201610 yr Author Question I've got VFS Recycle plugin installed which creates a folder (.Recycle.Bin) when I delete files. Can I enter .Recyle.Bin in the custom field to exclude these temp folders? I haven't seen them appear as shares. Doesn't really create an issue one way or the other. Just seems a waste to check files waiting to be deleted. I've made version 2016.02.18b which allows to exclude the folder. You need to enter its name .Recycle.Bin in the custom field.
February 18, 201610 yr I've made version 2016.02.18b which allows to exclude the folder. You need to enter its name .Recycle.Bin in the custom field. You Rock
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.