wgstarks Posted February 5, 2016 Share Posted February 5, 2016 I would think that it is, but I can't find any info one way or the other. Guess I can try one of the other methods just to test. Quote Link to comment
ljm42 Posted February 5, 2016 Share Posted February 5, 2016 I can confirm that exclusions work for me, but none of them have a dash in the share name. Since a dash is a special character in a regex, it is possible that there is a bug related to that. Quote Link to comment
gundamguy Posted February 5, 2016 Share Posted February 5, 2016 I would think that it is, but I can't find any info one way or the other. Guess I can try one of the other methods just to test. See this post by Squid. It might not be what is going on with you... but it can't hurt to look into it. Quote Link to comment
wgstarks Posted February 5, 2016 Share Posted February 5, 2016 I would think that it is, but I can't find any info one way or the other. Guess I can try one of the other methods just to test. See this post by Squid. It might not be what is going on with you... but it can't hurt to look into it. Thanks. I'll run the command to test later. Quote Link to comment
Squid Posted February 5, 2016 Share Posted February 5, 2016 I would think that it is, but I can't find any info one way or the other. Guess I can try one of the other methods just to test. See this post by Squid. It might not be what is going on with you... but it can't hurt to look into it. Thanks. I'll run the command to test later. the path isn't going to be the same as what I posted Quote Link to comment
wgstarks Posted February 5, 2016 Share Posted February 5, 2016 the path isn't going to be the same as what I posted I would just need to replace /checksum/ with /fileintegrity/ right? Might have remembered the exact syntax wrong. Regardless, could I still just look at one of the exported hash files? Quote Link to comment
wgstarks Posted February 6, 2016 Share Posted February 6, 2016 I would think that it is, but I can't find any info one way or the other. Guess I can try one of the other methods just to test. See this post by Squid. It might not be what is going on with you... but it can't hurt to look into it. I have removed the dashes from the names of the user shares. Doesn't appear to have had any effect. The exported hash files all have hashes in them so looks like Blake2 is working properly. They also include hashes for excluded shares so those shares weren't really excluded. Quote Link to comment
wgstarks Posted February 8, 2016 Share Posted February 8, 2016 I would think that it is, but I can't find any info one way or the other. Guess I can try one of the other methods just to test. See this post by Squid. It might not be what is going on with you... but it can't hurt to look into it. I have removed the dashes from the names of the user shares. Doesn't appear to have had any effect. The exported hash files all have hashes in them so looks like Blake2 is working properly. They also include hashes for excluded shares so those shares weren't really excluded. Looks like I finally got this plugin to work. Changed to MD5 and now exclusions are working. I notice that nothing is being logged though??? Maybe only errors are logged??? It would also be nice if the recycle bin could be excluded. Not really a problem, just seems like a waste. Quote Link to comment
tr0910 Posted February 9, 2016 Share Posted February 9, 2016 I have had this running for several weeks now. Yesterday I received a notification of unRAID file corruption: 07-02-2016 08:39 Notice [KIM] - bunker command Found 2 files with SHA256 hash key corruption However expanding message from the archived notifications page, it only lists one file as found not matching the SHA256 hash. SHA256 hash key mismatch, */mnt/disk1/downloads/Hash_Documentation/Hash process/IMG_2456.JPG is corrupted Does this archived notification only show 1 of the files corrupted? Here is the related log file from the plugins directory disks.export.20160208.bad.hash Shouldn't this stuff match? SHA256 hash key mismatch, /mnt/disk2/Documents/Design Stuff/Images/Backdrops?Flooring/Design backdrops/Backdrops with logo/psF601.tmp is corrupted SHA256 hash key mismatch (updated), /mnt/disk2/Documents/Design Stuff/Images/Backdrops?Flooring/Design backdrops/Backdrops with logo/ps9C.tmp was modified SHA256 hash key mismatch, /mnt/disk2/Documents/Design Stuff/Images/Backdrops?Flooring/Design backdrops/Backdrops with logo/psB9CF.tmp is corrupted SHA256 hash key mismatch, /mnt/disk2/Documents/Design Stuff/Images/Backdrops?Flooring/Design Quote Link to comment
Adam64 Posted February 9, 2016 Share Posted February 9, 2016 I was doing a defrag on one of my disks which ended up locking up my unRAID box and I had to hard reset it. I'd like to use this plugin to confirm that all the data is good on the disk. Do I just do the "check" command on the disk? What if I've updated the build and exported in the meantime, would that have exported a new hash with bad data? Quote Link to comment
wgstarks Posted February 12, 2016 Share Posted February 12, 2016 I would think that it is, but I can't find any info one way or the other. Guess I can try one of the other methods just to test. See this post by Squid. It might not be what is going on with you... but it can't hurt to look into it. I have removed the dashes from the names of the user shares. Doesn't appear to have had any effect. The exported hash files all have hashes in them so looks like Blake2 is working properly. They also include hashes for excluded shares so those shares weren't really excluded. Looks like I finally got this plugin to work. Changed to MD5 and now exclusions are working. I notice that nothing is being logged though??? Maybe only errors are logged??? It would also be nice if the recycle bin could be excluded. Not really a problem, just seems like a waste. Guess I spoke too soon. The next scheduled check was yesterday and I got flooded with notifications of bad hashes in the excluded folders. Tried switching to sha2 and re-ran the check this morning with the same result. Checked the log, but this plugin doesn't log anything other than the bad hash warning. Not sure what else to try? Quote Link to comment
TheDragon Posted February 12, 2016 Share Posted February 12, 2016 Quick question... Can this plugin be used after bunker has already created hashes and used the file extended attributes? Quote Link to comment
ljm42 Posted February 14, 2016 Share Posted February 14, 2016 I can confirm that exclusions work for me, but none of them have a dash in the share name. Since a dash is a special character in a regex, it is possible that there is a bug related to that. Hmm, looks like I'm having problems with exclusions now too. Here is my dynamix.file.integrity.cfg file: disks="disk1,disk2,disk3" service="1" method="-md5" cmd="A" exclude=".fsr,CrashplanTower,VMISO,cache_backup" schedule="2" parity="1" notify="-n" log="-L -f" disk1-1="on" disk2-2="on" disk3-3="on" priority="-n19,-c2 -n7" day="0" hour="0" min="0" But this showed up in my log today, about one of the excluded directories: Feb 14 06:19:43 Tower bunker: warning: MD5 hash key mismatch (updated), /mnt/disk1/CrashplanTower/470662345684159084/cpbf0000000000000000000/cpbmf was modified <snip> Feb 14 06:36:14 Tower bunker: warning: MD5 hash key mismatch (updated), /mnt/disk1/CrashplanTower/470662345684159084/cpbf0000000000003747658/cpbdf was modified Feb 14 06:36:14 Tower bunker: verified 661488 files from /mnt/disk1 with mask *. Found: 75 mismatches (updated), 0 corruptions. Duration: 06:36:13 Average speed: 47.5 MB/s Since it just started, I wonder if the problem is that ".fsr" is excluded but that directory only exists during a defrag. Quote Link to comment
bonienl Posted February 16, 2016 Author Share Posted February 16, 2016 I can confirm that exclusions work for me, but none of them have a dash in the share name. Since a dash is a special character in a regex, it is possible that there is a bug related to that. Hmm, looks like I'm having problems with exclusions now too. Here is my dynamix.file.integrity.cfg file: disks="disk1,disk2,disk3" service="1" method="-md5" cmd="A" exclude=".fsr,CrashplanTower,VMISO,cache_backup" schedule="2" parity="1" notify="-n" log="-L -f" disk1-1="on" disk2-2="on" disk3-3="on" priority="-n19,-c2 -n7" day="0" hour="0" min="0" But this showed up in my log today, about one of the excluded directories: Feb 14 06:19:43 Tower bunker: warning: MD5 hash key mismatch (updated), /mnt/disk1/CrashplanTower/470662345684159084/cpbf0000000000000000000/cpbmf was modified <snip> Feb 14 06:36:14 Tower bunker: warning: MD5 hash key mismatch (updated), /mnt/disk1/CrashplanTower/470662345684159084/cpbf0000000000003747658/cpbdf was modified Feb 14 06:36:14 Tower bunker: verified 661488 files from /mnt/disk1 with mask *. Found: 75 mismatches (updated), 0 corruptions. Duration: 06:36:13 Average speed: 47.5 MB/s Since it just started, I wonder if the problem is that ".fsr" is excluded but that directory only exists during a defrag. Can you post the output of command: ps -ef | grep inotify Quote Link to comment
ljm42 Posted February 16, 2016 Share Posted February 16, 2016 I can confirm that exclusions work for me, but none of them have a dash in the share name. Since a dash is a special character in a regex, it is possible that there is a bug related to that. Hmm, looks like I'm having problems with exclusions now too. Here is my dynamix.file.integrity.cfg file: disks="disk1,disk2,disk3" service="1" method="-md5" cmd="A" exclude=".fsr,CrashplanTower,VMISO,cache_backup" schedule="2" parity="1" notify="-n" log="-L -f" disk1-1="on" disk2-2="on" disk3-3="on" priority="-n19,-c2 -n7" day="0" hour="0" min="0" But this showed up in my log today, about one of the excluded directories: Feb 14 06:19:43 Tower bunker: warning: MD5 hash key mismatch (updated), /mnt/disk1/CrashplanTower/470662345684159084/cpbf0000000000000000000/cpbmf was modified <snip> Feb 14 06:36:14 Tower bunker: warning: MD5 hash key mismatch (updated), /mnt/disk1/CrashplanTower/470662345684159084/cpbf0000000000003747658/cpbdf was modified Feb 14 06:36:14 Tower bunker: verified 661488 files from /mnt/disk1 with mask *. Found: 75 mismatches (updated), 0 corruptions. Duration: 06:36:13 Average speed: 47.5 MB/s Since it just started, I wonder if the problem is that ".fsr" is excluded but that directory only exists during a defrag. Can you post the output of command: ps -ef | grep inotify Sure thing! Thanks for looking: root@Tower:~# ps -ef | grep inotify root 5522 5288 0 13:26 pts/0 00:00:00 grep inotify root 6484 1 0 Feb09 ? 00:00:01 inotifywait -mrq --exclude (^/mnt/disk[0-9]+/(\S+\.img$|.fsr|CrashplanTower|VMISO|cache_backup)/|\.AppleDB|\.DS_Store) --format %w%f -e close_write /mnt/disk1 /mnt/disk2 /mnt/disk3 Hmm, maybe it should be \.fsr instead of .fsr? Any chance you might be willing to exclude that by default, along with the Apple stuff? https://lime-technology.com/forum/index.php?topic=44989.msg442250#msg442250 Quote Link to comment
wgstarks Posted February 16, 2016 Share Posted February 16, 2016 Can you post the output of command: ps -ef | grep inotify I know this wasn't really directed at me but I thought I'd run it in case it might help with fixing the issue with exclusions. Brunnhilde login: root Password: Linux 4.1.17-unRAID. Last login: Mon Feb 15 20:07:47 -0500 2016 on /dev/pts/0 from 10.0.1.190. root@Brunnhilde:~# ps -ef | grep inotify root 21768 21082 0 17:03 pts/0 00:00:00 grep inotify root 32233 1 0 Feb13 ? 00:00:31 inotifywait -mrq --exclude (^/mnt/disk[0-9]+/(\S+\.img$|TM_Jasper|TM_MBP|TM_Maggie|TM_Mini)/|\.AppleDB|\.DS_Store) --format %w%f -e close_write /mnt/disk1 /mnt/disk2 /mnt/disk3 /mnt/disk4 /mnt/disk5 /mnt/disk6 /mnt/disk7 /mnt/disk8 /mnt/disk9 root@Brunnhilde:~# I've also attached one of the exported bad hash files. Not sure it really shows much of anything other than the fact that the excluded folder was still checked. disk9.export.20160213.bad.hash.zip Quote Link to comment
ljm42 Posted February 16, 2016 Share Posted February 16, 2016 All of the inotifywait exclude examples I can find have the regex wrapped in single quotes. Maybe that would help? Quote Link to comment
bonienl Posted February 17, 2016 Author Share Posted February 17, 2016 All of the inotifywait exclude examples I can find have the regex wrapped in single quotes. Maybe that would help? The original string is between quotes but these are not shown with the 'ps' command. I'll try to reproduce the issue, but so far the exclude works as expected on my system. Quote Link to comment
ljm42 Posted February 18, 2016 Share Posted February 18, 2016 So from what I can tell, once a file *somehow* gets bunker's file attributes added to it, that file will be processed by bunker's export and verify routines, even if the directory it is in has been excluded. To prevent this, I think we need a "Clean" option that loops through the files in the currently excluded directories and strips off any file attributes that bunker added in the past. It could either be run manually by the user or bunker could run it automatically before any export or verify routine. As to how the wrong files got the attributes in the first place, I can think of two possibilities: 1) If there are special characters in the share name (such as periods or dashes), they could confuse the regex parsers. Ideally the file integrity plugin would escape those characters before passing the share names to inotifywait or bunker. 2) It is also possible that user did a build or enabled the watch feature before they set the exclusions. Even though I really don't think I did this I'll admit it is a possibility. Either way, the "Clean" routine would fix it. Quote Link to comment
bonienl Posted February 18, 2016 Author Share Posted February 18, 2016 So from what I can tell, once a file *somehow* gets bunker's file attributes added to it, that file will be processed by bunker's export and verify routines, even if the directory it is in has been excluded. To prevent this, I think we need a "Clean" option that loops through the files in the currently excluded directories and strips off any file attributes that bunker added in the past. It could either be run manually by the user or bunker could run it automatically before any export or verify routine. As to how the wrong files got the attributes in the first place, I can think of two possibilities: 1) If there are special characters in the share name (such as periods or dashes), they could confuse the regex parsers. Ideally the file integrity plugin would escape those characters before passing the share names to inotifywait or bunker. 2) It is also possible that user did a build or enabled the watch feature before they set the exclusions. Even though I really don't think I did this I'll admit it is a possibility. Either way, the "Clean" routine would fix it. Good points. I'll work on an update to address these. Quote Link to comment
wgstarks Posted February 18, 2016 Share Posted February 18, 2016 So from what I can tell, once a file *somehow* gets bunker's file attributes added to it, that file will be processed by bunker's export and verify routines, even if the directory it is in has been excluded. To prevent this, I think we need a "Clean" option that loops through the files in the currently excluded directories and strips off any file attributes that bunker added in the past. It could either be run manually by the user or bunker could run it automatically before any export or verify routine. As to how the wrong files got the attributes in the first place, I can think of two possibilities: 1) If there are special characters in the share name (such as periods or dashes), they could confuse the regex parsers. Ideally the file integrity plugin would escape those characters before passing the share names to inotifywait or bunker. 2) It is also possible that user did a build or enabled the watch feature before they set the exclusions. Even though I really don't think I did this I'll admit it is a possibility. Either way, the "Clean" routine would fix it. I know this definatly applies to my case. I originally included the TimeMachine backups because the incompatibility hadn't been reported yet. Afterwards, I was not able to exclude the TM shares. Quote Link to comment
bonienl Posted February 18, 2016 Author Share Posted February 18, 2016 I've created an updated version. Please use 'Check for Updates' in the plugin manager to get it. This release has the following changes: - fixed regular expression when share and file names have 'special' characters - added *Clear* button, which allows to remove extended attributes from excluded folders and files only - added *Custom* field, which allows to enter custom file names and/or file extensions - make exclusion of Apple metadata (netatalk) user selectable IT IS MANDATORY TO RE-APPLY THE SETTINGS AFTER THIS UPDATE Quote Link to comment
wgstarks Posted February 18, 2016 Share Posted February 18, 2016 I've created an updated version. Please use 'Check for Updates' in the plugin manager to get it. This release has the following changes: - fixed regular expression when share and file names have 'special' characters - added *Clear* button, which allows to remove extended attributes from excluded folders and files only - added *Custom* field, which allows to enter custom file names and/or file extensions - make exclusion of Apple metadata (netatalk) user selectable IT IS MANDATORY TO RE-APPLY THE SETTINGS AFTER THIS UPDATE That was quick. I'll give it a shot later tonight. Thanks. Question I've got VFS Recycle plugin installed which creates a folder (.Recycle.Bin) when I delete files. Can I enter .Recyle.Bin in the custom field to exclude these temp folders? I haven't seen them appear as shares. Doesn't really create an issue one way or the other. Just seems a waste to check files waiting to be deleted. Quote Link to comment
bonienl Posted February 18, 2016 Author Share Posted February 18, 2016 Question I've got VFS Recycle plugin installed which creates a folder (.Recycle.Bin) when I delete files. Can I enter .Recyle.Bin in the custom field to exclude these temp folders? I haven't seen them appear as shares. Doesn't really create an issue one way or the other. Just seems a waste to check files waiting to be deleted. I've made version 2016.02.18b which allows to exclude the folder. You need to enter its name .Recycle.Bin in the custom field. Quote Link to comment
wgstarks Posted February 18, 2016 Share Posted February 18, 2016 I've made version 2016.02.18b which allows to exclude the folder. You need to enter its name .Recycle.Bin in the custom field. You Rock Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.