bonienl Posted March 19, 2016 Author Share Posted March 19, 2016 Thanks all, that was quick! Shouldn't FIP automatically update the export file? I believe it should if you have "Save new hashing results to flash" enabled. I don't. I do have "Save new hashing results to flash" enabled, however, it does not update the existing export file. I have changed the behaviour of the daily script "exportrotate' to reporting only, this means that it will signal that "build' or "export" are not up-to-date anymore, but it requires a user action to sync. Quote Link to comment
archedraft Posted March 19, 2016 Share Posted March 19, 2016 So FIP did not run this morning. I even tried the "update_cron" command after I changed the schedule yesterday just for good measure... Is everyone else not experiencing this? Quote Link to comment
bonienl Posted March 19, 2016 Author Share Posted March 19, 2016 So FIP did not run this morning. I even tried the "update_cron" command after I changed the schedule yesterday just for good measure... Is everyone else not experiencing this? Can you show the content of the following files: # cat [b]/etc/cron.d/root[/b] # Generated file integrity check schedule: 50 11 * * 1 /boot/config/plugins/dynamix.file.integrity/integrity-check.sh &> /dev/null cat [b][b]/boot/config/plugins/dynamix.file.integrity/integrity-check.sh[/b][/b] #!/bin/bash # This is an auto-generated file, do not change manually! # bunker=/usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker [[ $((10#$(date +%W)%5)) -eq 0 ]] && $bunker -Vqj -b2 -n -L /mnt/disk1 >/dev/null & [[ $((10#$(date +%W)%5)) -eq 0 ]] && $bunker -Vqj -b2 -n -L /mnt/disk2 >/dev/null & [[ $((10#$(date +%W)%5)) -eq 1 ]] && $bunker -Vqj -b2 -n -L /mnt/disk3 >/dev/null & [[ $((10#$(date +%W)%5)) -eq 1 ]] && $bunker -Vqj -b2 -n -L /mnt/disk4 >/dev/null & [[ $((10#$(date +%W)%5)) -eq 2 ]] && $bunker -Vqj -b2 -n -L /mnt/disk5 >/dev/null & [[ $((10#$(date +%W)%5)) -eq 2 ]] && $bunker -Vqj -b2 -n -L /mnt/disk6 >/dev/null & [[ $((10#$(date +%W)%5)) -eq 3 ]] && $bunker -Vqj -b2 -n -L /mnt/disk7 >/dev/null & [[ $((10#$(date +%W)%5)) -eq 3 ]] && $bunker -Vqj -b2 -n -L /mnt/disk8 >/dev/null & [[ $((10#$(date +%W)%5)) -eq 4 ]] && $bunker -Vqj -b2 -n -L /mnt/disk9 >/dev/null & [[ $((10#$(date +%W)%5)) -eq 4 ]] && $bunker -Vqj -b2 -n -L /mnt/disk10 >/dev/null & exit 0 With notifications ON (see settings) you would see messages as indicated below: 17-03-2016 19:41 Dynamix file integrity [disk3] Notice [VESTA] - Verify task for disk3 finished Duration: 5 hr, 10 min, 1 sec. 17-03-2016 19:22 Dynamix file integrity [disk5] Notice [VESTA] - Verify task for disk5 finished Duration: 4 hr, 50 min, 41 sec. 17-03-2016 14:31 Dynamix file integrity [disk3] Notice [VESTA] - Verify task for disk3 started Total files size: 2.22 TB 17-03-2016 14:31 Dynamix file integrity [disk5] Notice [VESTA] - Verify task for disk5 started Total files size: 2.13 TB Quote Link to comment
JorgeB Posted March 19, 2016 Share Posted March 19, 2016 Not a very big deal but if a disk is removed from the array (disk10 in this example) it still appears on the plugin selection. Quote Link to comment
John_M Posted March 19, 2016 Share Posted March 19, 2016 I'm still getting error messages (highlighted in red) in my syslog like this Mar 19 16:05:06 Lapulapu bunker: error: no export of file: /mnt/disk1/L_Public/Shared Music/.DS_Store despite (or, possibly because of) having Apple metadata excluded in File Integrity settings. Quote Link to comment
bonienl Posted March 19, 2016 Author Share Posted March 19, 2016 Not a very big deal but if a disk is removed from the array (disk10 in this example) it still appears on the plugin selection. You need to redo the disk verification tasks on the settings page. Quote Link to comment
bonienl Posted March 19, 2016 Author Share Posted March 19, 2016 I'm still getting error messages (highlighted in red) in my syslog like this Mar 19 16:05:06 Lapulapu bunker: error: no export of file: /mnt/disk1/L_Public/Shared Music/.DS_Store despite (or, possibly because of) having Apple metadata excluded in File Integrity settings. Can you show the output of: # [b]ps -ef | grep inotify[/b] root 23783 1 0 16:49 ? 00:00:00 inotifywait -mrqe close_write --exclude ^/mnt/disk[0-9]+/(.*\.Recycle\.Bin/|.*\.tmp$) --format %w%f /mnt/disk1 /mnt/disk10 /mnt/disk2 /mnt/disk3 /mnt/disk4 /mnt/disk5 /mnt/disk6 /mnt/disk7 /mnt/disk8 /mnt/disk9 Quote Link to comment
JorgeB Posted March 19, 2016 Share Posted March 19, 2016 You need to redo the disk verification tasks on the settings page. And how do I do that? This is the settings page from my test server, at the moment it only has 4 data disks. Pressing default doesn't remove unused disks. Quote Link to comment
bonienl Posted March 19, 2016 Author Share Posted March 19, 2016 You need to redo the disk verification tasks on the settings page. And how do I do that? This is the settings page from my test server, at the moment it only has 4 data disks. Pressing default doesn't remove unused disks. Hmm, in version 6.2 the number of disks presented in the array depends on the setting of "slots". Stop the array and change the slots number to 6 (if you are running dual parity). This is different behaviour, I am going to make an additional check in the plugin to explicitely exclude empty slots. Alternatively it might be considered a 'bug' in unRAID 6.2. Quote Link to comment
archedraft Posted March 19, 2016 Share Posted March 19, 2016 So FIP did not run this morning. I even tried the "update_cron" command after I changed the schedule yesterday just for good measure... Is everyone else not experiencing this? Can you show the content of the following files: # cat [b]/etc/cron.d/root[/b] # Generated file integrity check schedule: 50 11 * * 1 /boot/config/plugins/dynamix.file.integrity/integrity-check.sh &> /dev/null cat [b][b]/boot/config/plugins/dynamix.file.integrity/integrity-check.sh[/b][/b] #!/bin/bash # This is an auto-generated file, do not change manually! # bunker=/usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker [[ $((10#$(date +%W)%5)) -eq 0 ]] && $bunker -Vqj -b2 -n -L /mnt/disk1 >/dev/null & [[ $((10#$(date +%W)%5)) -eq 0 ]] && $bunker -Vqj -b2 -n -L /mnt/disk2 >/dev/null & [[ $((10#$(date +%W)%5)) -eq 1 ]] && $bunker -Vqj -b2 -n -L /mnt/disk3 >/dev/null & [[ $((10#$(date +%W)%5)) -eq 1 ]] && $bunker -Vqj -b2 -n -L /mnt/disk4 >/dev/null & [[ $((10#$(date +%W)%5)) -eq 2 ]] && $bunker -Vqj -b2 -n -L /mnt/disk5 >/dev/null & [[ $((10#$(date +%W)%5)) -eq 2 ]] && $bunker -Vqj -b2 -n -L /mnt/disk6 >/dev/null & [[ $((10#$(date +%W)%5)) -eq 3 ]] && $bunker -Vqj -b2 -n -L /mnt/disk7 >/dev/null & [[ $((10#$(date +%W)%5)) -eq 3 ]] && $bunker -Vqj -b2 -n -L /mnt/disk8 >/dev/null & [[ $((10#$(date +%W)%5)) -eq 4 ]] && $bunker -Vqj -b2 -n -L /mnt/disk9 >/dev/null & [[ $((10#$(date +%W)%5)) -eq 4 ]] && $bunker -Vqj -b2 -n -L /mnt/disk10 >/dev/null & exit 0 With notifications ON (see settings) you would see messages as indicated below: 17-03-2016 19:41 Dynamix file integrity [disk3] Notice [VESTA] - Verify task for disk3 finished Duration: 5 hr, 10 min, 1 sec. 17-03-2016 19:22 Dynamix file integrity [disk5] Notice [VESTA] - Verify task for disk5 finished Duration: 4 hr, 50 min, 41 sec. 17-03-2016 14:31 Dynamix file integrity [disk3] Notice [VESTA] - Verify task for disk3 started Total files size: 2.22 TB 17-03-2016 14:31 Dynamix file integrity [disk5] Notice [VESTA] - Verify task for disk5 started Total files size: 2.13 TB cat /etc/cron.d/root # Generated file integrity check schedule: 5 0 19 * * /boot/config/plugins/dynamix.file.integrity/integrity-check.sh &> /de v/null # Generated docker monitoring schedule: 10 0 * * * /usr/local/emhttp/plugins/dynamix.docker.manager/scripts/dockerupdate .php check &> /dev/null # Generated system monitoring schedule: */1 * * * * /usr/local/emhttp/plugins/dynamix/scripts/monitor &> /dev/null # Generated mover schedule: 0 1 * * * /usr/local/sbin/mover |& logger # Generated parity check schedule: 0 0 1 * * /usr/local/sbin/mdcmd check &> /dev/null # Generated plugins version check schedule: 10 0 * * * /usr/local/emhttp/plugins/dynamix.plugin.manager/scripts/plugincheck &> /dev/null # Generated array status check schedule: 20 0 * * * /usr/local/emhttp/plugins/dynamix/scripts/statuscheck &> /dev/null cat /boot/config/plugins/dynamix.file.integrity/integrity-check.sh #!/bin/bash # This is an auto-generated file, do not change manually! # bunker=nice -n19 ionice -c2 -n7 /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker [[ $(grep -Po '^mdResync=\K\S+' /proc/mdcmd) -ne 0 ]] && exit 0 [[ $((10#$(date +%m)%1)) -eq 0 ]] && $bunker -Vqj -b2 -n -L -f /boot/config/plugins/dynamix.file.integrity/logs/disk1.export.$(date +%Y%m%d).bad.log /mnt/disk1 >/dev/null & [[ $((10#$(date +%m)%1)) -eq 0 ]] && $bunker -Vqj -b2 -n -L -f /boot/config/plugins/dynamix.file.integrity/logs/disk2.export.$(date +%Y%m%d).bad.log /mnt/disk2 >/dev/null & [[ $((10#$(date +%m)%1)) -eq 0 ]] && $bunker -Vqj -b2 -n -L -f /boot/config/plugins/dynamix.file.integrity/logs/disk3.export.$(date +%Y%m%d).bad.log /mnt/disk3 >/dev/null & exit 0 Quote Link to comment
bonienl Posted March 19, 2016 Author Share Posted March 19, 2016 This line in the file integrity-check.sh: [[ $(grep -Po '^mdResync=\K\S+' /proc/mdcmd) -ne 0 ]] && exit 0 Will abort the verification process if a parity check is running at the same time. Did you have a parity check running? (you may want to change the setting of When parity operation is running to Continue). Quote Link to comment
archedraft Posted March 19, 2016 Share Posted March 19, 2016 This line in the file integrity-check.sh: [[ $(grep -Po '^mdResync=\K\S+' /proc/mdcmd) -ne 0 ]] && exit 0 Will abort the verification process if a parity check is running at the same time. Did you have a parity check running? (you may want to change the setting of When parity operation is running to Continue). No, parity check only happens on the first of the month. I can change it to "continue" if that is the issue? Quote Link to comment
bonienl Posted March 19, 2016 Author Share Posted March 19, 2016 No, parity check only happens on the first of the month. I can change it to "continue" if that is the issue? Strange, then it should have continued, can you change the setting and see if that makes a difference? Quote Link to comment
John_M Posted March 19, 2016 Share Posted March 19, 2016 I'm still getting error messages (highlighted in red) in my syslog like this Mar 19 16:05:06 Lapulapu bunker: error: no export of file: /mnt/disk1/L_Public/Shared Music/.DS_Store despite (or, possibly because of) having Apple metadata excluded in File Integrity settings. Can you show the output of: # [b]ps -ef | grep inotify[/b] root 23783 1 0 16:49 ? 00:00:00 inotifywait -mrqe close_write --exclude ^/mnt/disk[0-9]+/(.*\.Recycle\.Bin/|.*\.tmp$) --format %w%f /mnt/disk1 /mnt/disk10 /mnt/disk2 /mnt/disk3 /mnt/disk4 /mnt/disk5 /mnt/disk6 /mnt/disk7 /mnt/disk8 /mnt/disk9 Here you go: root@Lapulapu:~# ps -ef | grep inotify root 13646 1 0 16:03 ? 00:00:00 inotifywait -mrqe close_write --exclude ^/mnt/disk[0-9]+/(.*\.icns$|.*\.AppleDB/|.*\.DS_Store$) --format %w%f /mnt/disk1 /mnt/disk2 /mnt/disk7 root 28563 24193 0 18:38 pts/0 00:00:00 grep inotify root@Lapulapu:~# I'm excluding *.icns and Apple metadata. And I'm only including three disks at the moment. Quote Link to comment
bonienl Posted March 19, 2016 Author Share Posted March 19, 2016 That looks alright and should work as expected. Perhaps there is some left-over from earlier days. Have you tried to delete the file '/mnt/disk1/L_Public/Shared Music/.DS_Store' ? Quote Link to comment
archedraft Posted March 19, 2016 Share Posted March 19, 2016 Wow alright I figured it out after 60 mins of changes settings. If priority is set to "low" it doesn't start but if it is set to "normal" then it runs! Quote Link to comment
John_M Posted March 19, 2016 Share Posted March 19, 2016 That looks alright and should work as expected. Perhaps there is some left-over from earlier days. Have you tried to delete the file '/mnt/disk1/L_Public/Shared Music/.DS_Store' ? Yes, I suspect they are left-overs since before exclusions were possible. I found a neat little command for deleting them all from each of my shares. Don't worry, OS X will re-create them soon enough and they will then be properly excluded! Many thanks. find /mnt/user/L_Public -name .DS_Store -exec rm '{}' \; find /mnt/user/L_Private -name .DS_Store -exec rm '{}' \; Quote Link to comment
bonienl Posted March 19, 2016 Author Share Posted March 19, 2016 Wow alright I figured it out after 60 mins of changes settings. If priority is set to "low" it doesn't start but if it is set to "normal" then it runs! Interesting, what happens when you manually execute the command? nice -n19 ionice -c2 -n7 /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker Quote Link to comment
archedraft Posted March 19, 2016 Share Posted March 19, 2016 Wow alright I figured it out after 60 mins of changes settings. If priority is set to "low" it doesn't start but if it is set to "normal" then it runs! Interesting, what happens when you manually execute the command? nice -n19 ionice -c2 -n7 /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker I will test and find out. Before I do, is there a command to stop bunker from verifying each disk? I would hate to have multiple instances running. Quote Link to comment
bonienl Posted March 19, 2016 Author Share Posted March 19, 2016 Wow alright I figured it out after 60 mins of changes settings. If priority is set to "low" it doesn't start but if it is set to "normal" then it runs! Interesting, what happens when you manually execute the command? nice -n19 ionice -c2 -n7 /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker I will test and find out. Before I do, is there a command to stop bunker from verifying each disk? I would hate to have multiple instances running. 1. Running bunker without arguments will let it show help only and quit immediately. 2. Select the disks you want to verify in the "Disk verification tasks" table. If 'none' are selected it will actually verify all ! Quote Link to comment
JorgeB Posted March 19, 2016 Share Posted March 19, 2016 Hmm, in version 6.2 the number of disks presented in the array depends on the setting of "slots". Stop the array and change the slots number to 6 (if you are running dual parity). This is different behaviour, I am going to make an additional check in the plugin to explicitely exclude empty slots. Alternatively it might be considered a 'bug' in unRAID 6.2. Changing the number of slots did it, and that’s a good enough fix for me, I was thinking the plugin was getting info from the previous array config. Quote Link to comment
archedraft Posted March 19, 2016 Share Posted March 19, 2016 Wow alright I figured it out after 60 mins of changes settings. If priority is set to "low" it doesn't start but if it is set to "normal" then it runs! Interesting, what happens when you manually execute the command? nice -n19 ionice -c2 -n7 /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker This is what happens: nice -n19 ionice -c2 -n7 /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker bunker v2.5 - Copyright (c) 2015-2016 Bergware International Usage: bunker -a|A|v|V|u|e|t|i|c|C|r|R [-fdDsSlLnq] [-md5|-b2] path [!] [mask] -a add hash key attribute for files, specified in path and optional m ask -A same as -a option with implicit export function (may use -f) -v verify hash key attribute and report mismatches (may use -f) -V same as -v option with updating of mismatched keys (may use -f) -u update mismatched or corrupted hash keys with new hash key attribu te (may use -f) -e export hash key attributes to the export file (may use -f) -t touch file, i.e. copy file modified time to extended attribute -i import hash key attributes from file and restore them (must use -f ) -c check hash key attributes from input file (must use -f) -C same as -c option and correct mismatched hash key in extended attr ibute (must use -f) -r remove hash key extended attribute from specified selection (may u se -f) -R same as -r option and remove all other values too (may use -f) -f <file> optional set file reference to <file>. Defaults to /tmp/bunker.sto re.log -d <days> optional only verify/update/remove files which were scanned <days> or longer ago -D <time> optional only add/verify/update/export/remove files newer than <ti me>, time = NNs,m,h,d,w -s <size> optional only include files smaller than <size> -S <size> optional only include files greater than <size> -l optional create log entry in the syslog file -L optional, same as -l but only create log entry when changes are pr esent -n optional send notifications when file corruption is detected -q optional quiet mode, suppress all output. Use for background proce ssing -md5 optional use md5 hashing algorithm instead of sha256 -b2 optional use blake2 hashing algorithm instead of sha256 path path to starting directory, mandatory with some exceptions (see ex amples) mask optional filter for file selection. Default is all files when path or mask names have spaces, then place names between quot es precede mask with ! to change its operation from include to exclud e Examples: bunker -a /mnt/user/tv add SHA key for files in share tv bunker -a -S 10M /mnt/user/tv add SHA key for files gre ater than 10 MB in share tv bunker -a /mnt/user/tv *.mov add SHA key for .mov file s only in share tv bunker -a /mnt/user/tv ! *.mov add SHA key for all files in share tv except .mov files bunker -A -f /tmp/keys.hash /mnt/user/tv add SHA key for files in share tv and export to file keys.hash bunker -v -n /mnt/user/files verify SHA key for previo usly scanned files and send notifications bunker -V /mnt/user/files verify SHA key for scanne d files and update any mismatches bunker -v -d 90 /mnt/user/movies verify SHA key for files scanned 90 days or longer ago bunker -v -f /tmp/errors.hash /mnt/user/movies verify SHA key and save m ismatches in file errors.hash bunker -u /mnt/disk1 update SHA key for mismat ching files bunker -u -D 12h /mnt/disk1 update SHA key for mismat ching files created in the last 12 hours bunker -e -f /tmp/disk1_keys.hash /mnt/disk1 export SHA key to file di sk1_keys.hash bunker -i -f /tmp/disk1_keys.hash import and restore SHA ke y from user defined file - no path bunker -c -f /tmp/disk1_keys.hash check SHA key from user d efined input file - no path bunker -C -f /tmp/disk1_keys.hash check SHA key and correct mismatched attribute (omit corruptions) - no path bunker -r /mnt/user/tv remove SHA key for files in share tv bunker -r -f /tmp/errors.hash remove SHA key for files listed in file errors.hash - no path Quote Link to comment
bonienl Posted March 20, 2016 Author Share Posted March 20, 2016 Wow alright I figured it out after 60 mins of changes settings. If priority is set to "low" it doesn't start but if it is set to "normal" then it runs! Ok, found the issue and made a correction. New version 2016.03.20 is available. This version also addresses compatibility with unRAID v6.2. Users are encouraged to update to this version. Please re-apply settings after upgrading to make them effective. Quote Link to comment
archedraft Posted March 20, 2016 Share Posted March 20, 2016 Wow alright I figured it out after 60 mins of changes settings. If priority is set to "low" it doesn't start but if it is set to "normal" then it runs! Ok, found the issue and made a correction. New version 2016.03.20 is available. This version also addresses compatibility with unRAID v6.2. Users are encouraged to update to this version. Please re-apply settings after upgrading to make them effective. Just tested with low and it works. Thank you. Another question for you from your quote below. Does this mean that new files do not get a checksum or exported automatically? Meaning any new files we add must be manually added to FIP? I have changed the behaviour of the daily script "exportrotate' to reporting only, this means that it will signal that "build' or "export" are not up-to-date anymore, but it requires a user action to sync. Quote Link to comment
WizADSL Posted March 21, 2016 Share Posted March 21, 2016 I've just noticed that if I use rsync to copy files from a non-array location (it may not matter where the files are coming from, but this is the case for me) to a disk on the array that is being monitored by the file integrity plugin no checksums are being added to the written files. The rsync command I am using is: rsync -avHAX --progress /mnt/SomeDrive/ /mnt/disk1/Destination/ Also, would it be possible to add additional locations outside the array to have checksums added/checked? It would be great if I could monitor all of the files on my system. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.