bonienl Posted March 21, 2016 Author Share Posted March 21, 2016 Another question for you from your quote below. Does this mean that new files do not get a checksum or exported automatically? Meaning any new files we add must be manually added to FIP? Checksums are always automatically generated and placed in the extended attributes and verified when protection and verification are enabled on the settings page. The status of the exported files is now only monitored. Quote Link to comment
John_M Posted March 21, 2016 Share Posted March 21, 2016 Another question for you from your quote below. Does this mean that new files do not get a checksum or exported automatically? Meaning any new files we add must be manually added to FIP? Checksums are always automatically generated and placed in the extended attributes and verified when protection and verification are enabled on the settings page. The status of the exported files is now only monitored. I have found these two different modes quite confusing until very recently. I'm actually only interested in the checksums that are stored in the extended attributes and verified on a regular basis. I have no need for the export function so is it possible to disable alerts associated with it? If I delete the exported checksum files, perhaps? Quote Link to comment
bonienl Posted March 21, 2016 Author Share Posted March 21, 2016 I have found these two different modes quite confusing until very recently. I'm actually only interested in the checksums that are stored in the extended attributes and verified on a regular basis. I have no need for the export function so is it possible to disable alerts associated with it? If I delete the exported checksum files, perhaps? Good suggestion. I have made a new version 2016.03.21 available, which allows for disabling of notifications about the export status when no export file exists for the corresponding disk. Quote Link to comment
John_M Posted March 21, 2016 Share Posted March 21, 2016 I have found these two different modes quite confusing until very recently. I'm actually only interested in the checksums that are stored in the extended attributes and verified on a regular basis. I have no need for the export function so is it possible to disable alerts associated with it? If I delete the exported checksum files, perhaps? Good suggestion. I have made a new version 2016.03.21 available, which allows for disabling of notifications about the export status when no export file exists for the corresponding disk. That's brilliant. Thank you! I like the blue crosses - a good choice because they are not as scary as red ones. Quote Link to comment
Vr2Io Posted March 22, 2016 Share Posted March 22, 2016 File Integrity plugin was one of must install plugin. Quote Link to comment
quinnmjcj Posted March 22, 2016 Share Posted March 22, 2016 An issue I just ran across - I have "when parity operations are running" set to Don't start. But I'm rebuilding a data disk, and it started update new files - it really shouldn't do that, since that's a parity operation as well (it's really slowing down my rebuild!). Quote Link to comment
quinnmjcj Posted March 22, 2016 Share Posted March 22, 2016 Even after shutting it down (i.e. setting "search for new files" or whatever it's called to NO", it continued running multiple find commands - I finally had to uninstall the plugin to get it to stop - if I killed a find, it just started back up..... It really needs to stop entirely when parity operations are going on (at least, if that's what I ask for). Quote Link to comment
bonienl Posted March 23, 2016 Author Share Posted March 23, 2016 Even after shutting it down (i.e. setting "search for new files" or whatever it's called to NO", it continued running multiple find commands - I finally had to uninstall the plugin to get it to stop - if I killed a find, it just started back up..... It really needs to stop entirely when parity operations are going on (at least, if that's what I ask for). If the option "When parity operation is running" is set to "Don't start" then the verification tasks won't start if parity is running at the time. Once the verification tasks are started they can't be terminated. This is also true for a parity check which kicks in afterwards. My advice: schedule the parity check and verification tasks in such a way that they can't interfere with each other. Quote Link to comment
almarma Posted March 23, 2016 Share Posted March 23, 2016 Hi, This plugin seems to be great, but I have many questions about this before installing it, and I think many of us also have them. I would love to find a place in this plugin first post with FAQ like: - Does this replace the necessity of the monthly parity check? - Does it needs to be periodically checked too? If yes, how often is it recommended? - How long it takes compared to parity? - Aren't too many integrity/parity checks dangerous to the drives themselves? - Does this plugin affects the server performance when reading/writing many small files? I have several uses for my unRAID server: one is to store media, and while movies and series are easily replaceable, music would be worse, and I also store important files and work folders, so being able to just focus in the "Protect these folders" selection would be fantastic to me. Quote Link to comment
trurl Posted March 23, 2016 Share Posted March 23, 2016 - Does this replace the necessity of the monthly parity check?Parity validity and file integrity are such different categories that you might as well say they are unrelated. - Does it needs to be periodically checked too? If yes, how often is it recommended?It will notify you Quote Link to comment
bosiwow Posted March 23, 2016 Share Posted March 23, 2016 The plugin looks great, will I be able to see the checksums that are created in the samba shares on my windows computer. I've read that the checksums are being stored in the metadata of the files, can I view it somehow? Quote Link to comment
itimpi Posted March 23, 2016 Share Posted March 23, 2016 The plugin looks great, will I be able to see the checksums that are created in the samba shares on my windows computer. I've read that the checksums are being stored in the metadata of the files, can I view it somehow? It is not that easy to view the attributes directly from Windows. However the plugin includes an export feature to write the checksums out to a text file (in the same format as used by some Windows utilises) Quote Link to comment
bosiwow Posted March 23, 2016 Share Posted March 23, 2016 Oww that's a bummer, but it's a windows problem I guess. How can you view the hash from the linux console? Quote Link to comment
itimpi Posted March 23, 2016 Share Posted March 23, 2016 Oww that's a bummer, but it's a windows problem I guess. How can you view the hash from the linux console? As far as I know you cannot - you have to do it at the command line level. However it only takes seconds to create the export file that can be viewed in any text editor. Quote Link to comment
bosiwow Posted March 23, 2016 Share Posted March 23, 2016 Yeah, but I'm wondering about the command line level, because it doesn't seem to hard to write a script that ssh's into the server and fetches the hash and presents it to you on windows. Quote Link to comment
quinnmjcj Posted March 23, 2016 Share Posted March 23, 2016 Even after shutting it down (i.e. setting "search for new files" or whatever it's called to NO", it continued running multiple find commands - I finally had to uninstall the plugin to get it to stop - if I killed a find, it just started back up..... It really needs to stop entirely when parity operations are going on (at least, if that's what I ask for). If the option "When parity operation is running" is set to "Don't start" then the verification tasks won't start if parity is running at the time. Once the verification tasks are started they can't be terminated. This is also true for a parity check which kicks in afterwards. My advice: schedule the parity check and verification tasks in such a way that they can't interfere with each other. That's not my point - I shut down the server and restarted it to rebuild a data disk that had died - I had "don't start" set - but it still started - so clearly, the plugin doesn't consider rebuilding a data disk the same as a parity rebuild, and it should...... Quote Link to comment
bonienl Posted March 23, 2016 Author Share Posted March 23, 2016 That's not my point - I shut down the server and restarted it to rebuild a data disk that had died - I had "don't start" set - but it still started - so clearly, the plugin doesn't consider rebuilding a data disk the same as a parity rebuild, and it should...... A data disk rebuild is not the same as a parity check, as you notice. The current option is for parity check only. Note: when you stop the array it will terminate any running verification tasks immediately. Quote Link to comment
bonienl Posted March 23, 2016 Author Share Posted March 23, 2016 - Does this replace the necessity of the monthly parity check? As already noted by trurl, parity check and file integrity check are two different things. They can complement each other but not replace. - Does it needs to be periodically checked too? If yes, how often is it recommended? The purpose of the file integrity plugin is to detect silent file corruption, my experience shows this is very rarily happening, hence the period check interval doesn't have to be high. Set it to a value you feel comfortable with. - How long it takes compared to parity? These can't be compared directly. Parity check will scan the complete disk regardless of content, while file integrity looks at the files present. A 4TB disk can take consirable time in a parity check, but if it holds a few files then a file integrity check is done in a few seconds. - Aren't too many integrity/parity checks dangerous to the drives themselves? No. - Does this plugin affects the server performance when reading/writing many small files? For each file written, a hash value needs to be calculated, this will take processor time. Depends on your processor how noticable this is going to be. I have several uses for my unRAID server: one is to store media, and while movies and series are easily replaceable, music would be worse, and I also store important files and work folders, so being able to just focus in the "Protect these folders" selection would be fantastic to me. Remember that "protect these folders" will add a hash value to the file, which allows for corruption detection, but it will NOT repair any corruption. Quote Link to comment
ljm42 Posted March 24, 2016 Share Posted March 24, 2016 How can you view the hash from the linux console? You can get the md5 hash with this command: getfattr -n user.md5 /path/to/filename If you are using something other than md5, do this first to see what attributes are available: getfattr /path/to/filename Quote Link to comment
brettm357 Posted March 25, 2016 Share Posted March 25, 2016 What did I do wrong I am getting - /config/plugins/dynamix.file.integrity/disks.ini no such file or directory - the build never finishes and grep proc/mdcmd no such file or directory Quote Link to comment
Kir Posted March 26, 2016 Share Posted March 26, 2016 What did I do wrong I am getting - /config/plugins/dynamix.file.integrity/dicks.ini no such file or directory - the build never finishes ... dicks.ini, really? Quote Link to comment
CHBMB Posted March 26, 2016 Share Posted March 26, 2016 What did I do wrong I am getting - /config/plugins/dynamix.file.integrity/dicks.ini no such file or directory - the build never finishes ... dicks.ini, really? hehehee..... Quote Link to comment
brettm357 Posted March 27, 2016 Share Posted March 27, 2016 Its fixed - now how about my errors Quote Link to comment
ljm42 Posted March 27, 2016 Share Posted March 27, 2016 Hey bonienl, I was looking at these scripts: https://lime-technology.com/forum/index.php?topic=35183.0 https://lime-technology.com/forum/index.php?topic=7018.msg68073;topicseen#msg68073 and I realized both could efficiently be replaced by this plugin if it did some parsing of the export files: It could look for files with the same name (not including the path). This would identify issues with a file existing at both /mnt/disk1/share/file and /mnt/disk2/share/file, for instance. The hashes wouldn't necessarily be the same. It could look for files with identical hashes, regardless of file name, to identify true duplicates. I think this would be a nice way to get extra value out of these export files, if you are up for it Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.