March 21, 201610 yr Author Another question for you from your quote below. Does this mean that new files do not get a checksum or exported automatically? Meaning any new files we add must be manually added to FIP? Checksums are always automatically generated and placed in the extended attributes and verified when protection and verification are enabled on the settings page. The status of the exported files is now only monitored.
March 21, 201610 yr Another question for you from your quote below. Does this mean that new files do not get a checksum or exported automatically? Meaning any new files we add must be manually added to FIP? Checksums are always automatically generated and placed in the extended attributes and verified when protection and verification are enabled on the settings page. The status of the exported files is now only monitored. I have found these two different modes quite confusing until very recently. I'm actually only interested in the checksums that are stored in the extended attributes and verified on a regular basis. I have no need for the export function so is it possible to disable alerts associated with it? If I delete the exported checksum files, perhaps?
March 21, 201610 yr Author I have found these two different modes quite confusing until very recently. I'm actually only interested in the checksums that are stored in the extended attributes and verified on a regular basis. I have no need for the export function so is it possible to disable alerts associated with it? If I delete the exported checksum files, perhaps? Good suggestion. I have made a new version 2016.03.21 available, which allows for disabling of notifications about the export status when no export file exists for the corresponding disk.
March 21, 201610 yr I have found these two different modes quite confusing until very recently. I'm actually only interested in the checksums that are stored in the extended attributes and verified on a regular basis. I have no need for the export function so is it possible to disable alerts associated with it? If I delete the exported checksum files, perhaps? Good suggestion. I have made a new version 2016.03.21 available, which allows for disabling of notifications about the export status when no export file exists for the corresponding disk. That's brilliant. Thank you! I like the blue crosses - a good choice because they are not as scary as red ones.
March 22, 201610 yr An issue I just ran across - I have "when parity operations are running" set to Don't start. But I'm rebuilding a data disk, and it started update new files - it really shouldn't do that, since that's a parity operation as well (it's really slowing down my rebuild!).
March 22, 201610 yr Even after shutting it down (i.e. setting "search for new files" or whatever it's called to NO", it continued running multiple find commands - I finally had to uninstall the plugin to get it to stop - if I killed a find, it just started back up..... It really needs to stop entirely when parity operations are going on (at least, if that's what I ask for).
March 23, 201610 yr Author Even after shutting it down (i.e. setting "search for new files" or whatever it's called to NO", it continued running multiple find commands - I finally had to uninstall the plugin to get it to stop - if I killed a find, it just started back up..... It really needs to stop entirely when parity operations are going on (at least, if that's what I ask for). If the option "When parity operation is running" is set to "Don't start" then the verification tasks won't start if parity is running at the time. Once the verification tasks are started they can't be terminated. This is also true for a parity check which kicks in afterwards. My advice: schedule the parity check and verification tasks in such a way that they can't interfere with each other.
March 23, 201610 yr Hi, This plugin seems to be great, but I have many questions about this before installing it, and I think many of us also have them. I would love to find a place in this plugin first post with FAQ like: - Does this replace the necessity of the monthly parity check? - Does it needs to be periodically checked too? If yes, how often is it recommended? - How long it takes compared to parity? - Aren't too many integrity/parity checks dangerous to the drives themselves? - Does this plugin affects the server performance when reading/writing many small files? I have several uses for my unRAID server: one is to store media, and while movies and series are easily replaceable, music would be worse, and I also store important files and work folders, so being able to just focus in the "Protect these folders" selection would be fantastic to me.
March 23, 201610 yr - Does this replace the necessity of the monthly parity check?Parity validity and file integrity are such different categories that you might as well say they are unrelated. - Does it needs to be periodically checked too? If yes, how often is it recommended?It will notify you
March 23, 201610 yr The plugin looks great, will I be able to see the checksums that are created in the samba shares on my windows computer. I've read that the checksums are being stored in the metadata of the files, can I view it somehow?
March 23, 201610 yr The plugin looks great, will I be able to see the checksums that are created in the samba shares on my windows computer. I've read that the checksums are being stored in the metadata of the files, can I view it somehow? It is not that easy to view the attributes directly from Windows. However the plugin includes an export feature to write the checksums out to a text file (in the same format as used by some Windows utilises)
March 23, 201610 yr Oww that's a bummer, but it's a windows problem I guess. How can you view the hash from the linux console?
March 23, 201610 yr Oww that's a bummer, but it's a windows problem I guess. How can you view the hash from the linux console? As far as I know you cannot - you have to do it at the command line level. However it only takes seconds to create the export file that can be viewed in any text editor.
March 23, 201610 yr Yeah, but I'm wondering about the command line level, because it doesn't seem to hard to write a script that ssh's into the server and fetches the hash and presents it to you on windows.
March 23, 201610 yr Even after shutting it down (i.e. setting "search for new files" or whatever it's called to NO", it continued running multiple find commands - I finally had to uninstall the plugin to get it to stop - if I killed a find, it just started back up..... It really needs to stop entirely when parity operations are going on (at least, if that's what I ask for). If the option "When parity operation is running" is set to "Don't start" then the verification tasks won't start if parity is running at the time. Once the verification tasks are started they can't be terminated. This is also true for a parity check which kicks in afterwards. My advice: schedule the parity check and verification tasks in such a way that they can't interfere with each other. That's not my point - I shut down the server and restarted it to rebuild a data disk that had died - I had "don't start" set - but it still started - so clearly, the plugin doesn't consider rebuilding a data disk the same as a parity rebuild, and it should......
March 23, 201610 yr Author That's not my point - I shut down the server and restarted it to rebuild a data disk that had died - I had "don't start" set - but it still started - so clearly, the plugin doesn't consider rebuilding a data disk the same as a parity rebuild, and it should...... A data disk rebuild is not the same as a parity check, as you notice. The current option is for parity check only. Note: when you stop the array it will terminate any running verification tasks immediately.
March 23, 201610 yr Author - Does this replace the necessity of the monthly parity check? As already noted by trurl, parity check and file integrity check are two different things. They can complement each other but not replace. - Does it needs to be periodically checked too? If yes, how often is it recommended? The purpose of the file integrity plugin is to detect silent file corruption, my experience shows this is very rarily happening, hence the period check interval doesn't have to be high. Set it to a value you feel comfortable with. - How long it takes compared to parity? These can't be compared directly. Parity check will scan the complete disk regardless of content, while file integrity looks at the files present. A 4TB disk can take consirable time in a parity check, but if it holds a few files then a file integrity check is done in a few seconds. - Aren't too many integrity/parity checks dangerous to the drives themselves? No. - Does this plugin affects the server performance when reading/writing many small files? For each file written, a hash value needs to be calculated, this will take processor time. Depends on your processor how noticable this is going to be. I have several uses for my unRAID server: one is to store media, and while movies and series are easily replaceable, music would be worse, and I also store important files and work folders, so being able to just focus in the "Protect these folders" selection would be fantastic to me. Remember that "protect these folders" will add a hash value to the file, which allows for corruption detection, but it will NOT repair any corruption.
March 24, 201610 yr How can you view the hash from the linux console? You can get the md5 hash with this command: getfattr -n user.md5 /path/to/filename If you are using something other than md5, do this first to see what attributes are available: getfattr /path/to/filename
March 25, 201610 yr What did I do wrong I am getting - /config/plugins/dynamix.file.integrity/disks.ini no such file or directory - the build never finishes and grep proc/mdcmd no such file or directory
March 26, 201610 yr What did I do wrong I am getting - /config/plugins/dynamix.file.integrity/dicks.ini no such file or directory - the build never finishes ... dicks.ini, really?
March 26, 201610 yr What did I do wrong I am getting - /config/plugins/dynamix.file.integrity/dicks.ini no such file or directory - the build never finishes ... dicks.ini, really? hehehee.....
March 27, 201610 yr Hey bonienl, I was looking at these scripts: https://lime-technology.com/forum/index.php?topic=35183.0 https://lime-technology.com/forum/index.php?topic=7018.msg68073;topicseen#msg68073 and I realized both could efficiently be replaced by this plugin if it did some parsing of the export files: It could look for files with the same name (not including the path). This would identify issues with a file existing at both /mnt/disk1/share/file and /mnt/disk2/share/file, for instance. The hashes wouldn't necessarily be the same. It could look for files with identical hashes, regardless of file name, to identify true duplicates. I think this would be a nice way to get extra value out of these export files, if you are up for it
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.