bonienl Posted December 29, 2015 Share Posted December 29, 2015 WARNING: USING THIS PLUGIN ON DISKS FORMATTED IN REISERFS MAY LEAD TO SYSTEM INSTABILITY. IT IS ADVISED TO USE XFS. UPDATE Version 2016.01.05 marks the official release of this plugin This plugin is still work in progress and provides a GUI interface to the (new) bunker utility. It has two sections: settings and control The settings page specifies how automatic file hashing and verification is done. These are processes which - once started - will keep running in the background and notify the user when a silent file corruption is detected. For the automation of the hashing of new and modified files, the program inotifywait is used, this is part of the inotify-tools package which is installed together with the plugin as needed (it is not present on unRAID by default unless the NerdPack package is installed). It is possible to exclude both disks and folders (shares) from the hashing system. The verification process is based on cron entries. The user can define groups of disks which are verified at the given schedule. For example if a weekly schedule is chosen then every other group is verified the following week. With setting up groups it is possible to do all in once or spread out the load over several windows. This is done because the disk verification processes are started in parallel, the bigger a group the more horse power is required from your processor. The control page is used for general operations. Here it is possible to select the disks and choose from the commands: build, remove, export, import, check. An initial build needs to be done to have a checksum value added to the existing files. Building the checksums can be done either in a single go (when all disks are selected), or with intermediate steps (each time another disk(s) is selected. Once the initial build is complete it is possible to export the checksum values to a text file (normal operation is that a checksum value is linked to a file directly using its extended attributes). With the export files present (they are stored on flash), it is possible to import then back into the extended attributes or do a manual check to find file corruption. Once an operation is started its progress is shown in real-time, and includes an ETA for those operations which tend to take a long time. There is no need to stay at the same page or keep the browser open all the time. Whenever the control page is revisited from the same browser, it will update itself to the latest state. Attached the settings and control screens to give an indication of what it looks like If you want to use this functionality, you can install the plugin using the plugin manager. Copy and paste this URL https://raw.githubusercontent.com/bergware/dynamix/master/unRAIDv6/dynamix.file.integrity.plg People who are using an earlier version of bunker and associated cron jobs should stop and uninstall them, as they will interfere with this package. Extensive HELP is available on the GUI pages. Please read this carefully before setting up and using this new functionality. Hope you like this new addition to the Dynamix family. 1 2 3 Quote Link to comment
archedraft Posted December 29, 2015 Share Posted December 29, 2015 Awesome! Very excited! Quote Link to comment
bonienl Posted December 29, 2015 Author Share Posted December 29, 2015 I have updated the OT - a beta version is now available... Quote Link to comment
jumperalex Posted December 29, 2015 Share Posted December 29, 2015 Wow very excited as well. Subbed Quote Link to comment
abs0lut.zer0 Posted December 29, 2015 Share Posted December 29, 2015 for those of us trying to understand file integrity apps.. is this the same ? http://lime-technology.com/forum/index.php?topic=43396.msg414410#msg414410 Quote Link to comment
itimpi Posted December 29, 2015 Share Posted December 29, 2015 for those of us trying to understand file integrity apps.. is this the same ? http://lime-technology.com/forum/index.php?topic=43396.msg414410#msg414410 no. This is an alternative approach and only provides file integrity checking via checksumming (it has no recovery capability). It also stores the checksum information within the file system metadata so you do not see separate checksum files on the system. Quote Link to comment
bonienl Posted December 29, 2015 Author Share Posted December 29, 2015 The checksum utility of squid is also dealing with file integrity (or bit rot) detection, but is a different approach, as explained by itimpi. The approach of bunker is to store the relevant information in the extended attributes which go with the file. Besides the checksum value also the file modification date, file size and hash check date are stored in the extended attributes. This allows bunker to check for silent file corruption, i.e. the file content has changed but not the file modification date. A notification is sent and optional logging is done when such an event occurs. This utility only reports file corruption but does not have any means to repair it. This action is with the user. To use this utility you need to build first the extended attributes for all files you want to monitor, once this is in a place, any newly created or modified files get automatically their extended attribute information added. This is the function of inotifywait which tracks file changes and triggers bunker when this happens. The GUI plugin has the ability to visualize the background operations carried out by bunker. On the control page this is made visible, e.g. when starting a build operation you can follow here its proceedings, including an estimated time to finish for each disk which was included. And no worries leaving the page, all keeps on running and a revisit will show the current status. I've made a few fixes and released version 2015.12.29a. Those who do an upgrade, you must disable/enable the automatic hash function after installation to ensure that inotifywait is running (again). 1 Quote Link to comment
archedraft Posted December 30, 2015 Share Posted December 30, 2015 To use this utility you need to build first the extended attributes for all files you want to monitor, once this is in a place, any newly created or modified files get automatically their extended attribute information added. This is the function of inotifywait which tracks file changes and triggers bunker when this happens. Just installed and it looks great! Awesome work as always. I removed my old checksums and decided to use blake2 so it is recreating them as I type this. So inotifywait will add any new or modified file automatically? Quote Link to comment
bonienl Posted December 30, 2015 Author Share Posted December 30, 2015 To use this utility you need to build first the extended attributes for all files you want to monitor, once this is in a place, any newly created or modified files get automatically their extended attribute information added. This is the function of inotifywait which tracks file changes and triggers bunker when this happens. Just installed and it looks great! Awesome work as always. I removed my old checksums and decided to use blake2 so it is recreating them as I type this. So inotifywait will add any new or modified file automatically? inotifywait watches for file changes and whenever a new file or modified file is saved to disk, it triggers bunker with the necessary options to do the checksum calculation and add this to the extended attributes. This all goes transparantly for the user, and there should be minimum performance impact in the normal file operations. Quote Link to comment
bonienl Posted December 30, 2015 Author Share Posted December 30, 2015 I have made a new beta version 2015.12.30b available. Please update to this version. After installation you need to stop and start the automatic protection function (see settings page) to allow the updated version to work. Thanks for testing. Quote Link to comment
archedraft Posted December 30, 2015 Share Posted December 30, 2015 inotifywait watches for file changes and whenever a new file or modified file is saved to disk, it triggers bunker with the necessary options to do the checksum calculation and add this to the extended attributes. This all goes transparantly for the user, and there should be minimum performance impact in the normal file operations. Wow that's pretty slick. From your pictures above it looks like you are using blake2; however, the help section says that "this is a proprietary solution though". What (if any) are the potential risks of using blake2 being proprietary? Quote Link to comment
JorgeB Posted December 30, 2015 Share Posted December 30, 2015 I have to confess I didn’t really like the idea of adding the checksums to the file itself, maybe because I don’t understand how that works, but after trying it I really like how the plugin works. Great work as usual! PS: I believe that my AMD neon based HP N54Ls can’t handle Blake2, I will use SHA2, but I can select Blake2, shouldn’t this option be disable? Quote Link to comment
bonienl Posted December 30, 2015 Author Share Posted December 30, 2015 There is no risk in using BLAKE2, it is just not acknowledged as an official standard, though these guys promote it a lot. I am using BLAKE2 indeed, it is blistering fast. Quote Link to comment
bonienl Posted December 30, 2015 Author Share Posted December 30, 2015 I have to confess I didn’t really like the idea of adding the checksums to the file itself, maybe because I don’t understand how that works, but after trying it I really like how the plugin works. Great work as usual! PS: I believe that my AMD neon based HP N54Ls can’t handle Blake2, I will use SHA2, but I can select Blake2, shouldn’t this option be disable? The BLAKE2 package is installed together with the plugin, but the plugin can't tell whether it runs on your processor or not. Ps. The checksums are not added to the file itself, but to the extended attributes which are maintained by the OS, the original file is never touched. 1 Quote Link to comment
Squid Posted December 30, 2015 Share Posted December 30, 2015 I have to confess I didn’t really like the idea of adding the checksums to the file itself, maybe because I don’t understand how that works, but after trying it I really like how the plugin works. Great work as usual! PS: I believe that my AMD neon based HP N54Ls can’t handle Blake2, I will use SHA2, but I can select Blake2, shouldn’t this option be disable? The BLAKE2 package is installed together with the plugin, but the plugin can't tell whether it runs on your processor or not. Ps. The checksums are not added to the file itself, but to the extended attributes which are maintained by the OS, the original file is never touched. You can check to see if blake2 works at startup by running it against something and then disabling it depending upon the error code returned. I'll look at my code for this later tonight Quote Link to comment
bonienl Posted December 30, 2015 Author Share Posted December 30, 2015 Excellent idea, will add that and let me know what you have in mind. Meanwhile another update version 2015.12.30c is available with several corrections and enhancements. After installing this update you need to stop/start the service (see settings page) to make the changes active. I have changed the naming of the export files. If you have exported files earlier, you need to redo and discard the old ones (see also the release notes for more details). Quote Link to comment
JorgeB Posted December 30, 2015 Share Posted December 30, 2015 Little bug, with SHA2 selected in settings, Hashing method in File integrity control appears as none, works as expect with MD5 or BLAKE2. Quote Link to comment
bonienl Posted December 30, 2015 Author Share Posted December 30, 2015 Little bug, with SHA2 selected in settings, Hashing method in File integrity control appears as none, works as expect with MD5 or BLAKE2. ok, thanks it's corrected for a next release. Quote Link to comment
JorgeB Posted December 30, 2015 Share Posted December 30, 2015 I have to confess I didn’t really like the idea of adding the checksums to the file itself, maybe because I don’t understand how that works, but after trying it I really like how the plugin works. Great work as usual! PS: I believe that my AMD neon based HP N54Ls can’t handle Blake2, I will use SHA2, but I can select Blake2, shouldn’t this option be disable? The BLAKE2 package is installed together with the plugin, but the plugin can't tell whether it runs on your processor or not. Ps. The checksums are not added to the file itself, but to the extended attributes which are maintained by the OS, the original file is never touched. Well, it looks like BLAKE2 works after all on my CPU, I remember failing a test on Squid’s Checksum plugin, because SSE4.1 was required. Quote Link to comment
bonienl Posted December 30, 2015 Author Share Posted December 30, 2015 Hope you guys can keep up with the pace Version 2015.12.25d adds a new option, does a check on BLAKE2 support and some bug fixing. See the release notes for more details. Quote Link to comment
JorgeB Posted December 30, 2015 Share Posted December 30, 2015 Ps. The checksums are not added to the file itself, but to the extended attributes which are maintained by the OS, the original file is never touched. So if I were to copy some hashed files to a backup server running the same plugin I'd have to rehash them? Quote Link to comment
bonienl Posted December 30, 2015 Author Share Posted December 30, 2015 Ps. The checksums are not added to the file itself, but to the extended attributes which are maintained by the OS, the original file is never touched. So if I were to copy some hashed files to a backup server running the same plugin I'd have to rehash them? If you have the plugin running on your backup server, as soon as you copy files to it, they will get their checksum automatically added. One quick way to set all extended attributes on your backup server is to do an export command on your main server, copy the hash file over to your backup server and perform an import command. Quote Link to comment
JorgeB Posted December 30, 2015 Share Posted December 30, 2015 Ok, thanks, I think I get it now. Quote Link to comment
soana Posted December 31, 2015 Share Posted December 31, 2015 Whenever the control page is revisited it will update itself to the latest state. While running a build the statement above works for the same browser only. If you load the control page on a different browser or computer it will not show the latest state. Quote Link to comment
trurl Posted December 31, 2015 Share Posted December 31, 2015 I followed the bunker thread but never tried bunker because it didn't really fit with what I needed. I create NTFS disks containing specific unRAID shares for offsite backups. I assume the extended attributes hash would not transfer to the NTFS files, and I didn't want to have to manually export these every time I want to make a backup. I am currently using Squid's checksum since it creates separate hash files which get copied to NTFS along with everything else, and it works with shares or disks. I do like some aspects of the UI you have created for this though. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.