Squid Posted October 12, 2015 Share Posted October 12, 2015 THIS PLUGIN IS DEPRECATED Squid's Checksum Creator A new tool to automatically (and manually if so desired) to create md5 / sha1 / sha256 / blake2s checksums for user selected shares. IE: To verify the integrity of the data store within your shares. (This is a more "old school" / traditional way to generate checksums vs the approach that bunker utilizes, along with offering a GUI for control, and is purposely designed with a limited feature set to be as user friendly as possible. If you require the advanced filtering controls that bunker offers (eg: by file size, etc) install that script) Paste this link into the Install Plugin Text Box: https://raw.githubusercontent.com/Squidly271/checksum/master/plugins/checksum.plg This plugin is only compatible wth 6.1+ (will not work at all under 6.0 due to design) and also requires the Nerd Pack Plugin (specifically inotify tools). If inotify tools is not installed, the plugin will complain and direct you to the forum thread for Nerd Pack. The plugin is specifically designed to output its data files to be compatible with corz checksum for windows (http://corz.org/windows/software/checksum/), although its generated files are compatible with md5sum, sha1sum, sha256sum, etc. Description of Settings: Share Settings http://i46.photobucket.com/albums/f109/squidaz/Untitled_zpssp76gt4y.png Share To Monitor A drop down list of unRaid's user shares. Note that appdata if it exists will not be displayed (to prevent constant scanning and creation of this share. If you set the share to Custom, then you can manually enter a path below. Note that "/" and "/mnt" are not allowed paths under any circumstances. This is because using /mnt would for instances scan every folder a minimum of 3 times, and a maximum of # of drives present + 2. Just a waste of resources. Note that for monitored shares, only the folder within the share that has changes is recalculated. Algorithm Either md5, sha1, sha256, or blake2 (actually blake2s for compatibility with corz). md5 is arguably the most portable of the bunch. Speed wise for calculations, there is very little difference between them. Note: if you will be using corz to check any file in the future, do NOT use sha256. Corz does't support it. Update Changed Files If when scanning the folder, the plugin detects that a file has changed (based upon the last scan time and the current file modification time), you have the option of either updating the checksum for the file or not. Single Checksum file per folderIf yes, then a single file containing all of the generated checksums within that folder will be generated. If no, a separate checksum file for each file will be generated Monitor Sets whether the share is automatically monitored for changes to it. Extension for checksum fileEither .hash or the .$algorithmUsed. If using corz checksum, you'd be better off with .hash since its already associated with corz. If not, it really doesn't matter which you choose Include all filesScan all files within the folder for new additions or changes. If set to no, then the Included / Excluded files are available. Included / Excluded FilesWildcards for files to include / exclude separated by a space EG: *.mkv *.avi *.mp3 etc. Global Settings http://i46.photobucket.com/albums/f109/squidaz/Untitled_zpsmvnqjpdi.png Pause during parity check / rebuild Self explanatory. Will pause the current job after the current file is finished (see Queue below) if it detects a parity check / rebuild is in progress. After the parity check is finished, the job queue will continue (within 10 minutes). Note: the current job in the queue must be completed before any changes to this setting take effect (IE: if its already paused for a parity check, you can't unpause it unless you stop the parity check) Pause before calculating This setting will pause a queued job before beginning for the selected amount of time. This is useful for waiting for the current writes to a new folder to finish before beginning to calculate the checksum's for it. It is also useful for cutting and pasting within windows to avoid having the plugin recalculate the checksums of the cut folder before its actually deleted. Changes to this setting take effect after the current job in the queue is completed. Apply The apply button operates a little out of the ordinary here. Each section (ie: global settings / each share section) has its own apply button. Any changes to that section requires that you hit the apply button for that section. Still working on alternatives here. Add To Queue (See queue below) Manually adds the selected share to the job queue. Note that without preexisting checksum files, only new / changed folders will be scanned. (may be useful for some). To generate checksum files for all of your existing files, you need to add it to the queue. Monitor Control http://i46.photobucket.com/albums/f109/squidaz/Untitled_zpssjxvlwq8.png After certain changes to share settings (monitor or not, and the share monitored - the plugin will tell you), the background process must be started over. Note: during this process, any jobs already queued will be lost (ie: wait for the status to say idle), and any changes that happen during the process will not be handled. Also, depending upon the complexity of your folder structure, it may take a minute or so. The on screen log display will show a message that its generating watches. After its completed, it will state Watches Established At that point the system is back up and running. Queue Because this plugin is designed to automatically monitor for changes to folders, everything operates under a FIFO queue system. (Otherwise you could potentially have a couple of hundred of instances of checksum generation running concurrently if you had Plex save the .nfo files to the array). The queue works like this: The monitoring script sends a message to the checksum generation. If the time between the message being sent and the job being executed is less than the Pause before calculation setting, then the script will pause for the appropriate time. In English: Assuming that the pause time is set to 10minutes: If at 1pm a job is started, that job will not start until 1:10pm. Let's say that the job took 3 minutes to run. But, if at 1:01pm another job was sent, then at the completion of the first job, it will only have to pause 8 minutes prior to it beginning. Every change to any monitored folder is classed as a job. The net result is that if you are updating tons of folders concurrently (eg: Plex writing out its .nfo files), is that the total delay is only going to be 10 minutes. Manually queuing up a folder is exempt from the pause (ie: if the plugin is idle for generating any checksums, then it will run immediately) Speed of Generation On both my systems, this plugin can pretty much match the sustained read speed of the drives with any of the algorithms. IE: On average for my media files I get between 160MB/s to 195MB/s (for drives which are rated at 210MB/s max transfer rate). The big upshot of monitoring changes as they happen is that the odds are still decent that unRaid may still have the freshly written file still cached in memory) In my case, for freshly written media files less than around 5GB, I get checksums generated at ~ 480MB/s for a file which is technically stored on my ancient 5400rpm hard drive. Your mileage will vary depending upon your hard drives, controllers, and available RAM. Preexisting checksum files If you already have preexisting files generated by corz or by another program, this plugin will read them to save time. The preexisting filename must match the file that this plugin would have generated. IE: If in single checksum mode, the preexisting file should be either filename.ext.hash (or .md5 / .sha1 / etc) depending upon what your extension settings are. In folder mode, the file name should be folder.hash (or .md5 / .sha1 / etc). Side note: In the course of generating this I discovered a bug with corz where the timestamp of the file does not always match the timestamp of the file. Sometimes its out by exactly an hour up or down. The timestamps that this plugin generates always matches the timestamp reported by windows and by unRaid. Net result is that this plugin thinks the file has been modified since corz ran against it, and if set to update mode it will regenerate it. Side note: If someone really wants me to incorporate bunker support, send me a copy of the exported file, and I should be able to generate a script to pre-write the checksum files. Coming within the next week: cron jobs for each non-monitored share to be added to the queue. Incorporate an apply-all button. GUI for automatically checking a portion (or all) of the generated checksum files for consistency. (cron job and percentage to check). (And whatever else the user-base suggests that meets the design goals of simplicity to use and manage) Technical Notes In order to limit this plugin's impact upon the responsiveness of your system, the generation of checksum files runs at a low priority (nice=12). The log file generated by this plugin is automatically capped at 500K, at which point it is deleted and started over. If you have the separate log window open and it stops responding yet the log display within the plugin is still continuing, the log has restarted. Close the window and re-open it. Change Log 2015.10.31 - Revised GUI 2015.10.29 - Fix monitor not starting after a reboot 2015.10.24 - Add in manual verifications, revised logging 2015.10.18 - Add in inotifywait settings 2015.10.12 - Initial Release 1 Quote Link to comment
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.