Checksum Suite


Squid

Recommended Posts

Updated to 2015.11.06

 

- Complete rewrite of verification script

Note Because of the new script, in order for pause on parity checks and save logs to flash to work within the verification system, you have to go to global settings, change any value and change it back then hit apply.

 

 

Tech Notes:

 

Due to an intermittent bug within PHP itself (not my code), an intermittent "Bus Error" (basically an attempt for PHP to access an odd memory location - 100% impossible for my code to directly cause, hence its a PHP bug) would happen in certain circumstances (Large number of hash files stored in an array, large number of hash files to actually check, and low memory), necessitated a rewrite of the actual verification loop into a bash script.

Link to comment

Updated to 2015.11.10

 

This release has optional PAR2 creation / verification / repair.

 

The PAR2 side of things would be Beta.  I released it just to see if I'm on the right track here.

 

Currently, its 100% a manual operation (no scheduling yet)

 

It is however fully functional and (basically) bug free.  There are some issues with regards to the logging, and verify / repair buttons, but it does work as advertised.

 

In a couple of days once its closer to RC status I'll actually explain what PAR2 is (for those who don't know) and how it may save your bacon, but in the meantime if you know and understand PAR2 feel free to play around.

 

Also, this is now a "Suite" of related apps, so rather than throw a couple of icons into User Utilities, a new section (Checksum Suite) is created within Settings.

 

Link to comment

Played a little with the new par2 creation, working great, many thanks!

 

I know this is a very early version but is it possible to save the settings like recursive and redundancy %?

Coming.  Also number of blocks.  More advanced features like block size, etc which are only useful for usenet probably won't make it into the GUI.
Link to comment

I haven't even gotten around to installing this yet and all I want to say is

Shut-up-and-take-my-money-300x188.jpg

 

Seriously at some point, somewhere even almost hidden, you should be accepting donations. I know I know, you're giving back, its a labor of love, etc etc etc. Like I said, shut up and ...

You can always email etransfers to [email protected] 

 

Seriously though, I really don't care.  This is how I relax.

Link to comment

Just a compatibility note here.

 

It appears that Blake2 requires the SSE4.1 CPU extensions.  Unfortunately, not all CPU's have these instruction extensions (in my case, my A6-3500, whereas my other 2 CPU's do have those extensions)

 

I will be issuing a patch to disallow the use of Blake2 on systems where it is incompatible.

 

If you have been using blake2 via this plugin, you can tell if you are affected by either looking at one of the generated hash files (there will be no hash within the file) or by entering in the following command:

 

/usr/local/emhttp/plugins/checksum/include/b2sum /usr/local/emhttp/plugins/checksum/include/b2sum

(an attempt to generate a checksum of itself)

 

If the command returns "Illegal Operation", then blake2 is incompatible with your system and you should switch to a different hash algorithm.

Link to comment

Just checked and my HP N54L servers (with AMD CPU) also can't do blake2, I was using the default md5 because I hadn't notice this from your notes on the 1st post:

 

Squid's Checksum Creator

Algorithm Either md5, sha1, sha256, or blake2 (actually blake2s for compatibility with corz)...

 

Since md5 checksums can still be checked by corz what's the disadvantage of using it instead of blake2?

Link to comment

md5 has the advantage of being arguably the most portable of all the algorithms (and the fastest at generation)

 

But, there is a greater chance of "collisions" with it than with sha1 / 256 or blake2.

 

A collision means that the original file if corrupted could wind up generating the same hash value as the non corrupted.  Collisions are possible under any hashing method.  Its just a tad more likely with md5 versus the other algorithms.

 

That being said, this is all mathematical abstractions, and in the real-world (ie: non cryptography applications) the chances of a corruption in a file generating the same hash value are slim to none.

 

In other words, unless you're working for the NSA I wouldn't particularly worry about it.  The flaws in MD5 / SHA1 are more concerning if someone is trying to crack your password than with file corruption.

Link to comment

Hi Squid,

 

quick question on my tower the plugin has now run for over 2 weeks and the initial manual movie run is still not finished, but I almost have the feeling that it's going over existing folders again, is there anything I can verify that it will finish? My drives have now been running non stop for over 2 weeks and are getting quite hot... so any help would be great.

Link to comment

At work right now (and life has been getting in the way of updates to the plugin), but the only way it would continually recalculate everything over and over again is if it was doing it as Blake2 but the system isn't capable of running Blake (i just found that a couple of days ago and have been very busy so haven't had a chance to release an update)

Link to comment

Hi Squid,

 

quick question on my tower the plugin has now run for over 2 weeks and the initial manual movie run is still not finished, but I almost have the feeling that it's going over existing folders again, is there anything I can verify that it will finish? My drives have now been running non stop for over 2 weeks and are getting quite hot... so any help would be great.

The way to tell would be to stop the monitor, look at one of your movie folders to see if the hash file is in there.

 

Then start the monitor, wait for the watches to be established and recreate the checksums.  Assuming that nothing has changed in the folders, it should recreate the hash files (the checksum log won't show the skipped folders, only the ones with changes)

Link to comment

Updated to 2015.11.16

 

- Continuing improvements to UI for PAR2

- Laid ground work for integrations of PAR2 into the scheduler system (no scheduled tasks for this just yet however)

- Disallow creation of Blake2 checksums for systems not able to support it.

 

The blake2 issue was a fluke that I found it, and unfortunately after the fix was programmed for it, that particular server got repurposed, so I am currently unable to easily test if it still properly disallows Blake2.  If someone could test that fails the command given here: http://lime-technology.com/forum/index.php?topic=43396.msg420079#msg420079.  When going to the share creation tab, Blake2 should no longer appear on systems that cannot support it.

 

If you were using blake2, and the system doesn't support it, then the net result is that an infinite loop on creation would happen which you can fix by stopping the monitor.  But, you will have to switch any shares that you were having using Blake2 to MD5 / SHA.  (To do this, you'll have to change the settings for that share - even though it will already say MD5).  Note that this only applies to users who fail the blake2 test and were using blake2.  If you weren't using Blake2 or you pass the test then nothing needs to be done.

 

Link to comment

 

 

The blake2 issue was a fluke that I found it, and unfortunately after the fix was programmed for it, that particular server got repurposed, so I am currently unable to easily test if it still properly disallows Blake2.  If someone could test that fails the command given here: http://lime-technology.com/forum/index.php?topic=43396.msg420079#msg420079.  When going to the share creation tab, Blake2 should no longer appear on systems that cannot support it.

 

Confirmed,  no longer appears on my server that doesn't support blake2

 

 

Link to comment

 

 

The blake2 issue was a fluke that I found it, and unfortunately after the fix was programmed for it, that particular server got repurposed, so I am currently unable to easily test if it still properly disallows Blake2.  If someone could test that fails the command given here: http://lime-technology.com/forum/index.php?topic=43396.msg420079#msg420079.  When going to the share creation tab, Blake2 should no longer appear on systems that cannot support it.

 

 

Confirmed,  no longer appears on my server that doesn't support blake2

Thanks.  If you had a share using it, make sure to change the algorithm from md5 to something else then back to md5 so that the change takes effect in the stored settings

 

Link to comment

small bug?

 

the timeout settings seem not to stick when you change them ...

i have several time set the timeout on my movie server to 30 minutes ... but every time i open the window to the global settings or checksum tools it always shows 1 minute....

 

so not sure if the changes stick for that option....

I'll check it out.  I'm assuming that you're hitting apply

 

Link to comment

yep... change the setting and click apply...

same happens with the inotify settings...

change it to 728000 hit apply....move to main window... come back ... nothing there any more

Works fine on both my servers  (Although there's a minor issue where changing the # queued events by itself doesn't activate the apply button).  Can you actually write to the flash drive?  Try editing config/plugins/checksum/settings/global.json  (the delay setting is in seconds ("Pause"))
Link to comment

HI

this is the content of the json file

 

{

    "Parity": true,

    "IgnoreHour": true,

    "Notify": true,

    "Success": true,

    "LogSave": true,

    "NumWatches": "720000",

    "NumQueue": "",

    "Pause": "1800"

}

 

as you can see that is correct

but see attached screenshot .. the values are not correct

 

 

 

checksum_screenshot.JPG.64d9d01175ddcd82faabecdd92515554.JPG

Link to comment
  • Squid locked this topic
  • Squid unlocked this topic

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.