MD5 checksum sofware - what do you use?

[grandprix]

Even if you don't choose the MD5 algorithm, if you go with SHA, Blake, etc. whatever.  What do you use to recursively create, verify, etc. the checksums of your files on your unRAID?

 

I've come up with only two threads which seem to apply and those are:

 

http://corz.org/windows/software/checksum/

 

and

 

http://md5deep.sourceforge.net/

 

I've taken a peek at the first one, though I cannot immediately tell where it saves the hashes nor where it saves it (I'd like to be able to specify the location so that it doesn't write the hash files onto the array if possible?).  I haven't taken a look at md5deep yet.

 

I just wanted to get some ideas tossed out there so I can wake to them tomorrow and get busy. TBH.

 

Also, surprised there isn't a plug-in for this very thing yet?  Or is there and I just totally played ignorant in not finding it?

 

I know there were a few that claimed they made the MD5's for their array(s), but I'd rather not PM those people, as then only I would benefit from it and that's pointless/selfish.

 

[garycase]

I use the Corz Checksum utility.

 

By default it stores the checksums for all the files in a folder as a ".hash" file in that folder.  You can also configure it to save a single .hash file for an entire disk.    Or you can create a checksum for a single file (the checksum is stored in the same folder as the file).  I haven't modified this, but there is a "d" option that allows you to set the output directory for your hashes, so this should allow you to store the checksums in a location of your choice.  I actually prefer to have the checksum in each folder, so if I want to verify a specific folder I can just right-click on it in Explorer, and select "Verify checksums" and it will validate the checksums for that folder.    You can also do this at a higher level in the hierarchy (i.e. an entire disk) and it will validate the checksums for all folders below what you selected.

 

Clearly I do all this from Windows.    There are some utilities that have been listed on the forum that do this from Linux ... but I don't use those, so others will have to comment on how they use those.

 

As for creating checksums for newly added media -- you could do that after you add it to the array; but I always create the checksum for the folder I'm adding BEFORE I copy it to the UnRAID server - then copy it to both UnRAID and my backup disk ... that way I can validate either very easily should it ever be necessary.

 

 

 

[garycase]

... You may want to look over the details on the Corz utility here:

http://corz.org/windows/software/checksum/checksum-tricks.php

 

Click on "Command-line switches" to see a list of all of the options you can set for the utility.  As I noted earlier, I'm very happy with its default behavior, but you can make some significant modifications to this with these options.    The only option I use is the verify option, so it always does a verify after it's created the checksum.

 

[jumperalex]

Another thumbs up for Checksum.  Go ahead and look into md5deep but I can tell you right now (and probably repeating the threads you read and my own posts) md5deep does not have all the built in functions that Checksum does.  It generates and validates hases, that is all.  It will not append new hashes (vs. regenerating the entire set of hashes), delete hashes of removed files, nor update hashes of changed files (vs just adding a new hash to the list leaving an orphaned hash). 

 

You can certainly script all that, but that is what Checksum does for you.  I've pinged Corz to encourage him to re-attack the linux version of the checksum.  It is missing a lot of functionality because he doesn't use it personally anymore.  That said, the linux version can help you generate that initial set of hashes vice moving everything over your network for windows to generate.  It is indeed a good bit faster.  Then use the windows version for later management (synchronization) where only new files must move over the network.  You can also use the linux version for a faster disaster check (i.e. after a parity fail).

 

Finally ... if I get the big, or if you're so inclined, you could create a docker (or VM) with wine and checksum installed and then you can operate it all on unraid using VNC or RDS.

[grandprix]

Excellent replies, I thank you all.

 

As for Dockers, I haven't a clue about those.  That is on the short list of things I want to accomplish with this "upgrade"/new build.  However, in case something went awry, I wanted to ensure I had hashes of the files.  I'll follow your notes concerning Checksum Gary, thank you much for that.  I gave it a whirl late last night, while bobble heading, due to being so fatigued.

 

I'll poke around with it more extensively tonight.  Thanks again.

 

[grandprix]

Yep.  I must have been dead to the world.  It was right there in front of my face (on the GUI), under output.  Output directory.  Well I feel foolish.

 

I have it saving to one hash file (with full paths I assume) recursively.  May take a little while on 1000+ files all 7GB+, though, I assume I'll be able to simply right click on the hash then select Checksum Verify whenever I need to verify the files?

 

[garycase]

Yes, it will take a "while" (many hours) to do 1000+ files of over 7GB each !!    The status line at the upper left of your monitor (assuming you haven't changed where to display it) will show you how far along it is ... but I wouldn't be surprised if it take 3-4 minutes/file, so you're likely looking at 2-3 days for it to finish.

 

Verification is just as simple -- right-click and select "Verify checksums".  Takes about 5 seconds of "your time" ... but of course many hours of the computer's time if you're verifying them all.

 

[garycase]

... by the way, WeeboTech's utility to catalog all the files, generate checksums, verify them, etc. sounds like a VERY nice addition !!  Looking forward to when this is completed, primarily because running verifications natively on the UnRAID server will be appreciably faster than across the network.

 

[WeeboTech]

Forked off here.

RFC: MD5 checksum/Hash Software

http://lime-technology.com/forum/index.php?topic=34988.0

[kegler]

Here is the one I have been using for the last couple years:

 

HashCheck Shell Extension

 

http://code.kliu.org/hashcheck/

 

It is a shell extension for Windows Explorer.  But it can also generate a hashcheck file that can be used for verification.  If the target is a different path you can just edit the file.  Double clicking on the file launches the verification.

 

It is Windows only, but it is Open Source.  It is not near as complete as the Corz software appears to be, but it works for me.

[Vetteman]

Do these utilities work in the same way as QuickPar or MultiiPar and allow you to recreate a corrupted file due to bitrot or malware?

[JonathanM]

No, they simply show whether a file has changed since the checksum was originally generated.

[Vetteman]

Can they be used along with MultiPar?

 

[JonathanM]

Unclear what you are asking. Checksums are used to verify the file is unaltered. Multipar / Quickpar add extra files with additional data to allow replacement of a subset of the original files. If you have a valid multipar / quickpar set, there is no need for additional checksum utilities.

[Vetteman]

Multipar can read / open svf amd md5 files. 

 

Anyone know if there is a docker version of QPAR or Multipar? 

I checked the app store and couldn't find anything.