grandprix Posted September 2, 2014 Share Posted September 2, 2014 Even if you don't choose the MD5 algorithm, if you go with SHA, Blake, etc. whatever. What do you use to recursively create, verify, etc. the checksums of your files on your unRAID? I've come up with only two threads which seem to apply and those are: http://corz.org/windows/software/checksum/ and http://md5deep.sourceforge.net/ I've taken a peek at the first one, though I cannot immediately tell where it saves the hashes nor where it saves it (I'd like to be able to specify the location so that it doesn't write the hash files onto the array if possible?). I haven't taken a look at md5deep yet. I just wanted to get some ideas tossed out there so I can wake to them tomorrow and get busy. TBH. Also, surprised there isn't a plug-in for this very thing yet? Or is there and I just totally played ignorant in not finding it? I know there were a few that claimed they made the MD5's for their array(s), but I'd rather not PM those people, as then only I would benefit from it and that's pointless/selfish. Quote Link to comment
garycase Posted September 2, 2014 Share Posted September 2, 2014 I use the Corz Checksum utility. By default it stores the checksums for all the files in a folder as a ".hash" file in that folder. You can also configure it to save a single .hash file for an entire disk. Or you can create a checksum for a single file (the checksum is stored in the same folder as the file). I haven't modified this, but there is a "d" option that allows you to set the output directory for your hashes, so this should allow you to store the checksums in a location of your choice. I actually prefer to have the checksum in each folder, so if I want to verify a specific folder I can just right-click on it in Explorer, and select "Verify checksums" and it will validate the checksums for that folder. You can also do this at a higher level in the hierarchy (i.e. an entire disk) and it will validate the checksums for all folders below what you selected. Clearly I do all this from Windows. There are some utilities that have been listed on the forum that do this from Linux ... but I don't use those, so others will have to comment on how they use those. As for creating checksums for newly added media -- you could do that after you add it to the array; but I always create the checksum for the folder I'm adding BEFORE I copy it to the UnRAID server - then copy it to both UnRAID and my backup disk ... that way I can validate either very easily should it ever be necessary. Quote Link to comment
garycase Posted September 2, 2014 Share Posted September 2, 2014 ... You may want to look over the details on the Corz utility here: http://corz.org/windows/software/checksum/checksum-tricks.php Click on "Command-line switches" to see a list of all of the options you can set for the utility. As I noted earlier, I'm very happy with its default behavior, but you can make some significant modifications to this with these options. The only option I use is the verify option, so it always does a verify after it's created the checksum. Quote Link to comment
jumperalex Posted September 2, 2014 Share Posted September 2, 2014 Another thumbs up for Checksum. Go ahead and look into md5deep but I can tell you right now (and probably repeating the threads you read and my own posts) md5deep does not have all the built in functions that Checksum does. It generates and validates hases, that is all. It will not append new hashes (vs. regenerating the entire set of hashes), delete hashes of removed files, nor update hashes of changed files (vs just adding a new hash to the list leaving an orphaned hash). You can certainly script all that, but that is what Checksum does for you. I've pinged Corz to encourage him to re-attack the linux version of the checksum. It is missing a lot of functionality because he doesn't use it personally anymore. That said, the linux version can help you generate that initial set of hashes vice moving everything over your network for windows to generate. It is indeed a good bit faster. Then use the windows version for later management (synchronization) where only new files must move over the network. You can also use the linux version for a faster disaster check (i.e. after a parity fail). Finally ... if I get the big, or if you're so inclined, you could create a docker (or VM) with wine and checksum installed and then you can operate it all on unraid using VNC or RDS. Quote Link to comment
grandprix Posted September 2, 2014 Author Share Posted September 2, 2014 Excellent replies, I thank you all. As for Dockers, I haven't a clue about those. That is on the short list of things I want to accomplish with this "upgrade"/new build. However, in case something went awry, I wanted to ensure I had hashes of the files. I'll follow your notes concerning Checksum Gary, thank you much for that. I gave it a whirl late last night, while bobble heading, due to being so fatigued. I'll poke around with it more extensively tonight. Thanks again. Quote Link to comment
grandprix Posted September 2, 2014 Author Share Posted September 2, 2014 Yep. I must have been dead to the world. It was right there in front of my face (on the GUI), under output. Output directory. Well I feel foolish. I have it saving to one hash file (with full paths I assume) recursively. May take a little while on 1000+ files all 7GB+, though, I assume I'll be able to simply right click on the hash then select Checksum Verify whenever I need to verify the files? Quote Link to comment
garycase Posted September 2, 2014 Share Posted September 2, 2014 Yes, it will take a "while" (many hours) to do 1000+ files of over 7GB each !! The status line at the upper left of your monitor (assuming you haven't changed where to display it) will show you how far along it is ... but I wouldn't be surprised if it take 3-4 minutes/file, so you're likely looking at 2-3 days for it to finish. Verification is just as simple -- right-click and select "Verify checksums". Takes about 5 seconds of "your time" ... but of course many hours of the computer's time if you're verifying them all. Quote Link to comment
garycase Posted September 2, 2014 Share Posted September 2, 2014 ... by the way, WeeboTech's utility to catalog all the files, generate checksums, verify them, etc. sounds like a VERY nice addition !! Looking forward to when this is completed, primarily because running verifications natively on the UnRAID server will be appreciably faster than across the network. Quote Link to comment
WeeboTech Posted September 2, 2014 Share Posted September 2, 2014 Forked off here. RFC: MD5 checksum/Hash Software http://lime-technology.com/forum/index.php?topic=34988.0 Quote Link to comment
kegler Posted September 10, 2014 Share Posted September 10, 2014 Here is the one I have been using for the last couple years: HashCheck Shell Extension http://code.kliu.org/hashcheck/ It is a shell extension for Windows Explorer. But it can also generate a hashcheck file that can be used for verification. If the target is a different path you can just edit the file. Double clicking on the file launches the verification. It is Windows only, but it is Open Source. It is not near as complete as the Corz software appears to be, but it works for me. Quote Link to comment
Vetteman Posted December 30, 2022 Share Posted December 30, 2022 Do these utilities work in the same way as QuickPar or MultiiPar and allow you to recreate a corrupted file due to bitrot or malware? Quote Link to comment
JonathanM Posted December 30, 2022 Share Posted December 30, 2022 No, they simply show whether a file has changed since the checksum was originally generated. Quote Link to comment
Vetteman Posted December 30, 2022 Share Posted December 30, 2022 Can they be used along with MultiPar? Quote Link to comment
JonathanM Posted December 30, 2022 Share Posted December 30, 2022 Unclear what you are asking. Checksums are used to verify the file is unaltered. Multipar / Quickpar add extra files with additional data to allow replacement of a subset of the original files. If you have a valid multipar / quickpar set, there is no need for additional checksum utilities. 1 Quote Link to comment
Vetteman Posted December 30, 2022 Share Posted December 30, 2022 Multipar can read / open svf amd md5 files. Anyone know if there is a docker version of QPAR or Multipar? I checked the app store and couldn't find anything. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.