bitrot - a utility for generating sha256 keys for integrity checks (version 1.0)

[WeeboTech]

If memory serves, the issues I had were in the use of bracket [ ] characters in the file name.

 

 

You may have to escape them. It's a bit of a pain.  I remember having to do this this in perl and PHP when getting brackets in user forms.

[FreeMan]

I just ran

bitrot.sh -e -f <file> -p <path>

and it told me

All attributes created via bitrot.sh removed

I thought it was supposed to export the keys to a file, not remove them...

 

Obviously, I'm totally misunderstanding something, what's the value here?

[itimpi]

I just ran

bitrot.sh -e -f <file> -p <path>

and it told me

All attributes created via bitrot.sh removed

I thought it was supposed to export the keys to a file, not remove them...

 

Obviously, I'm totally misunderstanding something, what's the value here?

Are you sure you did not accidentally use -r instead? (the 'r' key is next to the 'e' key so could easily be mistyped).

 

It occurs to me that perhaps using both -e and -r in the same invocation should be treated as an error to avoid this type of problem.  Perhaps -r should also be invalid if the -0f option is present?

[FreeMan]

Hmmm... could have. I restarted the generation immediately, so the season is tied up right now. I'll check when it's done.

[jbartlett]

Does someone have a sample of the hash file extract that I can look at?

[bonienl]

Does someone have a sample of the hash file extract that I can look at?

 

There is no single format when it comes to hash output, even the same program can generate different output.

 

I settled for the following:

 

<hash value>|<full path/filename>|<scandate>

 

Using the single separator '|' makes it easy to split a line within the script. Also the <scandate> can be optional. And no risk that the 'pipe' character occurs in the path or file name.

 

This format can be easily translated to something else, for example for md5sum (if hashes are md5 values) you can do:

 

cut -d'|' -f1,2 hashfile | sed 's/|/  /' | md5sum -c

 

or

 

cut -d'|' -f1,2 hashfile | sed 's/|/ */' | md5sum -c

 

[jbartlett]

To fit that format, I'd just need to swap the hash & file name

[WeeboTech]

I would have to ask this, what's the purpose of exporting the 'other' extracted fields.

 

I think the export should be simpler for the end users.

 

The simple export should have the ability to pipe directly into the hasher without teaching the end user how to take apart the exported file.

An extended export could be something specific to the application storing data for importing on a copied file system, but for simpler end user history or testing it may be better to keep it simple.

 

I happen to like the way corz stores his extended fields as comments.

It's a little more complex to parse/reimport, but it keeps the data relevant for the hasher without interfering.

 

root@unRAID:/mnt/disk3/Music/music.mp3/Chill/Various Artists/Zen Deep Lounge# head folder.hash 
# made with checksum.. point-and-click hashing for windows (64-bit edition).
# from corz.org.. [url]http://corz.org/windows/software/checksum/[/url]
#
#md5#11 Dj Santero - Dont Go.mp3#[email protected]:20
00ef0e37ab33b07a91a32140f9ecb33d *11 Dj Santero - Dont Go.mp3
#md5#13 Dr Drummer Feat Maxim Beitan - Soul Cello.mp3#[email protected]:22
9decd38c6f34a79c938a1681004f1e56 *13 Dr Drummer Feat Maxim Beitan - Soul Cello.mp3
#md5#09 Eulises Gonzales - Deep Percussion.mp3#[email protected]:17
4a67a44d49cd34d18a5d1d4551b352e5 *09 Eulises Gonzales - Deep Percussion.mp3
#md5#10 The Sun Warrios - Sensual Groove.mp3#[email protected]:17

[bonienl]

I would have to ask this, what's the purpose of exporting the 'other' extracted fields.

 

The scandate is an extended attribute stored together with the hash value. It is used as a condition when using the '-d <days>' option to verify, update or remove attributes only for files scanned <days> or longer ago. Note this is not the same thing as the regular modified/accessed times.

 

The focus of 'my' tool isn't that much to rely on exported files, instead it verifies directly on the extended attributes and reports mismatches in separate files (with date identifier). These files in turn are used to do further action.

 

I guess I follow a bit different strategy as yours.

 

[FreeMan]

I just ran

bitrot.sh -e -f <file> -p <path>

and it told me

All attributes created via bitrot.sh removed

I thought it was supposed to export the keys to a file, not remove them...

 

Obviously, I'm totally misunderstanding something, what's the value here?

Are you sure you did not accidentally use -r instead? (the 'r' key is next to the 'e' key so could easily be mistyped).

 

It occurs to me that perhaps using both -e and -r in the same invocation should be treated as an error to avoid this type of problem.  Perhaps -r should also be invalid if the -0f option is present?

 

That's exactly what I did... "-r" instead of "-e".  Well, the good news is that the script isn't broken! 

 

I did, however, discover that the "--import" function has not yet been fully fleshed out. 

[jbartlett]

I did, however, discover that the "--import" function has not yet been fully fleshed out. 

 

Can you give details? I did test it pretty well and it worked then.

 

Though I suspect that programs suffer bitrot over time and stop working when they once did. If only there was a way to hash itself!

 

Edit: D'oh, I just recalled it was the recover function I tested and the import part slipped my mind

[FreeMan]

Whew! You had me doubting myself for a minute...

 

Glad I found this and ran it on all 10 of my physical drives. Now I get to test it because my UPS took a sudden dump on me. It just simply powered everything off without any power failure or indication of a battery issue.

 

I've brought the server back up without any UPS (prayers for smooth power for the next 48 hours or so would be appreciated), and a parity check has started. I'm going to run a -v to see if it reports any issues.

 

EDIT: One thought might be to have an option to write any mismatches to a file when doing a "-v" verify. I'm redirecting the entire output into a file, and that's workable. If it was just the mismatches, that would be easier.

[pkn]

... I use the find tool to locate files and I believe it only allows for one mask. ...

I'm not sure about slackware, but this usually works in bash on Redhat and SuSE:

find . \( -name '*.f' -o -name '*.F' -o -name '*.FIN' -o -name '*.f90' -o -name '*.fi' -o -name '*.fa' \) -print

[FreeMan]

I ran:

root@NAS:/boot/Scripts# bitrot.sh -v -p /mnt/disk2 > /mnt/cache/SHAverify_disk2.txt

and it gave me:

./bitrot.sh: line 465: [: -lt: unary operator expected

 

However, I haven't hit that error every time. So far, I've received the error on disk2 & disk3, but it ran cleanly on disk4 and so far on disk6.

 

Any ideas?

[BRiT]

I ran:

root@NAS:/boot/Scripts# bitrot.sh -v -p /mnt/disk2 > /mnt/cache/SHAverify_disk2.txt

and it gave me:

./bitrot.sh: line 465: [: -lt: unary operator expected

 

However, I haven't hit that error every time. So far, I've received the error on disk2 & disk3, but it ran cleanly on disk4 and so far on disk6.

 

Any ideas?

 

You most likely have a file name or directory name on disk 2 and disk3 with special characters in it, specifically a Single Quote ['], a Double Quote ["], a Space [ ], an Ampersand [&], or perhaps brackets '[', ']'.

[bonienl]

I ran:

root@NAS:/boot/Scripts# bitrot.sh -v -p /mnt/disk2 > /mnt/cache/SHAverify_disk2.txt

and it gave me:

./bitrot.sh: line 465: [: -lt: unary operator expected

 

However, I haven't hit that error every time. So far, I've received the error on disk2 & disk3, but it ran cleanly on disk4 and so far on disk6.

 

Any ideas?

 

You may want to try my alternative bunker, it uses the same attributes as bitrot, though syntax is a bit different. In your case it would be:

 

bunker -v -f /mnt/cache/SHAverify_disk2.txt /mnt/disk2

 

This would store any mismatches found on disk 2 in the specified file.

 

[jbartlett]

I ran:

root@NAS:/boot/Scripts# bitrot.sh -v -p /mnt/disk2 > /mnt/cache/SHAverify_disk2.txt

and it gave me:

./bitrot.sh: line 465: [: -lt: unary operator expected

 

However, I haven't hit that error every time. So far, I've received the error on disk2 & disk3, but it ran cleanly on disk4 and so far on disk6.

 

Any ideas?

 

You most likely have a file name or directory name on disk 2 and disk3 with special characters in it, specifically a Single Quote ['], a Double Quote ["], a Space [ ], an Ampersand [&], or perhaps brackets '[', ']'.

 

If you find that to be true, please post the file name in issue. You can scan certain directories to do a quick check to see if you can isolate which one.

[bonienl]

I ran:

root@NAS:/boot/Scripts# bitrot.sh -v -p /mnt/disk2 > /mnt/cache/SHAverify_disk2.txt

and it gave me:

./bitrot.sh: line 465: [: -lt: unary operator expected

 

However, I haven't hit that error every time. So far, I've received the error on disk2 & disk3, but it ran cleanly on disk4 and so far on disk6.

 

Any ideas?

 

You most likely have a file name or directory name on disk 2 and disk3 with special characters in it, specifically a Single Quote ['], a Double Quote ["], a Space [ ], an Ampersand [&], or perhaps brackets '[', ']'.

 

If you find that to be true, please post the file name in issue. You can scan certain directories to do a quick check to see if you can isolate which one.

 

The problem appears when there is no scandate attribute stored while the shakey attribute is present. You need to add an "empty" check first, e.g.:

if [ -n "$ShaDate" ] &&  [ $ShaDate -lt $ignoretoepoch ]; then

[tr0910]

Is the following true?

 

Bitrot works best on static files.  Not so good on working files that get edited frequently. 

 

If a text file has been edited by a text editor, the hash will fail on the next check.  And you won't know if the reason it fails is because of a simple edit, or a deeper and darker problem in your system. 

 

The only way to make sure hash fails are not false alarms would be to have the operating system generate a new hash upon every save of the file.

[jbartlett]

Is the following true?

 

Bitrot works best on static files.  Not so good on working files that get edited frequently. 

 

If a text file has been edited by a text editor, the hash will fail on the next check.  And you won't know if the reason it fails is because of a simple edit, or a deeper and darker problem in your system. 

 

The only way to make sure hash fails are not false alarms would be to have the operating system generate a new hash upon every save of the file.

 

Yes, and it is also true for any of the other bit hash programs too.