Jump to content
bonienl

bunker - yet another utility for file integrity checks

212 posts in this topic Last Reply

Recommended Posts

I'll see if I can find that nail  :)

This will make bunker awesome.

 

Then a plug in gui will make it accessible to the masses.

 

I too have struggled with the update of changed files issue. Thought it was just me not understanding it properly.

 

Thanks for continuing to move this forward.

 

#################

@weebo, @jbartlett you guys pay attention.  :) 

 

Bitrot needs this as well.

Share this post


Link to post

I'll see if I can find that nail  :)

Then a plug in gui will make it accessible to the masses.

 

In the short term it could just be a plugin installed through the unRAID GUI that just downloads the file and puts it in a standard location. That way we would get notifications when Bunker had an update.  ;D

 

But I do agree with @tr0910, a front end GUI would certainly make this much easier for new users. Honestly I believe checksums should be a core function of unRAID.

 

Just thought I would also say THANK YOU for your work! I appropriate you working on this in any capacity.

Share this post


Link to post

And I thought, I was only looking for a nail  ;)

Share this post


Link to post

And I thought, I was only looking for a nail  ;)

 

Maybe a nail and a lick of paint!! ;)

Share this post


Link to post

We will take the nail.

 

Then we can validate the tool.

 

If it's as awesome as I think it will be lots of people will clamor for a gui.

Share this post


Link to post

I've found a nail ... and started polishing it  :)

 

Updated version of bunker now also stores the last modified time of a file in the extended attributes. This allows bunker to determine whether a hash mismatch occured due to file corruption or due to file content modification.

 

To make use of this feature it is necessary that existing attribute information gets the new filedate attribute added. The easiest way is to use the new command -t (touch), e.g.

  bunker -t /mnt/disk1
  bunker -t /mnt/disk2
  ...
  bunker -t /mnt/disk9

Once the new filedate attribute is added, you may need to make new export files, e.g.

  bunker -e -f /boot/hash/disk1.txt /mnt/disk1
  bunker -e -f /boot/hash/disk2.txt /mnt/disk2
  ...
  bunker -e -f /boot/hash/disk9.txt /mnt/disk9

Using the -a (add) command will now automatically include the new attribute

  bunker -a /mnt/disk1

Updating and checking of files can follow two strategies, e.g

 

1. A complete check+update of all files on disk 1

  bunker -u -f /boot/hash/disk1.errors.txt /mnt/disk1

2. A partial check+update of files on disk 1 only newer than last scandate

  bunker -U -f /boot/hash/disk1.errors.txt /mnt/disk1

Please see the OT for downloading of the new version.

 

At this moment in time there is no GUI available, but who knows perhaps one day that nail becomes goldplated...

 

Share this post


Link to post

WAY TO GO!  ;D

 

So here is my strategy to get this automated. Would you mind double checking my logic to make sure it is actually doing what my intent is?

 

1. Schedule a daily cron that adds a checksum to new files :

bunker -a /mnt/disk1
bunker -a /mnt/disk2

 

2. Run a weekly cron script that checks the last scan date and updates if scan date has changed:

bunker -U -f /boot/hash/disk1.user-changes.txt /mnt/disk1
bunker -U -f /boot/hash/disk2.user-changes.txt /mnt/disk2

 

3. then after step 2 I can add a verify to the end of the script and it should only find corrupt files correct?

bunker -v -f /boot/hash/disk1.errors.txt /mnt/disk1
bunker -v -f /boot/hash/disk2.errors.txt /mnt/disk2

Share this post


Link to post

I think I found a bug or at least unexpected behavior on my end.

 

I created a test folder and created 7 blank .docx files.

 

I ran

/boot/bunker -a /mnt/user/BACKUP/TEST/

 

and it added all 7 files.

 

Then I opened up one file and added some text and saved it.

 

I ran

/boot/bunker -U /mnt/user/BACKUP/TEST/

 

but it now only shows 6 files and no updated?? I have to run another bunker -a for the modified file to be added, so it seems like whenever I edit my .docx file it looses its checksum. That shouldn't happen correct?

Share this post


Link to post

1. Daily update - ok, I do the same

 

2. Depending on how often you change/update files, you may want to make the bunker -U execution also daily.

 

3. bunker -V is the better choice here (this will update the scandate). Since this command will verify ALL files on the disk, it is better to run this at lower frequency, perhaps once a week or month.

 

Two tips:

- If you are going to put this in a cron job then adding the options -q (quiet) -l (logging) is helpful

- In the cron job you can start several instances of bunker, e.g. for each disk, and let them run concurrently by pushing the command to the background (&)

 

  bunker -a -q -l /mnt/disk1 &

  bunker -a -q -l /mnt/disk2 &

 

Share this post


Link to post

I think I found a bug or at least unexpected behavior on my end.

 

I created a test folder and created 7 blank .docx files.

 

I ran

/boot/bunker -a /mnt/user/BACKUP/TEST/

 

and it added all 7 files.

 

Then I opened up one file and added some text and saved it.

 

I ran

/boot/bunker -U /mnt/user/BACKUP/TEST/

 

but it now only shows 6 files and no updated?? I have to run another bunker -a for the modified file to be added, so it seems like whenever I edit my .docx file it looses its checksum. That shouldn't happen correct?

 

How do you edit and save the file, it sounds like the extended attributes are wiped ...

 

Share this post


Link to post

How do you edit and save the file, it sounds like the extended attributes are wiped ...

 

I agree it does sound like the extended attributes are getting wiped, I am opening the files with MS Word 2010 and then clicking save when I exit the file.

Share this post


Link to post

How do you edit and save the file, it sounds like the extended attributes are wiped ...

 

I agree it does sound like the extended attributes are getting wiped, I am opening the files with MS Word 2010 and then clicking save when I exit the file.

 

I suppose you have mapped the share as a disk under windows, but I am afraid that Windows applications are not aware of extended attributes (read: destroy them) when working this way.

 

Share this post


Link to post

 

How do you edit and save the file, it sounds like the extended attributes are wiped ...

 

I agree it does sound like the extended attributes are getting wiped, I am opening the files with MS Word 2010 and then clicking save when I exit the file.

 

I suppose you have mapped the share as a disk under windows, but I am afraid that Windows applications are not aware of extended attributes (read: destroy them) when working this way.

 

Dang it Windows! Always got to be a pain. I suppose that this shouldn't really matter because if I run a daily bunker -a any Windows modified files will be added again. Since the user modified the files they should be good for one day.

 

So I am still not fully understanding the difference between -v & -V. I'm confused on why/how the scan date effects a verify?

Share this post


Link to post

So I am still not fully understanding the difference between -v & -V. I'm confused on why/how the scan date effects a verify?

 

The difference happens when using the -U command. This command compares the file modification date against the scandate in the extended attributes.

 

When using the -v command the scandate is never updated, it stays as was made by the -a command, while the -V command updates the scandate to the date and time when the verification was done.

 

Though in your case - when loosing the extended attributes and recreating them - it doesn't make a difference !

 

Share this post


Link to post

So just to verify I modified the file with nano in CLI and bunker did update the file with -U so Windows is at fault.

 

Is there a way to test corrupt a file and then use bunker to see what it reports? I tried

dd if=/dev/urandom of=/mnt/user/BACKUP/TEST/New\ Microsoft\ Word\ Document.docx bs=1023 count=1

and MS Word will not open the file anymore but linux updated the time of the file when I did that so bunker said the file was good. Any way that you know of to corrupt a file without modifying the file timestamp?

Share this post


Link to post

So just to verify I modified the file with nano in CLI and bunker did update the file with -U so Windows is at fault.

 

Is there a way to test corrupt a file and then use bunker to see what it reports? I tried

dd if=/dev/urandom of=/mnt/user/BACKUP/TEST/New\ Microsoft\ Word\ Document.docx bs=1023 count=1

and MS Word will not open the file anymore but linux updated the time of the file when I did that so bunker said the file was good. Any way that you know of to corrupt a file without modifying the file timestamp?

 

When you use the -v command in this case, it should report a corruption -> file content has changed, but file timestamp stays the same.

 

Share this post


Link to post

-v reports the following if I run that dd command

Finished - verified 5 files, skipped 0 files. Found: 1 mismatches (0 corruptions).

 

I would have thought it should read 1 corruption

Share this post


Link to post

-v reports the following if I run that dd command

Finished - verified 5 files, skipped 0 files. Found: 1 mismatches (0 corruptions).

 

I would have thought it should read 1 corruption

 

It means the modified time of the file has been adjusted.

Share this post


Link to post

You would need to use touch to set the modification date back.  Try -

 

touch --date="2015-01-01 01:00:00" file_name

dd blah blah file_name

touch --date="2015-01-01 01:00:00" file_name

 

Share this post


Link to post

How do you edit and save the file, it sounds like the extended attributes are wiped ...

 

I agree it does sound like the extended attributes are getting wiped, I am opening the files with MS Word 2010 and then clicking save when I exit the file.

 

I suppose you have mapped the share as a disk under windows, but I am afraid that Windows applications are not aware of extended attributes (read: destroy them) when working this way.

 

Actually, that sounds ideal.  Is there any way to get unraid to wipe the attributes whenever the file is modified?  Then you wouldn't have to worry about checking file modification times at all.

Share this post


Link to post

You would need to use touch to set the modification date back.  Try -

 

touch --date="2015-01-01 01:00:00" file_name

dd blah blah file_name

touch --date="2015-01-01 01:00:00" file_name

 

Very nice  :D

verified 5 files, skipped 0 files. Found: 1 mismatches (1 corruptions)

Share this post


Link to post

Now to figure out a way to write a script that sends me an email with the log attached if it finds corruption.

Share this post


Link to post

Now to figure out a way to write a script that sends me an email with the log attached if it finds corruption.

 

I can make that an option in bunker to send notifications when corruption is detected.

 

Share this post


Link to post

Now to figure out a way to write a script that sends me an email with the log attached if it finds corruption.

 

I can make that an option in bunker to send notifications when corruption is detected.

That would be immense! :D

 

Can you also share with us what flags you're using on your setup?

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.