bunker - yet another utility for file integrity checks


Recommended Posts

I'll see if I can find that nail  :)

This will make bunker awesome.

 

Then a plug in gui will make it accessible to the masses.

 

I too have struggled with the update of changed files issue. Thought it was just me not understanding it properly.

 

Thanks for continuing to move this forward.

 

#################

@weebo, @jbartlett you guys pay attention.  :) 

 

Bitrot needs this as well.

Link to comment

I'll see if I can find that nail  :)

Then a plug in gui will make it accessible to the masses.

 

In the short term it could just be a plugin installed through the unRAID GUI that just downloads the file and puts it in a standard location. That way we would get notifications when Bunker had an update.  ;D

 

But I do agree with @tr0910, a front end GUI would certainly make this much easier for new users. Honestly I believe checksums should be a core function of unRAID.

 

Just thought I would also say THANK YOU for your work! I appropriate you working on this in any capacity.

Link to comment

I've found a nail ... and started polishing it  :)

 

Updated version of bunker now also stores the last modified time of a file in the extended attributes. This allows bunker to determine whether a hash mismatch occured due to file corruption or due to file content modification.

 

To make use of this feature it is necessary that existing attribute information gets the new filedate attribute added. The easiest way is to use the new command -t (touch), e.g.

  bunker -t /mnt/disk1
  bunker -t /mnt/disk2
  ...
  bunker -t /mnt/disk9

Once the new filedate attribute is added, you may need to make new export files, e.g.

  bunker -e -f /boot/hash/disk1.txt /mnt/disk1
  bunker -e -f /boot/hash/disk2.txt /mnt/disk2
  ...
  bunker -e -f /boot/hash/disk9.txt /mnt/disk9

Using the -a (add) command will now automatically include the new attribute

  bunker -a /mnt/disk1

Updating and checking of files can follow two strategies, e.g

 

1. A complete check+update of all files on disk 1

  bunker -u -f /boot/hash/disk1.errors.txt /mnt/disk1

2. A partial check+update of files on disk 1 only newer than last scandate

  bunker -U -f /boot/hash/disk1.errors.txt /mnt/disk1

Please see the OT for downloading of the new version.

 

At this moment in time there is no GUI available, but who knows perhaps one day that nail becomes goldplated...

 

Link to comment

WAY TO GO!  ;D

 

So here is my strategy to get this automated. Would you mind double checking my logic to make sure it is actually doing what my intent is?

 

1. Schedule a daily cron that adds a checksum to new files :

bunker -a /mnt/disk1
bunker -a /mnt/disk2

 

2. Run a weekly cron script that checks the last scan date and updates if scan date has changed:

bunker -U -f /boot/hash/disk1.user-changes.txt /mnt/disk1
bunker -U -f /boot/hash/disk2.user-changes.txt /mnt/disk2

 

3. then after step 2 I can add a verify to the end of the script and it should only find corrupt files correct?

bunker -v -f /boot/hash/disk1.errors.txt /mnt/disk1
bunker -v -f /boot/hash/disk2.errors.txt /mnt/disk2

Link to comment

I think I found a bug or at least unexpected behavior on my end.

 

I created a test folder and created 7 blank .docx files.

 

I ran

/boot/bunker -a /mnt/user/BACKUP/TEST/

 

and it added all 7 files.

 

Then I opened up one file and added some text and saved it.

 

I ran

/boot/bunker -U /mnt/user/BACKUP/TEST/

 

but it now only shows 6 files and no updated?? I have to run another bunker -a for the modified file to be added, so it seems like whenever I edit my .docx file it looses its checksum. That shouldn't happen correct?

Link to comment

1. Daily update - ok, I do the same

 

2. Depending on how often you change/update files, you may want to make the bunker -U execution also daily.

 

3. bunker -V is the better choice here (this will update the scandate). Since this command will verify ALL files on the disk, it is better to run this at lower frequency, perhaps once a week or month.

 

Two tips:

- If you are going to put this in a cron job then adding the options -q (quiet) -l (logging) is helpful

- In the cron job you can start several instances of bunker, e.g. for each disk, and let them run concurrently by pushing the command to the background (&)

 

  bunker -a -q -l /mnt/disk1 &

  bunker -a -q -l /mnt/disk2 &

 

Link to comment

I think I found a bug or at least unexpected behavior on my end.

 

I created a test folder and created 7 blank .docx files.

 

I ran

/boot/bunker -a /mnt/user/BACKUP/TEST/

 

and it added all 7 files.

 

Then I opened up one file and added some text and saved it.

 

I ran

/boot/bunker -U /mnt/user/BACKUP/TEST/

 

but it now only shows 6 files and no updated?? I have to run another bunker -a for the modified file to be added, so it seems like whenever I edit my .docx file it looses its checksum. That shouldn't happen correct?

 

How do you edit and save the file, it sounds like the extended attributes are wiped ...

 

Link to comment

How do you edit and save the file, it sounds like the extended attributes are wiped ...

 

I agree it does sound like the extended attributes are getting wiped, I am opening the files with MS Word 2010 and then clicking save when I exit the file.

 

I suppose you have mapped the share as a disk under windows, but I am afraid that Windows applications are not aware of extended attributes (read: destroy them) when working this way.

 

Link to comment

 

How do you edit and save the file, it sounds like the extended attributes are wiped ...

 

I agree it does sound like the extended attributes are getting wiped, I am opening the files with MS Word 2010 and then clicking save when I exit the file.

 

I suppose you have mapped the share as a disk under windows, but I am afraid that Windows applications are not aware of extended attributes (read: destroy them) when working this way.

 

Dang it Windows! Always got to be a pain. I suppose that this shouldn't really matter because if I run a daily bunker -a any Windows modified files will be added again. Since the user modified the files they should be good for one day.

 

So I am still not fully understanding the difference between -v & -V. I'm confused on why/how the scan date effects a verify?

Link to comment

So I am still not fully understanding the difference between -v & -V. I'm confused on why/how the scan date effects a verify?

 

The difference happens when using the -U command. This command compares the file modification date against the scandate in the extended attributes.

 

When using the -v command the scandate is never updated, it stays as was made by the -a command, while the -V command updates the scandate to the date and time when the verification was done.

 

Though in your case - when loosing the extended attributes and recreating them - it doesn't make a difference !

 

Link to comment

So just to verify I modified the file with nano in CLI and bunker did update the file with -U so Windows is at fault.

 

Is there a way to test corrupt a file and then use bunker to see what it reports? I tried

dd if=/dev/urandom of=/mnt/user/BACKUP/TEST/New\ Microsoft\ Word\ Document.docx bs=1023 count=1

and MS Word will not open the file anymore but linux updated the time of the file when I did that so bunker said the file was good. Any way that you know of to corrupt a file without modifying the file timestamp?

Link to comment

So just to verify I modified the file with nano in CLI and bunker did update the file with -U so Windows is at fault.

 

Is there a way to test corrupt a file and then use bunker to see what it reports? I tried

dd if=/dev/urandom of=/mnt/user/BACKUP/TEST/New\ Microsoft\ Word\ Document.docx bs=1023 count=1

and MS Word will not open the file anymore but linux updated the time of the file when I did that so bunker said the file was good. Any way that you know of to corrupt a file without modifying the file timestamp?

 

When you use the -v command in this case, it should report a corruption -> file content has changed, but file timestamp stays the same.

 

Link to comment

How do you edit and save the file, it sounds like the extended attributes are wiped ...

 

I agree it does sound like the extended attributes are getting wiped, I am opening the files with MS Word 2010 and then clicking save when I exit the file.

 

I suppose you have mapped the share as a disk under windows, but I am afraid that Windows applications are not aware of extended attributes (read: destroy them) when working this way.

 

Actually, that sounds ideal.  Is there any way to get unraid to wipe the attributes whenever the file is modified?  Then you wouldn't have to worry about checking file modification times at all.

Link to comment

Now to figure out a way to write a script that sends me an email with the log attached if it finds corruption.

 

I can make that an option in bunker to send notifications when corruption is detected.

That would be immense! :D

 

Can you also share with us what flags you're using on your setup?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.