Jump to content
bonienl

bunker - yet another utility for file integrity checks

212 posts in this topic Last Reply

Recommended Posts

Added new extended attribute: file size

 

We will need to re-run bunker on all our existing files to add the new extended attribute correct? bunker -u  /mnt/disk1 is all that is needed to update the existing checksums?

Share this post


Link to post

Added new extended attribute: file size

 

We will need to re-run bunker on all our existing files to add the new extended attribute correct? bunker -u  /mnt/disk1 is all that is needed to update the existing checksums?

 

bunker -u /mnt/disk1 will work, but requires to recalculate the hash value for each file, a lengthy process.

 

There is a hidden command -T to just add file size to the extended attributes without recalculating the hash: bunker -T /mnt/disk1 will add the missing attribute. Afterwards you may want to do an export command to save all attributes including the new one in a file, e.g. bunker -e -f /boot/hashes/disk1.txt /mnt/disk1

Share this post


Link to post

There is a hidden command -T to just add file size to the extended attributes without recalculating the hash: bunker -T /mnt/disk1 will add the missing attribute.

 

Very nice, I feel so cool using hidden commands  8)

 

Afterwards you may want to do an export command to save all attributes including the new one in a file, e.g. bunker -e -f /boot/hashes/disk1.txt /mnt/disk1

 

I have been saving exports of my files on a regular basis per your advice, but I do have a question for you. Is the point of the export to compare the extended attributes checksum with the exported text file? Or what is the reasoning behind having a separate export file?

Share this post


Link to post

I have been saving exports of my files on a regular basis per your advice, but I do have a question for you. Is the point of the export to compare the extended attributes checksum with the exported text file? Or what is the reasoning behind having a separate export file?

 

The exported file can be used for different purposes:

 

1. Restore the extended attributes when they went missing (import command)

2. Check the extended attributes after file moves (check command)

 

An example: I converted my data disks from RFS to XFS, in this process I started with an empty disk formatted in XFS, then copied the contents of the next disk to the empty disk and repeated this action for all data disks.

 

I had made an export file of each disk before starting the above procedure.

After a disk was copied I run bunker -a /mnt/diskX to reapply the extended attributes (altenatively you can use a copy method which preserves the extended attributes).

Next I replaced in the export file the original disk number for the new disk number to reflect the new location

Next step was to check the files after the copy. i.e. bunker -c /mnt/diskX and ensure no file corruption

 

Share this post


Link to post

It looks like a small modification is required for 6.1 compatibility.

 

Currently the script is looking for 'notify' under /usr/local/sbin.    In the 6.1 release this is now at /usr/local/emhttp/scripts.    Sounds like an upfront check is needed for the location of this command to handle running on different unRAID versions?

Share this post


Link to post

It looks like a small modification is required for 6.1 compatibility.

 

Currently the script is looking for 'notify' under /usr/local/sbin.    In the 6.1 release this is now at /usr/local/emhttp/scripts.    Sounds like an upfront check is needed for the location of this command to handle running on different unRAID versions?

 

Thanks for noticing.

 

I've made an update which verifies the location of the 'notify' script and use appropriately.

 

Share this post


Link to post

This utility looks really useful.

 

Is there any GUI front-end (dynamix, docker??) for this?

No, not at the moment.

 

I put the idea of creating a plugin to run this on my list of 'nice-to-try-creating' items a few weeks ago.  However demands on my time have have meant I have not got any further, but it is still something I would like to try my hand at if time becomes available.

Share this post


Link to post

This utility looks really useful.

 

Is there any GUI front-end (dynamix, docker??) for this?

No, not at the moment.

 

I put the idea of creating a plugin to run this on my list of 'nice-to-try-creating' items a few weeks ago.  However demands on my time have have meant I have not got any further, but it is still something I would like to try my hand at if time becomes available.

 

It actually seems fairly easy to use via CLI, but a GUI would probably open it up to a lot more non-techie users.

 

I'm looking around for an easy solution to protect my unraid data from data degradation. I have a full copy of every unraid hard drive on another hard drive, but there's no integrity checking, hence my interest in this solution or something else (snapraid?).

Share this post


Link to post

Is anyone still using this in conjunction with the excellent dynamix plugin?  I have an external server that I want to verify the rsyncing process via bunker command line after exporting the keys from the production server and doing a

 

bunker -c -f /tmp/disk1_keys.txt

Share this post


Link to post

Is anyone still using this in conjunction with the excellent dynamix plugin?  I have an external server that I want to verify the rsyncing process via bunker command line after exporting the keys from the production server and doing a

 

bunker -c -f /tmp/disk1_keys.txt

 

bunker version 2 comes together with the plugin Dynamix File Integrity. There is no need to install it separately. The export files of bunker v1 and v2 are compatible.

Share this post


Link to post

bunker version 2 comes together with the plugin Dynamix File Integrity. There is no need to install it separately. The export files of bunker v1 and v2 are compatible.

 

Thank-you.  For those following this, it would appear that bunker no longer exists on your flash drive in executable format.  It gets installed here at boot time.  You can find bunker in:

 

root@Tower:/usr/local/emhttp/plugins/dynamix.file.integrity/scripts# bunker
bunker v2.5.1 - Copyright (c) 2015-2016 Bergware International

Usage: bunker -a|A|v|V|u|e|t|i|c|C|r|R [-fdDsSlLnq] [-md5|-b2] path [!] [mask]
  -a          add hash key attribute for files, specified in path and optional mask
  -A          same as -a option with implicit export function (may use -f)
  -v          verify hash key attribute and report mismatches (may use -f)
  -V          same as -v option with updating of mismatched keys (may use -f)
  -u          update mismatched or corrupted hash keys with new hash key attribute (may use -f)
  -e          export hash key attributes to the export file (may use -f)
  -t          touch file, i.e. copy file modified time to extended attribute
  -i          import hash key attributes from file and restore them (must use -f)
  -c          check hash key attributes from input file (must use -f)
  -C          same as -c option and correct mismatched hash key in extended attribute (must use -f)
  -r          remove hash key extended attribute from specified selection (may use -f)
  -R          same as -r option and remove all other values too (may use -f)

  -f <file>   optional set file reference to <file>. Defaults to /tmp/bunker.store.log
  -d <days>   optional only verify/update/remove files which were scanned <days> or longer ago
  -D <time>   optional only add/verify/update/export/remove files newer than <time>, time = NNs,m,h,d,w
  -s <size>   optional only include files smaller than <size>
  -S <size>   optional only include files greater than <size>
  -l          optional create log entry in the syslog file
  -L          optional, same as -l but only create log entry when changes are present
  -n          optional send notifications when file corruption is detected
  -q          optional quiet mode, suppress all output. Use for background processing
  -md5        optional use md5 hashing algorithm instead of sha256
  -b2         optional use blake2 hashing algorithm instead of sha256

  path        path to starting directory, mandatory with some exceptions (see examples)
  mask        optional filter for file selection. Default is all files
              when path or mask names have spaces, then place names between quotes
              precede mask with ! to change its operation from include to exclude

Examples:
bunker -a /mnt/user/tv                                 add SHA key for files in share tv
bunker -a -S 10M /mnt/user/tv                          add SHA key for files greater than 10 MB in share tv
bunker -a /mnt/user/tv *.mov                           add SHA key for .mov files only in share tv
bunker -a /mnt/user/tv ! *.mov                         add SHA key for all files in share tv except .mov files
bunker -A -f /tmp/keys.hash /mnt/user/tv               add SHA key for files in share tv and export to file keys.hash
bunker -v -n /mnt/user/files                           verify SHA key for previously scanned files and send notifications
bunker -V /mnt/user/files                              verify SHA key for scanned files and update any mismatches
bunker -v -d 90 /mnt/user/movies                       verify SHA key for files scanned 90 days or longer ago
bunker -v -f /tmp/errors.hash /mnt/user/movies         verify SHA key and save mismatches in file errors.hash
bunker -u  /mnt/disk1                                  update SHA key for mismatching files
bunker -u -D 12h /mnt/disk1                            update SHA key for mismatching files created in the last 12 hours
bunker -e -f /tmp/disk1_keys.hash /mnt/disk1           export SHA key to file disk1_keys.hash
bunker -i -f /tmp/disk1_keys.hash                      import and restore SHA key from user defined file - no path
bunker -c -f /tmp/disk1_keys.hash                      check SHA key from user defined input file - no path
bunker -C -f /tmp/disk1_keys.hash                      check SHA key and correct mismatched attribute (omit corruptions) - no path
bunker -r  /mnt/user/tv                                remove SHA key for files in share tv
bunker -r -f /tmp/errors.hash                          remove SHA key for files listed in file errors.hash - no path
root@Tower:/usr/local/emhttp/plugins/dynamix.file.integrity/scripts#

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.