bunker - yet another utility for file integrity checks


Recommended Posts

sooo I am doing some testing with the bunker-daily script (sorry) and whenever I add a test file to disk1 and run bunker-daily I get the following errors. I think it is because it is trying and failing to create the new txt log of the added file. I will also attach my modified bunker-daily file, just in case I messed it up.

 

Your daily script looks alright.

 

The error comes from bunker itself, but I can't say what causes it. As a verification what version of bunker are you using (or perhaps redownload and try again) ?

 

Link to comment

I have a question with the bunker-monthly script. Does it ever get triggered at the same time as parity check? If so how do you handle that?

 

The line

 

if [[ $array == STARTED && $rsync -eq 0 ]]; then

 

Ensures that execution only takes place when the array is started and no parity operation is in progress.

 

The monthly cron is executed on the first day of the month at 04:40am, the easiest way to avoid conflicts is to schedule the parity check on another day.

 

Link to comment

The error comes from bunker itself, but I can't say what causes it. As a verification what version of bunker are you using (or perhaps redownload and try again) ?

Just a thought - should there be a parameter option that makes bunker display its version number?  Alternatively perhaps it could be standard behaviour if the -q option is not used?  That would make it much easier to determine what version of bunker one is actually running.

 

I can see that with the provision of some standardised scripts of the sort covered in the recent forum posts that setting up bunker becomes much easier for the average user.  You could perhaps include the latest version of the scripts in the standard bunker download to help those people just getting started?

 

FYI:  When I created my own version of your scripts I added to the top as comments the meaning of the various bunker parameter  options used in the script.  Provides a useful 'quick reminder' of what they are without the need to revert to the master bunker documentation.  If you include sample scripts in the standard download you might consider adding such comments to these sample scripts?

Link to comment
The error comes from bunker itself, but I can't say what causes it. As a verification what version of bunker are you using (or perhaps redownload and try again) ?

I have the same error messages and have just started with bunker (Thank you very much for all of your efforts bonienl), hence I'm running your latest version 1.10.

Link to comment

 

sooo I am doing some testing with the bunker-daily script (sorry) and whenever I add a test file to disk1 and run bunker-daily I get the following errors. I think it is because it is trying and failing to create the new txt log of the added file. I will also attach my modified bunker-daily file, just in case I messed it up.

 

Your daily script looks alright.

 

The error comes from bunker itself, but I can't say what causes it. As a verification what version of bunker are you using (or perhaps redownload and try again) ?

 

I'm running 1.10

Link to comment

Any luck on finding the culprit in bunker?

 

EDIT: so it has something to do with bunker and newly added files. Even if I remove the part of your daily script that adds the new files to the .txt file in the hash folder I get the error messages.

 

EDIT 2: Well it also happens if I modify the file. This is the last entry in my syslog before the errors occur.

bunker - warning: /mnt/disk3/MEDIA/test2.txt was modified, update sha256 hash. 

 

So moral of the story is don't add or modify files on your server and all is OK  ;D

 

EDIT 3: OK I do not get the error if I remove the "-q" flag! Question can I run theses scripts in cron without the -q flag?

Link to comment

Any luck on finding the culprit in bunker?

 

EDIT: so it has something to do with bunker and newly added files. Even if I remove the part of your daily script that adds the new files to the .txt file in the hash folder I get the error messages.

 

EDIT 2: Well it also happens if I modify the file. This is the last entry in my syslog before the errors occur.

bunker - warning: /mnt/disk3/MEDIA/test2.txt was modified, update sha256 hash. 

 

So moral of the story is don't add or modify files on your server and all is OK  ;D

 

EDIT 3: OK I do not get the error if I remove the "-q" flag! Question can I run theses scripts in cron without the -q flag?

 

Ok, your observations have helped me to determine the error. As a workaround you can run without the -q option.

 

Link to comment

Any luck on finding the culprit in bunker?

 

EDIT: so it has something to do with bunker and newly added files. Even if I remove the part of your daily script that adds the new files to the .txt file in the hash folder I get the error messages.

 

EDIT 2: Well it also happens if I modify the file. This is the last entry in my syslog before the errors occur.

bunker - warning: /mnt/disk3/MEDIA/test2.txt was modified, update sha256 hash. 

 

So moral of the story is don't add or modify files on your server and all is OK  ;D

 

EDIT 3: OK I do not get the error if I remove the "-q" flag! Question can I run theses scripts in cron without the -q flag?

 

Ok, your observations have helped me to determine the error. As a workaround you can run without the -q option.

 

Awesome, glad I could be of assistance.

Link to comment

I have also updated to v.1.11 and the errors from 1.10 disappeared. The advise that I would like to get is how to deal with my drive #1 that only contains 2 sparsebundle files for 2 Mac Time Machine backups:

Jul 17 11:12:49 Tower root: bunker - exported 5495 files from /mnt/disk1 with mask *. Duration: 00:05:03
Jul 17 11:13:56 Tower root: bunker - added 10 files from /mnt/disk2 with mask *. Duration: 00:01:07 Average speed: 149. MB/s
Jul 17 11:13:57 Tower root: bunker - updated 0 files from /mnt/disk2 with mask * (0 corruptions). Duration: 00:00:01
Jul 17 11:13:57 Tower root: bunker - exported 10 files from /mnt/disk2 with mask *. Duration: 00:00:00
Jul 17 11:13:57 Tower root: bunker - added 0 files from /mnt/disk3 with mask *. Duration: 00:00:00
Jul 17 11:13:57 Tower root: bunker - updated 0 files from /mnt/disk3 with mask * (0 corruptions). Duration: 00:00:00
Jul 17 11:13:57 Tower root: bunker - exported 0 files from /mnt/disk3 with mask *. Duration: 00:00:00
Jul 17 11:13:58 Tower root: bunker - added 0 files from /mnt/disk4 with mask *. Duration: 00:00:00
Jul 17 11:13:58 Tower root: bunker - updated 0 files from /mnt/disk4 with mask * (0 corruptions). Duration: 00:00:00
Jul 17 11:13:58 Tower root: bunker - exported 0 files from /mnt/disk4 with mask *. Duration: 00:00:00
Jul 17 11:13:58 Tower root: bunker - added 0 files from /mnt/disk5 with mask *. Duration: 00:00:00
Jul 17 11:13:58 Tower root: bunker - updated 0 files from /mnt/disk5 with mask * (0 corruptions). Duration: 00:00:00
Jul 17 11:13:58 Tower root: bunker - exported 0 files from /mnt/disk5 with mask *. Duration: 00:00:00
Jul 17 11:13:58 Tower root: bunker - added 0 files from /mnt/disk6 with mask *. Duration: 00:00:00
Jul 17 11:13:58 Tower root: bunker - updated 0 files from /mnt/disk6 with mask * (0 corruptions). Duration: 00:00:00
Jul 17 11:13:59 Tower root: bunker - exported 0 files from /mnt/disk6 with mask *. Duration: 00:00:00
Jul 17 11:13:59 Tower root: bunker - added 0 files from /mnt/disk7 with mask *. Duration: 00:00:00
Jul 17 11:13:59 Tower root: bunker - updated 0 files from /mnt/disk7 with mask * (0 corruptions). Duration: 00:00:00

 

So two questions:

  1. [*]would you exclude disk1 (the Time Machine drive)?

[*]why is the amount of files new/updated so low? This was the first time that I ran this script

 

Remark: I had to shorten the file disk1.export.txt - too large to attach.

Link to comment

Looks like there could be a buglet in displaying the initial estimated time to complete?

 

I am creating the initial checksums for a 6TB disk, and I started by getting the following displayed:

Currently processing file 2 of 3539. Completed: 0.1% [ETA: 13695459:24:16]

which is rather an excessive time :)

 

This later corrected itself to:

Currently processing file 10 of 3539. Completed: 0.3% [ETA: 29:03:57]

 

Link to comment

So two questions:

  1. [*]would you exclude disk1 (the Time Machine drive)?

[*]why is the amount of files new/updated so low? This was the first time that I ran this script

 

1. It is  entirely up to you which files you like to include/exclude. The bunker-daily script allows for skipping of disks if needed, while bunker itself can be given an exclude filter (use !)

 

2. The option -D 1 tells bunker only to search for files created in the last 24 hours , for an initial calculation you need to remove this option.

 

Link to comment

Could I request and enhancement to the -l option (perhaps -L) where only lines with non-zero values are written to the log.  At the moment with 20 disks in the system I get a lot of entries reporting 0 changes.  It would be easier to pick out significant entries if only the non-zero entries were logged.

Link to comment

Here is a write error from the middle of an add operation on an RFS disk. 

 

/boot/config/bunker -A -l -f /boot/logs/cronlogs/bunker21keys.log /mnt/disk2
Currently processing file 1923 of 3880. Completed: 18.7% [ETA: 02:16:55]
Calculating sha256 hash... - /boot/config/bunker: line 86: echo: write error: InFinished - added 3880 files. Duration: 00:38:32 Average speed: 3.76 MB/s

Link to comment

Could I request and enhancement to the -l option (perhaps -L) where only lines with non-zero values are written to the log.  At the moment with 20 disks in the system I get a lot of entries reporting 0 changes.  It would be easier to pick out significant entries if only the non-zero entries were logged.

 

I created version 1.12 which has a new option -L, this allows bunker to create log entries only when changes are present. See OT for download.

 

As a convenience, together with the bunker script I have also included the bunker-daily and bunker-monthly scripts in the zip file.

 

Link to comment

Here is a write error from the middle of an add operation on an RFS disk. 

 

/boot/config/bunker -A -l -f /boot/logs/cronlogs/bunker21keys.log /mnt/disk2
Currently processing file 1923 of 3880. Completed: 18.7% [ETA: 02:16:55]
Calculating sha256 hash... - /boot/config/bunker: line 86: echo: write error: InFinished - added 3880 files. Duration: 00:38:32 Average speed: 3.76 MB/s

 

This looks like an error while writing the extended file attributes. Are you using ReiserFS ?

 

Link to comment

Could I request and enhancement to the -l option (perhaps -L) where only lines with non-zero values are written to the log.  At the moment with 20 disks in the system I get a lot of entries reporting 0 changes.  It would be easier to pick out significant entries if only the non-zero entries were logged.

 

I created version 1.12 which has a new option -L, this allows bunker to create log entries only when changes are present. See OT for download.

Thanks for adding that option.  I will test it out today.

Link to comment

A small improvement to your daily script is to include the lines at the top of the for loop

  for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 ; do
    # Only operate on disks that are actually present in this unRAID system
    if [ -e /mnt/disk$i ] ; then

as that removes any need to alter the script to specify the exact number of disks in the system.

 

Also I was wondering if the -D 1 line should specify a larger value in case a parity check was running the last time this script was run and it therefore exited without doing anything?  It seems to me that the current version of the script would omit calculating the checksum for that missed day.  In fact on a disk where all the checksums are already added, that parameter does not seem to speed things up much at all so maybe it is mainly useful when first building up the checksums?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.