bunker - yet another utility for file integrity checks


Recommended Posts

A small improvement to your daily script is to include the lines at the top of the for loop

  for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 ; do
    # Only operate on disks that are actually present in this unRAID system
    if [ -e /mnt/disk$i ] ; then

as that removes any need to alter the script to specify the exact number of disks in the system.

 

Also I was wondering if the -D 1 line should specify a larger value in case a parity check was running the last time this script was run and it therefore exited without doing anything?  It seems to me that the current version of the script would omit calculating the checksum for that missed day.  In fact on a disk where all the checksums are already added, that parameter does not seem to speed things up much at all so maybe it is mainly useful when first building up the checksums?

 

Thanks for the suggestion about the disks , I will update my examples. Small gotcha ... omitting disks still requires to alter the list!

 

It is true that -D 1 assumes the script runs on daily basis and if a day is skipped for whatever reason it would miss information. Changing to -D 2 would prevent that situation.

 

Leaving the -D option out has only an impact on execution time when there are many files to be examined. For example I have one disk with 300K+ files and running bunker without -D option takes about 45 minutes, while the same thing with -D 1 is a few seconds.

 

Link to comment

Thanks for the suggestion about the disks , I will update my examples. Small gotcha ... omitting disks still requires to alter the list!

Good point.  I will have to bear that in mind.

 

It is true that -D 1 assumes the script runs on daily basis and if a day is skipped for whatever reason it would miss information. Changing to -D 2 would prevent that situation.

At least my assumption is correct and I was not misreading things.

 

Leaving the -D option out has only an impact on execution time when there are many files to be examined. For example I have one disk with 300K+ files and running bunker without -D option takes about 45 minutes, while the same thing with -D 1 is a few seconds.

Fair enough.  I must admit I had only tested it on disks which were primarily video type media files and the total was less than about 10,000 files.  Not tried it yet on my music files or software archives where the number would be much higher.

 

I have been taking your scripts and reworking them so that I have

  • a bunker-update.sh script based on your daily one that runs checking for recent changes and handles the adding/updating of the checksums.  I am assuming initially that it is run hourly and using -D 6h so that there is quite a bit of overlap but the execution time should still be quite short.[br][br]I was thinking of also having a parameter option to put no time limit on it.  The idea is that this one could be used for an initial creation of checksums. It is likely to run for days/weeks on my system with 20 drives to work on but it will take the tedium out of typing commands.
     
  • a bunker-verify.sh script based on your monthly one that handles verification.  The idea is that it is run daily and verifies one disk per day and the disk selected varies each day.  This should give a more frequent check than your monthly script as each disk is checked once per month, while still ensuring that there is not too much load on the system.  To avoid the most likely times for parity checks I am avoiding the first and last days of the months as candidates for checking the disks.  Can you see any reason why this is not a good approach to handling the verification?

I am more than happy to pass these scripts back for others to use if they are likely to prove useful.  Any thoughts on this, and also am I perhaps likely to be duplicating work you are already planning?

 

Assuming the above scripts work out OK, then I was thinking of trying my hand at a plugin to put a GUI front-end on the installation, configuring and scheduling execution of bunker. If I get that far (which is not at all guaranteed :) ) do you have any objections about me including the bunker binary as a download in such a plugin or should users just be prompted to install it manually? 

 

BTW:  Are there any useful exit codes from the bunker binary that it might be worth checking for?  I was thinking in particular of codes indicating error conditions.

Link to comment

Here is a write error from the middle of an add operation on an RFS disk. 

 

/boot/config/bunker -A -l -f /boot/logs/cronlogs/bunker21keys.log /mnt/disk2
Currently processing file 1923 of 3880. Completed: 18.7% [ETA: 02:16:55]
Calculating sha256 hash... - /boot/config/bunker: line 86: echo: write error: InFinished - added 3880 files. Duration: 00:38:32 Average speed: 3.76 MB/s

 

This looks like an error while writing the extended file attributes. Are you using ReiserFS ?

 

Yes, I am using RFS on this disk.

 

Can we conclude with certainty that the integrity of the data on our unRaid system requires that we run bunker on XFS drives? 

 

This is a big job to convert all those disks from RFS to XFS on my main server and all backup servers.  (40 drives and millions of JPG files)

 

Link to comment

...  I am more than happy to pass these scripts back for others to use if they are likely to prove useful.  Any thoughts on this, and also am I perhaps likely to be duplicating work you are already planning?

 

Put your ideas forward, there are definitely more ways to automate the script. Mine certainly is not the only (or best) way.

 

Assuming the above scripts work out OK, then I was thinking of trying my hand at a plugin to put a GUI front-end on the installation, configuring and scheduling execution of bunker. If I get that far (which is not at all guaranteed :) ) do you have any objections about me including the bunker binary as a download in such a plugin or should users just be prompted to install it manually? 

 

When you make a GUI then a good approach is to let the plugin download the zip file from the forum, I have done something similar for cache_dirs. This ensures that people installing the plugin get the latest version of bunker.

 

BTW:  Are there any useful exit codes from the bunker binary that it might be worth checking for?  I was thinking in particular of codes indicating error conditions.

 

There is no exit codes to indicate different failures, results are stored in a log file.

 

Btw bunker is a shell script not a true binary.

 

 

Link to comment

Here is a write error from the middle of an add operation on an RFS disk. 

 

/boot/config/bunker -A -l -f /boot/logs/cronlogs/bunker21keys.log /mnt/disk2
Currently processing file 1923 of 3880. Completed: 18.7% [ETA: 02:16:55]
Calculating sha256 hash... - /boot/config/bunker: line 86: echo: write error: InFinished - added 3880 files. Duration: 00:38:32 Average speed: 3.76 MB/s

 

This looks like an error while writing the extended file attributes. Are you using ReiserFS ?

 

Yes, I am using RFS on this disk.

 

Can we conclude with certainty that the integrity of the data on our unRaid system requires that we run bunker on XFS drives? 

 

This is a big job to convert all those disks from RFS to XFS on my main server and all backup servers.  (40 drives and millions of JPG files)

 

I can only share my experiences with you. Moving from RFS to XFS in my case improved stability, I did not encounter issues with running bunker on XFS, but had some lock ups in the past with RFS.

 

Link to comment

...  I am more than happy to pass these scripts back for others to use if they are likely to prove useful.  Any thoughts on this, and also am I perhaps likely to be duplicating work you are already planning?

 

Put your ideas forward, there are definitely more ways to automate the script. Mine certainly is not the only (or best) way.

OK - I should have something to look at shortly.

 

Assuming the above scripts work out OK, then I was thinking of trying my hand at a plugin to put a GUI front-end on the installation, configuring and scheduling execution of bunker. If I get that far (which is not at all guaranteed :) ) do you have any objections about me including the bunker binary as a download in such a plugin or should users just be prompted to install it manually? 

When you make a GUI then a good approach is to let the plugin download the zip file from the forum, I have done something similar for cache_dirs. This ensures that people installing the plugin get the latest version of bunker.

That was what I was thinking of doing - just checking you were happy before I made any assumptions.

 

BTW:  Are there any useful exit codes from the bunker binary that it might be worth checking for?  I was thinking in particular of codes indicating error conditions.

 

There is no exit codes to indicate different failures, results are stored in a log file.

 

Btw bunker is a shell script not a true binary.

I had not realised it was a script.  I guess that rather than simply suggesting changes I can try providing a patch for you to approve and/or validate.  If I still think some exit codes might be useful I at least know how to add such things to a script  :)

Link to comment

For your net revision (if any) of bunker I would suggest a small tweak:

  • Adding '--tag bunker' as a parameter to logger in your Logger function
  • Removing the prefix 'bunker -' from the text of the message in all the calls to Logger

It makes the syslog output look a little tidier and is more consistent with the style of other messages written to the syslog.  If you would prefer a patch to save you work I can produce one for you.

Link to comment

For your net revision (if any) of bunker I would suggest a small tweak:

  • Adding '--tag bunker' as a parameter to logger in your Logger function
  • Removing the prefix 'bunker -' from the text of the message in all the calls to Logger

It makes the syslog output look a little tidier and is more consistent with the style of other messages written to the syslog.  If you would prefer a patch to save you work I can produce one for you.

 

Yeah, make your changes and send an update to me. I will make a new version available in the OT.

 

Link to comment

For your net revision (if any) of bunker I would suggest a small tweak:

  • Adding '--tag bunker' as a parameter to logger in your Logger function
  • Removing the prefix 'bunker -' from the text of the message in all the calls to Logger

It makes the syslog output look a little tidier and is more consistent with the style of other messages written to the syslog.  If you would prefer a patch to save you work I can produce one for you.

OK - here is the bunker script with the changes I mentioned.  I also changed the version number to 1.13 on the assumption that was likely to be what was wanted.

 

Yeah, make your changes and send an update to me. I will make a new version available in the OT.

bunker.zip

Link to comment

This looks like an error while writing the extended file attributes. Are you using ReiserFS ?

 

Yes, I am using RFS on this disk.

Can we conclude with certainty that the integrity of the data on our unRaid system requires that we run bunker on XFS drives? 

This is a big job to convert all those disks from RFS to XFS on my main server and all backup servers.  (40 drives and millions of JPG files)

 

I have converted my first disk from RFS to XFS and then verified the copied over files with

 

rsync -avXi --log-file=/boot/XFSdisk2.txt /mnt/disk2 /mnt/disk12

bunker -v /mnt/disk12

 

40 corruptions found, but checking the files with Windows File Compare (FC /b), they compare exactly.  I expect the hashes are wrong on these 40 files because of the RFS write to extended attr bug??  I still have the 2 disks and can do some testing if you like.

 

Event: unRAID file corruption
Subject: Notice [TOWER5] - bunker verify command
Description: Found 40 files with sha256 hash corruption
Importance: alert

f6235c5deb3fa2d6689a9e676199f1f29763a35c911de47a0ff5452751b3a5b4|/mnt/disk12/disk2/Backups/WorkstationCatalog/WorkstationCatalog.sqlite|1437369145|1437165044.
e14f9927844234413cf951cd21e094c3d354592bb6f4a6daa67ec5d380099a7c|/mnt/disk12/disk2/Documents/Design Revolution's Stuff/Images/Templates/._Wordpress Images_facebook.psd|1437369145|1436548377.
50621fef5d0228d3e9c6aedd92b3daff42e94bd092d34d7e7ba1b602a22a761a|/mnt/disk12/disk2/Documents/Design Revolution's Stuff/Images/Templates/Wordpress Images_facebook.psd|1437369145|1436548375.
8dc7793c56f865a0f3a392a119c48b93a262255a12d552ff94a5d28ebb3fe4bb|/mnt/disk12/disk2/Documents/Design Revolution's Stuff/Images/Videos/Cake Smash/.DS_Store|1437369145|1436538960.
a3a5e44e49293eb6001043630d21d1e3212e95f846a64e80db1e1b8610b5afb2|/mnt/disk12/disk2/Documents/Design Revolution's Stuff/Images/Videos/.DS_Store|1437369145|1436544146.
93010123aa3d20a78ab26d0e8524ea837a1e3949ee213a9463d4b1985507a351|/mnt/disk12/disk2/Documents/Design Revolution's Stuff/Images/.DS_Store|1437369145|1436909757.
ec1082dd4dc74e546355c9be398bc9751a053a906fc4baaf0f4e116ad9cd28ae|/mnt/disk12/disk2/Documents/Design Revolution's Stuff/New DR website images/.DS_Store|1437369145|1435085224.
6fe5bdb664363567e3b244ae5ea05fcf7b887377a3ad36774e5be24159770e5c|/mnt/disk12/disk2/Documents/Design Revolution's Stuff/Possibilities for backdrops/.DS_Store|1437369145|1435086211.
5808e109be5c8d33a823f280194e8ed570596fb17d9a16e990de85de78594d55|/mnt/disk12/disk2/Documents/Design Revolution's Stuff/.DS_Store|1437369145|1436895456.
df0fa2db44c5431d285ce70e5ae21580f8094d05d56030046fe1fd3da3390159|/mnt/disk12/disk2/Documents/Marketing & Graphics/LightroomCatalogBackups/CS5-5_Jan2012/CS5-5_Jan2012-2.lrcat|1437369145|1436026983.
282839be4bb9158ff36123ba104198b41822140e0cee537eb452c7510263d7cf|/mnt/disk12/disk2/Documents/.DS_Store|1437369145|1436895451.
fd552124d5da4875acf9a137f1e8f44b0d3df6c37b5b719116311ec83428fbe5|/mnt/disk12/disk2/Pix2012/2012-11 Nov/[2012-11-29] Vbrams, Melissa/to send/Thumbs.db|1437369145|1435095750.
        

 

 

Link to comment

I still have the 2 disks and can do some testing if you like.

 

Can you do an export of both disks, e.g.

 

bunker -e -f export.disk2.txt /mnt/disk2
bunker -e -f export.disk12.txt /mnt/disk12

This gives you the hash values of both disks as read from the attended attribute. Compare the hash values of those files reported as corrupted.

 

Next run sha256sum to recalculate and display the hash value from CLI and see what the current hash value is, again compare to the export file.

 

Thanks for testing.

 

Link to comment

Can you do an export of both disks, e.g.

 

Here you go (bunker 1.11)

 

From the initial email of corruption here is one file.
f6235c5deb3fa2d6689a9e676199f1f29763a35c911de47a0ff5452751b3a5b4|/mnt/disk12/disk2/Backups/WorkstationCatalog/WorkstationCatalog.sqlite|1437369145|1437165044.
  
Exported values from disk
78656945a19a4bfc1d27e8482c2afb44de24686c8f251fb36bd1cd0ef9267816|/mnt/disk12/Backups/WorkstationCatalog/WorkstationCatalog.sqlite|1433682574|1437165044
78656945a19a4bfc1d27e8482c2afb44de24686c8f251fb36bd1cd0ef9267816|/mnt/disk2/Backups/WorkstationCatalog/WorkstationCatalog.sqlite|1433682574|1437165044

CLI calculated values from  (bunker -r / bunker -a / bunker -e)
f6235c5deb3fa2d6689a9e676199f1f29763a35c911de47a0ff5452751b3a5b4|/mnt/disk12/Backups/WorkstationCatalog/WorkstationCatalog.sqlite|1437395967|1437165044
f6235c5deb3fa2d6689a9e676199f1f29763a35c911de47a0ff5452751b3a5b4|/mnt/disk2/Backups/WorkstationCatalog/WorkstationCatalog.sqlite|1437397806|1437165044

Link to comment

Can we compare the hash date to the file date?  That way we would know.

 

scan date = 1437369145

file date = 1437165044

 

The above says that the scan date is about 2 days later than the file date, in other words the scan took place after the last file modification.

 

This makes extended attribute value suspicious ...

 

Link to comment

I have gone through the RFS to XFS conversion too. Made hash checks before and after, all went without a glitch.

 

Did you find any corruption in the extended attributes?

 

Bunker, once we have verified that it never fails, will make it possible for me to get rid of a backup server.  I'll still keep one though.

Link to comment

Bunker, once we have verified that it never fails, will make it possible for me to get rid of a backup server.  I'll still keep one though.

 

If you get rid of your backup server, how will you recover the file if it gets corrupted? Bunker only tells you a file is corrupt but cannot replace it.

Link to comment

Bunker, once we have verified that it never fails, will make it possible for me to get rid of a backup server.  I'll still keep one though.

 

If you get rid of your backup server, how will you recover the file if it gets corrupted? Bunker only tells you a file is corrupt but cannot replace it.

 

Today I have one live server and 3 backup servers.  Definitely will drop to 2, maybe one backup server once Crashplan settles down.

 

(Crashplan has 3tb of my files that are more critical and it thinks that is too much)

Link to comment

Did you find any corruption in the extended attributes?

 

When I moved from RFS to XFS I lost the extended attributes, the way I verified the correctness of the files afterwards was by using the -C command - this recalculates the hash and compares it to the value stored in the export file, and restoring it in the extended attributes.

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.