July 18, 201510 yr Author A small improvement to your daily script is to include the lines at the top of the for loop for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 ; do # Only operate on disks that are actually present in this unRAID system if [ -e /mnt/disk$i ] ; then as that removes any need to alter the script to specify the exact number of disks in the system. Also I was wondering if the -D 1 line should specify a larger value in case a parity check was running the last time this script was run and it therefore exited without doing anything? It seems to me that the current version of the script would omit calculating the checksum for that missed day. In fact on a disk where all the checksums are already added, that parameter does not seem to speed things up much at all so maybe it is mainly useful when first building up the checksums? Thanks for the suggestion about the disks , I will update my examples. Small gotcha ... omitting disks still requires to alter the list! It is true that -D 1 assumes the script runs on daily basis and if a day is skipped for whatever reason it would miss information. Changing to -D 2 would prevent that situation. Leaving the -D option out has only an impact on execution time when there are many files to be examined. For example I have one disk with 300K+ files and running bunker without -D option takes about 45 minutes, while the same thing with -D 1 is a few seconds.
July 18, 201510 yr Thanks for the suggestion about the disks , I will update my examples. Small gotcha ... omitting disks still requires to alter the list! Good point. I will have to bear that in mind. It is true that -D 1 assumes the script runs on daily basis and if a day is skipped for whatever reason it would miss information. Changing to -D 2 would prevent that situation. At least my assumption is correct and I was not misreading things. Leaving the -D option out has only an impact on execution time when there are many files to be examined. For example I have one disk with 300K+ files and running bunker without -D option takes about 45 minutes, while the same thing with -D 1 is a few seconds. Fair enough. I must admit I had only tested it on disks which were primarily video type media files and the total was less than about 10,000 files. Not tried it yet on my music files or software archives where the number would be much higher. I have been taking your scripts and reworking them so that I have a bunker-update.sh script based on your daily one that runs checking for recent changes and handles the adding/updating of the checksums. I am assuming initially that it is run hourly and using -D 6h so that there is quite a bit of overlap but the execution time should still be quite short.[br][br]I was thinking of also having a parameter option to put no time limit on it. The idea is that this one could be used for an initial creation of checksums. It is likely to run for days/weeks on my system with 20 drives to work on but it will take the tedium out of typing commands. a bunker-verify.sh script based on your monthly one that handles verification. The idea is that it is run daily and verifies one disk per day and the disk selected varies each day. This should give a more frequent check than your monthly script as each disk is checked once per month, while still ensuring that there is not too much load on the system. To avoid the most likely times for parity checks I am avoiding the first and last days of the months as candidates for checking the disks. Can you see any reason why this is not a good approach to handling the verification? I am more than happy to pass these scripts back for others to use if they are likely to prove useful. Any thoughts on this, and also am I perhaps likely to be duplicating work you are already planning? Assuming the above scripts work out OK, then I was thinking of trying my hand at a plugin to put a GUI front-end on the installation, configuring and scheduling execution of bunker. If I get that far (which is not at all guaranteed ) do you have any objections about me including the bunker binary as a download in such a plugin or should users just be prompted to install it manually? BTW: Are there any useful exit codes from the bunker binary that it might be worth checking for? I was thinking in particular of codes indicating error conditions.
July 18, 201510 yr Here is a write error from the middle of an add operation on an RFS disk. /boot/config/bunker -A -l -f /boot/logs/cronlogs/bunker21keys.log /mnt/disk2 Currently processing file 1923 of 3880. Completed: 18.7% [ETA: 02:16:55] Calculating sha256 hash... - /boot/config/bunker: line 86: echo: write error: InFinished - added 3880 files. Duration: 00:38:32 Average speed: 3.76 MB/s This looks like an error while writing the extended file attributes. Are you using ReiserFS ? Yes, I am using RFS on this disk. Can we conclude with certainty that the integrity of the data on our unRaid system requires that we run bunker on XFS drives? This is a big job to convert all those disks from RFS to XFS on my main server and all backup servers. (40 drives and millions of JPG files)
July 18, 201510 yr Author ... I am more than happy to pass these scripts back for others to use if they are likely to prove useful. Any thoughts on this, and also am I perhaps likely to be duplicating work you are already planning? Put your ideas forward, there are definitely more ways to automate the script. Mine certainly is not the only (or best) way. Assuming the above scripts work out OK, then I was thinking of trying my hand at a plugin to put a GUI front-end on the installation, configuring and scheduling execution of bunker. If I get that far (which is not at all guaranteed ) do you have any objections about me including the bunker binary as a download in such a plugin or should users just be prompted to install it manually? When you make a GUI then a good approach is to let the plugin download the zip file from the forum, I have done something similar for cache_dirs. This ensures that people installing the plugin get the latest version of bunker. BTW: Are there any useful exit codes from the bunker binary that it might be worth checking for? I was thinking in particular of codes indicating error conditions. There is no exit codes to indicate different failures, results are stored in a log file. Btw bunker is a shell script not a true binary.
July 18, 201510 yr Author Here is a write error from the middle of an add operation on an RFS disk. /boot/config/bunker -A -l -f /boot/logs/cronlogs/bunker21keys.log /mnt/disk2 Currently processing file 1923 of 3880. Completed: 18.7% [ETA: 02:16:55] Calculating sha256 hash... - /boot/config/bunker: line 86: echo: write error: InFinished - added 3880 files. Duration: 00:38:32 Average speed: 3.76 MB/s This looks like an error while writing the extended file attributes. Are you using ReiserFS ? Yes, I am using RFS on this disk. Can we conclude with certainty that the integrity of the data on our unRaid system requires that we run bunker on XFS drives? This is a big job to convert all those disks from RFS to XFS on my main server and all backup servers. (40 drives and millions of JPG files) I can only share my experiences with you. Moving from RFS to XFS in my case improved stability, I did not encounter issues with running bunker on XFS, but had some lock ups in the past with RFS.
July 18, 201510 yr ... I am more than happy to pass these scripts back for others to use if they are likely to prove useful. Any thoughts on this, and also am I perhaps likely to be duplicating work you are already planning? Put your ideas forward, there are definitely more ways to automate the script. Mine certainly is not the only (or best) way. OK - I should have something to look at shortly. Assuming the above scripts work out OK, then I was thinking of trying my hand at a plugin to put a GUI front-end on the installation, configuring and scheduling execution of bunker. If I get that far (which is not at all guaranteed ) do you have any objections about me including the bunker binary as a download in such a plugin or should users just be prompted to install it manually? When you make a GUI then a good approach is to let the plugin download the zip file from the forum, I have done something similar for cache_dirs. This ensures that people installing the plugin get the latest version of bunker. That was what I was thinking of doing - just checking you were happy before I made any assumptions. BTW: Are there any useful exit codes from the bunker binary that it might be worth checking for? I was thinking in particular of codes indicating error conditions. There is no exit codes to indicate different failures, results are stored in a log file. Btw bunker is a shell script not a true binary. I had not realised it was a script. I guess that rather than simply suggesting changes I can try providing a patch for you to approve and/or validate. If I still think some exit codes might be useful I at least know how to add such things to a script
July 19, 201510 yr For your net revision (if any) of bunker I would suggest a small tweak: Adding '--tag bunker' as a parameter to logger in your Logger function Removing the prefix 'bunker -' from the text of the message in all the calls to Logger It makes the syslog output look a little tidier and is more consistent with the style of other messages written to the syslog. If you would prefer a patch to save you work I can produce one for you.
July 19, 201510 yr Author For your net revision (if any) of bunker I would suggest a small tweak: Adding '--tag bunker' as a parameter to logger in your Logger function Removing the prefix 'bunker -' from the text of the message in all the calls to Logger It makes the syslog output look a little tidier and is more consistent with the style of other messages written to the syslog. If you would prefer a patch to save you work I can produce one for you. Yeah, make your changes and send an update to me. I will make a new version available in the OT.
July 19, 201510 yr For your net revision (if any) of bunker I would suggest a small tweak: Adding '--tag bunker' as a parameter to logger in your Logger function Removing the prefix 'bunker -' from the text of the message in all the calls to Logger It makes the syslog output look a little tidier and is more consistent with the style of other messages written to the syslog. If you would prefer a patch to save you work I can produce one for you. OK - here is the bunker script with the changes I mentioned. I also changed the version number to 1.13 on the assumption that was likely to be what was wanted. Yeah, make your changes and send an update to me. I will make a new version available in the OT. bunker.zip
July 19, 201510 yr Author Ok, had a look at your changes, thanks for the update. I have updated the OT with the new version v1.13.
July 20, 201510 yr This looks like an error while writing the extended file attributes. Are you using ReiserFS ? Yes, I am using RFS on this disk. Can we conclude with certainty that the integrity of the data on our unRaid system requires that we run bunker on XFS drives? This is a big job to convert all those disks from RFS to XFS on my main server and all backup servers. (40 drives and millions of JPG files) I have converted my first disk from RFS to XFS and then verified the copied over files with rsync -avXi --log-file=/boot/XFSdisk2.txt /mnt/disk2 /mnt/disk12 bunker -v /mnt/disk12 40 corruptions found, but checking the files with Windows File Compare (FC /b), they compare exactly. I expect the hashes are wrong on these 40 files because of the RFS write to extended attr bug?? I still have the 2 disks and can do some testing if you like. Event: unRAID file corruption Subject: Notice [TOWER5] - bunker verify command Description: Found 40 files with sha256 hash corruption Importance: alert f6235c5deb3fa2d6689a9e676199f1f29763a35c911de47a0ff5452751b3a5b4|/mnt/disk12/disk2/Backups/WorkstationCatalog/WorkstationCatalog.sqlite|1437369145|1437165044. e14f9927844234413cf951cd21e094c3d354592bb6f4a6daa67ec5d380099a7c|/mnt/disk12/disk2/Documents/Design Revolution's Stuff/Images/Templates/._Wordpress Images_facebook.psd|1437369145|1436548377. 50621fef5d0228d3e9c6aedd92b3daff42e94bd092d34d7e7ba1b602a22a761a|/mnt/disk12/disk2/Documents/Design Revolution's Stuff/Images/Templates/Wordpress Images_facebook.psd|1437369145|1436548375. 8dc7793c56f865a0f3a392a119c48b93a262255a12d552ff94a5d28ebb3fe4bb|/mnt/disk12/disk2/Documents/Design Revolution's Stuff/Images/Videos/Cake Smash/.DS_Store|1437369145|1436538960. a3a5e44e49293eb6001043630d21d1e3212e95f846a64e80db1e1b8610b5afb2|/mnt/disk12/disk2/Documents/Design Revolution's Stuff/Images/Videos/.DS_Store|1437369145|1436544146. 93010123aa3d20a78ab26d0e8524ea837a1e3949ee213a9463d4b1985507a351|/mnt/disk12/disk2/Documents/Design Revolution's Stuff/Images/.DS_Store|1437369145|1436909757. ec1082dd4dc74e546355c9be398bc9751a053a906fc4baaf0f4e116ad9cd28ae|/mnt/disk12/disk2/Documents/Design Revolution's Stuff/New DR website images/.DS_Store|1437369145|1435085224. 6fe5bdb664363567e3b244ae5ea05fcf7b887377a3ad36774e5be24159770e5c|/mnt/disk12/disk2/Documents/Design Revolution's Stuff/Possibilities for backdrops/.DS_Store|1437369145|1435086211. 5808e109be5c8d33a823f280194e8ed570596fb17d9a16e990de85de78594d55|/mnt/disk12/disk2/Documents/Design Revolution's Stuff/.DS_Store|1437369145|1436895456. df0fa2db44c5431d285ce70e5ae21580f8094d05d56030046fe1fd3da3390159|/mnt/disk12/disk2/Documents/Marketing & Graphics/LightroomCatalogBackups/CS5-5_Jan2012/CS5-5_Jan2012-2.lrcat|1437369145|1436026983. 282839be4bb9158ff36123ba104198b41822140e0cee537eb452c7510263d7cf|/mnt/disk12/disk2/Documents/.DS_Store|1437369145|1436895451. fd552124d5da4875acf9a137f1e8f44b0d3df6c37b5b719116311ec83428fbe5|/mnt/disk12/disk2/Pix2012/2012-11 Nov/[2012-11-29] Vbrams, Melissa/to send/Thumbs.db|1437369145|1435095750.
July 20, 201510 yr Author I still have the 2 disks and can do some testing if you like. Can you do an export of both disks, e.g. bunker -e -f export.disk2.txt /mnt/disk2 bunker -e -f export.disk12.txt /mnt/disk12 This gives you the hash values of both disks as read from the attended attribute. Compare the hash values of those files reported as corrupted. Next run sha256sum to recalculate and display the hash value from CLI and see what the current hash value is, again compare to the export file. Thanks for testing.
July 20, 201510 yr Can you do an export of both disks, e.g. Here you go (bunker 1.11) From the initial email of corruption here is one file. f6235c5deb3fa2d6689a9e676199f1f29763a35c911de47a0ff5452751b3a5b4|/mnt/disk12/disk2/Backups/WorkstationCatalog/WorkstationCatalog.sqlite|1437369145|1437165044. Exported values from disk 78656945a19a4bfc1d27e8482c2afb44de24686c8f251fb36bd1cd0ef9267816|/mnt/disk12/Backups/WorkstationCatalog/WorkstationCatalog.sqlite|1433682574|1437165044 78656945a19a4bfc1d27e8482c2afb44de24686c8f251fb36bd1cd0ef9267816|/mnt/disk2/Backups/WorkstationCatalog/WorkstationCatalog.sqlite|1433682574|1437165044 CLI calculated values from (bunker -r / bunker -a / bunker -e) f6235c5deb3fa2d6689a9e676199f1f29763a35c911de47a0ff5452751b3a5b4|/mnt/disk12/Backups/WorkstationCatalog/WorkstationCatalog.sqlite|1437395967|1437165044 f6235c5deb3fa2d6689a9e676199f1f29763a35c911de47a0ff5452751b3a5b4|/mnt/disk2/Backups/WorkstationCatalog/WorkstationCatalog.sqlite|1437397806|1437165044
July 20, 201510 yr Author So the stored hash value in the extended attribute is different, was the file modified after this calculation because that would explain the difference.
July 20, 201510 yr Author Can we compare the hash date to the file date? That way we would know. scan date = 1437369145 file date = 1437165044 The above says that the scan date is about 2 days later than the file date, in other words the scan took place after the last file modification. This makes extended attribute value suspicious ...
July 20, 201510 yr So the latest changes to bunker would classify this as corruption. Even though the file is actually fine.
July 20, 201510 yr Author So the latest changes to bunker would classify this as corruption. Even though the file is actually fine. Yes, this is seen as corruption because the hash doesn't match and the file isn't modified after the last scan date.
July 20, 201510 yr I am in the midst of pre-clearing a 2nd drive and will do another RFS -> XFS conversion. Is there any further testing you want to try on my XFS drives? I plan on doing a full (bunker -v -n) on a number of drives and seeing what I find.
July 20, 201510 yr Author I have gone through the RFS to XFS conversion too. Made hash checks before and after, all went without a glitch. Hope bunker can be useful to you too!
July 20, 201510 yr I have gone through the RFS to XFS conversion too. Made hash checks before and after, all went without a glitch. Did you find any corruption in the extended attributes? Bunker, once we have verified that it never fails, will make it possible for me to get rid of a backup server. I'll still keep one though.
July 20, 201510 yr Bunker, once we have verified that it never fails, will make it possible for me to get rid of a backup server. I'll still keep one though. If you get rid of your backup server, how will you recover the file if it gets corrupted? Bunker only tells you a file is corrupt but cannot replace it.
July 20, 201510 yr Bunker, once we have verified that it never fails, will make it possible for me to get rid of a backup server. I'll still keep one though. If you get rid of your backup server, how will you recover the file if it gets corrupted? Bunker only tells you a file is corrupt but cannot replace it. Today I have one live server and 3 backup servers. Definitely will drop to 2, maybe one backup server once Crashplan settles down. (Crashplan has 3tb of my files that are more critical and it thinks that is too much)
July 20, 201510 yr Today I have one live server and 3 backup servers. Definitely will drop to 2, maybe one backup server once Crashplan settles down. (Crashplan has 3tb of my files that are more critical and it thinks that is too much) ohhh... OK I understand now!
July 20, 201510 yr Author Did you find any corruption in the extended attributes? When I moved from RFS to XFS I lost the extended attributes, the way I verified the correctness of the files afterwards was by using the -C command - this recalculates the hash and compares it to the value stored in the export file, and restoring it in the extended attributes.
Archived
This topic is now archived and is closed to further replies.