July 14, 201510 yr Author sooo I am doing some testing with the bunker-daily script (sorry) and whenever I add a test file to disk1 and run bunker-daily I get the following errors. I think it is because it is trying and failing to create the new txt log of the added file. I will also attach my modified bunker-daily file, just in case I messed it up. Your daily script looks alright. The error comes from bunker itself, but I can't say what causes it. As a verification what version of bunker are you using (or perhaps redownload and try again) ?
July 14, 201510 yr Author I have a question with the bunker-monthly script. Does it ever get triggered at the same time as parity check? If so how do you handle that? The line if [[ $array == STARTED && $rsync -eq 0 ]]; then Ensures that execution only takes place when the array is started and no parity operation is in progress. The monthly cron is executed on the first day of the month at 04:40am, the easiest way to avoid conflicts is to schedule the parity check on another day.
July 14, 201510 yr The error comes from bunker itself, but I can't say what causes it. As a verification what version of bunker are you using (or perhaps redownload and try again) ? Just a thought - should there be a parameter option that makes bunker display its version number? Alternatively perhaps it could be standard behaviour if the -q option is not used? That would make it much easier to determine what version of bunker one is actually running. I can see that with the provision of some standardised scripts of the sort covered in the recent forum posts that setting up bunker becomes much easier for the average user. You could perhaps include the latest version of the scripts in the standard bunker download to help those people just getting started? FYI: When I created my own version of your scripts I added to the top as comments the meaning of the various bunker parameter options used in the script. Provides a useful 'quick reminder' of what they are without the need to revert to the master bunker documentation. If you include sample scripts in the standard download you might consider adding such comments to these sample scripts?
July 14, 201510 yr The error comes from bunker itself, but I can't say what causes it. As a verification what version of bunker are you using (or perhaps redownload and try again) ? I have the same error messages and have just started with bunker (Thank you very much for all of your efforts bonienl), hence I'm running your latest version 1.10.
July 14, 201510 yr sooo I am doing some testing with the bunker-daily script (sorry) and whenever I add a test file to disk1 and run bunker-daily I get the following errors. I think it is because it is trying and failing to create the new txt log of the added file. I will also attach my modified bunker-daily file, just in case I messed it up. Your daily script looks alright. The error comes from bunker itself, but I can't say what causes it. As a verification what version of bunker are you using (or perhaps redownload and try again) ? I'm running 1.10
July 16, 201510 yr Any luck on finding the culprit in bunker? EDIT: so it has something to do with bunker and newly added files. Even if I remove the part of your daily script that adds the new files to the .txt file in the hash folder I get the error messages. EDIT 2: Well it also happens if I modify the file. This is the last entry in my syslog before the errors occur. bunker - warning: /mnt/disk3/MEDIA/test2.txt was modified, update sha256 hash. So moral of the story is don't add or modify files on your server and all is OK EDIT 3: OK I do not get the error if I remove the "-q" flag! Question can I run theses scripts in cron without the -q flag?
July 16, 201510 yr Author Any luck on finding the culprit in bunker? EDIT: so it has something to do with bunker and newly added files. Even if I remove the part of your daily script that adds the new files to the .txt file in the hash folder I get the error messages. EDIT 2: Well it also happens if I modify the file. This is the last entry in my syslog before the errors occur. bunker - warning: /mnt/disk3/MEDIA/test2.txt was modified, update sha256 hash. So moral of the story is don't add or modify files on your server and all is OK EDIT 3: OK I do not get the error if I remove the "-q" flag! Question can I run theses scripts in cron without the -q flag? Ok, your observations have helped me to determine the error. As a workaround you can run without the -q option.
July 16, 201510 yr Just wanted to say "Thanks" for this! I'm working on updating all my offsite backups and I'm running this first on all my drives.
July 16, 201510 yr Any luck on finding the culprit in bunker? EDIT: so it has something to do with bunker and newly added files. Even if I remove the part of your daily script that adds the new files to the .txt file in the hash folder I get the error messages. EDIT 2: Well it also happens if I modify the file. This is the last entry in my syslog before the errors occur. bunker - warning: /mnt/disk3/MEDIA/test2.txt was modified, update sha256 hash. So moral of the story is don't add or modify files on your server and all is OK EDIT 3: OK I do not get the error if I remove the "-q" flag! Question can I run theses scripts in cron without the -q flag? Ok, your observations have helped me to determine the error. As a workaround you can run without the -q option. Awesome, glad I could be of assistance.
July 16, 201510 yr Author Thx archedraft, I have made version 1.11 available with the bug fix for the reporting. See OT.
July 16, 201510 yr Great, I'll give it a test run here in a few hours. I started a verify check on all my disks so that is going to take a few more hours. Thanks yet again for working on this!
July 16, 201510 yr Is there a way to stop bunker from verifying all the files? I tried htop and F9 but it just starts on the next file on the disk.
July 16, 201510 yr Author Is there a way to stop bunker from verifying all the files? I tried htop and F9 but it just starts on the next file on the disk. Ctrl-C
July 16, 201510 yr Is there a way to stop bunker from verifying all the files? I tried htop and F9 but it just starts on the next file on the disk. Ctrl-C That just skips to the next file as well. I guess I could just hit Ctrl-c a few thousand times...
July 17, 201510 yr I have also updated to v.1.11 and the errors from 1.10 disappeared. The advise that I would like to get is how to deal with my drive #1 that only contains 2 sparsebundle files for 2 Mac Time Machine backups: Jul 17 11:12:49 Tower root: bunker - exported 5495 files from /mnt/disk1 with mask *. Duration: 00:05:03 Jul 17 11:13:56 Tower root: bunker - added 10 files from /mnt/disk2 with mask *. Duration: 00:01:07 Average speed: 149. MB/s Jul 17 11:13:57 Tower root: bunker - updated 0 files from /mnt/disk2 with mask * (0 corruptions). Duration: 00:00:01 Jul 17 11:13:57 Tower root: bunker - exported 10 files from /mnt/disk2 with mask *. Duration: 00:00:00 Jul 17 11:13:57 Tower root: bunker - added 0 files from /mnt/disk3 with mask *. Duration: 00:00:00 Jul 17 11:13:57 Tower root: bunker - updated 0 files from /mnt/disk3 with mask * (0 corruptions). Duration: 00:00:00 Jul 17 11:13:57 Tower root: bunker - exported 0 files from /mnt/disk3 with mask *. Duration: 00:00:00 Jul 17 11:13:58 Tower root: bunker - added 0 files from /mnt/disk4 with mask *. Duration: 00:00:00 Jul 17 11:13:58 Tower root: bunker - updated 0 files from /mnt/disk4 with mask * (0 corruptions). Duration: 00:00:00 Jul 17 11:13:58 Tower root: bunker - exported 0 files from /mnt/disk4 with mask *. Duration: 00:00:00 Jul 17 11:13:58 Tower root: bunker - added 0 files from /mnt/disk5 with mask *. Duration: 00:00:00 Jul 17 11:13:58 Tower root: bunker - updated 0 files from /mnt/disk5 with mask * (0 corruptions). Duration: 00:00:00 Jul 17 11:13:58 Tower root: bunker - exported 0 files from /mnt/disk5 with mask *. Duration: 00:00:00 Jul 17 11:13:58 Tower root: bunker - added 0 files from /mnt/disk6 with mask *. Duration: 00:00:00 Jul 17 11:13:58 Tower root: bunker - updated 0 files from /mnt/disk6 with mask * (0 corruptions). Duration: 00:00:00 Jul 17 11:13:59 Tower root: bunker - exported 0 files from /mnt/disk6 with mask *. Duration: 00:00:00 Jul 17 11:13:59 Tower root: bunker - added 0 files from /mnt/disk7 with mask *. Duration: 00:00:00 Jul 17 11:13:59 Tower root: bunker - updated 0 files from /mnt/disk7 with mask * (0 corruptions). Duration: 00:00:00 So two questions: [*]would you exclude disk1 (the Time Machine drive)? [*]why is the amount of files new/updated so low? This was the first time that I ran this script Remark: I had to shorten the file disk1.export.txt - too large to attach.
July 17, 201510 yr Looks like there could be a buglet in displaying the initial estimated time to complete? I am creating the initial checksums for a 6TB disk, and I started by getting the following displayed: Currently processing file 2 of 3539. Completed: 0.1% [ETA: 13695459:24:16] which is rather an excessive time This later corrected itself to: Currently processing file 10 of 3539. Completed: 0.3% [ETA: 29:03:57]
July 17, 201510 yr Author The calculation of the ETA works progressive and may be off with the first files, but should become more accurate as more files are processed.
July 17, 201510 yr Author So two questions: [*]would you exclude disk1 (the Time Machine drive)? [*]why is the amount of files new/updated so low? This was the first time that I ran this script 1. It is entirely up to you which files you like to include/exclude. The bunker-daily script allows for skipping of disks if needed, while bunker itself can be given an exclude filter (use !) 2. The option -D 1 tells bunker only to search for files created in the last 24 hours , for an initial calculation you need to remove this option.
July 17, 201510 yr Could I request and enhancement to the -l option (perhaps -L) where only lines with non-zero values are written to the log. At the moment with 20 disks in the system I get a lot of entries reporting 0 changes. It would be easier to pick out significant entries if only the non-zero entries were logged.
July 17, 201510 yr Here is a write error from the middle of an add operation on an RFS disk. /boot/config/bunker -A -l -f /boot/logs/cronlogs/bunker21keys.log /mnt/disk2 Currently processing file 1923 of 3880. Completed: 18.7% [ETA: 02:16:55] Calculating sha256 hash... - /boot/config/bunker: line 86: echo: write error: InFinished - added 3880 files. Duration: 00:38:32 Average speed: 3.76 MB/s
July 18, 201510 yr Author Could I request and enhancement to the -l option (perhaps -L) where only lines with non-zero values are written to the log. At the moment with 20 disks in the system I get a lot of entries reporting 0 changes. It would be easier to pick out significant entries if only the non-zero entries were logged. I created version 1.12 which has a new option -L, this allows bunker to create log entries only when changes are present. See OT for download. As a convenience, together with the bunker script I have also included the bunker-daily and bunker-monthly scripts in the zip file.
July 18, 201510 yr Author Here is a write error from the middle of an add operation on an RFS disk. /boot/config/bunker -A -l -f /boot/logs/cronlogs/bunker21keys.log /mnt/disk2 Currently processing file 1923 of 3880. Completed: 18.7% [ETA: 02:16:55] Calculating sha256 hash... - /boot/config/bunker: line 86: echo: write error: InFinished - added 3880 files. Duration: 00:38:32 Average speed: 3.76 MB/s This looks like an error while writing the extended file attributes. Are you using ReiserFS ?
July 18, 201510 yr Could I request and enhancement to the -l option (perhaps -L) where only lines with non-zero values are written to the log. At the moment with 20 disks in the system I get a lot of entries reporting 0 changes. It would be easier to pick out significant entries if only the non-zero entries were logged. I created version 1.12 which has a new option -L, this allows bunker to create log entries only when changes are present. See OT for download. Thanks for adding that option. I will test it out today.
July 18, 201510 yr A small improvement to your daily script is to include the lines at the top of the for loop for i in 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 ; do # Only operate on disks that are actually present in this unRAID system if [ -e /mnt/disk$i ] ; then as that removes any need to alter the script to specify the exact number of disks in the system. Also I was wondering if the -D 1 line should specify a larger value in case a parity check was running the last time this script was run and it therefore exited without doing anything? It seems to me that the current version of the script would omit calculating the checksum for that missed day. In fact on a disk where all the checksums are already added, that parameter does not seem to speed things up much at all so maybe it is mainly useful when first building up the checksums?
Archived
This topic is now archived and is closed to further replies.