archedraft Posted July 9, 2015 Share Posted July 9, 2015 I'll see if I can find that nail WOOT!! Quote Link to comment
tr0910 Posted July 9, 2015 Share Posted July 9, 2015 I'll see if I can find that nail This will make bunker awesome. Then a plug in gui will make it accessible to the masses. I too have struggled with the update of changed files issue. Thought it was just me not understanding it properly. Thanks for continuing to move this forward. ################# @weebo, @jbartlett you guys pay attention. Bitrot needs this as well. Quote Link to comment
archedraft Posted July 9, 2015 Share Posted July 9, 2015 I'll see if I can find that nail Then a plug in gui will make it accessible to the masses. In the short term it could just be a plugin installed through the unRAID GUI that just downloads the file and puts it in a standard location. That way we would get notifications when Bunker had an update. But I do agree with @tr0910, a front end GUI would certainly make this much easier for new users. Honestly I believe checksums should be a core function of unRAID. Just thought I would also say THANK YOU for your work! I appropriate you working on this in any capacity. Quote Link to comment
bonienl Posted July 9, 2015 Author Share Posted July 9, 2015 And I thought, I was only looking for a nail Quote Link to comment
TheDragon Posted July 9, 2015 Share Posted July 9, 2015 And I thought, I was only looking for a nail Maybe a nail and a lick of paint!! Quote Link to comment
tr0910 Posted July 9, 2015 Share Posted July 9, 2015 We will take the nail. Then we can validate the tool. If it's as awesome as I think it will be lots of people will clamor for a gui. Quote Link to comment
bonienl Posted July 10, 2015 Author Share Posted July 10, 2015 I've found a nail ... and started polishing it Updated version of bunker now also stores the last modified time of a file in the extended attributes. This allows bunker to determine whether a hash mismatch occured due to file corruption or due to file content modification. To make use of this feature it is necessary that existing attribute information gets the new filedate attribute added. The easiest way is to use the new command -t (touch), e.g. bunker -t /mnt/disk1 bunker -t /mnt/disk2 ... bunker -t /mnt/disk9 Once the new filedate attribute is added, you may need to make new export files, e.g. bunker -e -f /boot/hash/disk1.txt /mnt/disk1 bunker -e -f /boot/hash/disk2.txt /mnt/disk2 ... bunker -e -f /boot/hash/disk9.txt /mnt/disk9 Using the -a (add) command will now automatically include the new attribute bunker -a /mnt/disk1 Updating and checking of files can follow two strategies, e.g 1. A complete check+update of all files on disk 1 bunker -u -f /boot/hash/disk1.errors.txt /mnt/disk1 2. A partial check+update of files on disk 1 only newer than last scandate bunker -U -f /boot/hash/disk1.errors.txt /mnt/disk1 Please see the OT for downloading of the new version. At this moment in time there is no GUI available, but who knows perhaps one day that nail becomes goldplated... Quote Link to comment
archedraft Posted July 10, 2015 Share Posted July 10, 2015 WAY TO GO! So here is my strategy to get this automated. Would you mind double checking my logic to make sure it is actually doing what my intent is? 1. Schedule a daily cron that adds a checksum to new files : bunker -a /mnt/disk1 bunker -a /mnt/disk2 2. Run a weekly cron script that checks the last scan date and updates if scan date has changed: bunker -U -f /boot/hash/disk1.user-changes.txt /mnt/disk1 bunker -U -f /boot/hash/disk2.user-changes.txt /mnt/disk2 3. then after step 2 I can add a verify to the end of the script and it should only find corrupt files correct? bunker -v -f /boot/hash/disk1.errors.txt /mnt/disk1 bunker -v -f /boot/hash/disk2.errors.txt /mnt/disk2 Quote Link to comment
archedraft Posted July 10, 2015 Share Posted July 10, 2015 I think I found a bug or at least unexpected behavior on my end. I created a test folder and created 7 blank .docx files. I ran /boot/bunker -a /mnt/user/BACKUP/TEST/ and it added all 7 files. Then I opened up one file and added some text and saved it. I ran /boot/bunker -U /mnt/user/BACKUP/TEST/ but it now only shows 6 files and no updated?? I have to run another bunker -a for the modified file to be added, so it seems like whenever I edit my .docx file it looses its checksum. That shouldn't happen correct? Quote Link to comment
bonienl Posted July 10, 2015 Author Share Posted July 10, 2015 1. Daily update - ok, I do the same 2. Depending on how often you change/update files, you may want to make the bunker -U execution also daily. 3. bunker -V is the better choice here (this will update the scandate). Since this command will verify ALL files on the disk, it is better to run this at lower frequency, perhaps once a week or month. Two tips: - If you are going to put this in a cron job then adding the options -q (quiet) -l (logging) is helpful - In the cron job you can start several instances of bunker, e.g. for each disk, and let them run concurrently by pushing the command to the background (&) bunker -a -q -l /mnt/disk1 & bunker -a -q -l /mnt/disk2 & Quote Link to comment
bonienl Posted July 10, 2015 Author Share Posted July 10, 2015 I think I found a bug or at least unexpected behavior on my end. I created a test folder and created 7 blank .docx files. I ran /boot/bunker -a /mnt/user/BACKUP/TEST/ and it added all 7 files. Then I opened up one file and added some text and saved it. I ran /boot/bunker -U /mnt/user/BACKUP/TEST/ but it now only shows 6 files and no updated?? I have to run another bunker -a for the modified file to be added, so it seems like whenever I edit my .docx file it looses its checksum. That shouldn't happen correct? How do you edit and save the file, it sounds like the extended attributes are wiped ... Quote Link to comment
archedraft Posted July 10, 2015 Share Posted July 10, 2015 How do you edit and save the file, it sounds like the extended attributes are wiped ... I agree it does sound like the extended attributes are getting wiped, I am opening the files with MS Word 2010 and then clicking save when I exit the file. Quote Link to comment
bonienl Posted July 10, 2015 Author Share Posted July 10, 2015 How do you edit and save the file, it sounds like the extended attributes are wiped ... I agree it does sound like the extended attributes are getting wiped, I am opening the files with MS Word 2010 and then clicking save when I exit the file. I suppose you have mapped the share as a disk under windows, but I am afraid that Windows applications are not aware of extended attributes (read: destroy them) when working this way. Quote Link to comment
archedraft Posted July 10, 2015 Share Posted July 10, 2015 How do you edit and save the file, it sounds like the extended attributes are wiped ... I agree it does sound like the extended attributes are getting wiped, I am opening the files with MS Word 2010 and then clicking save when I exit the file. I suppose you have mapped the share as a disk under windows, but I am afraid that Windows applications are not aware of extended attributes (read: destroy them) when working this way. Dang it Windows! Always got to be a pain. I suppose that this shouldn't really matter because if I run a daily bunker -a any Windows modified files will be added again. Since the user modified the files they should be good for one day. So I am still not fully understanding the difference between -v & -V. I'm confused on why/how the scan date effects a verify? Quote Link to comment
bonienl Posted July 10, 2015 Author Share Posted July 10, 2015 So I am still not fully understanding the difference between -v & -V. I'm confused on why/how the scan date effects a verify? The difference happens when using the -U command. This command compares the file modification date against the scandate in the extended attributes. When using the -v command the scandate is never updated, it stays as was made by the -a command, while the -V command updates the scandate to the date and time when the verification was done. Though in your case - when loosing the extended attributes and recreating them - it doesn't make a difference ! Quote Link to comment
archedraft Posted July 10, 2015 Share Posted July 10, 2015 So just to verify I modified the file with nano in CLI and bunker did update the file with -U so Windows is at fault. Is there a way to test corrupt a file and then use bunker to see what it reports? I tried dd if=/dev/urandom of=/mnt/user/BACKUP/TEST/New\ Microsoft\ Word\ Document.docx bs=1023 count=1 and MS Word will not open the file anymore but linux updated the time of the file when I did that so bunker said the file was good. Any way that you know of to corrupt a file without modifying the file timestamp? Quote Link to comment
bonienl Posted July 10, 2015 Author Share Posted July 10, 2015 So just to verify I modified the file with nano in CLI and bunker did update the file with -U so Windows is at fault. Is there a way to test corrupt a file and then use bunker to see what it reports? I tried dd if=/dev/urandom of=/mnt/user/BACKUP/TEST/New\ Microsoft\ Word\ Document.docx bs=1023 count=1 and MS Word will not open the file anymore but linux updated the time of the file when I did that so bunker said the file was good. Any way that you know of to corrupt a file without modifying the file timestamp? When you use the -v command in this case, it should report a corruption -> file content has changed, but file timestamp stays the same. Quote Link to comment
archedraft Posted July 10, 2015 Share Posted July 10, 2015 -v reports the following if I run that dd command Finished - verified 5 files, skipped 0 files. Found: 1 mismatches (0 corruptions). I would have thought it should read 1 corruption Quote Link to comment
bonienl Posted July 10, 2015 Author Share Posted July 10, 2015 -v reports the following if I run that dd command Finished - verified 5 files, skipped 0 files. Found: 1 mismatches (0 corruptions). I would have thought it should read 1 corruption It means the modified time of the file has been adjusted. Quote Link to comment
RobJ Posted July 10, 2015 Share Posted July 10, 2015 You would need to use touch to set the modification date back. Try - touch --date="2015-01-01 01:00:00" file_name dd blah blah file_name touch --date="2015-01-01 01:00:00" file_name Quote Link to comment
ljm42 Posted July 10, 2015 Share Posted July 10, 2015 How do you edit and save the file, it sounds like the extended attributes are wiped ... I agree it does sound like the extended attributes are getting wiped, I am opening the files with MS Word 2010 and then clicking save when I exit the file. I suppose you have mapped the share as a disk under windows, but I am afraid that Windows applications are not aware of extended attributes (read: destroy them) when working this way. Actually, that sounds ideal. Is there any way to get unraid to wipe the attributes whenever the file is modified? Then you wouldn't have to worry about checking file modification times at all. Quote Link to comment
archedraft Posted July 10, 2015 Share Posted July 10, 2015 You would need to use touch to set the modification date back. Try - touch --date="2015-01-01 01:00:00" file_name dd blah blah file_name touch --date="2015-01-01 01:00:00" file_name Very nice verified 5 files, skipped 0 files. Found: 1 mismatches (1 corruptions) Quote Link to comment
archedraft Posted July 10, 2015 Share Posted July 10, 2015 Now to figure out a way to write a script that sends me an email with the log attached if it finds corruption. Quote Link to comment
bonienl Posted July 10, 2015 Author Share Posted July 10, 2015 Now to figure out a way to write a script that sends me an email with the log attached if it finds corruption. I can make that an option in bunker to send notifications when corruption is detected. Quote Link to comment
TheDragon Posted July 10, 2015 Share Posted July 10, 2015 Now to figure out a way to write a script that sends me an email with the log attached if it finds corruption. I can make that an option in bunker to send notifications when corruption is detected. That would be immense! Can you also share with us what flags you're using on your setup? Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.