_Shorty Posted November 17, 2018 Share Posted November 17, 2018 *shrug* I'm not the only one that has it not working properly. If it's working for you, that's great for you. Doesn't change the fact that it is broken for me and other people. Have you tested it versus other hashing tools, or are you just assuming it is working because it hasn't reported any errors to you on its own? Quote Link to comment
FreeMan Posted November 17, 2018 Share Posted November 17, 2018 What's the difference between "Hash file mismatch" and "Hash file corruption"? I get notifications of both on a somewhat regular basis (they're .NFO files and 99.99% sure it's because Emby is constantly updating the NFOs). I've never had an issue with either Emby or Kodi complaining about the NFO files, so I'm pretty sure it's not an actual problem with them, just changes. Also, since all I ever get is reports on NFO changes, is it possible to exclude a particular file type? Quote Link to comment
bonienl Posted November 17, 2018 Author Share Posted November 17, 2018 Really man! There are plenty of satisfied users without issues. Sorry if doesn't work for you or you don't know how to use it properly. In the past four years using my plugin it reported a few corrupted files due to a power outage, a few mismatched files due to a flaky program and other then that no errors. It works as designed. 1 Quote Link to comment
bonienl Posted November 17, 2018 Author Share Posted November 17, 2018 3 minutes ago, FreeMan said: What's the difference between "Hash file mismatch" and "Hash file corruption"? A hash mismatch occurs when a program updates a file (different content) while not updating the file creation time. A hash corruption occurs when the file contents hasn't changed, but the stored hash value and newly calculated hash value don't match. 1 Quote Link to comment
_Shorty Posted November 17, 2018 Share Posted November 17, 2018 It doesn't work as designed when it says the hashes don't match but all the other hash tools say the hashes match. Sorry if you don't like to hear that, or if you don't care to believe user reports, but it is occasionally giving errors stating bad hashes when the file still matches the original kept elsewhere. Being rude because it works on your machine is kinda humourous. Quote Link to comment
FreeMan Posted November 17, 2018 Share Posted November 17, 2018 1 hour ago, bonienl said: A hash mismatch occurs when a program updates a file (different content) while not updating the file creation time. A hash corruption occurs when the file contents hasn't changed, but the stored hash value and newly calculated hash value don't match. Thanks for the clarification. Quote Link to comment
jammer Posted November 17, 2018 Share Posted November 17, 2018 4 hours ago, _Shorty said: Jammer, no, I haven't even bothered looking. It would be a nice additional safety net, for sure. But with the Dynamix one acting flaky I just turned it off. Not much point when it isn't properly doing what it should be. I haven't looked to see if there are any alternatives that perform a similar job. _Shorty, thanks for responding. That's unfortunate to hear. 2 hours ago, bonienl said: Really man! There are plenty of satisfied users without issues. Sorry if doesn't work for you or you don't know how to use it properly. In the past four years using my plugin it reported a few corrupted files due to a power outage, a few mismatched files due to a flaky program and other then that no errors. It works as designed. bonienl, I don't doubt that it works perfectly for the vast majority of users otherwise I would expect to find more discussion about issues. Can you please suggest next steps on how to troubleshoot? I suspect it may have something to do with lots of files with long Windows filepaths and additional attributes that may lead up to this issue happening. But then after copying to Unraid, I've even tried using mc to make a copy and other files encounter the same issue. Thanks in advance. Quote Link to comment
bonienl Posted November 17, 2018 Author Share Posted November 17, 2018 12 minutes ago, jammer said: Can you please suggest next steps on how to troubleshoot? I am not sure what the problem exactly is, perhaps you don't mind explaining what goes wrong for you. A couple of general recommendations. 1. Exclude folders and files which change frequently. The plugin is intended for files which don't get touched often and hence may incure bitrot over time. Frequently accessed files are automatically checked by the file system. 2. This plugin uses inotifywait to detect new or changed files, but is limited in how many files can change at the same time, this can result in files not getting a hash update. Try to minimize the number of concurrent changed files. 3. Certain OSes, like Apple, generate specific metadata files to keep file system information. Exclude such metadata files, the same is true for temporary files. 4. When the recycle.bin plugin is installed then exclude the recycle.bin folder because recycling and hashing interfere with each other Quote Link to comment
jammer Posted November 17, 2018 Share Posted November 17, 2018 2 hours ago, bonienl said: I am not sure what the problem exactly is, perhaps you don't mind explaining what goes wrong for you. A couple of general recommendations. 1. Exclude folders and files which change frequently. The plugin is intended for files which don't get touched often and hence may incure bitrot over time. Frequently accessed files are automatically checked by the file system. 2. This plugin uses inotifywait to detect new or changed files, but is limited in how many files can change at the same time, this can result in files not getting a hash update. Try to minimize the number of concurrent changed files. 3. Certain OSes, like Apple, generate specific metadata files to keep file system information. Exclude such metadata files, the same is true for temporary files. 4. When the recycle.bin plugin is installed then exclude the recycle.bin folder because recycling and hashing interfere with each other bonienl, thanks for taking the time to help. Here are some details and additional observations (i.e. incomplete filenames/paths in the logs, and pausing during copy). I'm guessing your #1, #3, and #4 recommendations don't apply in my situation. Setup: 1. Unraid 6.6.5 running on Dell PowerEdge T110 II, Intel® Xeon® CPU E3-1240 @ 3.30GHz, 16GB ECC, 2x4TB WD Red, 2TB WD, 1TB Hitachi, 128GB SSD (Unassigned). CyberPower CP1000PFCLCD UPS. Wired network. 2. Unraid array filesystem is encrypted XFS using passphrase 3. Memtest86 passed. SMART and Parity tests pass. 4. Windows 10 Desktop PC (NTFS) containing family photos/videos to be backed up. JPG and MOV files. 5. Created a Public SMB Share (/photos) that only includes one empty disk of the array. (keeping it simple for testing purposes). Copy-on-write on Auto. Cache disk disabled. Enhanced OS X interoperability disabled. 6. Plugins installed: Unassigned Devices, Community Applications, Dynamix File Integrity, 7. Docker and VM enabled but none installed. 8. Dynamix File Integrity options: Auto protect new files=Enabled. Hashing method=BLAKE2 (also tried SHA2). Save hash to flash=Disabled. Excluded folders/files=docker.img,libvirt.img and Apple metadata. Verification schedule=Daily. Procedure: 1. On Windows PC, I navigate to //towerl/photos and then drag/copy a directory of photos (inside are subdirectories of photos separated by date/event) from the local drive to the Unraid share. There are ~40,000 files totalling 160GB. This takes a few hours. 2. Wait for the daily hash verification check and the parity check (I set this to be more frequent for testing purposes) Symptoms: SHA256 hash key mismatch, is corrupted SHA256 hash key mismatch, /Jimmy Samsung/20160518_215300.jpg is corrupted SHA256 hash key mismatch, bbean cruise/00013.MTS is corrupted SHA256 hash key mismatch, ion/IMG_0408.JPG is corrupted SHA256 hash key mismatch, ion/IMG_0815.JPG is corrupted SHA256 hash key mismatch, thering at Janice/DSC_8135.JPG is corrupted SHA256 hash key mismatch, up shots/DSC_7719.JPG is corrupted 1. Firstly, not sure if it's related but while copying I notice that the transfer speed is 40+MB/s but drops to 0MB/s for several seconds every few minutes or so. During those times when it's dropped to 0MB/s I notice that one to two CPU cores is at 100% in the Unraid dashboard. 2. Daily hash verification shows ?? files hash mismatch. (see log excerpt above). These are for JPG and MTS (Video) files that don't change. 3. Note that in the first entry the filename is empty and for the other files, the filepaths are incomplete. 4. Subsequent hash verifications show identical errors and log output. 5. I've compared file hashes using other tools and also bitwise compare (cmp) to confirm that these so-called corrupted files are indeed identical to the source files. 6. Parity check shows 0 errors. I repeated the above. That is, I deleted the files and copied the same directory again. This time a different set of files show errors and subsequent hash verifications consistently show errors on that new set of files. As another test, I used mc on Unraid to make a copy of the folder locally on the array (i.e. taking Windows out of the loop). Similar result, the new copy had another set of files that show errors consistently. Again, file compares showed that the files were bitwise identical. I then changed the hash method from SHA2 to BLAKE2 and did a manual Build in the Integrity menu to recalculate the hashes for all files. The next daily hash verification showed no errors. However, after copying more files from the Windows PC, a few hash errors were found again in the new files. Any other things for me to try? Thanks. Quote Link to comment
bonienl Posted November 18, 2018 Author Share Posted November 18, 2018 You have CPU starvation or in other words CPU overload. This explains the "corruptions" you are seeing. This can happen because the plugin tries to start as many concurrent hashing sessions as possible, but too many sessions result in abortion. A couple of things you can do: Spread your disks over more verification tasks. When all disks are put under a single verification task, they all run in parallel, which may be too much for your processor to handle Lower the frequency of the disk verification schedule. A daily schedule with many disks keeps your server continueously busy. A typical schedule is weekly or monthly, which is frequent enough to detect bitrot. Combine this with point 1 When copying thousands of (small) files it is better to disable the automatic protection first and do a manual update afterwards. This avoids an overload on events and let the copying finish first before calculating/updating the hashes Quote Link to comment
limetech Posted November 18, 2018 Share Posted November 18, 2018 Bilged the last two posts in this topic - please keep it civil people. 1 2 Quote Link to comment
jammer Posted November 18, 2018 Share Posted November 18, 2018 13 hours ago, bonienl said: You have CPU starvation or in other words CPU overload. This explains the "corruptions" you are seeing. This can happen because the plugin tries to start as many concurrent hashing sessions as possible, but too many sessions result in abortion. A couple of things you can do: Spread your disks over more verification tasks. When all disks are put under a single verification task, they all run in parallel, which may be too much for your processor to handle Lower the frequency of the disk verification schedule. A daily schedule with many disks keeps your server continueously busy. A typical schedule is weekly or monthly, which is frequent enough to detect bitrot. Combine this with point 1 When copying thousands of (small) files it is better to disable the automatic protection first and do a manual update afterwards. This avoids an overload on events and let the copying finish first before calculating/updating the hashes bonienl, I hadn't thought of that, but it makes sense now. Thanks for pointing that out! That pausing issue was something I was going to look at separately since it was occasional and I only thought it affected speed and not functionality. Strangely, only one of the 8 CPU readings on the Dashboard would suddenly shoot up to 100% for a few seconds while all 8 would normally hover around 10% or so during the copy operation. It wasn't as if one or more were hovering around 70+%. I think that at least for my case, the problem occurs in the automatic hashing part rather than hash verification. Scheduled hash verifications perform consistently and run off hours but I'll decrease the frequency nonetheless once I verify that everything is working properly. Is there a way to schedule the hash Build operation to happen automatically say on a nightly basis off for any new unhashed files? This way, I can leave auto real-time protection disabled. Thanks! Quote Link to comment
Vr2Io Posted November 19, 2018 Share Posted November 19, 2018 (edited) 9 hours ago, jammer said: Is there a way to schedule the hash Build operation to happen automatically say on a nightly basis off for any new unhashed files? This way, I can leave auto real-time protection disabled. If tuen-off automatically protect, then previous hashed file may not hash again (build) if it by overwrite operation, but file which delete then write will hash again. Those missing file will hash again and update in (Check). Pls correct me if I am wrong. Nov 19 14:43:03 X370 bunker: checked 1 file from source /boot/config/plugins/dynamix.file.integrity/export/disk9.export.hash. Found: 1 mismatch (updated), 0 corruptions. Duration: 00:00:01. Average speed: 157 KB/s For my understanding, FIP and Checksum suit won't always work ideal, this is limitation of catch ionotify method. Edited November 19, 2018 by Benson Quote Link to comment
bonienl Posted November 19, 2018 Author Share Posted November 19, 2018 9 hours ago, jammer said: Is there a way to schedule the hash Build operation to happen automatically say on a nightly basis off for any new unhashed files? This way, I can leave auto real-time protection disabled. It is not automated at the moment, but I can add a new option to automatically update missing hashes on a scheduled (daily) basis. Quote Link to comment
bonienl Posted November 19, 2018 Author Share Posted November 19, 2018 35 minutes ago, Benson said: For my understanding, FIP and Checksum suit won't always work ideal, this is limitation of catch ionotify method. inotifywait doesn't always catch the files which are new or changed, this is especially true when many files 'change' at the same time. It might happen that real-time updates are not complete and doing a manual 'build' will restore the situation. Quote Link to comment
Vr2Io Posted November 19, 2018 Share Posted November 19, 2018 Just now, bonienl said: inotifywait doesn't always catch the files which are new or changed, this is especially true when many files 'change' at the same time. It might happen that real-time updates are not complete and doing a manual 'build' will restore the situation. Understand, thats why I say it is limitation, thanks 😉 Quote Link to comment
jammer Posted November 19, 2018 Share Posted November 19, 2018 5 hours ago, bonienl said: It is not automated at the moment, but I can add a new option to automatically update missing hashes on a scheduled (daily) basis. That would be great. Thanks! Quote Link to comment
gacpac Posted November 26, 2018 Share Posted November 26, 2018 I'm currently using the plugin and something came into my mind. This tool is basically to tell you, if some files were modified and give you an idea if the file is broken or not. But at the time that happens, if the file is truly corrupted there's not way to repair it anyway I'm I right? But also the build and check goes based in the first build for integrity and the following new files created. Quote Link to comment
_Shorty Posted November 26, 2018 Share Posted November 26, 2018 This tool does nothing but report whether or not the hash for the file is still the same as the last time it checked the hash or created the hash. If the hash is now different the tool does nothing except inform you of it thinking the hash is now different. No, it does nothing to help you repair the file. If the file is truly corrupt because of some problem with your hard drive you might be able to cure things by rebuilding from parity, perhaps after replacing a faulty drive if that's really the cause. You could also use some kind of file-level parity utility if you wanted to add some more redundancy to safeguard against things that may happen in the future. QuickPAR comes to mind. It is similar in theory to a parity drive. You can build parity files of a selectable amount of redundancy, and rebuild/repair if anything ever gets corrupt. This has to be done before you have any suspect files, though. You build parity files when everything is fine, and then if there is ever a time where things are not fine then you have redundant info to hopefully help you out of a jam. Quote Link to comment
fithwum Posted January 6, 2019 Share Posted January 6, 2019 (edited) will this plugin follow symlinks? and if so can that be turned off? Edited January 6, 2019 by fithwum Quote Link to comment
jeffreywhunter Posted January 7, 2019 Share Posted January 7, 2019 UR 6.5.5. I've been using File Integrity for a long time. No real issues. Today, I received an email notice with the following error displayed. Jan 6 14:58:06 HunterNAS bunker: error: BLAKE2 hash key mismatch, /mnt/disk1/My Backups/_gsdata_/a9ebea1d339a02ae40d547cf8068834d.tib is corrupted Jan 6 14:58:08 HunterNAS bunker: error: BLAKE2 hash key mismatch, /mnt/disk1/My Backups/_gsdata_/3bef27d30a51b20186a922805386f65a.tib is corrupted This happened back in Oct with the same files. Oct 21 15:09:53 HunterNAS bunker: error: BLAKE2 hash key mismatch, /mnt/disk1/My Backups/_gsdata_/a9ebea1d339a02ae40d547cf8068834d.tib is corrupted Oct 21 15:09:54 HunterNAS bunker: error: BLAKE2 hash key mismatch, /mnt/disk1/My Backups/_gsdata_/3bef27d30a51b20186a922805386f65a.tib is corrupted These are backups of one of my workstations. Large file size. Anything to worry about? Diagnostics file attached. Thanks in advance! hunternas-diagnostics-20190106-2330.zip Quote Link to comment
landS Posted January 7, 2019 Share Posted January 7, 2019 I run into the same issue with backups from my wife's machine. The trigger is her Mac backups and one of her pc work apps saves updates pre-existing data without properly modifying acces/modify datetime stamps. No issue, but glad this catches it. Quote Link to comment
JohnGAG Posted January 25, 2019 Share Posted January 25, 2019 On 11/17/2018 at 6:29 PM, bonienl said: 3. Certain OSes, like Apple, generate specific metadata files to keep file system information. Exclude such metadata files, the same is true for temporary files. So I guess (Apple) Time machine backups are not a good idea? Quote Link to comment
dheg Posted February 12, 2019 Share Posted February 12, 2019 Hi, I got my first file corruption error message. What do I do with this now? 🤔 Thanks! Event: unRAID file corruption Subject: Notice [GAIA] - bunker verify command Description: Found 4 files with BLAKE2 hash key corruption Importance: alert BLAKE2 hash key mismatch, /mnt/disk6/media/2017-05-28_01.mkv is corrupted BLAKE2 hash key mismatch, /mnt/disk6/media/2017-05-28_02.mkv is corrupted BLAKE2 hash key mismatch, /mnt/disk6/media/2017-05-28_04.mkv is corrupted BLAKE2 hash key mismatch, /mnt/disk6/media/2017-05-28_03.mkv is corrupted Quote Link to comment
JonathanM Posted February 13, 2019 Share Posted February 13, 2019 4 hours ago, dheg said: I got my first file corruption error message. What do I do with this now? Compare with your backups and restore if necessary. Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.