Dynamix File Integrity plugin


bonienl

Recommended Posts

*shrug* I'm not the only one that has it not working properly.  If it's working for you, that's great for you.  Doesn't change the fact that it is broken for me and other people.  Have you tested it versus other hashing tools, or are you just assuming it is working because it hasn't reported any errors to you on its own?

Link to comment

What's the difference between "Hash file mismatch" and "Hash file corruption"?

 

I get notifications of both on a somewhat regular basis (they're .NFO files and 99.99% sure it's because Emby is constantly updating the NFOs). I've never had an issue with either Emby or Kodi complaining about the NFO files, so I'm pretty sure it's not an actual problem with them, just changes.

 

Also, since all I ever get is reports on NFO changes, is it possible to exclude a particular file type?

Link to comment

Really man! There are plenty of satisfied users without issues. Sorry if doesn't work for you or you don't know how to use it properly.

 

In the past four years using my plugin it reported a few corrupted files due to a power outage, a few mismatched files due to a flaky program and other then that no errors. It works as designed.

  • Like 1
Link to comment
3 minutes ago, FreeMan said:

What's the difference between "Hash file mismatch" and "Hash file corruption"?

A hash mismatch occurs when a program updates a file (different content) while not updating the file creation time.

 

A hash corruption occurs when the file contents hasn't changed, but the stored hash value and newly calculated hash value don't match.

  • Like 1
Link to comment

It doesn't work as designed when it says the hashes don't match but all the other hash tools say the hashes match.  Sorry if you don't like to hear that, or if you don't care to believe user reports, but it is occasionally giving errors stating bad hashes when the file still matches the original kept elsewhere.  Being rude because it works on your machine is kinda humourous.

Link to comment
1 hour ago, bonienl said:

A hash mismatch occurs when a program updates a file (different content) while not updating the file creation time.

 

A hash corruption occurs when the file contents hasn't changed, but the stored hash value and newly calculated hash value don't match.

Thanks for the clarification.

Link to comment
4 hours ago, _Shorty said:

Jammer, no, I haven't even bothered looking.  It would be a nice additional safety net, for sure.  But with the Dynamix one acting flaky I just turned it off.  Not much point when it isn't properly doing what it should be.  I haven't looked to see if there are any alternatives that perform a similar job.

_Shorty, thanks for responding.  That's unfortunate to hear.  

 

2 hours ago, bonienl said:

Really man! There are plenty of satisfied users without issues. Sorry if doesn't work for you or you don't know how to use it properly.

 

In the past four years using my plugin it reported a few corrupted files due to a power outage, a few mismatched files due to a flaky program and other then that no errors. It works as designed.

bonienl, I don't doubt that it works perfectly for the vast majority of users otherwise I would expect to find more discussion about issues.  

 

Can you please suggest next steps on how to troubleshoot?  I suspect it may have something to do with lots of files with long Windows filepaths and additional attributes that may lead up to this issue happening.  But then after copying to Unraid, I've even tried using mc to make a copy and other files encounter the same issue.  Thanks in advance.   

Link to comment
12 minutes ago, jammer said:

Can you please suggest next steps on how to troubleshoot?

I am not sure what the problem exactly is, perhaps you don't mind explaining what goes wrong for you.

 

A couple of general recommendations.

1. Exclude folders and files which change frequently. The plugin is intended for files which don't get touched often and hence may incure bitrot over time. Frequently accessed files are automatically checked by the file system.

2. This plugin uses inotifywait to detect new or changed files, but is limited in how many files can change at the same time, this can result in files not getting a hash update. Try to minimize the number of concurrent changed files.

3. Certain OSes, like Apple, generate specific metadata files to keep file system information. Exclude such metadata files, the same is true for temporary files.

4. When the recycle.bin plugin is installed then exclude the recycle.bin folder because recycling and hashing interfere with each other

Link to comment
2 hours ago, bonienl said:

I am not sure what the problem exactly is, perhaps you don't mind explaining what goes wrong for you.

 

A couple of general recommendations.

1. Exclude folders and files which change frequently. The plugin is intended for files which don't get touched often and hence may incure bitrot over time. Frequently accessed files are automatically checked by the file system.

2. This plugin uses inotifywait to detect new or changed files, but is limited in how many files can change at the same time, this can result in files not getting a hash update. Try to minimize the number of concurrent changed files.

3. Certain OSes, like Apple, generate specific metadata files to keep file system information. Exclude such metadata files, the same is true for temporary files.

4. When the recycle.bin plugin is installed then exclude the recycle.bin folder because recycling and hashing interfere with each other

 

bonienl, thanks for taking the time to help.  Here are some details and additional observations (i.e. incomplete filenames/paths in the logs, and pausing during copy).  I'm guessing your #1, #3, and #4 recommendations don't apply in my situation. 

 

Setup: 

1. Unraid 6.6.5 running on Dell PowerEdge T110 II, Intel® Xeon® CPU E3-1240 @ 3.30GHz, 16GB ECC, 2x4TB WD Red, 2TB WD, 1TB Hitachi, 128GB SSD (Unassigned). CyberPower CP1000PFCLCD UPS. Wired network.   

2. Unraid array filesystem is encrypted XFS using passphrase  

3. Memtest86 passed. SMART and Parity tests pass.

4. Windows 10 Desktop PC (NTFS) containing family photos/videos to be backed up. JPG and MOV files. 

5. Created a Public SMB Share (/photos) that only includes one empty disk of the array.  (keeping it simple for testing purposes). Copy-on-write on Auto. Cache disk disabled.  Enhanced OS X interoperability disabled.  

6. Plugins installed: Unassigned Devices, Community Applications, Dynamix File Integrity, 

7. Docker and VM enabled but none installed.  

8. Dynamix File Integrity options: Auto protect new files=Enabled. Hashing method=BLAKE2 (also tried SHA2). Save hash to flash=Disabled. Excluded folders/files=docker.img,libvirt.img and Apple metadata. Verification schedule=Daily. 

 

Procedure: 

1. On Windows PC, I navigate to //towerl/photos and then drag/copy a directory of photos (inside are subdirectories of photos separated by date/event) from the local drive to the Unraid share. There are ~40,000 files totalling 160GB. This takes a few hours. 

2. Wait for the daily hash verification check and the parity check (I set this to be more frequent for testing purposes) 

 

Symptoms: 

SHA256 hash key mismatch,  is corrupted
SHA256 hash key mismatch, /Jimmy Samsung/20160518_215300.jpg is corrupted
SHA256 hash key mismatch, bbean cruise/00013.MTS is corrupted
SHA256 hash key mismatch, ion/IMG_0408.JPG is corrupted
SHA256 hash key mismatch, ion/IMG_0815.JPG is corrupted
SHA256 hash key mismatch, thering at Janice/DSC_8135.JPG is corrupted
SHA256 hash key mismatch, up shots/DSC_7719.JPG is corrupted

1. Firstly, not sure if it's related but while copying I notice that the transfer speed is 40+MB/s but drops to 0MB/s for several seconds every few minutes or so.  During those times when it's dropped to 0MB/s I notice that one to two CPU cores is at 100% in the Unraid dashboard.  

2. Daily hash verification shows ?? files hash mismatch. (see log excerpt above). These are for JPG and MTS (Video) files that don't change. 

3. Note that in the first entry the filename is empty and for the other files, the filepaths are incomplete.

4. Subsequent hash verifications show identical errors and log output. 

5. I've compared file hashes using other tools and also bitwise compare (cmp) to confirm that these so-called corrupted files are indeed identical to the source files. 

6. Parity check shows 0 errors. 

 

I repeated the above. That is, I deleted the files and copied the same directory again. This time a different set of files show errors and subsequent hash verifications consistently show errors on that new set of files.  

 

As another test, I used mc on Unraid to make a copy of the folder locally on the array (i.e. taking Windows out of the loop). Similar result, the new copy had another set of files that show errors consistently. Again, file compares showed that the files were bitwise identical. 

 

I then changed the hash method from SHA2 to BLAKE2 and did a manual Build in the Integrity menu to recalculate the hashes for all files. The next daily hash verification showed no errors.  However, after copying more files from the Windows PC, a few hash errors were found again in the new files. 

 

Any other things for me to try?  Thanks. 

 

Link to comment

You have CPU starvation or in other words CPU overload. This explains the "corruptions" you are seeing. This can happen because the plugin tries to start as many concurrent hashing sessions as possible, but too many sessions result in abortion. A couple of things you can do:

  1. Spread your disks over more verification tasks. When all disks are put under a single verification task, they all run in parallel, which may be too much for your processor to handle
  2. Lower the frequency of the disk verification schedule. A daily schedule with many disks keeps your server continueously busy. A typical schedule is weekly or monthly, which is frequent enough to detect bitrot. Combine this with point 1
  3. When copying thousands of (small) files it is better to disable the automatic protection first and do a manual update afterwards. This avoids an overload on events and let the copying finish first before calculating/updating the hashes

 

Link to comment
13 hours ago, bonienl said:

You have CPU starvation or in other words CPU overload. This explains the "corruptions" you are seeing. This can happen because the plugin tries to start as many concurrent hashing sessions as possible, but too many sessions result in abortion. A couple of things you can do:

  1. Spread your disks over more verification tasks. When all disks are put under a single verification task, they all run in parallel, which may be too much for your processor to handle
  2. Lower the frequency of the disk verification schedule. A daily schedule with many disks keeps your server continueously busy. A typical schedule is weekly or monthly, which is frequent enough to detect bitrot. Combine this with point 1
  3. When copying thousands of (small) files it is better to disable the automatic protection first and do a manual update afterwards. This avoids an overload on events and let the copying finish first before calculating/updating the hashes

 

bonienl, I hadn't thought of that, but it makes sense now. Thanks for pointing that out! 

 

That pausing issue was something I was going to look at separately since it was occasional and I only thought it affected speed and not functionality. 

 

Strangely, only one of the 8 CPU readings on the Dashboard would suddenly shoot up to 100% for a few seconds while all 8 would normally hover around 10% or so during the copy operation. It wasn't as if one or more were hovering around 70+%. 

 

I think that at least for my case, the problem occurs in the automatic hashing part rather than hash verification. Scheduled hash verifications perform consistently and run off hours but I'll decrease the frequency nonetheless once I verify that everything is working properly. 

 

Is there a way to schedule the hash Build operation to happen automatically say on a nightly basis off for any new unhashed files? This way, I can leave auto real-time protection disabled. 

 

Thanks! 

Link to comment
9 hours ago, jammer said:

Is there a way to schedule the hash Build operation to happen automatically say on a nightly basis off for any new unhashed files? This way, I can leave auto real-time protection disabled. 

If tuen-off automatically protect, then previous hashed file may not hash again (build) if it by overwrite operation, but file which delete then write will hash again. Those missing file will hash again and update in (Check).  Pls correct me if I am wrong.

 

Nov 19 14:43:03 X370 bunker: checked 1 file from source /boot/config/plugins/dynamix.file.integrity/export/disk9.export.hash. Found: 1 mismatch (updated), 0 corruptions. Duration: 00:00:01. Average speed: 157 KB/s

 

For my understanding, FIP and Checksum suit won't always work ideal, this is limitation of catch ionotify method.

Edited by Benson
Link to comment
9 hours ago, jammer said:

Is there a way to schedule the hash Build operation to happen automatically say on a nightly basis off for any new unhashed files? This way, I can leave auto real-time protection disabled. 

It is not automated at the moment, but I can add a new option to automatically update missing hashes on a scheduled (daily) basis.

Link to comment
35 minutes ago, Benson said:

For my understanding, FIP and Checksum suit won't always work ideal, this is limitation of catch ionotify method.

inotifywait doesn't always catch the files which are new or changed, this is especially true when many files 'change' at the same time.

It might happen that real-time updates are not complete and doing a manual 'build' will restore the situation.

Link to comment
Just now, bonienl said:

inotifywait doesn't always catch the files which are new or changed, this is especially true when many files 'change' at the same time.

It might happen that real-time updates are not complete and doing a manual 'build' will restore the situation.

Understand, thats why I say it is limitation, thanks 😉

Link to comment

I'm currently using the plugin and something came into my mind. 

This tool is basically to tell you, if some files were modified and give you an idea if the file is broken or not. 

 

But at the time that happens, if the file is truly corrupted there's not way to repair it anyway I'm I right?

 

But also the build and check goes based in the first build for integrity and the following new files created. 

Link to comment

This tool does nothing but report whether or not the hash for the file is still the same as the last time it checked the hash or created the hash.  If the hash is now different the tool does nothing except inform you of it thinking the hash is now different.  No, it does nothing to help you repair the file.  If the file is truly corrupt because of some problem with your hard drive you might be able to cure things by rebuilding from parity, perhaps after replacing a faulty drive if that's really the cause.  You could also use some kind of file-level parity utility if you wanted to add some more redundancy to safeguard against things that may happen in the future.  QuickPAR comes to mind.  It is similar in theory to a parity drive.  You can build parity files of a selectable amount of redundancy, and rebuild/repair if anything ever gets corrupt.  This has to be done before you have any suspect files, though.  You build parity files when everything is fine, and then if there is ever a time where things are not fine then you have redundant info to hopefully help you out of a jam.

Link to comment
  • 1 month later...

UR 6.5.5.  I've been using File Integrity for a long time.  No real issues. Today, I received an email notice with the following error displayed.

Jan 6 14:58:06 HunterNAS bunker: error: BLAKE2 hash key mismatch, /mnt/disk1/My Backups/_gsdata_/a9ebea1d339a02ae40d547cf8068834d.tib is corrupted
Jan 6 14:58:08 HunterNAS bunker: error: BLAKE2 hash key mismatch, /mnt/disk1/My Backups/_gsdata_/3bef27d30a51b20186a922805386f65a.tib is corrupted

 

This happened back in Oct with the same files.  

Oct 21 15:09:53 HunterNAS bunker: error: BLAKE2 hash key mismatch, /mnt/disk1/My Backups/_gsdata_/a9ebea1d339a02ae40d547cf8068834d.tib is corrupted
Oct 21 15:09:54 HunterNAS bunker: error: BLAKE2 hash key mismatch, /mnt/disk1/My Backups/_gsdata_/3bef27d30a51b20186a922805386f65a.tib is corrupted

These are backups of one of my workstations.  Large file size.  Anything to worry about?  Diagnostics file attached.

 

Thanks in advance!

hunternas-diagnostics-20190106-2330.zip

Link to comment
  • 3 weeks later...
  • 3 weeks later...

Hi,

I got my first file corruption error message.

What do I do with this now? 🤔

Thanks!

Event: unRAID file corruption
Subject: Notice [GAIA] - bunker verify command
Description: Found 4 files with BLAKE2 hash key corruption
Importance: alert

BLAKE2 hash key mismatch, /mnt/disk6/media/2017-05-28_01.mkv is corrupted
BLAKE2 hash key mismatch, /mnt/disk6/media/2017-05-28_02.mkv is corrupted
BLAKE2 hash key mismatch, /mnt/disk6/media/2017-05-28_04.mkv is corrupted
BLAKE2 hash key mismatch, /mnt/disk6/media/2017-05-28_03.mkv is corrupted

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.