Dynamix File Integrity plugin


bonienl

Recommended Posts

7 hours ago, je82 said:

i understand that blake3 is best if you have a cpu that supports it?

No special CPU need, blake3 super fast.

 

7 hours ago, je82 said:

my worry by the plugin writing to "extended attributes" is that the file is detected as changed by rsync resulting in my backup script having to overwrite all files to the backup server but maybe i am wrong here?

I always use rsync, extend attributes won't affect it.

Link to comment
  • 2 weeks later...

Anybody know how to manually correct hashes for false positives? File Integrity keeps warning me about those files and I know they're okay – probably some sort of change that didn't affect the file change timestamp.

 

I'd like to regenerate hashes for these files only instead of regenerating for the entire drive. Anybody know what command I'd have to run?

  • Like 1
Link to comment
32 minutes ago, ericswpark said:

Anybody know how to manually correct hashes for false positives? File Integrity keeps warning me about those files and I know they're okay – probably some sort of change that didn't affect the file change timestamp.

 

I'd like to regenerate hashes for these files only instead of regenerating for the entire drive. Anybody know what command I'd have to run?

So, is this util still broken and people are still using it?  Anyway...  If you only have a handful of files you'd like to do this for, just move them somewhere else, and then move them back.

Link to comment
Quote

So, is this util still broken and people are still using it?

 

DFI is in the repository, and I see no mention of it being broken.  However, I have recently received something about "BLAKE3 hash key has unexpected length" on good files.  I know the files are good because I create other hash files not by this plugin.

 

I don't trust this plugin 100%.  I don't see any issues posted on Github.  And support threads on Unraid is not the best place for support/issues (prefer Github).

 

My DFI version is 2021.08.24e.

 

 

Link to comment
4 hours ago, Jaybau said:

 

DFI is in the repository, and I see no mention of it being broken.  However, I have recently received something about "BLAKE3 hash key has unexpected length" on good files.  I know the files are good because I create other hash files not by this plugin.

 

I don't trust this plugin 100%.  I don't see any issues posted on Github.  And support threads on Unraid is not the best place for support/issues (prefer Github).

 

My DFI version is 2021.08.24e.

 

 

Well, I first posted about this when I saw it happening to me back in June of 2018 here: 

And I'm not the only one that has posted in this thread reporting that this is happening on their machines.  I don't think anyone wants to know that some random percentage of their files are probably ok.  Seems to me the whole point to begin with is to be able to trust that 100% of your files are ok if it reports that they are, and if there happen to actually be files that are not ok then they should be reported as not being ok.  But if it is falsely reporting hash mismatches when a handful of other hashing utils report that the reportedly changed file has not in fact changed at all then that points to the util not operating correctly.  And if it is not operating correctly then it can't be trusted.  And if it can't be trusted then it fails to meet its goal.

I'm rather confused as to how this issue hasn't received any attention in all this time.  We're talking over four years of a major bug seemingly being ignored.  Perhaps it should get some dev attention so that it can eventually be remedied, and the util then rendered actually useful.  I'm afraid it isn't useful while it is not operating correctly.  To be fair, perhaps it did receive some dev attention at some point, but apparently it is still exhibiting this flaw, so this would indicate the root of the problem was never actually fixed.  Hopefully it will be fixed at some point, because it would be nice to actually be able to use and trust such a thing.

Link to comment
4 hours ago, _Shorty said:

Well, I first posted about this when I saw it happening to me back in June of 2018 here: 

And I'm not the only one that has posted in this thread reporting that this is happening on their machines.  I don't think anyone wants to know that some random percentage of their files are probably ok.  Seems to me the whole point to begin with is to be able to trust that 100% of your files are ok if it reports that they are, and if there happen to actually be files that are not ok then they should be reported as not being ok.  But if it is falsely reporting hash mismatches when a handful of other hashing utils report that the reportedly changed file has not in fact changed at all then that points to the util not operating correctly.  And if it is not operating correctly then it can't be trusted.  And if it can't be trusted then it fails to meet its goal.

I'm rather confused as to how this issue hasn't received any attention in all this time.  We're talking over four years of a major bug seemingly being ignored.  Perhaps it should get some dev attention so that it can eventually be remedied, and the util then rendered actually useful.  I'm afraid it isn't useful while it is not operating correctly.  To be fair, perhaps it did receive some dev attention at some point, but apparently it is still exhibiting this flaw, so this would indicate the root of the problem was never actually fixed.  Hopefully it will be fixed at some point, because it would be nice to actually be able to use and trust such a thing.

As far as I am aware there are no Major bugs with this plugin. It seems like most people, myself included, have been running it without issue for several years. If you look back through this thread for posts from Bonienl you will see discussion of some of the limitations of this plugin that can occasionally produce false positives for hash mismatch. Things like too many files changing at once or certain programs changing files in a way that doesnt trigger a rehash. There are also issues I think with running to many disk verifications at once causing hashing tasks to be aborted and return as errors. There are also certain file types that dont really work with this plugin, like large vm disk images which take a long time to hash and can have many small random changes. In my experience though playing in the bounds of what this plugin can handle things work just fine. 

Link to comment
8 hours ago, primeval_god said:

As far as I am aware there are no Major bugs with this plugin. It seems like most people, myself included, have been running it without issue for several years. If you look back through this thread for posts from Bonienl you will see discussion of some of the limitations of this plugin that can occasionally produce false positives for hash mismatch. Things like too many files changing at once or certain programs changing files in a way that doesnt trigger a rehash. There are also issues I think with running to many disk verifications at once causing hashing tasks to be aborted and return as errors. There are also certain file types that dont really work with this plugin, like large vm disk images which take a long time to hash and can have many small random changes. In my experience though playing in the bounds of what this plugin can handle things work just fine. 

Well, in my case, nothing uses any of my shares directly except for Kodi accessing files for playback.  All files that are copied to the unRAID box are either copied there manually, or backed up occasionally via a scheduled robocopy.  The file in question in the screenshot I posted way back then is an image file that belongs to a game.  It would have been copied over to the unRAID box once and never changed.  Nothing would have updated it.  Nothing would have edited it.  And after initially being put there by robocopy, robocopy would not have put it there again or touched it in any way that I'm aware of.  It is a file that the game devs did not change, so there would be no reason for it to be updated during one of those backup runs.  I could be wrong, but I see no reason for a false positive other than something within this util itself, because as far as I'm aware the file was copied to unRAID once and never touched again.

Link to comment
4 hours ago, _Shorty said:

Well, in my case, nothing uses any of my shares directly except for Kodi accessing files for playback.  All files that are copied to the unRAID box are either copied there manually, or backed up occasionally via a scheduled robocopy.  The file in question in the screenshot I posted way back then is an image file that belongs to a game.  It would have been copied over to the unRAID box once and never changed.  Nothing would have updated it.  Nothing would have edited it.  And after initially being put there by robocopy, robocopy would not have put it there again or touched it in any way that I'm aware of.  It is a file that the game devs did not change, so there would be no reason for it to be updated during one of those backup runs.  I could be wrong, but I see no reason for a false positive other than something within this util itself, because as far as I'm aware the file was copied to unRAID once and never touched again.

 

Below are simple test to show when a file not completely transfer/generate, if hash it will got different hash, but once completed then the hash won't change. This also the major reason for false negative. That's why I suggest disable "Automatically protect" and I almost trouble free.

 

image.png.5cb2abfec81fc68a5b902d426280bcb3.png

Edited by Vr2Io
  • Like 1
Link to comment
16 minutes ago, Vr2Io said:

 

Below are simple test to show when a file not completely transfer/generate, if hash it will got different hash, but once completed then the hash won't change. This also the major reason for false negative. That's why I suggest disable "Automatically protect" and I almost trouble free.

 

image.png.5cb2abfec81fc68a5b902d426280bcb3.png

Are you supposing that a file that is a mere 2945 bytes was somehow hashed by the util before it completed writing and that's why the hash was incorrect?  Heh.  I sincerely doubt that.  Saying "A file's hash is different if you hash only a portion of the file." is kind of a silly thing to say.  Of course it is going to give a different hash.  It is different data.  I'd be incredibly surprised if we were talking about incomplete files being hashed, espectially given my initial report involving just 2945 bytes of data.  By your own admission, the util does not function correctly.  No point trying to defend it by saying it is mostly ok a lot of the time.  This is something that is supposed to be 100% correct 100% of the time.  If it is not, there's no point in using it.

Link to comment
3 hours ago, Vr2Io said:

 

Below are simple test to show when a file not completely transfer/generate, if hash it will got different hash, but once completed then the hash won't change. This also the major reason for false negative. That's why I suggest disable "Automatically protect" and I almost trouble free.

 

image.png.5cb2abfec81fc68a5b902d426280bcb3.png

 

Thank you...I believe this explains how I might have received my DFI integrity issue (files did not completely transfer, which I believe is caused by an Unraid defect for allowing/default 0 free space for a drive or the way Unraid pool handles 0 free space).

 

I do wish DFI had the option to regenerate hashes for the suspect files.  

Edited by Jaybau
Link to comment
11 hours ago, _Shorty said:

This is something that is supposed to be 100% correct 100% of the time.  If it is not, there's no point in using it.

If you found hash mismatch, there are nothing hurt, you can completely ignore it like it never exist, and I would surprise file haven't hash will better then have. If you don't like FIP, you can use another tools, or use command line to call bunker or generate hash in your way. This just what tools you have and how the solution work.

 

I am not selling you use FIP, I just point out one of the issue why got false negative.

Link to comment
7 hours ago, Vr2Io said:

 If you don't like FIP, you can use another tools, or use command line to call bunker or generate hash in your way. 

 

What can you do to regenerate the hash and store the hash in the file's extended attributes, so that future DFI checks will be valid?

Link to comment

You can manually set the extended attribute, although I do think this is quite the inconvenient workaround. Ideally the plugin should have a button where I can indicate that it was wrong, that the files are okay and it messed up.

 

Anyway, once you find the file, run:

 

getfattr -d <file>

 

The command should return all the extended attributes of the file. Find the one for "user.sha256".

 

Now generate a new SHA256:

 

sha256sum <file>

 

Copy the SHA256 hash, then type:

 

setfattr -n user.sha256 -v <hash value here> <file>

 

You may wish to re-run the `getfattr -d` command to verify that the update was successful.

 

You may also need to re-run the export hash function inside the plugin.

  • Like 1
Link to comment
12 hours ago, Vr2Io said:

If you found hash mismatch, there are nothing hurt, you can completely ignore it like it never exist, and I would surprise file haven't hash will better then have. If you don't like FIP, you can use another tools, or use command line to call bunker or generate hash in your way. This just what tools you have and how the solution work.

 

I am not selling you use FIP, I just point out one of the issue why got false negative.

Do you honestly not recognize how silly this response is?  The whole point of the util is to inform you of hash mismatches if there ever is one, because a hash mismatch means you now have a corrupt file that you need to deal with.  You're saying to ignore hash mismatch reports because they mean nothing and everything is actually fine.  Perhaps you should take the weekend and think about why this is ridiculous to say.

  • Upvote 1
Link to comment
9 hours ago, _Shorty said:

Do you honestly not recognize how silly this response is?  The whole point of the util is to inform you of hash mismatches if there ever is one, because a hash mismatch means you now have a corrupt file that you need to deal with.  You're saying to ignore hash mismatch reports because they mean nothing and everything is actually fine.  Perhaps you should take the weekend and think about why this is ridiculous to say.

Pls don't misleading, I am talking about false negative. Don't waste both time. And I haven't false negative issue on static file by FIP or Bunker.

Edited by Vr2Io
Link to comment
8 hours ago, Vr2Io said:

Pls don't misleading, I am talking about false negative. Don't waste both time. And I haven't false negative issue on static file by FIP or Bunker.

Perhaps it is a language/communication issue.  You seem to be saying specifically not to worry if it reports errors because errors are ok.

Link to comment
On 8/19/2022 at 7:43 PM, Jaybau said:

 

What can you do to regenerate the hash and store the hash in the file's extended attributes, so that future DFI checks will be valid?

If you have automatic protection on then I think you should just be able to touch the file and then the plugin will detect the change and rehash.

Link to comment
On 8/22/2022 at 8:31 AM, primeval_god said:

If you have automatic protection on then I think you should just be able to touch the file and then the plugin will detect the change and rehash.

Didn't think of that; thanks for the easy workaround!

 

EDIT: just tried this easy workaround and while it does work, it will throw up a one-time warning in the DFI plugin about mismatched hashes. It will update the hash after that so you shouldn't see this warning again.

Edited by ericswpark
Added caveat
Link to comment
  • 4 weeks later...
On 6/11/2022 at 10:04 AM, Interstellar said:

 

I am also having this problem, bonienl can you add some code that gives us the option to increase the PHP memory limit?

 

This problem isn't going away, disks are getting larger, thus filling with more files.

 

So two bugs with the script at the moment:

 

1. SHA256 "Check Export" button does not work correctly.

2. 128MB PHP limit is too low. (not strictly a bug, but it doesn't work with it set at 128MB so might as well be one)

 

@bonienl

 

Should I raise these bugs on GitHub for resolution? Still suffering from them...

Link to comment

Hello.

 

I have a Disk No.7 (14TB) with very many Files (from small TXT to big Acronis TI Images).
File Integrity tells me, when I start Export: 1805124 Files.
Since it takes several Minutes, I leave the tool working.


After some Minutes I look on the screen, and it seems it is almost ready (Screenshot 1).
And some 30 Seconds or so later I get an error (Screenshot 2).

Where is the Error/How to solve this?

 

FI-ERROR-0-2022-09-24 12_42_41-102 Tessa Main.png

FI-ERROR-2022-09-24 12_42_41-102 Tessa Main.png

Edited by DataCollector
Link to comment
On 4/7/2022 at 10:51 AM, bonienl said:

Note for those who want to uninstall this plugin, before uninstalling the plugin itself you need to remove the calculated hashes, this is done by the "Clear" operation (select all disks and then Clear, see also the built-in Help).

After the plugin is removed, you can delete the folder /config/plugins/dynamix.file.integrity on your flash devices to remove all log and other files.

I wanted now to clear all those to uninstall the FI plugin.
But it looks like it did not clear anything.

I dont understand this plugin at all.

 

FI-CLEAR-Nothing-2022-09-24 13_11_48-102 Tessa Main.png

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.