Dynamix File Integrity plugin


bonienl

Recommended Posts

I have 3 servers and this plug in has been installed on each for years. Everything has worked well. So much so that I frequently forget to check back on them for months. I happened to be doing other things in the tools tab and had an 'oh yeah FIP' moment. So I started just looking around to be sure everything was still running smoothly and found that 2 of my machines were running MD5 and one was running SHA2 and I couldn't for the life of me remember why. I turned on the help setting (consistently amazed at how awesome the help is) and it reminded me of the blake2 option which I think I had passed on originally because it said it was proprietary and I had some aspirations of checking my backups externally or some such that of course I never did.  I did a little quick googling of blake2 and learned that there is now a blake3 that is approx 5x as fast as blake2.  Has any thought been given to adding this as an option?

Edited by jasonfox
typos
Link to comment

My scheduled verify job reported "bunker verify command Found 1 file with BLAKE2 hash key corruption" but doesn't tell me what file is corrupted. I had error logging set to syslog in settings, but I don't see the file listed there either. How can I see what file is causing the problem?

 

Here is the relevant part of syslog:

bunker: error: BLAKE2 hash key mismatch,  is corrupted
bunker: Verify task for disk1 finished, duration: 12 hr, 2 min, 0 sec.
bunker: verified 455797 files from /mnt/disk1. Found: 0 mismatches, 1 corruption. Duration: 12:02:00. Average speed: 98.3 MB/s

 

  • Like 1
Link to comment
12 hours ago, jasonfox said:

I have 3 servers and this plug in has been installed on each for years. Everything has worked well. So much so that I frequently forget to check back on them for months. I happened to be doing other things in the tools tab and had an 'oh yeah FIP' moment. So I started just looking around to be sure everything was still running smoothly and found that 2 of my machines were running MD5 and one was running SHA2 and I couldn't for the life of me remember why. I turned on the help setting (consistently amazed at how awesome the help is) and it reminded me of the blake2 option which I think I had passed on originally because it said it was proprietary and I had some aspirations of checking my backups externally or some such that of course I never did.  I did a little quick googling of blake2 and learned that there is now a blake3 that is approx 5x as fast as blake2.  Has any thought been given to adding this as an option?

Speedup should be multiple core/thread only. For array even SSD ( single disk limitation ), expect no much benefit. ( current max speed ~500MB/s+ )

Edited by Vr2Io
Link to comment

I received an unraid alert for "Found 1 file with SHA256 hash key corruption". How do I tell which file is corrupt? No log file was created. My last dynamix log file is from 2019. Thanks.

 

EDIT: I ran this cmd to find it, but surely there has to be a better way? Why is it not logging? "cat /var/log/syslog | grep corrupt"

Edited by pman
Link to comment
  • 2 weeks later...

same here:

two of the corrupt files are not mentioned and I don't know what/where the third one is

Jan  6 03:31:59  bunker: error: BLAKE2 hash key mismatch,  is corrupted
Jan  6 03:51:41  bunker: error: BLAKE2 hash key mismatch,  is corrupted
Jan  6 03:58:50  bunker: error: BLAKE2 hash key mismatch, ng/fs.json is corrupted
Jan  6 04:02:59  bunker: Verify task for disk1 finished, duration: 1 hr, 2 min, 58 sec.
Jan  6 04:02:59  bunker: verified 198104 files from /mnt/disk1. Found: 0 mismatches, 3 corruptions. Duration: 01:02:58. Average speed: 36.1 MB/s

 

  • Like 2
Link to comment
  • 5 weeks later...

Me i'm kind of confused with this plugin. I've used it since forever. But it's one of those that well, if you don't get errors then you are good. But I'm kind of hoping to see a log with all good. All I have are the exports from 2019 and no logs. Currently i'm doing a manual check to see. I'm hoping to see a notification or something. 

Link to comment
On 12/17/2020 at 2:51 AM, Fizz said:

My scheduled verify job reported "bunker verify command Found 1 file with BLAKE2 hash key corruption" but doesn't tell me what file is corrupted. I had error logging set to syslog in settings, but I don't see the file listed there either. How can I see what file is causing the problem?

 

Here is the relevant part of syslog:


bunker: error: BLAKE2 hash key mismatch,  is corrupted
bunker: Verify task for disk1 finished, duration: 12 hr, 2 min, 0 sec.
bunker: verified 455797 files from /mnt/disk1. Found: 0 mismatches, 1 corruption. Duration: 12:02:00. Average speed: 98.3 MB/s

 

Mine got sent in an email from the smtp that i previously setup

Link to comment

I'm looking for some clarification on the Disk Verification Schedule and the Tasks.

 

So the help says

Quote

when you have set up a weekly schedule with 3 tasks then every week another task is executed and after 3 weeks the verification function has done all selected disks.

 

So in this example you have 3 disks and you have put each disk on a separate task. My understanding of the above is that Task 1 executes on the Monday, then the following Monday Task 2, and the following Monday after that Task 3. But then when does it restart? The next Monday after the week with Task 3 finishing its back to Task 1?

 

If I select Monthly, is that going to run Task 1 in Jan, Task 2 in Feb, Task 3 in March?

 

Is there anyway to set it up so it checks all disks once a month but separate the tasks by a day or a week?

 

Also on the Integrity page in the help it states "0 represents an open operation", does that mean a running operation? Like it's actively doing something in the background?

Edited by iarp
Link to comment

I've just installed and started using the File Integrity plugin today.

  • I manually started a `Build` process on 7 (out of 28) drives in my array
  • disk1 had least amount of files, so it has already finished

But, the UI shows some nonsense 🤔

  • it shows disk1 as a circle, not green checkmark, even though it has just finished the build, and is up-to-date
  • it shows disks 4, 5, 9 and 10 with a green checkmark, even though the builds are clearly still running and aren't finished
  • it shows disks 7, 12, 17, 18, 19, 22, 23, 24, 26, 27 and 28 with a green checkmark, even though the build process has never been run on these disk...

I mean, WAT? 😳

Can it be, that am I really not understanding what the circle/checkmark/cross means?

 

Or is this a bug?

 

2021-02-18_11-08-46__chrome.png

Link to comment
  • 2 weeks later...

I'm seeing similar notifications about 3 supposedly corrupted files, but no names!

 

BLAKE2 hash key mismatch,  is corrupted
BLAKE2 hash key mismatch,  is corrupted
BLAKE2 hash key mismatch,  is corrupted

 

The corresponding syslog entries are also missing file info, for example:

 

Feb 28 18:17:44 Tower bunker: error: BLAKE2 hash key mismatch, is corrupted

 

What is the cause of this (apparently I'm not the only one seeing it), and how can I fix it - is there a way to reset the file hashes?

 

Link to comment

Another false positive on a bluray dump (different image).  I've verified it with the original and everything hashes 100%.  I'm only getting 1 mismatch every month, but this is 2 for 2 where it has been a false positive.  Is there any way to address this or am I just stuck at having to double check files when it says they are corrupt?

Link to comment
20 minutes ago, Ranmacanada said:

Another false positive on a bluray dump (different image).  I've verified it with the original and everything hashes 100%.  I'm only getting 1 mismatch every month, but this is 2 for 2 where it has been a false positive.  Is there any way to address this or am I just stuck at having to double check files when it says they are corrupt?

This is know issue, I naver have bluray image ( or static file ) false positive, because I hash file by "build" in manual not automatic.

Link to comment
  • 1 month later...
On 10/13/2019 at 10:15 PM, Ouze said:

 

Happened again - did a check, out the output:

 

"bunker: error: SHA256 hash key mismatch, rnal.720p.bluray.x264-reward.mkv is corrupted"

 

According to a search, that narrows it down to 125 files. 

 

There has got to be a way to get this plugin to output more useful (specific) information, right? Why are these error messages being truncated this way?

 

I did an md5 check against all 125 files individual checksums and of course, they pass just fine.  These files got dropped onto the server once and have not been touched since then. If this plugin is going to generate output vague as it is, and as prone to false positives as it seems to be; then ultimately it's just contributing noise, not value. 

 

 

 

Did you finally found out why file names/paths are truncated? It also happens to me, and can be very annoying to find the exact file which is "supposed" to be corrupted.

Link to comment
  • 4 weeks later...

Is there a way to change the location where the hash results for each file are stored? I see they're stored in plain text on my usb drive. I'd like the results to be in an encrypted location such as my cache or array.

Link to comment

Files dont seem to be ignored in my set up I have tried .log, .nfo, .tgz, .tmp and *.log, *.nfo, *.tgz, *.tmp. Keep getting alerts about .nfos changing. Either ts a bug or I am doing something wrong.  

Link to comment
  • 4 weeks later...
MD5 hash key mismatch, /stats/combnk.html is corrupted
MD5 hash key mismatch, box/fixedpoint/ui/nedVisuals/fxpHistogram/release/images/toolstrip/liveeditor/left-align-24.png is corrupted
MD5 hash key mismatch, box/matlab/datatypes/duration/+matlab/+internal/+duration/getDetectionFormats.m is corrupted
MD5 hash key mismatch (updated), /mnt/disk1/system/docker/docker.img was modified
MD5 hash key mismatch (updated), /mnt/disk1/system/libvirt/libvirt.img was modified

 

I got the above response from an automated check. I understand docker and libvirt normally change but the corruptions I don't understand. I cant figure out what these files are and how to resolve this. 

Link to comment

Does anyone know why my scans always fail to scan any files? I have tried different folder exclusions and even removed all exclusions altogether any yet anytime I select "BUILD" or "EXPORT" for any disk, it scans 0 files for several hours and always has zero files at the end of the scan.

 

I assume I'm doing something wrong but I've tried all different types of settings and hash methods and no dice. Screenshot below.

 

Link to comment
2 hours ago, sunbear said:

I assume I'm doing something wrong but I've tried all different types of settings and hash methods and no dice. Screenshot below.

 

Your system got OOM ( out of memory ) call trace, you need found out what cause this.

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.