I recently changed Error Logging from Syslog Only to Syslog and Output file.

 

 

After a check I see the entries in syslog but can't find the "output file" - any idea where it's stored? (I saw a couple of posters ask but didn't see an answer.)

 

Now I get mismatches because Nextcloud files_trashbin too. This plugin likes to give false positives when you have Nextcloud.

I have 3 servers and this plug in has been installed on each for years. Everything has worked well. So much so that I frequently forget to check back on them for months. I happened to be doing other things in the tools tab and had an 'oh yeah FIP' moment. So I started just looking around to be sure everything was still running smoothly and found that 2 of my machines were running MD5 and one was running SHA2 and I couldn't for the life of me remember why. I turned on the help setting (consistently amazed at how awesome the help is) and it reminded me of the blake2 option which I think I had passed on originally because it said it was proprietary and I had some aspirations of checking my backups externally or some such that of course I never did.  I did a little quick googling of blake2 and learned that there is now a blake3 that is approx 5x as fast as blake2.  Has any thought been given to adding this as an option?

Edited December 17, 20205 yr by jasonfox
typos

My scheduled verify job reported "bunker verify command Found 1 file with BLAKE2 hash key corruption" but doesn't tell me what file is corrupted. I had error logging set to syslog in settings, but I don't see the file listed there either. How can I see what file is causing the problem?

 

Here is the relevant part of syslog:

bunker: error: BLAKE2 hash key mismatch,  is corrupted
bunker: Verify task for disk1 finished, duration: 12 hr, 2 min, 0 sec.
bunker: verified 455797 files from /mnt/disk1. Found: 0 mismatches, 1 corruption. Duration: 12:02:00. Average speed: 98.3 MB/s

 

12 hours ago, jasonfox said:

I have 3 servers and this plug in has been installed on each for years. Everything has worked well. So much so that I frequently forget to check back on them for months. I happened to be doing other things in the tools tab and had an 'oh yeah FIP' moment. So I started just looking around to be sure everything was still running smoothly and found that 2 of my machines were running MD5 and one was running SHA2 and I couldn't for the life of me remember why. I turned on the help setting (consistently amazed at how awesome the help is) and it reminded me of the blake2 option which I think I had passed on originally because it said it was proprietary and I had some aspirations of checking my backups externally or some such that of course I never did.  I did a little quick googling of blake2 and learned that there is now a blake3 that is approx 5x as fast as blake2.  Has any thought been given to adding this as an option?

Speedup should be multiple core/thread only. For array even SSD ( single disk limitation ), expect no much benefit. ( current max speed ~500MB/s+ )

Edited December 17, 20205 yr by Vr2Io

I received an unraid alert for "Found 1 file with SHA256 hash key corruption". How do I tell which file is corrupt? No log file was created. My last dynamix log file is from 2019. Thanks.

 

EDIT: I ran this cmd to find it, but surely there has to be a better way? Why is it not logging? "cat /var/log/syslog | grep corrupt"

Edited December 20, 20205 yr by pman

I posted about a similar problem above. For me, syslog does not say what the corrupted file is.

On 4/3/2019 at 8:24 PM, jeffreywhunter said:

I'm see the following error in my log, but no reference to any bad files.

 

BLAKE2 hash key mismatch,  is corrupted

Was this ever resolved? I'm seeing it now.

same here:

two of the corrupt files are not mentioned and I don't know what/where the third one is

Jan  6 03:31:59  bunker: error: BLAKE2 hash key mismatch,  is corrupted
Jan  6 03:51:41  bunker: error: BLAKE2 hash key mismatch,  is corrupted
Jan  6 03:58:50  bunker: error: BLAKE2 hash key mismatch, ng/fs.json is corrupted
Jan  6 04:02:59  bunker: Verify task for disk1 finished, duration: 1 hr, 2 min, 58 sec.
Jan  6 04:02:59  bunker: verified 198104 files from /mnt/disk1. Found: 0 mismatches, 3 corruptions. Duration: 01:02:58. Average speed: 36.1 MB/s

 

Can confirm what others have said about false positives.  Said one of my bluray images was corrupt, and I rechecked it against the original and the image is 100% fine.

Me i'm kind of confused with this plugin. I've used it since forever. But it's one of those that well, if you don't get errors then you are good. But I'm kind of hoping to see a log with all good. All I have are the exports from 2019 and no logs. Currently i'm doing a manual check to see. I'm hoping to see a notification or something. 

On 12/17/2020 at 2:51 AM, Fizz said:

My scheduled verify job reported "bunker verify command Found 1 file with BLAKE2 hash key corruption" but doesn't tell me what file is corrupted. I had error logging set to syslog in settings, but I don't see the file listed there either. How can I see what file is causing the problem?

 

Here is the relevant part of syslog:

bunker: error: BLAKE2 hash key mismatch,  is corrupted
bunker: Verify task for disk1 finished, duration: 12 hr, 2 min, 0 sec.
bunker: verified 455797 files from /mnt/disk1. Found: 0 mismatches, 1 corruption. Duration: 12:02:00. Average speed: 98.3 MB/s

 

Mine got sent in an email from the smtp that i previously setup

I'm looking for some clarification on the Disk Verification Schedule and the Tasks.

 

So the help says

Quote

when you have set up a weekly schedule with 3 tasks then every week another task is executed and after 3 weeks the verification function has done all selected disks.

 

So in this example you have 3 disks and you have put each disk on a separate task. My understanding of the above is that Task 1 executes on the Monday, then the following Monday Task 2, and the following Monday after that Task 3. But then when does it restart? The next Monday after the week with Task 3 finishing its back to Task 1?

 

If I select Monthly, is that going to run Task 1 in Jan, Task 2 in Feb, Task 3 in March?

 

Is there anyway to set it up so it checks all disks once a month but separate the tasks by a day or a week?

 

Also on the Integrity page in the help it states "0 represents an open operation", does that mean a running operation? Like it's actively doing something in the background?

Edited February 8, 20215 yr by iarp

Is there a solution regarding false positives?

I've just installed and started using the File Integrity plugin today.

• I manually started a `Build` process on 7 (out of 28) drives in my array
• disk1 had least amount of files, so it has already finished

But, the UI shows some nonsense 🤔

• it shows disk1 as a circle, not green checkmark, even though it has just finished the build, and is up-to-date
• it shows disks 4, 5, 9 and 10 with a green checkmark, even though the builds are clearly still running and aren't finished
• it shows disks 7, 12, 17, 18, 19, 22, 23, 24, 26, 27 and 28 with a green checkmark, even though the build process has never been run on these disk...

I mean, WAT? 😳

Can it be, that am I really not understanding what the circle/checkmark/cross means?

 

Or is this a bug?

 

I'm seeing similar notifications about 3 supposedly corrupted files, but no names!

 

BLAKE2 hash key mismatch,  is corrupted
BLAKE2 hash key mismatch,  is corrupted
BLAKE2 hash key mismatch,  is corrupted

 

The corresponding syslog entries are also missing file info, for example:

 

Feb 28 18:17:44 Tower bunker: error: BLAKE2 hash key mismatch, is corrupted

 

What is the cause of this (apparently I'm not the only one seeing it), and how can I fix it - is there a way to reset the file hashes?

 

Another false positive on a bluray dump (different image).  I've verified it with the original and everything hashes 100%.  I'm only getting 1 mismatch every month, but this is 2 for 2 where it has been a false positive.  Is there any way to address this or am I just stuck at having to double check files when it says they are corrupt?

20 minutes ago, Ranmacanada said:

Another false positive on a bluray dump (different image).  I've verified it with the original and everything hashes 100%.  I'm only getting 1 mismatch every month, but this is 2 for 2 where it has been a false positive.  Is there any way to address this or am I just stuck at having to double check files when it says they are corrupt?

This is know issue, I naver have bluray image ( or static file ) false positive, because I hash file by "build" in manual not automatic.

On 10/13/2019 at 10:15 PM, Ouze said:

 

Happened again - did a check, out the output:

 

"bunker: error: SHA256 hash key mismatch, rnal.720p.bluray.x264-reward.mkv is corrupted"

 

According to a search, that narrows it down to 125 files. 

 

There has got to be a way to get this plugin to output more useful (specific) information, right? Why are these error messages being truncated this way?

 

I did an md5 check against all 125 files individual checksums and of course, they pass just fine.  These files got dropped onto the server once and have not been touched since then. If this plugin is going to generate output vague as it is, and as prone to false positives as it seems to be; then ultimately it's just contributing noise, not value. 

 

 

 

Did you finally found out why file names/paths are truncated? It also happens to me, and can be very annoying to find the exact file which is "supposed" to be corrupted.

Is there a way to change the location where the hash results for each file are stored? I see they're stored in plain text on my usb drive. I'd like the results to be in an encrypted location such as my cache or array.

Files dont seem to be ignored in my set up I have tried .log, .nfo, .tgz, .tmp and *.log, *.nfo, *.tgz, *.tmp. Keep getting alerts about .nfos changing. Either ts a bug or I am doing something wrong.  

MD5 hash key mismatch, /stats/combnk.html is corrupted
MD5 hash key mismatch, box/fixedpoint/ui/nedVisuals/fxpHistogram/release/images/toolstrip/liveeditor/left-align-24.png is corrupted
MD5 hash key mismatch, box/matlab/datatypes/duration/+matlab/+internal/+duration/getDetectionFormats.m is corrupted
MD5 hash key mismatch (updated), /mnt/disk1/system/docker/docker.img was modified
MD5 hash key mismatch (updated), /mnt/disk1/system/libvirt/libvirt.img was modified

 

I got the above response from an automated check. I understand docker and libvirt normally change but the corruptions I don't understand. I cant figure out what these files are and how to resolve this. 

Does anyone know why my scans always fail to scan any files? I have tried different folder exclusions and even removed all exclusions altogether any yet anytime I select "BUILD" or "EXPORT" for any disk, it scans 0 files for several hours and always has zero files at the end of the scan.

 

I assume I'm doing something wrong but I've tried all different types of settings and hash methods and no dice. Screenshot below.

 

Attached Diagnostics.

minint-petb45-diagnostics-20210610-2021.zip

2 hours ago, sunbear said:

I assume I'm doing something wrong but I've tried all different types of settings and hash methods and no dice. Screenshot below.

 

Your system got OOM ( out of memory ) call trace, you need found out what cause this.