Dynamix File Integrity plugin


bonienl

Recommended Posts

1 hour ago, JustinChase said:

I couldn't find any "log tab", but there is a "log files" folder on the plugin page.  The only thing in that folder is a file called duplicate_file_hashes.txt

 

That only lists all the duplicate hashes it found, but nothing about any errors, or help figuring out which file triggered the error.

 

This is an example in log tab, it show a file "test" at /mnt/disk12 have problem, not in plugin tab (page).

 

l.png.829e090d5a1f06863d2aef5e4e74e62a.png

 

And you need enable logging in plugin setting.

 

s.png.52a4cd25610cfc818e1c8b1c4b622de9.png

Edited by Benson
Link to comment

Ah, that helps; thank you!

 

doesn't make as much sense as being in the log folder on the plugin page, but now I know.

 

it actually shows tons of warnings and errors about SHA256 mismatches and/or missing files.

 

I may have been doing some moving of files when the process was originally running, so maybe that's the issue.  It took about a day to run, and I'm not sure when I can not use the server for that long to run it again, but I'll try to do that soon, and have it re-evaluate all files during that time and see if errors continue.

 

I just installed a brand new parity drive and system drive, both of which passed the preclear, so I'd be surprised if there was bit rot happening already.

 

I guess we'll see soon enough.

 

thanks again for your help.

Link to comment

I deleted all data, and am just rebuiling now, while I use the server, but am not moving files around this time.  We'll see how it goes.  If any issues, I'll try the export function next time.  I read all the descriptions of all the buttons, and there was no obvious 'rebuild' function I could decipher.

 

thanks again for the help.

Link to comment
  • 4 weeks later...

I recently ran a dynamix file integrity test on a drive that has read errors. According to the log there are over 90 MKV files with a SHA256 hash key mismatch but even though it states that the files are corrupt they play perfectly. I used a hash file manager to rewrite the hash keys for the files but this did not work. How can I fix these files?

Link to comment
  • 3 weeks later...
  • 3 weeks later...
  • 1 month later...

I ran build and export on one of the discs in my server. During the export process many files were skipped and many corrupt files were found. I am unable to locate any corrupt files because no log was posted after the export process. After this,  I ran verification on disc 4. The process ran for 31 hours and no log was posted. How can I locate the corrupt files?

Link to comment
On 5/21/2020 at 8:42 AM, cdixon said:

I used a hash file manager to rewrite the hash keys for the files but this did not work. How can I fix these files?

I am not sure what means ( or how ) the hash file manager do it. If too much file show corrupt, then best way was "Remove" extended attributes and "Build" it again.

 

13 hours ago, cdixon said:

During the export process many files were skipped

Export won't skip file, only Import will have this warning.

 

13 hours ago, cdixon said:

How can I locate the corrupt files?

If no logging enable, then you may never know which file have hash mismatch.

 

 

Pls understand this plugin won't work perfect in some case which already discuss before and plugin should stop develop.

You need understand why wrong hash will generate for a file .... if file still in writing / transfer but FIP hash it, then wrong hash value will generate. I would simple suggest never hash file during file transfer, so best stop auto hash function ( manual execute hash or periodic during no any file in transfer ) . Otherwise, there are no meaning always "Remove" and "Build".

 

In my use case, I store large size media file always, those file are write once, so I haven't issue for those file. But something like frequency update file (i.e. subtitle) will auto-update (correct) during hash check.

I also ( advance manual operation ) export the hash to backup, every-time "Import" hash for all backup file from master. i.e. 10000+ file in backup, this also can confirm does any file in missing backup too.

 

* PS I also use checksum plugin, so I have two ref. source for file(s) really corrupt or not. *

 

1.thumb.PNG.cd6b3fea3b78234fb4f88281079c6a9c.PNG

 

Edited by Benson
Link to comment
  • 2 weeks later...

@bonienl

 

I ran a manual check on a drive and got a lot of BLAKE2 hash key mismatches (mostly .nfo files modified).  I received the details in a notification email, but there's no log file in /boot/config/plugins/dynamix.file.integrity/logs , even though I do have error logging set to "Syslog & Output File".  I also have log files and email notifications from 2018/2019, so I know it was working correctly at some point.  

 

Are log files not created when the check is run manually (as opposed scheduled daily/weekly/monthly)?  I have 14GB free on my flash drive, so it's not an issue of usable space.  What do you suggest?

Edited by nukem492
Link to comment
  • 2 weeks later...

I  am interested in starting using this plugin, but i have some questions:

 

1. If the hashing reults are NOT saved to flash, where are they stored?

2. Is it possible to store hash results log to another path then flash? I have for example used fstab to statically mount a smb share on a remote logserver which holds all my logs, i would want the logs to go directly there instead of moving them from the flash each day with a script (avoiding writing as much as possible to flash) is this possible or do i need to modify the plugin to do this?

Link to comment
  • 2 weeks later...
1 hour ago, dgwharrison said:

Hi, thanks for the plugin, just a minor bug I noticed when running the Find Duplicates command, it says it's checking Disk 8, I've actually removed that disk from the array as it was faulty. Screenshot:

2020-09-12_18-56-43.thumb.png.6a30b24a34d1e49863218b65b7c2983b.png

It sort/check all hash files, it won't check actual disk exist or not. You can simple delete disk8 hash file.

Edited by Benson
Link to comment
  • 2 weeks later...

I searched the thread and didn't find an answer so apologies if this question's been asked.

 

I'm new to this plugin and have my first verification running after the initial check. It's been going for around 8 hours and I'm wondering how far it's progressed. Is there an indicator somewhere? The initial check showed a handy progress bar in the page under Tools.

 

I'm also confused about whether "check" and "verify" are used interchangeably in the documentation. The help says Use the Check command to verify files against a previously exported file but the plugin can be configured to verify with Save new hashing results to flash (which I assume is the referenced "export") disabled, suggesting they're not interchangeable.

Edited by CS01-HS
typo
Link to comment
  • 4 weeks later...

got a question

i had a few blake2  corruptions now i replaced those files. and did a normal check which says ok. SO i did a check with the plugin again it took 19 hours to do all 45k files. The check says finish and it has less corruptions now but when i check the disk2.export.20201011.bad.log

it still shows the old count and the date is also not updated on it. While it finsh the check files this morning.

 

Is there something i missed why it didnt updated this bad.log file? i also dont see any other with a newer date.

Link to comment
  • 2 weeks later...

Found 25 files with BLAKE2 hash key mismatch
warning
BLAKE2 hash key mismatch (updated), blah/blah/aom-u-move/kowai/catacodec.data was modified
BLAKE2 hash key mismatch (updated), blah/blah/aom-u-move/kowai/local_files.data was modified

 

i had in the plugin excluded teh aom-u-move directory but it still shows the directory under that.  these files change often zo i excluded the whole directory but that didnt seems to work.

 

Link to comment
3 hours ago, KoNeko said:

Found 25 files with BLAKE2 hash key mismatch
warning
BLAKE2 hash key mismatch (updated), blah/blah/aom-u-move/kowai/catacodec.data was modified
BLAKE2 hash key mismatch (updated), blah/blah/aom-u-move/kowai/local_files.data was modified

 

i had in the plugin excluded teh aom-u-move directory but it still shows the directory under that.  these files change often zo i excluded the whole directory but that didnt seems to work.

 

If file already hashed before exclusion, you need clear its attribute. You can clear it manually or edit the export file then clear all by GUI and import again.

Link to comment
  • 4 weeks later...

@bonienl

I started using this plugin on a different array the other day and I think I found a bug with the reporting mechanism for the Export being up-to-date with the Build.

 

When mid-Build, I accidently hit the Export button the the webUI with all the disks selected.  The build continued, but an export was also done on all the devices in my array with the current files where hashes were computed.  I figured, these would almost certainly contain incomplete data, but I'd make a new export when the build completed the next day.  Before I did however, I wanted to see if the Export was incomplete AND the webUI reported that the state was stale.  When checking the export files, I was correct that the data was incomplete and not all the hashes were in the export files.  BUT, the webUI greeted me with all green check marks on the export status for each device.

 

This left me with a bigger question.  How does the plugin determine if the build and export are up-to-date?

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.