Dynamix File Integrity plugin


bonienl

Recommended Posts

6 hours ago, Benson said:

FIP won't cause write to data disk, just some after hash build.

 

So is it finding some hashes that are not built?  It seems very consistent, and FIP always tells me that all the hashes are up to date.  Some of the disks where this writing while checking has happened have not had new files written to them in some months. 

Edited by S80_UK
Link to comment
  • 2 weeks later...

The use is explained multiple times in this thread already.  But briefly...  You install the plugin and it generates additional data (hashes) that are mathematically derived from the contents of each file.   The plugin monitors your server for added files and deliberate changes to files so that the hashes are kept up to date.  You also set up a schedule that the plugin uses to determine when to check each file's contents against its hash.  Typically you set up the schedule to split the checking to a few disks each time, depending on your system capabilities.  If an error is found while checking then that file has become corrupted due to some unknown event (could be a disk problem, for example).  Reporting includes notifications of checks, errors, etc.

 

This software cannot tell you if a file is corrupted if you have not already installed it and generated the hashes, since that process requires you to have known good files.  So it is too late for you to use it to find the files that have already been damaged by your power cut.  By the way, that is a very unlikely event, unless the file was being modified at the time of the power failure.  If you have a small UPS to protect your server that situation is easily avoided. 

Edited by S80_UK
Link to comment
28 minutes ago, S80_UK said:

The use is explained multiple times in this thread already.  But briefly...  You install the plugin and it generates additional data (hashes) that are mathematically derived from the contents of each file.   The plugin monitors your server for added files and deliberate changes to files so that the hashes are kept up to date.  You also set up a schedule that the plugin uses to determine when to check each file's contents against its hash.  Typically you set up the schedule to split the checking to a few disks each time, depending on your system capabilities.  If an error is found while checking then that file has become corrupted due to some unknown event (could be a disk problem, for example).  Reporting includes notifications of checks, errors, etc.

 

This software cannot tell you if a file is corrupted if you have not already installed it and generated the hashes, since that process requires you to have known good files.  So it is too late for you to use it to find the files that have already been damaged by your power cut.  By the way, that is a very unlikely event, unless the file was being modified at the time of the power failure.  If you have a small UPS to protect your server that situation is easily avoided. 

 

Perfectly explained.

I got the plugin late :-(

I also got the UPS.

 

Thankyou
Gus

 

Link to comment

Is there another way then to find those corrupted files I got?

If the corrupted files are movies, shows or some junk I have of my disks I can simply delete or re-download it.

But if the corrupted files are familiar pictures or videos then that's more annoying.
I have backups in crashplan of those folders, but I don't really know what to re-download from backup.

 

If I simply try to copy folders from my server to my server, if it finds a corrupted file while I'm copying the file will it simply hang or will I get any nothification or will not get anything at all?

 

How would you guys proceed?

 

Thankyou
Gus

 

Link to comment
45 minutes ago, zzgus said:

How would you guys proceed?

 

Small question - complicated answer...

 

I don't use Crashplan, so my method will not be exactly what you would need.  What I have done in the past (before using this plugin) is to use rsync to generate a checksum of each file on the server and its backup and then compare them.  You will either need to do something similar, or possibly have some other utility do a binary compare of all files in specified folders.  That may not be so trivial if the backup is using Crashplan.  If both sets of files are local, there are various utilities for comparying files at the binary level.  Probably you can use diff at the command line, although occasionally I have even resorted to usng the DOS command line on a Windows PC... fc /b file1 file2   You would probably need to fetch the files or folders from Crashplan first so that you cuold then compare them locally.

 

This is what I used in the past...  My backups are on a second unRAID server,  powered off for most of the time, but which runs the rsync daemon in the background when it is powered up.   I have each disk drive on the main server backed up to a different share on the backup server.  I don't map drives to drives, but drives to shares over the network.  (Don't try mixing access to drives and shares on the same unRAID server, by the way - it can cause problems.)  If I want to compare a disk1 on my main server against its backup, I run a command like this...

 

rsync -rvnc --progress --delete --timeout=18000 /mnt/disk1/ BackupServer::mnt/user/Disk1-backup 2>&1 > /mnt/cache/Disk1_differences.txt & 

 

The -c option tells rsync to use checksums to determine whether to copy a file.  It will do nothing if they are the same.  In my case I also use the -n option so that nothing happens other than a report being produced.  Note that the output of the command is directed into a file on my cache drive for later checking.

 

Using the File Integrity plugin i will not need to do this in the future since the hashes for each file are stored in that file's extended attributes, and they will get updated on my backup server as well.  That allows me to check both servers independently for errors of course.  

Link to comment

As aleady noted, since you didn't have any hash values for your files, there's nothing this plugin or any other checksum utlity can do to help identify corrupted files.   You're also not likely get any notification of errors when copying files from the disk.

 

For those files you have backups of, the best way to confirm you have good copies is to do a binary comparison between the files on the UnRAID server and your backups.   Any files that don't match, you'll need to manually check to see if you can tell which is the good one -- and then replace the other one.    Note that since you don't have checksums for either set of files you have no way of confirming computationally which is actually the correct file.    Doing the comparison is easy -- just any any good comparison utility that does a binary compare.  I use FolderMatch  (not free), but there are several free utilities that can do this as well.   [ http://www.foldermatch.com/ ]     If there are any mismatches, it's clearly a bit more tedious, but hopefully you won't have any (or at least not many) of those.     Since your backups are apparently not local, I'd download a bunch at once onto a PC; then do the comparisons for those;  then repeat as needed until you've checked everything.

 

Adding a UPS was the most important improvement you made -- I've told folks for decades this is MANDATORY accessory for EVERY PC ... a power-protected PC that never undergoes an unexpected "yank" of the power cord (which is basically what a power failure is without a UPS)  has FAR fewer hardware issues than an unprotected PC.   And for a server it's even more important.    Hopefully you bought a quality UPS unit -- as a minimum it should have AVR, and ideally you want a true sine wave output.

 

One other thing you can do to give you some idea of whether there's definitely corruption or not is to see if parity is good on the system.    Do a parity check and see if there are any sync errors detected.    Note that many would suggest doing a non-correcting check, but I see no reason to do that.   You WANT parity to be good, so even if UnRAID's assumption that the error is in the parity isn't correct, that's really the only place it can be corrected, since you have no way to knowing which other disk it might be on.    If there are sync errors, you'll know for sure you had some corruption, and you need to then do the testing outlined above r.e. file comparisons.   Of course if you don't find any, then all you'll know is that the corruption was either on the parity disk (that's good, as it will be resolved by the parity check); or it's on a file you don't have backedup -- in which case there's no way to identify that file.    Unfortunately, even if there are NO parity errors, that doesn't absolutely mean you didn't have some corruption, as a write sequence may have been completed immediately before the power loss but before all file operations were completed.   But if that's the case, there's likely very few files involved.

 

 

 

Link to comment

@S80_UK @garycase

Thankyou for your time to answer and try to help me.

 

Then, I will try to download my crashplan backup and try to compare will my local copy.

I have to check when was the last crashplan backup copy and if crashplan has versions. If the crashplan copy is up to day then there's nothing to do as the two versions will be the same.

 

@S80_UK

57 minutes ago, S80_UK said:

rsync -rvnc --progress --delete --timeout=18000 /mnt/disk1/ BackupServer::mnt/user/Disk1-backup 2>&1 > /mnt/cache/Disk1_differences.txt & 

 

What are the "2>&1" and the last "&" stand for?

 

 

@garycase

Quote

Hopefully you bought a quality UPS unit -- as a minimum it should have AVR, and ideally you want a true sine wave output.

 

I think I got the correct one :-)       (APC BACK-UPS 700VA, 230V, AVR, SCHUKO Sockets)

 

Quote

Do a parity check and see if there are any sync errors detected. 

 

Yes I got 712 sync errors. Corrected parity later.

 

Thankyou

Gus

 

Link to comment
25 minutes ago, zzgus said:

 

@S80_UK

 

What are the "2>&1" and the last "&" stand for?

 

 

2>&1 redicts the stderr output from rsync to the destination file rather than the regular stdout which by default is what would appear on the screen. See here https://superuser.com/questions/436586/why-redirect-output-to-21-and-12

 

The & at the end causes this command to be spawned as a separate task which will complete in the background.  You can leave that off if you want to run in the foreground with the command prompt not returning until the end of the process.  I'm still learning - I know just enough Linux to be dangerous!

Edited by S80_UK
Link to comment
  • 3 weeks later...
  • 3 weeks later...

Question for everyone. I have been using this plugin for a while but completely missed the setting that allows you to ignore specific file extensions and now when it runs the verification task my .nfo's are almost always flagged as having been modified. Whats my best course of action to prevent getting that every time it runs? Is there a way to remove the hash from a specific file type in batches?

Link to comment
On 24/03/2018 at 7:06 PM, hermy65 said:

Question for everyone. I have been using this plugin for a while but completely missed the setting that allows you to ignore specific file extensions and now when it runs the verification task my .nfo's are almost always flagged as having been modified. Whats my best course of action to prevent getting that every time it runs? Is there a way to remove the hash from a specific file type in batches?

 

It's all explained if you turn on Help.

 

First exclude the files ending with .nfo by going to Settings -> File Integrity Settings and typing *.nfo in the Excluded Folders and Files -> Custom Files box. You can have a comma-separated list, such as *.tmp,*.nfo,*.temp. Click Apply.

 

Then go to Tools -> File Integrity and select the affected disk(s). Then click the Clear button. If you export your checksums then click Export to update the export file.

 

Link to comment
  • 4 weeks later...

Got a weird error today after my weekly integrity check. I seem to have a blank file giving a hash key mismatch.

BLAKE2 hash key mismatch,  is corrupted

I tried doing a rebuild and export before another check but the error still came up. Any idea what it could be?

Link to comment
1 hour ago, John_M said:

 

A file with an illegal character in its name, perhaps?

 

I've thought about that too so I've tried running an "Extended Test" using the Fix Common Problems plugin which checks every file/folder for invalid characters  and permission issues. Nothing came up.

Link to comment
  • 1 month later...
  • 1 month later...

I had a failing drive (since replaced), and now I'm running a Check on that drive. It's already reported 295 corruptions after about ~42k of ~256k files have been checked. I have backups (CrashPlan) which I hope haven't picked up any corrupted files.

 

Once the check has completed and I know how many files are corrupted, how do I go about finding out which ones are corrupted so I know what to restore from CP?

 

-- I did reread the OP and didn't see anything about it there. My apologies for not reading all 28 pages of the thread. If this has been answered, a pointer to where will be sufficient and I'll happily read the already written response.

Link to comment

Frankly, I don't think this thing can be trusted to do what it is intended to do.  If it's flagging even one file as being corrupt, as in my case, when other file hashing tools come back and say both copies actually have the same hash then there's a problem somewhere with this tool.  I wouldn't necessarily put too much stock into it saying you've got 295 corrupted files.  If you just replaced the drive and consequently things were just rebuilt from parity I would be more inclined to think your data is just fine.

Edited by _Shorty
Link to comment
10 hours ago, _Shorty said:

If you just replaced the drive and consequently things were just rebuilt from parity I would be more inclined to think your data is just fine.

 

Except for the fact that the drive was throwing errors prior to me being able to get it replaced. I think parity was good beforehand and that the new parity drive has valid data on it, but I have no way of being sure.

 

Plus, I was going through pictures and I've already found one that Photoshop says has bad headers so it won't open it. :(

Link to comment
  • 2 weeks later...

Im sure its super simple however I have excluded all the folders that will cause lots of issues, like incomplete downloads and appdata etc...
image.thumb.png.b7daf4a7ed263ff98cff8e8e40568462.png
As shown, but i'm still getting false errors like so:
SHA256 hash key mismatch (updated), /mnt/disk1/appdata/binhex-minidlna/files.db was modified
SHA256 hash key mismatch (updated)

and lots of them, how do i ignore the appdata folder?

 

 

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.