Dynamix File Integrity plugin

[Gizmotoy]

Hmm.. Just got a notification that Disk 2 needs updating = 2 files didn't have the checksum. No indication which ones.

 

Updated, again, nothing. The only record in the syslog:

bunker: added 2 files from /mnt/disk2. Duration: 00:10:02. Average speed: 96.4 MB/s

 

A suggestion: when the plugin notices that a disk needs updating because some files don't have checksums, could the Build process (and the Export process, if enabled - ie. orange circle, not blue cross) be triggered automatically? Perhaps include an option to enable this behaviour or leave it to be done manually.

Sounds like a good idea, but with the problems noted above, I think this could end up just thrashing our drives in an endless loop.  Mine are endlessly showing files missing checksums.  Making it optional sounds ok, though.

[bonienl]

Hmm.. Just got a notification that Disk 2 needs updating = 2 files didn't have the checksum. No indication which ones.

 

Updated, again, nothing. The only record in the syslog:

bunker: added 2 files from /mnt/disk2. Duration: 00:10:02. Average speed: 96.4 MB/s

 

A suggestion: when the plugin notices that a disk needs updating because some files don't have checksums, could the Build process (and the Export process, if enabled - ie. orange circle, not blue cross) be triggered automatically? Perhaps include an option to enable this behaviour or leave it to be done manually.

Sounds like a good idea, but with the problems noted above, I think this could end up just thrashing our drives in an endless loop.  Mine are endlessly showing files missing checksums.  Making it optional sounds ok, though.

 

I'll have a look and see if I come up with something workable 

 

[hermy65]

Is there any way to leverage this plugin or maybe some other solution to notify you if files that already have checksums are missing?

 

Use case: Im 99% positive i moved all of our baby pictures to the server weeks ago and yet today i cannot find them anywhere. Maybe i accidentally deleted them somehow or some other magic happened. Basically im looking for something that can tell me if something that was there is no longer there anymore.

[trurl]

Is there any way to leverage this plugin or maybe some other solution to notify you if files that already have checksums are missing?

 

Use case: Im 99% positive i moved all of our baby pictures to the server weeks ago and yet today i cannot find them anywhere. Maybe i accidentally deleted them somehow or some other magic happened. Basically im looking for something that can tell me if something that was there is no longer there anymore.

Google linux find command. You should be able to come up with something to search /mnt.

[John_M]

Is there any way to leverage this plugin or maybe some other solution to notify you if files that already have checksums are missing?

 

If you exported the checksums you can search the resulting hash file for your missing photos, since it consists of a list of filenames with associated checksums. If you didn't export the checksums the copy held in the extended attributes will have been lost when the files were deleted.

 

[FreeMan]

I just got a notification that the plugin found 1 file with hash key corruption.  I'm not terribly worried as I just re-formatted the disk from reiserfs to xfs, and I'm copying data to it, so I have the full set of original files.

 

However, I'm wondering if the corruption could have come from DFI calculating and storing a hash on a partially copied file, and the check calculating the hash on the fully copied file.

 

On the bright side, I do really appreciate this plug in and the fact that my Pushover message specified the file that failed (even though the WebGUI pop-up doesn't, and I'm OK with that)!

[drumstyx]

So does automatic really mean automatic?

 

I have it set to automatically protect new/modified files, but I get "needs Build & Export updating" notifications every day, even with no changes.

[John_M]

It is automatic, but I think some files slip through unnoticed. Hence my comment here. If you do a manual build after getting the notification that should bring things up to date, at least until you write more files.

 

[drumstyx]

It is automatic, but I think some files slip through unnoticed. Hence my comment here. If you do a manual build after getting the notification that should bring things up to date, at least until you write more files.

 

I wonder if there's any way to schedule a build after a successful schedule validation...

[Msan]

It also seems to miss a lot of files on my system on a regular basis..

 

Here is a wierd display issue.. I am running a manual build, but the icon's don't update

 

see attached..

[John_M]

Mine seems to miss the regular backups made by the Community Apps appdata Backup plugin. A limitation of inotify, perhaps? The icons not updating might be a browser cache issue - try clearing it.

[tr0910]

Is the last bunker 1.16 still compatible with this tool? Using bunker for exporting and checking that files are not corrupted after rsyncing to 2nd server.  I have an external backup unRaid server rsynced over the internet that I want to verify by exporting the keys from the production machine and verifying with.

 

bunker -c -f /tmp/disk1_keys.txt

 

Where the disk1_keys.txt is exported from the server running this plugin and the destination backup server uses bunker to validate the rsynced backup files.  I have done this several times in the last year successfully, but I want to know if I am just lucky, or that this is expected behavior?

[tr0910]

Is the last bunker 1.16 still compatible with this tool? Using bunker for exporting and checking that files are not corrupted after rsyncing to 2nd server.  I have an external backup unRaid server rsynced over the internet that I want to verify by exporting the keys from the production machine and verifying with.

 

bunker -c -f /tmp/disk1_keys.txt

 

Where the disk1_keys.txt is exported from the server running this plugin and the destination backup server uses bunker to validate the rsynced backup files.  I have done this several times in the last year successfully, but I want to know if I am just lucky, or that this is expected behavior?

 

Answered here by @bonienl:

 

http://lime-technology.com/forum/index.php?topic=37290.msg524470#msg524470

 

Inside "\\tower\flash\config\plugins\dynamix.file.integrity\integrity-check.sh" I find the following

 

/usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -Vqj  -n -L /mnt/disk1 >/dev/null

 

What is the new -j option in bunker?  I don't see it documented in the initial option run list.  I'm guessing there are lots more undocumented features to bunker.  You  must have an exclude option now, and ...

 

Pretty please could we have these documented???

[bonienl]

It is documented in the source code of bunker...

 

  1) one=1 ;;           # hidden option for GUI front-end - process single file
  x) mon=1 ;;           # hidden option for GUI front-end - output progress to monitor file
  j) job=1 ;;           # hidden option for GUI front-end - give start-stop signalling
  z) not=1 ;;           # hidden option for GUI front-end - invert file selection (exclude only)
  E) exclude=$OPTARG ;; # hidden option for GUI front-end - list of excluded folders (shares)
  F) folders=$OPTARG ;; # hidden option for GUI front-end - list of custom folder exclusions

 

These options are for the GUI front-end and not used in CLI mode.

 

[tr0910]

I have 12 disks with between 50,000 and 300,000 files each.  One disk has the ransomware plugin installed with close to 1MM files from it in the bait directories.  It seems to me that the integrity file plugin is not getting all legitimate files.  When I exported disk1, I only got 10,000 files exported keys hash file, but when I use create a listing of all the files on disk1 I get 35,000 files doing it this way...

 

find /mnt/disk3 -type f -fprintf /mnt/disk3/disk3.filelist "%P\n"

 

How is the best way to verify that the integrity plugin isn't missing files??

 

Might it have something to do with inotify limits??  (this disk13 is a snapshot drive with hard links containing millions of links to about 100,000 files).  At the time this drive wasn't excluded.  I've also bumped the size max_user_watches from 500,000 to 2,000,000 files but haven't restarted the server since I have done it.

 

Dec 23 14:16:11 Kim inotifywait[14210]: Failed to watch /mnt/disk13; upper limit on inotify watches reached!
Dec 23 14:16:11 Kim inotifywait[14210]: Please increase the amount of inotify watches allowed per user via `/proc/sys/fs/inotify/max_user_watches'.
Dec 23 14:16:18 Kim kernel: sd 1:0:1:2: [sdl] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
Dec 23 14:16:18 Kim kernel: sd 1:0:1:2: [sdl] tag#0 Sense Key : 0x5 [current] 
Dec 23 14:16:18 Kim kernel: sd 1:0:1:2: [sdl] tag#0 ASC=0x20 ASCQ=0x0 
Dec 23 14:16:18 Kim kernel: sd 1:0:1:2: [sdl] tag#0 CDB: opcode=0x85 85 06 20 00 00 00 00 00 00 00 00 00 00 40 e5 00
Dec 23 14:16:18 Kim kernel: sd 1:0:0:3: [sde] tag#0 UNKNOWN(0x2003) Result: hostbyte=0x00 driverbyte=0x08
Dec 23 14:16:18 Kim kernel: sd 1:0:0:3: [sde] tag#0 Sense Key : 0x5 [current] 
Dec 23 14:16:18 Kim kernel: sd 1:0:0:3: [sde] tag#0 ASC=0x20 ASCQ=0x0 

 

(Background: I was using bunker before using this plugin.  All disks were fully hashed with bunker prior to this plugin being installed but the disk with the missing files has had significant new files added over the last few months.  This plugin was installed as soon as it was available.  So, what I am saying, is there is legacy stuff here possibly hanging around from the olden days??  )

[RAINMAN]

Im curious on what could cause this.  I got a corruption message on a file. 

 

BLAKE2 hash key mismatch, /mnt/disk5/xxxxxxxxxxxxxxxxxxxx.1080p.BluRay.X264-AMIABLE.mkv is corrupted

 

I check the hash in the export

 

ab681975c59300f875b81763f4e3ca7e1186240c04400abf71939aa43d762c803cd6a43530a05ceb2d869250e32388045600c0fa26b135b2e69c034fbd939c95 */mnt/disk5/Movies-HD/xxxxxxxxxxxxxx.1080p.BluRay.X264-AMIABLE.mkv

 

I re-downloaded the file from my online backup move it to my array and it gets rehashed into disks.export.20161230.new.hash

 

ab681975c59300f875b81763f4e3ca7e1186240c04400abf71939aa43d762c803cd6a43530a05ceb2d869250e32388045600c0fa26b135b2e69c034fbd939c95 */mnt/disk1/Movies-HD/xxxxxxxxxxxxxxxxxxx.1080p.BluRay.X264-AMIABLE (2).mkv

 

The hashes are identical...

[bonienl]

This may happen when the hash computation gets interrupted, leaving an incomplete result. Usually only happens when the system gets (too) busy.

 

[FreeMan]

I've come across a strange issue where File Integrity is showing my disks as barely used, running the checks within seconds to minutes, then claiming to be done (attachment 1), while the disks (1 & 2 TB in this particular case) are stuffed to within an inch of their lives (attachment 2).

 

The third attachment shows that the build for both of these disks is currently up to date (just noticed that Disk1 needs to be exported, so I'm doing that now).

 

The fourth attachment shows my current DFIT config - it's set to run daily with 3 groups (which may be overkill to check that often, I get that, but it's how I've got it for now...). Disks 3 & 9 have not been converted from ReiserFS to XFS yet, so they're excluded from the checks.

 

Is there something in the config that might be throwing this off? Do you need to see a log from somewhere to track it down? These are not the only 2 disks this is happening to. There are a couple of others, but I don't remember which ones off the top of my head, so I'll have to wait for the daily scans to come around again to see which ones they are.

[John_M]

According to your pushover notifications disk1 has only 17.8 GB of files to verify, so 4 minutes 24 seconds seems reasonable; similarly disk4 has only 22 MB of files to verify, which takes 2 seconds. So it looks as though many of your files are being missed or excluded from being hashed in the first place. Have a look at the exported hash files to see what's happening.

 

[bonienl]

Doing a manual Build will add any files missing previously.

 

It is recommended to exclude folders/files which change frequently, here the normal error detection of the disk will take care. Detection of bitrot is more meaningful on folders/files which are not accessed or modified frequently.

 

[jeffreywhunter]

I'm getting syslog errors stating I need to increase the number of inotifywatches so that dynamix file integrity will work properly. 

 

I've looked at both the File Integrity Control page and the File Integrity Settings page, but not seeing any parameter called inotifywatches.  I've not seen any advanced tabs on any of those pages.  Where should I look?

 

Thanks!

[FreeMan]

OK, so it's been 2 weeks, but I'm back on this now.

 

in /boot/config/plugins/export/disk1.export.hash, I have 41 files listed. Now, a couple of these are movies, but none of them are big enough to account for a 1TB drive with 20.5kb free. Is the 20.5kb free the issue? Also, the export.hash file claims that it has a hash for a file that doesn't exist on that disk.

 

It seems that the disks that are having issues are the ones with about 1MB or less free space. I'm not entirely sure how all my disks have gotten this full, since all the shares are set to a minimum of 1GB.

 

Would I be best served by manually rebalancing the disk usage to free up some space, then force all disks to rebuild & reexport?

[trurl]

It seems that the disks that are having issues are the ones with about 1MB or less free space. I'm not entirely sure how all my disks have gotten this full, since all the shares are set to a minimum of 1GB.

Minimum free doesn't keep a disk from filling up. It just keeps unRAID from writing to a disk unless it has the minimum free. If you set it to 1GB, and it has 1.5GB free, then it will write to the disk regardless of how large the file will be because it doesn't know how large the file will be.

 

If you have minimum set to 1GB, and you have 1.5GB free, and you try to write a 2GB file, it will fail due to not enough space.

 

If you have minimum set to 1GB, and you have 1.5GB free, and you try to write a 1.4GB file, it will succeed because there is enough space, but then it won't write any other files because it no longer has 1GB free.

 

So, you must set minimum free to be larger than the largest file you ever expect to write.

[jeffreywhunter]

I'm getting syslog errors stating I need to increase the number of inotifywatches so that dynamix file integrity will work properly. 

 

I've looked at both the File Integrity Control page and the File Integrity Settings page, but not seeing any parameter called inotifywatches.  I've not seen any advanced tabs on any of those pages.  Where should I look?

 

Thanks!

 

Just a bump on this.  Is the solution to add a line to the go file to increase max_user_watches?

 

I currently have 524288 max_user_watches (from "cat /proc/sys/fs/inotify/max_user_watches")

 

How many should I increase them to?

 

echo ??? > /proc/sys/fs/inotify/max_user_watches

(change the "???" to what?, Double?  50% more?)

 

Thanks in advance...

[FreeMan]

So, you must set minimum free to be larger than the largest file you ever expect to write.

 

Thanks, trurl.

 

Would freeing up space on the drives help with this DFI issue? Since, as I read it in the OP, the hashes are being added to the files, I can see where the additional disk space requirement might be exceeding what's physically available and thus preventing the hashes from being added. I can see where DFI would then think there were significantly fewer files on the disk since only a few have hashes added to them.