Dynamix File Integrity plugin

[itimpi]

5 minutes ago, isvein said:

So this plugin wont add much on zfs formatted array drives as it has the scrub and checksum already?
and if a drive goes down, we always have good old parity as usual

 

Yes.  It really only adds value if you have XFS (or RieserFS) formatted drives in the array.

[Duckers]

hoi, what's "task 1-4" in file integrity in settings for?

As i want to queue the disks as all 4 data drives when building integrity caps out my poor i5-7400 lol. So i want to take them in turn to get the max speed per disk without fully capping the cpu until it's upgraded to handle anything.

[Swirl3208]

19 hours ago, itimpi said:

 

In terms of having them as array drives then ZFS and BTRFS offer very similar capabilities. in protecting against bitrot.   An unresolved question at the moment is whether ZFS or BTRFS will prove more resilient in such a configuration if hardware problems are encountered, and if performance is the same (I expect this is likely).

 

If we are using ZFS or BTRFS in the array, and bit rot is detected (assuming only on 1 disk), does unraid automatically recover and rebuild the corrupted block using the parity information? If not, then how do we recover when bit rot is detected on 1 drive?

On the other hand if we are using xfs with the file integrity plugin, how do we recover from bit rot detection?

[Swirl3208]

14 hours ago, Duckers said:

hoi, what's "task 1-4" in file integrity in settings for?

As i want to queue the disks as all 4 data drives when building integrity caps out my poor i5-7400 lol. So i want to take them in turn to get the max speed per disk without fully capping the cpu until it's upgraded to handle anything.

 

@Duckers

 

I believe the tasks run one after another. They used to be called groups as well in the past if that helps you understand a bit better.

 

Lets say in Task 1 you put disk 1 and disk 2. In Task 2 you put disk 3 and disk 4. The plugin will start calculating the hash for disk 1 and disk 2 in parallel. When those 2 disks are finished running, the plugin will calculate the hash for disk 3 and disk 4 in parallel.

[JorgeB]

1 hour ago, Swirl3208 said:

If not, then how do we recover when bit rot is detected on 1 drive?

You restore from a backup, same for xfs with the plugin. 

[Duckers]

How do i know if anything has been indexing? Indexed one disk at a time, but nothing indicates that disk 1 has been indexed.  No files or anything. I indexed disk 1 for 2 days. 

Edited June 25, 2023 by Duckers

[Terebi]

Is there a way to configure the plugin to do more incremental checks? Rather than a weekly/monthly check that takes 8 hours, can I do an hour a night?

[Nutt]

Hello,

I have 3 issues with this plugin and I cannot find any answers.

 

1. Build and verification progress (and ETA) are calculated only for part of files. After that, progress is 100%, but build and verification continue for the rest of the files (screen attached).

2. Automatically protect new and modified files is not working. When I add or modify a file, "Export up-to-date" status changes from "completed operation" to "open operation," but the file has no extended attributes generated or updated hash. I have to run build manually to generate missing hashes.
3. I think as a consequence of 2., a new hashing result daily file is not generated.

 

My configuration:

 

I wonder if rsnapshots and multiple links to the same file can be sources of problems.

Edited June 28, 2023 by Nutt

[milouz]

 

On 11/15/2022 at 5:51 PM, iarp said:

It won't. The move script that mover uses is not standard, its custom written and seems to copy to .partial, deletes the original and then renames the partial to the real name. That renaming does not trigger inotifywait.

is there any news on that?
What can I do to make the plugin automatically calculate hash for new files that where moved by the mover from cache to array?
is it possible to make the plugin calculate hash for file even if it is on cache and not on the array?
I discovered that I have a lot of files without blake3 hashes despite '"Automatically protect new and modified files" option is enabled.

[Interstellar]

Is this plugin still under development?

 

I'm still suffering from the 128MB PHP limit and there is still no way of manually triggering a disk verify.

 

Is there an alternative plugin that can achieve the same thing and works without editing stuff for the memory limit and setting a temporary scheduled verify?

Edited July 22, 2023 by Interstellar

[Waddoo]

I am currently trying to iron out my file integrity plugin's settings however, I currently have multiple excluded shares and custom files that continue to appear on my verification's logs. Namely Nextcloud and media. Linux ISOs. and *.log files appearing in the logs results. 

 

Is there something I am not selecting correctly in the checkbox for excluded shares?

 

What is the proper format to exclude multiple custom files? It reads:

Quote

Multiple exclusions are separated by a comma.

However I am finding that `*.nfo, *.log` files continue to appear in the logs, even for modifications?

 

I am running Unraid 6.11.5 and have attached a screenshot of my settings and a snippet of the verification log. Thanks and any help would be greatly appreciated.

 

Event: unRAID file corruption
Subject: Notice [ALEXANDRIA] - bunker verify command
Description: Found 2 files with MD5 hash key corruption
Importance: alert

MD5 hash key mismatch, /mnt/disk2/backups/Unraid/CA_Backup/Appdata_Backup/[email protected]/CA_backup.tar is corrupted
MD5 hash key mismatch, /mnt/disk2/backups/Unraid/CA_Backup/Appdata_Backup/[email protected]/backup.log is corrupted
MD5 hash key mismatch (updated), /mnt/disk2/media/LINUXISO/mint.nfo was modified

 

[Waddoo]

12 hours ago, Waddoo said:

I am currently trying to iron out my file integrity plugin's settings however, I currently have multiple excluded shares and custom files that continue to appear on my verification's logs. Namely Nextcloud and media. Linux ISOs. and *.log files appearing in the logs results. 

 

Is there something I am not selecting correctly in the checkbox for excluded shares?

 

What is the proper format to exclude multiple custom files? It reads:

However I am finding that `*.nfo, *.log` files continue to appear in the logs, even for modifications?

 

I am running Unraid 6.11.5 and have attached a screenshot of my settings and a snippet of the verification log. Thanks and any help would be greatly appreciated.

 

Event: unRAID file corruption
Subject: Notice [ALEXANDRIA] - bunker verify command
Description: Found 2 files with MD5 hash key corruption
Importance: alert

MD5 hash key mismatch, /mnt/disk2/backups/Unraid/CA_Backup/Appdata_Backup/[email protected]/CA_backup.tar is corrupted
MD5 hash key mismatch, /mnt/disk2/backups/Unraid/CA_Backup/Appdata_Backup/[email protected]/backup.log is corrupted
MD5 hash key mismatch (updated), /mnt/disk2/media/LINUXISO/mint.nfo was modified

 

Ran verification last night, same results where I continue to get emails of updated .nfo files from the media and Nextcloud shares. Likewise, exclusion rules do not seem to be respected.

 

Will there be upcoming updates to the app to not have updated hashes email and only include those that are corrupted?

[Vr2Io]

13 hours ago, Waddoo said:

However I am finding that `*.nfo, *.log` files continue to appear in the logs

Exclude means FIP will exclude generate the hash, any file if have corresponding attributes then it will verify, you need use the "Clear" function to clear the attributes.

 

 

Edited August 7, 2023 by Vr2Io

[Goldmaster]

Will there be an update or feature added that would allow for exporting all current checksums to 1 single checksum?

 

It would be very useful if the addon could pickup duplicate files if multiple files have the exact same checksums, that way it would free up storage.

[Waddoo]

1 minute ago, Goldmaster said:

Will there be an update or feature added that would allow for exporting all current checksums to 1 single checksum?

Highly doubt it. Checksum relies on a single file to have a checksum against. I think the only way of generating a single checksum of your files is to zip/tar/compress them and checksum that each time.

[sunwind]

How about using this with btfrs? or should I just use the scrub option on the disks? (anyone know if that is per disk? i'm used to zfs scrubs so seems weird a pool would scrub 1 disk at a time.)

[JorgeB]

37 minutes ago, sunwind said:

How about using this with btfrs?

Seems unnecessary to me.

 

38 minutes ago, sunwind said:

(anyone know if that is per disk? i'm used to zfs scrubs so seems weird a pool would scrub 1 disk at a time.)

If it's array devices it's by disk, same as if you use zfs, each disk is an independent filesystem.

[wacko37]

Hi all, with all the talk of ZFS it has lead me down the path of using this plugin for my Unraid array (zfs is not suitable for my home use case on main array)

 

Have a few question which I have searched for answers over the past few days but to no avail, time to ask those that do know.

 

I believe I fully understand how this plugin works on the Unraid side, by applying a checksum to the extend attributes of each file. Verification involves crossing checking the actual file against this checksum on a defined schedule. Any discrepancies found will be notified. Perfect, now puts my mind to rest if something gets corrupted.

 

So....  if something gets corrupted I would need to manually replace the file with a backup, this is where I get stuck on how to proceed. 

 

What I have been doing up until now is "rsync" every once in a while to an offline backup which is basically a large drive formatted in NTFS. This could then be used to replace the corrupted file. But how do I ensure that the backup is not corrupted, meaning how do I rsync to the external drive and verify the checksums to ensure that.

 

I utilise Unassigned Devices device script to run the rsync, I'm aware that rsync has "--checksum" command but now I would be double creating checksums. 

 

Is there a command that I can add the the Unassigned Devices device script that would cross check the checksum after sync. I have a feeling NTFS is going to be an issue here.

[tazman]

Dear all,

 

I have .nfo files excluded, as they change frequently. So the Export correctly states that files are skipped:

 

Finished - exported 216628 files, skipped 122 files. Duration: 00:01:59

 

Yet, upon export, the syslog is flooded with error messages like:

 

Aug 30 17:08:47 SS bunker: error: no export of file: /mnt/disk8/Movies/Brooklyn (2015)/movie.nfo

 

Yes, Mr. Fileintegrity, .nfos should not be exported as configured. Why do you complain about this?

 

It was so bad that the syslog partition (standard size: 128mb) overflowed and nothing was logged anymore. Yet, unraid continued to run like a charm (!!!!).

 

The error message is only generated for excluded files (ie. *.nfo) , not folders. 

 

Bug? Please fix! Feature? Please make it configurable! Thanks!

 

The bigger issue at hand seems to be that files handled by Mover are not recognized as modified. Like .nfo files when they are touched by Sonar when a new episode is added.

 

So excluding them is my attempt to prevent me getting flooded by bunker verify warnings. Now, my syslog gets flooded. Can't avoid the flood. Maybe it's global warming!

 

Ultimately, this makes the plugin less useful as you constantly need to remember to run manual Builds and Exports. Could this be schedulable?

 

Tazman

Edited August 30, 2023 by tazman

[wacko37]

On 8/28/2023 at 12:08 PM, wacko37 said:

Hi all, with all the talk of ZFS it has lead me down the path of using this plugin for my Unraid array (zfs is not suitable for my home use case on main array)

 

Have a few question which I have searched for answers over the past few days but to no avail, time to ask those that do know.

 

I believe I fully understand how this plugin works on the Unraid side, by applying a checksum to the extend attributes of each file. Verification involves crossing checking the actual file against this checksum on a defined schedule. Any discrepancies found will be notified. Perfect, now puts my mind to rest if something gets corrupted.

 

So....  if something gets corrupted I would need to manually replace the file with a backup, this is where I get stuck on how to proceed. 

 

What I have been doing up until now is "rsync" every once in a while to an offline backup which is basically a large drive formatted in NTFS. This could then be used to replace the corrupted file. But how do I ensure that the backup is not corrupted, meaning how do I rsync to the external drive and verify the checksums to ensure that.

 

I utilise Unassigned Devices device script to run the rsync, I'm aware that rsync has "--checksum" command but now I would be double creating checksums. 

 

Is there a command that I can add the the Unassigned Devices device script that would cross check the checksum after sync. I have a feeling NTFS is going to be an issue here.

 

Ok - I've had some good success implementing this into an existing script and YES NTFS retains the extended attributes

 

 

The last piece of this puzzle is getting a log file with content/info. This is my line of code for calling bunker:

/usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -v -b3 -l -n "$dst_path" &>"$MOUNTPOINT"/bunker-logs/log.`date '+%Y_%m_%d__%H_%M_%S'`.txt >> $LOGFILE

 

Now it does everything correctly, it verifies the required destination and creates a log file. BUT there is never any content / info within the log file.

 

I've exhausted all avenues to try to get it to write the results but to no avail. I'm using Unassigned Devices to run the script and all the required information can be viewed in the UD "view device script log"

 

Any help is greatly appreciated 

[wacko37]

1 hour ago, wacko37 said:

 

Ok - I've had some good success implementing this into an existing script and YES NTFS retains the extended attributes

 

 

The last piece of this puzzle is getting a log file with content/info. This is my line of code for calling bunker:

/usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -v -b3 -l -n "$dst_path" &>"$MOUNTPOINT"/bunker-logs/log.`date '+%Y_%m_%d__%H_%M_%S'`.txt >> $LOGFILE

 

Now it does everything correctly, it verifies the required destination and creates a log file. BUT there is never any content / info within the log file.

 

I've exhausted all avenues to try to get it to write the results but to no avail. I'm using Unassigned Devices to run the script and all the required information can be viewed in the UD "view device script log"

 

Any help is greatly appreciated 

Its ok I got it - was trying to write the log in 2 places at the same time. All sorted for now...... I think.

 

[Vr2Io]

3 minutes ago, wacko37 said:

Its ok I got it - was trying to write the log in 2 places at the same time. All sorted for now...... I think.

 

Awesome script, will try later. 👍

Edited September 1, 2023 by Vr2Io

[wacko37]

@Vr2Io

 

Attached is the script, it rsync's to external drive with --xattrs enabled, it then calls bunker to verify the files within sync folder.

 

I have successfully tested this on an NTFS drive.

 

 

0_-_TestUSBNTFS_-_SYNC--VERIFY.sh

Edited September 1, 2023 by wacko37

[wacko37]

Just wondering when DFI builds its checksums for new files.

 

Meaning are checksums generated when new files have been created in the cache pool or after Mover has transfered the new files to the main array?

 

Thanks

[T0rqueWr3nch]

On 9/12/2023 at 2:19 AM, wacko37 said:

Just wondering when DFI builds its checksums for new files.

 

Meaning are checksums generated when new files have been created in the cache pool or after Mover has transfered the new files to the main array?

 

Thanks

 

Hello!

 

I'm glad you asked this question, as I recently contributed to implementing this functionality in Dynamix File Integrity (DFI). With the recent plugin update at the beginning of this month, DFI now supports auto-hashing for cache-to-disk moves. This means that the checksums are generated after the Mover has transferred the files to the main array.

 

Before this update, automatic hashing was limited; it only occurred on main array disk writes. This would include new files being written directly to the main disk array or modifications being made to files already present on the array. Therefore, checksums weren’t automatically generated for files moved from the cache pool, necessitating users to manually trigger a build.

 

With this update, users should hopefully notice that they no longer need to manually trigger builds when automatic checksums are enabled.

 

If you have any other questions, feel free to ask!