Dynamix File Integrity plugin

[Shantarius]

Hello,

 

what is with this spinning symbol meant?

 

 

[feins]

Dear All,

 

Why Disk 2 i got a pink box?

Also i notice that i got alot of hash key mismatch.

 

 

ts-p500-diagnostics-20231004-1823.zip

[Mainfrezzer]

2 hours ago, feins said:

Dear All,

 

Why Disk 2 i got a pink box?

Also i notice that i got alot of hash key mismatch.

 

 

ts-p500-diagnostics-20231004-1823.zip 171.62 kB · 1 download

the mismatch lets you know that the file was modified, got a new checksum but you have no current export of that new checksum(thats expected with .nfo files, there is no point in checksumming them).

the pink box is intriguing, no clue tho

[T0rqueWr3nch]

I just did a very quick pass through the code. The CSS seems fairly limited, so my guess (and that's really all it is) is that this pink checkbox is an artifact of the overall Unraid theme you have applied.

 

If you want to feel better about it- in your mismatch logs, are you seeing hits on files that are on that disk? If so, then your integrity checks are working and I wouldn't concern myself about it (unless @bonienl says otherwise).

[wacko37]

On 9/30/2023 at 11:26 AM, T0rqueWr3nch said:

 

Hello!

 

I'm glad you asked this question, as I recently contributed to implementing this functionality in Dynamix File Integrity (DFI). With the recent plugin update at the beginning of this month, DFI now supports auto-hashing for cache-to-disk moves. This means that the checksums are generated after the Mover has transferred the files to the main array.

 

Before this update, automatic hashing was limited; it only occurred on main array disk writes. This would include new files being written directly to the main disk array or modifications being made to files already present on the array. Therefore, checksums weren’t automatically generated for files moved from the cache pool, necessitating users to manually trigger a build.

 

With this update, users should hopefully notice that they no longer need to manually trigger builds when automatic checksums are enabled.

 

If you have any other questions, feel free to ask!

Sorry for the late reply, thankyou for the detailed response and your efforts in making the Mover auto-hashing possible.... this community is so blessed by the hard work of others.

 

My main reason for asking when auto-hashing is generated is (from my limited understanding) if there where to be a risk of corruption it would be at its highest when doing that transfer. Meaning any transfer that would entail the exchange of a file between to locations/disks and most likely filesystems in my case ZFS to XFS. (or does ECC ram remove that risk altogether?)

 

As all files being written the the main array would be via the cache (setup depending of course) It would be nice/peace of mind if the auto-hashing was to be done when the files are 1st written to the cache via the /mnt/user0/ folder structure. "user0" from my understanding is the temp folder on the cache prior to the Mover transfer to main array /mnt/user/, that way when DFI runs a verification is would be against checksums made prior to the mover transfer.

 

Or maybe this is not at all possible due to my lack of knowledge. Either way I now know how it works atm and very grateful none the less

Edited October 6, 2023 by wacko37

[Shantarius]

Hello,

 

in my syslog I have >15000 Entries like this:

 

Oct 17 05:52:51 Avalon bunker: error: BLAKE3 hash key mismatch, /mnt/disk1/Backup_zpool/zpool/nextcloud_data/20231007_031906/nextcloud_data/christian/files/Obsidian_Vault/.git/objects/ba/edc24a7db2d2f8cc372d12a6cfaa5a7b3b9206 is corrupted

 

At 10th Oktober the same for disk3. What is happened here and how can I solve this?

 

Thanks!

 

[psychic99]

I have the DFI plugin working, and I also use luck backup to archive data off the array (which now has hash) which copies to a drive using the unassigned devices plugin.  When the data gets copied over (using fuse) it archives the entire directories (cache and array) so this means newer files do not have hashes against them.   What I want to do however is when they are archived to this mount point I would like to compare the hashes created w/ DFI plugin versus what has been archived for obvious reasons (if i need to restore I want a good copy!).  

 

1.  What method would recommend to do this (I am sure someone has done this)

2.  Is there a reason why unassigned plugin mount points are excluded from the DFI (only disks in the array).

 

I don't think what I am trying to accomplish is unique -- I suppose I could nix luck and go to something like rsync w/ checksum.  Let me know what your thoughts are.  I am simply trying to ensure what is archived is archived with 100% fidelity.   BTW I also send most of this data to the cloud but the local disk is the first restore candidate.

Thanks.

[johnsanc]

I am cleaning up my array and switched to BLAKE3. I want to ensure ALL files have a BLAKE3 hash and ideally wipe out all the older hashes since I have no use for them. What is the best way to do this?

 

I noticed that the Remove action doesn't actually remove old hashes, it only seems to remove the timestamps + whatever hash is currently selected. Is this a bug or by design?

 

Based on my current observations it seems the only way to start from a clean slate is to do a Clear and Remove on every disk for every hashing method. 

[isvein]

I got this one yesterday, I guess this is a kind of file that should be excluded as it changes often?
Is there a list of filetypes that should be excluded?
Tried to search but no luck

[Interstellar]

Still looking for a way to manually start a verification process rather than temporarily setting a verification schedule and then once its started turning it off again....

[wacko37]

Hi All, I searched high and low for an answer but to no avail, basacially I would like a more custom cron schedule for DFI, what I want to do is 0 6 */4 * * which is on the 6th hour of every 4 days of a month run a check on a single disk.

So there is no setting to do this via the Webui, so what I need to know is it safe to manually edit  "/UNRAID Flash Dir/config/plugins/dynamix.file.integrity/integrity-check.cron" to reflect the above cron scheduale.

 

Thanks

[wacko37]

On 12/6/2023 at 8:27 AM, wacko37 said:

Hi All, I searched high and low for an answer but to no avail, basacially I would like a more custom cron schedule for DFI, what I want to do is 0 6 */4 * * which is on the 6th hour of every 4 days of a month run a check on a single disk.

So there is no setting to do this via the Webui, so what I need to know is it safe to manually edit  "/UNRAID Flash Dir/config/plugins/dynamix.file.integrity/integrity-check.cron" to reflect the above cron scheduale.

 

Thanks

Ok - I manually edited both files in the below locations to 0 6 */4 * * but to no avail.

 

"config/plugins/dynamix.file.integrity/integrity-check.cron"   &   "config/plugins/dynamix.file.integrity/integrity-check.cron"

 

Is what I'm trying to achieve possible

[Oublieux]

Hi everyone, I set up DFI for the first time today. It seems straightforward enough, but I seem to be running into an issue (?) that is puzzling me:

• I hit "Build" to start generating BLAKE3 hash files for my all four of my disks at once.
• Two of the disks are completed and the remaining two are still in the build phase.
• I assumed that the hash files for the two disks that completed would be present, but clicking the "hash files" icon that opens the directory shows that there are no files at all.
   

Has anyone run into this before? Or, should I be waiting for the build to complete across all four disks first?

Edited December 19, 2023 by Oublieux

[foo_fighter]

6 hours ago, Oublieux said:

Hi everyone, I set up DFI for the first time today. It seems straightforward enough, but I seem to be running into an issue (?) that is puzzling me:

• I hit "Build" to start generating BLAKE3 hash files for my all four of my disks at once.
• Two of the disks are completed and the remaining two are still in the build phase.
• I assumed that the hash files for the two disks that completed would be present, but clicking the "hash files" icon that opens the directory shows that there are no files at all.
   

Has anyone run into this before? Or, should I be waiting for the build to complete across all four disks first?

 

hashes are stored as Xattrs in the files. try getfattr -d filename. You can also export them to files on the boot usb.

 

[foo_fighter]

On 10/31/2023 at 9:36 AM, johnsanc said:

I am cleaning up my array and switched to BLAKE3. I want to ensure ALL files have a BLAKE3 hash and ideally wipe out all the older hashes since I have no use for them. What is the best way to do this?

 

I noticed that the Remove action doesn't actually remove old hashes, it only seems to remove the timestamps + whatever hash is currently selected. Is this a bug or by design?

 

Based on my current observations it seems the only way to start from a clean slate is to do a Clear and Remove on every disk for every hashing method. 

Use something like:  find .  -exec setfattr --remove=user.md5 {} \;

Edited December 20, 2023 by foo_fighter

[feins]

On 10/4/2023 at 8:44 PM, Mainfrezzer said:

the mismatch lets you know that the file was modified, got a new checksum but you have no current export of that new checksum(thats expected with .nfo files, there is no point in checksumming them).

the pink box is intriguing, no clue tho

Recently the file integrity just finish checksum and this time no Pink Check box. but I'm still getting prompt that

unRAID file corruption: 19-12-2023 10:53

Notice [TS] - bunker verify command
Found 154 files with BLAKE3 hash key mismatch

How do i find what type of files are those?

 

[Oublieux]

3 hours ago, foo_fighter said:

 

hashes are stored as Xattrs in the files. try getfattr -d filename. You can also export them to files on the boot usb.

 

Thank you, the getfattr -d -n command worked and output a blake3 value. The "Export" button doesn't seem to do anything at the moment... I do wonder if it's because the BLAKE3 hashes are still being generated on the other two drives.

Edit: Had a "duh" moment and realized I didn't select any disks before running "Export". Can confirm that exports work after selecting a disk, and I am just being a bit of an old fogey. Appreciate the help!

Edited December 20, 2023 by Oublieux

[foo_fighter]

@bonienl May I make an enhancement request?

Some of us like to run bunker from the command line and output STDOUT to a log file.

I'd like a mode similar to -q (surpress all output) but to remove the "Scanning... spinning icon" output so it only  outputs  relevant information and also remove escape characters so that the output can go into a plain readable text file for a .log.

 

 

[foo_fighter]

Bunker does not like spaces in dirnames when the dir is double quoted "/dir name/":

\ /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker: line 91: $monitor: ambiguous redirect

/usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker: line 133: $monitor: ambiguous redirect

 

but seems okay with it backslashed as /dir\ name/.

 

 

Edited December 24, 2023 by foo_fighter

[feins]

Each time when the file integrity runs i keep getting this type of errors.

How to resolve this? Ive already excluded *.nfo

[Vr2Io]

On 12/26/2023 at 9:38 PM, feins said:

Each time when the file integrity runs i keep getting this type of errors.

How to resolve this? Ive already excluded *.nfo

You also need clear attribute on those .nfo file. and re-export the hash.

Edited December 29, 2023 by Vr2Io

[feins]

22 hours ago, Vr2Io said:

You also need clear attribute on those .nfo file. and re-export the hash.

Could you be able to guide me how to do that?

[Vr2Io]

19 hours ago, feins said:

Could you be able to guide me how to do that?

Use "Clear" button.

 

 

[Oublieux]

This might sound silly, but how does this plugin handle deleted files? I had this plugin build a database of hashes, but deleted some files afterwards. When running a verification, the plugins note that there are problems because files are now missing. Is this expected behavior?

[Vr2Io]

14 hours ago, Oublieux said:

This might sound silly, but how does this plugin handle deleted files? I had this plugin build a database of hashes, but deleted some files afterwards. When running a verification, the plugins note that there are problems because files are now missing. Is this expected behavior?

You need perform export again.