Dynamix File Integrity plugin


bonienl

Recommended Posts

Hi,

 

Could somebody answer this topic, it was asked twice on page 41, I think the query got lost in the rest of the discussion.

I have exactly the same issue, leaving me nervous.

I've started a rebuild of my disk 4, as I cannot export. I have a feeling that's not the right thing to do though??

 

Can somebody please elaborate on Sven's below issue, which is identical to mine?

 

 

On 10/10/2021 at 8:36 AM, sven said:

I am a new user of the File Integrity Plugin. I have used the BLAKE3 algorithm and so far the performance seems great.

 

One (small?) problem: at the end of running the Export step for one of the disks, I get a PHP error that a memory limit has been reached:

 

Quote

<br /> <b>Fatal error</b>: Allowed memory size of 134217728 bytes exhausted (tried to allocate 168201344 bytes) in <b>/usr/local/emhttp/plugins/dynamix.file.integrity/include/ProgressInfo.php</b> on line <b>40</b><br />

 

I have gathered that this has to do with the memory limit of php which is set at 128MB. When I go to the export directory, I see that a 168MB file has been created called disk1.export.blake3.hash. When I open the file it contains the hashes for all files from disk 1.

 

Since the export file seems to be fine, is this just an error in the progress display?

 

There were 862892 files in the export. 6TB drive, many large (500MB+) files, but also a lot of smaller files.

 

UnraidFileIntegrityPHPerror.thumb.JPG.a5e2e69ab5ac05fc645191f872899114.JPG

 

Edited by Geck0
So, after rebuilding, I've tried to export again. I get the same out of memory error. The system log says: Sep 25 19:56:30 Nexus bunker: added 756 files from /mnt/disk4. Duration: 00:36:04. Average speed: 280 MB/s Sep 25 20:05:05 Nexus bunker: error: no
Link to comment
28 minutes ago, Geck0 said:

Hi,

 

Could somebody answer this topic, it was asked twice on page 41, I think the query got lost in the rest of the discussion.

I have exactly the same issue, leaving me nervous.

I've started a rebuild of my disk 4, as I cannot export. I have a feeling that's not the right thing to do though??

 

Can somebody please elaborate on Sven's below issue, which is identical to mine?

 

 

 

Try below solution at page 43.

 

Edited by Vr2Io
Link to comment
4 hours ago, Vr2Io said:

 

Sample check files by "getfattr -d <file>"

Thank you.

I did check one of the files:

getfattr -d noiseman_sound_insect.avi

And it looks like the blake3 checksum is still there (see screenshot below).

But I did follow the instructions:
"for those who want to uninstall this plugin,
 before uninstalling the plugin itself you need to remove the calculated hashes,
 this is done by the "Clear" operation (select all disks and then Clear, see also the built-in Help)."
And after I did this I removed the plugin and deleted the folder /config/plugins/dynamix.file.integrity

 

FI-ATTRIB.jpg

Edited by DataCollector
Link to comment
14 minutes ago, DataCollector said:

Thank you.

I did check one of the files:

getfattr -d noiseman_sound_insect.avi

And it looks like the blake3 checksum is still there (see screenshot below).

But I did follow the instructions:
"for those who want to uninstall this plugin,
 before uninstalling the plugin itself you need to remove the calculated hashes,
 this is done by the "Clear" operation (select all disks and then Clear, see also the built-in Help)."
And after I did this I removed the plugin and deleted the folder /config/plugins/dynamix.file.integrity

 

 

Strange, clear operation haven't clear extend attribute. If you want clear it, pls try install FIP again ( set blake3 ) for that or left those attribute there should be no hurt.

Link to comment
42 minutes ago, Vr2Io said:

 

Strange, clear operation haven't clear extend attribute. If you want clear it, pls try install FIP again ( set blake3 ) for that or left those attribute there should be no hurt.

Yes. Strange.

 

Now I installed FI again,
"Automatically protect new and modified files": Enabled
Method: "BLAKE3"
<Apply> (see screenshot 1)
==
Then I spun up the first 5 Disks of my '1st System' (look at signature) manually and started in FI "Clear" for those Disks #1-5.
(see screenshot 2)
==
FI toook around 1 Second and then showed in green "cleared" (see screenshot 3)
==
And I then looked again at the attributes and Blake3 is there.
(see screenshot 4)

 

I am very sad.

 

 

FI-Again-00-2022-09-25 13_50_11-102 Tessa Main.png

FI-AGAIN-01-CLEAR-PRESET-2022-09-25 13_48_46-102 Tessa Main.png

FI-Again-02-0Seconds-2022-09-25 13_49_28-102 Tessa Main.png

FI-Again-Attrib-2022-09-25 13_56_33-102 Tessa Main.png

Link to comment

Pls try

 

/bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -r -b3 /mnt/diskX

 

or

 

/bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -R -b3 /mnt/diskX

 

, where X is disk no.

 

 

/bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -h , would got bunker help

 

Edit : Overlook you should use remove instead clear

 

image.png.ab58fcae0e4166d311a83d5e22dbdb0e.png

Edited by Vr2Io
  • Thanks 1
Link to comment

Unfortunately this plugin isn't working correctly for me. Today it gave me a notification saying that 82 files on my array were corrupt. When I checked the files deemed corrupt by the plugin the hashes matched up to the values in the extended file attributes, so I have no idea where the plugin got the idea that my files were corrupt. It honestly shows how unreliable the plugin is when these "corruption" notifications don't even faze me in the slightest.

Link to comment
  • 2 weeks later...
On 9/25/2022 at 2:03 PM, DataCollector said:

Yes. Strange.

 

Now I installed FI again,
"Automatically protect new and modified files": Enabled
Method: "BLAKE3"
<Apply> (see screenshot 1)
==
Then I spun up the first 5 Disks of my '1st System' (look at signature) manually and started in FI "Clear" for those Disks #1-5.
(see screenshot 2)
==
FI toook around 1 Second and then showed in green "cleared" (see screenshot 3)
==
And I then looked again at the attributes and Blake3 is there.
(see screenshot 4)

 

I am very sad.

 

 

FI-Again-00-2022-09-25 13_50_11-102 Tessa Main.png

FI-AGAIN-01-CLEAR-PRESET-2022-09-25 13_48_46-102 Tessa Main.png

FI-Again-02-0Seconds-2022-09-25 13_49_28-102 Tessa Main.png

FI-Again-Attrib-2022-09-25 13_56_33-102 Tessa Main.png

 

 

EDIT: it seems "Clear" is the wrong command.   Today I tried "Remove" and that command did work on the disks for some time.

 

Link to comment
  • 3 weeks later...
  • 2 weeks later...

Hi,

 

I have this plugin installed with Automatically protect new and modified files: enabled. but when I check new files with "getfattr -d <file>" the hash information is missing until I go and run the build again manually. Am I doing something wrong or can I check if it's ruining somehow?

  • Upvote 1
Link to comment

I've been trying to track down why so many NEW files do not have a hash. I get emails every single time due to either hash mismatch (usually on nextcloud.log despite excluding *.log files) or hash missing altogether. I dug into the source code of the plugin and replicated the inotifywait command so that i could watch for myself what was going on.

 

inotifywait -mr -e close_write --format '%w%f' /mnt/disk1 /mnt/disk2

and after running mover here is what i see

/mnt/disk2/documents/ubuntu-18.04.1-server-amd64.iso.partial
/mnt/disk1/backup/unraid/STORAGE/flash/config/editor.cfg.partial
/mnt/disk1/backup/unraid/STORAGE/flash/config/super.dat.CA_BACKUP.partial
/mnt/disk1/backup/SQL/mariadb/vmosa/2022-11-12-08.00.01.sql.tgz.partial
/mnt/disk1/backup/SQL/postgres11/airsonic/airsonic_2022-11-12-08.05.02.sql.tgz.partial
/mnt/disk1/backup/SQL/postgres11/authelia/authelia_2022-11-12-08.05.03.sql.tgz.partial
/mnt/disk1/backup/SQL/postgres11/family_photos_dev/family_photos_dev_2022-11-12-08.05.03.sql.tgz.partial
/mnt/disk1/backup/SQL/postgres11/family_photos_prod/family_photos_prod_2022-11-12-08.05.04.sql.tgz.partial
/mnt/disk1/backup/SQL/postgres11/film_convert_dev/film_convert_dev_2022-11-12-08.05.04.sql.tgz.partial
/mnt/disk1/backup/SQL/postgres11/film_convert_prod/film_convert_prod_2022-11-12-08.05.05.sql.tgz.partial
/mnt/disk1/backup/unraid/STORAGE/flash/config/plugins/dynamix.file.manager/dynamix.file.manager.txz.partial
/mnt/disk1/backup/unraid/STORAGE/flash/config/plugins/dynamix.file.manager.plg.partial
/mnt/disk1/backup/unraid/STORAGE/flash/config/plugins/unassigned.devices.plg.partial
/mnt/disk1/backup/unraid/STORAGE/flash/config/plugins/dynamix.file.integrity/disks.ini.partial

As you can see, every file moved by mover ends with .partial and if i run getfattr on the actual file (since there is no .partial file)

root@storage:~# getfattr -d /mnt/disk1/backup/SQL/postgres11/airsonic/airsonic_2022-11-12-08.05.02.sql.tgz
root@storage:~# 

When I navigate into the airsonic backup folder and run getfattr on any file in there (because backup always saves to cache and then mover moves) getfattr -d airsonic* is blank for all files.

  • Like 2
Link to comment
  • 2 months later...

Has anyone else ever gotten this error?

 

clean install, getting this error when doing an initial build:

 

 Finished - added 0 files. Duration: 00:00:01

 

I had the plugin installed before and it successfully did an initial build and export.  But after upgrading the plugin I began getting this error.  I removed the plugin, deleted the plugin folder under \boot\config\plugins and rebooted, but still getting the error.  Logs are empty.

 

 

Link to comment

Just want to jump in here too, I get the same thing. I manually this on all my disks to remove previous hashes:

/bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -r -b2 /mnt/diskX
/bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -r -b3 /mnt/diskX 

Removed all old preferences and started fresh but every time I hit build its completes immediately with the same log as the previous post.

 

For now the plugin stays disabled. I'm not quite sure what's going on.

 

EDIT:

it also looks like "bunker -R -b3 /mnt/diskX" is broken? it only clears the hash but not the other "user." attrs looking in to the script. Wonder if the new unraid version is causing issues?

Edited by JimmyGerms
Link to comment

Plugin just call bunker to perform different task, pls check does file permission issue, try use tools - NEW PERMISSIONS to fix that and re-run bunker.

 

Daily add & verify file ( bunker CLI ) without issue.

 

image.thumb.png.1f4791c5ae49a05a2b8a93ad9d296b55.png

 

 

3 hours ago, JimmyGerms said:

it only clears the hash but not the other "user."

This normal.

Edited by Vr2Io
Link to comment

I've run "New Permissions" not too long ago and everything matches what the help docs have for those Shares.

 

Looking at this code, I'm reading it that if the "user.scandate" attr is empty then using -R should remove the three extra attrs but it doesn't seem to...Not sure where my setup is going wrong.

      if [[ -z $userdate || $((userdate/86400)) -le $epoch ]]; then
        ((count++))
        setfattr -x user.$hash "$file" 2>/dev/null
        [[ $cmd == R ]] && setfattr -x user.scandate "$file" 2>/dev/null
        [[ $cmd == R ]] && setfattr -x user.filedate "$file" 2>/dev/null
        [[ $cmd == R ]] && setfattr -x user.filesize "$file" 2>/dev/null

 

EDIT: Yeah, not seeing anything wrong in the code. Even if the user.scandate was 0 it still should respect the flags...What's going on with my system then. Cannot get this to kick off at all anymore.

Edited by JimmyGerms
  • Like 1
Link to comment
11 hours ago, JimmyGerms said:

Nice find, will try that as I am also getting the "0 files added" and no log files created....

Edited by Kretsche
Edit: Does not work for me as its not compatible with Unraid 6.11.5
Link to comment
On 2/17/2023 at 3:21 AM, Kretsche said:

Nice find, will try that as I am also getting the "0 files added" and no log files created....

I'm on 6.11.5 too. The trick for me was I downloaded the corresponding txz with that version of the plugin and placed it on the flash drive. Next I manually installed the plugin by using that link and it worked. Here's the link to the .txz for ya!

https://github.com/bergware/dynamix/raw/fbd863f64ce1af8ea97ac2d13cc906a410925b9f/archive/dynamix.file.integrity.txz

 

EDIT: Honestly, it's still not functioning right. New files don't get hashed even though I have the setting "save new files hashes to flash" enabled. Think overall I'm just going to remove this and write a little script to remove left over hashes that are stuck on files.

Edited by JimmyGerms
Link to comment

Same issue as above, no files being added. 6.11.5

Unlike Jimmy, I was unable to revert to a previous version;

 

plugin: installing: dynamix.file.integrity.plg
Executing hook script: pre_plugin_checks
plugin: downloading: dynamix.file.integrity.plg ... done

Executing hook script: pre_plugin_checks
plugin: skipping: inotify-tools-3.14-x86_64-1.txz - Unraid version too high, requires at most version 6.2.9
plugin: skipping: b2sum-20130305-x86_64-1.txz - Unraid version too high, requires at most version 6.3.9

+==============================================================================
| Skipping package b3sum-1.0.0-x86_64-1 (already installed)
+==============================================================================
plugin: downloading: dynamix.file.integrity.txz ... done

plugin: bad file MD5: /boot/config/plugins/dynamix.file.integrity/dynamix.file.integrity.txz
Executing hook script: post_plugin_checks

 

Link to comment

Hello, Same issue for me for the newer version of this plugin on UnRaid 6.11.5

 

no files been added, also tried with every hashing methods

 

No Files Added (it run when we do Build and after few minutes, it said 0 file added

No Export files (file at 0 so we can click on Export, it run but nothing been exported)...

 

it seem to just scan all files without scanning the hash

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.