19 hours ago, DataCollector said:

I wanted now to clear all those to uninstall the FI plugin.
But it looks like it did not clear anything.

I dont understand this plugin at all.

 

Sample check files by "getfattr -d <file>"

Hi,

 

Could somebody answer this topic, it was asked twice on page 41, I think the query got lost in the rest of the discussion.

I have exactly the same issue, leaving me nervous.

I've started a rebuild of my disk 4, as I cannot export. I have a feeling that's not the right thing to do though??

 

Can somebody please elaborate on Sven's below issue, which is identical to mine?

 

 

On 10/10/2021 at 8:36 AM, sven said:

I am a new user of the File Integrity Plugin. I have used the BLAKE3 algorithm and so far the performance seems great.

 

One (small?) problem: at the end of running the Export step for one of the disks, I get a PHP error that a memory limit has been reached:

 

Quote

<br /> <b>Fatal error</b>: Allowed memory size of 134217728 bytes exhausted (tried to allocate 168201344 bytes) in <b>/usr/local/emhttp/plugins/dynamix.file.integrity/include/ProgressInfo.php</b> on line <b>40</b><br />

 

I have gathered that this has to do with the memory limit of php which is set at 128MB. When I go to the export directory, I see that a 168MB file has been created called disk1.export.blake3.hash. When I open the file it contains the hashes for all files from disk 1.

 

Since the export file seems to be fine, is this just an error in the progress display?

 

There were 862892 files in the export. 6TB drive, many large (500MB+) files, but also a lot of smaller files.

 

 

Edited September 25, 20223 yr by Geck0
So, after rebuilding, I've tried to export again. I get the same out of memory error. The system log says: Sep 25 19:56:30 Nexus bunker: added 756 files from /mnt/disk4. Duration: 00:36:04. Average speed: 280 MB/s Sep 25 20:05:05 Nexus bunker: error: no

28 minutes ago, Geck0 said:

Hi,

 

Could somebody answer this topic, it was asked twice on page 41, I think the query got lost in the rest of the discussion.

I have exactly the same issue, leaving me nervous.

I've started a rebuild of my disk 4, as I cannot export. I have a feeling that's not the right thing to do though??

 

Can somebody please elaborate on Sven's below issue, which is identical to mine?

 

 

 

Try below solution at page 43.

 

Edited September 25, 20223 yr by Vr2Io

@Vr2Io

This didn't work, however I'm guessing I need to server restart for this to take effect. I'm also in the middle of a parity check. I'll have to try again tomorrow after the parity ends.

Thanks for responding, it looks like there are a few people raising this issue.

4 hours ago, Vr2Io said:

 

Sample check files by "getfattr -d <file>"

Thank you.

I did check one of the files:

getfattr -d noiseman_sound_insect.avi

And it looks like the blake3 checksum is still there (see screenshot below).

But I did follow the instructions:
"for those who want to uninstall this plugin,
 before uninstalling the plugin itself you need to remove the calculated hashes,
 this is done by the "Clear" operation (select all disks and then Clear, see also the built-in Help)."
And after I did this I removed the plugin and deleted the folder /config/plugins/dynamix.file.integrity

 

Edited September 25, 20223 yr by DataCollector

14 minutes ago, DataCollector said:

Thank you.

I did check one of the files:

getfattr -d noiseman_sound_insect.avi

And it looks like the blake3 checksum is still there (see screenshot below).

But I did follow the instructions:
"for those who want to uninstall this plugin,
 before uninstalling the plugin itself you need to remove the calculated hashes,
 this is done by the "Clear" operation (select all disks and then Clear, see also the built-in Help)."
And after I did this I removed the plugin and deleted the folder /config/plugins/dynamix.file.integrity

 

 

Strange, clear operation haven't clear extend attribute. If you want clear it, pls try install FIP again ( set blake3 ) for that or left those attribute there should be no hurt.

42 minutes ago, Vr2Io said:

 

Strange, clear operation haven't clear extend attribute. If you want clear it, pls try install FIP again ( set blake3 ) for that or left those attribute there should be no hurt.

Yes. Strange.

 

Now I installed FI again,
"Automatically protect new and modified files": Enabled
Method: "BLAKE3"
<Apply> (see screenshot 1)
==
Then I spun up the first 5 Disks of my '1st System' (look at signature) manually and started in FI "Clear" for those Disks #1-5.
(see screenshot 2)
==
FI toook around 1 Second and then showed in green "cleared" (see screenshot 3)
==
And I then looked again at the attributes and Blake3 is there.
(see screenshot 4)

 

I am very sad.

 

 

Pls try

 

/bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -r -b3 /mnt/diskX

 

or

 

/bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -R -b3 /mnt/diskX

 

, where X is disk no.

 

 

/bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -h , would got bunker help

 

Edit : Overlook you should use remove instead clear

 

Edited September 25, 20223 yr by Vr2Io

Unfortunately this plugin isn't working correctly for me. Today it gave me a notification saying that 82 files on my array were corrupt. When I checked the files deemed corrupt by the plugin the hashes matched up to the values in the extended file attributes, so I have no idea where the plugin got the idea that my files were corrupt. It honestly shows how unreliable the plugin is when these "corruption" notifications don't even faze me in the slightest.

On 9/25/2022 at 2:03 PM, DataCollector said:

Yes. Strange.

 

Now I installed FI again,
"Automatically protect new and modified files": Enabled
Method: "BLAKE3"
<Apply> (see screenshot 1)
==
Then I spun up the first 5 Disks of my '1st System' (look at signature) manually and started in FI "Clear" for those Disks #1-5.
(see screenshot 2)
==
FI toook around 1 Second and then showed in green "cleared" (see screenshot 3)
==
And I then looked again at the attributes and Blake3 is there.
(see screenshot 4)

 

I am very sad.

 

 

 

 

EDIT: it seems "Clear" is the wrong command.   Today I tried "Remove" and that command did work on the disks for some time.

 

I wanted to added ".Recycle.Bin,backup/SQL" to the excluded folders and i accidentally hit my enter key. Doing so resets all settings to default.

Hi,

 

I have this plugin installed with Automatically protect new and modified files: enabled. but when I check new files with "getfattr -d <file>" the hash information is missing until I go and run the build again manually. Am I doing something wrong or can I check if it's ruining somehow?

I've been trying to track down why so many NEW files do not have a hash. I get emails every single time due to either hash mismatch (usually on nextcloud.log despite excluding *.log files) or hash missing altogether. I dug into the source code of the plugin and replicated the inotifywait command so that i could watch for myself what was going on.

 

inotifywait -mr -e close_write --format '%w%f' /mnt/disk1 /mnt/disk2

and after running mover here is what i see

/mnt/disk2/documents/ubuntu-18.04.1-server-amd64.iso.partial
/mnt/disk1/backup/unraid/STORAGE/flash/config/editor.cfg.partial
/mnt/disk1/backup/unraid/STORAGE/flash/config/super.dat.CA_BACKUP.partial
/mnt/disk1/backup/SQL/mariadb/vmosa/2022-11-12-08.00.01.sql.tgz.partial
/mnt/disk1/backup/SQL/postgres11/airsonic/airsonic_2022-11-12-08.05.02.sql.tgz.partial
/mnt/disk1/backup/SQL/postgres11/authelia/authelia_2022-11-12-08.05.03.sql.tgz.partial
/mnt/disk1/backup/SQL/postgres11/family_photos_dev/family_photos_dev_2022-11-12-08.05.03.sql.tgz.partial
/mnt/disk1/backup/SQL/postgres11/family_photos_prod/family_photos_prod_2022-11-12-08.05.04.sql.tgz.partial
/mnt/disk1/backup/SQL/postgres11/film_convert_dev/film_convert_dev_2022-11-12-08.05.04.sql.tgz.partial
/mnt/disk1/backup/SQL/postgres11/film_convert_prod/film_convert_prod_2022-11-12-08.05.05.sql.tgz.partial
/mnt/disk1/backup/unraid/STORAGE/flash/config/plugins/dynamix.file.manager/dynamix.file.manager.txz.partial
/mnt/disk1/backup/unraid/STORAGE/flash/config/plugins/dynamix.file.manager.plg.partial
/mnt/disk1/backup/unraid/STORAGE/flash/config/plugins/unassigned.devices.plg.partial
/mnt/disk1/backup/unraid/STORAGE/flash/config/plugins/dynamix.file.integrity/disks.ini.partial

As you can see, every file moved by mover ends with .partial and if i run getfattr on the actual file (since there is no .partial file)

root@storage:~# getfattr -d /mnt/disk1/backup/SQL/postgres11/airsonic/airsonic_2022-11-12-08.05.02.sql.tgz
root@storage:~# 

When I navigate into the airsonic backup folder and run getfattr on any file in there (because backup always saves to cache and then mover moves) getfattr -d airsonic* is blank for all files.

I restarted the plugin after months of pause and I face the same issue.

I tried to exclude .partial files in the hope that the plugin will skip it and only consider the real one once the transfer is done.

I'll report back.

It won't. The move script that mover uses is not standard, its custom written and seems to copy to .partial, deletes the original and then renames the partial to the real name. That renaming does not trigger inotifywait.

Has anyone else ever gotten this error?

 

clean install, getting this error when doing an initial build:

 

 Finished - added 0 files. Duration: 00:00:01

 

I had the plugin installed before and it successfully did an initial build and export.  But after upgrading the plugin I began getting this error.  I removed the plugin, deleted the plugin folder under \boot\config\plugins and rebooted, but still getting the error.  Logs are empty.

 

 

Just want to jump in here too, I get the same thing. I manually this on all my disks to remove previous hashes:

/bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -r -b2 /mnt/diskX
/bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -r -b3 /mnt/diskX 

Removed all old preferences and started fresh but every time I hit build its completes immediately with the same log as the previous post.

 

For now the plugin stays disabled. I'm not quite sure what's going on.

 

EDIT:

it also looks like "bunker -R -b3 /mnt/diskX" is broken? it only clears the hash but not the other "user." attrs looking in to the script. Wonder if the new unraid version is causing issues?

Edited February 16, 20233 yr by JimmyGerms

Plugin just call bunker to perform different task, pls check does file permission issue, try use tools - NEW PERMISSIONS to fix that and re-run bunker.

 

Daily add & verify file ( bunker CLI ) without issue.

 

 

 

3 hours ago, JimmyGerms said:

it only clears the hash but not the other "user."

This normal.

Edited February 16, 20233 yr by Vr2Io

I've run "New Permissions" not too long ago and everything matches what the help docs have for those Shares.

 

Looking at this code, I'm reading it that if the "user.scandate" attr is empty then using -R should remove the three extra attrs but it doesn't seem to...Not sure where my setup is going wrong.

      if [[ -z $userdate || $((userdate/86400)) -le $epoch ]]; then
        ((count++))
        setfattr -x user.$hash "$file" 2>/dev/null
        [[ $cmd == R ]] && setfattr -x user.scandate "$file" 2>/dev/null
        [[ $cmd == R ]] && setfattr -x user.filedate "$file" 2>/dev/null
        [[ $cmd == R ]] && setfattr -x user.filesize "$file" 2>/dev/null

 

EDIT: Yeah, not seeing anything wrong in the code. Even if the user.scandate was 0 it still should respect the flags...What's going on with my system then. Cannot get this to kick off at all anymore.

Edited February 16, 20233 yr by JimmyGerms

Ok, rolled back the plugin to this version and it's working again...

 

https://raw.githubusercontent.com/bergware/dynamix/fbd863f64ce1af8ea97ac2d13cc906a410925b9f/unRAIDv6/dynamix.file.integrity.plg

11 hours ago, JimmyGerms said:

Ok, rolled back the plugin to this version and it's working again...

 

https://raw.githubusercontent.com/bergware/dynamix/fbd863f64ce1af8ea97ac2d13cc906a410925b9f/unRAIDv6/dynamix.file.integrity.plg

Nice find, will try that as I am also getting the "0 files added" and no log files created....

Edited February 17, 20233 yr by Kretsche
Edit: Does not work for me as its not compatible with Unraid 6.11.5

On 2/17/2023 at 3:21 AM, Kretsche said:

Nice find, will try that as I am also getting the "0 files added" and no log files created....

I'm on 6.11.5 too. The trick for me was I downloaded the corresponding txz with that version of the plugin and placed it on the flash drive. Next I manually installed the plugin by using that link and it worked. Here's the link to the .txz for ya!

https://github.com/bergware/dynamix/raw/fbd863f64ce1af8ea97ac2d13cc906a410925b9f/archive/dynamix.file.integrity.txz

 

EDIT: Honestly, it's still not functioning right. New files don't get hashed even though I have the setting "save new files hashes to flash" enabled. Think overall I'm just going to remove this and write a little script to remove left over hashes that are stuck on files.

Edited February 19, 20233 yr by JimmyGerms

New 6.11.5 Unraid install with the plugin freshly installed and the build command is not doing anything for me, it keeps repeating that 0 files have been added even though the disk is not empty

Same issue as above, no files being added. 6.11.5

Unlike Jimmy, I was unable to revert to a previous version;

 

plugin: installing: dynamix.file.integrity.plg
Executing hook script: pre_plugin_checks
plugin: downloading: dynamix.file.integrity.plg ... done

Executing hook script: pre_plugin_checks
plugin: skipping: inotify-tools-3.14-x86_64-1.txz - Unraid version too high, requires at most version 6.2.9
plugin: skipping: b2sum-20130305-x86_64-1.txz - Unraid version too high, requires at most version 6.3.9

+==============================================================================
| Skipping package b3sum-1.0.0-x86_64-1 (already installed)
+==============================================================================
plugin: downloading: dynamix.file.integrity.txz ... done

plugin: bad file MD5: /boot/config/plugins/dynamix.file.integrity/dynamix.file.integrity.txz
Executing hook script: post_plugin_checks

 

Hello, Same issue for me for the newer version of this plugin on UnRaid 6.11.5

 

no files been added, also tried with every hashing methods

 

No Files Added (it run when we do Build and after few minutes, it said 0 file added

No Export files (file at 0 so we can click on Export, it run but nothing been exported)...

 

it seem to just scan all files without scanning the hash