Vr2Io Posted September 25, 2022 Share Posted September 25, 2022 19 hours ago, DataCollector said: I wanted now to clear all those to uninstall the FI plugin. But it looks like it did not clear anything. I dont understand this plugin at all. Sample check files by "getfattr -d <file>" Quote Link to comment
Geck0 Posted September 25, 2022 Share Posted September 25, 2022 (edited) Hi, Could somebody answer this topic, it was asked twice on page 41, I think the query got lost in the rest of the discussion. I have exactly the same issue, leaving me nervous. I've started a rebuild of my disk 4, as I cannot export. I have a feeling that's not the right thing to do though?? Can somebody please elaborate on Sven's below issue, which is identical to mine? On 10/10/2021 at 8:36 AM, sven said: I am a new user of the File Integrity Plugin. I have used the BLAKE3 algorithm and so far the performance seems great. One (small?) problem: at the end of running the Export step for one of the disks, I get a PHP error that a memory limit has been reached: Quote <br /> <b>Fatal error</b>: Allowed memory size of 134217728 bytes exhausted (tried to allocate 168201344 bytes) in <b>/usr/local/emhttp/plugins/dynamix.file.integrity/include/ProgressInfo.php</b> on line <b>40</b><br /> I have gathered that this has to do with the memory limit of php which is set at 128MB. When I go to the export directory, I see that a 168MB file has been created called disk1.export.blake3.hash. When I open the file it contains the hashes for all files from disk 1. Since the export file seems to be fine, is this just an error in the progress display? There were 862892 files in the export. 6TB drive, many large (500MB+) files, but also a lot of smaller files. Edited September 25, 2022 by Geck0 So, after rebuilding, I've tried to export again. I get the same out of memory error. The system log says: Sep 25 19:56:30 Nexus bunker: added 756 files from /mnt/disk4. Duration: 00:36:04. Average speed: 280 MB/s Sep 25 20:05:05 Nexus bunker: error: no Quote Link to comment
Vr2Io Posted September 25, 2022 Share Posted September 25, 2022 (edited) 28 minutes ago, Geck0 said: Hi, Could somebody answer this topic, it was asked twice on page 41, I think the query got lost in the rest of the discussion. I have exactly the same issue, leaving me nervous. I've started a rebuild of my disk 4, as I cannot export. I have a feeling that's not the right thing to do though?? Can somebody please elaborate on Sven's below issue, which is identical to mine? Try below solution at page 43. Edited September 25, 2022 by Vr2Io Quote Link to comment
Geck0 Posted September 25, 2022 Share Posted September 25, 2022 @Vr2Io This didn't work, however I'm guessing I need to server restart for this to take effect. I'm also in the middle of a parity check. I'll have to try again tomorrow after the parity ends. Thanks for responding, it looks like there are a few people raising this issue. Quote Link to comment
DataCollector Posted September 25, 2022 Share Posted September 25, 2022 (edited) 4 hours ago, Vr2Io said: Sample check files by "getfattr -d <file>" Thank you. I did check one of the files: getfattr -d noiseman_sound_insect.avi And it looks like the blake3 checksum is still there (see screenshot below). But I did follow the instructions: "for those who want to uninstall this plugin, before uninstalling the plugin itself you need to remove the calculated hashes, this is done by the "Clear" operation (select all disks and then Clear, see also the built-in Help)." And after I did this I removed the plugin and deleted the folder /config/plugins/dynamix.file.integrity Edited September 25, 2022 by DataCollector Quote Link to comment
Vr2Io Posted September 25, 2022 Share Posted September 25, 2022 14 minutes ago, DataCollector said: Thank you. I did check one of the files: getfattr -d noiseman_sound_insect.avi And it looks like the blake3 checksum is still there (see screenshot below). But I did follow the instructions: "for those who want to uninstall this plugin, before uninstalling the plugin itself you need to remove the calculated hashes, this is done by the "Clear" operation (select all disks and then Clear, see also the built-in Help)." And after I did this I removed the plugin and deleted the folder /config/plugins/dynamix.file.integrity Strange, clear operation haven't clear extend attribute. If you want clear it, pls try install FIP again ( set blake3 ) for that or left those attribute there should be no hurt. Quote Link to comment
DataCollector Posted September 25, 2022 Share Posted September 25, 2022 42 minutes ago, Vr2Io said: Strange, clear operation haven't clear extend attribute. If you want clear it, pls try install FIP again ( set blake3 ) for that or left those attribute there should be no hurt. Yes. Strange. Now I installed FI again, "Automatically protect new and modified files": Enabled Method: "BLAKE3" <Apply> (see screenshot 1) == Then I spun up the first 5 Disks of my '1st System' (look at signature) manually and started in FI "Clear" for those Disks #1-5. (see screenshot 2) == FI toook around 1 Second and then showed in green "cleared" (see screenshot 3) == And I then looked again at the attributes and Blake3 is there. (see screenshot 4) I am very sad. Quote Link to comment
Vr2Io Posted September 25, 2022 Share Posted September 25, 2022 (edited) Pls try /bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -r -b3 /mnt/diskX or /bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -R -b3 /mnt/diskX , where X is disk no. /bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -h , would got bunker help Edit : Overlook you should use remove instead clear Edited September 25, 2022 by Vr2Io 1 Quote Link to comment
ericswpark Posted September 30, 2022 Share Posted September 30, 2022 Unfortunately this plugin isn't working correctly for me. Today it gave me a notification saying that 82 files on my array were corrupt. When I checked the files deemed corrupt by the plugin the hashes matched up to the values in the extended file attributes, so I have no idea where the plugin got the idea that my files were corrupt. It honestly shows how unreliable the plugin is when these "corruption" notifications don't even faze me in the slightest. Quote Link to comment
DataCollector Posted October 9, 2022 Share Posted October 9, 2022 On 9/25/2022 at 2:03 PM, DataCollector said: Yes. Strange. Now I installed FI again, "Automatically protect new and modified files": Enabled Method: "BLAKE3" <Apply> (see screenshot 1) == Then I spun up the first 5 Disks of my '1st System' (look at signature) manually and started in FI "Clear" for those Disks #1-5. (see screenshot 2) == FI toook around 1 Second and then showed in green "cleared" (see screenshot 3) == And I then looked again at the attributes and Blake3 is there. (see screenshot 4) I am very sad. EDIT: it seems "Clear" is the wrong command. Today I tried "Remove" and that command did work on the disks for some time. Quote Link to comment
iarp Posted October 29, 2022 Share Posted October 29, 2022 I wanted to added ".Recycle.Bin,backup/SQL" to the excluded folders and i accidentally hit my enter key. Doing so resets all settings to default. Quote Link to comment
Clayto Posted November 9, 2022 Share Posted November 9, 2022 Hi, I have this plugin installed with Automatically protect new and modified files: enabled. but when I check new files with "getfattr -d <file>" the hash information is missing until I go and run the build again manually. Am I doing something wrong or can I check if it's ruining somehow? 1 Quote Link to comment
iarp Posted November 12, 2022 Share Posted November 12, 2022 I've been trying to track down why so many NEW files do not have a hash. I get emails every single time due to either hash mismatch (usually on nextcloud.log despite excluding *.log files) or hash missing altogether. I dug into the source code of the plugin and replicated the inotifywait command so that i could watch for myself what was going on. inotifywait -mr -e close_write --format '%w%f' /mnt/disk1 /mnt/disk2 and after running mover here is what i see /mnt/disk2/documents/ubuntu-18.04.1-server-amd64.iso.partial /mnt/disk1/backup/unraid/STORAGE/flash/config/editor.cfg.partial /mnt/disk1/backup/unraid/STORAGE/flash/config/super.dat.CA_BACKUP.partial /mnt/disk1/backup/SQL/mariadb/vmosa/2022-11-12-08.00.01.sql.tgz.partial /mnt/disk1/backup/SQL/postgres11/airsonic/airsonic_2022-11-12-08.05.02.sql.tgz.partial /mnt/disk1/backup/SQL/postgres11/authelia/authelia_2022-11-12-08.05.03.sql.tgz.partial /mnt/disk1/backup/SQL/postgres11/family_photos_dev/family_photos_dev_2022-11-12-08.05.03.sql.tgz.partial /mnt/disk1/backup/SQL/postgres11/family_photos_prod/family_photos_prod_2022-11-12-08.05.04.sql.tgz.partial /mnt/disk1/backup/SQL/postgres11/film_convert_dev/film_convert_dev_2022-11-12-08.05.04.sql.tgz.partial /mnt/disk1/backup/SQL/postgres11/film_convert_prod/film_convert_prod_2022-11-12-08.05.05.sql.tgz.partial /mnt/disk1/backup/unraid/STORAGE/flash/config/plugins/dynamix.file.manager/dynamix.file.manager.txz.partial /mnt/disk1/backup/unraid/STORAGE/flash/config/plugins/dynamix.file.manager.plg.partial /mnt/disk1/backup/unraid/STORAGE/flash/config/plugins/unassigned.devices.plg.partial /mnt/disk1/backup/unraid/STORAGE/flash/config/plugins/dynamix.file.integrity/disks.ini.partial As you can see, every file moved by mover ends with .partial and if i run getfattr on the actual file (since there is no .partial file) root@storage:~# getfattr -d /mnt/disk1/backup/SQL/postgres11/airsonic/airsonic_2022-11-12-08.05.02.sql.tgz root@storage:~# When I navigate into the airsonic backup folder and run getfattr on any file in there (because backup always saves to cache and then mover moves) getfattr -d airsonic* is blank for all files. 2 Quote Link to comment
ChatNoir Posted November 15, 2022 Share Posted November 15, 2022 I restarted the plugin after months of pause and I face the same issue. I tried to exclude .partial files in the hope that the plugin will skip it and only consider the real one once the transfer is done. I'll report back. Quote Link to comment
iarp Posted November 15, 2022 Share Posted November 15, 2022 It won't. The move script that mover uses is not standard, its custom written and seems to copy to .partial, deletes the original and then renames the partial to the real name. That renaming does not trigger inotifywait. Quote Link to comment
quattro Posted February 11, 2023 Share Posted February 11, 2023 Has anyone else ever gotten this error? clean install, getting this error when doing an initial build: Finished - added 0 files. Duration: 00:00:01 I had the plugin installed before and it successfully did an initial build and export. But after upgrading the plugin I began getting this error. I removed the plugin, deleted the plugin folder under \boot\config\plugins and rebooted, but still getting the error. Logs are empty. Quote Link to comment
JimmyGerms Posted February 16, 2023 Share Posted February 16, 2023 (edited) Just want to jump in here too, I get the same thing. I manually this on all my disks to remove previous hashes: /bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -r -b2 /mnt/diskX /bin/bash /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -r -b3 /mnt/diskX Removed all old preferences and started fresh but every time I hit build its completes immediately with the same log as the previous post. For now the plugin stays disabled. I'm not quite sure what's going on. EDIT: it also looks like "bunker -R -b3 /mnt/diskX" is broken? it only clears the hash but not the other "user." attrs looking in to the script. Wonder if the new unraid version is causing issues? Edited February 16, 2023 by JimmyGerms Quote Link to comment
Vr2Io Posted February 16, 2023 Share Posted February 16, 2023 (edited) Plugin just call bunker to perform different task, pls check does file permission issue, try use tools - NEW PERMISSIONS to fix that and re-run bunker. Daily add & verify file ( bunker CLI ) without issue. 3 hours ago, JimmyGerms said: it only clears the hash but not the other "user." This normal. Edited February 16, 2023 by Vr2Io Quote Link to comment
JimmyGerms Posted February 16, 2023 Share Posted February 16, 2023 (edited) I've run "New Permissions" not too long ago and everything matches what the help docs have for those Shares. Looking at this code, I'm reading it that if the "user.scandate" attr is empty then using -R should remove the three extra attrs but it doesn't seem to...Not sure where my setup is going wrong. if [[ -z $userdate || $((userdate/86400)) -le $epoch ]]; then ((count++)) setfattr -x user.$hash "$file" 2>/dev/null [[ $cmd == R ]] && setfattr -x user.scandate "$file" 2>/dev/null [[ $cmd == R ]] && setfattr -x user.filedate "$file" 2>/dev/null [[ $cmd == R ]] && setfattr -x user.filesize "$file" 2>/dev/null EDIT: Yeah, not seeing anything wrong in the code. Even if the user.scandate was 0 it still should respect the flags...What's going on with my system then. Cannot get this to kick off at all anymore. Edited February 16, 2023 by JimmyGerms 1 Quote Link to comment
JimmyGerms Posted February 16, 2023 Share Posted February 16, 2023 Ok, rolled back the plugin to this version and it's working again... https://raw.githubusercontent.com/bergware/dynamix/fbd863f64ce1af8ea97ac2d13cc906a410925b9f/unRAIDv6/dynamix.file.integrity.plg Quote Link to comment
Kretsche Posted February 17, 2023 Share Posted February 17, 2023 (edited) 11 hours ago, JimmyGerms said: Ok, rolled back the plugin to this version and it's working again... https://raw.githubusercontent.com/bergware/dynamix/fbd863f64ce1af8ea97ac2d13cc906a410925b9f/unRAIDv6/dynamix.file.integrity.plg Nice find, will try that as I am also getting the "0 files added" and no log files created.... Edited February 17, 2023 by Kretsche Edit: Does not work for me as its not compatible with Unraid 6.11.5 Quote Link to comment
JimmyGerms Posted February 17, 2023 Share Posted February 17, 2023 (edited) On 2/17/2023 at 3:21 AM, Kretsche said: Nice find, will try that as I am also getting the "0 files added" and no log files created.... I'm on 6.11.5 too. The trick for me was I downloaded the corresponding txz with that version of the plugin and placed it on the flash drive. Next I manually installed the plugin by using that link and it worked. Here's the link to the .txz for ya! https://github.com/bergware/dynamix/raw/fbd863f64ce1af8ea97ac2d13cc906a410925b9f/archive/dynamix.file.integrity.txz EDIT: Honestly, it's still not functioning right. New files don't get hashed even though I have the setting "save new files hashes to flash" enabled. Think overall I'm just going to remove this and write a little script to remove left over hashes that are stuck on files. Edited February 19, 2023 by JimmyGerms Quote Link to comment
Krikro Posted February 20, 2023 Share Posted February 20, 2023 New 6.11.5 Unraid install with the plugin freshly installed and the build command is not doing anything for me, it keeps repeating that 0 files have been added even though the disk is not empty Quote Link to comment
Brandon_K Posted February 23, 2023 Share Posted February 23, 2023 Same issue as above, no files being added. 6.11.5 Unlike Jimmy, I was unable to revert to a previous version; plugin: installing: dynamix.file.integrity.plg Executing hook script: pre_plugin_checks plugin: downloading: dynamix.file.integrity.plg ... done Executing hook script: pre_plugin_checks plugin: skipping: inotify-tools-3.14-x86_64-1.txz - Unraid version too high, requires at most version 6.2.9 plugin: skipping: b2sum-20130305-x86_64-1.txz - Unraid version too high, requires at most version 6.3.9 +============================================================================== | Skipping package b3sum-1.0.0-x86_64-1 (already installed) +============================================================================== plugin: downloading: dynamix.file.integrity.txz ... done plugin: bad file MD5: /boot/config/plugins/dynamix.file.integrity/dynamix.file.integrity.txz Executing hook script: post_plugin_checks Quote Link to comment
FTW Posted February 26, 2023 Share Posted February 26, 2023 Hello, Same issue for me for the newer version of this plugin on UnRaid 6.11.5 no files been added, also tried with every hashing methods No Files Added (it run when we do Build and after few minutes, it said 0 file added No Export files (file at 0 so we can click on Export, it run but nothing been exported)... it seem to just scan all files without scanning the hash Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.