Dynamix File Integrity plugin


bonienl

Recommended Posts

Sometime last week, I set my Validation scheduled to weekly on Sunday. Disk 1 kicked off and completed but none of the others did.

 

I'm also wondering how does the Task function work. You can set up One drive per Task which seems the logical choice but you can also assign multiple drives to a single Task.

 

What exactly *is* as "Task" in this case? It seems like a single thread that will do stuff. There's no built-in help around it.

Link to comment

I think I've got this. There is a built-in help when you click on "Disk verification schedule" label and it says:

 

Quote

One or more tasks can be created, each containing one or more disks. All disks under the same task are verified in parallel. This decreases the total run-time, but increases the processor load. At the other end every disk can be put in its own task, resulting in the lowest processor load.

 

But mostly important:

 

Quote

When multiple tasks exist then each task is executed in the next schedule in a rotating fashion. E.g. when you have set up a weekly schedule with 3 tasks then every week another task is executed and after 3 weeks the verification function has done all selected disks.

 

  • Like 3
  • Thanks 1
Link to comment

Hi, 

 

I have downloaded this plugin a few days ago and am now trying to get it to work, but have some questions. 

I have it setup in a way that it should (if I understand it correctly) scan my drives on the first day of every month:

image.thumb.png.bdd676480e04d3875124e984fd290eae.png

 

I then went to the "Tools" tab and started the first build of all the drives that I want:

image.thumb.png.07b7c97b9828c646f646770797383c2c.png

 

This leaves me with a few questions;

1. I don't understand why some disks already say that the build is up to date when this is the first build I am performing. 

2. I wonder how it will tell me if a file has experienced bitrot.

3. How can I manually check to see if a file has experienced bitrot?

 

 

 

  • Like 2
Link to comment

Pretty sure you just uninstall it from the plugins page and then restart your server like any other plugin. I dont think the uninstall process removes hashes from already hashed files though. And the logs are probably not removed either. Any chance those are just leftover from the last time you had it installed?

Link to comment
14 hours ago, superderpbro said:

I reinstalled it and set it to do nothing. Removed all hashes and logs. Be nice to actually uninstall it though!

I get the feeling that this plugin is not actively in development or support anymore. The last pages are filled with people having issues and no solutions. 

 

@bonienl is this plugin still supported? We are experiencing a few issues with it. 

  • Like 1
Link to comment
  • 4 weeks later...
On 3/10/2022 at 1:08 PM, workermaster said:

I get the feeling that this plugin is not actively in development or support anymore. The last pages are filled with people having issues and no solutions. 

 

@bonienl is this plugin still supported? We are experiencing a few issues with it. 

 

All the Dynamix plugins are maintained and supported on best-effort basis.

Right now I am pretty busy and focus is on the next Unraid release.

I am using this plugin myself too and don't see any issues.

 

Note for those who want to uninstall this plugin, before uninstalling the plugin itself you need to remove the calculated hashes, this is done by the "Clear" operation (select all disks and then Clear, see also the built-in Help).

After the plugin is removed, you can delete the folder /config/plugins/dynamix.file.integrity on your flash devices to remove all log and other files.

 

  • Like 1
Link to comment
30 minutes ago, bonienl said:

 

All the Dynamix plugins are maintained and supported on best-effort basis.

Right now I am pretty busy and focus is on the next Unraid release.

I am using this plugin myself too and don't see any issues.

 

Note for those who want to uninstall this plugin, before uninstalling the plugin itself you need to remove the calculated hashes, this is done by the "Clear" operation (select all disks and then Clear, see also the built-in Help).

After the plugin is removed, you can delete the folder /config/plugins/dynamix.file.integrity on your flash devices to remove all log and other files.

 

Thanks for the reply. I think that the main issue is that we simply don't know how to use it. The post I made on the 6th of March shows the issue that I have, and it boils down to me not knowing how te application will inform me of bitrot. 

 

I will uninstall it by following your instructions, but it would be nice if you could find some time in the future to help us out. I would love to use the plugin, but would need to know how it works in order to do so. 

 

Please keep up the good work!

  • Like 1
Link to comment
3 hours ago, workermaster said:

The post I made on the 6th of March shows the issue that I have, and it boils down to me not knowing how te application will inform me of bitrot. 

After the initial build the plugin will watch your file system for any writes made your files. When a write happens it will refresh the hash of the changed file. Then every time the verification task runs it will recalculate all of the hashes and compare them to the hashes saved with the file. If it finds any that dont match it will send a notification (through unraids notification system) alerting you of the file with the mismatched hash. Additionally you can trigger a verification run with the buttons in the plugin. 

Edited by primeval_god
  • Like 1
Link to comment
1 hour ago, primeval_god said:

After the initial build the plugin will watch your file system for any writes made your files. When a write happens it will refresh the hash of the changed file. Then every time the verification task runs it will recalculate all of the hashes and compare them to the hashes saved with the file. If it finds any that dont match it will send a notification (through unraids notification system) alerting you of the file with the mismatched hash. Additionally you can trigger a verification run with the buttons in the plugin. 

Thanks. I will install it again and have it run the first check on all disks later this week. Can I manually brick a file to test the notification and functionality of the tool?

Link to comment
2 hours ago, workermaster said:

Thanks. I will install it again and have it run the first check on all disks later this week. Can I manually brick a file to test the notification and functionality of the tool?

Not sure, you might be able to disable "automatically protect new and modified files" after your initial build then modify a file and attempt a validation. Since the file is modified but its hash is not updated it should trigger an alert. Just be sure to reenable the setting afterword and rebuild the disk you modified a file on. Also word of advice since validation tasks run in parallel, with a lot of disks it is best to space validation out into several tasks that run at different times. For my 5 disks i run verification on one disk every friday (each disk is only verified every 5 weeks).

 

Also a quick fyi for you and anyone who reads this post. Dont forget to check the help text for this plugin. Many people, myself included, forget that the help button in unRAID often reveals useful information, and many of our great plugin authors make use of it.

Edited by primeval_god
  • Like 2
Link to comment
On 4/7/2022 at 6:37 PM, primeval_god said:

Not sure, you might be able to disable "automatically protect new and modified files" after your initial build then modify a file and attempt a validation. Since the file is modified but its hash is not updated it should trigger an alert. Just be sure to reenable the setting afterword and rebuild the disk you modified a file on. Also word of advice since validation tasks run in parallel, with a lot of disks it is best to space validation out into several tasks that run at different times. For my 5 disks i run verification on one disk every friday (each disk is only verified every 5 weeks).

 

Also a quick fyi for you and anyone who reads this post. Dont forget to check the help text for this plugin. Many people, myself included, forget that the help button in unRAID often reveals useful information, and many of our great plugin authors make use of it.

I tried to test the functionality but think that I did something wrong. 

 

I started a build of a drive and let it complete. After the build was done, I exported the result. I then turned off the option to automatically update the hashfile when a file changes. Then I opened a picture file on the drive with Notepad. This gave me a garbled mess in Notepad and I delted a few characters. This corrupts the file itself. In the plugin settings menu I told it to start checking the drive every day and waited for the check to be completed, but never got a warning or error about the corrupted file. I have no idea why this didn't work. 

 

After this failed test, I turned on the auto file hash update option so all files would stay up to date. Then I started building all the drives and exported them. Today I let the system run an integrity check and got a few errors. But I think these were false positives.

 

Here is one of the errors;

image.png.4b4d27fc6b3ea2a4fa6a3b6dd864e095.png

I think this is a false positive because that file changes all the time. I also do not want to monitor .log files anymore. I will try to set a rule to not check .log files. 

 

Here is another false positive that I am sure of is a false positive;

image.thumb.png.08e7604c1b985cd25818a7dac413a8b7.png

This file was being written to the disk while the integrity check was in progress. All 4 files being corrupt after sitting on the disk for a few minutes seems to unlikely for it to be true. 

 

Lastly I get a lot of these messages;

image.thumb.png.b7334c72995dffa7061082ce9050052c.png

I do not know where to find the that path since /tmp was the ram right?

 

 

So the plugin seems to work and do the checks that it should do, but there are some false positives.

 

Link to comment

I let the integrity check finish and got some more results. All 11 of my drives got either hash mismatches or corruptions. And not just 1 file, but multiple files. One drive even has 9 corruptions. This would mean that all 11 of the drives have corrupted multiple files within 24 hours. This also happened with my previous testing. 

 

It could be that I am doing something wrong, but when I Google for "Unraid file integrity plugin false positives", I get a lot of search results for false positives. A lot of other people are having the same issue as I am right now.

 

I also read about people thinking that the option to automatically update the hashes of new files does not work. I share that feeling since new files are mentioned as "corrupt" by the plugin, even though the option is turned on. 

 

I have uninstalled the plugin by first deleting the hash files, then removing the plugin and then removing the plugin folder from the boot drive. After doing some more reading online, I want to keep testing this plugin, so I decided to install it again (something about not being able to makeup my mind).

 

I have added some exclusions. Can anyone tell me if this is the correct way of doing them?;

image.thumb.png.76b0596bbe241b4e252225b608e7f703.png

I don't want any of the .xml, .srt or .log files to be protected, since they change a lot and I don't care if they break. 

 

 

image.png

Edited by workermaster
Link to comment

I kept getting data corruption and missing files and a parity sync kept reporting errors if I wait a few days after the last parity check. I managed to isolate it to one drive I swapped it out and did a parity rebuild but the issue persisted so I assumed a corrupted XFS. I was using this utility to help track corruptions and it was a clear benefit there. It seemed that removing all files fixed whatever corruption was happening, had no issues even after filling up the drive with large NULL files in the event I needed to 7zip the drive up to give someone to check but it never failed in the next month.

 

Since I needed to reformat, I decided to switch to btrfs to use it's own built in scrub for bitrot, and I was thinking of taking advantage of btrfs features.

Link to comment
  • 2 weeks later...

This is in settings 

 

/mnt/user/BlackTowerMain/storj,/mnt/user/storjnode2 (second entry I just added)

 

 

for folders, and 

 

*.db,*.v1,*.sj1,*.log,*.tmp,.storj*,.Recycle.Bin 

 

in Custom files.

 

Storj nodes update so much (and are self healing probably) but I was still getting log lines such as 

 

BLAKE3 hash key mismatch, /mnt/disk3/BlackTowerMain/storj/storage/blobs/pmw6tvzmf2jv6giyybmmvl4o2ahqlaldsaeha4yx74n5aaaaaaaa/2q/ouw7yyys2wfigjx2vfmmns4if2naa77erhiudde3p6y2apqhaa.sj1 is corrupted

(and LOTS of them!)

 

Or am I missing some syntax/formatting?

 

Basically, I really don't care about corruption in these directories at all. It's all storj slices so I doubt it'd even matter. The other nodes, I presume, would just heal it and I'd get a CRC/parity error etc.


 

Link to comment

I have started using the plugin again after fixing the things happening in my previous posts. Now the plugin checks the integrity every first day of the month. There is 1 disk in my system that is very slow. It should be just as fast as the other (same type) disk in my system, but instead of taking about 16 to 18 hours to finish, it takes over 3 days to finish a check. 

 

How can I figure out why that drive is so much slower? Is has a lot of big files on it, just like all the other drives in the system. During my previous testing, this drive was just as fast as the other drives. 

Link to comment

Hi Guys and Gals..

Longtime user but not any kind of programmer, though I normally find solutions (eventually).

I find myself in need of some assistance from far more competent minds than mine.

Scenario:

Xeon 4C/8T  64GB Ram

Disks:  Parity & Disk1 are WD Ultrastar DC HC330 (CMR 7200) - fast changing data

Disk2 to Disk6 contiguous are Seagate ST4000DM004-2CV104 (SMR 5400) - slow changing data

Dynamix file integrity installed and set to watch new/modified files.

I wish to use xfs_fsr on disks that are monitored by Dynamix file integrity.

 

Testing:

Ran xfs_fsr -v /dev/md1

Server went NUTS:

htop showed load average ABOVE 16

and 8+ files being accessed by md5sum

so I waited until it had all completed and Disk1 had stopped thrashing, then I started digging...

 

Found rc.watcher supports stop and start: so...

 

cd /usr/local/emhttp/plugins/dynamix.file.integrity/scripts

./rc.watcher stop >> /mnt/cache/Transcode/fsr.log

xfs_fsr -v /dev/md2 | tee -a /mnt/cache/Transcode/fsr.log

./rc.watcher start >> /mnt/cache/Transcode/fsr.log

 

Seemed to work, no thrashing, no md5sum processes running, EXCEPT after a while in terminal this appeared:

( screenshot titled bunker error.png)

 

Something wrong there, nothing in logs or notifications...

 

Performed a clean restart of server just to make sure it is stable and post this cry for help!

 

What service(s) do I need to stop/pause/suspend/kill before running xfs_fsr?

What service(s) do I need to start/unpause/resume afterwards?

 

Thanks for your consideration & help.

 

bunker error.png

Link to comment

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.