Dynamix File Integrity plugin

[clevoir]

I have now taken my Unraid array (6.2b19) back to the basic install

 

I have installed the latest version of the plugin, build & exported all drives, and set it automatically monitor new / changed files. I have also added the cache directories plugin + Plex docker.

 

Everything appears to be working on OK, but I have two entries shown on a monitor attached to the array which read:-

 

grep: /boot/config/plugins/dynamix.file.integrity/disks.ini: No such file or directory

 

However on checking the contents of the flash drive in Windows, the disks.ini file exists?

[BRiT]

You really should be running the latest beta21 first. Earlier betas of 6.2 are no longer supported by LimeTech and thus shouldnt be supported by the community either.

[clevoir]

I have now added around 12 files to a share, which moved OK, and then tried to add around another 12 files to the same share, Terracopy crashes in doing so

 

I have closed Terracopy using Task Manager and found that Windows Explorer has crashed too

 

On trying to stop the array, the GUI does not refresh and therefore I am unable to stop or reboot the array.

 

I was expriencing the same behaviour using b21 using the plugin, and though that I would try and go back to b19 to determine if it is a problem with the Unraid or the plugin.

 

After adding the first batch of files I looked at the processor load, which was running @ 45% and I did not recieve any further messages on the monitor which I have attached to the array.

 

I am able to ping the array OK, but it appears that the GUI has hung. I honest don't know if this is a plugin or unraid problem, but I was able to alter / add files OK before reinstalling the plugin.

[clevoir]

I have now rebooted the array manually, and found that although Terracopy had reported the first batch of files had moved OK to the share, they were in fact missing.

 

I have therefore tried to move the files again to the share, and Terracopy & Winows Explorer hang again.

 

On checking the log, the following entries were shown regarding the plugin after the array had rebooted. Looking back I should have got a diagnostics log instead

 

Apr 9 17:07:32 Tower emhttp: shcmd (122): /usr/local/emhttp/plugins/dynamix.docker.manager/scripts/dockerupdate.php |& logger

Apr 9 17:07:45 Tower root: Updating templates... Updating info... Done.

Apr 9 17:07:45 Tower emhttp: nothing to sync

 

On stopping the array, the GUI hung again and I had to do another manual reboot. Each manual reboot does not trigger a parity check though?

 

I have now started the array in safe mode without any plugins, and I am able to move the same batches of files OK without Terracopy / Windows Explorer hanging.

 

I appreciate that I have the cache directories plugin installed, which would kick in after a reboot. However when I attempted to add the first batch of files, this was some 6 hours after I rebooted the array and as such don't think this is connected.

[clevoir]

I now think this is UnRaid problem has Windows Explorer is hanging on 2 network PC's on trying to access shares on the array. As no plugins are running I am sure that this can't be a problem with this plug. I also recall reading another post about problems with GUi hanging in b20 / b21.

 

I'll have to do some more testing

[jakea333]

I've never had any luck with this plugin keeping itself up to date. I have the setting "Automatically protect new and modified files:" set to "Enabled" but it doesn't seem to ever correctly work. So far, I've just manually rebuilt every few days which works fine, but I'd really prefer the near real-time protection enabled.

 

I think it might be related to the inotifywait component. When I look at the config I see:

root@Tower:~# cat /etc/inotifywait.conf
cmd="a"
method="-md5"
exclude="(Domain_Backups/|Podcasts/)"
disks=""

 

Is that "disks" parameter supposed to contain all of my array disks?

 

I've intermittently seen an error in my Syslog that's related I believe.

Apr 14 20:06:08 Tower inotifywait[22757]: Failed to watch /mnt/disk5; upper limit on inotify watches reached!
Apr 14 20:06:08 Tower inotifywait[22757]: Please increase the amount of inotify watches allowed per user via `/proc/sys/fs/inotify

 

I went to that location and increased the watches from the default 524288 to 1524288 as a rough test. I then reapplied the settings in File Integrity but saw no change. Disk5 contains a backup of my Plex Library, which has a large number of small files. My guess is that it's larger than the default watches, but my quick change didn't seem to work. I've got the Share it's stored in excluded, but I don't believe that'll matter to the disk watches.

 

Any suggestions for what I might do to troubleshoot next? I haven't found much in the way of logs to help parse what's going on, so I'm a little out of my depth.

 

***UPDATE

It seems I spoke too soon. Since updating the max # of watches my files have been staying up to date. I added a line to my Go file to set my max watches to ~2 million each time unRAID boots. I figure that's enough so that I won't have to worry about it. Looks like inotifywait uses ~225 MB of RAM on my system, but that seems a small price to pay for the functionality of this plug-in. I still see the disks="" but the plug-in seems to be functioning correctly now.

[GoChris]

I'm sure this has been answered, but it's not the first post and I don't have time to read 30 more pages.

 

When it tells me there are 3 files with hash key corruption, where does it list the files?

 

Seems a bit odd that I only started using this a few days ago, and now there are corrupted files. But I want to investigate.

[John_M]

I'm sure this has been answered, but it's not the first post and I don't have time to read 30 more pages.

 

When it tells me there are 3 files with hash key corruption, where does it list the files?

 

Seems a bit odd that I only started using this a few days ago, and now there are corrupted files. But I want to investigate.

 

Look in the syslog.

 

[John_M]

I'm still getting syslog entries like these:

 

Apr 24 06:00:14 Lapulapu bunker: Verify task for disk2 started, total files size 3.40 TB
Apr 24 11:44:27 Lapulapu bunker: warning: BLAKE2 hash key mismatch (updated), /mnt/disk2/L_Public/.AppleDB/db_errlog was modified
Apr 24 11:44:27 Lapulapu bunker: warning: BLAKE2 hash key mismatch (updated), /mnt/disk2/L_Public/.AppleDB/log.0000000001 was modified
Apr 24 11:44:27 Lapulapu bunker: warning: BLAKE2 hash key mismatch (updated), /mnt/disk2/L_Public/.AppleDB/cnid2.db was modified
Apr 24 11:44:27 Lapulapu bunker: Verify task for disk2 finished, duration: 5 hr, 44 min, 26 sec.
Apr 24 11:44:27 Lapulapu bunker: verified 2082 files from /mnt/disk2. Found: 3 mismatches (updated), 0 corruptions. Duration: 05:44:26. Average speed: 164 MB/s

 

despite having excluded Apple metadata and starting with a fresh database by having first run

 

rm -rf /mnt/user/L_Public/.AppleDB

 

to remove the database, then connecting a Macintosh client via AFP and allowing it to recreate the .AppleDB folder and contents. Does the exclusion of Apple metadata actually exclude the three files listed?

 

EDIT: I looked back to this post earlier in the thread and ran the command again:

 

root@Lapulapu:~# ps -ef | grep inotify
root     16813     1  0 Apr12 ?        00:00:00 inotifywait -dsrqo /var/run/hash.pipe -e close_write --exclude '' --format %w%f /mnt/disk1 /mnt/disk2 /mnt/disk6
root     17207 17175  0 14:13 pts/6    00:00:00 grep inotify
root@Lapulapu:~#

 

which shows no exclusions: --exclude ''

So I went back to Settings and unchecked then rechecked the Apple metadata exclusion box and ran the command again:

 

root@Lapulapu:~# ps -ef | grep inotify
root     17899     1  0 14:15 ?        00:00:00 inotifywait -dsrqo /var/run/hash.pipe -e close_write --exclude ^/mnt/disk[0-9]+/(.*\.icns$|.*\.AppleDB/|.*\.DS_Store$) --format %w%f /mnt/disk1 /mnt/disk2 /mnt/disk6
root     18102 17175  0 14:15 pts/6    00:00:00 grep inotify
root@Lapulapu:~#

 

showing that the exclusions had been reset: --exclude ^/mnt/disk[0-9]+/(.*\.icns$|.*\.AppleDB/|.*\.DS_Store$)

 

I guess the settings had been lost during recent plugin updates, though I always follow the instructions to re-apply the settings after an update of this plugin. I usually do this by toggling off the Send notifications option, then toggling it back on and clicking Apply, but I suppose that hasn't always been enough.

 

EDIT2: I thought I'd better check my other main server:

 

root@Northolt:~# ps -ef | grep inotify
root     28977 28922  0 14:43 pts/0    00:00:00 grep inotify
root     40726     1  0 Apr09 ?        00:00:00 inotifywait -dsrqo /var/run/hash.pipe -e close_write --exclude ^/mnt/disk[0-9]+/(.*\.icns$|.*\.AppleDB/|.*\.DS_Store$) --format %w%f /mnt/disk1 /mnt/disk2 /mnt/disk3
root@Northolt:~#

 

where the settings have obviously stuck. The only software difference is that Northolt is running 6.1.9 (plus) and Lapulapu is running 6.2.0beta21 (pro).

 

[Kir]

Is there a way to manually check a particular disk?

[John_M]

Is there a way to manually check a particular disk?

Tools -> File Integrity. Then check the box for the disk you want to verify and click the Check button. That will check the disk against a previously exported checksum file. I'm not aware of a simple way of forcing the checking of a disk against the checksums stored in extended attributes without resorting to the command line or fiddling with the crontab.

 

[Kir]

Is there a way to manually check a particular disk?

Tools -> File Integrity. Then check the box for the disk you want to verify and click the Check button. That will check the disk against a previously exported checksum file. I'm not aware of a simple way of forcing the checking of a disk against the checksums stored in extended attributes without resorting to the command line or fiddling with the crontab.

 

Yes, that much I figured out myself. However, I don't export checksums, so it leaves me in limbo.

[JorgeB]

Is there a way to manually check a particular disk?

Tools -> File Integrity. Then check the box for the disk you want to verify and click the Check button. That will check the disk against a previously exported checksum file. I'm not aware of a simple way of forcing the checking of a disk against the checksums stored in extended attributes without resorting to the command line or fiddling with the crontab.

 

Yes, that much I figured out myself. However, I don't export checksums, so it leaves me in limbo.

 

Do an export before running the check, it should only take a few seconds.

[gshlomi]

Can't install the plugin:

 

plugin: installing: https://raw.github.com/bergware/dynamix/master/unRAIDv6/dynamix.file.integrity.plg
plugin: downloading https://raw.github.com/bergware/dynamix/master/unRAIDv6/dynamix.file.integrity.plg
plugin: downloading: https://raw.github.com/bergware/dynamix/master/unRAIDv6/dynamix.file.integrity.plg ... done
plugin: downloading: http://mirrors.slackware.com/slackware/slackware64-14.1/slackware64/a/inotify-tools-3.14-x86_64-1.txz ... failed (Invalid URL / Server error response)
plugin: wget: http://mirrors.slackware.com/slackware/slackware64-14.1/slackware64/a/inotify-tools-3.14-x86_64-1.txz download failure (Invalid URL / Server error response

[bonienl]

Can't install the plugin:

 

plugin: installing: https://raw.github.com/bergware/dynamix/master/unRAIDv6/dynamix.file.integrity.plg
plugin: downloading https://raw.github.com/bergware/dynamix/master/unRAIDv6/dynamix.file.integrity.plg
plugin: downloading: https://raw.github.com/bergware/dynamix/master/unRAIDv6/dynamix.file.integrity.plg ... done
plugin: downloading: http://mirrors.slackware.com/slackware/slackware64-14.1/slackware64/a/inotify-tools-3.14-x86_64-1.txz ... failed (Invalid URL / Server error response)
plugin: wget: http://mirrors.slackware.com/slackware/slackware64-14.1/slackware64/a/inotify-tools-3.14-x86_64-1.txz download failure (Invalid URL / Server error response

 

Looks like the download site isn't reachable from your location, perhaps retry later.

 

[gshlomi]

Can't install the plugin:

 

plugin: installing: https://raw.github.com/bergware/dynamix/master/unRAIDv6/dynamix.file.integrity.plg
plugin: downloading https://raw.github.com/bergware/dynamix/master/unRAIDv6/dynamix.file.integrity.plg
plugin: downloading: https://raw.github.com/bergware/dynamix/master/unRAIDv6/dynamix.file.integrity.plg ... done
plugin: downloading: http://mirrors.slackware.com/slackware/slackware64-14.1/slackware64/a/inotify-tools-3.14-x86_64-1.txz ... failed (Invalid URL / Server error response)
plugin: wget: http://mirrors.slackware.com/slackware/slackware64-14.1/slackware64/a/inotify-tools-3.14-x86_64-1.txz download failure (Invalid URL / Server error response

 

Looks like the download site isn't reachable from your location, perhaps retry later.

 

Trying to access http://mirrors.slackware.com/slackware/slackware64-14.1/slackware64/a/inotify-tools-3.14-x86_64-1.txz manually, I'm getting redirected to http://mirror.bjtu.edu.cn/slackware/slackware64-14.1/slackware64/a/inotify-tools-3.14-x86_64-1.txz, getting a 404.

Is there a way to receive the files from a specific mirror or something?

[bonienl]

Try: http://slackware.cs.utah.edu/pub/slackware/slackware64-14.1/slackware64/a/inotify-tools-3.14-x86_64-1.txz

 

[gshlomi]

Try: http://slackware.cs.utah.edu/pub/slackware/slackware64-14.1/slackware64/a/inotify-tools-3.14-x86_64-1.txz

How do I incorporate it into the plugin installation process ?

I'm installing using Community Apps...

Thanks

[bonienl]

Try: http://slackware.cs.utah.edu/pub/slackware/slackware64-14.1/slackware64/a/inotify-tools-3.14-x86_64-1.txz

How do I incorporate it into the plugin installation process ?

I'm installing using Community Apps...

Thanks

 

Copy the .txz file to the folder /packages on your flash disk (you need to create the folder if not existing). Afterwards you can do an installation of the plugin and it will recognize the presence of the .txz file.

 

[gshlomi]

Try: http://slackware.cs.utah.edu/pub/slackware/slackware64-14.1/slackware64/a/inotify-tools-3.14-x86_64-1.txz

How do I incorporate it into the plugin installation process ?

I'm installing using Community Apps...

Thanks

 

Copy the .txz file to the folder /packages on your flash disk (you need to create the folder if not existing). Afterwards you can do an installation of the plugin and it will recognize the presence of the .txz file.

Thanks.

That did the trick...

[bobbintb]

I installed the system temp plugin because I keep getting message from the kernel about temps above threshold and throttled CPU. I think these are fast positives but trying to make sure. When temperature do I look at for CPU?

 

coretemp - Physical id 0 - 99.0 C
coretemp - Core 0 - 99.0 C
coretemp - Core 1 - 100.0 C
coretemp - Core 2 - 96.0 C
coretemp - Core 3 - 91.0 C
nct6776 - MB Temp - 27.0 C
nct6776 - CPU Temp - 39.0 C
nct6776 - AUXTIN - 56.0 C
nct6776 - PECI Agent - 95.0 C

 

I'm just confused at how the CPU core temperatures can be so much higher than the CPU temperature.

[LolHens]

Hi!

 

Would it be possible to add an "Errors" option to the notifications setting so that it only shows file integrity errors? 

The info messages really clutter the interface after a few days. I wish unRAID would have an option to only show warnings and errors...

 

[RobJ]

I installed the system temp plugin because I keep getting message from the kernel about temps above threshold and throttled CPU. I think these are fast positives but trying to make sure. When temperature do I look at for CPU?

 

coretemp - Physical id 0 - 99.0 C
coretemp - Core 0 - 99.0 C
coretemp - Core 1 - 100.0 C
coretemp - Core 2 - 96.0 C
coretemp - Core 3 - 91.0 C
nct6776 - MB Temp - 27.0 C
nct6776 - CPU Temp - 39.0 C
nct6776 - AUXTIN - 56.0 C
nct6776 - PECI Agent - 95.0 C

 

I'm just confused at how the CPU core temperatures can be so much higher than the CPU temperature.

 

Perhaps you meant to post this in the System Temp thread? 

 

Edit:  hmm...  I thought there was a System Temp support thread.  It appears to be the catchall thread instead, Dynamix - V6 Plugins.

[jeffreywhunter]

Just found this interesting tool and having some questions about setup.  Followed the settings found in this thread, but I don't see any discussion around the "Tasks" in the Disk Verification Tasks section.  Is there documentation on the tasks?  Do I need to set something up?  Are there additional utilities I need to install to get it working?

 

Sorry, another newbie trying to learn...

[bonienl]

Just found this interesting tool and having some questions about setup.  Followed the settings found in this thread, but I don't see any discussion around the "Tasks" in the Disk Verification Tasks section.  Is there documentation on the tasks?  Do I need to set something up?  Are there additional utilities I need to install to get it working?

 

Sorry, another newbie trying to learn...

 

Did you have a look at the included help?

 

The tasks are used to define a background verification schedule. This allows the plugin to detect silent corruption.