Dynamix File Integrity plugin

[bonienl]

Started 3 build operations for disks 1, 2 and 3 (see first picture)

 

After a few minutes canceled disk 3, both disk 1 and 2 continue to work (see second picture)

 

You might want to clear your browser cookies, and start fresh, perhaps that helps?

 

Also try the latest version of today. Several corrections have been made in the past days.

[JorgeB]

Can o please repeat your test but canceling  disk1 instead of 3?  I'm not close to a server right now but I believe it didn't happened if I canceled the last disk.

[bonienl]

Can o please repeat your test but canceling  disk1 instead of 3?  I'm not close to a server right now but I believe it didn't happened if I canceled the last disk.

 

Good catch, canceling the first disk 1 gives your situation.

 

Let me check and correct that ...

[JorgeB]

Nice to know I'm not imagining things

 

Thanks for looking into it.

 

[bonienl]

Made a correction and fixed the incorrect display update when a running disk operation is canceled. There is a mistake corrected too in the process termination logic (it didn't always find the corresponding pid).

 

Please update to version 2016.01.01b if you want to use the Cancel functionality on the control page.

 

 

[archedraft]

I am getting the following from my unRAID terminal on upon reboot:

unRAID login: /etc/inotifywait.conf: line 3: /mnt/disk2: Is a directory

 

I am guessing it has to do with this plugin?

[bonienl]

I am getting the following from my unRAID terminal on upon reboot:

unRAID login: /etc/inotifywait.conf: line 3: /mnt/disk2: Is a directory

 

I am guessing it has to do with this plugin?

 

Yes, but wondering why the message appears. Can't find a clear reason.

 

Can you show the content of this file on your system (below is mine):

# cat /etc/inotifywait.conf
cmd="a"
method="-b2"
exclude=""
disks="/mnt/disk1 /mnt/disk2 /mnt/disk3 /mnt/disk4 /mnt/disk5 /mnt/disk6 /mnt/disk7 /mnt/disk8 /mnt/disk9 /mnt/disk10 /mnt/disk11"

 

[archedraft]

# cat /etc/inotifywait.conf

method=-b2

exclude=

disks=/mnt/disk1 /mnt/disk2

 

I also got an email from my server. Same message but a little more info:

Subject:

cron for user root /usr/bin/run-parts /etc/cron.daily 1> /dev/null

Body:

/etc/inotifywait.conf: line 3: /mnt/disk2: Is a directory

[bonienl]

# cat /etc/inotifywait.conf

method=-b2

exclude=

disks=/mnt/disk1 /mnt/disk2

 

I also got an email from my server. Same message but a little more info:

Subject:

cron for user root /usr/bin/run-parts /etc/cron.daily 1> /dev/null

Body:

/etc/inotifywait.conf: line 3: /mnt/disk2: Is a directory

 

In the latest version I have quoted the values of the conf file, e.g. disks="/mnt/disk1 /mnt/disk2". This is needed when spaces are present.

 

You may want to upgrade. As an extra measure I suppress error readings.

 

Thanks for testing.

 

[archedraft]

In the latest version I have quoted the values of the conf file, e.g. disks="/mnt/disk1 /mnt/disk2". This is needed when spaces are present.

 

You may want to upgrade. As an extra measure I suppress error readings.

 

Thanks for testing.

 

Thanks, I was 1 version off  Latest version added the "".

[LateNight]

I'm trying to upgrade from 2016.0.01a to 2016.01.02 but receive the following error:

 

plugin: bad file MD5: /boot/config/plugins/dynamix.file.integrity/dynamix.file.integrity.txz

 

[JorgeB]

There appears to be something wrong in latest version, build works ok but trying to run a check completes without any file being verified, syslog fills with file not found errors, file names on log are incomplete, probably the cause of the problem.

[bonienl]

There appears to be something wrong in latest version, build works ok but trying to run a check completes without any file being verified, syslog fills with file not found errors, file names on log are incomplete, probably the cause of the problem.

 

You may want to use the "really" latest version which is 2016.01.03.

 

Can you check the format of your export files, is it still in the old format with 5 fields separated by a | character, or is it the new format which is 2 fields separated by a * character.

 

All commands now expect the new format and are not able to read the old format.

 

Thanks for testing.

 

[bonienl]

I'm trying to upgrade from 2016.0.01a to 2016.01.02 but receive the following error:

 

plugin: bad file MD5: /boot/config/plugins/dynamix.file.integrity/dynamix.file.integrity.txz

 

This may be a bad download. There is a newer version available now. Do a "Check for updates" and do an installation. Should go all fine.

 

Thanks for testing.

[JorgeB]

Updated but still not working, I’m only using new format only, from a quick test:

 

Exported file:

fe893c472a452790f85f7bb720fea856 */mnt/disk4/test/james.corden.2015.11.12.saoirse.ronan.hdtv.x264-crooks.mp4
b2acaee8d6152c5fbaf27ad1abca9074 */mnt/disk4/test/james.corden.2015.11.16.natalie.dormer.hdtv.x264-crooks.mp4
61b6e399a721af7aa6ee53e55e5e05b8 */mnt/disk4/test/james.corden.2015.11.17.ethan.hawke.hdtv.x264-crooks.mp4
4e66607e129855e777f68c7c40e128cd */mnt/disk4/test/james.corden.2015.11.18.anthony.mackie.hdtv.x264-crooks.mp4
52b644ea8298b38b2891c7d89339db90 */mnt/disk4/test/james.corden.2015.11.19.bryan.cranston.hdtv.x264-crooks.mp4

 

From syslog after pressing check:

Jan  3 10:22:53 Testv6 bunker: warning: 5.11.12.saoirse.ronan.hdtv.x264-crooks.mp4 is missing
Jan  3 10:22:53 Testv6 bunker: warning: 5.11.16.natalie.dormer.hdtv.x264-crooks.mp4 is missing
Jan  3 10:22:53 Testv6 bunker: warning: 5.11.17.ethan.hawke.hdtv.x264-crooks.mp4 is missing
Jan  3 10:22:53 Testv6 bunker: warning: 5.11.18.anthony.mackie.hdtv.x264-crooks.mp4 is missing
Jan  3 10:22:53 Testv6 bunker: warning: 5.11.19.bryan.cranston.hdtv.x264-crooks.mp4 is missing

 

[bonienl]

Can you give me a step by step overview how to produce?

 

I just started a 'check' here and works as expected.

 

[JorgeB]

Select a disk and press build, wait for build to finish.

 

Select same disk and press check.

 

I even tried with simpler file names just in case and got same result:

 

disk4.export.hash

fe893c472a452790f85f7bb720fea856 */mnt/disk4/test/1.mp4
b2acaee8d6152c5fbaf27ad1abca9074 */mnt/disk4/test/2.mp4
61b6e399a721af7aa6ee53e55e5e05b8 */mnt/disk4/test/3.mp4
4e66607e129855e777f68c7c40e128cd */mnt/disk4/test/4.mp4
52b644ea8298b38b2891c7d89339db90 */mnt/disk4/test/5.mp4

 

syslog

Jan 3 10:39:43 Testv6 bunker: added 5 files from /mnt/disk4 with mask *. Duration: 00:00:06 Average speed: 329. MB/s
Jan 3 10:39:49 Testv6 bunker: warning: is missing
Jan 3 10:39:49 Testv6 bunker: warning: is missing
Jan 3 10:39:49 Testv6 bunker: warning: is missing
Jan 3 10:39:49 Testv6 bunker: warning: is missing
Jan 3 10:39:49 Testv6 bunker: warning: is missing

 

[bonienl]

Can you try:

 

build - export -check

 

Build means the extended attributes are created.

Export means extended atrributes are exported to a file

Check means export file is used to check against calculated hash of file(s)

 

Also: after upgrading to the latest version, you need to stop/start the protect function on the settings page, in order to make the update work.

[JorgeB]

OK, believe I found something, check works on hashs done with BLAKE2, fails on hashs done with MD5 or SHA256.

[bonienl]

OK, believe I found something, check works on hashs done with BLAKE2, fails on hashs done with MD5 or SHA256.

 

Ok, need to investigate that (I am using BLAKE2 all the time).

 

Thanks for testing and helping out.

 

[bonienl]

Well a 'stupid' mistake from my side. The key lengths of MD5 and SHA256 were mixed up. Causing both methods to use the wrong key size when performing their operations.

 

I've made a correction for this, plus some more updates. See the release notes of version 2013.01.03a

 

Please update to this version.

 

 

[JorgeB]

That was it, thanks.

 

One little suggestion if you don’t mind, at the moment when the check is finish it only displays “Check finished”, I’d like to request more info, for example you could take the info displayed during a check and modify it a litle to display at the end:

 

Check finished, processed xxx of xxx files. Skipped: 0 files. Found: 0 mismatches, 0 corruptions

 

I believe you’re taking a simpler approach and just notifying in case of errors but I would feel better having more info of what was actually checked in the end.

 

[bonienl]

One little suggestion if you don’t mind, at the moment when the check is finish it only displays “Check finished”, I’d like to request more info, for example you could take the info displayed during a check and modify it a litle to display at the end:

 

Sure, that is possible and available. Please update to version 2016.01.04.

 

Below a screenshot of what it looks like.

[landS]

Really excited for this - initial check nearly done on my i5 server (3 4Tb data disks).  Is it normal for a lower specd server to become fully unresponsive during the initial check (specs in signature)

[bonienl]

Really excited for this - initial check nearly done on my i5 server (3 4Tb data disks).  Is it normal for a lower specd server to become fully unresponsive during the initial check (specs in signature)

 

If you started all three disks simultaneously, it can introduce a full load on your processor. Alternatively you can build one disk at the time to lower the load, but it will take longer to complete all disks.