Dynamix File Integrity plugin

[clevoir]

I have just saved some files to a share named MP3 on disk 2, and have just recieved the attached messages regarding setfattr

 

Are these messages related to this plugin?

[bonienl]

I have just saved some files to a share named MP3 on disk 2, and have just recieved the attached messages regarding setfattr

 

Are these messages related to this plugin?

 

Disable the file integrity protection and verification, and retest.

 

[clevoir]

I have now copied 6 test files, totaling 8Gb, to a test folder under the MP3 share. I have not recieved any error reports regarding these files since disabling creation and verification of files.

 

On coming back to my server, I had noticed more error messages regarding another share - this is before I disabled creation / verification of files

 

An example is:-

 

stat: cannot stat '  /mnt/disk2/Hi Def 2/The Big Bang Theory/Series 6/mymovies.ing : No such file or directory

/bin/md5sum: ' /mnt/disk2/Hi Def 2 /The Big Bang Theory/Series 6/mymovies.ign' : No such file or directory

 

On looking at The Big Bang Theory / Series 6 folder I don't have a mymovies.ign file?

 

Just to confirm I am running version 2016.04.02

[RAINMAN]

My main concern now is the FIND feature.  This is using a significant portion of my CPU when I am not wanting to do anything with finding duplicates.

 

When you open the integrity control page it will start every 3 minutes a background job which is used to update the status of build and export of each disk. This is unrelated to the find command.

 

You can check if the background job has issues by running it in the CLI:

/etc/cron.daily/exportrotate -q

 

Any suggestions why 2 of my disks need to rebuilt every day?  Possibly I have to many files for innotify?  I have a few million files (due to plex app data daily backups)

 

I am pretty sure I figured this out with a lot of testing.  I'm running rsnapshot sync each night to backup my appdata folder to my array.  This is not getting picked up by the plugin.  Why though.  I am not sure.  This is run directly on unraid via cron.  Its not within a plugin.  Maybe that's it?

 

I'm going to try excluding this folder tonight and see if it fixed the unbuilt disk.

 

This solved it.  It seems that anything transfered with rsnapshot sync is not getting picked up by the plugin.

[EdgarWallace]

The plugin keeps complaining about the files you see below in spite they are all ok. Does is has something to do with the file names?

 

Event: unRAID file corruption
Subject: Notice [TOWER] - bunker verify command
Description: Found 8 files with BLAKE2 hash key corruption
Importance: alert

BLAKE2 hash key mismatch, /mnt/disk6/iTunes/Richard Wagner; Bayreuther Festspiele, Pierre Boulez/Das Rheingold/05 Szene I. Lugt, Schwestern! Die Weckerin lacht in den Grund.aiff is corrupted
BLAKE2 hash key mismatch, /mnt/disk6/iTunes/Richard Wagner; Bayreuther Festspiele, Pierre Boulez/Das Rheingold/07 Szene I. Orchesterzwischenspiel.aiff is corrupted
BLAKE2 hash key mismatch, /mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/01 If I Were A Bell.aif is corrupted
BLAKE2 hash key mismatch, /mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/02 You're My Everything.aif is corrupted
BLAKE2 hash key mismatch, /mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/03 I Could Write A Book.aif is corrupted
BLAKE2 hash key mismatch, /mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/04 Oleo.aif is corrupted
BLAKE2 hash key mismatch, /mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/05 It Could Happen To You.aif is corrupted
BLAKE2 hash key mismatch, /mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/06 Woody'n You.aif is corrupted

 

Thanks for looking into this.

[clevoir]

Just for more information, I have now revcieved a notication that Disks 2 & 14 need to be built and exported, from your earlier post I understand that messages shouldn't be sent about build after the intial build has been carried out. However I assume this message was issued as I have turned off the ceration / verification of hash file?

 

Event: Dynamix file integrity daily update

Subject: Notice [TOWER] - Disk 2, Disk 4, Disk 14 needs Build & Export updating

Description: 19 new files without hash value

Importance: normal

[bobbintb]

There are about 29 pages to this. I didn't read them all so sorry if it was already asked. My main reason for going with btrfs was for the hashing protection against data rot, even though I've heard the recovery tools for it aren't as mature as XFS. So how does using this with XFS compare to using btrfs in UnRAID?

[Moussa]

There are about 29 pages to this. I didn't read them all so sorry if it was already asked. My main reason for going with btrfs was for the hashing protection against data rot, even though I've heard the recovery tools for it aren't as mature as XFS. So how does using this with XFS compare to using btrfs in UnRAID?

 

This plugin only detects bit corruption, it can't repair it. You would still need backups.

[EdgarWallace]

The plugin keeps complaining about the files you see below in spite they are all ok. Does is has something to do with the file names?

 

Event: unRAID file corruption
Subject: Notice [TOWER] - bunker verify command
Description: Found 8 files with BLAKE2 hash key corruption
Importance: alert

BLAKE2 hash key mismatch, /mnt/disk6/iTunes/Richard Wagner; Bayreuther Festspiele, Pierre Boulez/Das Rheingold/05 Szene I. Lugt, Schwestern! Die Weckerin lacht in den Grund.aiff is corrupted
BLAKE2 hash key mismatch, /mnt/disk6/iTunes/Richard Wagner; Bayreuther Festspiele, Pierre Boulez/Das Rheingold/07 Szene I. Orchesterzwischenspiel.aiff is corrupted
BLAKE2 hash key mismatch, /mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/01 If I Were A Bell.aif is corrupted
BLAKE2 hash key mismatch, /mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/02 You're My Everything.aif is corrupted
BLAKE2 hash key mismatch, /mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/03 I Could Write A Book.aif is corrupted
BLAKE2 hash key mismatch, /mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/04 Oleo.aif is corrupted
BLAKE2 hash key mismatch, /mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/05 It Could Happen To You.aif is corrupted
BLAKE2 hash key mismatch, /mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/06 Woody'n You.aif is corrupted

 

Thanks for looking into this.

Any idea what is causing this?

[BRiT]

The plugin keeps complaining about the files you see below in spite they are all ok. Does is has something to do with the file names?

 

Event: unRAID file corruption
Subject: Notice [TOWER] - bunker verify command
Description: Found 8 files with BLAKE2 hash key corruption
Importance: alert

BLAKE2 hash key mismatch, /mnt/disk6/iTunes/Richard Wagner; Bayreuther Festspiele, Pierre Boulez/Das Rheingold/05 Szene I. Lugt, Schwestern! Die Weckerin lacht in den Grund.aiff is corrupted
BLAKE2 hash key mismatch, /mnt/disk6/iTunes/Richard Wagner; Bayreuther Festspiele, Pierre Boulez/Das Rheingold/07 Szene I. Orchesterzwischenspiel.aiff is corrupted
BLAKE2 hash key mismatch, /mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/01 If I Were A Bell.aif is corrupted
BLAKE2 hash key mismatch, /mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/02 You're My Everything.aif is corrupted
BLAKE2 hash key mismatch, /mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/03 I Could Write A Book.aif is corrupted
BLAKE2 hash key mismatch, /mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/04 Oleo.aif is corrupted
BLAKE2 hash key mismatch, /mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/05 It Could Happen To You.aif is corrupted
BLAKE2 hash key mismatch, /mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/06 Woody'n You.aif is corrupted

 

Thanks for looking into this.

Any idea what is causing this?

 

If i had to guess I'd say its special characters in the directory filenames, the apostrophe and the semicolon causing script issues.

[bonienl]

The plugin keeps complaining about the files you see below in spite they are all ok. Does is has something to do with the file names?

 

Event: unRAID file corruption
Subject: Notice [TOWER] - bunker verify command
Description: Found 8 files with BLAKE2 hash key corruption
Importance: alert

BLAKE2 hash key mismatch, /mnt/disk6/iTunes/Richard Wagner; Bayreuther Festspiele, Pierre Boulez/Das Rheingold/05 Szene I. Lugt, Schwestern! Die Weckerin lacht in den Grund.aiff is corrupted
BLAKE2 hash key mismatch, /mnt/disk6/iTunes/Richard Wagner; Bayreuther Festspiele, Pierre Boulez/Das Rheingold/07 Szene I. Orchesterzwischenspiel.aiff is corrupted
BLAKE2 hash key mismatch, /mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/01 If I Were A Bell.aif is corrupted
BLAKE2 hash key mismatch, /mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/02 You're My Everything.aif is corrupted
BLAKE2 hash key mismatch, /mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/03 I Could Write A Book.aif is corrupted
BLAKE2 hash key mismatch, /mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/04 Oleo.aif is corrupted
BLAKE2 hash key mismatch, /mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/05 It Could Happen To You.aif is corrupted
BLAKE2 hash key mismatch, /mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/06 Woody'n You.aif is corrupted

 

Thanks for looking into this.

Any idea what is causing this?

 

Can you show the output of:

getfattr -R -d "/mnt/disk6/iTunes/Richard Wagner; Bayreuther Festspiele, Pierre Boulez/Das Rheingold"
getfattr -R -d "/mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet"

All files should have 4 entries, as in the example below

# file: mnt/disk1/photos/Richard Wagner; Bayreuther Festspiele, Pierre Boulez/Das Rheingold/05 Szene I. Lugt, Schwestern! Die Weckerin lacht in den Grund.aiff
user.blake2="786a02f742015903c6c6fd852552d272912f4740e15847618a86e217f71f5419d25e1031afee585313896444934eb04b903a685b1448b755d56f701afe9be2ce"
user.filedate="1459949087"
user.filesize="0"
user.scandate="1459950914"

 

[EdgarWallace]

Thanks bonienl. Here is the output (I moved the Wagner stuff but left Miles Davis where he was :-))

 

root@Tower:~# getfattr -R -d "/mnt/disk6/iTunes/Richard Wagner/Das Rheingold/05 Szene I. Lugt, Schwestern! Die Weckerin lacht in den Grund.aiff"
getfattr: Removing leading '/' from absolute path names
# file: mnt/disk6/iTunes/Richard Wagner/Das Rheingold/05 Szene I. Lugt, Schwestern! Die Weckerin lacht in den Grund.aiff
user.blake2="17ad470f522b40999f7dece7199cf854cdb039cffd182bd754fd1233f6785f640d305fda5ae2fb07582089d9cd94dbe110c7a2127d8f6db950fa87eba266fed5"
user.filedate="1454754812"
user.filesize="64525942"
user.scandate="1459789201"

root@Tower:~# getfattr -R -d "/mnt/disk6/iTunes/Richard Wagner/Das Rheingold/07 Szene I. Orchesterzwischenspiel.aiff"
getfattr: Removing leading '/' from absolute path names
# file: mnt/disk6/iTunes/Richard Wagner/Das Rheingold/07 Szene I. Orchesterzwischenspiel.aiff
user.blake2="cb312ef20185a9f5c44d27ccfe1dc9bd11a9f7545218ba15e854db7aa918f19b22185b4f93fafd54db31951944b626d8cf0d5e8bf43be39fce5d13dc5f6791da"
user.filedate="1454754816"
user.filesize="30892288"
user.scandate="1459789201"

root@Tower:~# getfattr -R -d "/mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/01 If I Were A Bell.aif"
getfattr: Removing leading '/' from absolute path names
# file: mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/01 If I Were A Bell.aif
user.blake2="caf7b1382282519b061a47253f80686ee8ca5249e2bcff8425650ee6c0130d89b626464266d7afb0785d58b91fc0312b7141f2fb42b80b83a18d547e42b4066a"
user.filedate="1457559927"
user.filesize="88088986"
user.scandate="1459789201"

root@Tower:~# getfattr -R -d "/mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/02 You're My Everything.aif"
getfattr: Removing leading '/' from absolute path names
# file: mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/02 You're My Everything.aif
user.blake2="2a97a0553dd5fc0a45dd8abae7faa72385b7d58052ced4b2ef3bfdecf5a7557659e486fdfb854d20bc293dd347475882dc20d62ab0dd71e4457c15295f6e70b9"
user.filedate="1457559929"
user.filesize="56847394"
user.scandate="1459789201"

root@Tower:~# getfattr -R -d "/mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/03 I Could Write A Book.aif"
getfattr: Removing leading '/' from absolute path names
# file: mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/03 I Could Write A Book.aif
user.blake2="aac664b60cde279b5064510ee570f420d9c986ca2e6ef9e6076c05e150c0deb979b6334b7a324ea66c391f46a6c3c5e9a1e274eea7f25dfa06cec5dde39f5870"
user.filedate="1457559931"
user.filesize="55255084"
user.scandate="1459789201"

root@Tower:~# getfattr -R -d "/mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/04 Oleo.aif"
getfattr: Removing leading '/' from absolute path names
# file: mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/04 Oleo.aif
user.blake2="5cc639c98986a63270ec2c56a48c8a4ff8daef3db01d065ef1887388da6c314cf4861e1f68522e61fc6baefdaf4e50ebf039856fe242c440f251837e5f8092d5"
user.filedate="1457559933"
user.filesize="62859070"
user.scandate="1459789201"

root@Tower:~# getfattr -R -d "/mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/05 It Could Happen To You.aif"
getfattr: Removing leading '/' from absolute path names
# file: mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/05 It Could Happen To You.aif
user.blake2="2a196043f37a46ae5596b8e9643ca9838a8ee875a4824f7dccdce6d40af4a511700e36ff33e090f0c9e6acc2e32726aa1a91a24698a7f3c75cc1141dde957a18"
user.filedate="1457559935"
user.filesize="70931164"
user.scandate="1459789201"

root@Tower:~# getfattr -R -d "/mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/06 Woody'n You.aif"
getfattr: Removing leading '/' from absolute path names
# file: mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet/06 Woody'n You.aif
user.blake2="6228cbba3e158e557eab8058b1b496c419f91aecea24c3b6bd1e02b6e10435d88c6802fd8d1047f49057e91133a44c783557923d858e66dc6d887bea1fea2f24"
user.filedate="1457559937"
user.filesize="53542808"
user.scandate="1459789201"

[bonienl]

That all looks alright. The program believes the stored hash value in the extended attributes is not the same as the calculated hash value, while the file size and modification time haven't changed.

 

You can force a update of the extended attributes by doing a manual command from CLI:

/usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -ub2 "/mnt/disk6/iTunes/Richard Wagner; Bayreuther Festspiele, Pierre Boulez/Das Rheingold"
/usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -ub2 "/mnt/disk6/iTunes/The Miles Davis Quintet/Relaxin' with the Miles Davis Quintet"

And compare afterwards the hash values using the 'getfattr' command again.

[EdgarWallace]

This was reported for Wagner:

Finished - verified 38 files, skipped 0 files. Found: 38 mismatches (updated), 0 corruptions. Duration: 00:00:35. Average speed: 44.2 MB/s

 

...and this for Miles Davis:

Finished - verified 9 files, skipped 0 files. Found: 0 mismatches, 6 corruptions (updated). Duration: 00:00:03. Average speed: 129 MB/s

 

Here is a compare of the same file before and after using bunker -ub2:

root@Tower:~# getfattr -R -d "/mnt/disk6/iTunes/Richard Wagner/Das Rheingold/07 Szene I. Orchesterzwischenspiel.aiff"
# file: mnt/disk6/iTunes/Richard Wagner/Das Rheingold/07 Szene I. Orchesterzwischenspiel.aiff
user.blake2="cb312ef20185a9f5c44d27ccfe1dc9bd11a9f7545218ba15e854db7aa918f19b22185b4f93fafd54db31951944b626d8cf0d5e8bf43be39fce5d13dc5f6791da"
user.filedate="1454754816"
user.filesize="30892288"
user.scandate="1459789201"

 

root@Tower:~# getfattr -R -d "/mnt/disk6/iTunes/Richard Wagner/Das Rheingold/07 Szene I. Orchesterzwischenspiel.aiff"
# file: mnt/disk6/iTunes/Richard Wagner/Das Rheingold/07 Szene I. Orchesterzwischenspiel.aiff
user.blake2="8e75ad9d6cf8b0e1d6804f9cedcad7cb54f76f8484de736d0afbdb1db639f2d6398876367af9c9794e2acf8e206d31ea4166d5cd5653dba810d52f7330e0f665"
user.filedate="1459845249"
user.filesize="30892288"
user.scandate="1459955608"

[clevoir]

bonienl have you had a chance to investigate the issue which I highlighted, I see that an updated version is now availible?

[bobbintb]

There are about 29 pages to this. I didn't read them all so sorry if it was already asked. My main reason for going with btrfs was for the hashing protection against data rot, even though I've heard the recovery tools for it aren't as mature as XFS. So how does using this with XFS compare to using btrfs in UnRAID?

 

This plugin only detects bit corruption, it can't repair it. You would still need backups.

 

Apart from not being able to repair it, how does it compare? Can btrfs export file hashes to a file? Is the blake2 hashing in this plugin faster or more efficient than the btrfs hashing? Would there be any issue or reason to use this plugin with btrfs?

 

edit: also, I just found this:

https://lime-technology.com/forum/index.php?topic=44400.msg424508#msg424508

So, unless this has changed, it seems like UnRAID currently cannot repair bitrot anyway. Is that correct?

[EdgarWallace]

bonienl, I received another ~190 corruption messages, e.g.

BLAKE2 hash key mismatch, /mnt/disk3/Kino/Mighty Heart, A (2007).nfo is corrupted
BLAKE2 hash key mismatch, /mnt/disk3/Kino/Se7en (1995).nfo is corrupted
BLAKE2 hash key mismatch, /mnt/disk3/Kino/Panic Room (2002).nfo is corrupted
BLAKE2 hash key mismatch, /mnt/disk3/Kino/Silence of the Lambs, The (1991).nfo is corrupted
BLAKE2 hash key mismatch, /mnt/disk3/Kino/Oculus (2013).nfo is corrupted
BLAKE2 hash key mismatch, /mnt/disk3/Kino/Oblivion (2013).nfo is corrupted
BLAKE2 hash key mismatch, /mnt/disk3/Kino/Unbreakable (2000).nfo is corrupted

 

Emby docker was modifying almost all info files so I would rather have expected messages like these:

BLAKE2 hash key mismatch (updated), /mnt/disk3/Kino/Savages (2012).nfo was modified
BLAKE2 hash key mismatch (updated), /mnt/disk3/Kino/Prometheus (2012).nfo was modified
BLAKE2 hash key mismatch (updated), /mnt/disk3/Kino/Resident Evil (2002).nfo was modified
BLAKE2 hash key mismatch (updated), /mnt/disk3/Kino/Stereo (2014).nfo was modified

 

It is showing the same pattern as yesterday. Before having forced the update:

root@Tower:/# getfattr -R -d "/mnt/disk3/Kino/Mighty Heart, A (2007).nfo"
# file: mnt/disk3/Kino/Mighty Heart, A (2007).nfo
user.blake2="e51e01038906c6a80e8f6cc6a13b7b7399f3128fc00e778ab97eb5dba0d74251d0d745d56c2ceff70c84d216b0b28292e2cf075bbb6c51237bb2b204ef8a9147"
user.filedate="1459962347"
user.filesize="9841"
user.scandate="1459964230"

 

After the update:

root@Tower:/# /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -ub2 "/mnt/disk3/Kino"
Finished - verified 6796 files, skipped 0 files. Found: 0 mismatches, 0 corruptions. Duration: 02:23:13. Average speed: 47.9 MB/s
root@Tower:/# getfattr -R -d "/mnt/disk3/Kino/Mighty Heart, A (2007).nfo"
# file: mnt/disk3/Kino/Mighty Heart, A (2007).nfo
user.blake2="e51e01038906c6a80e8f6cc6a13b7b7399f3128fc00e778ab97eb5dba0d74251d0d745d56c2ceff70c84d216b0b28292e2cf075bbb6c51237bb2b204ef8a9147"
user.filedate="1459962347"
user.filesize="9841"
user.scandate="1460020802"

 

Would it be save to force updates for all these incidents (/usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -ub2)? Your advise is highly appreciated. Thanks a lot.

[bonienl]

Would it be save to force updates for all these incidents (/usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -ub2)? Your advise is highly appreciated. Thanks a lot.

 

The script declares a file corrupted when its hash key has changed but the file size and file modification date remain the same. I am not sure how Emby makes modifications, but it seems conflicting with the rule.

 

I would suggest to make an exception rule for *.nfo files and clear their extended attributes.

 

[bonienl]

Just for more information, I have now revcieved a notication that Disks 2 & 14 need to be built and exported, from your earlier post I understand that messages shouldn't be sent about build after the intial build has been carried out. However I assume this message was issued as I have turned off the ceration / verification of hash file?

 

I did some testing to analyze the behavior. There is a limit in the number of files which can be processed at a given time. If for example you copy thousands of small files to your unRAID system, chances are big that not all files get a calculated hash value attached. If on the other hand you copy a few large files to your system, then all files will get their hash value attached.

 

My advice is to exclude those folders which change regularly. If something goes wrong, it will probably be detected in the copying process already. The virtue of the plugin is to examine those files which are rarely touched, and hence may get silent corruption.

 

You can of course attach a hash value to all files and use it for verification purposes, but you have to keep in mind the risk of not keeping up-to-date when too many files change at the same time.

 

[EdgarWallace]

Thanks for the advise bonienl. I removed the attributes by using:

/usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -r /mnt/user/Movies ! *.nfo

 

and excluded *.nfo's in Settings afterwards. I might have overlooked it but can you explain, what's excluded by using the Apple metadata switch?

 

Thanks for the great plugin!!!

[clevoir]

Thanks for your testing and reply

 

I have 28.5TB of files spread over 14 disks, 99% of these are static files, with new files being added at small intervals e.g. new music / films & TV series.

 

The first error was caused by me adding 128 small MP3 files, and I think the second bythe plugin doing something in the background

 

The plugin really does what I want, which is to spot silent corruption in the background and I am very keen to pursue it's use

 

Would a solution be to carry out the initial build one disk at a time, leaving a suitable period, perhaps a day in between disks to ensure that all files are hashed?   

[RobJ]

The second problem, as below occured due to ign files that do not appear to exist, which I am not sure are connected to the use of the plugin.

 

stat: cannot stat '  /mnt/disk2/Hi Def 2/The Big Bang Theory/Series 6/mymovies.ing : No such file or directory

/bin/md5sum: ' /mnt/disk2/Hi Def 2 /The Big Bang Theory/Series 6/mymovies.ign' : No such file or directory

 

Please excuse my OCD-like observations.  When you first posted the lines above, there were so many typos, I didn't know what to say.  I realize typos don't matter to some, but as a programmer, I can't help noticing them, because they are critical.  So I can't tell if they were introduced by hand typing the error messages, or are actually correct quotes.  If so, they do explain the error messages.  I'll just list them, you can decide if they're an issue -

 

* stat: cannot stat '  /mnt/disk2/Hi Def 2/The Big Bang Theory/Series 6/mymovies.ing : No such file or directory

- 2 spaces after the leading apostrophe, can't be correct

- file extension is .ing, is that correct?

 

* /bin/md5sum: ' /mnt/disk2/Hi Def 2 /The Big Bang Theory/Series 6/mymovies.ign' : No such file or directory

- 1 space after the leading apostrophe, can't be correct

- has "Hi Def 2 " (extra space after the 2) instead of "Hi Def 2", both can't be correct

- file extension is .ign, should it be .ign or .ing?

 

Sorry, some of us are just annoying purists.  But without accurate info, it's hard to help.

[clevoir]

Thanks I understand your point

 

I was copying from a monitor attached to the array, it appears that I made some typo's

[bonienl]

The latest version could easily overcommit the processor when many files are copied/created. A new version 2016.04.08 is available to overcome this issue. All users are encouraged to update to this version.

 

Please re-apply settings to make the changes effective after the update.

 

[firetime]

you don't say: PIC

 

Good timing on the update as I came here to ask for assistance with this. The mover just grabbed a whole pile of rar part files and everything got real slow.