Dynamix File Integrity plugin

[t3]

sry if this was already asked (haven't seen it though)... i guess these extra attributes are lost in most cases when files are copied/moved around/backed up, by/from whatever tool and/or source, right?

 

for example, if a file is copied from a client machine, which even may put it into the cache (if used, so then mover later places it into the fnial location)... is the hash kept or rebuilt? i know that in this case for example squid's checksum tool will also "fail", but imo every time a file that wasn't altered is just re-hashed only because it is copied or moved, any attempt to provide accurate integrity info is broken (at least if the underlying file system does not guarantee that the destination file 100% matches the input).

 

maybe you can give a short breakdown under which circumstances the hash/integrity info is guaranteed to stay with the file (as i guess this would be the shorter list in comparison with the cases where it gets lost)...

[bonienl]

It depends on the command given to copy or move files. Some commands have an option to include the extended attributes in the execution process, like rsync or cp, but this option needs to be explicitely given otherwise extended attributes are lost.

 

When a file is copied or moved to a data disk and it doesn't have the extended attributes then a (new) checksum value is calculated and stored. If it does have extended attributes then a verification is done to detect corruption.

 

[Squid]

It depends on the command given to copy or move files. Some commands have an option to include the extended attributes in the execution process, like rsync or cp, but this option needs to be explicitely given otherwise extended attributes are lost.

 

When a file is copied or moved to a data disk and it doesn't have the extended attributes then a (new) checksum value is calculated and stored. If it does have extended attributes then a verification is done to detect corruption.

What about copying between shares over the network

[bonienl]

It depends on the command given to copy or move files. Some commands have an option to include the extended attributes in the execution process, like rsync or cp, but this option needs to be explicitely given otherwise extended attributes are lost.

 

When a file is copied or moved to a data disk and it doesn't have the extended attributes then a (new) checksum value is calculated and stored. If it does have extended attributes then a verification is done to detect corruption.

What about copying between shares over the network

 

Again will depend on the command / utility used to do the copying. If you are doing this from a Windows machine then extended attributes get lost, but will be recalculated once the copy is finished. This assumes that files are copied between physical disks. When source and destination disk are the same, chances are that the extended attributes remain untouched (haven't tested that though).

 

In case of doubt one could compare the new checksum value in the extended attributes against the existing checksum value in the export file, but this is a manual action (=CLI).

 

[t3]

In case of doubt one could compare the new checksum value in the extended attributes against the existing checksum value in the export file, but this is a manual action (=CLI).

well, there will always be doubt

... since no operation, even done internally in the box, can guarantee that input and output is identical; my concern for example would be what happens if a sync from the main box to the failover/backup box. of course there is only a 0.000000001% chance for corruption to happen during that process, but when handling terabytes of data this must be a concern. a "boring" extra hash file per folder has actually some advantages in e.g. this case;

 

 

i understand that any form of (ideally end-to-end) integrity checking wihtout proper support from underlying file systems and transports is more or less impossible to implement, so, anyway, thanks for the effort with this - will definitely look into possible use cases...

[unevent]

The essence of this plugin is to use the extended attributes, which allows to store the checksum information together with the file instead of storing it in separate files.

 

I've put out the warning so that people are aware of this potential instability when ReiserFS is used as file system. Personally I moved over to XFS a long time ago and find this to work absolutely stable.

 

Sorry par2 is not considered, the intention of this plugin is to detect only.

 

Thanks for the info.  I subscribe to the philosophy of it ain't broke don't fix it...to a degree.  Existing drives for me will remain reiserfs while new will be XFS.  Thanks again.

[archedraft]

 

Thanks for the info.  I subscribe to the philosophy of it ain't broke don't fix it...to a degree.  Existing drives for me will remain reiserfs while new will be XFS.  Thanks again.

But what's the fun in that?? I've always preferred the, If it ain't broke then how can we make it better, approach. Much more interesting

[bonienl]

well, there will always be doubt

... since no operation, even done internally in the box, can guarantee that input and output is identical; my concern for example would be what happens if a sync from the main box to the failover/backup box. of course there is only a 0.000000001% chance for corruption to happen during that process, but when handling terabytes of data this must be a concern. a "boring" extra hash file per folder has actually some advantages in e.g. this case;

 

You need to use the right tools.

 

When transferring data and you want to ensure the transfer is correct then use a utility which can verify the transfer afterwards. This is already the current situation and best way to verify transfers.

 

The main purpose of this plugin is to detect silent file corruption. Check that files which haven't touched or moved do not deteriate over time.

 

[bonienl]

Thanks for the info.  I subscribe to the philosophy of it ain't broke don't fix it...to a degree.  Existing drives for me will remain reiserfs while new will be XFS.  Thanks again.

 

There are a number of known issues with RFS but it depends on the situation whether it affects the set up or not. Nothing obliges you to change, and if you are happy with the current set up, just let it as is.

 

F.I.P allows selective disk selection. If you do have a mix of RFS and XFS formatted disks then select only the latter ones to participate in the checksum build and verification.

 

[danioj]

I have noticed something in the last couple of days. My Parity Drive seems to be constantly spun up. The only thing I have done with the Server in those few days is enable this plugin. I did a scan and also enabled the option "Automatically protect new and modified files:" using BLAKE.

 

No other drives are spun up and IF it is the plugin it doesn't seem to be reading or writing to the drive either. Counts on both (as recorded by unRAID) are not increasing - at least not that I can see anyway!!

 

Is there any reason that this plugin would keep the Parity drive spun up - perhaps monitoring - or should I be looking elsewhere?

[bonienl]

I have noticed something in the last couple of days. My Parity Drive seems to be constantly spun up. The only thing I have done with the Server in those few days is enable this plugin. I did a scan and also enabled the option "Automatically protect new and modified files:" using BLAKE.

 

No other drives are spun up and IF it is the plugin it doesn't seem to be reading or writing to the drive either. Counts on both (as recorded by unRAID) are not increasing - at least not that I can see anyway!!

 

Is there any reason that this plugin would keep the Parity drive spun up - perhaps monitoring - or should I be looking elsewhere?

 

The "automatic protect" is passive monitoring and should not spin up your parity disk.

 

The verification tasks may spin up the parity disk together with the disk(s) which is/are examined at that time.

 

Looking at my own system, all spins down as expected.

 

[danioj]

I have noticed something in the last couple of days. My Parity Drive seems to be constantly spun up. The only thing I have done with the Server in those few days is enable this plugin. I did a scan and also enabled the option "Automatically protect new and modified files:" using BLAKE.

 

No other drives are spun up and IF it is the plugin it doesn't seem to be reading or writing to the drive either. Counts on both (as recorded by unRAID) are not increasing - at least not that I can see anyway!!

 

Is there any reason that this plugin would keep the Parity drive spun up - perhaps monitoring - or should I be looking elsewhere?

 

The "automatic protect" is passive monitoring and should not spin up your parity disk.

 

The verification tasks may spin up the parity disk together with the disk(s) which is/are examined at that time.

 

Looking at my own system, all spins down as expected.

 

 

Thanks for the quick reply. I manually spun down the Parity disk and left it for 20 mins. Seems nothing happened to spin it back up again which makes me feel better.

 

Perhaps it was a freak happening / bug / some other issue that I have not yet found.

 

Either way, as you suggested it doesn't appear it is this plugin causing it! 

[John_M]

Check your parity drive's settings (Main|Parity) and make sure that the Spin down delay hasn't inadvertently been set to Never. I had a data drive that would never spin down and I discovered this setting had somehow been changed from Default. It may not be the case for you but it's an easy thing to check.

[danioj]

Check your parity drive's settings (Main|Parity) and make sure that the Spin down delay hasn't inadvertently been set to Never. I had a data drive that would never spin down and I discovered this setting had somehow been changed from Default. It may not be the case for you but it's an easy thing to check.

 

Dude - I know this is slightly off topic BUT that was a great catch. You were right. For WHATEVER reason it was set to Never. I have no idea HOW that happened, I guess it must have been me who did it at some point recently but I don't remember. Unless a few wines happened while I did it - who knows.

 

Thank you very much! Set back to Default like the other drives.

 

Anyway - clearly the issue had nothing to do with this plugin. Case closed.

[danioj]

I have just updated to v2016.01.06a.

 

With reference to the new feature "added initial build and initial export status of each disk on control page"

 

When I installed this a couple of days ago I did a build. Under the "File Integrity and Control Tab" I now have orange circles against each disk for "Initial Build" and "Initial Export" (as attached). I have no issue with these orange circles identifying a positive (although I have never seen this symbol before for a positive confirmation) but any chance you can release a key for the possible status symbols?

 

Thank you in advance!

[JorgeB]

I have just updated to v2016.01.06a.

 

With reference to the new feature "added initial build and initial export status of each disk on control page"

 

When I installed this a couple of days ago I did a build. Under the "File Integrity and Control Tab" I now have orange circles against each disk for "Initial Build" and "Initial Export" (as attached). I have no issue with these orange circles identifying a positive (although I have never seen this symbol before for a positive confirmation) but any chance you can release a key for the possible status symbols?

 

Thank you in advance!

The confirmation symbol is a green check mark, since you updated after initial check you need to press build & export all disks to get it,  should take a few seconds.

[danioj]

I have just updated to v2016.01.06a.

 

With reference to the new feature "added initial build and initial export status of each disk on control page"

 

When I installed this a couple of days ago I did a build. Under the "File Integrity and Control Tab" I now have orange circles against each disk for "Initial Build" and "Initial Export" (as attached). I have no issue with these orange circles identifying a positive (although I have never seen this symbol before for a positive confirmation) but any chance you can release a key for the possible status symbols?

 

Thank you in advance!

The confirmation symbol is a green check mark, since you updated after initial check you need to press build & export all disks to get it,  should take a few seconds.

 

Perfect. Thank you.

[bonienl]

Another update of Dynamix File Intregity plugin is available..

 

In this version there is a daily verification of the build and export status, and in the GUI you can see if everything is up-to-date. A manual action is required to update the specific item.

 

[taalas]

Regarding ReiserFS:

 

Is this a hard limitation or is it possible for this to be made stable at some point via a workaround/fix?

[bonienl]

Regarding ReiserFS:

 

Is this a hard limitation or is it possible for this to be made stable at some point via a workaround/fix?

 

It doesn't look like there is active development on RFS in the Linux kernel and I am afraid the situation will stay as is.

[landS]

Bonienl, I believe you said new files checksum is created when the file is written to the system.  Given a Cache of BTRS and array disks of XFS, where all new files are written the the Cache as able ---  does this plugin ignore the cache, create a checksum on both the cache and once moved the data disk, or does it just retain the checksum of the BTRS even after it is moved to the XFS disk?

 

Thanks

[bonienl]

Bonienl, I believe you said new files checksum is created when the file is written to the system.  Given a Cache of BTRS and array disks of XFS, where all new files are written the the Cache as able ---  does this plugin ignore the cache, create a checksum on both the cache and once moved the data disk, or does it just retain the checksum of the BTRS even after it is moved to the XFS disk?

 

Thanks

 

The cache disk/pool is not included in the checksumming. Only when a file is moved/copied to a data disk a calculation takes place for the destination.

[landS]

Perfect!

[karateo]

Hi!

I am trying to find out how to include only some folders (for example /mnt/user/Photos) but I am missing that option.

I could exclude all others but for example I have docker.img in /mnt/disk1/ so I can not exclude it and it keeps saying

 

Jan 12 16:33:10 TOWER bunker: warning: BLAKE2 hash key mismatch (updated), /mnt/disk1/docker.img was modified

 

Is there a settings combination I am missing?

[bonienl]

Hi!

I am trying to find out how to include only some folders (for example /mnt/user/Photos) but I am missing that option.

I could exclude all others but for example I have docker.img in /mnt/disk1/ so I can not exclude it and it keeps saying

 

Jan 12 16:33:10 TOWER bunker: warning: BLAKE2 hash key mismatch (updated), /mnt/disk1/docker.img was modified

 

Is there a settings combination I am missing?

 

The underlying utilities only work with exclude lists, the GUI follows the same approach. Can't really change that.

 

docker.img isn't a folder and is not selectable. I presume you don't have a cache drive, which usually holds the docker image.

 

I need to add a solution which will ignore the docker image if it is located on a data disk instead of the cache disk.

 

As a workaround for now, you may want to consider to move the docker file into a folder, e.g. /mnt/disk1/docker/docker.img and exclude the docker folder.