So, the plugin complain about file corruption in the preview folder of Nextcloud. As I have told I should probably just exclude the Nextcloud appdata folder, I have tried to do that. I have added it to the custom folder field like this : /mnt/user/NCloud/appdata_[Some Weird Letter and number combination] because thats the Nextcloud appdata and location is called. I then have tried both clicking remove and export and remove and build, but it still complains about the same corruption inside the appdata preview folder. 

What do I have to do different to get it to work?

On 6/30/2019 at 9:02 AM, Mihle said:

So, the plugin complain about file corruption in the preview folder of Nextcloud. As I have told I should probably just exclude the Nextcloud appdata folder, I have tried to do that. I have added it to the custom folder field like this : /mnt/user/NCloud/appdata_[Some Weird Letter and number combination] because thats the Nextcloud appdata and location is called. I then have tried both clicking remove and export and remove and build, but it still complains about the same corruption inside the appdata preview folder. 

What do I have to do different to get it to work?

I thought this place was more active than this, but ok.

On 6/30/2019 at 3:02 AM, Mihle said:

So, the plugin complain about file corruption in the preview folder of Nextcloud. As I have told I should probably just exclude the Nextcloud appdata folder, I have tried to do that. I have added it to the custom folder field like this : /mnt/user/NCloud/appdata_[Some Weird Letter and number combination] because thats the Nextcloud appdata and location is called. I then have tried both clicking remove and export and remove and build, but it still complains about the same corruption inside the appdata preview folder. 

What do I have to do different to get it to work?

Are you absolutely sure you have that path correct? That is a nonstandard path for docker appdata. Usually it would be something like /mnt/user/appdata/dockername

19 hours ago, trurl said:

Are you absolutely sure you have that path correct? That is a nonstandard path for docker appdata. Usually it would be something like /mnt/user/appdata/dockername

Yes, Nextcloud seem to have two appdata folders. The other one, the one is already excluded, via just excluding the appdata forlder. but putting in my own location does not seem to work as I mentioned so I may do something wrong?

On 6/10/2019 at 10:42 AM, shiarua said:

Well, looking through the source, I guess I could just make some wrapper scripts around the bunker script for exporting hash files and running the verification checks. Seems like a feasible workaround. 

Do you have exporting to custom location working? if so, would you post the scripts you made?

 

I can't have filenames exported to the unencrypted USB drive because the actual filenames might include sensitive information.

Believe it or not, I read this whole thread. I think I even maintained some of the information I read.

 

Some one asked for only manual/scheduled hash creation, rather than using inotify to do the hashes when files are created. Was that feature ever added? I'd rather just swoop through nightly and do them then during the day when the server is busy doing other things. Plus, inotify-tools has issues where it can miss adding watches or even perform erroneous watches when files are moved. I think that might explain the issues some people have had while I was reading this thread.

I was wondering if it is worth changing the text on the FIND button to read DUPLICATES instead?   Although I have had the plugin installed for ages it was only recently that I realized this button was about detecting duplicate files on the server. 

 

Since it works off the hash files generated by the plugin it is also very fast once you have the hash files generated.   Knowing this capability exists might be a reason that encourages more users to make use of the plugin

Do you need to export the hashes to run a check? If not, does it just read each file and compare it to the extended metadata checksum?

I'm having an issue that since I installed the plugin again like 1-2 months ago. I had used this plugin a long time ago and decided to install it again. It has consistently shows 5 files that are corrupted ever since I reinstalled it. Not sure if these are because there were some files left behind from the previous install of this plugin. 4 of them are just NFO's and one is an MKV. Is there a way to acknowledge the error so it stops reporting it or what is the best way to handle it.

8 hours ago, DDock said:

I'm having an issue that since I installed the plugin again like 1-2 months ago. I had used this plugin a long time ago and decided to install it again. It has consistently shows 5 files that are corrupted ever since I reinstalled it. Not sure if these are because there were some files left behind from the previous install of this plugin. 4 of them are just NFO's and one is an MKV. Is there a way to acknowledge the error so it stops reporting it or what is the best way to handle it.

 

That means the hash result not match the one which store in extend attributes, you can simple delete from attributes and then build and export again.

 

 

 

16 hours ago, Benson said:

 

That means the hash result not match the one which store in extend attributes, you can simple delete from attributes and then build and export again.

 

 

 

Just to make sure I'm running the proper command I'm just deleting the attribute user.sha256 and not the other user.xxx attributes?

 

Thank You for sending me down the correct path.

1 hour ago, DDock said:

Just to make sure I'm running the proper command I'm just deleting the attribute user.sha256 and not the other user.xxx attributes?

 

Thank You for sending me down the correct path.

 

Delete some may be ok, but just delete all.

 

i.e.

user.filedate="1547312879"

user.filesize="85302"
user.md5="a7184f925c3bc9eb2f8a917f7f26f9c8"
user.scandate="1565683848"

Edited August 13, 20197 yr by Benson

I've been trying to set up file integrity to get it to work but I'm not sure if I'm doing this correctly or not. I keep on getting BLAKE2 key mismatch on my libvirt.img and a lot of my nextcloud files: See example:

BLAKE2 hash key mismatch (updated), /mnt/disk1/nextcloud/.htaccess was modified
BLAKE2 hash key mismatch (updated), /mnt/disk1/nextcloud/appdata_ocdoy1vwt49l/appstore/apps.json was modified
BLAKE2 hash key mismatch (updated), /mnt/disk1/nextcloud/appdata_ocdoy1vwt49l/appstore/categories.json was modified
BLAKE2 hash key mismatch (updated), /mnt/disk1/nextcloud/appdata_ocdoy1vwt49l/appstore/future-apps.json was modified
BLAKE2 hash key mismatch (updated), /mnt/disk5/vm backup/libvirt.img was modified

Are these true errors?? I have these excluded and clearing it but every time integrity check runs it always finds these files mismatched.

 

Here are my settings:

35 minutes ago, bobokun said:

I've been trying to set up file integrity to get it to work but I'm not sure if I'm doing this correctly or not. I keep on getting BLAKE2 key mismatch on my libvirt.img and a lot of my nextcloud files

Yeah, don't do that. The whole point of this plugin is to keep tabs on seldom used archival type files, files that should never change. Active files should not even be monitored, it's a waste of resources.

19 minutes ago, jonathanm said:

Yeah, don't do that. The whole point of this plugin is to keep tabs on seldom used archival type files, files that should never change. Active files should not even be monitored, it's a waste of resources.

I understand and that's why I've added them on my exclude folders (as displayed on the screenshot) but it's still being monitored? I must be configuring something wrong but I'm not sure how I should fix it.

1 hour ago, bobokun said:

Are these true errors?? I have these excluded and clearing it but every time integrity check runs it always finds these files mismatched.

You need do clearing again and export, this to reflect the change.

BLAKE2 hash key mismatch, (720p_30fps_H264-192kbit_AAC).mp4 is corrupted

I got this error on a Disk 3, is there any way I can find can out where on disk 3 this file is?

37 minutes ago, Koshy said:

BLAKE2 hash key mismatch, (720p_30fps_H264-192kbit_AAC).mp4 is corrupted

I got this error on a Disk 3, is there any way I can find can out where on disk 3 this file is?

Suppose be Unicode file name display issue. You may check disk3 hash file which store at export directory. Grep it and found which file(s) have similar name then further confirm by running b2sum <file> and compare the hash value.

Edited September 4, 20196 yr by Benson

15 hours ago, Benson said:

Suppose be Unicode file name display issue. You may check disk3 hash file which store at export directory. Grep it and found which file(s) have similar name then further confirm by running b2sum <file> and compare the hash value.

Thank you, I found the file.

On every run the server is providing a wrong error message:

BLAKE2 hash key mismatch, /mnt/disk5/iTunes/Music/Wolfgang Haffner/Kind of Cool/02 So What.aif is corrupted

However the file is perfectly ok. Is there any way to update the hash key manually?

34 minutes ago, EdgarWallace said:

On every run the server is providing a wrong error message:

BLAKE2 hash key mismatch, /mnt/disk5/iTunes/Music/Wolfgang Haffner/Kind of Cool/02 So What.aif is corrupted

However the file is perfectly ok. Is there any way to update the hash key manually?

You can run the CLI command (this updates the files in the specific folder)

/usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -b2 -a "/mnt/disk5/iTunes/Music/Wolfgang Haffner/Kind of Cool"

 

You can run the CLI command (this updates the files in the specific folder)

/usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -b2 -a "/mnt/disk5/iTunes/Music/Wolfgang Haffner/Kind of Cool"

 

Oh great@bonienl. Thanks a lot.


Gesendet von iPad mit Tapatalk

48 minutes ago, EdgarWallace said:

Oh great@bonienl. Thanks a lot.

I made a typo instead of "-a" (add) you should use "-u" (update)

Great @bonienl

Worked very well, thank you very much:

root@Tower:~# /usr/local/emhttp/plugins/dynamix.file.integrity/scripts/bunker -b2 -u "/mnt/disk2/iTunes/Music/Eric Clapton/I Still Do"
Finished - verified 12 files, skipped 0 files. Found: 0 mismatches, 1 corruption (updated). Duration: 00:00:04. Average speed: 149 MB/s

 

I am doing a mass move from my old unraid server over to my new one. I expected some likely file transfer damage, so no surprise when I did a manual check and saw errors. What was a surprise was how... incomplete those errors were.

 

Quote

 

SHA256 hash key mismatch, 1e03.mkv is corrupted

SHA256 hash key mismatch, 20p.AMZN.WEB-DL.mkv is corrupted

SHA256 hash key mismatch, DTV.XviD-NoTV.avi is corrupted

SHA256 hash key mismatch, es s02e06.avi is corrupted

 

 

I saw a few pages back someone else had a similar problem, and the instructions were to go into the hash log for the disk and find candidates, then run a b2sum (path) against each candidate. For some that is easy, but for like that first one and second one, there are a lot of candidates.

 

Is there a less tedious way of configuring this output so I don't need to check 39 false positives to find the one I want manually?

 

Also, that last one only has a single candidate, so I checked it. Those files have individual md5 checksums, one for each file. That directory passes an md5 check on both the source and the allegedly corrupt destination. I don't understand how that can be. I don't mean that I don't believe it, I just don't quite understand it. 

 

Edited October 1, 20196 yr by Ouze