Vr2Io Posted April 27, 2020 Share Posted April 27, 2020 (edited) 1 hour ago, JustinChase said: I couldn't find any "log tab", but there is a "log files" folder on the plugin page. The only thing in that folder is a file called duplicate_file_hashes.txt That only lists all the duplicate hashes it found, but nothing about any errors, or help figuring out which file triggered the error. This is an example in log tab, it show a file "test" at /mnt/disk12 have problem, not in plugin tab (page). And you need enable logging in plugin setting. Edited April 27, 2020 by Benson Quote Link to comment
JustinChase Posted April 27, 2020 Share Posted April 27, 2020 Ah, that helps; thank you! doesn't make as much sense as being in the log folder on the plugin page, but now I know. it actually shows tons of warnings and errors about SHA256 mismatches and/or missing files. I may have been doing some moving of files when the process was originally running, so maybe that's the issue. It took about a day to run, and I'm not sure when I can not use the server for that long to run it again, but I'll try to do that soon, and have it re-evaluate all files during that time and see if errors continue. I just installed a brand new parity drive and system drive, both of which passed the preclear, so I'd be surprised if there was bit rot happening already. I guess we'll see soon enough. thanks again for your help. Quote Link to comment
Vr2Io Posted April 27, 2020 Share Posted April 27, 2020 9 minutes ago, JustinChase said: it actually shows tons of warnings and errors about SHA256 mismatches and/or missing files. That would be normal if file move to other path. You should perform export to update it. Quote Link to comment
JustinChase Posted April 28, 2020 Share Posted April 28, 2020 I deleted all data, and am just rebuiling now, while I use the server, but am not moving files around this time. We'll see how it goes. If any issues, I'll try the export function next time. I read all the descriptions of all the buttons, and there was no obvious 'rebuild' function I could decipher. thanks again for the help. Quote Link to comment
cdixon Posted May 21, 2020 Share Posted May 21, 2020 I recently ran a dynamix file integrity test on a drive that has read errors. According to the log there are over 90 MKV files with a SHA256 hash key mismatch but even though it states that the files are corrupt they play perfectly. I used a hash file manager to rewrite the hash keys for the files but this did not work. How can I fix these files? Quote Link to comment
snuffy47 Posted June 11, 2020 Share Posted June 11, 2020 (edited) nvm Edited June 11, 2020 by snuffy47 Quote Link to comment
kaffesugen Posted July 2, 2020 Share Posted July 2, 2020 Hi! During my build of one harddisk the green checkmark next to "Build up-to-date" appeared, but the build process is still ongoing. Should I click the cancel button? Quote Link to comment
cdixon Posted August 10, 2020 Share Posted August 10, 2020 I ran build and export on one of the discs in my server. During the export process many files were skipped and many corrupt files were found. I am unable to locate any corrupt files because no log was posted after the export process. After this, I ran verification on disc 4. The process ran for 31 hours and no log was posted. How can I locate the corrupt files? Quote Link to comment
Vr2Io Posted August 11, 2020 Share Posted August 11, 2020 (edited) On 5/21/2020 at 8:42 AM, cdixon said: I used a hash file manager to rewrite the hash keys for the files but this did not work. How can I fix these files? I am not sure what means ( or how ) the hash file manager do it. If too much file show corrupt, then best way was "Remove" extended attributes and "Build" it again. 13 hours ago, cdixon said: During the export process many files were skipped Export won't skip file, only Import will have this warning. 13 hours ago, cdixon said: How can I locate the corrupt files? If no logging enable, then you may never know which file have hash mismatch. Pls understand this plugin won't work perfect in some case which already discuss before and plugin should stop develop. You need understand why wrong hash will generate for a file .... if file still in writing / transfer but FIP hash it, then wrong hash value will generate. I would simple suggest never hash file during file transfer, so best stop auto hash function ( manual execute hash or periodic during no any file in transfer ) . Otherwise, there are no meaning always "Remove" and "Build". In my use case, I store large size media file always, those file are write once, so I haven't issue for those file. But something like frequency update file (i.e. subtitle) will auto-update (correct) during hash check. I also ( advance manual operation ) export the hash to backup, every-time "Import" hash for all backup file from master. i.e. 10000+ file in backup, this also can confirm does any file in missing backup too. * PS : I also use checksum plugin, so I have two ref. source for file(s) really corrupt or not. * Edited August 11, 2020 by Benson Quote Link to comment
bonienl Posted August 11, 2020 Author Share Posted August 11, 2020 20 hours ago, cdixon said: How can I locate the corrupt files? See the setting "Error logging" under Settings -> File Integrity Quote Link to comment
cdixon Posted August 11, 2020 Share Posted August 11, 2020 3 minutes ago, bonienl said: See the setting "Error logging" under Settings -> File Integrity That's the problem. In the settings, I have Error logging set to Syslog and Output file but I am still unable to find the corrupt files with this option. Quote Link to comment
nukem492 Posted August 19, 2020 Share Posted August 19, 2020 (edited) @bonienl I ran a manual check on a drive and got a lot of BLAKE2 hash key mismatches (mostly .nfo files modified). I received the details in a notification email, but there's no log file in /boot/config/plugins/dynamix.file.integrity/logs , even though I do have error logging set to "Syslog & Output File". I also have log files and email notifications from 2018/2019, so I know it was working correctly at some point. Are log files not created when the check is run manually (as opposed scheduled daily/weekly/monthly)? I have 14GB free on my flash drive, so it's not an issue of usable space. What do you suggest? Edited August 19, 2020 by nukem492 Quote Link to comment
je82 Posted August 28, 2020 Share Posted August 28, 2020 I am interested in starting using this plugin, but i have some questions: 1. If the hashing reults are NOT saved to flash, where are they stored? 2. Is it possible to store hash results log to another path then flash? I have for example used fstab to statically mount a smb share on a remote logserver which holds all my logs, i would want the logs to go directly there instead of moving them from the flash each day with a script (avoiding writing as much as possible to flash) is this possible or do i need to modify the plugin to do this? Quote Link to comment
danielocdh Posted September 11, 2020 Share Posted September 11, 2020 Are folder exclusions recursive? So for example if I exclude folder _abc, will that exclude /mnt/user/x/_abc/some_file.zip ? If not, how can I achieve this, thanks. Quote Link to comment
dgwharrison Posted September 12, 2020 Share Posted September 12, 2020 Hi, thanks for the plugin, just a minor bug I noticed when running the Find Duplicates command, it says it's checking Disk 8, I've actually removed that disk from the array as it was faulty. Screenshot: Quote Link to comment
Vr2Io Posted September 12, 2020 Share Posted September 12, 2020 (edited) 1 hour ago, dgwharrison said: Hi, thanks for the plugin, just a minor bug I noticed when running the Find Duplicates command, it says it's checking Disk 8, I've actually removed that disk from the array as it was faulty. Screenshot: It sort/check all hash files, it won't check actual disk exist or not. You can simple delete disk8 hash file. Edited September 12, 2020 by Benson Quote Link to comment
dgwharrison Posted September 15, 2020 Share Posted September 15, 2020 Hi, just wondering how I reset the File Integrity plugin? I deleted the hash files however that seems to have procuded unexpected behaviour... (my bad). Quote Link to comment
itimpi Posted September 15, 2020 Share Posted September 15, 2020 Deleting the hash files should have no effect, so what is the unexpected behaviour? Quote Link to comment
dgwharrison Posted September 15, 2020 Share Posted September 15, 2020 (edited) Sorry @itimpi just realised I forgot to export. 🤦♂️ Edited September 15, 2020 by dgwharrison Quote Link to comment
CS01-HS Posted September 23, 2020 Share Posted September 23, 2020 (edited) I searched the thread and didn't find an answer so apologies if this question's been asked. I'm new to this plugin and have my first verification running after the initial check. It's been going for around 8 hours and I'm wondering how far it's progressed. Is there an indicator somewhere? The initial check showed a handy progress bar in the page under Tools. I'm also confused about whether "check" and "verify" are used interchangeably in the documentation. The help says Use the Check command to verify files against a previously exported file but the plugin can be configured to verify with Save new hashing results to flash (which I assume is the referenced "export") disabled, suggesting they're not interchangeable. Edited September 23, 2020 by CS01-HS typo Quote Link to comment
KoNeko Posted October 15, 2020 Share Posted October 15, 2020 got a question i had a few blake2 corruptions now i replaced those files. and did a normal check which says ok. SO i did a check with the plugin again it took 19 hours to do all 45k files. The check says finish and it has less corruptions now but when i check the disk2.export.20201011.bad.log it still shows the old count and the date is also not updated on it. While it finsh the check files this morning. Is there something i missed why it didnt updated this bad.log file? i also dont see any other with a newer date. Quote Link to comment
KoNeko Posted October 26, 2020 Share Posted October 26, 2020 Found 25 files with BLAKE2 hash key mismatch warning BLAKE2 hash key mismatch (updated), blah/blah/aom-u-move/kowai/catacodec.data was modified BLAKE2 hash key mismatch (updated), blah/blah/aom-u-move/kowai/local_files.data was modified i had in the plugin excluded teh aom-u-move directory but it still shows the directory under that. these files change often zo i excluded the whole directory but that didnt seems to work. Quote Link to comment
Vr2Io Posted October 26, 2020 Share Posted October 26, 2020 3 hours ago, KoNeko said: Found 25 files with BLAKE2 hash key mismatch warning BLAKE2 hash key mismatch (updated), blah/blah/aom-u-move/kowai/catacodec.data was modified BLAKE2 hash key mismatch (updated), blah/blah/aom-u-move/kowai/local_files.data was modified i had in the plugin excluded teh aom-u-move directory but it still shows the directory under that. these files change often zo i excluded the whole directory but that didnt seems to work. If file already hashed before exclusion, you need clear its attribute. You can clear it manually or edit the export file then clear all by GUI and import again. Quote Link to comment
KoNeko Posted October 27, 2020 Share Posted October 27, 2020 19 hours ago, Vr2Io said: If file already hashed before exclusion, you need clear its attribute. You can clear it manually or edit the export file then clear all by GUI and import again. sounds a very big hassle? How to do that ? or is there a other way. Quote Link to comment
JesterEE Posted November 25, 2020 Share Posted November 25, 2020 @bonienl I started using this plugin on a different array the other day and I think I found a bug with the reporting mechanism for the Export being up-to-date with the Build. When mid-Build, I accidently hit the Export button the the webUI with all the disks selected. The build continued, but an export was also done on all the devices in my array with the current files where hashes were computed. I figured, these would almost certainly contain incomplete data, but I'd make a new export when the build completed the next day. Before I did however, I wanted to see if the Export was incomplete AND the webUI reported that the state was stale. When checking the export files, I was correct that the data was incomplete and not all the hashes were in the export files. BUT, the webUI greeted me with all green check marks on the export status for each device. This left me with a bigger question. How does the plugin determine if the build and export are up-to-date? Quote Link to comment
Recommended Posts
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.