March 27, 201610 yr Thanks for the explanation, I didn't realise that Export was optional I have carried out an intial build for each disk, but thereafter on a couple of occassions I have noticed that the build is out of date?
March 27, 201610 yr Author I have carried out an intial build for each disk, but thereafter on a couple of occassions I have noticed that the build is out of date? Perhaps, you want to keep an eye on that with the latest version, under normal circumstances that should not happen.
March 28, 201610 yr Request: Option for selecting which page displays by default when clicking the plugin. I find that except for the initial config 99% of the time I want the "Control" page, so it would be nice to have the option. Thx.
March 28, 201610 yr Request: Option for selecting which page displays by default when clicking the plugin. I find that except for the initial config 99% of the time I want the "Control" page, so it would be nice to have the option. I agree with you. Perhaps the simpler solution will be to make the Control page the default rather than having to set an option? I wonder if any others have a view on this?
March 28, 201610 yr Author Another approach can be to separate settings and control. The latter one would then be placed under Tools.
March 28, 201610 yr That would be better, 2 clicks instead of 3, was just thinking that I would really like to have it in the main bar, I know the space is limited but for my NAS only servers that don't use dockes/vms there's still a lot of space and would just need a single click, naturally this should be optional as many users don't have the space or would not want it there.
March 28, 201610 yr I like the idea of making it less clicks to verify data. Another idea based on johnnie's post. It would be pretty awesome to add a row for each disk on the dashboard that had a green check or whatever symbol stating that all data on that disk was good and a red X if it found that data was corrupt.
March 28, 201610 yr I like the idea of making it less clicks to verify data. Another idea based on johnnie's post. It would be pretty awesome to add a row for each disk on the dashboard that had a green check or whatever symbol stating that all data on that disk was good and a red X if it found that data was corrupt. I like this as on my VM server I am at the dashboard most of the time anyway.
March 28, 201610 yr Hey bonienl, I was looking at these scripts: https://lime-technology.com/forum/index.php?topic=35183.0 https://lime-technology.com/forum/index.php?topic=7018.msg68073;topicseen#msg68073 and I realized both could efficiently be replaced by this plugin if it did some parsing of the export files: It could look for files with the same name (not including the path). This would identify issues with a file existing at both /mnt/disk1/share/file and /mnt/disk2/share/file, for instance. The hashes wouldn't necessarily be the same. It could look for files with identical hashes, regardless of file name, to identify true duplicates. I think this would be a nice way to get extra value out of these export files, if you are up for it Interesting... I'll have a look at that. Yes you have the keys to solving a lot of data issues using file integrity data. Statistics of many different kinds would be possible. 1 duplicate file analysis 2 show me different views of all my stuff 3 who knows. . . Now you may not want to create all these things. But creating an API to the file integrity extended attributes might enable all kinds of good things.
March 29, 201610 yr I have come across some behaviour with this plugin that i'd like to see if others are experiencing the same thing as me. I started with a full scan and export, checked to monitor external files and also have an exclusion on .nfo's and apple metadata. My settings are as indicated in the screenshot below: So, I am expecting that once I have scanned and exported that my interaction with this plugin will be limited to any issues it finds on a monthly basis. Is this a realistic expectation? When the scan executed a couple of days ago (27th) I received the following email: Event: unRAID file corruption Subject: Notice [MAIN] - bunker verify command Description: Found 5 files with BLAKE2 hash key mismatch Importance: warning BLAKE2 hash key mismatch (updated), /mnt/disk4/nas/Documents/.DS_Store was modified BLAKE2 hash key mismatch (updated), /mnt/disk4/nas/Documents/Daniel/.DS_Store was modified BLAKE2 hash key mismatch (updated), /mnt/disk4/nas/Documents/Daniel/development/scripts/.DS_Store was modified BLAKE2 hash key mismatch (updated), /mnt/disk4/nas/Documents/Daniel/development/.DS_Store was modified BLAKE2 hash key mismatch (updated), /mnt/disk4/nas/.DS_Store was modified I was expecting that .DS_Store files would be ignored as I have selected ignore apple metadata? Is there something I am missing? The ignore of .nfo files seems to be working just fine. In addition I keep getting emails saying the following: Event: Dynamix file integrity daily update Subject: Notice [MAIN] - Disk 2, Disk 4, Disk 6 needs Export updating Description: 29 non-existing files in export file Importance: normal Why if I have things set up to monitor new files should I have to keep going in and doing a manual export? I am expecting more from the plugin than it is designed to do? Last but not least, when I DO go in there and go to do the manual export I find (each time) that there is not only an export that is required but there needs to be a scan done on one or more disks too. See screenshot below: Now, I can do a Build and Export again BUT feel like I am having to manage this Plugin and keep monitoring the status to ensure that everything remains Built and Exported. Is there anything I need to be doing differently or do my expectations of the plugin need to change? For now Build and Export status is back up to date: EDIT: And now almost 3 hours later, the Build is showing as not up to date for Disk 4. There have been some files copied to Disk 4 BUT I thought that new files were being monitored and things updated accordingly
March 29, 201610 yr Author To set some expectations right... 1. If you experience issues with .DS_Store files, it is likely a left-over from the past. Delete those files (command can be found in an earlier post in this thread) 2. The operation of this plugin does NOT require external files. After an initial build and enabling of the protection and verification feature, all should stay up-to-date automatically. This includes warnings when silent corruption is deteced. 3. If files are found afterwards which didn't get a hash checksum in their extended attributes then a manual Build is required. This is checked once a day, and should be more exception than rule. If this happens frequently, ask yourself how you are creating files and let me know if there is a true omission. 4. Export files are an option and NOT required for the operation of the plugin. These files can be used for recovery or other purposes. These files require manual action to stay up to date, i.e. the user decides when to update the export files. This is a change from the earlier versions of this plugin. And taking this oppertunity to announce a new version 2016.03.29. This new version adds the feature to search for duplicate file names and file hashes, these are read from the export files. At the moment pretty basic and not a replacement of existing tools. Try it out and see if this is a step in the right direction.
March 29, 201610 yr To set some expectations right... 1. If you experience issues with .DS_Store files, it is likely a left-over from the past. Delete those files (command can be found in an earlier post in this thread) 2. The operation of this plugin does NOT require external files. After an initial build and enabling of the protection and verification feature, all should stay up-to-date automatically. This includes warnings when silent corruption is deteced. 3. If files are found afterwards which didn't get a hash checksum in their extended attributes then a manual Build is required. This is checked once a day, and should be more exception than rule. If this happens frequently, ask yourself how you are creating files and let me know if there is a true omission. 4. Export files are an option and NOT required for the operation of the plugin. These files can be used for recovery or other purposes. These files require manual action to stay up to date, i.e. the user decides when to update the export files. This is a change from the earlier versions of this plugin. And taking this oppertunity to announce a new version 2016.03.29. This new version adds the feature to search for duplicate file names and file hashes, these are read from the export files. At the moment pretty basic and not a replacement of existing tools. Try it out and see if this is a step in the right direction. Right, that makes more sense. I shall get rid of ALL .DS_Store files and see if I have any issues going forward. Understanding now what the export function is and how external files are used I don't think I need it, I have a full daily backup to restore from which is also being checked regularly AND i doubt I would ever get rot on two files at the same time so even if there was rot I will always have one copy of the file to restore from so I shall not worry about it. Thank you for taking the time to explain that to me
March 29, 201610 yr And taking this oppertunity to announce a new version 2016.03.29. This new version adds the feature to search for duplicate file names and file hashes, these are read from the export files. At the moment pretty basic and not a replacement of existing tools. Try it out and see if this is a step in the right direction. Wow that was fast, thanks bonienl! I pressed "find" and got: Reading and sorting hash files Including... disk1.export.hash Including... disk1.export.20160214.bad.hash Including... disk2.export.hash Including... disk3.export.hash Including... disks.export.20160109.bad.hash Including... disks.export.20160111.bad.hash Including... disks.export.20160115.bad.hash Including... disks.export.20160118.bad.hash Including... disks.export.20160129.bad.hash Including... disks.export.20160130.bad.hash Finding duplicate file names Duplicate file names found! See log file: duplicate_file_names.txt. Even though it says "Duplicate file names found", my duplicate_file_names.txt file just contains: [other content] So I purposefully created two files: /mnt/disk1/Movies/todo/test.txt /mnt/disk2/Movies/todo/test.txt and ran it again. Now duplicate_file_names.txt contains: disk3,disk2 [other content] Movies/todo/test.txt So I'm glad that file shows up, but it seems like the first run should have picked up some too, especially after I saw the data from the next run... I ran "find" a third time with the "include duplicate file hashes" option. It ran for about 15 minutes and created a 243k line file! This is amazing data. I see I need to exclude a few more directories (such as anything SVN related), but it did identify duplicate photos, videos, and other files that I need to clean up. I am definitely surprised by what it dug up. I need to figure out how to parse the file for groups of files (so I can work on all photos at once, etc) but overall this is very cool.
March 29, 201610 yr I shall get rid of ALL .DS_Store files and see if I have any issues going forward. Actually, you can just go to the tools page of this plugin and press the Clear button. This will delete any attributes that were added before the files were excluded, and once the attributes are gone the files will no longer be processed.
March 29, 201610 yr I shall get rid of ALL .DS_Store files and see if I have any issues going forward. Actually, you can just go to the tools page of this plugin and press the Clear button. This will delete any attributes that were added before the files were excluded, and once the attributes are gone the files will no longer be processed. No matter, deleted them now! Thanks though!
March 29, 201610 yr Author I pressed "find" and got: Reading and sorting hash files Including... disk1.export.hash Including... disk1.export.20160214.bad.hash Including... disk2.export.hash Including... disk3.export.hash Including... disks.export.20160109.bad.hash Including... disks.export.20160111.bad.hash Including... disks.export.20160115.bad.hash Including... disks.export.20160118.bad.hash Including... disks.export.20160129.bad.hash Including... disks.export.20160130.bad.hash Finding duplicate file names Duplicate file names found! See log file: duplicate_file_names.txt. I am afraid you need to do some manual housekeeping. The *.bad.hash files need to be moved to the logs folder. Actually their name have changed too, and end on .log now (but not necssary to rename the old files, just move them). The export folder should only contain the exported files from the disks.
March 29, 201610 yr Author That would be better, 2 clicks instead of 3, was just thinking that I would really like to have it in the main bar, I know the space is limited but for my NAS only servers that don't use dockes/vms there's still a lot of space and would just need a single click, naturally this should be optional as many users don't have the space or would not want it there. The new version allows you to have the feature just one click away (that is put the file integrity control page in the header menu). See file integrity settings to change this.
March 29, 201610 yr The new version allows you to have the feature just one click away (that is put the file integrity control page in the header menu). See file integrity settings to change this. Thanks!
March 29, 201610 yr Author New version 2016.03.29a 2016.03.29b 2016.03.30 with some improvements and corrections is available. Users are encouraged to upgrade to this version.
March 30, 201610 yr I am afraid you need to do some manual housekeeping. The *.bad.hash files need to be moved to the logs folder. Actually their name have changed too, and end on .log now (but not necssary to rename the old files, just move them). The export folder should only contain the exported files from the disks. Thanks, that's much cleaner I moved the files and updated to the 2016.03.30 version Can you clarify what the Find button is looking for? My duplicate_file_names.txt file still just shows this: disk3,disk2 [other content] Movies/todo/test.txt So if it is looking for files at the same path on different disks, then it is working correctly. But if it is looking for duplicate filenames (without considering the path) then it is missing a lot.
March 30, 201610 yr Author Can you clarify what the Find button is looking for? My duplicate_file_names.txt file still just shows this: disk3,disk2 [other content] Movies/todo/test.txt So if it is looking for files at the same path on different disks, then it is working correctly. But if it is looking for duplicate filenames (without considering the path) then it is missing a lot. For duplicate file names, it looks for files with the same path and name on different disks. This is the same behavior as how unRAID sees duplicate names. To find duplicate files regardless of their location, look at the duplicate hashes file. Ps. "other content" in your example above denotes two files with the same path/name but different content (their hashes aren't equal).
March 30, 201610 yr Can you clarify what the Find button is looking for? My duplicate_file_names.txt file still just shows this: disk3,disk2 [other content] Movies/todo/test.txt So if it is looking for files at the same path on different disks, then it is working correctly. But if it is looking for duplicate filenames (without considering the path) then it is missing a lot. For duplicate file names, it looks for files with the same path and name on different disks. This is the same behavior as how unRAID sees duplicate names. To find duplicate files regardless of their location, look at the duplicate hashes file. Ps. "other content" in your example above denotes two files with the same path/name but different content (their hashes aren't equal). In that case, it is working perfectly. Thanks for this!
March 30, 201610 yr Im not sure this is working right for me. I have 6 disks and built and exported all of them. A few days later i noticed disk 5 or 6 (i forget which) was not up to date on the build and export. I recreated them. fine. Now last night I had a scheduled check at 3am. It started disk 3 and 4 only. Why not 1 and 2 and 5 and 6? And then i also noticed that 5 and 6 are not up to date again. What gives? I thought this is supposed to keep itself up to date as files are copied to the array. Anyone help me get this setup right? or what can i check to see why its not working. Also, when I do an export of one that the build is up to date but the export is not I get a lot of error messages: /mnt/disk6/TV/**PRIVATE** Mar 30 16:16:28 FILESERVER bunker: error: no export of file: /mnt/disk6/TV/**PRIVATE** Mar 30 16:16:28 FILESERVER bunker: error: no export of file: /mnt/disk6/TV/**PRIVATE** Mar 30 16:16:28 FILESERVER bunker: error: no export of file: /mnt/disk6/TV/**PRIVATE** Mar 30 16:16:28 FILESERVER bunker: error: no export of file: /mnt/disk6/TV/**PRIVATE** Mar 30 16:16:28 FILESERVER bunker: error: no export of file: /mnt/disk6/TV/**PRIVATE** Mar 30 16:16:28 FILESERVER bunker: error: no export of file: /mnt/disk6/TV/**PRIVATE** Mar 30 16:16:28 FILESERVER bunker: error: no export of file: /mnt/disk6/TV/**PRIVATE** One more thing. Now that I updated the plugin, the find command keeps spiking my CPU usage for 2 -3 mins each 2-3 mins. What is this doing? After removing the plugin this stopped. Installing it again creates the same behaviour. Any way to go back to a version without the find feature?
March 31, 201610 yr Hi bonienl, thank you very much for the great plugin! I converted all my disks to xfs and I'm very happy with the interface, usability and extended data security. I know your plugin uses the extended attributes to save the hash information and I really like that. But as I often check single directories for correctness using corz from my windows machine I would like an user option to create corz compatible hash files per directory, too. I know this is redundant information and I know there is another plugin doing that, but as the author discontinued development it might break over time leaving me stranded. Please consider implementing that option into your plugin. Also another question. I didn't tested long enough to know, but what happens if a time slot for a check is missed? I guess you are using cron to shedule the tasks, which would skip such tasks and wait till the next execution time. As my server is sleeping rather often, this prevents task execution most of the time with the other plugin, which is a big disadvantage for me. IMHO it would be better to start missed tasks as soon as the server wakes, which will keep it awake until completion and only then will let it sleep again, if no other things are going on. But maybe your plugin already does this? If not, there was a post for installing anacron on unraid recently, so it should be possible to get another sheduler running on unraid. Also I remember there beeing php alternatives, which could be a possibility, too. cheers, SlrG
March 31, 201610 yr Author Im not sure this is working right for me. I have 6 disks and built and exported all of them. A few days later i noticed disk 5 or 6 (i forget which) was not up to date on the build and export. I recreated them. fine. Now last night I had a scheduled check at 3am. It started disk 3 and 4 only. Why not 1 and 2 and 5 and 6? And then i also noticed that 5 and 6 are not up to date again. What gives? I thought this is supposed to keep itself up to date as files are copied to the array. Anyone help me get this setup right? or what can i check to see why its not working. Also, when I do an export of one that the build is up to date but the export is not I get a lot of error messages: /mnt/disk6/TV/**PRIVATE** Mar 30 16:16:28 FILESERVER bunker: error: no export of file: /mnt/disk6/TV/**PRIVATE** Mar 30 16:16:28 FILESERVER bunker: error: no export of file: /mnt/disk6/TV/**PRIVATE** Mar 30 16:16:28 FILESERVER bunker: error: no export of file: /mnt/disk6/TV/**PRIVATE** Mar 30 16:16:28 FILESERVER bunker: error: no export of file: /mnt/disk6/TV/**PRIVATE** Mar 30 16:16:28 FILESERVER bunker: error: no export of file: /mnt/disk6/TV/**PRIVATE** Mar 30 16:16:28 FILESERVER bunker: error: no export of file: /mnt/disk6/TV/**PRIVATE** Mar 30 16:16:28 FILESERVER bunker: error: no export of file: /mnt/disk6/TV/**PRIVATE** One more thing. Now that I updated the plugin, the find command keeps spiking my CPU usage for 2 -3 mins each 2-3 mins. What is this doing? After removing the plugin this stopped. Installing it again creates the same behaviour. Any way to go back to a version without the find feature? The hashes are stored in the extended attributes of a file, and should stay up to date automatically when protection is enabled. Export to a file is optional and a manual action, it does not affect the working of the plugin. The status shows when an existing export file gets outdated, but the user decides when to update. The disk verification tasks table defines how and when verification of disks take place. See the online help for more information. I suppose the name of the folder/file isn't really **PRIVATE** (asterisks in a folder/file name are not allowed). The message "no export of file" happens when no hash key value is stored in the extended attributes of the file. The usual approach is to rebuild. Sure you are using XFS as file system on disk 6 or have you changed the hashing method from one to another midway?
Join the conversation
You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.