Squid

That being said, the hash file does not contain any absolute path references to the file. In other words, in your case you could verify the files either through user shares or through disk shares and it would still work ok.

OK I see your point. On my system it takes around 20 seconds (after the drives spin up) to figure it all out before it starts actually checking the disk. I get it. We do things differently. I exhibit much more control in the layout of my data utilizing user shares for read only. I never have a directory with multiple files span multiple disks. I always keep like files self contained within a directory. With all my split points, they are only directories and they rarely have files that require a checksum. With this software, if a user access a disk share via windows and uses corz to verify a hash file, is it possible the hash file will contain references to files on other disks? Depends on how you set up the folders. If you use user shares then yes because then its up to unraid's split levels on where to put the hash file. If you set up a custom disk folder, then no. The hash file generated will always be on the particular disk. And related to this, I'll add to the list code to suppress all the errors if no user shares are present (since that is a valid use case for unRaid)

OK I see your point. On my system it takes around 20 seconds (after the drives spin up) to figure it all out before it starts actually checking the disk.

While I have user shares, I almost always access and organize with disk shares. My own gdbm managed md5sum DB files are all based on the disks. my sqlocate sqlite tables are also based on disks as they could migrate into and out of the array as well. Ideally when a disk goes bad and you question it's integrity, you'll be operating at the disk level. i.e. a failed disks/when replacing/rebuilding or migrating to other file systems. With hash files existing on a per directory level it does actually provide a valid check mechanism per disk. I have not used the Checksum Creator/Verify tool, but being able to validate a disk with a sweep of hash files contained only on that disk would be valuable after an event. Because with unRaid's user shares and split levels, there is no guarantee that the actual .hash file is stored on the disk being checked even if the files hashed are stored there. Because of that, when checking a disk the plugin first parses every hash on all the disks. It then checks to see what files that have been hashed are stored on the particular disk. It then checks those files. That way, no file stored on the disk gets missed. To only process .hash files stored on the particular disk being checked is pointless in my mind.

Correct. Was designed for user shares in mind and never thought that someone out there never used them

You are correct. The first two did start automatically on boot. That got broken in the last rev. Will update tonight (would have done it 2 days ago, but my boss is being unreasonable and insisting I earn a pay check )

ok. Narrows the problem down at least. Got an update coming out tonight because in the process of testing this I find a few minor but annoying (to me at least) bugs. There's nothing wrong with the path that you are using (replicated that path on mine with no problems), and the plugin picked up changes to it. My best guess right now as to what happened is this: You triggered a manual scan of the share. Because of the nature of how everything works (right now), each write to a folder ultimately triggers another scan (and a hash file being written counts as one). The cumulative amount of excess scans exceeded the maximum number of queued events (default is 16384). Not sure if you've got 16384 separate folders within that share, nor if anything else installed is also possibly using inotifywait either. If that maximum is exceeded, then events get dropped. This circumstance should only happen during initial creation of hashes, or if you happen to write out a ton of files at once (ie: save the plex metadata to each folder). Luckily, the forthcoming cron job will alleviate this high workload issue. Everyday (or whenever you choose) a manual scan gets triggered to catch up anything that may have been missed. ETA for that update is this weekend. Got some more testing to do with regards to some of the logging issues that I noticed in your logs, and also discovered that the monitor doesn't properly start up from a restart, and some annoying items that keep insisting on being output to an attached monitor. Those are todays's updates.

I actually just do fast. I found that its close enough.

Received your log. will check it out after work. But, I'm curious as what happens if you manually trigger another checksum on that share. Will it pick up those two files? (It won't rehash everything else)

You don't have .txt files excluded by chance (or not included)? Anything "weird" about the folder name? Can you send me the complete command log (stored in /tmp/checksum/checksumlog.txt) and the .hash file for that particular folder.

Assuming that the txt file was put into /mnt/user/Documents (because the plg did pick up changes in Pix2015 (several commands issued to scan (and presumably it did hash the folder -> the excess scan commands didn't pick anything up that had changed), was that folder already being monitored when you put the file in?

Looks to me like you have to create in advance the appropriate ini files. You might want to ask here about it. http://ogarproject.com/ Also, according to this: https://github.com/OgarProject/Ogar, you also need to have ports 80 and 443 mapped to the container (They're not exported in the dockerfile, so CA has no idea about that) Problems like this (and the impossibility to automatically pick up any environment variables) is why CA has dockerHub searches disabled by default. That feature of CA will always remain an "advanced" feature.

You already can on creation. Disable include all files and enter in whatever wild cards you choose (separated by a space) into the include / exclude sections.

Truth be told, the plugin should work, as that use case is quite possible. Gfjardim / Joe? probably just never thought about it.

Brand new installation? Syslog doesn't look like you've got any drives assigned at all, and apparently preclear doesn't like that fact.

Updated to 2015.10.24 - Ability to run manual verifications of shares / disks (either full or partial) - Revised logging - Ability to save log files when they rotate or at array stop. I am just in the beginning stages of testing cron job scheduling for verifications, but since the actual verification routine is done (and ultimately forms the basis for cron jobs), here is the ability to manually run them. You can run verifications against either a share or against a disk. Two options are available for either of these: % of the disk / share to check, and % to start at. IE: if you only want to check the oldest 10% of the files in a share, you would enter in 10 and 0. If you only want to check the new 10% of the files, you would enter in 10 and 90. Verifications are completely separate from the job queue. In otherwords, verifications will run regardless if a creation job is in progress, and you can run multiple verifications at the same time. Speed decrease on running multiple verifications concurrently will be dependent upon your included / excluded disks for a particular share, transfer speed of the drive(s), CPU speed, etc. On my main server, I can run around 6 concurrent disk verifications and the total net speed is greater than only running one at a time (although each verification is slower than if I only ran one at a time) At the end of each verification, if there are any failures (corrupted / updated files), a notification is sent out. If you have email notifications enabled on warnings, the email body will detail the files affected and the probable cause. If you do not have email notifications enabled, then you will just see that there were failures (I have a feature request in to rectify this: http://lime-technology.com/forum/index.php?topic=43615.0) In either case, once there is a failure, a new log button (Failures) will be enabled that will also show you all of the failures that have occurred since the last reset of the server. The logging has been revamped. There are now 4 log buttons: Failures - A list of all the verification failures since the last reset Checksum Log - A list of all the creation / updating of hash files Verify Log - A list of all the verification activity Command Log - A list of all the various commands given to the plugin (ie: update this folder's checksums, verify this folder, etc) All logs (with the exception of the failure log) will automatically rotate at a size of 500k (to not eat all your memory on the server if you have a ton of activity and long up-times). The Failures log does not rotate so that you can see any/all failures on the system since the last reset of the server. If this file should happen to grow to be an appreciable size, then you've got far more important issues happening with your server than the size of a log file. You also have the option of saving the log files when they rotate to the flash drive (/boot/config/plugins/checksum/logs). A failure log for a particular job is always saved to the flash drive (/boot/config/plugins/checksum/logs/failure) regardless of whether you have save logs enabled or not. Notes on Verifying a Disk This plugin makes zero assumptions about how your shares and split levels are set up. Because of this, unRaid may or may not store the hash files for a particular media folder on the same disk as the media folder. The plugin is aware of this, and when doing a disk verification, the following happens: - Plugin will find every hash file within /mnt/user - It will then parse each hash file to see what file(s) exist on the particular disk to be checked. - It will then only verify those files that exist on the disk. On my servers, this process only takes ~20 seconds before it gets rolling on the verification, but it guarantees that any file on a particular disk that has had a hash created for it will be checked regardless of which disk the actual hash file exists on. Next Week's Update: Cron Job scheduling of manual creation jobs and verifications (Verification scheduling will be user-set cumulative eg: Week 1 check first 10%, week 2 check second 10%, etc), and hopefully a rehash of the GUI (if not that will happen in two weeks)

You can hit log to see what's actually going on. (Or post a link to the log -> its stored at /tmp/checksum/log.txt) Today / Tomorrow's update has the option to automatically save the log(s) to the flash drive.

Wow. You must still be running an early 6.0 Beta. I don't think that the extensions tab has existed for quite a while

It's https://github.com/linuxserver/docker-templates. I think that when they created the OP, they pretty much assumed that everyone using docker was also using the Community Applications plugin (and why wouldn't you? It's way easier than dealing with repositories. http://lime-technology.com/forum/index.php?topic=40262.0

Try turning on dockerhub searches within community applications search for discourse and see what happens

http://lime-technology.com/forum/index.php?topic=34970.0 and changes required for 6.1 http://lime-technology.com/forum/index.php?topic=42259.0

99% of the time there's probably going to be nothing in the queue to save. If you have a share set to be monitored, then any change to the share (new file / changed file) will add that folder (not the entire share) to the queue. That particular folder will then be hashed according to the rules that you've set. No other folders within that share will be touched. My rational for this is that I want every new / changed file to be hashed as soon as possible. Not up to whenever then next cron job runs. My settings for the pause is set to 10 minutes. 10 minutes after a file is added, it has the hash created for it. If I just can't wait to watch the movie after its transferred, and the file happens to be in use (watched) when the plugin goes to hash it, it automatically is skipped (to avoid any potential viewing troubles like stuttering), and rescheduled for the next time that cron.hourly runs. To my way of thinking, this has a couple of advantages - The less amount of time the file sits on the hard drive without being hashed, the less likely that any silent corruption to that file could happen. - Any changes I make to the particular file(s) (stripping foreign languages, etc) are rehashed immediately without me having to think about it, and then trying to remember 6 months later when the file fails a verification test if I changed anything to it. - There is no lengthy process of having to hash your entire shares unless you choose to do it. That being said, I fully do realize that everyone does not want the same method of operation as I myself do. After the verification side of things is pumped out in the next day or two (work responsibilities permitting), the next updates will be to the GUI which will also allow scheduled hashing of entire shares (currently right now its a manual job -> It adds to the queue the share, and tells it to run against every folder. If that happens to get interrupted, then adding it to the queue again just picks up right back where it left off) I would think that the vast majority of writes to shares happen basically one folder at a time, with a lengthy pause before another write (eg: a movie gets transferred over, then another movie after a download, etc) In the interim, the queue has already been cleared out, so there's nothing to lose. About the only time a ton of queues would happen at a time is when you would have your PMS write out all of the .nfo files for each piece of media. In that case, a new item is going to get queued for just about every folder within your shares. The plugin runs through them all. In that case, its fast because the only new files it has hash are the very small .nfo files. But, you have to keep in mind that any folder that gets queued, will scan the entire folder for items that aren't already hashed. Anything not already hashed is going to get hashed. So, if you tell PMS to write all your .nfo's and you haven't already hashed the media files with either this plugin, or with corz, or something else, then the net result is that all of your folders will get fully hashed (basically no different than manually hitting add to queue) Simply put, you set a share to be monitored for changes. If a change happens within a folder of that share, then the folder is hashed for items not already hashed. Manually adding a share to the queue will hash every folder within that share. This was originally designed for how I want it to handle my media collection. Over the next month, things are going to become more general purpose and try and hit everyone's usage case (and a big part of that is scheduled hash creation, not relying upon a change queue). At that point, I would think it should be bullet proof. Changes to folders are picked up immediately, and in the off chance that something gets missed, then the scheduled scan will pick it up. But, saving the queue (if anything is there) in the event of a shutdown is a good idea, and its something that I'll think about how to implement. In the case of an outright power failure with no UPS to trigger a clean shutdown, there's probably not much I can do about it, because the queue is implemented as a linux pipe (to facilitate easy interprocess communication) and is not a file which is just "read" off the flash drive. But, clean shutdowns can be handled because I can read all remaining queued items, save them, and then re-write them at the next startup. Hopefully that made sense) The basic routine for verifications is done and tested. Next up is the GUI for it, for which it is also going to necessitate the frame work for scheduled cron jobs on verifications.

Have you added any shares to monitor? No, the instructions say that tou have to wait for the 'Watches Established' log message. No, the Start button stays active, the Stop button stays greyed out. Edit: Ah, the message to wait relates to queueing jobs, not adding shares - sorry, my confusion. Like I said, the GUI needs a little TLC. After I finish up with the verifier, then that's going to get overhauled.

Have you added any shares to monitor? Does it say in the log setting up watches? After hitting start monitor, does the button switch to being disabled (ie: you have to stop it then try and start it again) The GUI as-is is also a tad counter intuitive since with each share section, you have to apply it separately for it to take effect (on my to-do list to set up an apply all button)