April 17, 201016 yr Hi. I've recently read up on the added value of doing a pre_clear on any newly installed disk, whether it is a brand new disk or used, it gives the person some certainty that then new disk is clear of any problems. I will for any future disks that I install on my rig, I would run it to be sure. Seeing that though it isn't a requirement, but so many on the forum recommend and use this, why not tie it or bundle it with unRAID? Perhaps have it as a option on the default web GUI for only when unraid detects a new disk and has the option available as a button to click. It should also give some info on what it does and while it is pre clearing, display the verbose or status on the page like a percentage remaining or something.
May 31, 201016 yr Joe, how difficult would be to make and integrate an unMenu package for your preclear script? Would be great I think having it right in unMenu for the less experienced folks.
May 31, 201016 yr Joe, how difficult would be to make and integrate an unMenu package for your preclear script? Would be great I think having it right in unMenu for the less experienced folks. We can't just use the existing process, since most users will not wait 20+hours for a web-page to return with the completion status. It is basically a complete re-write, because the clearing itself must be performed by a process that disconnects itself from the web-server and runs to completion posting its status and progress to a yet-undefined inter-process communication area, one that can handle multiple drives being cleared at the same time. Then, the web-interface just needs to deal with the display of the progress and the initial request and any potential cancel. Finally, the current "diff" of before and after smart reports is far too difficult for most users to interpret. It needs a analytical module to advise of what happened during the clearing process. Now this same three part process is probably not a bad thing even for the command line version. We would then be able to start the clearing process and not worry about the terminal session being disconnected. We could even monitor it from the web interface, or from the command line interface. I've been giving parts of this a bit of thought, and its a bit more complicated when you need to be able to track multiple disks being cleared. Joe L.
May 31, 201016 yr I see what you mean! Can a screen session be used for a quick implementation. The web server could connect and pull the text that is displayed there and show it? Name the different sessions according to the disk name being cleared. Just a quick though, however...
June 1, 201016 yr Would it be easier to have the preclear session update a log file, and then have the web session check the output of that file at a set interval? The disconnection between the two would allow for navigation away from the page without losing session state. It would also let you house multiple sessions in different temporary log files. You could even allow for a trigger to send an email when the preclear is over, with the results.
June 1, 201016 yr Would it be easier to have the preclear session update a log file, and then have the web session check the output of that file at a set interval? The disconnection between the two would allow for navigation away from the page without losing session state. It would also let you house multiple sessions in different temporary log files. You could even allow for a trigger to send an email when the preclear is over, with the results. Your description of a "log file" is one potential "yet-undefined inter-process communication area" as I initially described. It certainly is a simple one to impliment, but I'll need to figure out just how big a given log might get if clearing a 2TB drive through 20 cycles of I simply append to it. (somebody will do it, trust me) If we write a date-time, a disk device, a percentage or progress, a step, a temperature we might have something like this 2010-05-29 08:00:00 /dev/sdg 035 01 40C I probably forgot something, so let's say 50 characters per line. If we update the status every 10 seconds, then we get 300 characters per minute. I've been hearing 20 to 30 hours for larger drives for a single cycle... It then would be 300 characters per minute * 60 minutes per hour * 30 hours = 540,000 characters to log just one disk through 1 cycle. That is way too long to constantly scan for the tail, and several disks in parallel through multiple cycles would just be eating up too much memory. If I just keep a single line per disk being cleared, I'll need to deal with concurrency issues, but I think I can work through that. It is why I originally said a yet-undefined inter-process communication channel needs to be defined. Optimally, I'd use shared memory... but then, the files in a ram disk are one version of "shared" memory.
June 1, 201016 yr You could even allow for a trigger to send an email when the preclear is over, with the results. Pre-clear already has a built-in e-mail for intermediate progress, and at completion. -m [email protected] = optional recipient address. If blank and -M option is used, it will default to default e-mail address of "root" -M 1 = Will send an e-mail message at the end of the final results (default if -m is used, but no other -M option given) -M 2 = Will send an e-mail same as 1 plus at the end of a cycle (if multiple cycles are specified) -M 3 = Will send an e-mail same as 2 plus at the start and end of the pre-read, zeroing, post-read -M 4 = Will send an e-mail same as 3 plus also at intervals of 25% during the long tests The -m, -M options requires that a valid working mail command is installed. One version that has worked (bashmail) is affilaited with the unraid_notify script. There are others that also will work. Any of these need to be configured to work with your mail server.
June 1, 201016 yr Pre-clear already has a built-in e-mail for intermediate progress, and at completion. I didn't realize this. Thanks for pointing it out. It certainly is a simple one to impliment, but I'll need to figure out just how big a given log might get if clearing a 2TB drive through 20 cycles of I simply append to it. (somebody will do it, trust me) ... I've been hearing 20 to 30 hours for larger drives for a single cycle... It then would be 300 characters per minute * 60 minutes per hour * 30 hours = 540,000 characters to log just one disk through 1 cycle. That is way too long to constantly scan for the tail, and several disks in parallel through multiple cycles would just be eating up too much memory. No need to append, as the only real interesting information during a preclear is the current progress, not the historic log results. If you do an overwrite of the data in the file, you'll only have one or two lines, total. When the preclear is finished, you can append the results to the file. If I just keep a single line per disk being cleared, I'll need to deal with concurrency issues, but I think I can work through that. It is why I originally said a yet-undefined inter-process communication channel needs to be defined. Why not make one file per disk, using the serial number as the file name? It will build in some fault tolerance in case one of the drives dumps out of its cycle (which happened to me recently).
June 1, 201016 yr I've been hearing 20 to 30 hours for larger drives for a single cycle... I recently precleared four 2 TB WD EARS (with jumpers) for a single cycle each, and each one took about 35 hours. You may want to call it 40 hours just to be safe. So your estimates may still be a bit shy of the real thing.
June 1, 201016 yr If we write a date-time, a disk device, a percentage or progress, a step, a temperature we might have something like this 2010-05-29 08:00:00 /dev/sdg 035 01 40C I probably forgot something, so let's say 50 characters per line. If we update the status every 10 seconds, then we get 300 characters per minute. I've been hearing 20 to 30 hours for larger drives for a single cycle... It then would be 300 characters per minute * 60 minutes per hour * 30 hours = 540,000 characters to log just one disk through 1 cycle. That is way too long to constantly scan for the tail, and several disks in parallel through multiple cycles would just be eating up too much memory. If I just keep a single line per disk being cleared, I'll need to deal with concurrency issues, but I think I can work through that. It is why I originally said a yet-undefined inter-process communication channel needs to be defined. Optimally, I'd use shared memory... but then, the files in a ram disk are one version of "shared" memory. Why not just have a log for each disk? preclear_sdg.log preclear_sda.log And just overwrite the log with the same as whats shown at the command line currently, you could tail to get current progress for an overview of all disks being precleared and when you select a disk to check progress it shows the more detailed version. Of course your perfectly reasonable reaction might be "that sounds like a great idea, let me know when you have finished coding it" at which point i'll have to just say sorry and keep my mouth shut
June 1, 201016 yr If we write a date-time, a disk device, a percentage or progress, a step, a temperature we might have something like this 2010-05-29 08:00:00 /dev/sdg 035 01 40C I probably forgot something, so let's say 50 characters per line. If we update the status every 10 seconds, then we get 300 characters per minute. I've been hearing 20 to 30 hours for larger drives for a single cycle... It then would be 300 characters per minute * 60 minutes per hour * 30 hours = 540,000 characters to log just one disk through 1 cycle. That is way too long to constantly scan for the tail, and several disks in parallel through multiple cycles would just be eating up too much memory. If I just keep a single line per disk being cleared, I'll need to deal with concurrency issues, but I think I can work through that. It is why I originally said a yet-undefined inter-process communication channel needs to be defined. Optimally, I'd use shared memory... but then, the files in a ram disk are one version of "shared" memory. Why not just have a log for each disk? preclear_sdg.log preclear_sda.log That's similar to what I have in mind, but I would use the disk model/serial number to be sure the results are mapped to the specific drive when reported later. I really need a directory in which to write the log files, and then one, or more likely two files per drive. One with the abbreviated current state of the clear, and the other with the historical data including time-stamps, so you would know how things progressed when you were not watching. And just overwrite the log with the same as whats shown at the command line currently, you could tail to get current progress for an overview of all disks being precleared and when you select a disk to check progress it shows the more detailed version. The small risk there is that the web-page might be reading the file at the exact same time it was just opened to write by the background clearing process (and therefore, it is empty) There is no easy to use semaphore or file-locking I can use to keep the web-browser waiting until the file itself is written, even if I'm only writing one line. If you know of one that can be used at the shell level, I'm all ears. Of course your perfectly reasonable reaction might be "that sounds like a great idea, let me know when you have finished coding it" at which point i'll have to just say sorry and keep my mouth shut Yup... I'm glad you understand... It is a bit more complicated than you describe, and the much more difficult part is displaying the output in a way where users do not constantly ask me to interpret their test results. That will take a lot more time to refine... (But it you wanted to help, I'm not so stupid as to ignore an offer to help.) Joe L.
June 1, 201016 yr Why not just have a log for each disk? preclear_sdg.log preclear_sda.log Kinda already said that... Why not make one file per disk, using the serial number as the file name? It will build in some fault tolerance in case one of the drives dumps out of its cycle (which happened to me recently). The small risk there is that the web-page might be reading the file at the exact same time it was just opened to write by the background clearing process (and therefore, it is empty) There is no easy to use semaphore or file-locking I can use to keep the web-browser waiting until the file itself is written, even if I'm only writing one line. If you know of one that can be used at the shell level, I'm all ears. If the browser initiated read finds an empty file, wait 5 seconds and reread, without updating the screen so there is no "blip" of 0% progress. Surely there's a way to open a file with read only permissions to ensure the browser isn't locking the preclear writes. I'm just bouncing ideas Joe, not telling you what/how/where to code. Yup... I'm glad you understand... It is a bit more complicated than you describe, and the much more difficult part is displaying the output in a way where users do not constantly ask me to interpret their test results. That will take a lot more time to refine... (But it you wanted to help, I'm not so stupid as to ignore an offer to help.) If I were more comfortable with Linux scripting I would be more than happy to offer help. I write code for a living, but it's in the Microsoft ASP/SQL world, so I don't exactly feel at home in Linux mode. But I am learning. Thanks for doing what you have done, and what you continue to do, which is develop useful programs that make unRAID more flexible and usable.
Archived
This topic is now archived and is closed to further replies.