I've run through a few more test cycles...  I'm pretty sure I have preclear_disk.sh clearing correctly.

 

I eliminated using sfdisk to do the actual partitioning... instead, I'm using a few lines of "awk" to generate the 16 bytes needed to partition the drive, and I am using "dd" to write to the drive's MBR.  It is easier, and less likely to run into a deadlock situation.

 

Attached is the newest version of "preclear_disk.sh"

 

To use it, unzip onto your flash drive, then log in via telnet, cd to the folder you unzipped it into, and then type:

preclear_disk.sh /dev/aaa

 

Where "/dev/aaa" =  /dev/hda, /dev/hdb, /dev/hdc, /dev/hdd ... or /dev/sda, /dev/sdb, /dev/sdc, etc.

 

If you are unsure of the device for your disk, type:

ls -l /dev/disk/by-id and look for the device affiliated with your disk's model/serial-number.

 

You will first be shown the existing partitioning of your disk (if it is partitioned) and asked to respond with "Yes" to continue.

You must respond with a capital "Y" and lower-case "es", or the pre-clear will not continue.

 

The preclear_disk.sh script will not clear a disk if it is part of the unRAID array, nor will it clear a drive that is mounted or busy.

It will take many hours if it is clearing a large drive.  You can follow along its progress on the screen.

 

Most of the steps of pre-clearing a disk take only a few seconds.  The exception is "step 2 of 10" in my process.  It will take a few hours or more for a larger drive.  My IDE based array wrote to an old IDE drive at about 24 MB/s.  It clears (writes zeros) at an average of 80MB/s to an much newer SATA drive.  (it takes over 3 1/2 hours on my IDE based array to clear a 1TB SATA drive at a rate of between 85Mb/s and 75Mb/s using an IDE-to-SATA converter)

 

Expect the screen to sit at this step for a while... you can monitor the progress as the bottom lines update every 10 seconds.

 

When done, the screen will look like this:

 

As before, this script is "beta" and although unlikely to harm your server, you could do something I've not anticipated and clear the wrong drive.   See my disclaimer in the post above.   

 

This shell script for pre-clearing a drive is not supported by Tom at Lime-Technology, so don't call him to ask how to use it... but he did share in an e-mail to me how unraid determines if a disk is pre-cleared, and what I would need to do to have a disk be considered pre-cleared.   I want to thank him publicly for the clues I needed.   If you have questions, ask here in this support forum thread.

 

Edit: newest version with pre-read/clear/post-read is attached here: http://lime-technology.com/forum/index.php?topic=2817.msg23246#msg23246

 

Joe L.

I am ashamed of asking that dumb question, but what is the pre-clearing does and why is it good to have before adding to the array?

All the time when I added new disks to the array (new or already formatted), the time of formatting was just quite a few times, so I didn't catch the "lengthy clearing process". Could you please bring some light in my head?

 

Thank you!

I am ashamed of asking that dumb question, but what is the pre-clearing does and why is it good to have before adding to the array?

All the time when I added new disks to the array (new or already formatted), the time of formatting was just quite a few times, so I didn't catch the "lengthy clearing process". Could you please bring some light in my head?

 

Thank you!

It is not a dumb question.

 

If I were to add a new 1T byte disk to my array, and it was not pre-cleared, it would first be cleared by the unRAID software, then formatted, then made available for your use.

 

While the "clearing" process is occurring my array will be off-line, unable to be used to play movies, do backups, etc.   On my server, the step of clearing a new 1TB drive can take nearly 4 hours.   After a disk is "cleared", it is added to the array, the array is "Started" and you are then asked if you wish to format the new disk.  The actual formatting takes only a minute or so, but it is done after the array is back on-line.

 

Rather than have my server off-line for 4 hours, if I pre-clear a disk before adding it to the array, I won't have 4 hours of time where my server is not available to play movies, etc.  Instead, my down time is minutes...

 

Pre-clearing writes zeros to most of the new drive, all except a few specific bytes in the master-boot-record that allow it to be identified as pre-cleared. Those bytes define the first partition, and partition type.  That process is a bit tricky and it is why I have gone through so many "beta" versions of my pre-clear script.

 

The only other way to add a drive to your array without having it "cleared" is to press the button labeled "Restore"  It forces a complete rebuild of parity based on the current assigned and working disks.  In my array, that takes over 12 hours.

 

Without a pre-clear process, I have either 4 hours of down-time while a new disk is cleared, or 12+ hours of time where I have no parity protection of my data.

 

Joe L.

Thank you Joe L. for taking your time to explain in that detail.

Te reason why I was't aware about how long the clearing process is, that all the time I added new disks, I reorganized my disks and slots at the same time, so I always did a restore and parity rebuilding as well. Therefore, there were no need of clearing, if I understood well. I know, that this wasn't the best approach, as I don't have partity protection for this time.

 

But now, I understand. Thank you again!

Joe, what I envision is an "add new disk" plugin. It would first verify that the disk is either really empty or new, if it's not, it would still offer the option to continue, but not without confirmation, maybe even a 30 second time out period between confirmations to avoid the "I'm really sure" double click followed by the Oh CRAP moment. It would then proceed to do a long smart test, a full burn in run, another logged smart test, followed by your pre-clear magic. Automate the whole lengthy process so you can shut down the array, physically install the new drive, boot the box back up to a usable array, start the new drive procedure, come back 2 days later and have a spiffy new tested drive waiting to be assigned to an array slot.

 

In principle I know that I should do the burn in and long test before trusting a drive, but I'm usually too much in a hurry to get running. If I had a button to push that would do all the right things to add the disk safely, I'd use it, even if the drive wouldn't be ready for a day or two.

 

For confirmation that you are operating on the correct drive, perhaps offer the option of manually entering a serial number. If I put in a new drive, I could easily write down the serial number, and enter it when asked so you could compare it to the electronic version on the drive. No match = further checks and prompts.

 

Jonathan

Joe, what I envision is an "add new disk" plugin. It would first verify that the disk is either really empty or new, if it's not, it would still offer the option to continue, but not without confirmation, maybe even a 30 second time out period between confirmations to avoid the "I'm really sure" double click followed by the Oh CRAP moment. It would then proceed to do a long smart test, a full burn in run, another logged smart test, followed by your pre-clear magic. Automate the whole lengthy process so you can shut down the array, physically install the new drive, boot the box back up to a usable array, start the new drive procedure, come back 2 days later and have a spiffy new tested drive waiting to be assigned to an array slot.

 

In principle I know that I should do the burn in and long test before trusting a drive, but I'm usually too much in a hurry to get running. If I had a button to push that would do all the right things to add the disk safely, I'd use it, even if the drive wouldn't be ready for a day or two.

Interesting idea...  I like it a lot... the difficult part is getting the periodic status we would like, because even though we would love to just press a button and walk away for a day or 2, we won't... we will peek every now and then.  I was thinking of almost the same test cycle... and I think that a plug-in that will initiate a "SMART" test, a "full read", "preclear-zero", "full read", "SMART test" series would be perfect.  One thing for sure... it would take a LONG time to test the series with a large drive. (your estimate of a day or two is not too far off.)

 

Whatever a plug-in does, it must simply initiate the action, return immediately and then allow you to monitor it.  The actual burn-in and test has to be a stand-alone process that can be monitored/managed by the plug-in.

 

For confirmation that you are operating on the correct drive, perhaps offer the option of manually entering a serial number. If I put in a new drive, I could easily write down the serial number, and enter it when asked so you could compare it to the electronic version on the drive. No match = further checks and prompts.

 

Jonathan

Serial numbers are pretty long... but how about entering the just the last 4 characters of the serial number and ensuring they match??  (lots easier to get it right as you type it in.)  The other tests to determine the drive is not otherwise in use are still important, as I  want it completely outside of the array and not mounted before proceeding to clear it.

 

Joe L.

 

Serial numbers are pretty long... but how about entering the just the last 4 characters of the serial number and ensuring they match??  (lots easier to get it right as you type it in.)  The other tests to determine the drive is not otherwise in use are still important, as I  want it completely outside of the array and not mounted before proceeding to clear it.

 

Joe L.

 

The last 4 would be fine. It's only a "warm'n'fuzzy" so _I_the_end_user_ know for sure which drive I'm about to blow away. Your logic to safety check my decision should take precedence. If I were to add 3 new drives at once, I'd want to be able to initiate the process on all 3, one at a time confirming each one, and allow them all to process at once.

 

Are there any drives that you know of that return a different serial number string than is printed on the label? I know that the internal serial number on HP printers for example have several digits extra added to the end compared to the printed label.

 

All the drives assigned in my array show serial numbers 8 characters long, but I have no clue what the actual numbers on the label are.

 

Jonathan