Jump to content
dalben

Windows based preclear

20 posts in this topic Last Reply

Recommended Posts

What would it take to create a windows based preclear app/script ?

 

As it stands, my server, hidden in the basement on the TOP shelf of a cupboard is the only box I have capable of running preclear.  I'd love the ability to drop a hdd into a caddy attached to my desk pic and run preclears from there.  It would make life a lot easier.

 

Is it possible ?

Share this post


Link to post

First, Someone who is a genius at writing .bat scripts like Joe L. is with .sh scripts.

 

Second, Someone who can figure out if Windows will even allow one access to OS level disk operations at .bat level.  (Remember, that for security reasons, access to those functions should be tightly controlled!) 

 

Third, There will have to a pool of alpha and beta testers willing to risk/entrust their computers during the software development phase because not everyone will be running the same version of Windows and wide cross-section of them will be necessary to find all of the gotcha's!

 

Actually, it might be easier and quicker to you to go to a few yard/garage sales and see if you can find a old computer that you could use to run a trial version of unRAID  on.  OR you could install a 'hot-change' caddy into that "hidden" server so you could easily slide HD's in-and-out for preclearing.  (How often do you preclear a disk anyway?)

Share this post


Link to post

So you're saying there's a chance then?

 

I have written a (very) few .bat programs over the years and I had some limited exposure (some thirty-five years ago) to Shell programing, and I can tell you that on the surface, they are similar.  However UNIX was initially written with the idea that the command line would be the only interface.  (IF you don't know, the basic interface was a teletype machine for the several years!)  Doing anything (and everything) on the command line became 'Badge of Honor' in the UNIX world.  I have the gut feeling that the Windows .bat language is a small subset of what can be done in a shell program but I don't have proof of that statement.  Remember that in a Shell program, the programmer calls on those command line programs like  sed  and a whole pile of others that simply don't exist in the DOS environment!  If a particular function  doesn't exist in DOS, then a program to duplicate  will have to be written and compiled that will do it and included in the package.  (Bell Labs probably had a couple of hundred guys who contributed one or more of these little programs over the years.) 

 

Anything can be done if you have enough resources to do.  The question is how do you assemble those resources.  Are you willing to devote twelve hours a day for the next six months?  I know that I won't be willing to do that...

Share this post


Link to post

What about a virtualbox container of unraid w/preclear plugin?

 

 

Config it for only access to the usb drive device?

 

 

Myk

 

 

Share this post


Link to post

What about a virtualbox container of unraid w/preclear plugin?

 

Config it for only access to the usb drive device?

 

This sounds like the best idea if possible.  A cut down, dedicated to preclear image just to make preclearing easier.  Or take it a step further, same image has all the tools needed for any disk repair/recovery as well.

 

Much rather sit in my office doing this than halfway up a stepladder in the basement

Share this post


Link to post

Would USB be too slow though? Wonder if VMWare or VirtualBox can pass through a whole drive.

Share this post


Link to post

I often thought how useful it would be to have a preclear on a Windows machine.  One of my Windows systems has a cageless removeable 3.5" drive bay.

 

To preclear safely on an unRAID server, you shouldn't start the array.  There are several disk errors that will hang your server necessitating a reboot.

Share this post


Link to post

Would USB be too slow though? Wonder if VMWare or VirtualBox can pass through a whole drive.

 

I've run my last 2 preclears off a USB3 Caddy attached to the Unraid server and it didnt seem too bad.

 

Also, in my use case, doing it from my PC means I'm not rushed for time where I need to keep going to the basement or transport the server upstairs.

 

I just saw another thread where someone is using a laptop for unraid.  That might be the hybrid solution I am looking for.

Share this post


Link to post

I use a little KVM, and I can verify that it will pass my USB to SATA adapter through to do a preclear through the unassigned devices plugin and the preclear plugin / script. It's USB 3.0 device, so it actually goes about as quick as I normally allocate for a drive to be done.

Share this post


Link to post

Make an unRAID 5.0.6 USB stick and boot your PC from it whenever you want to pre-clear a disk. You can use it in trial mode indefinitely.

 

Share this post


Link to post

Let me say that with Windows 6, preclearing of the drive is not really necessary as the unRAID can now preclear the drive without taking the array off-line. 

 

If you are just trying to see if the drive is good to begin with and put a few hours on the drive to get it by the infant mortality failure point, I believe both Seagate and WD provide utilities that will completely scan and test the entire disk surface in much the same manner as the preclear function.  And these utilities run on Windows machines...

Share this post


Link to post

Let me say that with unRAID 6, preclearing of the drive is not really necessary as the unRAID can now preclear the drive without taking the array off-line. 

 

If you are just trying to see if the drive is good to begin with and put a few hours on the drive to get it by the infant mortality failure point, I believe both Seagate and WD provide utilities that will completely scan and test the entire disk surface in much the same manner as the preclear function.  And these utilities run on Windows machines...

 

EDIT:  'unRAID 6' was 'Windows 6'

Share this post


Link to post

A few thoughts ...

 

(a)  Conceptually, it's not all that difficult to do.  There's no problem writing low-level disk I/O routines in Windows ... I've written quite a few over the years, but NOT in the last decade (programming is decidedly a young person's "game" ... it's all too easy to spend 20-30 hours focused on a problem and forget about minor details like food & sleep).    The pre-clear script is essentially a read test of the whole disk;  writing zeroes to the entire disk;  doing another read test along with some intensive seeks to exercise it a good bit; and writing a special "signature" on the disk that "tells" UnRAID that it's been pre-cleared.    The only thing that isn't pretty straight-forward is the last bit ... and I assume Tom would provide the details of that to anyone who was writing a utility to do this (as he did for Joe many years ago).

 

(b)  Pragmatically, it's simply not needed. 

 

Pre-clear serves two fundamental purposes:

 

      (1)  The original purpose was to eliminate the long down-time on an UnRAID array while a new disk was being cleared.  The array was unusable for the duration of this, so adding a new disk was a bit of a PITA.  Joe L. wrote the pre-clear script to provide a way to have a disk "pre-cleared", so it could be instantly added to an array ... UnRAID would recognize the special signature; write zeroes over that signature; and "knew" that the rest of the drive was already zeroes, so there was NO impact on parity, so the drive could simply be added to the array.

 

and 

 

(2)  As part of the process, Joe included some rudimentary testing -- the pre-read and post-read phases along with a fair amount of random seeks.  This 2nd part has become a de-factor way to "test" a new disk before incorporating it into the array.

 

Why isn't it really needed?

 

r.e. #1 =>  UnRAID no longer makes the array unavailable while clearing new disks.  It simply clears them; THEN adds them to the array; and the array is never unavailable for the user during this process.    So the primary initial motivation for pre-clearing drives is no longer a factor.

 

r.e. #2 =>  It's certainly a good idea to test a new disk before using it.  But the manufacturer's free testing utilities do a very good job of this.  WD's Data Lifeguard;  Seagate's SeaTools;  HGST's Drive Fitness Test; and several 3rd party tools are all good ways to test new drives -- and do every bit as much of a test as the pre-read and post-read phases of pre-clear.  These all can run on a Windows box with no problem.

 

Personally, I test new drives with WD's Data Lifeguard =>  I run the Quick Test; then the Extended Test; then I Write Zeroes to the entire drive; and then I repeat the Quick and Extended tests.    I've found a few "infant mortality" issues over the years with this regimen; but I've NEVER had a drive that passed this regimen cause any problems during it's first few years of life.

 

If I was going to add the drive to UnRAID as a new drive (not as a replacement, where clearing it makes no difference), I would then connect it to a spare PC;  boot a USB flash I have configured with the free version of v5 and the pre-clear script; and run one pass of pre-clear with the -n option, which will skip the pre- and post- read phases => i.e. just do the clear and mark it with the pre-cleared signature.  This takes about 1/4th as long as a normal pre-clear cycle.    However, with v6.2 this is no longer necessary, since UnRAID will clear it without any impact on the array operations.

 

 

Share this post


Link to post

A few thoughts ...

 

<< snip >>

r.e. #2 =>  It's certainly a good idea to test a new disk before using it.  But the manufacturer's free testing utilities do a very good job of this.  WD's Data Lifeguard;  Seagate's SeaTools;  HGST's Drive Fitness Test; and several 3rd party tools are all good ways to test new drives -- and do every bit as much of a test as the pre-read and post-read phases of pre-clear.  These all can run on a Windows box with no problem.

 

Personally, I test new drives with WD's Data Lifeguard =>  I run the Quick Test; then the Extended Test; then I Write Zeroes to the entire drive; and then I repeat the Quick and Extended tests.    I've found a few "infant mortality" issues over the years with this regimen; but I've NEVER had a drive that passed this regimen cause any problems during it's first few years of life.

 

<< snip >>

 

Quick question.  Have you used the WD Data Lifeguard utility on other manufacturer's disk  (like Seagate) without any issues? 

 

 

Share this post


Link to post

I used the preclear (via gui) to do 2 passes on 4 different USB3 based WD hdds. They all were mybooks, and I removed them from the case when done, and added them to unraid.

 

Preclear times were about ~125MB/sec, and took (as far as I can tell) about the same time as having it internally would have.

 

For full disclousre though, I ended up replacing a 2TB with a 4TB (and so on) so I don't actually know if the preclear would have worked correctly, since some of these external USB3 enclosures have built in encryption.

 

But if you had USB3 (I think 2 might drag on too long) you could run a cable and try that out?

Share this post


Link to post

No, it doesn't work with WD MyBooks precisely because of the encryption. It does work with Toshiba Canvio USB 3 disks though.

Share this post


Link to post

With WD Data Lifeguard, you can also run two (and presumably more) instances. I've done it successfully with two 8TB HGST Deskstars, and doing it now with another two. I don't know if there is a performance penalty, but I suspect it still takes much less time than running the tests consecutively.

Share this post


Link to post

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.
Note: Your post will require moderator approval before it will be visible.

Guest
Reply to this topic...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.