Pre Clear natively

[NAS]

I am requesting that the current code that detects an unformatted drive be enhanced to be "format and preclear." A global setting for the number of passes would need added. This setting could be 0 if a user does not want preclear.

 

This tool should be native and default for all users not just the technically literate and forum followers.

 

Specifically I am not suggesting this could/should be an addon but rather that it is absorbed into the default unRAID as the way a drive is added by default.

[Helmonder]

Actually I would +1 a plugin version... Saves the telnetting.. Basically would be a webinterface around the command line so probably not to difficult..

[NAS]

I don't mind it being a plugin actually IF its is shipped by default and supported by Limetech. Otherwise it must be native.

 

This is not a nice to have addon it should be core and default for every user... even the one who dont have a clue.

[chickensoup]

I agree, I also have a similar request- if it belongs in a new thread please feel free to move (feature request).

 

I'm aware that it is possible to preclear/format multiple disks at a time, just not via the WebUI. The question I have is why not? Would it be possible to allow multiple drives to be cleared etc. via the WebUI in order to make expanding an existing array a little easier (when more than one disk is involved) and also save the need to do this via terminal? This would not be something require for 5.0-Final however might be something worth integrating into 5.1.

[Helmonder]

Well... Basically the UNRAID system does not -need- preclear... It was build to accept a drive and do an initial run on the drive to check it..

 

Pre-Clear was build by someone else to give the drive a much more thorough testing.. It is not needed for the core functionality of unraid, although most community members recommend using it (I use it myself also).

 

Plugin would be the best way to go, it will also keep the unraid system nice and lean.. We do not want to end up with a system that has "everything in", if I wanted that I would have gone for a windows solution :-)

 

I do not see why the plugin could not be a community product.. It is pretty "safe" functionality in the sense that it does not mess with data drives.. Only thing that is pretty important is that it never fools with drives already in use...

[NAS]

Do not agree at all.

 

It is safer to use pre-clear than not. A user should not be expected to know about pre-clear, get it, learn how to use it etc

 

Default it to 0 passes (i.e. off) if that helps but some stuff in the community should be pushed upstream into unraid proper for official sanction support, testing and documentation.

 

If doing so stops one user losing one important file it is only a win for unRAID and justifies itself.

 

 

 

 

[Helmonder]

Not much to disagree with :-)

 

You are right in that preclear is safer... I did not say otherwise... Thing is that is not a -necessity- .. Whatever though.. Ofcourse I would be ok with it part of the base package...

[Dougy]

Not much to disagree with :-)

 

You are right in that preclear is safer... I did not say otherwise... Thing is that is not a -necessity- .. Whatever though.. Ofcourse I would be ok with it part of the base package...

 

You forget that a preclear also stops the array from needing to be offline as the disk is prepared. 

[Joe L.]

Well... Basically the UNRAID system does not -need- preclear... It was build to accept a drive and do an initial run on the drive to check it.

wrong. It blindly writes zeros to the disk. It does absolutely no checking.

[bbqninja]

Do not agree at all.

 

It is safer to use pre-clear than not. A user should not be expected to know about pre-clear, get it, learn how to use it etc

 

Default it to 0 passes (i.e. off) if that helps but some stuff in the community should be pushed upstream into unraid proper for official sanction support, testing and documentation.

 

If doing so stops one user losing one important file it is only a win for unRAID and justifies itself.

 

1. I agree 100% with what you've said.  It should be in there.

 

but

 

2. There are TONS of other linux based NAS distributions.  NONE of them pre-clear.  NONE of them have a huge amount of people complaining about data loss/etc.

[Helmonder]

Well... Basically the UNRAID system does not -need- preclear... It was build to accept a drive and do an initial run on the drive to check it.

wrong. It blindly writes zeros to the disk. It does absolutely no checking.

 

And the zero's will possibly trigger smart results that you can check..

 

This thread is somewhat weird, just to make sure people are not misunderstanding what I am saying:

 

- Preclear is good, nice and it is very wise to use it;

- Preclear will help in giving your array more uptime;

 

BUT

 

It is not functionality that is absolutely necessary for unraid to function so I do not agree it should be part of the core package..

 

Basically I would like to see that every part of the functionality I have added to unraid is part of the core package, preclear included.. However if Tom would do that he would end up with way to much depdendencies halting development. Therefor unraid should stick with its core functionality and have the community take care of the rest (which it is doing a nice job of)

[Dougy]

Well... Basically the UNRAID system does not -need- preclear... It was build to accept a drive and do an initial run on the drive to check it.

wrong. It blindly writes zeros to the disk. It does absolutely no checking.

 

And the zero's will possibly trigger smart results that you can check..

 

This thread is somewhat weird, just to make sure people are not misunderstanding what I am saying:

 

- Preclear is good, nice and it is very wise to use it;

- Preclear will help in giving your array more uptime;

 

BUT

 

It is not functionality that is absolutely necessary for unraid to function so I do not agree it should be part of the core package..

 

Basically I would like to see that every part of the functionality I have added to unraid is part of the core package, preclear included.. However if Tom would do that he would end up with way to much depdendencies halting development. Therefor unraid should stick with its core functionality and have the community take care of the rest (which it is doing a nice job of)

 

and I'm guessing that none of them make the array inaccessible for many many hours while they prepare a disk.  unless I am mistaken and this behavior has change in one of the version updates this really needs to change!

[Isolgrac]

Can I make an alternative suggestion?

 

I'm currently putting together my first server and so I'm commenting as a new & inexperienced user. I have personally found the large number of available plugins quite bewildering. I know they're all there in the User Customizations board, but coming at that for the first time is quite overwhelming. I can see there are some popular ones (Simple Features, unMenu, SNAP) but I'm struggling to understand what they all do and if I should choose one or can I run them together. (Don't answer that here or else we'll go off topic!)

 

What would be useful for newcomers would be if there was an official plugin selection that should contain all the normal useful stuff, whatever that may be. This could then be bundled with the official download, or made available alongside it. However, I think this should be a community led project, not a lime-tech one. That way lime-tech would be freed up to work on the core functionality and the UI and the useability tools would be developed and supported by the community.

 

I don't think this is significantly different to what we have at the moment. I'm sure the experienced people here would be able to suggest a selection of existing plugins that could be taken as a starting point. You then just need a team of people of oversee it. Development of plugins would carry on as it is, but a plugin would only get added to the official set once it has been firmly established in the community.

 

I think this would meet both goals - newcomers can get a standard set of valuable add-ons and lime-tech can concetrate on core functionality.

[NAS]

Its good to hear the viewpoint of a new user so thanks for posting.

 

I think your points reinforce mine, let me explain.

 

I am putting a case for pre clear to be made part of unraid proper. As a new user you would have been presented with either:

 

1. Two buttons. One saying "format" and another saying "preclear and format"

 

or

 

2. One button saying "format" and a setting for pre clear count

 

This puts this feature right there in front of you from day one in the normal GUI, you could not help but be introduced to it. There would be no searching around for what you should add (with conflicting viewpoints) and the manual would explain what these features are and what they give to you.

 

Many of the replys so far are from the two camps. Camp 1 that thinks pre clear is a must and camp 2 that don't see the need.

 

Pre clear does two high level things.

 

1. Checks for faulty drives to a level higher than most OS's

2. Makes the integration of a new drive into unRAID far more seemless

 

unRAID is a very specific NAS and with that comes some specific tasks that need done. It also focuses on making sure your data is safe beyond a level most out of the box OSs give you so accepting that you realise there will be some unRAID quirks

 

I am never going to convince users that dont see the need for pre clear that there is a need, and this thread is not the place to do that but at the very least half the community think the feature is crucial and IMO that is enough to warent it promoting to official supported status.

[boof]

There are a lot of community developed plugins I personally think should be adopted by limetech into the core.

 

Preclear wise just a way to prepare a disk for adding to the array quickly without any downtime would be my vote.

 

How that would be done - using the actual existing community preclear script or just another method behind the scenes wouldn't concern me.

 

It would be an interesting (but fairly academic) metric to know how many people use the current preclear script to verify their disks and how many use it just to avoid lengthy array downtime when adding a drive. I always just have it fill with zeros and exit. I've never used it (in conjunction with s.m.a.r.t of course) to verify the condition of a drive. Though I appreciate that functionality exists.

[Isolgrac]

Hi NAS, thanks for your reply.

While I undertand and agree with your points, I also agree with Helmonder that lime-tech should focus on the basic core functionality.

 

If this was an open-source project then I would 100% agree with you, this sort of thing should be part of the main project. Of course this isn't open source and (correct me if I'm wrong here) I understand there is only one person working on it. Given the very limited development resource and the already delayed release schedule, I would suggest that trying to push more work his way is not the way to go.

 

At the moment there are only two areas of functionality: 1- lime-tech developed and supported official stuff, and 2- User developed and supported plug-ins. I think we all agree that important stuff is easily lost in the user forums. What I'm suggesting is an in-between solution that identifes the best-of-class user plugins and bundles them up with the main download. The key bit is that the dev & support of these plugins remains in the unraid community, not with lime-tech. This meets the goals of getting these tools out to novice users and avoids the problems of placing additional work on lime-tech.

 

 

[NAS]

As far as I am concerned pre-clear is core functionality.

 

Thats the crux of this and its not something we can debate out unfortunately.

 

 

[BRiT]

I have to agree, pre-clear is basic core functionality that absolutely should be built into the official product.

 

From a feature standpoint, having your array being down for extensive periods of time (hours on end) when a new drive is being added to it should be a bit of an embarrassment.

[theone]

I use pre-clear for adding new drives as part of their veification.

 

The main reason IMO for adding it to the core package is exactly what "BRiT" wrote above (has nothing to do with drive verification)

 

From a feature standpoint, having your array being down for extensive periods of time (hours on end) when a new drive is being added to it should be a bit of an embarrassment.

[Joe L.]

I have to agree, pre-clear is basic core functionality that absolutely should be built into the official product.

 

From a feature standpoint, having your array being down for extensive periods of time (hours on end) when a new drive is being added to it should be a bit of an embarrassment.

Unless I'm way off base, adding a disk to an established array in other raid protected NAS involves dropping and re-building parity.  The array  might stay up, but the parity protection is not present during the operation.

 

The current button in unRAID could be changed minimally to perform the following:

check for pre-clear signature... if not present,  zero drive  (while unRAID array is online) ...add pre-clear-signature once zeroing is complete.

add to existing array (pre-clear signature should be recognized as it is now)

format drive

 

instead of

stop array

add to existing array

check for pre-clear signature... if not present,  zero drive  (while unRAID array is off-line)

format drive

start array

 

The argument over core functionality vs. add-on can be for the added features in the user developed add-on.

 

In both the current and proposed change the pre-clear signature is recognized, if you elected to perform the more comprehensive tests.

In the proposed case, the down-time of the array is minimal.  unRAID still zeros the disk, and still partitions/formats it.

 

The primary difference is that the disk is not part of the array until AFTER it has a pre-clear signature of some kind.

 

If parity protection is not yet established, the entire zeroing and writing of the pre-clear signature can be skipped (or, optionally NOT skipped, if users desire to exercise the SMART features to detect bad sectors.)

 

Joe L.

[Joe L.]

I use pre-clear for adding new drives as part of their veification.

 

The main reason IMO for adding it to the core package is exactly what "BRiT" wrote above (has nothing to do with drive verification)

 

From a feature standpoint, having your array being down for extensive periods of time (hours on end) when a new drive is being added to it should be a bit of an embarrassment.

And the extended down-time is exactly why I originally wrote the pre-clear script, after learning Tom's code skipped the clearing on the drives he had already pre-cleared and tested.

 

(interestingly, the time off-line was far less back then, as the biggest disks were only 500Gig.  The off-line time was equally annoying.)

 

[JackBauer]

I'd like a pre-clear plugin that analyzes the output and presents the data in a manner that is easy to understand.

[Joe L.]

I'd like a pre-clear plugin that analyzes the output and presents the data in a manner that is easy to understand.

me too. 

[JackBauer]

me too. 

 

Are you sure?  You might get bored not responding to all those "Does my pre-clear log look ok?"

 

(Including mine )

[Joe L.]

me too. 

 

Are you sure?  You might get bored not responding to all those "Does my pre-clear log look ok?"

 

(Including mine )

It could be as simple as two possible outputs:

 

    1. Your disk has already failed.

  or

    2. Your disk is in the process of failing.  (it just needs a bit more time.  repeat this test later to see if the process is complete.)

 

Joe L.