Understanding Problem - PreClear a HDD - what is it "exactly" doing

[Zonediver]

Hi Experts,

i am using unraid since 2010 and read some articles and user commends about preclear but i have still a real understanding problem with it.

The only usefull prepare of a HDD (in my opinion) is a low-level-format, which is "refreshing" the whole platter and brings back the magnetic orientation of each sector

on the disk.

So my question is now: does this preclear a reorientation of the magnetic surface or is it just writing zeros to the platters?

If it is a "silly" question, i opologize, but i must ask this question because i did never a preclear on any disk on my unraid and i never got a redballed disk, so it seems

useless - for me - because it wastes time - but i might be wrong.

Thanks for your help to explain

[Squid]

The low-level formatting that you're thinking about hasn't actually been possible on a hard drive for quite a number of years, and it was done because hard drives of the era used stepper motors which were mechanical devices and tended to drift or change their alignment over time.

 

What preclear basically does is read every byte on the drive, write every byte on the drive, and then read every byte on the drive.

 

Along the way, its comparing smart values for the drives at the end of each pass.  This way, you will know if the brand new drive you just bought has out of the box issues (and it does happen) and should be returned.

 

It also saves you time when you are adding a new drive to an array that is already parity protected.

 

If you don't run a preclear, then when you add a drive, unRaid itself must clear the drive (because there is no guarantee that the drive has nothing but zeroes already written to it) so that it is in a predefined state and the parity is still valid.  During this time period (say around 10 hours or so for a 3TB), your entire array is unavailable.  No streaming of videos or music.  With a precleared drive, not only do you know that the drive you are putting into the system is actually good, but the down time is only a minute or two.

[itimpi]

Another reason for doing a pre-clear is that the most likely time for a disk to fail is when it is brand new and has not yet been used.  Putting a new disk through the pre-clear helps give some confidence that you are not going to suffer from this 'early mortality' issue. 

 

If you do it with the disk in the position where it will finally be used you are also checking that the cabling and power supply to the disk are good.

[Zonediver]

Thanks for this detailed explanation

But the part with "unraid is not usable during rebuild" is not correct.

 

I do this normaly in the following way:

I take the new HDD and put in in my PC

then i read all SMART-values, do a short read test and after this, i format the disk with zeros

Afther this process, i delete the partition so it is RAW

Then i put it to my unraid and start the rebuild

And now: during this process, unraid is fully available - i can stream movies/music and tv-shows during the rebuild

 

If this procedure is not recommended, please let me know 

[SSD]

Thanks for this detailed explanation

But the part with "unraid is not usable during rebuild" is not correct.

 

I do this normaly in the following way:

I take the new HDD and put in in my PC

then i read all SMART-values, do a short read test and after this, i format the disk with zeros

Afther this process, i delete the partition so it is RAW

Then i put it to my unraid and start the rebuild

And now: during this process, unraid is fully available - i can stream movies/music and tv-shows during the rebuild

 

If this procedure is not recommended, please let me know 

 

There is a difference between using a new disk for a rebuild operation (which you described) and adding one as a new disk (which is what squid was describing). A rebuild operation does not care about preclearing the disk because it is about to overwrite every sector on the disk. Adding a new disk requires that a drive be full of zeroes. A drive precleared using the script contains a preclear signature that unRAID trusts to mean the disk is already full of zeroes. Lacking that signature, unRAID zeroes the disk and keeps the array offline during the process.

[SSD]

I do this normaly in the following way:

I take the new HDD and put in in my PC

then i read all SMART-values, do a short read test and after this, i format the disk with zeros

Afther this process, i delete the partition so it is RAW

Then i put it to my unraid and start the rebuild

And now: during this process, unraid is fully available - i can stream movies/music and tv-shows during the rebuild

 

If this procedure is not recommended, please let me know 

 

Sorry didn't respond to this part of the question. I am not sure exactly what you mean by "formatting a disk with zeroes", but assuming this writes the disk with zeroes.

 

By way of background, new disks are not tested extensively at the labs (maybe some enterprise class drives are, but not commercial drives). Drive failures follow a bathtub curve, meaning that there are more failures at the beginning and end of expected life, than in the middle. Testing the drive early can help identify some of the drives that would otherwise fail early in their lives, so the purchaser can return them for replacement just after purchase.

 

By reading, writing, and then re-reading each sector, while simultaneously performing extraneous background operations generating random head movements - unRAID is performing a much more thorough test to what you are describing. It was designed by Joe L. with knowledge of the way the smart system detects and remaps sectors. If there is a bad sector, the initial read allows the smart system to detect it. It will not remap after the read in most cases, but will "remember" that this is a weak sector by marking it as a sector pending reallocation. It does not remap because smart does not know whether this sector contains important data - data that might be retrievable through extraordinary means (e.g., a recovery service). (Smart will not reallocate a sector on a read unless it is able to successfully read the sector after several retries. At that point it can remap the sector and write the data elsewhere without risking data loss). The write stage will cause the smart subsystem to remap that sector, which it already knows is problematic. Since this is a write, there is no need to read or know the contents of that sector - it is about to be overwritten anyway. But smart does give one last variant attempt to access the pending sector. Sometimes, remarkably, the sector responds and smart unflags it as a pending sector and does no remap. But mostly it just confirms the sector is bad and performs the remap, writing the data to the newly mapped sector.

 

In this way, the pre-read and write phases of the preclear cycle should cause any bad sectors to be identified and remapped. The post-read step confirms that all sectors are full of zeroes. Having non-zero data after just filling the disk with zeroes would certainly be a problem. You might call this a data verify unrelated to smart. But the post read is also engaging the smart system, as the pre-read did. If the post-read identifies additional sectors that are problematic, it will mark them as pending. This is a bad situation - the first read didn't see these sectors as a problem, but now they are being seen as a problem. This is pointing towards a bad disk.

 

If a drive makes it through the pre-read, write, and post-read with no reallocated sectors, no pending sectors, and verifying the zeroes were written correctly, you can have pretty high confidence the drive is good. Some users prefer to run 2 or 3 (or more) cycles on disks. Unless a disk is suspect in some way, I stick with one and will endure the hardship of dealing with a drive failure if that drive does fail. But my experience has been that preclear does a fine job with 1 pass of weeding out a high percentage of drives that would otherwise experience early drive failure.

 

I feel it relevant to state that not all organizations endorse early drive testing. In fact most if not all consumers of huge numbers of disks do not do such testing, but simply put the new drives into service and depend on the redundancy of their storage arrays to identify and kick bad disks so that they can be replaced and rebuilt. The time to pretest each one just isn't economic. They also argue that you are using up useful life of the drive in the pretesting (filling a disk completely - 1, 2, or even 3 times - that might represent a lifetime of writes for many drives! Might consider it like taking a car on a 100,000 mile test test drive before starting to use it!)

 

We personal users, however, would rather weed out a bad one and return it for a new one, than have a disk get kicked from the array and have to deal with the rebuild and warranty claim. To us that add a few drives a year, it is worth it IMO.

[Perforator]

I've never pre-cleared any of the drives in my array. It's a waste of time in my opinion.

 

[SSD]

I've never pre-cleared any of the drives in my array. It's a waste of time in my opinion.

 

Sorry to be blunt, but this is an uninformed opinion.

 

Doing a preclear before adding a drive to an array avoids a lengthy downtime. Even if you don't want it for it's diagnostic purpose, every user should find that useful.

 

But it's diagnostic ability to weed out weak disks has been proven repeatedly. Not saying you haven't been lucky, but after it happens to you once, you will see the value.

[garycase]

I've never pre-cleared any of the drives in my array. It's a waste of time in my opinion.

 

It's certainly not a waste of time if you're adding a new drive to your array => just eliminating the effective downtime on the array while it clears the drive is worthwhile.    If you really don't want to bother with the testing aspect of the pre-clear script, you can do just the clearing operation (which takes about 1/4th the time of a full pass) by using the -n option.

 

If you're replacing a drive, then it's true that the only function pre-clear is providing is a test for infant mortality.  If you're willing to live with the potential infant mortality issues, then it's fine to skip the pre-clear => a drive rebuild will itself provide a degree of testing (without the SMART parameter monitoring that pre-clear does) ... and if you do a non-correcting parity check immediately after the rebuild you'll know that all went well.

 

But you SHOULD do at least some level of testing of your new drives -- a pass with the manufacturer's diagnostics on a different PC is fine if you don't care about the pre-cleared signature (i.e. you're not adding it as a new drive or just don't mind the long downtime for UnRAID to clear the drive itself).