7 minutes ago, noax said:

Thanks a lot for your explanation, now I understand, but if I format, wouldn't I lose everything?!

You can only add a cleared disk to a parity protected array. If the disk contains data that you want on your server you’ll need to move the data to the server first and then go through the process to add the disk to the array. You can use the Unassigned Devices plugin to mount the disk outside the array and transfer the data.

11 hours ago, noax said:

but if I format, wouldn't I lose everything?!

This is the first time in this discussion with you that the word FORMAT has been used. Format has a very specific meaning, and misunderstanding that meaning has caused some people to lose data.

 

Unraid must format any disk it will use in the parity array or cache pool. You must let Unraid do the format after adding the disk. And if you add a disk to an array that already has valid parity, it must be clear before it can be formatted.

 

A clear disk and a formatted disk are NOT the same thing. A clear disk is all zeros. A formatted disk has an empty filesystem written to it (this is what FORMAT has always meant in every operating system you have ever used). That empty filesystem is "metadata", data about the data. The metadata of a newly formatted disk represents an empty top level folder, ready to accept new files and subfolders.

 

A mistake people often make is telling Unraid to format a disk that is already in the array and already has data on it. They think they can recover that data from parity. But format is a write operation (writes an empty filesystem), and Unraid treats that write exactly as it does any other, by updating parity. So after a format of a disk in the array, parity agrees that the disk has an empty filesystem.

 

All this may be more than you wanted to know, but maybe it will make you reconsider before you make a mistake in the future.

9 minutes ago, trurl said:

All this may be more than you wanted to know, but maybe it will make you reconsider before you make a mistake in the future.

But both a format and a preclear will lose all data on the drive, yes?  Alarm bells started ringing when somebody is worried about losing data on a preclear topic (but the more I thought about it, the less I probably know)

8 minutes ago, Cessquill said:

But both a format and a preclear will lose all data on the drive, yes?  Alarm bells started ringing when somebody is worried about losing data on a preclear topic (but the more I thought about it, the less I probably know)

Clear or preclear will write zeros to the entire drive overwriting all existing data on the drive. This is done so that the parity knows exactly what is on the drive when it is added (zeros) and can maintain an accurate computation of parity. You can’t add data to a parity protected share by adding a drive that already contains the data. You’ll need to copy the data to the share first probably.

Thanks @wgstarks - that's what I was thinking (as in: if you're worried about losing data when formatting, why are you preclearing before taking care of the data?).  However, I've been wrong too many times to be confident!

Short answer, clear preclear and format will “erase” the drive. With format there are processes to possibly recover the data since formatting just changes the indexing. This is usually rather expensive though.

 

With clear and preclear your only choice would be to restore from a backup probably since these processes actually overwrite every sector.

6 hours ago, wgstarks said:

Short answer, clear preclear and format will “erase” the drive. With format there are processes to possibly recover the data since formatting just changes the indexing. This is usually rather expensive though.

 

With clear and preclear your only choice would be to restore from a backup probably since these processes actually overwrite every sector.

 

7 hours ago, wgstarks said:

Clear or preclear will write zeros to the entire drive overwriting all existing data on the drive. This is done so that the parity knows exactly what is on the drive when it is added (zeros) and can maintain an accurate computation of parity. You can’t add data to a parity protected share by adding a drive that already contains the data. You’ll need to copy the data to the share first probably.

If you have a disk formatted by unRAID (or possibly otherwise formatted correctly) that has data, it can be added to the array by resetting the array, adding the disk, and rebuilding parity.

 

A replacement disk can be swapped in without preparation because the disk will be written with the data and formatting existing on the previous disk. (Many would preclear as a test but it’s not required in any case)

 

Adding an an additional disk to the array will cause the disk to be zeroed(cleared), added to the array and then formatted. The new disk will be empty and any data it previously contained will be gone. (This is what preclear was originally for)

 

If if you have a random disk with data that you want to add to the array. You must first copy the data to another location, perhaps in the array, and then add the disk as an additional disk. If the data is not in the array it should be copied to the array at this point. 

 

So preclear is not needed and no longer appears to be supported. Does anyone know of a docker for disk test and “burn in”? Should we start removing references to preclear?  Should the OP be modified to reflect the current non-working status of preclear? If gfjardim returns to update the plugin they can update the the OP to say that the plugin works now.

Thanks everyone for your help, it seems to be working now I formatted it and it shows as disk 2 (i had disk 1 and parity) so I think it should be good and I just hope that my shares will automatically write on disk 2 when disk1 will be full

 

4 hours ago, noax said:

Thanks everyone for your help, it seems to be working now I formatted it and it shows as disk 2 (i had disk 1 and parity) so I think it should be good and I just hope that my shares will automatically write on disk 2 when disk1 will be full

Make sure you have Notifications setup to alert you immediately by email or other agent if a problem is detected.

I've cleared a few disks and sent the automated disk reports afterwards. Is there a public database you're building for drive reliability? Can this app serve as a reference for vetting quality drives?

Hey!

 

I've set up a test unraid server with some older disk I want to preclear. Two 1.5 TB disks and two 1 TB disk. The preclear plugin recognizes the two 1.5 TB disks as 2 TB disks

 

 

Any ideas? It is a bug? Diagnostics attached.

tower-diagnostics-20190813-1431.zip

So it looks like there is no need to really pre-clear anymore. Should have read this before I started a pre-clear on a brand new 10TB WD RED. Ah well may as well let it finish now lol

2 hours ago, z0ki said:

So it looks like there is no need to really pre-clear anymore.

There has not been for several years now   Some people still like to do it as an initial stress test as recovering adding a 'bad' drive to the array can be more trouble than doing the preclear.

4 hours ago, z0ki said:

So it looks like there is no need to really pre-clear anymore. Should have read this before I started a pre-clear on a brand new 10TB WD RED. Ah well may as well let it finish now lol

2 hours ago, itimpi said:

There has not been for several years now   Some people still like to do it as an initial stress test as recovering adding a 'bad' drive to the array can be more trouble than doing the preclear.

When you add a drive to a new slot, Unraid will clear it by writing zeros to the entire disk. If instead you use a new drive to replace an existing drive, Unraid will rebuild to it by writing the entire disk from the parity calculation. Either way the new drive gets a workout, though not as much as one (or more) preclear cycles.

 

If you are only doing a single disk instead of several, and you are careful (always double-check connections) and diligent (have Notifications setup to alert you immediately), then maybe the possible infant mortality of that single disk isn't that much trouble to recover from. Somewhat less risk if you have dual parity.

 

2 hours ago, trurl said:

When you add a drive to a new slot, Unraid will clear it by writing zeros to the entire disk. If instead you use a new drive to replace an existing drive, Unraid will rebuild to it by writing the entire disk from the parity calculation. Either way the new drive gets a workout, though not as much as one (or more) preclear cycles.

 

If you are only doing a single disk instead of several, and you are careful (always double-check connections) and diligent (have Notifications setup to alert you immediately), then maybe the possible infant mortality of that single disk isn't that much trouble to recover from. Somewhat less risk if you have dual parity.

 

For new drives I still prefer to do at least one pass (all 3 stages) of the preclear script/plugin. For drives that I've had in operation for a while, I'll typically only do a zero pass as long as the SMART report from the drive shows nothing concerning. Of course since unRAID will do that itself, it's not required to use the plugin. I just prefer being able to do a zero of the drive before adding it to the array. It takes the same amount of time whether you do the zero pass using the plugin or by adding the drive to the pool, but if by chance there's a failure during the zero, I'd rather know about it before adding it to the pool.

 

One question about just adding a drive to the pool and letting unRAID do the zero: if by chance there's errors or a failure of the drive while unRAID is zeroing the drive, does it still add it to the pool? If so this means obtaining a replacement drive right away. If you do the zero pass with the plugin, you'll have the option of delaying the addition of the drive until it's replaced/repaired. Most of us tend to buy drives only when on sale, so using the plugin seems safer to me, although I do have dual parity in use.

 

Good little insight Trurl and XXL. 

 

As I won't have a parity disk setup straight off the bat i thought I would just let the entire cycle run. As majority of my media data will be on this disk that is currently going through the Pre-Clear cycle. I will have the files backed up elsewhere regardless, so I may as well take the precautions to save headaches down the line. 

 

Step 1 of 5 - Pre-read verification: [17:32:07 @ 158 MB/s] SUCCESS
Step 2 of 5 - Zeroing in progress: (22% Done) Time elapsed: 3:03:30 | Write speed: 203 MB/s | Average speed: 203 MB/s

This is my progress thus far. Taken a while as you can see lol

 

Is it good practice to do the same for a Parity Drive? Or that one doesn't really matter as nothing is really written to it? 

 

Also, it sounds like the HDD is getting a real work out during the Zeroing phase. Is this normal? I assume it is??

Edited August 18, 20196 yr by z0ki

4 minutes ago, z0ki said:

Is it good practice to do the same for a Parity Drive? Or that one doesn't really matter as nothing is really written to it? 

With parity protected data everything will involve writes to parity.

7 minutes ago, z0ki said:

Also, it sounds like the HDD is getting a real work out during the Zeroing phase. Is this normal? I assume it is??

Yes. It’s why some people use this plugin for stress testing new drives.

On 8/17/2019 at 11:36 AM, AgentXXL said:

if by chance there's errors or a failure of the drive while unRAID is zeroing the drive, does it still add it to the pool?

I'm fairly sure it will bail out and allow you to remove it. I can't remember where I saw that, and I haven't personally tested it. Fairly easy to test if you wanted to risk invalidating parity for the sake of demonstration. Add a not precleared drive in a hot swap bay, let the clear process start, turn off the power to that specific bay and see what happens.

28 minutes ago, jonathanm said:

I'm fairly sure it will bail out and allow you to remove it. I can't remember where I saw that, and I haven't personally tested it. Fairly easy to test if you wanted to risk invalidating parity for the sake of demonstration. Add a not precleared drive in a hot swap bay, let the clear process start, turn off the power to that specific bay and see what happens.

Shouldn't invalidate parity if you remove a partially cleared disk. Parity will be unaffected by the clearing disk contents until it is finished clearing (the whole point of clearing) and gets formatted. Don't know if New Config would be required to remove it or not.

22 minutes ago, jonathanm said:

I'm fairly sure it will bail out and allow you to remove it. I can't remember where I saw that, and I haven't personally tested it. Fairly easy to test if you wanted to risk invalidating parity for the sake of demonstration. Add a not precleared drive in a hot swap bay, let the clear process start, turn off the power to that specific bay and see what happens.

If it does stop addition of the new drive, then that's great. For now I'll continue to use the plugin for the basic stress test on new drives, and for a simple zero pass for older but still functional drives. I'm potentially going to build a 2nd unRAID server to keep my media server isolated from my personal/work/backup data. I know I could just make a separate share with different credentials/access, but a 2nd unRAID box that I keep at a friends place would give me an offsite backup.

 

1 hour ago, AgentXXL said:

simple zero pass for older but still functional drives

TBH, I don't use the plugin.

 

But, IIRC a zero pass does nothing but write the signature to the drive.  Seeing as how at that point there is no guarantee that the drive isn't actually cleared, then if you add a drive to an array that all you've done is write the signature to it then by definition you have invalidated parity.

34 minutes ago, Squid said:

TBH, I don't use the plugin.

 

But, IIRC a zero pass does nothing but write the signature to the drive.  Seeing as how at that point there is no guarantee that the drive isn't actually cleared, then if you add a drive to an array that all you've done is write the signature to it then by definition you have invalidated parity.

A zero only pass of the plugin still takes the same amount of time as the pre-read and post-read stages would, so it's not just writing a signature... it's actually writing zeroes to every sector of the drive before it writes the signature. I agree that if all it did was write a signature, which would take less than a minute, you're never going to be sure.

Ok
I thought there was also an option to simply write the sig. But I'm the first to admit that I'm ignorant (even without my wife's prodding)

Ashamed to admit that I no longer live up to my teenage ethos of "it's better to burn out than fade away"

3 minutes ago, Squid said:

I thought there was also an option to simply write the sig.

I hope nobody would set such a thing loose in the world.